It’s all about testing…

There can be no proper development without proper validation. So testing is one of the most important tasks during development. And that includes the architecture. Not only must the architect ensure proper testing, (s)he also needs to validate the correct realization of the architecture design. I will get into detail on some of the tasks the architect needs to take care of later. At this point I would just like to point out that test consideration has to start from the beginning of architecture design and must continue all the way throughout development to integration.

Together with a colleague I created a checklist of the architect’s tasks regarding testing. I am using this checklist in my project to formally verify that all those tasks were considered. You can also formalize it in your project, if you like, or just use it as a reminder for yourself to check that you took care of everything. So, without further ado, here’s the checklist:

• Architecture Document(s)
  • The following documents exist:
    • Architecture Specification has been reviewed and released.
    • …
  • The architecture documentation contains the following information:
    • Lists of components of the system.
    • Responsibilities of the components.
    • Interfaces of the components.
    • Dependencies between the components.
    • Observation and control points for testing.
    • Tracing and logging mechanisms are defined.
    • Strategies for fault detection and prevention are considered.
    • Guidelines for development are listed.
    • Test automation is considered.
    • Documented design decisions containing information about testability.
    • A utility tree for the non-functional requirements.
• Risk List for Risk Based Testing
  • The risk list for risk based testing exists:
    • Risk list has been reviewed and released.
  • The risk list contains the following information:
    • The list contains all identified product risks.
    • The risks have a short description, a risk type, a list of affected quality criteria and a list of affected parts of the system.
    • For the risks the probability for the risk to occur, the damage the risk will cause and the test effectiveness are rated on a scale from 1 to 5.
    • The test priority number is calculated as product of the three values.
• Quality Model
  • The quality model exists:
    • Quality model has been reviewed and released.
  • The quality model contains the following information:
    • Quality attributes for functionality, assigned to the following sub-characteristics: suitability, accuracy, interoperability, security, and compliance.
    • Quality attributes for reliability, assigned to the following sub-characteristics: maturity, fault tolerance, and recoverability.
    • Quality attributes for usability, assigned to the following sub-characteristics: understandability, learnability, operability, and attractiveness.
    • Quality attributes for efficiency, assigned to the following sub-characteristics: time behaviour, and resource utilisation.
    • Quality attributes for maintainability, assigned to the following sub-characteristics: analyzability, changeability, stability, and testability.
    • Quality attributes for portability, assigned to the following sub-characteristics: adaptability, installability, co-existence, replaceability, and conformance.
    • For each attribute it is described why it shall be tested.
    • For each attribute it is described what the test criteria are.
    • For each attribute it is described how the test shall be measured.
    • For each attribute it is described how the results shall be interpreted.
• Component Design Specifications
  • The component design specifications exist:
    • Design Specification Component A has been reviewed and released.
    • …
  • The design documentation contains the following information:
    • Lists of sub-components of the component.
    • Responsibilities of the sub-components.
    • Interfaces of the sub-components.
    • Dependencies between the sub-components.
    • Observation and control points for testing.
    • Tracing and logging mechanisms are defined.
    • Strategies for fault detection and prevention are considered.
    • Guidelines for development are listed.
    • Test automation is considered.
    • Design decisions are documented and contain information about testability.
• Code
  The code quality is checked during the code reviews of the components:
  • Code reviews have been performed.
• Unit Test
  The unit tests of the components are performed before system integration:
  • Unit tests have been performed.
• System Integration
  • The system integration plan exists:
    • Integration plan has been reviewed and released.
  • The system integration plan contains the following information:
    • Sequence of integration steps.
    • Details of integration step content.
• System Integration Test
  • The system test plan exists:
    • System test plan has been reviewed and released.
  • The system test plan contains the following information:
    • Entry criteria for system test.
    • System integration test specifications.
• Test Scenarios
  Test scenarios exist for the quality attributes.
  • Quality Attribute 1
    The test scenario is considered:
    • The test scenario is covered by test cases.
  • …