Software Testing and Quality
Assurance White Papers
TABLE OF CONTENTS .......................................................................................................... 1
SOFTWARE TESTING AND SOFTWARE QUALITY ASSURANCE..................................... 1
QA Services ............................................................................................................................................1
Process and Procedures – Why follow them? ......................................................................................1
Standards and Templates – What is supposed to be in a document?....................................................1
Test Readiness Review – Who is ready for the next phase? ................................................................2
Different Levels of Testing.....................................................................................................................2
Unit Testing.........................................................................................................................................2
Parallel/Audit Testing..........................................................................................................................2
Functional Testing...............................................................................................................................2
Usability Testing .................................................................................................................................3
Incremental Integration Testing ...........................................................................................................3
Integration Testing ...............................................................................................................................3
System Testing ....................................................................................................................................3
End-to-End Testing ..............................................................................................................................3
Regression Testing ...............................................................................................................................3
Sanity Testing......................................................................................................................................4
Performance Testing............................................................................................................................4
Load Testing........................................................................................................................................4
Installation Testing ...............................................................................................................................4
Security/Penetration Testing ................................................................................................................4
Recovery/Error Testing ........................................................................................................................4
Compatibility Testing..........................................................................................................................4
Comparison Testing .............................................................................................................................4
Acceptance Testing ..............................................................................................................................4
Alpha Testing ......................................................................................................................................5
Beta Testing.........................................................................................................................................5
Resources................................................................................................................................................5
Test/QA Team Lead .............................................................................................................................5
Testers .................................................................................................................................................5
Test Build Manager/System Administrator/Database Administrator...................................................5
Technical Analyst & Test Configuration Manager ..............................................................................5
SOFTWARE TESTING METHODOLOGY .............................................................................. 6
Step 1 - Create Test Strategy.................................................................................................................7
Step 2 - Create Test Plan/Design...........................................................................................................7
Step 3 - Execute Tests............................................................................................................................9
FREQUENTLY ASKED QUESTIONS................................................................................... 11
Why is it often hard for management to get serious about quality assurance?..............................11
Why does software have bugs?............................................................................................................11
How can new Software QA processes be introduced in an existing organization? ........................12
What is verification? validation? ........................................................................................................12
What is a 'walkthrough'?....................................................................................................................12
What's an 'inspection'? ........................................................................................................................12
What are five common problems in the software development process?........................................13
What are five common solutions to software development problems? ...........................................13
What is software 'quality'? ..................................................................................................................13
What is 'good code'?............................................................................................................................14
What is 'good design'? .........................................................................................................................14
What is the 'software life cycle'?.........................................................................................................14
Will automated testing tools make testing easier?.............................................................................14
What makes a good test engineer?......................................................................................................15
What makes a good Software QA engineer?......................................................................................15
What makes a good QA or Test manager? ........................................................................................15
What's the role of documentation in QA?..........................................................................................15
What's the big deal about 'requirements'? ........................................................................................16
What's a 'test plan'? .............................................................................................................................16
What's a 'test case'? .............................................................................................................................16
What should be done after a bug is found?........................................................................................17
What is configuration management (CM)?........................................................................................17
What if the software is so buggy it can't really be tested at all? ......................................................17
How can it be known when to stop testing? .......................................................................................17
What if there isn't enough time for thorough testing?......................................................................18
What if the project isn't big enough to justify extensive testing?.....................................................18
What can be done if requirements are changing continuously? ......................................................19
What if the application has functionality that wasn't in the requirements?...................................19
How can Software QA processes be implemented without stifling productivity?..........................20
What if organization is growing fast that fixed QA processes are impossible? ..............................20
How does a client/server environment affect testing?.......................................................................20
How can Web sites be tested?..............................................................................................................21
How is testing affected by object-oriented designs? ..........................................................................22
Why is it recommended to test during the design phase?.................................................................22
What are the five approaches to integration testing? .......................................................................23
What are the three software development process models?.............................................................23
GLOSSARY OF TERMS ....................................................................................................... 24
Pointe Technology Group, Inc
Software Testing and Quality Assurance White Papers
Software Testing and Software Quality Assurance
Software Quality Assurance (QA) at Pointe Technology is oriented to ‘prevention’. It involves
the entire software development process. Process is monitoring and improving the process,
making sure that any agreed-upon standards and procedures are followed, and ensuring that
problems are found and dealt with.
Software Testing at Pointe Technology is oriented to ‘detection’. Testing involves the
operation of a system or application under controlled conditions and evaluating the results.
Organizations vary considerably in how they assign responsibility for QA and testing.
Sometimes they are the combined responsibility of one group or individual. Also common are
project teams that include a mix of testers and developers who work closely together, with
overall QA processes monitored by project managers. It will depend on what best fits an
organization's size and business structure.
Pointe can provide QA and/or Software Testing services. This document details some
aspects of how we can provide quality and web site testing services. For more information
please contact one of the Pointe Players listed in the last section of this document.
QA Services
QA ensures that all parties concerned with the project adhere to the process and procedures,
standards and templates, and test readiness reviews.
Our QA service depends on the customers and projects. A lot will depend on team leads or
managers, feedback to developers, and ensuring adequate communications among
customers, managers, developers, and testers.
Process and Procedures – Why follow them?
Detailed and well-written processes and procedures ensure that the correct steps are being
executed to facilitate a successful completion of a task. They also ensure that a process is
repeatable.
Once Pointe Technology has learned and reviewed your processes and procedures we will
recommend improvements and/or additions. Once you approved and signed off, CM will
enter them into the version control tool. Then a plan will be devised to 1) document, 2)
implement, and 3) monitor these processes and procedures.
Processes and procedures will be reviewed and updated periodically by QA and the process
owner to ensure that they are current and relevant.
Standards and Templates – What is supposed to be in a document?
All documents should be written to a certain standard and template. This gives documents
uniformity. It also helps in learning where information is located making it easier for a user to
find what they want. Lastly, with standards and templates information will not be accidentally
omitted from a document.
Once the Pointe Technology has learned and reviewed your standards and templates we will
recommend improvements and/or additions. Once they are approved and signed off by you,
CM will enter them into the version control tool. The standards and templates will be
enforced by QA during: all levels of testing.
Standards and templates will be reviewed and updated periodically by the process owners to
ensure that they are current and relevant.
Test Readiness Review – Who is ready for the next phase?
The test readiness review ensures that all artifacts are complete and correct & information is
being passed on to the next group. The group “passing off” must follow an Exit Criteria. The
group “receiving “ must follow an Entrance Criteria. The QA person ensures that all parties
are prepared and that the process is correctly performed.
Pointe can suggest an Entrance and Exit Criteria for review, approval, and eventually sign
off. Once Pointe and the customer have agreed upon the Entrance and Exit Criteria CM will
enter it into the version control tool. QA will then enforce these criteria during every Test
Readiness Review.
Both Entrance and Exit Criteria will be reviewed and updated periodically by QA and the
process owner to ensure that they are current and relevant.
Different Levels of Testing
Pointe has expertise in testing at all of the below testing levels. At each test level we will
document the results. It is recommended to get sign-off and check in all documentation and
code according to Configuration Management (CM) procedures to ensure quality testing.
Each level of testing is either considered black or white box testing.
• Black box testing: not based on any knowledge of internal design or code.
Tests are based on requirements and functionality.
• White box testing: based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, and conditions.
Unit Testing
Unit Testing is the first level of dynamic testing and is first the responsibility of the developers
than the testers. After the expected test results are met or differences are explainable/acceptable.
Parallel/Audit Testing
Testing where the user reconciles the output of the new system to the output of the current
system to verify the new system does the operations correctly.
Functional Testing
Black-box type of testing geared to functional requirements of an application. Testers should
do this type of testing.
Usability Testing
Testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted enduser
or customer. User interviews, surveys, video recording of user sessions, and other
techniques can be used. Programmers and testers are usually not appropriate as usability
testers.
Incremental Integration Testing
It is recommended that continuous testing of an application as new functionality is added.
This may require that various aspects of an application's functionality be independent enough
to work separately before all parts of the program are completed, or that test drivers be
developed as needed; done by programmers or by testers.
Integration Testing
Upon completion of unit testing, integration testing, which is black box testing, will begin. The
purpose is to ensure that distinct components of the application still work in accordance to
customer requirements. Test sets will be developed with the express purpose of exercising
the interfaces between the components. This activity is to be carried out by the Test Team.
Integration test will be termed complete when actual results and expected results are either in
line or differences are explainable/acceptable based on client input.
System Testing
Upon completion of integration testing, the Test Team will begin system testing. During
system testing, which is a black box test, the complete system is configured in a controlled
environment to validate its accuracy and completeness in performing the functions as
designed. The system test will simulate production in that it will occur in the “production-like”
test environment and test all of the functions of the system that will be required in production.
The Test Team will complete the system test.
Prior to the system test, the unit and integration test results will be reviewed by SQA to
ensure that all problems have been resolved. It is important for higher level testing efforts to
understand unresolved problems from the lower testing levels. System test is deemed
complete when actual results and expected results are either in line or differences are
explainable/acceptable based on client input.
End-to-End Testing
Similar to system testing; the 'macro' end of the test scale; involves testing of a complete
application environment in a situation that mimics real-world use, such as interacting with a
database, using network communications, or interacting with other hardware, applications, or
systems if appropriate.
Regression Testing
The objective of regression testing is to ensure that software remains intact. A baseline set
of data and scripts will be maintained and executed to verify that changes introduced during
the release have not “undone” any previous code. Expected results from the baseline are
compared to results of the software being regression tested. All discrepancies will be
highlighted and accounted for before testing proceeds to the next level.
Sanity Testing
Sanity testing will be performed whenever cursory testing is sufficient to prove that the
application is functioning according to specifications. This level of testing is a subset of
regression testing. It will normally include a set of core tests of basic GUI functionality to
demonstrate connectivity to the database, application servers, printers, etc.
Performance Testing
Although performance testing is described as a part of system test, it can be regarded as a
distinct level of testing. Performance testing will verify the load, volume, and response times
as defined by requirements.
Load Testing
Testing an application under heavy loads, such as testing of a web site under a range of
loads to determine at what point the system's response time degrades or fails.