Software Quality Testing

Easier to staff a team. We do not need software developers or other experts to perform these tests (note: expertise is required to identify which tests to run, etc). Manual testers are also easier to find at lower rates than developers - presenting an opportunity to save money, or test more, or both.

cons
Higher maintenance cost with automated testing. Application changes tend to break black-box tests, because of their reliance on the constancy of the interface.

Redundancy of tests. Without insight into the implementation, the same code paths can get tested repeatedly, while others are not tested at all.

White box software testing:

pros
More efficient automated testing. Unit tests can be defined that isolate particular areas of the code, and they can be tested independently. This enables faster test suite processing

More efficient debugging of problems. When a regression error is introduced during development, the source of the error can be more efficiently found - the tests that identify an error are closely related (or directly tied) to the troublesome code. This reduces the effort required to find the bug.

A key component of TDD. Test driven development (an Agile practice) depends upon the creation of tests during the development process - implicitly dependent upon knowledge of the implementation. Unit tests are also a critical element for continuous integration.

cons
Harder to use to validate requirements. White box tests incorporate (and often focus on) how something is implemented, not why it is implemented. Since product requirements express full system outputs, black box tests are better suited to validating requirements. Carefull white box tests can be designed to test requirements.

Hard to catch misinterpretation of requirements. Developers read the requirements. They also design the tests. If they implement the wrong idea in the code because the requirement is ambiguous, the white box test will also check for the wrong thing. Specifically, the developers risk testing that the wrong requirement is properly implemented.

Hard to test unpredictable behavior. Users will do the strangest things. If they are not anticipated, a white box test will not catch them. I recently saw this with a client, where a bug only showed up if the user visited all of the pages in an application (effectively caching them) before going back to the first screen to enter values in the controls.

Requires more expertise and training. Before someone can run tests that utilize knowledge of the implementation, that person needs to learn about how the software is implemented.

Which testing approach should we use?
There is also the concept of gray box testing, or layered testing - using both black box and white box techniques to balance the pros and cons for a project. We have seen this approach work very effectively for larger teams. Developers utilize white box tests to prevent submission of bugs to a testing team that uses black box tests to validate that requirements have been met (and to perform system level testing). This approach also allows for a mixture of manual and automated testing. Any continuous integration strategy should utilize both forms of testing.