Software Testing Notes

     I think in many ways the more recent revolution in how software is created comes not from programming languages or code organization but from how testing and quality assurance are accomplished. We learned a decade or so ago how an object-oriented approach helps in producing good software. To go further a testing structure is neccessary.

     Many companies only have "black-box" testing done by non-programmers who are part of a "Quality Assurance" or "Quality Engineering" department, which is separated from the department which programmers are in. Their testing consists of using the program as an end-user would, while carefully tracking the results.

     There are also various forms of automated "black-box" testing (i.e. external programs which simulate the actions of a user) as well as "white-box" testing which contacts internal pieces of the program not normally accessible to the end user.


Unit Testing

     Unit testing is the practice of writing small automated test for the functionality of self-contained units within the code -- generally a single class or small set of classes.

     For example, an Enterprise Java Bean which accesses a database might have a unit test which checks to make sure that basic get and put functions work.

     JUnit is a set of Java classes that aid in unit testing. However, formal tools are not required. A simple convention is to include a main method for a Java class, which instantiates and tests some simple functionality of that class. Additional information can be found in the following articles:


Automated Development Tests

     A set of tests should be bundled together as a minimum baseline for interim development versions. Ideally, this should be run as a single step after compiling, as a simple make test or ant test command.

     This can be as little as a handful of the unit tests discussed above. As a project approaches the point of hand-off to the QA department, the tests should probably be more extensive and establish at least a few of the minimum standards needed for hand-off (as described in the QA guidelines).

     Web projects should develop simple automated testing as well for link testing and so forth. General use tools (such as a linkbot) should be made available within the company.


Logging

     Logging is a neccessary process for debugging, testing, and maintenance. Like unit tests, log and info messages should be seen as an integral part of the product. They should not be considered as temporary code hacks to be thrown away later.

     For any large-scale projects (and many others), logging procedure should be multi-level and runtime-configurable. These mean:

     Logging with these features is a major step over the common practice of the programmer adding in "print" lines and then commenting them out when not needed. The most important change is that the programmer should look at log statements as a permanent part of the program rather than temporary hacks. This greatly aids the debugging process as well as use by external administrators.

     These features can be implemented by custom code or by using an existing logging package. For Java, the open-source project log4j is an example of a package which supports these features with minimal impact on processing speed.


Bug Tracking

     Once a software product is in the pipeline towards being made public, it frequently goes into a formal process where specific bugs are numbered and tracked. One of the pitfalls of such systems is that they can stifle open communication by being to formalized. In developing a product, the programmers need to keep others informed -- especially in communicating an accurate timeline of when fixes will be complete.

     In reality, not all of the intended product requirements are always going to be met. Too often, engineers work under increasing pressure to get requirements completed by the deadline. Desperate to meet all the requirements, they work furiously right up to the deadline, but often what is produced is not their best work. Then, when the product is turned over QA testers, it is full of bugs which cannot be fixed in time for the intended shipping date, thereby resulting in either a missed product launch date or a buggy product.

     The lesson is that programmers need to assess as early as possible which goals are realistic. At that point, either the requirements or the goals need to change, and other teams informed of these decisions.