Friday, July 30, 2010

The Art of SOA Testing

Testing an SOA system is more art than science. It's more like trying to untangle the many strands in a bowl of spaghetti that checking  for faulty auto parts in a production line. It follows that if you have a system that’s designed with circular interdependencies and referential loopbacks—as most sufficiently sophisticated SOA system are—classical testing methodologies that evaluate simple cause-effect relationships will simply not work.
What’s needed is a completely new approach; one that will take into consideration the dynamics of a complex SOA system. This approach should encompass comprehensive testing in each of these testing dimensions:
·         Functional Testing. The validation of functionality as perceived by the end-user.

·         Sanity Testing. Ensuring that the system will behave as expected when faced with corrupt data or out-of-bound conditions.

·         Performance Testing. The system is tested under various degrees of traffic loads; stressing it beyond its point of rupture to determine how the system will behave.

·         Availability/Recovery Testing. What happens if you unplug that cable or turn off that server? How well does the system recover from each scenario?
So, you may ask, “How is this different from traditional quality assurance?”  SOA testing does not remove the need to do testing along the traditional testing stages (Unit, Component, Integration and System testing), but it does change the emphasis placed on each.  When testing traditional systems, each component is unit tested to ensure that it does what it is supposed to do,  but there is an assumption that it is the integration testing phase that will ultimately reveal any missed functionality or bug in that component. This is natural since, in non-SOA systems, there is no real way to know if each component is truly performing according to specification prior to the integration. In non-SOA systems individual components can’t be seen as fully-functioning standalone units of work and holistic performance and robustness testing of individual subroutines or libraries is not really feasible.
Things are different with SOA. In SOA, unit testing becomes much more important. After all, if SOA excels at anything it is with the ease with which you are able to validate the sanity of any given service. With SOA, you can actually plan to test each service as if it were a complete deliverable on its own. That is, each service can be tested for performance, recoverability and sanity.

 With SOA, performing a sanity-check of the system, without first ensuring that each service is in compliance with all traditional testing benchmarks, would be an exercise in futility. If the integrity of the parts is not proved, you can’t expect to ensure the integrity of the whole. Each service should be completely tested for its ability to deliver the functionality it provides, its behavior when exposed to spurious requests, whether the request is garbled or contains request variables exceeding the valid data boundaries, and the efficiency of the service under all varied conditions.    
Ideally, component testing should be performed by an independent testing authority—the Service QA team—in order to validate that the programmer has actually delivered the component. This means that, in addition to the “unit testing” typically performed by the developer of the service, the Service QA team assumes responsibility for performing a “Service Test” that follows the same standards that would be followed on a complete delivery. That is, a service should not become part of the system unless it has first been certified by the QA team to be a part of the standard service development life-cycle.
Initially, the focus of this type of service testing of will be Functional and Sanity testing (including the testing of the agreed service interfaces), but as the testing process progresses, you will also need to do performance and recovering testing on a per-component basis. Having this level of testing granularity also has the additional side effect of allowing you to keep track of programmers’ performance.
Knowledge that the services have been well tested prior to moving to system level testing will greatly simplify the integration testing. In SOA testing, integration and system testing ought to be more about testing core service inter-dependencies, validating the overall system performance and dynamics, and testing the system capacity and scalability metrics.
If you see a thread here, it is that you should apply all testing cycles beginning with the service level. Think of testing as a fractal methodology whereby the various testing harnesses that are ultimately applied to the entire system are first tried on each granular service and then applied to broader and broader service conglomerates.  Testing with SOA is a true bottom-up exercise where your key testing focus shifts from functional validity (at the service level) to a comprehensive system-level validation as you move up into coarser integration testing. Still, you should allow for the reality that testing at any given level will reveal defects and the need to correct them. SOA testing is not only a fractal process but an iterative one as well (see diagram).  This testing approach aligns very well with rapid application development methodologies.
The concept of fractal testing demands the development of test scripts and test simulators that will enable the successive testing of services, components (represented by interoperating services), sub-systems (represented by an aggregation of services delivering well-defined business functionality), and finally system-level testing.
Testing thus becomes a recursive exercise that validates the viability of the system on a step-by-step basis. The functional testing, which includes the validation of the end-user functionality, including usability testing, can take place in parallel to the fractal testing exercise, but it is one that also requires a different focus. Since you will need to develop a comprehensive functional testing plan, you would do well to define the overall test patterns that you will apply to each gradient.  It is essential that you define all known use-cases and give special attention to the more obscure cases. Programmers tend to test the most typical scenarios, and for the most part you’ll rarely find bugs in these cases. The issues tend to appear when trying rarer use case situations or unusual user interactions. In fact, you should make it a point to first test these rare examples and then work your way up from the rarer to more typical situations.
Finally, SOA testing forces the introduction of suitable testing and system monitoring components that are not available for traditional non-SOA systems.  The loosely-coupled nature of SOA systems require they exercise all available tools for network monitoring, service logging and so forth. In the end, SOA testing is basically an exercise on system testing, even when simply checking the individual services. Because you can’t expect to have all cooperating services available at once, you will need to allocate some project time to the development of service scaffolds to support the earlier testing.
Remember, SOA testing is like dealing with Medusa—the mythical female Greek entity with hair made of serpents. Your job is to be the hero Perseus and confront the very scary challenger, not by cutting off her head, but by giving her a great new hairdo.