Friday, July 30, 2010

The Art of SOA Testing


Testing an SOA system is more art than science. It's more like trying to untangle the many strands in a bowl of spaghetti that checking  for faulty auto parts in a production line. It follows that if you have a system that’s designed with circular interdependencies and referential loopbacks—as most sufficiently sophisticated SOA system are—classical testing methodologies that evaluate simple cause-effect relationships will simply not work.
What’s needed is a completely new approach; one that will take into consideration the dynamics of a complex SOA system. This approach should encompass comprehensive testing in each of these testing dimensions:
·         Functional Testing. The validation of functionality as perceived by the end-user.

·         Sanity Testing. Ensuring that the system will behave as expected when faced with corrupt data or out-of-bound conditions.

·         Performance Testing. The system is tested under various degrees of traffic loads; stressing it beyond its point of rupture to determine how the system will behave.

·         Availability/Recovery Testing. What happens if you unplug that cable or turn off that server? How well does the system recover from each scenario?
So, you may ask, “How is this different from traditional quality assurance?”  SOA testing does not remove the need to do testing along the traditional testing stages (Unit, Component, Integration and System testing), but it does change the emphasis placed on each.  When testing traditional systems, each component is unit tested to ensure that it does what it is supposed to do,  but there is an assumption that it is the integration testing phase that will ultimately reveal any missed functionality or bug in that component. This is natural since, in non-SOA systems, there is no real way to know if each component is truly performing according to specification prior to the integration. In non-SOA systems individual components can’t be seen as fully-functioning standalone units of work and holistic performance and robustness testing of individual subroutines or libraries is not really feasible.
Things are different with SOA. In SOA, unit testing becomes much more important. After all, if SOA excels at anything it is with the ease with which you are able to validate the sanity of any given service. With SOA, you can actually plan to test each service as if it were a complete deliverable on its own. That is, each service can be tested for performance, recoverability and sanity.

 With SOA, performing a sanity-check of the system, without first ensuring that each service is in compliance with all traditional testing benchmarks, would be an exercise in futility. If the integrity of the parts is not proved, you can’t expect to ensure the integrity of the whole. Each service should be completely tested for its ability to deliver the functionality it provides, its behavior when exposed to spurious requests, whether the request is garbled or contains request variables exceeding the valid data boundaries, and the efficiency of the service under all varied conditions.    
Ideally, component testing should be performed by an independent testing authority—the Service QA team—in order to validate that the programmer has actually delivered the component. This means that, in addition to the “unit testing” typically performed by the developer of the service, the Service QA team assumes responsibility for performing a “Service Test” that follows the same standards that would be followed on a complete delivery. That is, a service should not become part of the system unless it has first been certified by the QA team to be a part of the standard service development life-cycle.
Initially, the focus of this type of service testing of will be Functional and Sanity testing (including the testing of the agreed service interfaces), but as the testing process progresses, you will also need to do performance and recovering testing on a per-component basis. Having this level of testing granularity also has the additional side effect of allowing you to keep track of programmers’ performance.
Knowledge that the services have been well tested prior to moving to system level testing will greatly simplify the integration testing. In SOA testing, integration and system testing ought to be more about testing core service inter-dependencies, validating the overall system performance and dynamics, and testing the system capacity and scalability metrics.
If you see a thread here, it is that you should apply all testing cycles beginning with the service level. Think of testing as a fractal methodology whereby the various testing harnesses that are ultimately applied to the entire system are first tried on each granular service and then applied to broader and broader service conglomerates.  Testing with SOA is a true bottom-up exercise where your key testing focus shifts from functional validity (at the service level) to a comprehensive system-level validation as you move up into coarser integration testing. Still, you should allow for the reality that testing at any given level will reveal defects and the need to correct them. SOA testing is not only a fractal process but an iterative one as well (see diagram).  This testing approach aligns very well with rapid application development methodologies.
The concept of fractal testing demands the development of test scripts and test simulators that will enable the successive testing of services, components (represented by interoperating services), sub-systems (represented by an aggregation of services delivering well-defined business functionality), and finally system-level testing.
Testing thus becomes a recursive exercise that validates the viability of the system on a step-by-step basis. The functional testing, which includes the validation of the end-user functionality, including usability testing, can take place in parallel to the fractal testing exercise, but it is one that also requires a different focus. Since you will need to develop a comprehensive functional testing plan, you would do well to define the overall test patterns that you will apply to each gradient.  It is essential that you define all known use-cases and give special attention to the more obscure cases. Programmers tend to test the most typical scenarios, and for the most part you’ll rarely find bugs in these cases. The issues tend to appear when trying rarer use case situations or unusual user interactions. In fact, you should make it a point to first test these rare examples and then work your way up from the rarer to more typical situations.
Finally, SOA testing forces the introduction of suitable testing and system monitoring components that are not available for traditional non-SOA systems.  The loosely-coupled nature of SOA systems require they exercise all available tools for network monitoring, service logging and so forth. In the end, SOA testing is basically an exercise on system testing, even when simply checking the individual services. Because you can’t expect to have all cooperating services available at once, you will need to allocate some project time to the development of service scaffolds to support the earlier testing.
Remember, SOA testing is like dealing with Medusa—the mythical female Greek entity with hair made of serpents. Your job is to be the hero Perseus and confront the very scary challenger, not by cutting off her head, but by giving her a great new hairdo.

Friday, July 23, 2010

On Documentation


One of my biggest professional upsets took place when, as a young programmer, I lost weeks-worth of documentation I had diligently created for a bank settlement system I had just finished coding.  I had entered the documentation via a vanilla text editor (those were the days before modern word processors!) and, sadly, I had neglected to follow proper backup procedures. I had no versioning or backups as I was always saving my file under the same name (I was young, remember!).  A fateful evening, just as I prepared to leave work, I pressed the Return key to save my nearly finished document back to disk. Literally, at the precise moment that the disk’s red light began to blink, indicating the save to disk was taking place, the office experienced a brief electrical brownout.  It was an outage that left me with a corrupt file, my extensive documentation gone, and with fits of cries and cursing.  
Other work priorities and accelerating deadlines prevented me from redoing the document with the gusto and detail I had invested in it originally. Upon delivery of the system, I went with a quick cheat-sheet that summarized the workings of the system and then I verbally explained to the users the key features and manner of use of my application. In the end, the customer was so satisfied than it dawned on me that the long lost documentation was never even really been needed!
Still, internally I was disappointed. I had a sense of missing the complete deliverable and even to this day I imagine how cool and grandiose my eighty page documentation manual would have looked sitting next to the eight and a half inch floppies containing my code. In time, though, I became grateful for this twist of fate. As I proceeded to improve upon and change the system, I realized that that original documentation stack would have been extremely cumbersome to maintain, and that with each successive new release would have become promptly outdated. In the end, the system worked so well and lasted so long that it never did require those eighty missing pages of documentation.
Keep in mind that my experience preceded those Apple Mac commercials from the mid eighties bragging about how skimpy their manual was compared to the PC’s.  I was an accidental pioneer in the school of thought that says ‘when it comes to documenting, less is more’. This view on documentation is now common. Nowadays, getting software with documentation is as rare as getting more than one bag of peanuts on a commercial flight.
The trend towards less documentation is only viable as long as you define and design the system with usability in mind.  Creating a complex and cumbersome system and then trying to deliver it with no documentation is a sure ticket to perennial support hell. Fact is that some documentation is always needed, but even when documentation is required you can now avoid paper manuals. That era is long gone. Focus instead on creating electronic files using modern collaboration systems.  The best systems are self-documenting in each of their layers. When it comes to source code, for example, ensure that programmers follow best coding practices and the automated documentation facilities provided by the programming language you’ve decided to use. The wealth of material created during the analysis of requirements phase, as well as in day to day communications (emails, file transfers, etc), regarding definition of features and functionality can also be seen as living documentation. Make sure this information is never lost in the limbo of disorganized mailboxes of the team, but is instead properly indexed, catalogued, kept, and propagated via Wikis, so that it is always available via a web portal to the appropriate users.
Ultimately, the online help screens, combined with the various Powerpoint presentations used for presentations to executives and the team plus demos and training materials, will constitute the core documentation for your system. Systems that are easy to use, capable of self-diagnosis and auto-recovery from exceptions, and that provide comprehensive and understandable status information need not rely on bulky external technical documentation. This is particularly true in the realm of SOA; where systems should naturally mirror the business processes they implement.
Beware of project plans with a standalone task for “System Documentation”. Documentation should be an organic element in the design and development process; not a separate task. If anything, the company’s business process documentation should serve as a suitable reference. Ultimately, the best documentation is an intrinsic part of the deliverable; not a fat manual collecting dust in a corner cabinet.

Friday, July 16, 2010

On Tools & Methodologies


In one of Pablo Picasso’s most famous quotes he says that it took him four years to paint like Raphael, but a lifetime to learn how to paint like a child. Indeed, a sign of expertise is mastering one’s field to the extent that there’s no longer a need to abide by “traditional” rules. Great songwriters like Jon Mitchell, Bridget St John, and Nick Drake (of “Pink Moon” fame) tuned their guitars in an unorthodox manner; inventing new chords which helped them create uniquely melodious sounds.  
Once during a job interview, I was asked if I preferred project management tool X over tool Y.  I could not refrain from answering, “The tool is irrelevant! What matters are the people who will be using it!” Perhaps my answer did not endear me to the interviewer as I didn’t land the job, but the point remains: A better tennis racquet won’t make someone a better player (I proved this pseudo-scientifically by purchasing an expensive state-of-the-art beauty made of graphite that didn’t save me from losing game after game). Expensive tools don’t necessarily ensure a project’s success. In fact, blindly applying processes or fixed methodologies can easily be more of a hindrance and a problem than an assist.
Make no mistake, tools and methodologies are needed, but as a colleague of mine once remarked, “Only beginners need to follow every step and use every tool. Experienced people have gained the necessary knowledge to know when to bypass steps.”  Along the same lines, author Phil G. Armour makes this great point in his First Law of Software Process[1]: “A process only allows us to do things we already know how to do”. The corollary to his law is that you cannot have a process for something you have never done or known how to do.  This scenario is especially relevant in the world of SOA as well as in the world of other emerging technologies, where tools and processes are still immature and evolving.
It follows that a large IT transformation project will require the use of repeatable processes in those areas that have been tried before. However, in an IT Transformation the biggest challenge is about creating something new. Processes in this context are not things that you must follow, but instead are things that you will have to create.  Indeed, a key element of your IT transformation deliverables is to define and document the required processes to ensure repeatable success and continued operations of the new system.
How do you know which processes are needed and which aren’t?  The answer is to assess you own experience, along with the experience of your team, and figure out whether there is a need for a process—any process at all. If there is no need for a process, then don’t have one!  You do not need to apologize for this. After all, experience is all about knowing how to separate the essential from the superfluous.  Do you still read your car’s user manual every time you are about to drive? (Did you even read the manual at all?)
If you are dealing with a senior development team you can only enforce so many processes, but if you are managing a team formed of very junior developers, they will actually benefit from the strong direction and clarity the following processes can provide.  Now, let’s not confuse processes with standards. You should definitely have a well-defined set of standards, and your developers will probably want to have them as well. Do, however, keep in mind an oh-so-true observation Mr. Armour makes in his book: “What all software developers really want is a rigorous, iron-clad, hide-bound, concrete, universal, absolute total, definite, and complete set of process rules that they can break”.
You will need to assess how flexible you should be about these rules on a case-by-case basis. Ultimately, the range of tools and methodologies you will need to consider for a complex transformation initiative is rather large. At a minimum you should concentrate on standardizing the use of the following:
·         Project Management Tools
·         Documentation
·         Collaboration
·         Libraries and repositories such as service registries and source code controls
There is some disagreement about whether to enforce standards on tools that are considered individual productivity enhancers. The line here is not always clear. Is a software development environment (otherwise known as IDE) something you should insist upon standardizing across your entire development group? I can assure you, programmers are typically compliant with much of the BS (Big Standards) they have to put up with (filling out those much-hated time reports, going through ridiculously convoluted change management processes, facing draconian purchase order approvals, having to change passwords every nanosecond, and so on.) but I have learned that trying to force them to abandon their preferred IDE tool is likely to set off a revolution of Bolshevik proportions.  If you can identify two or at most three IDEs that everyone can use, then settle for those. You should be prepared, however, to accept that a year down the road, these IDEs will no longer be “the best” and there’ll be a new kid on the block everyone’s dying to use.
Conceding a little on tools driven by personal preferences should not mean that you will give up on enforcing truly important enterprise rules. How you do version control, for example, should be standardized and not left to a single programmer’s decision. Version control is a shared function after all. Focus on putting your foot down on the use of tools that have an enterprise scope and have to be shared by several people, as well as tools that support repeatable processes. But make sure you indentify these preferred tools as quickly as possible during the tenet-definition exercise. Failure to do so could lead to the adoption and rampant propagation of a zoo of ad-hoc tools, making more difficult the alignment of your enterprise efforts. Still, there’s nothing wrong with keeping the pulse on advances of technology so that if a new valuable tool emerges you can evaluate its use in an objective manner and adopt it on a disciplined manner, should you wish to do so.


[1] “The Laws of Software Process.”—Philip G. Armour,  Auerbach Publications. ISBN: 0849314895

Friday, July 9, 2010

Milestone Based Project Management

Proposal for Posadas
The world is at a loss to find a thoroughly satisfactory reason for the persistence of the German in forcing the fighting about Verdun. To the world the German had been defeated in the battle long ago. Two months have passed since a German gain of any importance has been chronicled. In that time there have been many heavy attacks, prepared by artillery fire and delivered by infantry in their final stage, but all have shattered themselves against the wall of the French defense. The German losses have continued to be heavy, indisputably heavier than those of the French, yet the attacks continue. It is no wonder that the neutral world looks on in amazement and asks why." (NY Times. May 14, 1916, Sunday).
The battle of Verdun in 1916 during World War I lasted nearly nine months and resulted in the deaths of about a quarter million soldiers and the wounding of another one million. Then, as tragic as this battle was, another battle, the battle of Somme, designed as a means to draw German forces away from Verdun, resulted in even more casualties than the battle of Verdun (speak of “remediation”!). In the end the Battle of Somme resulted in 1.5 million casualties (around 400,000 killed).  In the entire battle that took place from July to September of 1916, the British and French managed to perform an “impressive” drive of 8 Km (5 miles) from the original battle position.
While these battles remain the paragon of human stubbornness and stupidity, many large IT projects eventually face their own version of the Verdun battle (all human considerations aside!).  One can imagine the Verdun battle being run with one of those Task-Oriented project management tools: “Delivery of supplies? Check!” “Trench digging? Check!” Transfer another division of 5,000 men to the front to serve a cannon fodder? Check!”  
From a task-oriented project management perspective everything was probably on track. Now, imagine if instead, they had established a timetable for quantifiable milestones “By week five we should have advanced two miles”. Perhaps that approach would have served the Generals better in understanding their project was headed for failure.
Milestone based project management can also maintain the sanity of remediation efforts. What happened in Verdun is also typical of many failing projects: a sub-system falters and the management team expends resources and time in an effort to save it. When this fails, more and more resources are applied, until eventually the entire project begins to be measured by the bad press of this particular failure. This type of behavior has been well studied under the guise of the Gambler’s paradox (the more one loses, the more one continues to gamble in the hope of recouping losses, resulting in even greater losses).
If the project is managed by appropriately partitioning it into discrete modules and deliverables, and with quantifiable milestone targets, then things need not be so dire. Also, it’s important to understand that a transformation project is typically so complex that it is unrealistic to expect all of its components to immediately work as expected.  Good project management entails tracking down and detecting when a sub-project is being derailed and acting upon this issue with a clear understanding of the problem.  This means approaching the management of the project via well-defined milestones. Some of the milestones may be there to serve as a kind of “canary in the mine”—something like the delivery of a simple demo to test nothing is rotten in the Kingdom of Projectmark. The key is to understand whether the component in trouble is critical to the overall success of the project, or whether an appropriate do-over is in order.
Needless to say, the key to tracking and assessing the status and impacts of the project dynamics is the Project Management function. If the methodologies and lifecycle of project management in the context of IT projects are not clearly defined, this area quickly tends to become a hotspot of friction. Why? Because of a mistaken idea that anyone with working knowledge of MS/Project can become an IT project manager. A major transformation project is not the right place for copying the style of shows like “The Apprentice”, where contestants (usually celebrities) are given project management roles as a part of their challenge. In this show it does not seem to matter whether the contestant is “project managing” a challenge outside his or her area of expertise. 

In the world of IT, delivery managers with systems and software background have traditionally been more successful in the project management role. The role of generalist project managers in IT projects is better left for peripheral or support functions (managing a training track, for instance). Moreover, IT development artifacts can be extremely detailed and if micro-managed (something very easily done with task-oriented project management), can cause situations where you are not able to see the forest for the trees.  Traditional Gantt-chart management with day-to-day track assessment of projects via red-yellow-green color codes may look good on status reports, but does little to ascertain the true status of a project.
Secondly, there are projects and there are projects. . .  Smaller projects need to be handled in a way that assures the necessary ingredients for success are available to the team leader with a minimum of red tape. Smaller programs can benefit greatly from rapid application development methodologies and from early prototyping. Just remember that Agile methodologies should only apply to the actual software development steps; and not to the architecture and design phases. Using pseudo-Agile approaches under the mantra "Code now, Design Later" often results in failure. 

What defines a small project?  Well, I’d say one where no more than five people working on a deliverable for a maximum of four months is as good a metric as any.  We are also talking about something that will not cost over one million dollars including labor, hardware, and software capital costs.
Then there are the medium size projects. These are projects that can take up to a year, or even a bit longer if the executives in your organization are prepared to go on Prozac and give you more leeway. The core development teams for this type of project should never exceed the magic number of ten. These projects tend to fall in the ball-park figure of around three million dollars. A medium size project needs to be managed with a savvy combination of high-level project management controls and the appropriate placement of deliverable milestones.
Larger projects should basically not even exist. Why not? Well, a large project should be suitably broken into as many small and medium sized projects as possible. Ultimately, a large project should only exist in the talking points of the individuals responsible for your company’s public relations, and in the radar of the very small team responsible for integrating all the various moving parts.
So what is Milestone Based project management all about? Basically, it’s the definition and tracking of measurable, well defined milestones. The difference between a milestone and a traditional project management event is the fact that the milestone should stand on its own as a deliverable. Completing a piece of code is not a milestone. Completing a prototype or being able to demonstrate a complete sub-component of the deliverable, are examples of milestones. If the event is something that can be shown to anyone outside the development group, and is considered to be at least a partial success, then it qualifies as a milestone.
Tracking milestones has a number of benefits:
·         Missing a milestone is a clear signal that the project is not on track.
·         A successful milestone motivates the team by providing a clear partial result
·         Milestone deliverables can provide some benefits sooner
·         A successful milestone can motivate the project sponsors to continue their support of  the project, even when faced with budgetary constraints
More importantly, there is no way to sugar-coat a missed milestone. While missing one milestone should not spell gloom-and-doom, a project stringing along two or more missed milestones is a project that needs to be seriously reviewed.
In fact, you should always allow room in a large project to anticipate the failure of a specific component, and to adjust for the overall delivery because of this failure. The solution to the quandary might range from starting again from scratch (assuming the initial implementation was proven wrong), to completely eliminating the subsystem and remediating by providing a suitable minimum set of capabilities from other subsystems.
Milestone driven project management is superior to task-oriented project management in large projects because it focuses on what’s important while simplifying the view from the top.



Friday, July 2, 2010

Interlude Four: Digital Evanescence


An Englishman, an American, and an Italian are sitting in a bar. The Englishman says, “They dug down 30 feet near Bristol and found very old copper telegraph wires.”
 “So?” says the American.
 “This is proof that we, the English, were the first to invent the telegraph.”
“In that case,” the American retorts, “you should know that during the excavation of a new subway line in New York they found rusted telephone wiring. It definitely proves that the telephone was invented in the good ol’ US of A!”
The Italian who up to this point had been quietly drinking his Cinzano retorts, “That’s nothing! We Italians dug a hole 40 feet deep next to the Tower of Pisa, and guess what we found. . .?”  He pauses, smiling mysteriously.
“OK. Do illuminate us.” Says the Englishman after the continued silence suggested the Italian’s question was not meant to be rhetorical.
“We found nooothing!” answers the Italian in a very excited tone.
After a shocked moment of silence, the American asks, “So?  What does that prove?”
Beaming with pride, the Italian says, “It proves that we Italians were the first to invent the wireless phone!”
Technology is a funny thing. The more pervasive and embedded it is in the fabric of our society, the more invisible it becomes.  This is actually a nice feature (I can envision a future where cell phones and computers will be embedded in our bodies), but the problem with invisibility is that it lacks solidity; it is evanescent.
It appears that writing developed about five thousand years ago when, according to Wikipedia, a set of discovered cuneiform Sumerian tablets represented the oldest known record of man’s writing. Predictably, most of the deciphered Sumerian tablets deal with business transactions such as payments for goods, accounting records and the like (religious writings including the Old Testament, the Indian Vedas, and the Pyramid Egyptian texts came later).  Obviously, most of what we know about the history of humanity comes from the ever increasing access we have to written history, whether the material is from scrolls, stones, or books. Even if much knowledge has been lost as a result of the destruction of 40,000 books after the Library of Alexandria was set on fire, or as a result of the Spaniards’ destruction of thousands of Mayan and Aztec scrolls, we are fortunate that some written historical records have been preserved.  The analog nature of our written legacy enables the archeologists to quickly recognize written text in any language for what it is, even if they are unable to immediately decipher their findings. 
Now think forward to a time in the future, say one thousand years from now. Will future archeologists be able to recognize the remnants of a CD or USB stick as something containing valuable information from our time? Taking into account the difficulty involved in translating the cuneiform of the Sumerian Tablets, how difficult will it be in the future to access the contents of an iPad tablet?
You may think that my concern is unfounded, but I still have a couple of old 8 ¼ inch floppy disks that contain information from some of my earlier computer jobs.  I should also mention the stack of punch cards I keep somewhere in my basement! I know the cards contain code I’ve written years ago (assembler) because I can visually decode the old EBCDIC encoding in the perforations, but who knows what I have stored on those floppies?
My point is this: Can we guarantee that in the future humans will be able to access and decode the information we’re storing so casually in digital form?  Digital information is intrinsically evanescent and unrecognizable. And nowadays, digital information can be anything: pictures, sounds, books.
 In my closet I have a compendium of ten or so photograph albums that contain photos from my childhood to. . .  er, a period in the mid nineties. At that time, I began to use a digital camera I had won in an expo raffle (business card on the bowl, it happens!). Problem is that that primitive camera captured the pictures in a format that I can no longer decipher and so I have no way of viewing those photos.  Anyway, this is probably not a great loss—the picture resolution back then was less than one megapixel.  To be fair, if you take a look at your old paper photos, you will probably notice that many prints are now suffering from discoloration and are fading under the weight of the years. Keeping stuff on paper is no assurance of permanence.
Naturally, today I have gigabytes of snapshots taken over the last ten years, images I never took the trouble to print and which will never be placed in an actual physical photo album. Now, digitally storing your pictures is both a blessing and a curse. Photoshop’s miraculous ability to remove wrinkles aside, the blessing is that, provided you judiciously back up your data, you will be able to keep and share your pictures indefinitely. The curse is that it is now so easy to take thousands upon thousands of pictures that it’s become more cumbersome to actually enjoy them. (This is similar to the Video-camera phenomena of the eighties when thousands of hours of video recording of your children running and jumping around mindlessly couldn’t be realistically watched.  Not enough time or patience.)
But I digress . . .  the main problem with digitally stored information such as pictures is that it can so easily become unreadable due to changes in technology or to its disappearance.  True, Internet based photographic libraries like Picassa, Webshots, and others can store tons of family albums, but do so under specific commercial restrictions. This is especially true if you, like most, use those services under the “free” subscription model.
Digital information stored on old media may become inaccessible forever and, in time, our current media will be superseded by new technology as well. I would suggest that what’s needed is an organization, international like UNESCO, or domestic like the US Library of Congress, to take responsibility for the storage and permanent accessibility of all the digital content being created. This would mean preserving the detailed knowledge of encoding standards used across all times and media; along with the preservation of equipment capable of quickly accessing this content.
This content is, after all, not unlike the Egyptian papyruses of yore, for it will likely be judged as the patrimony of humanity in the years to come.