Friday, November 27, 2009

Interlude Two: Y2K and the Fuzzy Nature of Success

Before the start of the millennium the world was abuzz with concern, as it held its collective breath fearing that the much touted Y2K bug would usher in an end to civilization.  Airplanes could fall from the sky, Nuclear Plants could melt, and financial assets could disappear.
Mind you, the so-called “Millennium Bug” was not actually a bug. Rather, it was a side effect of legacy applications written during a time when the cost to store one byte of data was about ten million times higher than it is today.  Storing only the last two digits of the year did in fact save significant money during those early days of super-expensive disk storage. The real issue was that legacy applications over-extended their welcome, as they are prone to do, and the need to prevent a devastating impact from Y2K became a critical imperative[1]
With a total worldwide estimated expense of $200B, the industry successfully confronted both the hype and the challenge of this problem. In the end, the only explosions seen were from the fireworks of the beautiful millennial ceremonies held around the world. January 1, 2000 came and went with barely a hiccup.
Sadly, there soon emerged the narrative that IT people had duped everyone with their “Y2K fear-mongering” and that the whole effort had been a waste of money. During a CNN interview, a congressman opposing a new budget request stated, “Let’s not have another Y2K project where we spent so much money on it and then . . . nothing happened anyway!”
All this got me thinking about the fuzzy nature of success. For the most part, failure is easy to recognize even if politicians and some Wall Street insiders are masters at concealing it (mortgage derivatives anyone?). Success, on the other hand, is often muffled or belittled.  Even clearly successful endeavors such as the moon landing had its share of detractors: “The project was over-budget”, “Astronauts died in the process”, “The whole mission was a fool’s-errand”, “That money could have been used to solve world-hunger”, “The whole thing was a hoax”, et cetera, et cetera. 
Yes, we frequently hear motivational arguments about how failure can be a good thing, how we can learn from it, and how we should all be allowed to fail, and so on, and so forth.  Truth be told, no one possessing any degree of rationality seeks to fail, and if you fail one too many times, chances are that you will suffer the consequences.  Setting all philosophizing side, I think we can all agree that success is preferable to failure.
But what exactly constitutes success?
A baseball batter is deemed wildly successful when he is able to hit the ball 40% of the time. What is the right percentage for a transformation project? Certainly not 40%.  Neither is 100%. A bigger problem is the expectation of perfection. One of my assigned goals in a previous job was to make sure that all technology worked well all of the time—this on a shoestring budget and limited resources to boot.  Yet, in the real world, a project that succeeds too well in all of its dimensions is either a chimera or something that was not ambitious enough to begin with.
Alas, I have witnessed projects that delivered and even exceeded their key goals, but because they failed to meet one hundred percent of the original expectations, they ended up being perceived as failures. Projects that truly set out to accomplish something must be measured against their key deliverables, even if they might miss a few non-essential objectives.  This is especially true with large and complex transformation initiatives.
As far as that congressman was concerned, the Y2K intervention was a failure because he looked only at the cost involved and did not understand that “nothing bad should happen” was actually the criteria for success. The hundreds of billions of dollars spent to avert the Y2K bug was money well spent, precisely because, in the end, “nothing happened”. The Y2K catastrophe was avoided; plus there’s anecdotal evidence that the Y2K remediation efforts also spear-headed beneficial transformation makeovers.
Many books have been written about failure but not many about success.  In my experience, success (actual or imagined) can take one of the following forms:
·         Bombastic.  When Charles Lindbergh landed in France after his solo trans-Atlantic flight, the feat was celebrated by a million people in Paris and between four and five million people when he paraded in New York City.
·         Celebrated. Naturally this is the kind of success that most of us hope for after a project is completed.
·         Contented.  Not all successes have to be celebrated. In fact, it is rather annoying when someone constantly highlights his or her accomplishments.  Contented success that takes place day by day has the value of making you feel inwardly warm and cozy knowing you have done your job right.
·         Unappreciated. As with Y2K, I have also seen projects deprived of recognition simply because they provided success without drama. It is ironic that projects that first fail and then eventually get fixed (usually at added expense and time) are the ones that tend to get credited and celebrated the most (e.g. Hubble Space Telescope). However, projects that deliver their promise right off the bat; especially when the promise is one of risk aversion tend to receive little or no recognition. 
·         Fake. Not real success, but I added it to this list for completeness sake. We all remember that landing on an aircraft carrier with a “Mission Accomplished” banner, and who actually thinks the Kardashians are talented?
Yes, no one wants to be unappreciated, but hopefully you will not make claims for fake successes. It is fine to let everyone know of your actual accomplishments, but do so in a manner that does not cross the bragging line. To achieve success worthy of celebration, it is best is to be objective about the essential metrics right from the start.  When creating a new project plan, make sure to include a Success Criteria section were you list the criteria that must be met in order to qualify it for accolades. You may need to educate everyone on the paradigm that while the success criteria does include the deliverables you and your sponsors deem mandatory, it does not necessarily reflect the totality of desired deliverables.  After all, Kennedy’s goal for the Moon Shot was a very simple "land a man on the Moon and return him safely to Earth by the end of the decade”.  While the original five billion dollar budget was easily exceeded, and failures occurred along the way, the moon landing mission was indeed a Bombastic success.
If the objective of a project is to deliver something tangible, you can at the very least make a compelling case for its success by showing the working deliverable. However, success will be more difficult to define for those projects developed to avoid risks. In these cases your criteria will be phrased around the concept of “avoidance”, and since “avoidance” is fairly open-ended (e.g. “Avoid hackers breaking into our Credit Card Data Base”), you will need to refine the parameters. For instance: How long will you keep the data safe?  At what cost? What specific kinds of intrusion will be prevented? The more precise and objective you are with metrics, the better.
True success should always be assessed fairly and realistically and then celebrated. It’s worth remembering that lack of recognition is one of the main reasons seasoned professionals search for greener pastures. Every accomplishment should be humbly recognized and used as a foundation for the next step up the ladder.  And for every successful step we should give contented thanks.
With that, I wish you all a very successful 2015!

[1] If moving from the year 1999 to the year 2000 was tough, can you imagine the pain the Chinese had to undergo to move from the year of the Rabbit to the year of the Dragon? (No more millennium jokes, I promise)

Friday, November 20, 2009

The Data Sentinel

Data is what we put into the system and information is what we expect to get out of it (actually, there’s an epistemological argument that what we really crave is knowledge. For now, however, I’ll use the term ‘information’ to refer to the system output). Data is the dough; Information the cake. When we seek information, we want it to taste good, to be accurate, relevant, current, and understandable. Data is another matter. Data must be acquired and stored in whatever is best from a utilitarian perspective. Data can be anything. This explains why two digits were used to store the date years in the pre-millennium system, leading to the big Y2K brouhaha (more on this later).  Also, data is not always flat and homogeneous. It can have a hierarchical structure and come from multiple sources. In fact, data is whatever we choose to call the source of our information.
Google has reputedly hundreds of thousands of servers with Petabytes of data (1 Petabyte = 1,024 Terabytes), which you and I can access in a manner of milliseconds by typing free context searches. For many, a response from Google represents information, but to others this output is data to be used in the cooking of new information. As a matter of fact, one of the most exciting areas of research today is the emergence of Collective Intelligence via the mining of free text information on the web. Or consider the very promising WolframAlpha knowledge engine effort ( which very ambitiously taps a variety of databases to provide consolidated knowledge to users. There are still other mechanisms to provide information that rely on the human element as a source of data. Sites such as or actually use carbon-based intelligent life forms to respond to questions.
Data can be stored in people’s neurons, spreadsheets, 3 x 5 index cards, papyrus scrolls, punched cards, magnetic media, optical disk or futuristic quantum storage. The point is that the user doesn’t care how the data is stored or how it is structured. In the end, Schemas, SQL, Rows, Columns, Indexes, Tables, are the ways we IT people store and manage data for our own convenience. But as long as the user can access data in a reliable, prompt, and comprehensive fashion, she could care less whether the data comes from a super-sophisticated object oriented data base or from a tattered printed copy of the World Almanac.
How should data be accessed then? I don’t recommend handling data in an explicit manner the way RDBMs vendors tell you to handle it. Data is at the core of the enterprise, but it does not have to be a “visible” core. You don’t visualize data with SQL. Instead, I suggest that you handle all access to data in an abstract way. You visualize data with services and this brings up the need via a Data Sentinel Layer. This layer should be, you guessed it, an SOA enabled component providing data accesses and maintenance services.
To put it simply, the Data Sentinel is the gatekeeper and abstraction layer for data. Nothing goes into the data storages without the Sentinel first passing it through; nothing gets out without the Sentinel allowing it. Furthermore, the Sentinel allows decoupling of how the data is ultimately stored from the way the data is perceived to be stored. Depending upon your needs, you may choose consolidated data storages or, alternatively, you may choose to follow a federated approach to heterogeneous data. It doesn’t matter. The Data Sentinel is responsible for presenting a common SOA fa├žade to the outside world. 
Clearly, a key tenet should be to not allow willy-nilly access to data by bypassing the Sentinel. You should not allow applications or services (whether atomic or composite) to fire their own SQL statements against a data base. If you want to maintain the integrity of your SOA design, make sure to access data via the data abstraction services provided by the Sentinel services only.
Then again, this being a world filled with frailty, there are three exceptions where you will have to allow SOA entities to bypass the abstraction layer provided by the Sentinel. Every castle has secret passageways. I will cover the situations where exceptions may apply later: Security/Monitoring, Batch/Reporting, and the Data Joiner Pattern.
Obviously, data abstraction requires attention to performance, data persistence, and data integrity aspects. Thankfully, there are off-the-shelf tools to help facilitate this abstraction and the implementation of a Sentinel layer, such as Object-Relational mapping, automated data replication, and data caching products (e.g. Hibernate). Whether you choose to use an off-the-shelf tool or to write your own will depend upon your needs, but the use of those tools is not always sufficient to implement a proper Sentinel.  Object-Relational mapping or use of Stored Procedures, for example, are means to more easily map data access into SOA-like services, but you still need to ensure that the interfaces comply with the SOA interface criteria covered earlier. In the end, the use of a Data Sentinel Layer is a case of applying abstraction techniques to deal with the challenges of an SOA-based system, but one that also demands engineering work in order to deploy the Sentinel services in front of the Data Bases/Sources. There are additional techniques and considerations that also apply, and these will be discussed later on.

Friday, November 13, 2009

ESB and the SOA Fabric

A number of new needs have emerged with the advent of SOA. First of all, there was no standard way for an application to construct and deliver a service call.  Secondly, there was no standard way to ensure the service would be delivered.  Thirdly, it was not clear how this SOA environment could be managed and operated effectively. Fourthly . . .  well, you get the idea; the list goes on and on.
SOA demands the existence of an enabling infrastructure layer known as middleware. Middleware provides all necessary services, independent of the underlying technical infrastructure.  To satisfy this need, vendors began to define SOA architectures around a relatively abstract concept: the Enterprise Service Bus, or ESB.   Now, there has never been disagreement about the need to have a foundational layer to support common SOA functions—an enterprise bus of sorts. The problem is that each vendor took it upon himself to define the specific capabilities and mechanisms of their proprietary ESB, oftentimes by repackaging preexisting products and rebranding them to better fit their sales strategies.
As a result, depending on the vendor, the concept of Enterprise Service Bus encompasses an amalgamation of integration and transformation technologies that enable the cooperative work of any number of environments: Service Location, Service Invocation, Service Routing, Security, Mapping, Asynchronous and Event Driven Messaging, Service Orchestration, Testing Tools, Pattern Libraries, Monitoring and Management, etc. Unfortunately, when viewed as an all-or-nothing proposition, ESB’s broad and fuzzy scope tends to make vendor offerings somewhat complex and potentially expensive.
The term ESB is now so generic and undefined that you should be careful not to get entrapped into buying a cornucopia of vendor products that are not going to be needed for your specific SOA environment.  ESBs resemble more a Swiss army knife, with its many accessories, of which only a few will ever be used. Don’t be deceived; vendors will naturally try to sell you the complete superhighway, including rest stops, gas stations and the paint for the road signs, when all you really need is a quaint country road. You can be choosy and build your base SOA foundation gradually.  Because of this, I am willfully avoiding use of the term “Enterprise Service Bus”, preferring instead to use the more neutral term, “SOA Fabric.”
Of all the bells and whistles provided by ESB vendors (data transformation, dynamic service location, etc.), the one key function the SOA Fabric should deliver is ensuring that the services and service delivery mechanisms are abstracted from the SOA clients.
A salient feature that vendors tell us ESBs are good for is their ability to integrate heterogeneous environments. However, if you think about it, since you are going through the process of transforming the technology in your company (the topic of this writings after all!), you should really strive to introduce a standard protocol and eliminate as many of legacy protocols as you can.
Ironically, a holistic transformation program should have the goal of deploying the most homogeneous SOA environment possible; thus obviating the need for most of the much touted ESB’s transformation and mapping functions. In a new system, SOA can be based upon canonical formats and common protocols; thus minimizing the need for data and service format conversion. This goal is most feasible when applied to the message flows occurring in you internal ecosystem.
Now, you may still need some of those conversion functions for several other reasons, migration and integration with external systems being the most obvious cases. If the migration will be gradual, and therefore requires the interplay of new services with legacy services, go ahead and enable some of the protocol conversion features provided by ESBs. The question would then be how important this feature is to you, and whether you wouldn’t be better off following a non-ESB integration mechanism in the interim.  At least, knowing you will be using this particular ESB function only for migration purposes, you can try to negotiate a more generous license with the vendor.
There are cases whereby, while striving for a homogeneous SOA environment, you may well conclude that your end state architecture must integrate a number of systems under a  federated view. Your end state architecture in this case will be a mix of hybrid technologies servicing autonomous problem domains. Under this scenario, it would be best to reframe the definition of the problem at hand from one of creating an SOA environment to one of applying Enterprise Application Integration (EAI) mechanisms. If your end state revolves more around integration EAI, it would be better suited to performing boundary-level mapping and transformation work.  In this case, go and shop for a great EAI solution; not for an ESB.
If the vendor gives you the option of acquiring specific subsets of their ESB offering (at a reduced price) then that’s something worth considering. At the very least, you will need to provide support for service deployment, routing, monitoring, and management, even if you won’t require many of the other functions in the ESB package. Just remember to focus in deploying the fabric that properly matches your SOA objectives and not the one that matches your vendor’s sales quota.
A quick word regarding Open Source ESBs. . . There are many, but the same caveats I’ve used for vendor-based ESB’s apply. Open Source ESBs are not yet as mature, and the quality of functions they provide varies significantly according to the component. Focus on using only those components you can be sure will work in a reliable and stable manner or those which are not critical to the system. Remember you are putting in place components that will become part of the core fabric. Ask yourself, does it make sense in order to save a few dollars to use a relatively unsupported ESB component for a critical role (Service Invocation or Messaging, come to mind), versus using a more stable vendor solution?
In the end, if you are planning to use the protocol conversion features packaged in a vendor-provided or open source ESBs, I suggest you use them in a discrete, case-by-case basis, and not as an inherent component of your SOA fabric. This way, even as you face having to solve integration problems associated with the lack of standards, at least you won’t be forced into drinking the Kool-Aid associated with a particular vendor’s view of ESB!

Friday, November 6, 2009

The Orchestrators

Back in the XIX century (that’s the 19th century for all you X-geners!), there was a composer who didn’t know how to play the piano. In fact, nor did he know how to play the violin, the flute, the trombone, or any other instrument for that matter. Yet, the man managed to compose symphonies that to this day are considered musical masterpieces. The composer’s name was Louis Hector Berlioz, and he achieved this feat by directing the orchestra through each step of his arrangement and composition. His most recognized work is called “Symphonie Fantastique” and, according to Wikipedia, the symphony is scored for an orchestra consisting of 2 flutes(2nd doubling piccolo), 2 oboes (2nd doubling English horn), 2 clarinets (1st doubling E-flat clarinet), 4 bassoons, 4 horns, 2 trumpets, 2 cornets, 3 trombones, 2 ophicleides (what the heck is an ophecleide? A forerunner of the euphonium, I found out. What the heck is a euphonium? Well, check it out in Wiki!), 2 pairs of timpani, snare drum, cymbals, bass drum, bells in C and G, 2 harps, and strings.
By now, you probably get the idea. Mr. Berlioz fully exemplifies the ultimate back-end composite services element: The Orchestrator. Berlioz composed some pretty cool stuff by knowing a) what he wanted to express, b) what specific set of instruments should be used at a particular point in time, and c) how to communicate the notes of his composition to the orchestra.
Every SOA-based system needs its Berliozes.
There are several dimensions involved in defining the role of an orchestrator for SOA. First, as discussed earlier, most orchestrator roles will be provided within the context of an application; not as a part of a service. That is, the orchestration is what defines an application and makes one application different from another. The orchestration is the brain of the application, and it is the entity that decides the manner and SOA services calling flow.
In some instances, you might even be able to reuse orchestration patterns and apply them across multiple applications. Better still, you can build orchestration patterns by utilizing the emerging Business Process Modeling technologies (BPM). BPM simplifies the work of creating orchestration logic by providing a visual and modular way of assembling orchestration flows. A small commentary of mine is this: BPM is not SOA, but BPM requires SOA to work properly.
An apropos question is to ask how much orchestration should be automated in the SOA system as opposed to letting the user manually orchestrate his or her own interactions. To answer this question it is best to remember the complexity rule I stated earlier:  the simpler the user interaction; the more complex the system, and vice-versa. 
Then again, there are limits to the complexity of an orchestration. A full-fledged Artificial Intelligence system could become the ultimate orchestration engine but, unfortunately, such a machine remains in the realm of science fiction.  Cost-Benefit compromises must be made.
Say we have a travel oriented system and need to find the coolest vacation spots for the month of September. Should we let the user manually orchestrate the various steps needed to reach a conclusion? Each step would indirectly generate the appropriate service calls for searching destinations, filtering unwanted responses, obtaining additional descriptions, getting prices, initiating the booking, and so forth. Or we could consider developing a sophisticated orchestration function that’s able to take care of those details and do the hard work on behalf of the prospective traveler. But should we do it?
The answer lies in the size of “the market” for a particular need. Clearly, there is a need for a travel orchestration capability that can take care of all the details mentioned. After all, isn’t this why Travel Agencies emerged in the first place? If the orchestration is need by only a few users, then it is best not to spend money and effort attempting to automate something that is too unique. On the other hand, if the request becomes common, then it is preferable to create an automated orchestration function that organizes and integrates the use of SOA services.
The orchestrators design should always accommodate the transparency tenets in order to allow horizontal scalability. In other words, if you provide the orchestration via servers located in the system membrane, you will need to design the solution in such a way that you can always add more front end servers to accommodate increased workloads, without disrupting the orchestration processes in existing servers. Because orchestration usually requires the server to maintain some form of state, at least for the duration of a transaction, you will need to incorporate some form of session-stickiness in the orchestration logic. Later on, I will write more about why I recommend that this is the one and only area where a “session state” between the user and the orchestration should exist, even as I still advice to keep backend services discrete and sessionless.