Sunday, November 22, 2009

Let's expose everything the backend provides!

Something I hear many times in my profession is when a customer is conservative on spending money but wants as much value as possible. Attitude sounds ok, right? Well here's an example when this does not work.

The problem:
The budget is limited, the back end has huge interfaces and I need only a fraction of that. My sponsor must be convinced that now is the time to expose more.

Case study:

  • The desired service is "Change Name&Address Details". This is the actual requirement (I know, not SMART but for this example it will do just fine) 
  • The back end system exposes a "UpdateCustomerProfile" service which allows changing Name, Address, Privacy settings, Credit Status, Collection Path and collection status and a whole bunch more... 
The team starts analyzing and finds that actually, the data this back end allows to change is from different logical/functional domains; the CM, Billing, Credit, Collections etc. domains. They discuss with the project sponsor that a number of these services are actually very useful and it's perhaps wise to expose more than just the changeNameAddress capability.

When further designing the services, they find that a whole bunch of services and even more capabilities can be exposed on this back end service UpdateCustomerProfile. When they discuss their findings with the sponsor, he actually dictates that instead of exposing the proposed 10+ granular capabilities, he says I will grant you the luxury of exposing extra functionality as I see the benefit, so you get some extra money however, you must expose it as a single 1:1 service and let the consumer decide which functionality to use. This should also increase the reusability of the service so it's a sure win-win situation and I get more value for money. Let's call this service "ChangeCustomerProfile".

Done deal, right? Why is this a problem? Everyone's happy, everyone gets what they want.

Let's see why this is not the right approach:
  • By exposing all the nitty-gritty back end interface details 1:1 they actually kill the decoupled contract pattern used in SOA. The service consumer must now (because of the bottom-up approach) know how the back end system works in order to effectively use the interface on the middleware. Actually, it is as if the middleware did not exist. With or without the middleware, no difference in controlling the back-end. Where the gain?
  • As you can see from the new service's name "ChangeCustomerProfile" it was hard to find a name which makes it intuitive for the service consumer to derive from the service name what the service does. The designer of the consumer of this service must now read extensive documentation and carefully decide how to use it in order to avoid invalidating data or breaking data consistency. Hmmm... I always thought that, when it comes to consuming SOA services, the understanding of the system would be better; but nothing changed. One of my next posts will be on service naming. Stay tuned for that one.
  • By exposing this extremely large interface to the service consumers (very 'reusable' right? So many consumers), in case something must change the interface, chances are, we have to update many consumers to make sure they all keep working with this service capability. For the people who are now arguing that you don't need to change all the consumers, let's see: you are partially right. But the cost of a system is not just determined by the service delivery effort; there's also something called operation & maintenance. Every operational old version of a service interface, increases your operation and maintenance cost. This is what is referred to as your "IT Burden", which in fact kills part of your SOA strategic goal to reduce the IT burden. So you loose either way:  either because next change on your interface triggers many service consumer changes, or because the number of versions of this interface explodes, making operations hard.
We were just forced to ignore many of the SOA design principles:
  1. Standardized service contracts: We did not do this at all. Instead of designing interface-first, we created the interface from the back-end exposed interface. This bottom-up approach results in tight coupling between consumer and service provider.
  2. Service loose coupling:  This bottom-up approach, which is also reached when ie. using interface generation tools like java2wsdl, results in tight coupling between consumer and service provider. We may have a decoupled interface from technology and transport point of view, we definitely did not gain loose coupling from exposed back end services.
  3. Service Abstraction: Instead of exposing capabilities abstracted from the back end, we exposed the back end interface 1:1: no abstraction at all on capability level.
  4. Service Reusability: This is the one we unintentionally failed despite the sponsors' "efforts". A service is a true candidate for reusability when many of the SOA design principles are applied properly, including #1, #2, #3, #7 (not a guarantee that reuse is reached, but at least the potential for reuse).
  5. Service Autonomy: because we exposed a too generic interface "can-do-all", we unfortunately disallowed future designers to only use parts of the system which are necessary. Instead, they need to hit that same huge service 'capability' in many situations. By doing this, the autonomy of the service decreases.
  6. Service Statelessness: Because this service can-do all, chances are there that a lot of state must be passed back and forth. Although the service on itself may be stateless, the knowledge of the back end's data is large on client side. Usually statelessness is a significant driver for the amount of scalability which can be obtained, but so is passing around a lot of data.Worst case, the client must read ALL data the service exposes, to be able to modify and update the back end data.
  7. Service Composability: Composability is low in this example, in general. The service fails most of the design principles, resulting in low composability.
  8. Service Discoverability: There is virtually none. The service name nor documentation could give distinguishable content to make it easy to find the appropriate service for the appropriate capability. What can I do with the customer? What is this 'profile'. Is it a form of aggregated data which allows deriving customer value? Is it something related to his visual appearance? What is this thing we're exposing???
    How did this cause us to fail most of the 7 strategic goals of SOA:
    1. Increased intrinsic interoperability: failed because the technical back end interface was exposed directly to the service consumers. Gone agnostic services. Gone interoperability as composing services can only be done with exhaustive reading and figuring out how this stuff actually works: in fact, instead of making it easy to use this service in other service compositions, you remain on the good ol' integration effort path and will have gained anything in the domain of this goal.
    2. Increased federation: there is no increased federation. Everything remained as before. Your middleware exposes what your back end provides, no chance of improving federation.
    3. Increased business and technology alignment: Sort of... on one side, the business is involved during the design process. But next time this service is available and then it remains questionable whether the business is consulted. Benefit of the doubt though I tend to think 'no'.
    4. Increased vendor diversification options: Actually, no, you are just locked in more to this vendor as now, not just the consumers, but also the middleware is tightly interlocked with the vendor's interfaces.
    5. Increased Return on Investment: Your initial investment: yes, you get a lot of service (which is not used right away) for your money. But the cost of next changes or the cost of operation will increase and must be included in the ROI calculations as well. So less ROI than expected. The issues typically arise after several years when the environment becomes unmanageable. Since ROI is impacted by the first four strategic goals which we gracefully failed, don't expect anything here.
    6. Increased organisational agility: We did not help the organisation wrt agility by building this service. In fact, by having yet another IT 'asset' exposing more of the same, in exactly the same way hardly helps improving in this area.
    7. Reduced IT burden: We failed the first six goals, so what do we get on this one? Right, nothing at all. Probably what we did made managing the solution harder, so your IT burden increases instead of reaching the intended decrease.

      Saturday, November 14, 2009

      Common causes of SOA failure

      A number of common misconceptions will make most SOA initiatives fail, hard to survive or not start at all. This post is not complete, and may be updated many times, depending on the feedback I get from you, the users. If you feel a cause is missing, drop a comment including your issue, cause and resolution. Don't forget to mention your name and contact details.

      • SOA is a silver bullet
      • Failure to implement proper governance
      • Failure to get management buy-in
      • SOA "the enterprise"
      • Specific vendor has the best SOA implementation
      • SOA quickly brings ROI
      • We are doing web services, so we are doing SOA

      SOA is a silver bullet

      Issue: SOA is not a tool but an architectural approach. Implementing a tool does not implement the SOA concepts into the business and technology.

      Cause: Vendors explain their SOA tools to be the silver bullet.

      Resolution: There is none. SOA is a mindset supported by tools; as long as you are aware about that, and live up to those concepts and patterns, you may be fine.

      Failure to implement proper governance

      Issue: Failure to consider/facilitate the governance burden in a project will increase the governance burden after the project, if not make it virtually impossible.

      Cause: Implementing a SOA in a project instead of a programme may easily let the project slip into the tactical approach instead of the strategical approach.

      Resolution: Setup some governance requirements prior to starting your first business project. Having these can force the project to include governance in the project, making sure the deliverables are not left 'floating' in the organization after project delivery.

      Failure to get management buy-in

      Issue: We, the IT department got this SOA installed, but no-one is willing to invest the extra money required with the implementation of SOA services. Every project is going for the quick and dirty approach, continuing as they were...

      Cause: No management buy-in, resulting in lack of appropriate governance.

      Resolution: Evangelize! Convince the senior management staff that the SOA thing they bought is really a good thing, resulting in long-term benefits, the traditional way of working will not provide. Result of management buy-in can be that the appropriate governance is implemented and a SOA way of working is enforced. Explain the same situation to everyone who's willing to hear your story.

      SOA "The enterprise"

      Issue: Companies try to change the entire enterprise at once. This is a definite recipe for failure. No single company can change all their IT systems, business processes, governance etc. in one go.

      Cause: This is a common misconception: Implementing a SOA involves people from the entire enterprise when the business analysis and service modeling project phases take place. This means people from the entire enterprise, multiple disciplinces. This does NOT mean that we need to involve EVERYONE and ALL systems.

      Resolution: To start very small: select specific business processes to change. Implement SOA services using the SOA approach which show a clear business value and can demonstrate that SOA can work. Typically management only chooses a technology if the proof is there. Maybe even demonstrate a few quick wins. Even then, SOA is a strategical approach so don't expect miracles.

      Specific vendor has the best SOA implementation

      Issue: Comparing vendors' SOA products based on their marketing and their product specifications gave me this best of breed product, but it's not doing what I want. It's not possible to 'get the best SOA products out there'.

      Cause: A lot of vendors deliver their SOA product with a lot of promises and marketing terminology. Sometimes it's even not true that certain vendors claim delivering SOA tools because people mistakenly assume 'web services' being 'services'. Also it's hard to do a product selection without requirements. It is similar to the 'get me the best car out there' discussion. If you don't tell your requirements you may end up with a Ferrari on an offroad track.

      Resolution: Define your company's requirements, define a strategy and then compare products with your requirements.

      SOA quickly brings ROI

      Issue: If I buy this SOA thing, I will quickly get a better ROI.

      Cause: Not understanding the strategical nature of a SOA.

      Resolution: Increased ROI, Better technology/organizational alignment are prerequisites for a reduced IT burden. All of these are indirectly obtainable SOA characteristics, depending on a number of other SOA concepts. This illustrates the strategical nature of the SOA approach.

      We are doing web services, so we are doing SOA

      Issue: Web services and SOA service terminology is mistakenly assumed "the same thing"

      Cause: Web services provide a standard technology for the service contract. Web services don't force the user into following SOA concepts and patterns delivered by the SOA approach, resulting in "SOA Services"

      Resolution: This needs a proper implementation of accurate terminology in people's minds. Once this is reached, a common understanding of the difference between a web service as technology and service as concept is reached and the mistake will not be made again. This may very well be one of the hardest things to achieve (my definition of service is better than yours, your definition of service does not deliver anything concrete etcetera).

      Thursday, November 5, 2009

      How does SOA fit in today's and tomorrow's distributed architecture

      This post describes briefly how SOA fits into current distributed computing architectures. Its purpose is not to give a full detailed positioning but is merely intended as a primer.

      silo - integration - soa - grid - cloud

      Traditionally, companies would implement "silo" based applications or application silos. By nature, these software stacks integrate poorly with other solutions because they are self-providing. Vendors selling these systems are not interested in opening up their systems as it allows their installed (functional) base to be reduced by other vendor's products with similar functionality. This makes it hard to make these systems work together with other systems. Mostly, the customers of these systems need to pay prime rates to have them opened up for their intended purposes, and the vendors will gladly charge another customer money to get the same or a similar interface implemented.

      Making these systems work together is called "integration". Integrating systems is bespoke work per definition. It is considered expensive as it is custom work for every new integration. An integration effort is considered a tactical approach. Also, integration is based on a typical point-to-point approach which must be prevented to prevent IT burden from skyrocketing. The more interfaces are built between systems, the more chaotic are changes to be implemented and the more unpredictable the cost of change becomes.

      This is where "SOA" comes into the picture. SOA tries to change this cluttered 'spaghetti architecture' into a more organized 'lasagna architecture' if you will. Why lasagna? Because a layered approach helps improving the predictability of your IT systems. Ever tried to take out a piece of spaghetti from your plate? You never know what comes with that small piece... With lasagna, you can take out a neat square (read: structured) piece and work from there. If you would read these neat pieces as services or service capabilities, it becomes apparent that the predictability of cost and impact of changes is increased, compared to spaghetti architectures. Yes, I hear you thinking, "I can do that with 'integration' as well" and you are probably right. The difference between integration and a service oriented architecture approach is mainly based on the approach SOA takes. SOA is based upon services with intrinsic interoperability which would be built to fulfill a specific business purpose. Furthermore they are intended to be reused/reconfigured aka (re)composed, and integration efforts are because of their custom efforts not easily or not at all reusable. To improve recomposability, SOA services should preferably not hold state, as holding state makes use of a SOA service limited to certain interaction patterns (their functionality depends on prior calls to the services or service compositions); without it, the intended business purpose cannot be fulfilled. Because a SOA service fairly easy (all in the eye of the beholder) can be used for other purposes in other service compositions, the SOA architecture approach is considered a strategical approach. For more SOA concepts I can recommend this book.

      A problem of traditional SOA architectures is that they are really good at processing large volumes of small messages, but they lack the make-up for handling large messages. Many SOA architectures are based on a message based infrastructure. This means that messages are to be read into memory and streamed on the network many times in order to execute the business goals. These are by nature expensive resources as they eat away at network bandwidth, memory and potentially disk I/O. In a SOA "grid", services are allowed to have state and state is available (shared) across a certain infrastructure which reduces the number of times reading into memory and flushing from memory to network and/or disk. Fundamentally, a SOA grid is a special kind of SOA architecture which works around some of its challenges. It makes the SOA architecture more efficient with respect to large messages. Also a grid architecture has improved availability and fault tolerance because of its make-up.

      A "cloud" architecture virtualizes resources like storage, memory, cpu, complete machine, java virtual machine etc. Basically, a cloud consists of containers with specific characteristics to meet a certain goal. For example you can have a database in the cloud, not knowing where it's hosted and you typically don't need to worry about it. It has a different cost model (ie. x cents per effective capacity unit per effective time unit) and you do not have to worry about hosting services. This also introduces other issues, like where is my customer data stored. Many countries legislation does not allow storing sensitive data outside the country, or in case of the EU, outside the EU. This poses different challenges. Advantage is however that you don't invest or plan a huge local SOA based infrastructure, but utilize (multiple) shared but isolated virtual infrastructure(s) to host your environment. You typically pay for what you actually use. And the environment is easily scalable as the resources are virtualized and would allow (on the fly) configuration changes (ie. assign more CPU or disk capacity). It is extremely simple to scale both vertically as well as horizontally. A step further in the cloud, is software, platform or application as a service, this is where provides will provide ie. a complete application environment as a service, resulting in a cloud of infrastructure, resources, platforms etc.

      Another nice detail is that, if the concept of a SOA based approach did not exist, the paradigms and services required for a SOA grid and a cloud would be way less flexible than present. Potentially, the SOA grid and cloud would not even exist in it's current form.

      Another (more traditional) application of grid computing / grid architecture (notice the difference in terminology: not "SOA Grid") is where a giant task is  split into smaller tasks and split across multiple systems which bite on smaller chunks of the huge data load. Very big companies would use supercomputers to do this before the idea of grid computing came along. The issue was that these computers would cost insane amounts of money and they could only be put to use on one task at a time if maximum system capacity should be available. In a grid computing environment, the computing units run on standalone computers which can be configured into groups of computing units each specializing in a different task. Because commodity hardware can be used instead of supercomputers, the cost per calculation will be significantly lower. Some initiatives have gone a step further at this and utilized public computers (individual user's computers) to increase the computing power to the greatest extent. Examples are the SETI initiative, or medical companies calculating the must effective treatment for a certain disease of virus. Funny detail is that complete public communities have formed competing to process the most parts of a giant problem - they have turned this into a competition much to the benefit of the companies putting up the calculations for work. If you would like to see such competition, view this score board.