Friday, April 23, 2010

Thoughts on the SOA Manifesto 2/2

Guiding Principles, important statements to make when thinking about SOA in general.

Respect the social and power structure of the organization.
We need to know that since every organization is different, and a SOA is business-driven, every SOA implementation is different. By respecting the social and power structure in the organization, we are allowed to continue developing a SOA architecture and implementation, whereas "rowing upstream" significantly reduces our productivity, as well as it may kill the entire SOA initiative.

Recognize that SOA ultimately demands change on many levels.
Since a SOA is business-driven and also aligning business / business and IT, a number of changes can be expected, not only in the IT landscape. For example, since processes are shared, processes and organizational departments must be aligned. Since infrastructure, service provided etc. are also shared, this requires different ways of funding projects, as well as new roles and responsibilities to be defined ie. for governance of the SOA, project and program management, program alignment, clear distinction for who is responsible for which aspects of the SOA and the organization etc. This has impact on all levels of the organization.

The scope of SOA adoption can vary. Keep efforts manageable and within  meaningful boundaries.
Well, keep it small and simple (K.I.S.S.) I guess... Try, when starting with SOA adoption define a meaningful boundary for the SOA adoption. Start small, start simple. Not only with the SOA implementation, but also when defining the scope. Try a proof of concept; try experiencing implementation of a few (sharable) services in production. Get your experience with them, and try to see how you can learn. Only then you should be ready to grow from there, in what is probably a multi-year effort, towards the intended initial scope. The scope itself - how far you adopt SOA - can be increased incrementally in a programme approach. In SOA, the big-bang approach is recipe for failure. It may be better to start off small and iteratively see how you can 'grow your garden' towards the 'end state'. But please remember, a garden is never finished and constantly needs maintenance and attention. In a garden usually there is also some waste; sometimes a bit more and sometimes a bit less: don't hesitate to discard some of your investments if there is no more clear business value in them.

Products and standards alone will neither give you SOA nor apply the service orientation paradigm for you.
SOA is not a tool, nor a silver bullet that you can buy, install and you have lift-off. It takes planning and skill to create a SOA. Actually, it is not something you build, but the way you do things. If you live by the service orientation principles, you have a fair chance of getting there. Buying tools without guidelines how to use them, without a proper plan in the appropriate programme, lacking management buy-in and governance to regulate, the product implementation will be ill-spent money.

SOA can be realized through a variety of technologies and standards.
As long as you stick to the principles of service orientation, you can realize any flavour of service orientation. Every technology and standard has its pros and cons. It all boils down to how you use them. That's why SOA can be implemented using most technologies and standards. SOAP is not the implementation of a SOA, but a SOA can be implemented using SOAP. And SOAP can be used on various transports like JMS, HTTP etc. But SOAP on itself, nor the transports mentioned are mandatory when "going SOA". Any middleware you use which allows utilizing the principles of SOA will suffice. And technologies can be mixed if it makes sense.

Establish a uniform set of enterprise standards and policies based on industry, de facto, and community standards.
Enterprise standards and policies help defining SOA elements in a consistent way. By applying the same guidelines, chances of (intrinsic) interoperability increase and this is key to a SOA foundation. Standards, policies and guidelines are not a necessary evil: without them there would not be a SOA. This implies that these must be properly enforced and governance is required to ensure they are applies consistently.

Pursue uniformity on the outside while allowing diversity on the inside.
This one is a bit more difficult and less obvious than the rest. By introducing uniformity on the outside (standardization at the interface level) you make interoperability easier, because more diversity increases a learning curve which reduces the achievable reuse. This is because when things get harder (to comprehend), the potential for reuse decreases. Having said that, internally, anything should be possible. Diversity should be allowed to be able to deliver effective core service logic. Effective core service logic contributes to the attractiveness of services and would inherently increase reuse potential. The one size fits all approach does not work on the inside as well as the outside. Some balancing and fair trade-offs may be required to make it work.

Identify services through collaboration with business and technology stakeholders.
No-one can understand an entire business. Noone can design a service or a system without understanding why he is doing it. That's where the business stakeholders come in. Also no-one can understand all the technology requirements and implementation consequences. This required technology stakeholders to participate. On the other hand, cooperation requires trust working both ways. The business participants should realize that in order to maximize enterprise effectiveness, the scope for the SOA architect can be larger than their own scope, and allow the technology people to design services which support the business as a whole.

Maximize service usage by considering the current and future scope of utilization.
Plan ahead. See how a service fits into the programme of projects and initiatives to shape the SOA. Try to predict how the SOA evolves and see how the service can be of greater benefit by predicting how the programme and long term plans affect the consumption of the service. Try to see how the potential of the service can be increased ie. by extrapolating how this service can be useful for other types of consumers, channels etcetera. This would ultimately result in a increased return on investment (ROI).

Verify that services satisfy business requirements and goals.
Obviously, business requirements drive a SOA. Without business requirements, there would not be a need for a SOA and services would probably not be the best approach as the effective implementation of services creates certain overhead which would cost a lot of money, resulting in decreased performance of the architecture. Without a goal, pursuing business requirements may result in a majority of tactical decisions preventing to reach the actual enterprise goals. So while implementing services from tactical point of view, try to never loose the strategic goals out of sight.

Evolve services and their organization in response to real use.
Let me come back to the gardening analogy I made earlier. When creating a garden, everything makes perfect sense. All plants and flowers look great together and everything is tuned. But no garden can survive without gardening (maintenance). In the course of time, taste (requirements) changes. Also sometimes, things that looked great initially, may seem trivial after a while, or may even explicitly be undesired (change of enterprise goals). Be prepared to review and maintain the (service) garden and make sure it's always aligned with the actually expected use. When a garden (SOA) is not aligned with the expectations, use decreases and in the end, has no right to survive and may even be replaced by something else.

Separate the different aspects of a system that change at different rates.
I feel this statement is a bit narrowing the actual potential it could have had. If the statement would be "Separate concerns" I could have better agreed with it. When separation of concerns, as a principle, is properly applied, this greatly increases the comprehensibility and decreases complexity (learning curve). Separation of aspects of a system that change at different rates would be a special version (implementation) of the statement to separate concerns.

Reduce implicit dependencies and publish all external dependencies to increase robustness and reduce the impact of change.
Reduction of implicit dependencies actually refers to the principle of abstraction. Abstraction leads to loose coupling. Loose coupling is key to a SOA as it increases flexibility and consequently the ability to change. When the ability to change increases, the impact of change is lower.

At every level of abstraction, organize each service around a cohesive and manageable unit of functionality.
Abstract each functional group of logic into a self-contained, isolated, cohesive logical unit. This makes the functionality more manageable by the separation of concerns principle. A group of related logical units can be encapsulated into a service. A service can have many capabilities it ultimately encapsulates a manageable unit of functionality. By having isolated units of work, autonomy and composability increases contributing to the (re)use of services.

Perhaps it's wise to make the following statement: Thomas Erl already described the principles for service orientation. The statements in these posts do not attempt to redefine these principles. If you want to review them, take a look at the website where you should find lots of information on the Prentice Hall SOA Books by Thomas Erl et al.

Phew... this is it for now. I hope this gives food for thought. If you agree or especially if you disagree with the statements made here, please be invited to drop me a line and explain your point of view. Perhaps I have to change my insights and way of thinking :)

Sunday, April 4, 2010

Thoughts on the SOA Manifesto 1/2

October 2009 I was at the 2nd SOA Symposium in Rotterdam where the final efforts were made and the actual presentation of the "SOA Manifesto" happened.

This post is how I interpret the SOA Manifesto statements and I would like to encourage people to post their concerns / remarks on this vision.

Business value over technical strategy
While having a planned technical strategy is extremely important, it's also the business who pays the bills. The cost of projects is often not really aligned with the expected business value, so we have to make a trade-off between our strategy and make tactical decisions to keep the business happy. When we keep the business happy, we are most likely allowed to continue our planned path to destiny and in the end get close to our technical strategy goals. If we push too much on the technical strategy end, we risk loosing our customer's commitment which may blow the entire SOA architecture efforts, since they invest in things that do not generate the appropriate business value.
Strategic goals over project-specific benefits
Traditionally, project point of view would strive for short term gains, to reach the project goals. But sometimes the project goals are not in the right direction of the strategic goals. Although the project manager has all interest and most value to gain from meeting the project benefits, some restraint must be taken in order to pursue these at all cost, especially when they are not aligned with the strategic goals. Usually a project is aligned with the strategic goals of the company, but sometimes these goals are not the way to deliver the project easiest. This is where mixed concerns get in the way of everyone. One should not allow project goals more significant than the strategic goals, the latter typically being the reason why the project was started in the first place. As SOA delivers enterprise strategic value, the projects should implement this value on a project-by-project basis whilst inside those projects, never loosing the enterprise strategic goals out of sight.
Intrinsic interoperability over custom integration
Although often a custom integration may be quicker to implement, it is typically as the name says, custom work - it cannot be reused somewhere else, give or take a few exceptions. By using standards and standardized interface definitions, it is easier to use and reuse these interfaces. This statement means that both the standardized technical interfaces as well as a shared data definition model can help. Particularly the latter helps in a semantic way - understanding the interface is made a lot easier. This results in quicker adoption of the interface and hence its used standards. Even the shared data model can be according to industry  standards. Using these industry standard data models typically requires a lot more ramp-up time and implementation time due to the learning curve, but once understood, they can be discussed about with people from ie. other companies, without their prior knowledge of your company - this could even make it easier to hire external help to finish your projects.
Shared services over specific-purpose implementations
When focussing on specific-purpose implementations, these often very easily delivered. Issue with the approach is that for every new business functionality requested, a new tailored service is created. This causes that no reuse happens: services are not shared. A typical approach is where service logic is copied, and adapter for a slightly different service. Although quicker to develop, the use is minimal if not negligible, you still increase the problem operational complexity whereas a shared service typically requires no modification at all, hence must be managed only once. Note that to work with shared services, considerable extra design/development time must be incorporated to cope for service discoverability and description of the service interaction in the context of new functionality. Then you have the issue of "i'm special", "not invented by me" syndrome. These are often symptoms of arrogance or distrust and are killing to any organization. When getting over these issues organizations allow for use of shared services, reducing the total cost of ownership as no custom designed implementations have to be managed.
Flexibility over optimization
What happens very often, is that logic is optimized for the current scope. This would typically be the case for any project-based approach. Any service or capability delivered often is a service with limited scope from project point of view. However from enterprise point of view it may be very wise to add some flexibility (at the cost of project budget and timelines). Don't think this will come for free. Issues like up-front planning and rethinking design from multi-use point of view may take extra time during design. Often the implementation time is not too severely impacted. For example, a service may be built to perform work for an online shop application. But this typically locks the service to be useful for the online shop channel only. If we were to foresee that this service can be used by other channels, or in other service compositions, we could cope for the future flexible use by ie. as simple as adding channel ID as a parameter, even though we do not need it for the current project. Any next project or service composition could use the service in a channel-aware fashion.
Evolutionary refinement over pursuit of initial perfection
Reaching initial perfection is probably utopia. Most SOA implementations are complex systems and all the ins and outs are learned over time. Trying to reach it when starting with a new solution will delay projects, cost a lot of money and ultimately may not fly at all because the cost of this is so high, that the business forces us to look for other alternatives. Remember, the business pays the bills :). Initial perfection is hard to reach anyway, if not unrealistic because it would require that all requirements, including future requirements, must be known at the start of the SOA initiative. As requirements change over time, the 'perfect' solution would grow out of sync with requirements soon. This would mean you spent a lot of effort on things that may never happen. Perhaps a better approach is to use KISS: "Keep It Small And Simple". Start small and be pragmatic. Do what you can now and leave what you can't until you can resolve in future iterations or projects. This is what is being referred to as evolutionary refinement

As you can see the value statements in the SOA manifesto are not defined in a "SMART" way. They are not measurable and this causes confusion and leaves room for discussion. Discussion is good however. As long as it does not stop us in our tracks because we cannot resolve our differences, there is progress and it forces us to think about what we are doing. Not one implementation of a SOA will be the same. There is no "one size fits all" solution. If this were the case we would be out of jobs. Use the statements made here to your own benefit when you need them and make up your own mind if you need to. Nothing stated here is "the unified truth for all", just use these statements as best practices whenever you find fit.