Part I of this series observed that 2006 was the year in which many large-scale SOA projects were kicked off in medium and large enterprises. Which means that an all-encompassing methodology that could evolve SOA to the enterprise was needed. Much has been written about service analysis and design for SOA but not so much about methodologies for creating an all-encompassing SOA, including the infrastructure - and, after all, it's the SOA infrastructure that supports service interactions and agility through the change management of services and metadata.

I posted in my first article that SOA derives its powerful complexity management traits from the tried and true divide-and-conquer approach to problem solving. I then set about trying to distill a "little-m" analysis method for en masse SOA enablement guided by this principle, which mines infrastructure requirements by extending existing service analysis methods, most notably IBM SOMA. Analysis results were said to be a set of system requirements and a set of infrastructure requirements, products of the same business domain analysis (and perhaps re-engineering) iterations.

System requirements in SOA specify applications and supporting business and component services while infrastructure requirements specify non-functional aspects, including all metadata management facilities. Part I focused exclusively on the creation of the infrastructure requirements set (the "enablement" of the enterprise for SOA), reflective of an overarching vision of business agility and efficiency. Also derived were three crucial data sets that will assist design greatly: A set of enterprise assets that should be reused/repurposed, a prioritization of infrastructure elements and services needed to expedite desired business functionality, and a view of the consumer base of the SOA and what it will take to rewire them to the new SOA.

We also have in hand (as part of the requirements set) a system change taxonomy...a categorization and specification of the metadata that must be changed in the new SOA, and its velocity, to support the business's overarching vision of agility. Inasmuch as Part I accepted service analysis methods axiomatically, this article focuses on an infrastructure creation method and only references service design techniques, though ensuring that services perform well and are policy-compliant is within the purview of any SOA infrastructure.

SOA analysis and design is iterative. Once service and infrastructure specification reaches a point where the requirements sets are specified enough ("enough" very much depends on the specific enterprise and level of complexity), then design of specific services and SOA infrastructure can join the iterations, with service design following infrastructure design to the extent that services depend on the infrastructure specification. The aptness of the divide-and-conquer paradigm for SOA enablement shows up all the more in the design phase. In fact, with respect to the infrastructure it can be said that it's the overarching design principle.

SOA standards and best practices like loose coupling and agility via metadata management mean that a meet-in-the middle approach to design can be taken as long as strong governance is in place to ensure that all parties adhere to specifications, standards, and best practices, and that everyone and every task is accountable. Being able to divide up complex design tasks into manageable subtasks and build out the SOA infrastructure while also building core services, and even starting an application on top of it all is the core competency of en masse SOA design.

I will now layout a series of broad work steps that should lead you to the end-point of having a future-state architecture plan and associated roadmap in hand. And I will conclude by folding in build-and-test cycles.

Create an SOA Design Center
It stands to reason that before we can send architects, developers, managers, and administrators off in different directions to design (and/or repurpose) our enterprise infrastructure components for SOA we need a way for individuals and teams to collaborate and align as they advance. Creating an SOA design center (SDC), an extension of the tried and true SOA center of excellence that adds collaboration and oversight capabilities is the way to achieve this.

In its collaboration aspect the SDC serves to allow team members to quickly find and exchange information crucial to their efforts. This will be relevant not only for infrastructure designers, but for service developers, consumers, and analysts alike. In its oversight aspect, the SDC serves as the crossroads of the project management and enterprise architecture disciplines. In a divide/conquer approach to a large enterprise build-out, it's not possible that all design decisions route through the old enterprise architecture council for approval.

Yet, governance of the design process is no less important. Likewise, project managers can't be expected to capture such an effort in a monolithic project plan, even though the management discipline is just as important. By allowing the design process to de-federate from strictures that artificially impede it you'll both empower the designers and the governance regime. In a sense, it's analogous to the governance of services themselves - by separating services from policy enforcement we empower both service designers and policy administrators. By eliminating hurdles and gates that would otherwise impede architects as they design and specify infrastructure elements, progress is quicker, even while oversight and administration is better.

How you implement the SDC and what tools you use are entirely dependent on the scope of your effort and your enterprise's architectural governance sensibilities. Seed the SDC with the asset map and change taxonomy you created in the analysis phase, and use it as the storehouse for design artifacts including the future-state architecture plan and roadmap. Consider coupling the SDC with a service design repository and solution repository to render a complete view of how your enterprise will meet its business objectives.

One caveat: Sending designers off in different directions on the premise that fewer strictures will empower them as they design individual piece parts of the larger SOA means that there's great onus on the testing regime to ensure the sum of the parts meets the holistic enterprise requirements you have derived.

Infrastructure Chemistry & Preliminary Design Choices
When you chose to organize your enterprise around services and their assembly under a metadata-driven management regime to build business solutions you are subscribing to a well-defined infrastructure reference model. The SOA infrastructure reference model has matured over the last couple of years, and continues to mature, but it has its roots at the service protocol level where policy is enforced by intermediaries against XML data, most often in SOAP headers and payloads. Everything in the reference model is about ensuring that the right services accurately answer the right requests with the right policies having been applied accurately and within the expected timeframe. What's "right" is communicated in the form of metadata, administered in the SOA infrastructure. Achieving such runtime governance of the service-based enterprise, and especially metadata change management facilities, is the major thrust of the SOA infrastructure design effort.

By mixing in some basic design assumptions and analysis artifacts from the infrastructure requirements set, concrete design choices are precipitated from the reference model, just like adding a precipitant to a solution in a chemistry lab experiment. For example, if you know from analysis that your business services will comprise new component data services and legacy EAI services, you might specify an SOA data services platform and an ESB for orchestration and legacy integration. On the other hand, if you're only creating business services from a portfolio of SOAP Web Services that are WS-I-compliant, you may want to pencil in a pure-intermediary governance solution with a standalone orchestration environment. Start by capturing these design decisions on a whiteboard and publish them as schematics to the SDC when they've matured from the "what if we did this" point to the "we should probably do this" stage. Feel free to publish several competing design choices...this is a best practice that focuses attention on the relevant pros and cons of different design concepts. Consider holding a preliminary design review (or several...or weekly) with the enterprise architecture team to exchange ideas verbally. The process of iterating over competing design choices (and systematically eliminating entire choices that don't measure up over time) should lead you to a small number of preliminary designs (or perhaps one).

For example, if you begin to factor in security policy enforcement requirements you might add policy enforcement points, agents, and a policy repository to your design choices. This might cause you to devalue a monolithic ESB-centered design and favor an infrastructure designed around intermediaries and policy management. Likewise, if your policy enforcement requirements are light, you might favor the ESB-centered design more. Note that you shouldn't make specific technology decisions at this point; evolving the preliminary design into a detailed design and making associated technology decisions includes several additional factors.

Repurpose, Rehash, and Reinvigorate
You might recall that a key part of the analysis phase for en masse SOA enablement was to identify existing enterprise assets that should be included in the new SOA. This includes both services (maybe not compliant Web Services) and enterprise aspects like identity and access management systems. This list needs to be segmented into assets that can "easily" participate in the new SOA - they're easy to include if a new version exists, or a delegate or mediator can be added so that they can quickly meet your SOA service or metadata standards - and those that aren't readily repurposed. Examples of easily repurposed assets are databases that can be introspected and service-enabled quickly or EAI-type interfaces that can be facaded by an ESB.

Notice I said easily, not cheaply...a value judgment will have to be made about what to bring in and at what cost. This is an example of a decision that should bubble up through your SOA design center and get kicked around. Other assets that were tagged for reuse may not be so easy to get at. They'll have to be reinvigorated by changing their core aspects or otherwise engaging in a project to repurpose them. Examples of such problematic assets include business rule repositories and policy repositories that may be scattered across the enterprise and that may need consolidation, upgrade, or outright replacement. Each preliminary design choice should clearly identify the assets being reused, even if they appear as a black box, for example, "SAML Token Provider," to encapsulate your identity management system. And over the course of design iterations, feel free to rehash the decision of reusing assets if it's clear it adds risk or undue cost. Such rehashing is vital feedback for the more advanced analysis iterations that are feeding design.

Create Ontologies of Change
From the analysis phase, one of the key artifacts of the infrastructure requirements set was the change taxonomy, a categorization of everything in the system that has to change (and at what speed) to accommodate the overarching business agility vision of the enterprise. SOA agility is easily understood in the abstract, e.g., "we could increase revenue and profit if we could interact with partners more easily," but it's impossible to do system design from such nebulous statements. For this reason, the analysis phase is particularly interested in enumerating the metadata that has to change for the system to be agile: rules, policies, service contracts, service compositions, transforms, etc.

The design phase should further this effort by creating ontologies of change - abstract models that detail who changes what metadata in the system and what effect the change has. Even though these models will be abstract at first, it will be apparent what changes are needed in the infrastructure to meet the requirements. By way of example, if a requirement has been expressed for security administrators and business managers to coordinate on managing federation partner access control policies, an ontology should be developed that includes all three actors (security administrator, business manager, and partner) and that includes system elements with which they interact to manage partner access.

The ontology should include a functional use case that captures the system interactions and particularly the system response time. If that system is expected to support partner onboarding a day from certificate exchange and suspension within one minute of revocation, this should be detailed in the functional use case. Such agile partner management requires a more intricate infrastructure design than a much slower (and perhaps manual) response.

Once the ontology has been fleshed out, write a "day in the life" scenario for each and every participant that the system will impact, publish it to the SDC, and solicit feedback from those involved from the business and technical teams. Once the ontology is finalized, it can be folded into the preliminary design choices and the infrastructure expanded to accommodate it, if need be. Look for synergies between ontologies to lessen the impact on the infrastructure of accommodating change, e.g., a single metadata repository to handle all service policies or a single administration portal for metadata management.

Integrate, Compose, Transform, Mediate
Once design is complete to the extent that it's clear how services are to be assembled and governed as dictated by metadata managed at sufficient change velocity to be agile, an integration view of the design can be considered. Such a view is concerned with the edge conditions in the enterprise that break the nice whiteboard view of the architecture where integration is as easy as drawing lines between boxes. The reality is often much different. Not only might the SOA infrastructure be expected to orchestrate services of differing protocols, standards, interpretations of standards, operations, and schemas, but it might even have to bridge different message exchange patterns (synchronous/asynchronous). Metadata formats may also need to be bridged, e.g., security token formats, access control policy formats, or service compositions. Designing for mediation is especially important in large enterprises where silo'd applications and service domains are most prevalent, along with entrenched enterprise service groups who provide security, management, and other orthogonal functions. And preparing to integrate new domains under a merger/acquisition scenario, or just expanding to new lines of business will serve to future-proof the infrastructure design.

As described in the analysis phase, it may also be necessary to bridge data domains by transforming and/or composing sub-domain service data into a unified data service. For example, a 'getCustomer' service could well comprise several sub-domain services by composition and a transform to a canonical form. As a best practice in SOA infrastructure design, abstract and encapsulate technical details for all forms of integration to the extent possible. For data integration this would mean leveraging an EII platform, where details of 'getCustomer' and the like can be managed as a black box. Ensure the design meets integration requirements, both implicit/future as well as those enumerated in the analysis phase.

Design for Consumption
A key topic of en masse SOA analysis was the focus on service consumption and how to ease the transition of service consumers away from local policy enforcement, and how to ease service and metadata changes in the UI and federation tiers. It's an unfortunate fact of life for user interfaces that authorization code is usually kept in portal repositories and, worse yet, hard-coded in the backing code itself. And service interaction code and screens are usually no more refined themselves. When migrating the enterprise to SOA, we can't hope to achieve agility and enhance reuse without easing the plight of interface designers and developers. We can go a long way towards achieving this end by accomplishing two main objectives:

1. Implement centralized UI policy management
2. Create reusable business service consumers

Step one in achieving these objectives is to create and implement a portal strategy. The maturity of the implementation of remote portlet standards (specifically WSRP) makes it practical and advisable to de-federate portlet design, development, and production, enabling the end-point aggregation of the portlets for specific applications. WSRP is an XML language with a SOAP protocol binding whose portlet producer interfaces are expressed with WSDL. By putting an intermediary between the WSRP producer and the consumer application, content-aware authorization policies can be enforced, as well as aspects like SLA monitoring and management. Leveraging the SOA infrastructure to enforce fine-grained, content-aware access control policies not only lessens the burden on UI developers but greatly eases the burden of changing policies.