Microservices Expo Authors: Liz McMillan, Pat Romanski, Carmen Gonzalez, Elizabeth White, Jason Bloomberg

Related Topics: Microservices Expo, Industrial IoT

Microservices Expo: Article

SOA to Reduce Complexity?

SOA has matured and tools have been developed to cope with complexity, complexity itself has become less of an issue

Whenever IT professionals discuss the benefits of SOA - full utilization of IT resources, reduced infrastructure costs, and the agility to deliver new services quickly - they usually issue a caveat. They warn potential adopters of the inherent complexity of SOA, and point out that unless SOA is properly managed and governed, it's apt to fail.

It's a valid concern, which is why I recommend that IT professionals seek a SOA solution that combines SOA governance, quality, and management.

But as SOA has matured and tools have been developed to cope with complexity, complexity itself has become less of an issue. In fact, here at HP, we've discovered something that may surprise IT managers who implement SOA to gain the usual benefits. When it comes to integrating large, complex IT environments, SOA has a significant additional benefit: It actually reduces complexity. And we use it for that purpose in integrating our comprehensive suite of IT management solutions by substituting business services for point integrations.

The HP IT Management Portfolio: A Use Case
To help our customers take a lifecycle approach to IT management, we use SOA to integrate a set of software that includes solutions for portfolio management, SOA governance, quality assurance, application management, business service management, IT service management, and business service automation.

In the process, we've encountered and solved a problem many of our large enterprise customers face - the overwhelming complexity of using traditional enterprise application integration (EAI) techniques.

The story of how we integrate our software portfolio is a compelling use case for SOA as both an example and a set of best practices for integrating large complex environments. Using our own SOA software solutions, we've substituted SOA-based services for EAI-style hub-and-spoke integrations. And in the process, we've given our developers best practices, visibility into the inherent complexity of integrated resources, and a mechanism for putting everyone's expectations on the table - time, cost, resources consumed, and impact on service levels.

Standards Are Not Enough
Before I go into detail about how we apply SOA concepts to integrations, I'd like to comment briefly on the role of standards. Standards are important in EAI, since the more standards-based systems are the less work needs to be done writing APIs that make disparate systems talk to one another.

And while this is extremely important, it's not sufficient to overcome the complexity of large-system integration. Using SOAP, SWDL, Atom, or similar standards can help systems interface with systems, but it doesn't make services more interoperable or useful. SOA processes and solutions add essential factors such as shared guidelines on how to externalize data using these standards and a way to express identifications, versions, business taxonomies, and relationships. It's very easy to serialize a Java object into a custom piece of XML code and then put it in a SOAP envelope. But the result can still be chaos if the process isn't guided by effective architectural governance.

In short, standards are essential, but standards aren't enough. They help systems talk to systems, but they don't do enough to help systems work well with other systems. They don't supply agreements on the meanings of terms, rules for data storage and format, or contracts that govern the interaction of organizations. And SOA does.

The Importance of Contracts
SOA brings several advantages to the process of integration, but if I had to pick one as the most important, I would point to contracts. The whole concept of SOA is based on business services, and services are based on contracts. Unfortunately, most people consider contracts in SOA to be based simply on the technical agreement at the service-interface or message-format level. But a real contract is much more. A SOA contract is an agreement between the provider of the service and the consumer of the service - an agreement that covers all the important aspects of the relationship. Examples include the following, although there are many more aspects that can and should be described in the contract:

  • When will the service be available?
  • What is the expected performance of the service?
  • Are there limits on the number of times I can call the service?
  • Who is responsible for correcting faults?
  • What is the change process?
  • What are the escalation procedures?
  • How can I request modifications to the service?

Contracts give visibility into services and their relationships, yield better alignment with business service management, and reveal integrations that might break or otherwise impact seemingly unrelated services. And they capture the roles and the responsibilities of both the consumer and provider.

Contracts are an important part of SOA governance precisely because of the inherent complexity that distributed and loosely-coupled systems create. They are a best practice developed to address and mitigate the problem of complexity. So SOA governance mechanisms work just as well to counter the complexity of enterprise integration, and that's why here at HP we use our own SOA governance approach and software in the process of integrating our portfolio.

SOA puts the business logic where it will be invoked automatically - in the business service rather than buried in an integration server. Contracts yield important benefits while an integration project is going on, and they build in the mechanism for successfully scaling and expanding later. They also support the lifecycle approach by extending visibility across development, testing, and monitoring.

The Importance of Policy Management
Another key benefit that SOA brings to integration is consistency, driven by policy management. Integration projects need to comply with corporate policies for interoperability, security, and performance. In many cases, they also have to comply with external policies imposed by regulations such as SOX and HIPAA. Enabling policy compliance and reporting are inherent capabilities of our SOA governance solution.

A Model to Address the Key Requirements of Integration
Combining the HP lifecycle approach to IT management, the principles of SOA, and the expertise we have built around SOA, HP has developed a model that enables us to integrate components of our portfolio quickly, cost-effectively, and in a way that supports future growth and change.

We take basic factors into consideration before we undertake an integration project, some of which go beyond software architecture:

  • Overall architecture
  • Solutions integration
  • Standards
  • Service model governance
  • Data model governance

Our working version of this model addresses all five factors by providing governance, a roadmap, communication, visibility, and alignment between our business units and our R&D organization. It also gives us a methodology, key services, and standards. And it helps our teams agree on priorities, roles, and responsibilities; select tools and define architecture styles; and agree on how services will be architected, described, and published.

Under this model, every integration effort is split into two parts: delivery and governance. The integrations are governed centrally by our Applications Strategy Team (AST), which is our SOA Center of Excellence. The actual delivery of the integrations remains in the hands of our Product Strategy Teams (PSTs) and are handled just as if they were any other product feature. This separation of roles allows for scalable execution of large portfolios of integrations.

Business Requirements First
Within this framework, we first deal with the "why" and "what," and then with the "how." We always try to design our integrations to minimize middleware infrastructure requirements. Even sending one message from service A to service B might require sophisticated middleware if the services have poor interfaces or inappropriate interaction options. On the other hand, well-designed services require nothing more than the ubiquitous Internet infrastructure. So we avoid the kind of high-level use case analysis often seen in EAI projects that contain long lists of low-level requirements containing WS-* acronyms.

While low-level requirements become critical at later stages of the lifecycle, we start from a different place. When defining solutions that span multiple products, we first define the use cases and agree on priorities of subordinate use cases (which are often individual integrations or product enhancements). These use cases help us define services and basic interaction patterns. At the service and interaction level, we insist on formal descriptions.

Governance is extremely important at this level. If the high-level decisions are wrong, we're apt to end up with services that are too granular or processes that are overly complicated. Good governance helps us identify and eliminate overlaps, minimize hardware requirements, consolidate databases where necessary, and form solution-level teams that involve business-level stakeholders for clear communication. Once the business requirements are firm, we can map them to the product level, integrating products in a way that addresses overall business requirements rather than simply bringing two systems together.

Combining SOA with EAI
While SOA delivers essential capabilities lacking in traditional EAI practices, we understand that for some integrations, EAI is still necessary. If very tight integration is required, or both applications require their own databases, we might still use EAI techniques. But even when we do, we give the integration a service façade so that we have an integration catalog for governance purposes and compliance reporting.

If your organization is still taking an EAI approach to integrating large, complex environments, and especially if you're experiencing high costs and low responsiveness to business needs, then I recommend that you consider SOA as an alternative approach. It can pay off in increased visibility, better decision-making, flexibility, change resilience, reduced ongoing maintenance, lower cost, and - believe it or not - less complexity.

More Stories By Radovan Janecek

Radovan Janecek is VP, Software Engineering at CA based in Prague, Czech Republic. Formerly he was the Chief Architect for HP Software’s largest product group: Business Technology Optimization. He has been involved in SOA for many years. Most recently, he was responsible for all development for HP SOA Systinet, the leader in the SOA governance market. He was also a key member of the team that founded Systinet, the company that defined the SOA governance space.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

Microservices Articles
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, discussed how to use Kubernetes to set up a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. H...
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, will discuss why containers should be paired with new architectural practices such as microservices ra...
In his session at 20th Cloud Expo, Scott Davis, CTO of Embotics, discussed how automation can provide the dynamic management required to cost-effectively deliver microservices and container solutions at scale. He also discussed how flexible automation is the key to effectively bridging and seamlessly coordinating both IT and developer needs for component orchestration across disparate clouds – an increasingly important requirement at today’s multi-cloud enterprise.
The Software Defined Data Center (SDDC), which enables organizations to seamlessly run in a hybrid cloud model (public + private cloud), is here to stay. IDC estimates that the software-defined networking market will be valued at $3.7 billion by 2016. Security is a key component and benefit of the SDDC, and offers an opportunity to build security 'from the ground up' and weave it into the environment from day one. In his session at 16th Cloud Expo, Reuven Harrison, CTO and Co-Founder of Tufin, ...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In their Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, and Mark Lav...
Many organizations are now looking to DevOps maturity models to gauge their DevOps adoption and compare their maturity to their peers. However, as enterprise organizations rush to adopt DevOps, moving past experimentation to embrace it at scale, they are in danger of falling into the trap that they have fallen into time and time again. Unfortunately, we've seen this movie before, and we know how it ends: badly.
TCP (Transmission Control Protocol) is a common and reliable transmission protocol on the Internet. TCP was introduced in the 70s by Stanford University for US Defense to establish connectivity between distributed systems to maintain a backup of defense information. At the time, TCP was introduced to communicate amongst a selected set of devices for a smaller dataset over shorter distances. As the Internet evolved, however, the number of applications and users, and the types of data accessed and...