Welcome!

Microservices Expo Authors: Pat Romanski, Elizabeth White, Steve Wilson, Stackify Blog, Derek Weeks

Related Topics: Containers Expo Blog, Microservices Expo, @CloudExpo

Containers Expo Blog: Blog Post

Managing Data Integrity in SOA and SaaS Based Environments

Techniques for managing transactions in the cloud

Data integrity is one of the most critical elements in any system. Data integrity is easily achieved in a standalone system with a single database. Data integrity in such a system is maintained via database constraints and transactions. Transactions should follow ACID (Atomicity, Consistency, Isolation, Durability) properties to ensure data integrity. Most databases support ACID transactions and can preserve data integrity.

Next in the complexity chain are distributed systems. In a distributed system, there are multiple databases and multiple applications. In order to maintain data integrity in a distributed system, transactions across multiple data sources need to be handled correctly in a fail-safe manner. This is usually done via a central global transaction manager. Each application in the distributed system should be able to participate in the global transaction via a resource manager. This is achieved using a 2-phase commit protocol as per the XA standard. Most databases and custom applications have the ability to participate in a global transaction. Many packaged applications can also participate in a global transaction via EAI adapters. In reality, in most environments, some of the applications may support participation in a global transaction via 2-phase commit, some may support only single phase commit transactions and some may not support any transaction capability at all.

Moving further up in complexity are distributed systems with a mix of on-premise and partner applications. In this case, not all applications in the system are under the control of the organization and partner application interface may not support XA. B2B integration standards such as EDI and ebXML are the primary methods of ensuring reliability and data integrity across partner systems.

Enter the world of SOA and Cloud computing, and the problem of data integrity gets magnified even more, as there is a mix of on-prem and SaaS applications exposed as services. SaaS applications are multi-tenant applications hosted by a third party. SaaS applications usually expose their functionality via XML based APIs over HTTP protocol. SOAP and REST based web services are the most common methods of implementing these APIs. Also, in SOA based environments, many on-prem applications expose their functionality via SOAP and REST web services as well. One of the biggest challenges with web services is transaction management. At the protocol level, HTTP doesn’t support transactions or guaranteed delivery, so the only option is to implement these at the API level. Although there are standards available for managing data integrity with web services such as WS-Transaction and WS-Reliability, these standards are not yet mature and not many vendors have implemented these. Most SaaS vendors expose their web services APIs without any support for transactions. Also, each SaaS application may have different levels of availability and SLA (Service Level Agreement), which further complicates management of transactions and data integrity across multiple SaaS applications. There are several techniques that can be applied to ensure data integrity in such environments.

Let’s take a simple scenario of new customer creation at a company. This company uses 2 SaaS vendors, one for Marketing and one for CRM. In addition, there is an on-premise ERP application. When a new customer places an order, the customer information needs to be sent to the Marketing service (for marketing campaigns), CRM service (for customer management) and ERP application (for order fulfillment). Both Marketing and CRM applications expose their customer creation APIs via SOAP web services over HTTP, whereas the ERP application exposes customer creation via a database API. Here is the sequence of operations in this transaction:

1. Customer creation in Marketing via SOAP web service (Doesn't support transaction)

2. Customer creation in CRM via SOAP web service (Doesn’t support transaction)

3. Customer creation in ERP via database insert (Supports transaction)

In order to maintain data integrity across the 3 applications, either all the steps should get successfully executed or none of them should get executed. In the above sequence of operations, if step 1 succeeds but step 2 fails, step 1 can’t be rolled back. If step 1 and 2 succeed but step 3 fails, steps 1 and 2 can’t be rolled back. So we have a data integrity issue at hand in various failure scenarios and customer record will exist in some systems but not in others. This is usually not acceptable in any production environment. So what can be done to handle this problem? There are several techniques that can be applied in this scenario:

Technique 1: Perform the operations that support transactions before the operations that don’t support transactions

In our example, step 3 should be moved to the beginning as follows:

1. Customer creation in ERP via database insert (Supports transaction)

2. Customer creation in Marketing via SOAP web service (Doesn't support transaction)

3. Customer creation in CRM via SOAP web service (Doesn’t support transaction)

With this change in the sequence of operations, if step 1 succeeds but step 2 fails, step 1 can just be rolled back. We still have a problem if step 1 and 2 succeed but step 3 fails. This is where the following techniques come in handy.

Technique 2: Use compensating transactions

In our new sequence as per technique 1, if steps 1 and 2 succeed but step 3 fails, rollback step 1 and issue a compensating transaction for step 2. Compensating transaction in this case will be to delete the customer. Of course, for this to work, the Marketing SaaS application needs to provide a “delete customer” API which should be requested before signing up with this SaaS vendor.

Technique 3: Break the transaction into multiple decoupled transactions

In our example, step 3 can be executed in a separate asynchronous transaction using a queue. Queue can be implemented using database or some messaging technology such as JMS. In either case, both write and read of messages from queue will support transactions. Here is the sequence of operations with this change:

First transaction:

1. Customer creation in ERP via database insert (Supports transaction)

2. Post message to a queue for customer creation in CRM (Supports transaction)

3. Customer creation in Marketing via SOAP web service (Doesn't support transaction)

In the above sequence, if step 2 fails, step 1 can be rolled back and if step3 fails, steps 1 and 2 can be rolled back. Note that posting message to queue is done before customer creation in Marketing to make sure the step that doesn’t support transaction is executed last (as per Technique 1).

Second transaction:

1. Queue listener retrieves message from queue (Supports transaction)

2. Customer creation in CRM via SOAP web service (Doesn't support transaction)

In the above sequence, if step 2 fails, step 1 can be rolled back.

So by breaking a transaction into multiple smaller transactions separated by queues, we are able to achieve data integrity.

Technique 4: Execute the transaction as a long-running transaction

If all the steps of the transaction are orchestrated as separate tasks of a long-running process using a state machine or BPM (Business process management) tool, then failure at any step will result in the process not progressing to the next step. Retries can be introduced at every step to ensure that every step is successful before the whole process is finished. This is the most reliable technique of all the techniques discussed but this can also introduce latency as the process can take a long-time to finish if any application or service is down for a long-time. This solution introduces more complexity into the environment and may not be acceptable in all situations but this is also the most reliable way to design distributed transactions in services based environments.

By applying the techniques discussed in this article, most failure scenarios can be handled effectively so that data integrity is not compromised. These techniques can be applied to any distributed system but are most useful (and almost mandatory) in SOA and SaaS based environments where interfaces are exposed via web services.

More Stories By Vinay Singla

Vinay Singla is a senior technology professional with extensive experience in the SaaS and SOA space.

@MicroservicesExpo Stories
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, discussed how to use Kubernetes to set up a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. H...
When you focus on a journey from up-close, you look at your own technical and cultural history and how you changed it for the benefit of the customer. This was our starting point: too many integration issues, 13 SWP days and very long cycles. It was evident that in this fast-paced industry we could no longer afford this reality. We needed something that would take us beyond reducing the development lifecycles, CI and Agile methodologies. We made a fundamental difference, even changed our culture...
As many know, the first generation of Cloud Management Platform (CMP) solutions were designed for managing virtual infrastructure (IaaS) and traditional applications. But that’s no longer enough to satisfy evolving and complex business requirements. In his session at 21st Cloud Expo, Scott Davis, Embotics CTO, will explore how next-generation CMPs ensure organizations can manage cloud-native and microservice-based application architectures, while also facilitating agile DevOps methodology. He wi...
Docker is sweeping across startups and enterprises alike, changing the way we build and ship applications. It's the most prominent and widely known software container platform, and it's particularly useful for eliminating common challenges when collaborating on code (like the "it works on my machine" phenomenon that most devs know all too well). With Docker, you can run and manage apps side-by-side - in isolated containers - resulting in better compute density. It's something that many developer...
These days, change is the only constant. In order to adapt and thrive in an ever-advancing and sometimes chaotic workforce, companies must leverage intelligent tools to streamline operations. While we're only at the dawn of machine intelligence, using a workflow manager will benefit your company in both the short and long term. Think: reduced errors, improved efficiency and more empowered employees-and that's just the start. Here are five other reasons workflow automation is leading a revolution...
We have Continuous Integration and we have Continuous Deployment, but what’s continuous across all of what we do is people. Even when tasks are automated, someone wrote the automation. So, Jayne Groll evangelizes about Continuous Everyone. Jayne is the CEO of the DevOps Institute and the author of Agile Service Management Guide. She talked about Continuous Everyone at the 2016 All Day DevOps conference. She describes it as "about people, culture, and collaboration mapped into your value streams....
Cloud adoption is often driven by a desire to increase efficiency, boost agility and save money. All too often, however, the reality involves unpredictable cost spikes and lack of oversight due to resource limitations. In his session at 20th Cloud Expo, Joe Kinsella, CTO and Founder of CloudHealth Technologies, tackled the question: “How do you build a fully optimized cloud?” He will examine: Why TCO is critical to achieving cloud success – and why attendees should be thinking holistically ab...
Docker is on a roll. In the last few years, this container management service has become immensely popular in development, especially given the great fit with agile-based projects and continuous delivery. In this article, I want to take a brief look at how you can use Docker to accelerate and streamline the software development lifecycle (SDLC) process.
We define Hybrid IT as a management approach in which organizations create a workload-centric and value-driven integrated technology stack that may include legacy infrastructure, web-scale architectures, private cloud implementations along with public cloud platforms ranging from Infrastructure-as-a-Service to Software-as-a-Service.
Did you know that you can develop for mainframes in Java? Or that the testing and deployment can be automated across mobile to mainframe? In his session and demo at @DevOpsSummit at 21st Cloud Expo, Dana Boudreau, a Senior Director at CA Technologies, will discuss how increasingly teams are developing with agile methodologies, using modern development environments, and automating testing and deployments, mobile to mainframe.
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory?
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single-threaded, you can effectively identify hot spots in your serverless code. In his session at @DevOpsSummit at 21st Cloud Expo, Dave Martin, Product owner at CA Technologies, will give a live demonstration and code walkthrough, showing how ...
There are several reasons why businesses migrate their operations to the cloud. Scalability and price are among the most important factors determining this transition. Unlike legacy systems, cloud based businesses can scale on demand. The database and applications in the cloud are not rendered simply from one server located in your headquarters, but is instead distributed across several servers across the world. Such CDNs also bring about greater control in times of uncertainty. A database hack ...
@DevOpsSummit at Cloud Expo taking place Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center, Santa Clara, CA, is co-located with the 21st International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is ...
API Security is complex! Vendors like Forum Systems, IBM, CA and Axway have invested almost 2 decades of engineering effort and significant capital in building API Security stacks to lockdown APIs. The API Security stack diagram shown below is a building block for rapidly locking down APIs. The four fundamental pillars of API Security - SSL, Identity, Content Validation and deployment architecture - are discussed in detail below.
With Cloud Foundry you can easily deploy and use apps utilizing websocket technology, but not everybody realizes that scaling them out is not that trivial. In his session at 21st Cloud Expo, Roman Swoszowski, CTO and VP, Cloud Foundry Services, at Grape Up, will show you an example of how to deal with this issue. He will demonstrate a cloud-native Spring Boot app running in Cloud Foundry and communicating with clients over websocket protocol that can be easily scaled horizontally and coordinate...
In his session at 20th Cloud Expo, Scott Davis, CTO of Embotics, discussed how automation can provide the dynamic management required to cost-effectively deliver microservices and container solutions at scale. He also discussed how flexible automation is the key to effectively bridging and seamlessly coordinating both IT and developer needs for component orchestration across disparate clouds – an increasingly important requirement at today’s multi-cloud enterprise.
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
IT organizations are moving to the cloud in hopes to approve efficiency, increase agility and save money. Migrating workloads might seem like a simple task, but what many businesses don’t realize is that application migration criteria differs across organizations, making it difficult for architects to arrive at an accurate TCO number. In his session at 21st Cloud Expo, Joe Kinsella, CTO of CloudHealth Technologies, will offer a systematic approach to understanding the TCO of a cloud application...
Most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes a lot of work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reduction in cost ...