|By Anshuk Pal Chaudhari, Bijoy Majumdar, Sunny Saxena||
|February 26, 2007 04:30 PM EST||
Most organizations that have tried have been successful in implementing a pliable Service Oriented Architecture (SOA) paradigm. Analysts have come out with strategies to translate existing applications into SOA-compliant systems using a staggered approach. The rewards reaped come in the form of low-cost maintenance and agility in their business, along with reusable and self-contained services. But there are still challenges in this form of service-based architecture and solutions need to be devised.
One of the biggest hurdles has been coordinating technology-agnostic services into a single long-running unit of work that produces predictable results. The transactions running across multiple services over multiple domains need to be synchronized to maintain business integrity. Currently organizations depend on proprietary solutions to coordinate transactions for data consistency. This article will walk you through the definition of long-running transaction in SOA and its challenges then talk about the various approaches to resolving the issue while retaining the characteristics of a service-based architecture.
Applications utilize multiple services across different modules or layers to serve a particular business need. For example, security authentication, service information, EIS information, updating services need to coordinate in a business unit termed a transaction thatcomprehends data consistency and business integrity in an organization.
Transactions are a set of operations that must be executed entirely or not at all. The fault-tolerance mechanism of managing transactions is to maintain the so-called ACID properties: A - Atomicity (all or none), C - Consistency (the resource must start and end in a consistent state), I - Isolation (make the transactions appear isolated from all the other operations) and D - Durability (once notified, the transaction will persist). ACID provides concurrency in operations and retains data integrity.
ACID properties are easier to implement on transactions that run only a short time because during a transaction the resources are held in a locked state. Transactions that run for a long time can't afford to lock up resources. Till date, an ACID transaction assumes closely coupled systems that aren't an SOA-mandated environment. So the ACID properties in a long-running activity need to be applied so that locking doesn't occur, or if it does, then the duration of the locking is as short as possible.
Long-Running Transactions in Service-Crowded Systems
To understand the concept of a long-running transaction, we need to look first into the various lifetimes of a transaction. A transaction lifetime can be defined as the minimum amount of time a transaction is kept open. This time period can be anywhere from a few seconds to several hours. A transaction with a short lifetime can begin and end in a matter of seconds, while a long-running transaction can be alive for minutes, hours, even days depending upon the underlying business requirements and implementation. Transactions with a short lifetime are easy to handle since the resources they use can be locked for the time required by maintaining the ACID properties. But the same strategy can't be applied to long-running transactions. Locking up resources for a long time can seriously hamper the application's performance bringing in unnecessary deadlock situations and long wait-times. Any transaction left in an open state for an indefinite amount of time qualifies as a long-running transaction.
The following scenarios make long-running transactions possible:
- A transaction with lots of database queries
- B2B transactions
- Batch processes
- Pseudo-Asynchronous activities within a transaction
Batch processes run for long periods of time, usually for hours. Regularly backing up sensitive data is an example. In most cases, batch processes only involve reading data and hence not many transactional issues are encountered. But in certain cases these long batch processes can include modifying the data. A failure during that operation would require an equally long rollback process.
Pseudo-asynchronous activities are used in concurrent activities but the transactions are resumed at some kind of notification. Such operations can be trivial to handle as the control is passed on completely and there is a complex or no way back to reach the sender once the activity is completed. This results in a complex scenario where an independent or intelligently handled rollback needs to be initiated on the source.
In a SOA each functionality is separated as a service. So, a certain application may use many services to provide a defined functionality. The principles of SOA define services as separate platform- and system-independent entities - each capable of functioning on their own, thus providing reusability and interoperability across disparate platforms.
A long-running transaction creates a number of problems in a SOA architecture. As long as a transaction is limited to a closed environment, catching faults or exceptions and triggering the appropriate rollback mechanism can easily be defined in the underlying application architecture. For example, a transaction involving a database as a resource would already have mechanisms defined in it to handle errors and do rollbacks. Even in a distributed database environment these things can be taken care of. Imagine the same situation in an open SOA scenario where each transactional query is executed on an altogether different platform or system. How a rollback would be implemented in such a case is just one of the immediate questions that comes to mind.
Let's consider a scenario where the transaction involves the participation of three different services - each performing a particular operation. Only if all three operations are successful would the transaction be deemed a success. Any other outcome would result in the transaction being marked as a failure. Then, if and when the transaction fails, appropriate recovery measures have to be implemented. And to top it off, we can lock a resource only for the time when the service local to the resource is operating.
Let's look into the problems encountered with long-running transactions in SOA. They can be referred to as failure cases:
- The participation of multiple services results in multiple endpoints being invoked during one cycle of the transaction. Any of these services can be down at the instance when the transaction is in process.
- SOA boasts of loosely coupled systems. Maintaining transactions is only possible in closely coupled systems.
- The services involved can be based on any platform. Because of the disparity among the underlying implementation of the services, a context can't be deployed across the services to manage the transactions.
- The current status in the flow of the transaction can't be known at a given instance.
- Ifasynchronous services are involved in the transaction they can't be reached back, unless the service information is explicitly passed on.
- Resources can't be kept in a locked state for long periods of time. To free a resource once the service is done with it, it must release it. Doing this can cause a problem later on if the service fails and a rollback is issued throughout all the services.
- Alternate methods need to be devised to perform the appropriate recovery operations. In most cases these methods are either platform-specific or too dependent on the underlying business process.
Any methodology that tries to implement transaction management for a long-running transaction scenario in a SOA needs to make sure to:
- Uniquely identify a transaction across the various participating services
- Guarantee that the data is delivered and the notifications are sent
- Some compensation must be provided in case something goes wrong during the transaction
- Errors in asynchronous services have to be addressed
- A compensation methodology
- Transaction coordinator
In an ideal situation any changes done during a long-running transaction must be reverted back to the original content in case there's a failure somewhere else along the flow of the transaction. This is precisely what happens in a closed environment and is known as a rollback. In a SOA architecture, a situation might arise where a rollback isn't an option. In that case, instead of a rollback, compensation is provided. For example, in WS choreography, the self-reliant services pass control messages back and forth to notify the participating services of a rollback operation.
Compensation may be defined as the most logical change applied to the resource to maintain data consistency and integrity. How it's constructed depends on the governing business rules and underlying technical implementation of the services. In certain cases, compensation can include a rollback. In the example above, if the transaction fails at the third service (the transaction is uniquely identified by an id throughout its lifetime), we need to perform a compensatory operation at the previous service to negate the effect caused by the transaction. So, if the second service sent out an e-mail announcing that it has implemented the changes, a compensatory operation would be to send another e-mail announcing the failure of the transaction and that the changes have been undone. A synchronous process showcasing the scenario is illustrated in Figure 1.
But what if the services participating in the process are asynchronous, as one would expect in a long-running transaction? One way would be to save states and service information.
Methodology 2: Transaction Coordinator
A more appropriate solution would be to orchestrate the process using a transaction manager or process coordinator. Instead of inter-service communication, the services would be answerable to the coordinator, which in turn would handle all the transaction and compensation scenarios. Once again the transaction will be uniquely identified throughout the transaction cycle by an id. This would help the coordinator perform compensatory operations on the required set of data. The coordinator can manage the service information as well. This would solve any issues with asynchronous services. Figure 2 illustrates the coordinator service. This kind of methodology is used mostly in service orchestration-type applications and is a more centralized approach unlike methodology 1.
Case study - Money Transfer Scenario
Consider a money transfer scenario (Figure 4) where a complete transaction process involves five different services. All five services are separated by virtue of both system and the language of implementation.
The first service, the initiation service, is exposed to the client to pick up the user input. It validates the necessary input parameters and processes the transaction by sequentially executing the credit service and the debit service. Then the system notifies the stakeholder and the internal logs for auditing.
With no transaction context involved in this processing, the services are executed independently with no knowledge of the member service status. There's no way for the executed services to rollback and for specific reasons:
- Service status isn't shared
- Non-availability of co-ordination federation in the processing
- Compensation services for revoking the services
The coordinator receives the input and generates an id to uniquely identify the transaction. An acknowledgement is sent to the initiation service as RECEIVED. The initiation service notifies the client about the start of the process and provides the unique transaction id. The client can use this transaction id to monitor and track the transaction. The initiation service is now ready to take further client input. The coordinator maintains a log to record each operation it performs. The log is created against the transaction id.
After generating the id for the transaction, the coordinator calls the external service of the bank, which accepts the money. This credit service takes the necessary input and starts updating the records in the database. Depending upon the style of the compensation, state information is saved before the update process initiates. Once the update takes place successfully, an acknowledgement to the coordinator is sent. (Figure 3)
The coordinator then logs the changes and proceeds to call the debit service. The debit service makes the necessary changes to the local database to reflect the debit. The debit process follows the same pattern as the credit process. On successful operation, a DEBITED acknowledgement is sent to the coordinator. The coordinator notifies each service involved of successful individual transactions at each step by means enacts the 2PC execution. When there's a failure, the coordinator runs the compensation service for each activity.
The long-running transaction is designed specifically for business interactions that take a long time. The intention is to tie the logical single business-to-business unit of work across heterogeneous domains. Each methodology depends on the architecture of the system and the existing assets in the organization. Technical analysts need to differentiate such special transaction in the SOA study and deal with them through special defined methodologies.
1. William Cox. "Transactional Web Services."
2. Pat Helland. "Why I hate the phrase Long running Transactions..."
3. Wikipedia Atomic Transactions:
Two-phase commit protocol:
As the software delivery industry continues to evolve and mature, the challenge of managing the growing list of the tools and processes becomes more daunting every day. Today, Application Lifecycle Management (ALM) platforms are proving most valuable by providing the governance, management and coordination for every stage of development, deployment and release. Recently, I spoke with Madison Moore at SD Times about the changing market and where ALM is headed.
May. 2, 2016 08:30 PM EDT Reads: 1,522
Struggling to keep up with increasing application demand? Learn how Platform as a Service (PaaS) can streamline application development processes and make resource management easy.
May. 2, 2016 08:30 PM EDT Reads: 2,129
If there is anything we have learned by now, is that every business paves their own unique path for releasing software- every pipeline, implementation and practices are a bit different, and DevOps comes in all shapes and sizes. Software delivery practices are often comprised of set of several complementing (or even competing) methodologies – such as leveraging Agile, DevOps and even a mix of ITIL, to create the combination that’s most suitable for your organization and that maximize your busines...
May. 2, 2016 07:45 PM EDT Reads: 1,871
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus inter...
May. 2, 2016 06:30 PM EDT Reads: 1,285
The goal of any tech business worth its salt is to provide the best product or service to its clients in the most efficient and cost-effective way possible. This is just as true in the development of software products as it is in other product design services. Microservices, an app architecture style that leans mostly on independent, self-contained programs, are quickly becoming the new norm, so to speak. With this change comes a declining reliance on older SOAs like COBRA, a push toward more s...
May. 2, 2016 06:00 PM EDT Reads: 1,488
From the conception of Docker containers to the unfolding microservices revolution we see today, here is a brief history of what I like to call 'containerology'. In 2013, we were solidly in the monolithic application era. I had noticed that a growing amount of effort was going into deploying and configuring applications. As applications had grown in complexity and interdependency over the years, the effort to install and configure them was becoming significant. But the road did not end with a ...
May. 2, 2016 12:30 PM EDT Reads: 230
You deployed your app with the Bluemix PaaS and it's gaining some serious traction, so it's time to make some tweaks. Did you design your application in a way that it can scale in the cloud? Were you even thinking about the cloud when you built the app? If not, chances are your app is going to break. Check out this webcast to learn various techniques for designing applications that will scale successfully in Bluemix, for the confidence you need to take your apps to the next level and beyond.
May. 2, 2016 12:00 PM EDT Reads: 1,548
Digital means customer preferences and behavior are driving enterprise technology decisions to be sure, but let’s not forget our employees. After all, when we say customer, we mean customer writ large, including partners, supply chain participants, and yes, those salaried denizens whose daily labor forms the cornerstone of the enterprise. While your customers bask in the warm rays of your digital efforts, are your employees toiling away in the dark recesses of your enterprise, pecking data into...
May. 2, 2016 10:15 AM EDT Reads: 1,097
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
May. 2, 2016 06:30 AM EDT Reads: 2,542
Wow, if you ever wanted to learn about Rugged DevOps (some call it DevSecOps), sit down for a spell with Shannon Lietz, Ian Allison and Scott Kennedy from Intuit. We discussed a number of important topics including internal war games, culture hacking, gamification of Rugged DevOps and starting as a small team. There are 100 gold nuggets in this conversation for novices and experts alike.
May. 2, 2016 06:30 AM EDT Reads: 797
With DevOps becoming more well-known and established practice in nearly every industry that delivers software, it is important to continually reassess its efficacy. This week’s top 10 includes a discussion on how the quick uptake of DevOps adoption in the enterprise has posed some serious challenges. Additionally, organizations who have taken the DevOps plunge must find ways to find, hire and keep their DevOps talent in order to keep the machine running smoothly.
May. 2, 2016 06:15 AM EDT Reads: 1,462
Call it DevOps or not, if you are concerned about releasing more code faster and at a higher quality, the resulting software delivery chain and process will look and smell like DevOps. But for existing development teams, no matter what the velocity objective is, getting from here to there is not something that can be done without a plan. Moving your release cadence from months to weeks is not just about learning Agile practices and getting some automation tools. It involves people, tooling and ...
May. 2, 2016 04:30 AM EDT Reads: 1,533
Between the mockups and specs produced by analysts, and resulting applications built by developers, there exists a gulf where projects fail, costs spiral, and applications disappoint. Methodologies like Agile attempt to address this with intensified communication, with partial success but many limitations. In his session at 18th Cloud Expo, Charles Kendrick, CTO & Chief Architect at Isomorphic Software, will present a revolutionary model enabled by new technologies. Learn how business and devel...
May. 2, 2016 04:30 AM EDT Reads: 1,776
The notion of customer journeys, of course, are central to the digital marketer’s playbook. Clearly, enterprises should focus their digital efforts on such journeys, as they represent customer interactions over time. But making customer journeys the centerpiece of the enterprise architecture, however, leaves more questions than answers. The challenge arises when EAs consider the context of the customer journey in the overall architecture as well as the architectural elements that make up each...
May. 2, 2016 03:15 AM EDT Reads: 1,994
In 2006, Martin Fowler posted his now famous essay on Continuous Integration. Looking back, what seemed revolutionary, radical or just plain crazy is now common, pedestrian and "just what you do." I love it. Back then, building and releasing software was a real pain. Integration was something you did at the end, after code complete, and we didn't know how long it would take. Some people may recall how we, as an industry, spent a massive amount of time integrating code from one team with another...
May. 2, 2016 12:30 AM EDT Reads: 975
SYS-CON Events announced today that Stratoscale, the software company developing the next generation data center operating system, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Stratoscale is revolutionizing the data center with a zero-to-cloud-in-minutes solution. With Stratoscale’s hardware-agnostic, Software Defined Data Center (SDDC) solution to store everything, run anything and scale everywhere...
May. 1, 2016 01:30 PM EDT Reads: 1,609
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
May. 1, 2016 12:15 PM EDT Reads: 2,362
Much of the discussion around cloud DevOps focuses on the speed with which companies need to get new code into production. This focus is important – because in an increasingly digital marketplace, new code enables new value propositions. New code is also often essential for maintaining competitive parity with market innovators. But new code doesn’t just have to deliver the functionality the business requires. It also has to behave well because the behavior of code in the cloud affects performan...
May. 1, 2016 02:45 AM EDT Reads: 1,417
This is not a small hotel event. It is also not a big vendor party where politicians and entertainers are more important than real content. This is Cloud Expo, the world's longest-running conference and exhibition focused on Cloud Computing and all that it entails. If you want serious presentations and valuable insight about Cloud Computing for three straight days, then register now for Cloud Expo.
Apr. 30, 2016 02:30 PM EDT Reads: 1,769
I had the opportunity to catch up with Chris Corriere - DevOps Engineer at AutoTrader - to talk about his experiences in the realm of Rugged DevOps. We discussed automation, culture and collaboration, and which thought leaders he is following. Chris Corriere: Hey, I'm Chris Corriere. I'm a DevOps Engineer AutoTrader. Derek Weeks: Today we're going to talk about Rugged DevOps. It's a subject that's gaining a lot of traction in the community but not a lot of people are really familiar with wh...
Apr. 30, 2016 09:45 AM EDT Reads: 1,597