|By Anshuk Pal Chaudhari, Bijoy Majumdar, Sunny Saxena||
|February 26, 2007 04:30 PM EST||
Most organizations that have tried have been successful in implementing a pliable Service Oriented Architecture (SOA) paradigm. Analysts have come out with strategies to translate existing applications into SOA-compliant systems using a staggered approach. The rewards reaped come in the form of low-cost maintenance and agility in their business, along with reusable and self-contained services. But there are still challenges in this form of service-based architecture and solutions need to be devised.
One of the biggest hurdles has been coordinating technology-agnostic services into a single long-running unit of work that produces predictable results. The transactions running across multiple services over multiple domains need to be synchronized to maintain business integrity. Currently organizations depend on proprietary solutions to coordinate transactions for data consistency. This article will walk you through the definition of long-running transaction in SOA and its challenges then talk about the various approaches to resolving the issue while retaining the characteristics of a service-based architecture.
Applications utilize multiple services across different modules or layers to serve a particular business need. For example, security authentication, service information, EIS information, updating services need to coordinate in a business unit termed a transaction thatcomprehends data consistency and business integrity in an organization.
Transactions are a set of operations that must be executed entirely or not at all. The fault-tolerance mechanism of managing transactions is to maintain the so-called ACID properties: A - Atomicity (all or none), C - Consistency (the resource must start and end in a consistent state), I - Isolation (make the transactions appear isolated from all the other operations) and D - Durability (once notified, the transaction will persist). ACID provides concurrency in operations and retains data integrity.
ACID properties are easier to implement on transactions that run only a short time because during a transaction the resources are held in a locked state. Transactions that run for a long time can't afford to lock up resources. Till date, an ACID transaction assumes closely coupled systems that aren't an SOA-mandated environment. So the ACID properties in a long-running activity need to be applied so that locking doesn't occur, or if it does, then the duration of the locking is as short as possible.
Long-Running Transactions in Service-Crowded Systems
To understand the concept of a long-running transaction, we need to look first into the various lifetimes of a transaction. A transaction lifetime can be defined as the minimum amount of time a transaction is kept open. This time period can be anywhere from a few seconds to several hours. A transaction with a short lifetime can begin and end in a matter of seconds, while a long-running transaction can be alive for minutes, hours, even days depending upon the underlying business requirements and implementation. Transactions with a short lifetime are easy to handle since the resources they use can be locked for the time required by maintaining the ACID properties. But the same strategy can't be applied to long-running transactions. Locking up resources for a long time can seriously hamper the application's performance bringing in unnecessary deadlock situations and long wait-times. Any transaction left in an open state for an indefinite amount of time qualifies as a long-running transaction.
The following scenarios make long-running transactions possible:
- A transaction with lots of database queries
- B2B transactions
- Batch processes
- Pseudo-Asynchronous activities within a transaction
Batch processes run for long periods of time, usually for hours. Regularly backing up sensitive data is an example. In most cases, batch processes only involve reading data and hence not many transactional issues are encountered. But in certain cases these long batch processes can include modifying the data. A failure during that operation would require an equally long rollback process.
Pseudo-asynchronous activities are used in concurrent activities but the transactions are resumed at some kind of notification. Such operations can be trivial to handle as the control is passed on completely and there is a complex or no way back to reach the sender once the activity is completed. This results in a complex scenario where an independent or intelligently handled rollback needs to be initiated on the source.
In a SOA each functionality is separated as a service. So, a certain application may use many services to provide a defined functionality. The principles of SOA define services as separate platform- and system-independent entities - each capable of functioning on their own, thus providing reusability and interoperability across disparate platforms.
A long-running transaction creates a number of problems in a SOA architecture. As long as a transaction is limited to a closed environment, catching faults or exceptions and triggering the appropriate rollback mechanism can easily be defined in the underlying application architecture. For example, a transaction involving a database as a resource would already have mechanisms defined in it to handle errors and do rollbacks. Even in a distributed database environment these things can be taken care of. Imagine the same situation in an open SOA scenario where each transactional query is executed on an altogether different platform or system. How a rollback would be implemented in such a case is just one of the immediate questions that comes to mind.
Let's consider a scenario where the transaction involves the participation of three different services - each performing a particular operation. Only if all three operations are successful would the transaction be deemed a success. Any other outcome would result in the transaction being marked as a failure. Then, if and when the transaction fails, appropriate recovery measures have to be implemented. And to top it off, we can lock a resource only for the time when the service local to the resource is operating.
Let's look into the problems encountered with long-running transactions in SOA. They can be referred to as failure cases:
- The participation of multiple services results in multiple endpoints being invoked during one cycle of the transaction. Any of these services can be down at the instance when the transaction is in process.
- SOA boasts of loosely coupled systems. Maintaining transactions is only possible in closely coupled systems.
- The services involved can be based on any platform. Because of the disparity among the underlying implementation of the services, a context can't be deployed across the services to manage the transactions.
- The current status in the flow of the transaction can't be known at a given instance.
- Ifasynchronous services are involved in the transaction they can't be reached back, unless the service information is explicitly passed on.
- Resources can't be kept in a locked state for long periods of time. To free a resource once the service is done with it, it must release it. Doing this can cause a problem later on if the service fails and a rollback is issued throughout all the services.
- Alternate methods need to be devised to perform the appropriate recovery operations. In most cases these methods are either platform-specific or too dependent on the underlying business process.
Any methodology that tries to implement transaction management for a long-running transaction scenario in a SOA needs to make sure to:
- Uniquely identify a transaction across the various participating services
- Guarantee that the data is delivered and the notifications are sent
- Some compensation must be provided in case something goes wrong during the transaction
- Errors in asynchronous services have to be addressed
- A compensation methodology
- Transaction coordinator
In an ideal situation any changes done during a long-running transaction must be reverted back to the original content in case there's a failure somewhere else along the flow of the transaction. This is precisely what happens in a closed environment and is known as a rollback. In a SOA architecture, a situation might arise where a rollback isn't an option. In that case, instead of a rollback, compensation is provided. For example, in WS choreography, the self-reliant services pass control messages back and forth to notify the participating services of a rollback operation.
Compensation may be defined as the most logical change applied to the resource to maintain data consistency and integrity. How it's constructed depends on the governing business rules and underlying technical implementation of the services. In certain cases, compensation can include a rollback. In the example above, if the transaction fails at the third service (the transaction is uniquely identified by an id throughout its lifetime), we need to perform a compensatory operation at the previous service to negate the effect caused by the transaction. So, if the second service sent out an e-mail announcing that it has implemented the changes, a compensatory operation would be to send another e-mail announcing the failure of the transaction and that the changes have been undone. A synchronous process showcasing the scenario is illustrated in Figure 1.
But what if the services participating in the process are asynchronous, as one would expect in a long-running transaction? One way would be to save states and service information.
Methodology 2: Transaction Coordinator
A more appropriate solution would be to orchestrate the process using a transaction manager or process coordinator. Instead of inter-service communication, the services would be answerable to the coordinator, which in turn would handle all the transaction and compensation scenarios. Once again the transaction will be uniquely identified throughout the transaction cycle by an id. This would help the coordinator perform compensatory operations on the required set of data. The coordinator can manage the service information as well. This would solve any issues with asynchronous services. Figure 2 illustrates the coordinator service. This kind of methodology is used mostly in service orchestration-type applications and is a more centralized approach unlike methodology 1.
Case study - Money Transfer Scenario
Consider a money transfer scenario (Figure 4) where a complete transaction process involves five different services. All five services are separated by virtue of both system and the language of implementation.
The first service, the initiation service, is exposed to the client to pick up the user input. It validates the necessary input parameters and processes the transaction by sequentially executing the credit service and the debit service. Then the system notifies the stakeholder and the internal logs for auditing.
With no transaction context involved in this processing, the services are executed independently with no knowledge of the member service status. There's no way for the executed services to rollback and for specific reasons:
- Service status isn't shared
- Non-availability of co-ordination federation in the processing
- Compensation services for revoking the services
The coordinator receives the input and generates an id to uniquely identify the transaction. An acknowledgement is sent to the initiation service as RECEIVED. The initiation service notifies the client about the start of the process and provides the unique transaction id. The client can use this transaction id to monitor and track the transaction. The initiation service is now ready to take further client input. The coordinator maintains a log to record each operation it performs. The log is created against the transaction id.
After generating the id for the transaction, the coordinator calls the external service of the bank, which accepts the money. This credit service takes the necessary input and starts updating the records in the database. Depending upon the style of the compensation, state information is saved before the update process initiates. Once the update takes place successfully, an acknowledgement to the coordinator is sent. (Figure 3)
The coordinator then logs the changes and proceeds to call the debit service. The debit service makes the necessary changes to the local database to reflect the debit. The debit process follows the same pattern as the credit process. On successful operation, a DEBITED acknowledgement is sent to the coordinator. The coordinator notifies each service involved of successful individual transactions at each step by means enacts the 2PC execution. When there's a failure, the coordinator runs the compensation service for each activity.
The long-running transaction is designed specifically for business interactions that take a long time. The intention is to tie the logical single business-to-business unit of work across heterogeneous domains. Each methodology depends on the architecture of the system and the existing assets in the organization. Technical analysts need to differentiate such special transaction in the SOA study and deal with them through special defined methodologies.
1. William Cox. "Transactional Web Services."
2. Pat Helland. "Why I hate the phrase Long running Transactions..."
3. Wikipedia Atomic Transactions:
Two-phase commit protocol:
As Enterprise business moves from Monoliths to Microservices, adoption and successful implementations of Microservices become more evident. The goal of Microservices is to improve software delivery speed and increase system safety as scale increases. Documenting hurdles and problems for the use of Microservices will help consultants, architects and specialists to avoid repeating the same mistakes and learn how and when to use (or not use) Microservices at the enterprise level. The circumstance w...
Feb. 25, 2017 02:45 PM EST Reads: 879
TechTarget storage websites are the best online information resource for news, tips and expert advice for the storage, backup and disaster recovery markets. By creating abundant, high-quality editorial content across more than 140 highly targeted technology-specific websites, TechTarget attracts and nurtures communities of technology buyers researching their companies' information technology needs. By understanding these buyers' content consumption behaviors, TechTarget creates the purchase inte...
Feb. 25, 2017 02:30 PM EST Reads: 1,612
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry’s single source for the cloud. Fusion’s advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including cloud...
Feb. 25, 2017 02:00 PM EST Reads: 3,958
@DevOpsSummit at Cloud taking place June 6-8, 2017, at Javits Center, New York City, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long developm...
Feb. 25, 2017 12:15 PM EST Reads: 1,860
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Feb. 25, 2017 11:45 AM EST Reads: 5,847
DevOps and microservices are permeating software engineering teams broadly, whether these teams are in pure software shops but happen to run a business, such Uber and Airbnb, or in companies that rely heavily on software to run more traditional business, such as financial firms or high-end manufacturers. Microservices and DevOps have created software development and therefore business speed and agility benefits, but they have also created problems; specifically, they have created software securi...
Feb. 25, 2017 11:00 AM EST Reads: 3,891
This week's news brings us further reminders that if you're betting on cloud, you're headed in the right direction. The cloud is growing seven times faster than the rest of IT, according to IDC, with a 25% spending increase just from 2016 to 2017. SaaS still leads the pack, with an estimated two-thirds of public cloud spending going that way. Large enterprises, with more than 1,000 employees, are predicted to account for more than half of cloud spending and have the fastest annual growth rate.
Feb. 25, 2017 10:15 AM EST Reads: 900
The emerging Internet of Everything creates tremendous new opportunities for customer engagement and business model innovation. However, enterprises must overcome a number of critical challenges to bring these new solutions to market. In his session at @ThingsExpo, Michael Martin, CTO/CIO at nfrastructure, outlined these key challenges and recommended approaches for overcoming them to achieve speed and agility in the design, development and implementation of Internet of Everything solutions with...
Feb. 25, 2017 10:00 AM EST Reads: 7,736
Cloud Expo, Inc. has announced today that Andi Mann and Aruna Ravichandran have been named Co-Chairs of @DevOpsSummit at Cloud Expo 2017. The @DevOpsSummit at Cloud Expo New York will take place on June 6-8, 2017, at the Javits Center in New York City, New York, and @DevOpsSummit at Cloud Expo Silicon Valley will take place Oct. 31-Nov. 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Feb. 25, 2017 09:30 AM EST Reads: 2,211
SYS-CON Events announced today that Outlyer, a monitoring service for DevOps and operations teams, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outlyer is a monitoring service for DevOps and Operations teams running Cloud, SaaS, Microservices and IoT deployments. Designed for today's dynamic environments that need beyond cloud-scale monitoring, we make monitoring effortless so you...
Feb. 25, 2017 09:15 AM EST Reads: 2,177
In his General Session at 16th Cloud Expo, David Shacochis, host of The Hybrid IT Files podcast and Vice President at CenturyLink, investigated three key trends of the “gigabit economy" though the story of a Fortune 500 communications company in transformation. Narrating how multi-modal hybrid IT, service automation, and agile delivery all intersect, he will cover the role of storytelling and empathy in achieving strategic alignment between the enterprise and its information technology.
Feb. 25, 2017 08:00 AM EST Reads: 5,846
All clouds are not equal. To succeed in a DevOps context, organizations should plan to develop/deploy apps across a choice of on-premise and public clouds simultaneously depending on the business needs. This is where the concept of the Lean Cloud comes in - resting on the idea that you often need to relocate your app modules over their life cycles for both innovation and operational efficiency in the cloud. In his session at @DevOpsSummit at19th Cloud Expo, Valentin (Val) Bercovici, CTO of Soli...
Feb. 25, 2017 07:30 AM EST Reads: 1,228
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In his general session at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, will explore...
Feb. 25, 2017 07:15 AM EST Reads: 1,528
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
Feb. 25, 2017 07:00 AM EST Reads: 9,064
After more than five years of DevOps, definitions are evolving, boundaries are expanding, ‘unicorns’ are no longer rare, enterprises are on board, and pundits are moving on. Can we now look at an evolution of DevOps? Should we? Is the foundation of DevOps ‘done’, or is there still too much left to do? What is mature, and what is still missing? What does the next 5 years of DevOps look like? In this Power Panel at DevOps Summit, moderated by DevOps Summit Conference Chair Andi Mann, panelists l...
Feb. 25, 2017 05:00 AM EST Reads: 7,179
Culture is the most important ingredient of DevOps. The challenge for most organizations is defining and communicating a vision of beneficial DevOps culture for their organizations, and then facilitating the changes needed to achieve that. Often this comes down to an ability to provide true leadership. As a CIO, are your direct reports IT managers or are they IT leaders? The hard truth is that many IT managers have risen through the ranks based on their technical skills, not their leadership abi...
Feb. 25, 2017 05:00 AM EST Reads: 8,251
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound e...
Feb. 25, 2017 04:00 AM EST Reads: 11,906
In recent years, containers have taken the world by storm. Companies of all sizes and industries have realized the massive benefits of containers, such as unprecedented mobility, higher hardware utilization, and increased flexibility and agility; however, many containers today are non-persistent. Containers without persistence miss out on many benefits, and in many cases simply pass the responsibility of persistence onto other infrastructure, adding additional complexity.
Feb. 25, 2017 12:45 AM EST Reads: 1,333
An overall theme of Cloud computing and the specific practices within it is fundamentally one of automation. The core value of technology is to continually automate low level procedures to free up people to work on more value add activities, ultimately leading to the utopian goal of full Autonomic Computing. For example a great way to define your plan for DevOps tool chain adoption is through this lens. In this TechTarget article they outline a simple maturity model for planning this.
Feb. 25, 2017 12:15 AM EST Reads: 2,444
The rise of containers and microservices has skyrocketed the rate at which new applications are moved into production environments today. While developers have been deploying containers to speed up the development processes for some time, there still remain challenges with running microservices efficiently. Most existing IT monitoring tools don’t actually maintain visibility into the containers that make up microservices. As those container applications move into production, some IT operations t...
Feb. 24, 2017 09:45 PM EST Reads: 1,277