|By Vinay Singla||
|August 19, 2009 02:00 AM EDT||
Data integrity is one of the most critical elements in any system. Data integrity is easily achieved in a standalone system with a single database. Data integrity in such a system is maintained via database constraints and transactions. Transactions should follow ACID (Atomicity, Consistency, Isolation, Durability) properties to ensure data integrity. Most databases support ACID transactions and can preserve data integrity.
Next in the complexity chain are distributed systems. In a distributed system, there are multiple databases and multiple applications. In order to maintain data integrity in a distributed system, transactions across multiple data sources need to be handled correctly in a fail-safe manner. This is usually done via a central global transaction manager. Each application in the distributed system should be able to participate in the global transaction via a resource manager. This is achieved using a 2-phase commit protocol as per the XA standard. Most databases and custom applications have the ability to participate in a global transaction. Many packaged applications can also participate in a global transaction via EAI adapters. In reality, in most environments, some of the applications may support participation in a global transaction via 2-phase commit, some may support only single phase commit transactions and some may not support any transaction capability at all.
Moving further up in complexity are distributed systems with a mix of on-premise and partner applications. In this case, not all applications in the system are under the control of the organization and partner application interface may not support XA. B2B integration standards such as EDI and ebXML are the primary methods of ensuring reliability and data integrity across partner systems.
Enter the world of SOA and Cloud computing, and the problem of data integrity gets magnified even more, as there is a mix of on-prem and SaaS applications exposed as services. SaaS applications are multi-tenant applications hosted by a third party. SaaS applications usually expose their functionality via XML based APIs over HTTP protocol. SOAP and REST based web services are the most common methods of implementing these APIs. Also, in SOA based environments, many on-prem applications expose their functionality via SOAP and REST web services as well. One of the biggest challenges with web services is transaction management. At the protocol level, HTTP doesn’t support transactions or guaranteed delivery, so the only option is to implement these at the API level. Although there are standards available for managing data integrity with web services such as WS-Transaction and WS-Reliability, these standards are not yet mature and not many vendors have implemented these. Most SaaS vendors expose their web services APIs without any support for transactions. Also, each SaaS application may have different levels of availability and
Let’s take a simple scenario of new customer creation at a company. This company uses 2 SaaS vendors, one for Marketing and one for CRM. In addition, there is an on-premise ERP application. When a new customer places an order, the customer information needs to be sent to the Marketing service (for marketing campaigns), CRM service (for customer management) and ERP application (for order fulfillment). Both Marketing and CRM applications expose their customer creation APIs via SOAP web services over HTTP, whereas the ERP application exposes customer creation via a database API. Here is the sequence of operations in this transaction:
1. Customer creation in Marketing via SOAP web service (Doesn't support transaction)
2. Customer creation in CRM via SOAP web service (Doesn’t support transaction)
3. Customer creation in ERP via database insert (Supports transaction)
In order to maintain data integrity across the 3 applications, either all the steps should get successfully executed or none of them should get executed. In the above sequence of operations, if step 1 succeeds but step 2 fails, step 1 can’t be rolled back. If step 1 and 2 succeed but step 3 fails, steps 1 and 2 can’t be rolled back. So we have a data integrity issue at hand in various failure scenarios and customer record will exist in some systems but not in others. This is usually not acceptable in any production environment. So what can be done to handle this problem? There are several techniques that can be applied in this scenario:
Technique 1: Perform the operations that support transactions before the operations that don’t support transactions
In our example, step 3 should be moved to the beginning as follows:
1. Customer creation in ERP via database insert (Supports transaction)
2. Customer creation in Marketing via SOAP web service (Doesn't support transaction)
3. Customer creation in CRM via SOAP web service (Doesn’t support transaction)
With this change in the sequence of operations, if step 1 succeeds but step 2 fails, step 1 can just be rolled back. We still have a problem if step 1 and 2 succeed but step 3 fails. This is where the following techniques come in handy.
Technique 2: Use compensating transactions
In our new sequence as per technique 1, if steps 1 and 2 succeed but step 3 fails, rollback step 1 and issue a compensating transaction for step 2. Compensating transaction in this case will be to delete the customer. Of course, for this to work, the Marketing SaaS application needs to provide a “delete customer” API which should be requested before signing up with this SaaS vendor.
Technique 3: Break the transaction into multiple decoupled transactions
In our example, step 3 can be executed in a separate asynchronous transaction using a queue. Queue can be implemented using database or some messaging technology such as JMS. In either case, both write and read of messages from queue will support transactions. Here is the sequence of operations with this change:
1. Customer creation in ERP via database insert (Supports transaction)
2. Post message to a queue for customer creation in CRM (Supports transaction)
3. Customer creation in Marketing via SOAP web service (Doesn't support transaction)
In the above sequence, if step 2 fails, step 1 can be rolled back and if step3 fails, steps 1 and 2 can be rolled back. Note that posting message to queue is done before customer creation in Marketing to make sure the step that doesn’t support transaction is executed last (as per Technique 1).
1. Queue listener retrieves message from queue (Supports transaction)
2. Customer creation in CRM via SOAP web service (Doesn't support transaction)
In the above sequence, if step 2 fails, step 1 can be rolled back.
So by breaking a transaction into multiple smaller transactions separated by queues, we are able to achieve data integrity.
Technique 4: Execute the transaction as a long-running transaction
If all the steps of the transaction are orchestrated as separate tasks of a long-running process using a state machine or BPM (Business process management) tool, then failure at any step will result in the process not progressing to the next step. Retries can be introduced at every step to ensure that every step is successful before the whole process is finished. This is the most reliable technique of all the techniques discussed but this can also introduce latency as the process can take a long-time to finish if any application or service is down for a long-time. This solution introduces more complexity into the environment and may not be acceptable in all situations but this is also the most reliable way to design distributed transactions in services based environments.
By applying the techniques discussed in this article, most failure scenarios can be handled effectively so that data integrity is not compromised. These techniques can be applied to any distributed system but are most useful (and almost mandatory) in SOA and SaaS based environments where interfaces are exposed via web services.
Building custom add-ons does not need to be limited to the ideas you see on a marketplace. In his session at 20th Cloud Expo, Sukhbir Dhillon, CEO and founder of Addteq, will go over some adventures they faced in developing integrations using Atlassian SDK and other technologies/platforms and how it has enabled development teams to experiment with newer paradigms like Serverless and newer features of Atlassian SDKs. In this presentation, you will be taken on a journey of Add-On and Integration ...
Mar. 27, 2017 08:15 AM EDT Reads: 3,050
Culture is the most important ingredient of DevOps. The challenge for most organizations is defining and communicating a vision of beneficial DevOps culture for their organizations, and then facilitating the changes needed to achieve that. Often this comes down to an ability to provide true leadership. As a CIO, are your direct reports IT managers or are they IT leaders? The hard truth is that many IT managers have risen through the ranks based on their technical skills, not their leadership abi...
Mar. 27, 2017 05:00 AM EDT Reads: 11,042
The essence of cloud computing is that all consumable IT resources are delivered as services. In his session at 15th Cloud Expo, Yung Chou, Technology Evangelist at Microsoft, demonstrated the concepts and implementations of two important cloud computing deliveries: Infrastructure as a Service (IaaS) and Platform as a Service (PaaS). He discussed from business and technical viewpoints what exactly they are, why we care, how they are different and in what ways, and the strategies for IT to transi...
Mar. 27, 2017 05:00 AM EDT Reads: 6,163
Without a clear strategy for cost control and an architecture designed with cloud services in mind, costs and operational performance can quickly get out of control. To avoid multiple architectural redesigns requires extensive thought and planning. Boundary (now part of BMC) launched a new public-facing multi-tenant high resolution monitoring service on Amazon AWS two years ago, facing challenges and learning best practices in the early days of the new service.
Mar. 27, 2017 03:45 AM EDT Reads: 2,973
All organizations that did not originate this moment have a pre-existing culture as well as legacy technology and processes that can be more or less amenable to DevOps implementation. That organizational culture is influenced by the personalities and management styles of Executive Management, the wider culture in which the organization is situated, and the personalities of key team members at all levels of the organization. This culture and entrenched interests usually throw a wrench in the work...
Mar. 27, 2017 03:00 AM EDT Reads: 2,995
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
Mar. 27, 2017 12:45 AM EDT Reads: 2,117
As software becomes more and more complex, we, as software developers, have been splitting up our code into smaller and smaller components. This is also true for the environment in which we run our code: going from bare metal, to VMs to the modern-day Cloud Native world of containers, schedulers and micro services. While we have figured out how to run containerized applications in the cloud using schedulers, we've yet to come up with a good solution to bridge the gap between getting your contain...
Mar. 26, 2017 09:45 PM EDT Reads: 7,641
As organizations realize the scope of the Internet of Things, gaining key insights from Big Data, through the use of advanced analytics, becomes crucial. However, IoT also creates the need for petabyte scale storage of data from millions of devices. A new type of Storage is required which seamlessly integrates robust data analytics with massive scale. These storage systems will act as “smart systems” provide in-place analytics that speed discovery and enable businesses to quickly derive meaningf...
Mar. 26, 2017 07:45 PM EDT Reads: 9,608
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In his Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, will explore t...
Mar. 26, 2017 03:15 PM EDT Reads: 2,844
DevOps has often been described in terms of CAMS: Culture, Automation, Measuring, Sharing. While we’ve seen a lot of focus on the “A” and even on the “M”, there are very few examples of why the “C" is equally important in the DevOps equation. In her session at @DevOps Summit, Lori MacVittie, of F5 Networks, explored HTTP/1 and HTTP/2 along with Microservices to illustrate why a collaborative culture between Dev, Ops, and the Network is critical to ensuring success.
Mar. 26, 2017 03:00 PM EDT Reads: 10,592
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Mar. 26, 2017 01:45 PM EDT Reads: 8,572
Everyone wants to use containers, but monitoring containers is hard. New ephemeral architecture introduces new challenges in how monitoring tools need to monitor and visualize containers, so your team can make sense of everything. In his session at @DevOpsSummit, David Gildeh, co-founder and CEO of Outlyer, will go through the challenges and show there is light at the end of the tunnel if you use the right tools and understand what you need to be monitoring to successfully use containers in your...
Mar. 26, 2017 01:00 PM EDT Reads: 1,612
What if you could build a web application that could support true web-scale traffic without having to ever provision or manage a single server? Sounds magical, and it is! In his session at 20th Cloud Expo, Chris Munns, Senior Developer Advocate for Serverless Applications at Amazon Web Services, will show how to build a serverless website that scales automatically using services like AWS Lambda, Amazon API Gateway, and Amazon S3. We will review several frameworks that can help you build serverle...
Mar. 26, 2017 12:45 PM EDT Reads: 1,942
The IT industry is undergoing a significant evolution to keep up with cloud application demand. We see this happening as a mindset shift, from traditional IT teams to more well-rounded, cloud-focused job roles. The IT industry has become so cloud-minded that Gartner predicts that by 2020, this cloud shift will impact more than $1 trillion of global IT spending. This shift, however, has left some IT professionals feeling a little anxious about what lies ahead. The good news is that cloud computin...
Mar. 26, 2017 10:30 AM EDT Reads: 1,285
SYS-CON Events announced today that HTBase will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. HTBase (Gartner 2016 Cool Vendor) delivers a Composable IT infrastructure solution architected for agility and increased efficiency. It turns compute, storage, and fabric into fluid pools of resources that are easily composed and re-composed to meet each application’s needs. With HTBase, companies can quickly prov...
Mar. 26, 2017 08:15 AM EDT Reads: 2,929
An overall theme of Cloud computing and the specific practices within it is fundamentally one of automation. The core value of technology is to continually automate low level procedures to free up people to work on more value add activities, ultimately leading to the utopian goal of full Autonomic Computing. For example a great way to define your plan for DevOps tool chain adoption is through this lens. In this TechTarget article they outline a simple maturity model for planning this.
Mar. 26, 2017 06:00 AM EDT Reads: 4,289
While DevOps most critically and famously fosters collaboration, communication, and integration through cultural change, culture is more of an output than an input. In order to actively drive cultural evolution, organizations must make substantial organizational and process changes, and adopt new technologies, to encourage a DevOps culture. Moderated by Andi Mann, panelists discussed how to balance these three pillars of DevOps, where to focus attention (and resources), where organizations might...
Mar. 26, 2017 05:15 AM EDT Reads: 6,191
The rise of containers and microservices has skyrocketed the rate at which new applications are moved into production environments today. While developers have been deploying containers to speed up the development processes for some time, there still remain challenges with running microservices efficiently. Most existing IT monitoring tools don’t actually maintain visibility into the containers that make up microservices. As those container applications move into production, some IT operations t...
Mar. 26, 2017 01:00 AM EDT Reads: 2,980
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
Mar. 25, 2017 09:45 PM EDT Reads: 3,653
Software development is a moving target. You have to keep your eye on trends in the tech space that haven’t even happened yet just to stay current. Consider what’s happened with augmented reality (AR) in this year alone. If you said you were working on an AR app in 2015, you might have gotten a lot of blank stares or jokes about Google Glass. Then Pokémon GO happened. Like AR, the trends listed below have been building steam for some time, but they’ll be taking off in surprising new directions b...
Mar. 25, 2017 01:30 PM EDT Reads: 5,841