Welcome!

Microservices Expo Authors: Elizabeth White, Liz McMillan, Yeshim Deniz, Pat Romanski, Zakia Bouachraoui

Related Topics: Microservices Expo, @CloudExpo, @DevOpsSummit

Microservices Expo: Blog Feed Post

Top Two Features of Self-Healing Microservices | @CloudExpo #Cloud #Microservices

Microservices-based environments are more complex than their monolithic counterparts

Top Two Features of Self Healing Microservices
By Martin Goodwell

Microservices-based environments are more complex than their monolithic counterparts. To operate microservices environments with the same level of convenience that you’ve come expect from operating self-contained monolithic application environments, you need to have the right tools in place and rely on best practices that will keep your microservices healthy.

We’ve noticing an increasing number of microservices environments deployed by our customers, and the trend only seems to be increasing. We recently asked some of these customers about their experiences. Most of their responses were in line with expectations. Two issues however caught us by surprise.

The two issues that surprised us
While increased complexity and the proliferation of services is a challenge that most customers contend with, it’s the orchestration layer that seems to cause the most trouble for customers. The complexity of microservices environments requires automated operation and self-healing capabilities. This means that some sort of orchestration layer is required.

Automated orchestration
The first issue for many customers is that they haven’t even considered using a tool for orchestration. Because they don’t have an orchestration layer in place they’ve been frustrated from the beginning. After a few deployments, maintaining different instances of services simply became too complicated.

This is too bad because plenty of good open-source tools are available to facilitate the operation of microservices. For example, Netflix OSS—one of the toolsets most commonly used by our customers—provides a number of utilities that Netflix has developed over the years.

Of the dozens of available Netflix OSS tools and libraries, Eureka service registry and Hystrix circuit breaker are the most popular and likely also the easiest to use (Spring Cloud is a real easy to getting started with those technologies quickly, by the way). The service registry allows your microservices to register themselves upon startup. Any client that intends to access a microservice can look up the available endpoint addresses via Eureka. This reduces the risk of accessing an unavailable service to near zero.

Netflix OSS Eureka and Hystrix at a glance

Circuit breakers keep your business going
Hystrix circuit breaker makes your service calls more resilient by keeping track of each endpoint’s status. Normally you face expensive request timeouts when an endpoint becomes unavailable. Hystrix saves you from such timeouts by “breaking” the connection to the endpoint (this is why Hystrix is called a “circuit breaker”). It then reports that the service as unavailable so that subsequent requests don’t run into the same timeouts. Hystrix then continues to poll the service in the background to see when it’s available again.

With the risk of accessing unavailable services reduced to near-zero and expensive timeouts reduced to a minimum, your website keeps your content online and your visitors happy. You probably already know how much money you lose with each additional second of wait time, so using a service registry and a circuit breaker is a no-brainer in my opinion.

The second problem reported by our customers really caught us by surprise: a large number of customers report that their orchestration layers cause increased troubleshooting effort when problems are encountered.

Self-healing systems impact troubleshooting?
We were puzzled by this at first because the orchestration layer is supposed to keep an environment healthy and self-healing—quite the opposite of what our customers were reporting. Once we dug into this issue a little deeper however, the cause of the additional troubleshooting effort became obvious. These customers were initially looking for problems only in their services, not the orchestration layer itself. So when the orchestration layer did fail, customers spent considerable time eliminating the possible root-causes in their own code before getting around to investigating the orchestration layer.

These experiences taught us and our customers an important lesson:

Orchestration layers need to be monitored
You need a proper monitoring tool in place to know when your orchestration layer fails. You’re probably used to monitoring your database connection pools, your service queues, and of course your API performance metrics. You need to apply the same monitoring strategy to your orchestration layer.

Eureka and Hystrix are well-crafted pieces of code that allow for monitoring of all the important orchestration-layer related metrics.

Increased service lookup times and failing connections indicate that your orchestration layer need maintenance.

The lessons we gleaned from our customers’ experiences leads us to the conclusion that monitoring of the orchestration layer should be an integral feature of all full-stack monitoring tools. Service registries, circuit breakers, and other orchestration tools should be part of all newly created environments, as should database connection pools and messaging queues.

Short interruptions are handled automatically by the orchestration layer. Ongoing problems indicate that the orchestration layer is in trouble.

Keep control over your environment
Ideally, your orchestration layer is a simple set of tools that hardly ever fails. Unfortunately, things that hardly ever fail are not the initial focus of troubleshooting. So, when your orchestration layer fails, you may have a hard time learning about it quickly.

Orchestration-layer metrics can be a great indicator of looming problems. In cases of bad performance, take a look at your orchestration layer metrics. The orchestration layer may not be your hottest troublespot, but when when there is trouble there, you’ll be glad to know about it.

Do you know your metrics?
Is your environment based on microservices? Do you actively use service registries and circuit breakers? If so, do you know how much time service lookups consume? Do you know how often fallback mechanisms need to jump in? And do you know if your environment is truly self-healing?

If you have Netflix OSS components in use, you can add Netflix OSS monitoring by downloading the plugin file from our GitHub repo at JMX-Extensions/Netflix OSS extensions/plugin.json. Activating it is just a simple matter of uploading the file to your Dynatrace Ruxit environment.

In one of our upcoming releases, we will include this feature to be available out-of-the-box. Ruxit provides immediate value to you out-of-the-box. Haven’t tried Dynatrace Ruxit yet? Take the free trial and see how resilient your environement is. You may be surprised at how Ruxit’s auto-detection and zero-configuration have simplified monitoring setup.

The post Top 2 features of self-healing microservices appeared first on #monitoringlife.

Read the original blog entry...

More Stories By Dynatrace Blog

Building a revolutionary approach to software performance monitoring takes an extraordinary team. With decades of combined experience and an impressive history of disruptive innovation, that’s exactly what we ruxit has.

Get to know ruxit, and get to know the future of data analytics.

Microservices Articles
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
Most DevOps journeys involve several phases of maturity. Research shows that the inflection point where organizations begin to see maximum value is when they implement tight integration deploying their code to their infrastructure. Success at this level is the last barrier to at-will deployment. Storage, for instance, is more capable than where we read and write data. In his session at @DevOpsSummit at 20th Cloud Expo, Josh Atwell, a Developer Advocate for NetApp, will discuss the role and value...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
Consumer-driven contracts are an essential part of a mature microservice testing portfolio enabling independent service deployments. In this presentation we'll provide an overview of the tools, patterns and pain points we've seen when implementing contract testing in large development organizations.
Adding public cloud resources to an existing application can be a daunting process. The tools that you currently use to manage the software and hardware outside the cloud aren’t always the best tools to efficiently grow into the cloud. All of the major configuration management tools have cloud orchestration plugins that can be leveraged, but there are also cloud-native tools that can dramatically improve the efficiency of managing your application lifecycle. In his session at 18th Cloud Expo, ...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
Enterprise architects are increasingly adopting multi-cloud strategies as they seek to utilize existing data center assets, leverage the advantages of cloud computing and avoid cloud vendor lock-in. This requires a globally aware traffic management strategy that can monitor infrastructure health across data centers and end-user experience globally, while responding to control changes and system specification at the speed of today’s DevOps teams. In his session at 20th Cloud Expo, Josh Gray, Chie...
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...