|By Gary Kaiser||
|July 13, 2014 01:00 PM EDT||
We know that losing packets is not a good thing; retransmissions cause delays. We also know that TCP ensures reliable data delivery, masking the impact of packet loss. So why are some applications seemingly unaffected by the same packet loss rate that seems to cripple others? From a performance analysis perspective, how do you understand the relevance of packet loss and avoid chasing red herrings?
In Part II, we examined two closely related constraints - bandwidth and congestion. In Part III, we discussed TCP slow-start and introduced the Congestion Window (CWD). In Part IV, we'll focus on packet loss, continuing the concepts from these two previous entries.
TCP ensures reliable delivery of data through its sliding window approach to managing byte sequences and acknowledgements; among other things, this sequencing allows a receiver to inform the sender of missing data caused by packet loss in multi-packet flows. Independently, a sender may detect packet loss through the expiration of its retransmission timer. We will look at the behavior and performance penalty associated with each of these cases; generally, the impact of packet loss will depend on both the characteristics of the flow and the position of the dropped packet within the flow.
The Retransmission Timer
Each packet a node sends is associated with a retransmission timer; if the timer expires before the sent data has been acknowledged, it is considered lost and retransmitted. There are two important characteristics of the retransmission timer that relate to performance. First, the default value for the initial retransmission timeout (RTO) is almost always 3000 milliseconds; this is adjusted to a more reasonable value as TCP observes actual path round-trip times. Second, the timeout value is doubled for subsequent retransmissions of a packet.
In small flows (a common characteristic of chatty operations - like web pages), the retransmission timer is the method used to detect packet loss. Consider a request or reply message of just 1000 bytes, sent in a single packet; if this packet is dropped, there will of course be no acknowledgement; the receiver has no idea the packet was sent. If the packet is dropped early in the life of a TCP connection - perhaps one of the SYN packets during the TCP 3-way handshake, or an initial GET request or a 304 Not Modified response - the dropped packet will be retransmitted only after 3 seconds have elapsed.
Triple Duplicate ACK
Within larger flows, a dropped packet may be detected before the retransmission time expires if the sender receives three duplicate ACKs; this is generally more efficient (faster) than waiting for the retransmission timer to expire. As the receiving node receives packets that are out of sequence (i.e., after the missing packet data should have been seen), it sends duplicate ACKs, the acknowledgement number repeatedly referencing the expected (missing) packet data. When the sending node receives the third duplicate ACK, it assumes the packet was in fact lost (not just delayed) and retransmits it. This event causes the sender to assume network congestion, reducing its congestion window by 50% to allow congestion to subside. Slow-start begins to increase the CWD from that new value, using a relatively conservative congestion avoidance ramp.
As an example, consider a server sending a large file to a client; the sending node is ramping up through slow-start. As the CWD reaches 24, earlier packet loss is detected via a triple duplicate ACK; the lost data is retransmitted, and the CWD is reduced to 12. Slow-start resumes from this point in its congestion avoidance mode.
While arguments abound about the inefficiency of existing congestion avoidance approaches, especially on high-speed networks, you can expect to see this behavior in today's networks.
Transaction Trace Illustration
Identifying retransmission timeouts using merged trace files is generally quite straightforward; we have proof the packet has been lost (because we see it on the sending side and not on the receiving side), and we know the delay between the dropped and retransmitted packets at the sending node. The Delta column in the Error Table indicates the retransmission delay.
Error Table entry showing a 3-second retransmission delay caused by a retransmission timeout (RTO)
For larger flows, you can illustrate the effect of dropped packets on the sender's Congestion Window by using the Time Plot view. For Series 1, graph the sender's Frames in Transit; this is essentially the CWD. For Series 2, graph the Cumulative Error Count in both directions. As errors (retransmitted packets or out-of-sequence packets) occur, the CWD will be reduced by about 50%.
Time Plot view showing the impact of packet loss (blue plot) on the Congestion Window (brown plot)
For more networking tips click here for the full article
Containers have changed the mind of IT in DevOps. They enable developers to work with dev, test, stage and production environments identically. Containers provide the right abstraction for microservices and many cloud platforms have integrated them into deployment pipelines. DevOps and Containers together help companies to achieve their business goals faster and more effectively. In his session at DevOps Summit, Ruslan Synytsky, CEO and Co-founder of Jelastic, reviewed the current landscape of...
Jul. 1, 2015 05:00 PM EDT Reads: 2,181
Conferences agendas. Event navigation. Specific tasks, like buying a house or getting a car loan. If you've installed an app for any of these things you've installed what's known as a "disposable mobile app" or DMA. Apps designed for a single use-case and with the expectation they'll be "thrown away" like brochures. Deleted until needed again. These apps are necessarily small, agile and highly volatile. Sometimes existing only for a short time - say to support an event like an election, the Wor...
Jul. 1, 2015 05:00 PM EDT Reads: 1,599
The cloud has transformed how we think about software quality. Instead of preventing failures, we must focus on automatic recovery from failure. In other words, resilience trumps traditional quality measures. Continuous delivery models further squeeze traditional notions of quality. Remember the venerable project management Iron Triangle? Among time, scope, and cost, you can only fix two or quality will suffer. Only in today's DevOps world, continuous testing, integration, and deployment upend...
Jul. 1, 2015 03:00 PM EDT Reads: 1,906
Discussions about cloud computing are evolving into discussions about enterprise IT in general. As enterprises increasingly migrate toward their own unique clouds, new issues such as the use of containers and microservices emerge to keep things interesting. In this Power Panel at 16th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the state of cloud computing today, and what enterprise IT professionals need to know about how the latest topics and trends affect t...
Jul. 1, 2015 02:30 PM EDT Reads: 1,160
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at @ThingsExpo, James Kirkland, Red Hat's Chief Arch...
Jul. 1, 2015 02:21 PM EDT Reads: 470
Summer is finally here and it’s time for a DevOps summer vacation. From San Francisco to New York City, our top summer conferences list is going to continuously deliver you to the summer destinations of your dreams. These DevOps parties are hitting all the hottest summer trends with Microservices, Agile, Continuous Delivery, DevSecOps, and even Continuous Testing. Move over Kanye. These are the top 5 Summer DevOps Conferences of 2015.
Jul. 1, 2015 01:00 PM EDT Reads: 843
Sharding has become a popular means of achieving scalability in application architectures in which read/write data separation is not only possible, but desirable to achieve new heights of concurrency. The premise is that by splitting up read and write duties, it is possible to get better overall performance at the cost of a slight delay in consistency. That is, it takes a bit of time to replicate changes initiated by a "write" to the read-only master database. It's eventually consistent, and it'...
Jul. 1, 2015 12:00 PM EDT Reads: 1,702
"Plutora provides release and testing environment capabilities to the enterprise," explained Dalibor Siroky, Director and Co-founder of Plutora, in this SYS-CON.tv interview at @DevOpsSummit, held June 9-11, 2015, at the Javits Center in New York City.
Jul. 1, 2015 11:45 AM EDT Reads: 1,004
Containers are changing the security landscape for software development and deployment. As with any security solutions, security approaches that work for developers, operations personnel and security professionals is a requirement. In his session at DevOps Summit, Kevin Gilpin, CTO and Co-Founder of Conjur, will discuss various security considerations for container-based infrastructure and related DevOps workflows.
Jul. 1, 2015 10:30 AM EDT Reads: 784
Cloud Migration Management (CMM) refers to the best practices for planning and managing migration of IT systems from a legacy platform to a Cloud Provider through a combination professional services consulting and software tools. A Cloud migration project can be a relatively simple exercise, where applications are migrated ‘as is’, to gain benefits such as elastic capacity and utility pricing, but without making any changes to the application architecture, software development methods or busine...
Jul. 1, 2015 10:00 AM EDT Reads: 1,914
Data center models are changing. A variety of technical trends and business demands are forcing that change, most of them centered on the explosive growth of applications. That means, in turn, that the requirements for application delivery are changing. Certainly application delivery needs to be agile, not waterfall. It needs to deliver services in hours, not weeks or months. It needs to be more cost efficient. And more than anything else, it needs to be really, dc infra axisreally, super focus...
Jul. 1, 2015 10:00 AM EDT Reads: 1,972
The most often asked question post-DevOps introduction is: “How do I get started?” There’s plenty of information on why DevOps is valid and important, but many managers still struggle with simple basics for how to initiate a DevOps program in their business. They struggle with issues related to current organizational inertia, the lack of experience on Continuous Integration/Delivery, understanding where DevOps will affect revenue and budget, etc. In their session at DevOps Summit, JP Morgenthal...
Jul. 1, 2015 09:32 AM EDT Reads: 570
Overgrown applications have given way to modular applications, driven by the need to break larger problems into smaller problems. Similarly large monolithic development processes have been forced to be broken into smaller agile development cycles. Looking at trends in software development, microservices architectures meet the same demands. Additional benefits of microservices architectures are compartmentalization and a limited impact of service failure versus a complete software malfunction. ...
Jul. 1, 2015 09:00 AM EDT Reads: 852
DevOps tends to focus on the relationship between Dev and Ops, putting an emphasis on the ops and application infrastructure. But that’s changing with microservices architectures. In her session at DevOps Summit, Lori MacVittie, Evangelist for F5 Networks, will focus on how microservices are changing the underlying architectures needed to scale, secure and deliver applications based on highly distributed (micro) services and why that means an expansion into “the network” for DevOps.
Jun. 30, 2015 08:30 AM EDT Reads: 2,429
Many people recognize DevOps as an enormous benefit – faster application deployment, automated toolchains, support of more granular updates, better cooperation across groups. However, less appreciated is the journey enterprise IT groups need to make to achieve this outcome. The plain fact is that established IT processes reflect a very different set of goals: stability, infrequent change, hands-on administration, and alignment with ITIL. So how does an enterprise IT organization implement change...
Jun. 29, 2015 12:45 PM EDT Reads: 2,807
While DevOps most critically and famously fosters collaboration, communication, and integration through cultural change, culture is more of an output than an input. In order to actively drive cultural evolution, organizations must make substantial organizational and process changes, and adopt new technologies, to encourage a DevOps culture. Moderated by Andi Mann, panelists discussed how to balance these three pillars of DevOps, where to focus attention (and resources), where organizations migh...
Jun. 28, 2015 05:00 PM EDT Reads: 2,044
At DevOps Summit NY there’s been a whole lot of talk about not just DevOps, but containers, IoT, and microservices. Sessions focused not just on the cultural shift needed to grow at scale with a DevOps approach, but also made sure to include the network ”plumbing” needed to ensure success as applications decompose into the microservice architectures enabling rapid growth and support for the Internet of (Every)Things.
Jun. 28, 2015 01:00 PM EDT Reads: 1,908
Mashape is bringing real-time analytics to microservices with the release of Mashape Analytics. First built internally to analyze the performance of more than 13,000 APIs served by the mashape.com marketplace, this new tool provides developers with robust visibility into their APIs and how they function within microservices. A purpose-built, open analytics platform designed specifically for APIs and microservices architectures, Mashape Analytics also lets developers and DevOps teams understand w...
Jun. 27, 2015 11:00 AM EDT Reads: 1,944
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists peeled away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud envir...
Jun. 26, 2015 12:00 PM EDT Reads: 2,243
Sumo Logic has announced comprehensive analytics capabilities for organizations embracing DevOps practices, microservices architectures and containers to build applications. As application architectures evolve toward microservices, containers continue to gain traction for providing the ideal environment to build, deploy and operate these applications across distributed systems. The volume and complexity of data generated by these environments make monitoring and troubleshooting an enormous chall...
Jun. 26, 2015 12:00 PM EDT Reads: 1,550