Microservices Expo Authors: Stackify Blog, Aruna Ravichandran, Dalibor Siroky, Kevin Jackson, PagerDuty Blog

Related Topics: @DevOpsSummit, Microservices Expo, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Post

DevOps Insights into Conway’s Law By @TheEbizWizard | @DevOpsSummit #DevOps #Microservices

Do organizational structures determine system design, or vice versa?

Both digital transformation and DevOps are organizational and cultural transformations more so than they are technology changes – although in both cases, technology plays a large part in driving the organizational change necessary to achieve the business value for either effort. Just how the organizational changes and technology changes work together, however, is a difficult question.

Fortunately for us, this question is an old one – 47 years old, in fact. In a 1968 paper, computer scientist Melvin Conway wrote, “Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization’s communication structure.”

Conway produced no evidence of this statement, and his paper was initially rejected as a result – but to this day, we refer to this statement as Conway’s Law.

Law, however, is decidedly an overstatement. Observation is actually more accurate. In fact, the Wikipedia page for Conway’s Law states that “Although sometimes construed as humorous, Conway’s law was intended as a valid sociological observation.”

Whether the statement be humorous observation or law, however, today Conway’s Law plays a central role in our efforts to break down organizational silos in order to improve business velocity and better meet the needs of customers – the purported goals of devops and digital transformation, respectively.

For Conway’s Law to be a useful tool, however, we need a better causal story. Can changing our organizational structures impact our technology? Or more importantly, how does technology impact our organizational structures? And why?

A Brief History of Conway’s Law
Melvin Conway
was an early computer scientist whose most interesting accomplishment other than the law itself may have been his part in creating the compiler for the original 1984 version of Mac Pascal.

His eponymous law, however, apparently languished until sometime before 1996 or perhaps even earlier than 1991, when open source guru Eric Raymond included Conway’s Law in his Jargon File. He restated it with the snarky example, “If you have four groups working on a compiler, you’ll get a 4-pass compiler.”

Our story then gets more interesting when the Harvard Business School attempted to put some meat on the bones in a 2008 study. They use an objective measure of the modularity of software to find evidence for Conway’s Law. They conclude:

  • A product’s architecture tends to mirror the structure of the organization within which it is developed.
  • New organizational arrangements can have a distinct impact on the nature of the resulting design, and hence may affect product performance in unintended ways.

It’s important to note two important updates to the context of Conway’s Law by the time of the 2008 study: an organization’s “communication structure” becomes the structure of the organization itself, while “system design” becomes “product architecture” – with the constraint to product architecture being the scope of the study, rather than any sort of judgment on the scope of Conway’s Law itself.

What the 2008 study is essentially lacking, furthermore, is a discussion of the underlying causal principles behind the law. Instead, it is more or less taken for granted that siloed technology teams will create modular systems with the modularity aligning to the team structure simply because that’s the way people behave when they’re in teams.

The reason this question of causality is so important is because it goes to the heart of how we might use Conway’s Law to actually improve things. In particular, will changing organizational structures enable us to build better software?

Jonny Leroy and Matt Simons of ThoughtWorks explored this question when they coined the term “Inverse Conway Maneuver” in an article in the 2010 issue of the Cutter IT Journal. They state:

Conway’s Law … can be summarized as “Dysfunctional organizations tend to create dysfunctional applications.” To paraphrase Einstein, you can’t fix a problem from within the same mindset that created it, so it is often worth investigating whether restructuring your organization or team would prevent the new application from displaying all the same structural dysfunctions as the original. In what could be termed an “inverse Conway maneuver,” you may want to begin by breaking down silos that constrain the team’s ability to collaborate effectively.

The ThoughtWorks article takes the context of Conway’s Law from observational to normative: not satisfied simply to reflect on the way things work, they take the important step of opining on how things should – and should not work. They also intentionally focus on the negative: rather than simply describing how to build good software, they note that Conway’s Law is about building dysfunctional software.

Reversing the Inverse Conway Maneuver
It’s no surprise that ThoughtWorks’ focus is on changing organizational structures in order to build better software – after all, ThoughtWorks is a software development organization, and most modern software development thinking, including both the Agile and Lean movements, focuses on organizational change in furtherance of better software.

For digital transformation efforts, however, we must reverse this discussion. Technology change is driving changing customer preferences and behavior, which in turn are driving organizational change across increasingly software-driven enterprises.

The causality question behind Conway’s Law, therefore, is less about how changing software organizations can lead to better software, but rather how companies can best leverage changing technology in order to transform their organizations.

Hints at how to answer this question surprisingly come from the world of devops – surprising because the focus of devops is ostensibly on building and deploying better software more quickly. Be that as it may, there’s no question that technology change is a primary facilitator and driving force for the devops cultural and organizational shifts.

Connecting the Dots to Conway’s Law
If we didn’t have the cloud computing example of fully automated deployment and operational environments, and if we didn’t have today’s dramatic innovations in continuous development, continuous integration, and continuous delivery tooling, then devops would never have left the whiteboard stage. There’s no question that the devops story is a tale of technology-driven organizational change.

The DevOps technology landscape, however, doesn’t have an end-to-end, seamless technology story. On the contrary, this landscape is cluttered with dozens of various tools, many open source, all of which are in various stages of maturity. Therefore, it’s not readily apparent how to apply Conway’s Law, since we’re trying to leverage diverse toolchains of tools and technologies in order to help evolve our organizational structures.

In fact, Conway’s Law describes how such a diverse tooling marketplace came to be in the first place, as open source teams are generally quite modular, and thus will produce modular software.

Once we’ve solved the organizational challenges of digital and DevOps, breaking down silos in order to deliver customer value at velocity, then we can expect Conway’s Law to kick in again, and predict the rise of end-to-end software solutions as the result of horizontally self-organized teams.

It’s the middle piece, however, we’re struggling with now: how do we leverage a diverse set of disparate technologies to facilitate the cross-cutting reorganization of our businesses, contrary to the observation of Conway’s Law? And do we really think going against Conway’s Law will actually work?

Not to fear. In fact, the technologies that underpin DevOps aren’t as diverse and disparate as the application of Conway’s Law might suggest. True, the creators of all the tools in our DevOps or digital tool belts are working mostly independently of each other, a pattern of behavior which in the past has led to incompatible software mishmashes.

Today, however, we’re seeing the rise of what we might call crowdsourced architecture, as all of these teams work within the context of mature communication protocols, RESTful interfaces, and other emerging architectural trends like containerization and microservices.

As a result, mostly independent doesn’t mean completely independent. Instead, we have the loosely connected communication structure that assures us that Conway’s Law is still alive and well. What’s changed is the context for this notion of an organizational communication structure.

Conway was referring to the communication structures of companies or software development organizations or perhaps individual software development teams. Today, however, we’re talking about the broader technology community itself.

The Intellyx Take
There was no way Melvin Conway could have observed such crowdsourced architectural maturity back in 1968, because the telephone and snail mail-based communication structures of the day weren’t conducive to crowdsourcing anything. In contrast, today we have a plethora of tools and processes for facilitating communication and collaboration across traditional projects, teams, and open source efforts.

The end result is essentially a two-level application of Conway’s Law: a collaborative extended community of technologists that creates not simply a collection of disparate tools but rather chainable tools that leverage crowdsourced architectural principles to facilitate a level of coordination and interactivity we’ve never seen before.

This coordinated technology environment, in turn, facilitates the reorganization within companies, as they now have the tools they need to break down organizational silos, and people within those companies self-organize along horizontal lines, connecting customer experience to back-office software development and operations.

Conway’s Law, therefore, does work both ways. Organizational structures impact system design, and system architectures impact organizational structures as well. In the final analysis, however, Conway’s Law remains stubbornly observational. The underlying causal story – why such observations are remarkably universal – remains to be told. Stay tuned!

Intellyx advises companies on their digital transformation initiatives and helps vendors communicate their agility stories. As of the time of writing, none of the organizations mentioned in this article are Intellyx customers. Image credit: Melvin Conway.

More Stories By Jason Bloomberg

Jason Bloomberg is the leading expert on architecting agility for the enterprise. As president of Intellyx, Mr. Bloomberg brings his years of thought leadership in the areas of Cloud Computing, Enterprise Architecture, and Service-Oriented Architecture to a global clientele of business executives, architects, software vendors, and Cloud service providers looking to achieve technology-enabled business agility across their organizations and for their customers. His latest book, The Agile Architecture Revolution (John Wiley & Sons, 2013), sets the stage for Mr. Bloomberg’s groundbreaking Agile Architecture vision.

Mr. Bloomberg is perhaps best known for his twelve years at ZapThink, where he created and delivered the Licensed ZapThink Architect (LZA) SOA course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, the leading SOA advisory and analysis firm, which was acquired by Dovel Technologies in 2011. He now runs the successor to the LZA program, the Bloomberg Agile Architecture Course, around the world.

Mr. Bloomberg is a frequent conference speaker and prolific writer. He has published over 500 articles, spoken at over 300 conferences, Webinars, and other events, and has been quoted in the press over 1,400 times as the leading expert on agile approaches to architecture in the enterprise.

Mr. Bloomberg’s previous book, Service Orient or Be Doomed! How Service Orientation Will Change Your Business (John Wiley & Sons, 2006, coauthored with Ron Schmelzer), is recognized as the leading business book on Service Orientation. He also co-authored the books XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996).

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting).

@MicroservicesExpo Stories
How is DevOps going within your organization? If you need some help measuring just how well it is going, we have prepared a list of some key DevOps metrics to track. These metrics can help you understand how your team is doing over time. The word DevOps means different things to different people. Some say it a culture and every vendor in the industry claims that their tools help with DevOps. Depending on how you define DevOps, some of these metrics may matter more or less to you and your team.
For many of us laboring in the fields of digital transformation, 2017 was a year of high-intensity work and high-reward achievement. So we’re looking forward to a little breather over the end-of-year holiday season. But we’re going to have to get right back on the Continuous Delivery bullet train in 2018. Markets move too fast and customer expectations elevate too precipitously for businesses to rest on their laurels. Here’s a DevOps “to-do list” for 2018 that should be priorities for anyone w...
If testing environments are constantly unavailable and affected by outages, release timelines will be affected. You can use three metrics to measure stability events for specific environments and plan around events that will affect your critical path to release.
In a recent post, titled “10 Surprising Facts About Cloud Computing and What It Really Is”, Zac Johnson highlighted some interesting facts about cloud computing in the SMB marketplace: Cloud Computing is up to 40 times more cost-effective for an SMB, compared to running its own IT system. 94% of SMBs have experienced security benefits in the cloud that they didn’t have with their on-premises service
DevOps failure is a touchy subject with some, because DevOps is typically perceived as a way to avoid failure. As a result, when you fail in a DevOps practice, the situation can seem almost hopeless. However, just as a fail-fast business approach, or the “fail and adjust sooner” methodology of Agile often proves, DevOps failures are actually a step in the right direction. They’re the first step toward learning from failures and turning your DevOps practice into one that will lead you toward even...
DevOps is under attack because developers don’t want to mess with infrastructure. They will happily own their code into production, but want to use platforms instead of raw automation. That’s changing the landscape that we understand as DevOps with both architecture concepts (CloudNative) and process redefinition (SRE). Rob Hirschfeld’s recent work in Kubernetes operations has led to the conclusion that containers and related platforms have changed the way we should be thinking about DevOps and...
The goal of Microservices is to improve software delivery speed and increase system safety as scale increases. Microservices being modular these are faster to change and enables an evolutionary architecture where systems can change, as the business needs change. Microservices can scale elastically and by being service oriented can enable APIs natively. Microservices also reduce implementation and release cycle time and enables continuous delivery. This paper provides a logical overview of the Mi...
While walking around the office I happened upon a relatively new employee dragging emails from his inbox into folders. I asked why and was told, “I’m just answering emails and getting stuff off my desk.” An empty inbox may be emotionally satisfying to look at, but in practice, you should never do it. Here’s why. I recently wrote a piece arguing that from a mathematical perspective, Messy Desks Are Perfectly Optimized. While it validated the genius of my friends with messy desks, it also gener...
The next XaaS is CICDaaS. Why? Because CICD saves developers a huge amount of time. CD is an especially great option for projects that require multiple and frequent contributions to be integrated. But… securing CICD best practices is an emerging, essential, yet little understood practice for DevOps teams and their Cloud Service Providers. The only way to get CICD to work in a highly secure environment takes collaboration, patience and persistence. Building CICD in the cloud requires rigorous ar...
The enterprise data storage marketplace is poised to become a battlefield. No longer the quiet backwater of cloud computing services, the focus of this global transition is now going from compute to storage. An overview of recent storage market history is needed to understand why this transition is important. Before 2007 and the birth of the cloud computing market we are witnessing today, the on-premise model hosted in large local data centers dominated enterprise storage. Key marketplace play...
The cloud revolution in enterprises has very clearly crossed the phase of proof-of-concepts into a truly mainstream adoption. One of most popular enterprise-wide initiatives currently going on are “cloud migration” programs of some kind or another. Finding business value for these programs is not hard to fathom – they include hyperelasticity in infrastructure consumption, subscription based models, and agility derived from rapid speed of deployment of applications. These factors will continue to...
Some people are directors, managers, and administrators. Others are disrupters. Eddie Webb (@edwardawebb) is an IT Disrupter for Software Development Platforms at Liberty Mutual and was a presenter at the 2016 All Day DevOps conference. His talk, Organically DevOps: Building Quality and Security into the Software Supply Chain at Liberty Mutual, looked at Liberty Mutual's transformation to Continuous Integration, Continuous Delivery, and DevOps. For a large, heavily regulated industry, this task ...
Following a tradition dating back to 2002 at ZapThink and continuing at Intellyx since 2014, it’s time for Intellyx’s annual predictions for the coming year. If you’re a long-time fan, you know we have a twist to the typical annual prediction post: we actually critique our predictions from the previous year. To make things even more interesting, Charlie and I switch off, judging the other’s predictions. And now that he’s been with Intellyx for more than a year, this Cortex represents my first ...
"Grape Up leverages Cloud Native technologies and helps companies build software using microservices, and work the DevOps agile way. We've been doing digital innovation for the last 12 years," explained Daniel Heckman, of Grape Up in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
The Toyota Production System, a world-renowned production system is based on the "complete elimination of all waste". The "Toyota Way", grounded on continuous improvement dates to the 1860s. The methodology is widely proven to be successful yet there are still industries within and tangential to manufacturing struggling to adopt its core principles: Jidoka: a process should stop when an issue is identified prevents releasing defective products
Defining the term ‘monitoring’ is a difficult task considering the performance space has evolved significantly over the years. Lately, there has been a shift in the monitoring world, sparking a healthy debate regarding the definition and purpose of monitoring, through which a new term has emerged: observability. Some of that debate can be found in blogs by Charity Majors and Cindy Sridharan.
We seem to run this cycle with every new technology that comes along. A good idea with practical applications is born, then both marketers and over-excited users start to declare it is the solution for all or our problems. Compliments of Gartner, we know it generally as “The Hype Cycle”, but each iteration is a little different. 2018’s flavor will be serverless computing, and by 2018, I mean starting now, but going most of next year, you’ll be sick of it. We are already seeing people write such...
It’s “time to move on from DevOps and continuous delivery.” This was the provocative title of a recent article in ZDNet, in which Kelsey Hightower, staff developer advocate at Google Cloud Platform, suggested that “software shops should have put these concepts into action years ago.” Reading articles like this or listening to talks at most DevOps conferences might make you think that we’re entering a post-DevOps world. But vast numbers of organizations still struggle to start and drive transfo...
Let's do a visualization exercise. Imagine it's December 31, 2018, and you're ringing in the New Year with your friends and family. You think back on everything that you accomplished in the last year: your company's revenue is through the roof thanks to the success of your product, and you were promoted to Lead Developer. 2019 is poised to be an even bigger year for your company because you have the tools and insight to scale as quickly as demand requires. You're a happy human, and it's not just...
"Opsani helps the enterprise adopt containers, help them move their infrastructure into this modern world of DevOps, accelerate the delivery of new features into production, and really get them going on the container path," explained Ross Schibler, CEO of Opsani, and Peter Nickolov, CTO of Opsani, in this SYS-CON.tv interview at DevOps Summit at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.