Microservices Expo Authors: Yeshim Deniz, Elizabeth White, Flint Brenton, Liz McMillan, Pat Romanski

Related Topics: @DevOpsSummit, Microservices Expo, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Post

DevOps Insights into Conway’s Law By @TheEbizWizard | @DevOpsSummit #DevOps #Microservices

Do organizational structures determine system design, or vice versa?

Both digital transformation and DevOps are organizational and cultural transformations more so than they are technology changes – although in both cases, technology plays a large part in driving the organizational change necessary to achieve the business value for either effort. Just how the organizational changes and technology changes work together, however, is a difficult question.

Fortunately for us, this question is an old one – 47 years old, in fact. In a 1968 paper, computer scientist Melvin Conway wrote, “Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization’s communication structure.”

Conway produced no evidence of this statement, and his paper was initially rejected as a result – but to this day, we refer to this statement as Conway’s Law.

Law, however, is decidedly an overstatement. Observation is actually more accurate. In fact, the Wikipedia page for Conway’s Law states that “Although sometimes construed as humorous, Conway’s law was intended as a valid sociological observation.”

Whether the statement be humorous observation or law, however, today Conway’s Law plays a central role in our efforts to break down organizational silos in order to improve business velocity and better meet the needs of customers – the purported goals of devops and digital transformation, respectively.

For Conway’s Law to be a useful tool, however, we need a better causal story. Can changing our organizational structures impact our technology? Or more importantly, how does technology impact our organizational structures? And why?

A Brief History of Conway’s Law
Melvin Conway
was an early computer scientist whose most interesting accomplishment other than the law itself may have been his part in creating the compiler for the original 1984 version of Mac Pascal.

His eponymous law, however, apparently languished until sometime before 1996 or perhaps even earlier than 1991, when open source guru Eric Raymond included Conway’s Law in his Jargon File. He restated it with the snarky example, “If you have four groups working on a compiler, you’ll get a 4-pass compiler.”

Our story then gets more interesting when the Harvard Business School attempted to put some meat on the bones in a 2008 study. They use an objective measure of the modularity of software to find evidence for Conway’s Law. They conclude:

  • A product’s architecture tends to mirror the structure of the organization within which it is developed.
  • New organizational arrangements can have a distinct impact on the nature of the resulting design, and hence may affect product performance in unintended ways.

It’s important to note two important updates to the context of Conway’s Law by the time of the 2008 study: an organization’s “communication structure” becomes the structure of the organization itself, while “system design” becomes “product architecture” – with the constraint to product architecture being the scope of the study, rather than any sort of judgment on the scope of Conway’s Law itself.

What the 2008 study is essentially lacking, furthermore, is a discussion of the underlying causal principles behind the law. Instead, it is more or less taken for granted that siloed technology teams will create modular systems with the modularity aligning to the team structure simply because that’s the way people behave when they’re in teams.

The reason this question of causality is so important is because it goes to the heart of how we might use Conway’s Law to actually improve things. In particular, will changing organizational structures enable us to build better software?

Jonny Leroy and Matt Simons of ThoughtWorks explored this question when they coined the term “Inverse Conway Maneuver” in an article in the 2010 issue of the Cutter IT Journal. They state:

Conway’s Law … can be summarized as “Dysfunctional organizations tend to create dysfunctional applications.” To paraphrase Einstein, you can’t fix a problem from within the same mindset that created it, so it is often worth investigating whether restructuring your organization or team would prevent the new application from displaying all the same structural dysfunctions as the original. In what could be termed an “inverse Conway maneuver,” you may want to begin by breaking down silos that constrain the team’s ability to collaborate effectively.

The ThoughtWorks article takes the context of Conway’s Law from observational to normative: not satisfied simply to reflect on the way things work, they take the important step of opining on how things should – and should not work. They also intentionally focus on the negative: rather than simply describing how to build good software, they note that Conway’s Law is about building dysfunctional software.

Reversing the Inverse Conway Maneuver
It’s no surprise that ThoughtWorks’ focus is on changing organizational structures in order to build better software – after all, ThoughtWorks is a software development organization, and most modern software development thinking, including both the Agile and Lean movements, focuses on organizational change in furtherance of better software.

For digital transformation efforts, however, we must reverse this discussion. Technology change is driving changing customer preferences and behavior, which in turn are driving organizational change across increasingly software-driven enterprises.

The causality question behind Conway’s Law, therefore, is less about how changing software organizations can lead to better software, but rather how companies can best leverage changing technology in order to transform their organizations.

Hints at how to answer this question surprisingly come from the world of devops – surprising because the focus of devops is ostensibly on building and deploying better software more quickly. Be that as it may, there’s no question that technology change is a primary facilitator and driving force for the devops cultural and organizational shifts.

Connecting the Dots to Conway’s Law
If we didn’t have the cloud computing example of fully automated deployment and operational environments, and if we didn’t have today’s dramatic innovations in continuous development, continuous integration, and continuous delivery tooling, then devops would never have left the whiteboard stage. There’s no question that the devops story is a tale of technology-driven organizational change.

The DevOps technology landscape, however, doesn’t have an end-to-end, seamless technology story. On the contrary, this landscape is cluttered with dozens of various tools, many open source, all of which are in various stages of maturity. Therefore, it’s not readily apparent how to apply Conway’s Law, since we’re trying to leverage diverse toolchains of tools and technologies in order to help evolve our organizational structures.

In fact, Conway’s Law describes how such a diverse tooling marketplace came to be in the first place, as open source teams are generally quite modular, and thus will produce modular software.

Once we’ve solved the organizational challenges of digital and DevOps, breaking down silos in order to deliver customer value at velocity, then we can expect Conway’s Law to kick in again, and predict the rise of end-to-end software solutions as the result of horizontally self-organized teams.

It’s the middle piece, however, we’re struggling with now: how do we leverage a diverse set of disparate technologies to facilitate the cross-cutting reorganization of our businesses, contrary to the observation of Conway’s Law? And do we really think going against Conway’s Law will actually work?

Not to fear. In fact, the technologies that underpin DevOps aren’t as diverse and disparate as the application of Conway’s Law might suggest. True, the creators of all the tools in our DevOps or digital tool belts are working mostly independently of each other, a pattern of behavior which in the past has led to incompatible software mishmashes.

Today, however, we’re seeing the rise of what we might call crowdsourced architecture, as all of these teams work within the context of mature communication protocols, RESTful interfaces, and other emerging architectural trends like containerization and microservices.

As a result, mostly independent doesn’t mean completely independent. Instead, we have the loosely connected communication structure that assures us that Conway’s Law is still alive and well. What’s changed is the context for this notion of an organizational communication structure.

Conway was referring to the communication structures of companies or software development organizations or perhaps individual software development teams. Today, however, we’re talking about the broader technology community itself.

The Intellyx Take
There was no way Melvin Conway could have observed such crowdsourced architectural maturity back in 1968, because the telephone and snail mail-based communication structures of the day weren’t conducive to crowdsourcing anything. In contrast, today we have a plethora of tools and processes for facilitating communication and collaboration across traditional projects, teams, and open source efforts.

The end result is essentially a two-level application of Conway’s Law: a collaborative extended community of technologists that creates not simply a collection of disparate tools but rather chainable tools that leverage crowdsourced architectural principles to facilitate a level of coordination and interactivity we’ve never seen before.

This coordinated technology environment, in turn, facilitates the reorganization within companies, as they now have the tools they need to break down organizational silos, and people within those companies self-organize along horizontal lines, connecting customer experience to back-office software development and operations.

Conway’s Law, therefore, does work both ways. Organizational structures impact system design, and system architectures impact organizational structures as well. In the final analysis, however, Conway’s Law remains stubbornly observational. The underlying causal story – why such observations are remarkably universal – remains to be told. Stay tuned!

Intellyx advises companies on their digital transformation initiatives and helps vendors communicate their agility stories. As of the time of writing, none of the organizations mentioned in this article are Intellyx customers. Image credit: Melvin Conway.

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

@MicroservicesExpo Stories
Without a clear strategy for cost control and an architecture designed with cloud services in mind, costs and operational performance can quickly get out of control. To avoid multiple architectural redesigns requires extensive thought and planning. Boundary (now part of BMC) launched a new public-facing multi-tenant high resolution monitoring service on Amazon AWS two years ago, facing challenges and learning best practices in the early days of the new service.
You often hear the two titles of "DevOps" and "Immutable Infrastructure" used independently. In his session at DevOps Summit, John Willis, Technical Evangelist for Docker, covered the union between the two topics and why this is important. He provided an overview of Immutable Infrastructure then showed how an Immutable Continuous Delivery pipeline can be applied as a best practice for "DevOps." He ended the session with some interesting case study examples.
Don’t go chasing waterfall … development, that is. According to a recent post by Madison Moore on Medium featuring insights from several software delivery industry leaders, waterfall is – while still popular – not the best way to win in the marketplace. With methodologies like Agile, DevOps and Continuous Delivery becoming ever more prominent over the past 15 years or so, waterfall is old news. Or, is it? Moore cites a recent study by Gartner: “According to Gartner’s IT Key Metrics Data report, ...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
All organizations that did not originate this moment have a pre-existing culture as well as legacy technology and processes that can be more or less amenable to DevOps implementation. That organizational culture is influenced by the personalities and management styles of Executive Management, the wider culture in which the organization is situated, and the personalities of key team members at all levels of the organization. This culture and entrenched interests usually throw a wrench in the work...
We all know that end users experience the internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices - not doing so will be a path to eventual ...
The next XaaS is CICDaaS. Why? Because CICD saves developers a huge amount of time. CD is an especially great option for projects that require multiple and frequent contributions to be integrated. But… securing CICD best practices is an emerging, essential, yet little understood practice for DevOps teams and their Cloud Service Providers. The only way to get CICD to work in a highly secure environment takes collaboration, patience and persistence. Building CICD in the cloud requires rigorous ar...
"This all sounds great. But it's just not realistic." This is what a group of five senior IT executives told me during a workshop I held not long ago. We were working through an exercise on the organizational characteristics necessary to successfully execute a digital transformation, and the group was doing their ‘readout.' The executives loved everything we discussed and agreed that if such an environment existed, it would make transformation much easier. They just didn't believe it was reali...
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his general session at @DevOpsSummit at 19th Cloud Expo, Eric Robertson, General Manager at CollabNet, will discuss how customers are able to achieve a level of transparency that e...
"DivvyCloud as a company set out to help customers automate solutions to the most common cloud problems," noted Jeremy Snyder, VP of Business Development at DivvyCloud, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
We all know that end users experience the Internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices – not doing so will be a path to eventual b...
"Opsani helps the enterprise adopt containers, help them move their infrastructure into this modern world of DevOps, accelerate the delivery of new features into production, and really get them going on the container path," explained Ross Schibler, CEO of Opsani, and Peter Nickolov, CTO of Opsani, in this SYS-CON.tv interview at DevOps Summit at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Docker is sweeping across startups and enterprises alike, changing the way we build and ship applications. It's the most prominent and widely known software container platform, and it's particularly useful for eliminating common challenges when collaborating on code (like the "it works on my machine" phenomenon that most devs know all too well). With Docker, you can run and manage apps side-by-side - in isolated containers - resulting in better compute density. It's something that many developer...
The “Digital Era” is forcing us to engage with new methods to build, operate and maintain applications. This transformation also implies an evolution to more and more intelligent applications to better engage with the customers, while creating significant market differentiators. In both cases, the cloud has become a key enabler to embrace this digital revolution. So, moving to the cloud is no longer the question; the new questions are HOW and WHEN. To make this equation even more complex, most ...
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
What's the role of an IT self-service portal when you get to continuous delivery and Infrastructure as Code? This general session showed how to create the continuous delivery culture and eight accelerators for leading the change. Don Demcsak is a DevOps and Cloud Native Modernization Principal for Dell EMC based out of New Jersey. He is a former, long time, Microsoft Most Valuable Professional, specializing in building and architecting Application Delivery Pipelines for hybrid legacy, and cloud ...
Many organizations are now looking to DevOps maturity models to gauge their DevOps adoption and compare their maturity to their peers. However, as enterprise organizations rush to adopt DevOps, moving past experimentation to embrace it at scale, they are in danger of falling into the trap that they have fallen into time and time again. Unfortunately, we've seen this movie before, and we know how it ends: badly.
"I focus on what we are calling CAST Highlight, which is our SaaS application portfolio analysis tool. It is an extremely lightweight tool that can integrate with pretty much any build process right now," explained Andrew Siegmund, Application Migration Specialist for CAST, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"We view the cloud not as a specific technology but as a way of doing business and that way of doing business is transforming the way software, infrastructure and services are being delivered to business," explained Matthew Rosen, CEO and Director at Fusion, in this SYS-CON.tv interview at 18th Cloud Expo (http://www.CloudComputingExpo.com), held June 7-9 at the Javits Center in New York City, NY.
In his session at Cloud Expo, Alan Winters, U.S. Head of Business Development at MobiDev, presented a success story of an entrepreneur who has both suffered through and benefited from offshore development across multiple businesses: The smart choice, or how to select the right offshore development partner Warning signs, or how to minimize chances of making the wrong choice Collaboration, or how to establish the most effective work processes Budget control, or how to maximize project result...