Microservices Expo Authors: Elizabeth White, Mehdi Daoudi, Pat Romanski, Flint Brenton, Gordon Haff

Related Topics: Microservices Expo, Java IoT, Containers Expo Blog, Agile Computing, @CloudExpo

Microservices Expo: Article

Scrum Buts, “Meta” Thinking, and Agile Architecture

How would we ever expect to architect an enterprise if we didn’t take an Agile Architecture approach that dealt with change?

ZapThink has long bemoaned the Agile Manifesto paradox: that the point to the Manifesto was to be less dogmatic about software development, but today people are overly dogmatic about Agile, defeating its entire purpose. In fact, this paradox has found its way into what is perhaps the most popular of the Agile methodologies: Scrum. Not to worry, all you Scrum aficionados out there; we’re not going to teach you how to do Scrum in this ZapFlash. Instead, this article is more about how to think about a broad set of problems in a particular way, starting with Scrum Buts.

Enter the Scrum But
The notion of a Scrum But arose when it became clear that thousands of organizations were attempting to follow Scrum for their software development projects, but many of them were having problems with one or another of its tenets. As a result, they would say things like:

“We use Scrum, but Retrospectives are a waste of time, so we don't do them.”


“We use Scrum, but we can't build a piece of functionality in a month, so our Sprints are 6 weeks long.”

Where Retrospectives and Sprints are well-known Scrum best practices. Note that both of these statements follow the same “We use Scrum, but X so Y” pattern, hence the term Scrum But.

The question with Scrum Buts, of course, is what to do with them—or more specifically, how to think about them. There are two schools of thought:

  1. Scrum Buts are simply excuses not to do Scrum properly, and if you’re not doing it properly, then you’re not really doing it at all.
  2. Resolving Scrum Buts when they come up in order to achieve the desired result (software that meets the stakeholders’ needs) is actually a part of the Scrum methodology. As a result, Scrum Buts are expected and even welcomed.

From ZapThink’s perspective, option #1 is an example of taking a dogmatic approach, which is inherently non-Agile. Option #2 basically says that you can modify the rules if necessary (one of the four principles of the Agile Manifesto), even the rules of Scrum itself.

In other words, principle #2 is self-referential—which may be more Agile to be sure, but people have problems with self-reference. It brings up uncomfortable visions of the liar’s paradox: “everything I say is a lie,” or in more modern terms, the first rule of Fight Club (“do not talk about Fight Club.”) How can we make sense of the world or anything in it if we have to deal with self-reference paradoxes? If a rule of Scrum is “you can change the rules of Scrum,” then couldn’t anything and everything be Scrum? What use is that?

“Meta” Thinking
Fortunately, such problems of self-reference have a straightforward, if subtle solution. Instead of thinking of such statements as referring to themselves, think of them as actually two separate but related statements, where one relates to the other. We call this “meta” thinking.

In the case of Scrum Buts, we have the Scrum methodology and the Scrum meta-methodology. A meta-methodology is a methodology for creating methodologies. Remember, the Scrum methodology is a methodology for creating software. The Scrum meta-methodology is a methodology for creating or improving the Scrum methodology. So when someone says:

“When you say, ‘We use Scrum, but we can't build a piece of functionality in a month, so our Sprints are 6 weeks long,’ my recommendation is to try three 30-day Sprints before extending the length of the Sprint.”

That entire statement is a Scrum meta-methodology statement. Furthermore, without such statements, your methodology wouldn’t be Agile. The obvious conclusion is that all Agile methodologies are actually meta-methodologies. Otherwise they wouldn’t be Agile!

Avoiding the Hall of Mirrors
I’m sure one of you wise guys out there is thinking, what about methodologies for creating meta-methodologies? We’d call meta-meta-methodologies, of course. And what about methodologies for creating those, ad infinitum? We call this the hall of mirrors problem, because you only need to have two mirrors facing each other to get the infinite tunnel effect.

Simple answer: we don’t need a methodology for creating meta-methodologies. Instead, an informal approach will do. In general, we only want to go to the meta-meta step if there’s a bona fide reason to do so, as with Model-Driven Architecture, when they talk about meta-meta-models. But even the brainiacs at the OMG don’t spend much time thinking about meta-meta-meta-models—at least, I hope not!

We often get this question in our Licensed ZapThink Architect SOA course when we discuss meta-policies. A meta-policy, of course, is a policy that applies to other policies. Because the automation of policy-related processes is a core enabler of SOA governance, polices for how an organization performs such automation are quite important—and such policies are meta-policies. An example of a useful meta-policy: “we’ll configure our registry/repository’s design-time workflow to conform to our existing Service lifecycle.” Such a configuration means configuring the policies in the tool as a way of automating the corresponding workflow, and thus that statement is a meta-policy.

In this example, the hall of mirrors question focuses more on automation then on the policies themselves. Does it make sense to automate meta-policy related processes? The answer is basically no. It’s best to create, communicate, and enforce meta-policies manually, through human communication activities. Yes, you could think of that last statement as a meta-meta-policy, but thinking about it that way doesn’t really help you, so don’t bother.

More Meta-Thinking
Meta-models are well known now, and other than those plus meta-methodologies and meta-policies, where does meta thinking come up? Way back in 2005, ZapThink discussed the notion of meta-processes. A meta-process, of course, is a process for dealing with processes, for example, the process of composing Services to implement a business process. While we pointed out in that article that such meta-processes should be handled manually, we also predicted increasing automation of Service composition processes—a capability that is still well in the future (although if you have any examples, please let us know.)

Another meta that is central to ZapThink’s thinking is the meta-requirement. In particular, we’ve been talking about the meta-requirement of business agility as a fundamental driver of SOA, and by extension, Agile Architecture in general. When the business asks for an agile system, they are asking for a system that can respond to changing requirements—which is what makes such agility a meta-requirement. And remember, it’s because the business agility meta-requirement is an emergent property of the enterprise that requires us to characterize the enterprise itself as a complex system.

Finally, at the risk of belaboring the point, let’s talk about meta-architecture: what does it mean to architect an architecture? Yes, ZapThink has been spending a lot of our brain cycles on meta-architecture as well. We’ve been putting our stamp on how best to do SOA for several years now. Our students may be architecting their organizations and their component systems, but ZapThink has been architecting SOA itself. And now that we can stick a fork in that, it’s time to work on architecting Cloud architecture and more broadly, Agile Architecture.

The ZapThink Take
Meta thinking about something means more than just thinking about the thing, it means thinking about how the thing might change. Meta-processes are processes for changing processes. The reason we need meta-policies is because policies might change—otherwise we’d just hard-code them into our software and be done with them. The business agility meta-requirement is fundamentally a requirement for change.

Complex systems are themselves self-adapting. They are always in a state of change. How would we ever expect to architect a complex system like an enterprise if we didn’t take an Agile Architecture approach that worked at the meta level to deal with change? Perhaps the lack of a complex systems approach to traditional Enterprise Architecture—architectural approaches that presume a final to-be state—is why all such approaches are inherently flawed.

Our Scrum But example is an illuminating illustration of what we mean by an Agile Architectural approach. You could look at a list of Scrum Buts and say, this team is doomed to failure. They’ve taken the good bits to Scrum and stripped those out, and now they’re screwed. Alternatively, you could look at the same list and say, with a few bits of advice about how to deal with the Scrum Buts (in other words, the right meta-methodology), this team might be successful after all.

This admittedly oversimplified scenario has two outcomes: team crashes and burns, or team is successful in spite of their Scrum Buts. In complex systems theory, these outcomes are called attractors: given a set of circumstances, the end result will usually follow one of a set of patterns, in spite of the fact that different people are involved, each with their own skills and preferences. The system is subject to perturbations (the Scrum Buts, in our example), as well as constraints (the advice that makes up the meta-methodology), and the various identities of the people.

Without the appropriate advice, the attractor that is most likely to describe the outcome is the failure attractor. Clearly, the more Scrum Buts you have, and the more serious they are, the more likely your project will fail (although failure is still not certain). But with the proper meta-methodology, you can steer the project toward the success attractor, in spite of all the Scrum Buts and the various people on the team, who may be disgruntled, incompetent, overworked, or whatever.

Note that the success attractor is not a final state in a traditional sense. Rather, it allows for the fact that perturbations, constraints, and identities are always subject to change. Generalize our Scrum meta-methodology example to the level of the enterprise, and you can get a sense of what we mean by Agile Architecture. Can we design the complex system of the enterprise, a system consisting of human and technology subsystems, to move toward desirable attractors through the introduction of appropriate meta-policies, meta-processes, and meta-methodologies? That’s the million dollar meta-architecture question.

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

@MicroservicesExpo Stories
"We started a Master of Science in business analytics - that's the hot topic. We serve the business community around San Francisco so we educate the working professionals and this is where they all want to be," explained Judy Lee, Associate Professor and Department Chair at Golden Gate University, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
For over a decade, Application Programming Interface or APIs have been used to exchange data between multiple platforms. From social media to news and media sites, most websites depend on APIs to provide a dynamic and real-time digital experience. APIs have made its way into almost every device and service available today and it continues to spur innovations in every field of technology. There are multiple programming languages used to build and run applications in the online world. And just li...
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
The general concepts of DevOps have played a central role advancing the modern software delivery industry. With the library of DevOps best practices, tips and guides expanding quickly, it can be difficult to track down the best and most accurate resources and information. In order to help the software development community, and to further our own learning, we reached out to leading industry analysts and asked them about an increasingly popular tenet of a DevOps transformation: collaboration.
We call it DevOps but much of the time there’s a lot more discussion about the needs and concerns of developers than there is about other groups. There’s a focus on improved and less isolated developer workflows. There are many discussions around collaboration, continuous integration and delivery, issue tracking, source code control, code review, IDEs, and xPaaS – and all the tools that enable those things. Changes in developer practices may come up – such as developers taking ownership of code ...
The dynamic nature of the cloud means that change is a constant when it comes to modern cloud-based infrastructure. Delivering modern applications to end users, therefore, is a constantly shifting challenge. Delivery automation helps IT Ops teams ensure that apps are providing an optimal end user experience over hybrid-cloud and multi-cloud environments, no matter what the current state of the infrastructure is. To employ a delivery automation strategy that reflects your business rules, making r...
Cloud Governance means many things to many people. Heck, just the word cloud means different things depending on who you are talking to. While definitions can vary, controlling access to cloud resources is invariably a central piece of any governance program. Enterprise cloud computing has transformed IT. Cloud computing decreases time-to-market, improves agility by allowing businesses to adapt quickly to changing market demands, and, ultimately, drives down costs.
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
How is DevOps going within your organization? If you need some help measuring just how well it is going, we have prepared a list of some key DevOps metrics to track. These metrics can help you understand how your team is doing over time. The word DevOps means different things to different people. Some say it a culture and every vendor in the industry claims that their tools help with DevOps. Depending on how you define DevOps, some of these metrics may matter more or less to you and your team.
"CA has been doing a lot of things in the area of DevOps. Now we have a complete set of tool sets in order to enable customers to go all the way from planning to development to testing down to release into the operations," explained Aruna Ravichandran, Vice President of Global Marketing and Strategy at CA Technologies, in this SYS-CON.tv interview at DevOps Summit at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"We are an integrator of carrier ethernet and bandwidth to get people to connect to the cloud, to the SaaS providers, and the IaaS providers all on ethernet," explained Paul Mako, CEO & CTO of Massive Networks, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Grape Up leverages Cloud Native technologies and helps companies build software using microservices, and work the DevOps agile way. We've been doing digital innovation for the last 12 years," explained Daniel Heckman, of Grape Up in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"Outscale was founded in 2010, is based in France, is a strategic partner to Dassault Systémes and has done quite a bit of work with divisions of Dassault," explained Jackie Funk, Digital Marketing exec at Outscale, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"I focus on what we are calling CAST Highlight, which is our SaaS application portfolio analysis tool. It is an extremely lightweight tool that can integrate with pretty much any build process right now," explained Andrew Siegmund, Application Migration Specialist for CAST, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Let's do a visualization exercise. Imagine it's December 31, 2018, and you're ringing in the New Year with your friends and family. You think back on everything that you accomplished in the last year: your company's revenue is through the roof thanks to the success of your product, and you were promoted to Lead Developer. 2019 is poised to be an even bigger year for your company because you have the tools and insight to scale as quickly as demand requires. You're a happy human, and it's not just...
The enterprise data storage marketplace is poised to become a battlefield. No longer the quiet backwater of cloud computing services, the focus of this global transition is now going from compute to storage. An overview of recent storage market history is needed to understand why this transition is important. Before 2007 and the birth of the cloud computing market we are witnessing today, the on-premise model hosted in large local data centers dominated enterprise storage. Key marketplace play...
Cavirin Systems has just announced C2, a SaaS offering designed to bring continuous security assessment and remediation to hybrid environments, containers, and data centers. Cavirin C2 is deployed within Amazon Web Services (AWS) and features a flexible licensing model for easy scalability and clear pay-as-you-go pricing. Although native to AWS, it also supports assessment and remediation of virtual or container instances within Microsoft Azure, Google Cloud Platform (GCP), or on-premise. By dr...
With continuous delivery (CD) almost always in the spotlight, continuous integration (CI) is often left out in the cold. Indeed, it's been in use for so long and so widely, we often take the model for granted. So what is CI and how can you make the most of it? This blog is intended to answer those questions. Before we step into examining CI, we need to look back. Software developers often work in small teams and modularity, and need to integrate their changes with the rest of the project code b...
Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Kubernetes was originally built by Google, leveraging years of experience with managing container workloads, and is now a Cloud Native Compute Foundation (CNCF) project. Kubernetes has been widely adopted by the community, supported on all major public and private cloud providers, and is gaining rapid adoption in enterprises. However, Kubernetes may seem intimidating and complex ...