Welcome!

Microservices Expo Authors: Zakia Bouachraoui, Elizabeth White, Pat Romanski, Liz McMillan, Yeshim Deniz

Related Topics: Microservices Expo, Java IoT, Containers Expo Blog, Agile Computing, @CloudExpo

Microservices Expo: Article

Scrum Buts, “Meta” Thinking, and Agile Architecture

How would we ever expect to architect an enterprise if we didn’t take an Agile Architecture approach that dealt with change?

ZapThink has long bemoaned the Agile Manifesto paradox: that the point to the Manifesto was to be less dogmatic about software development, but today people are overly dogmatic about Agile, defeating its entire purpose. In fact, this paradox has found its way into what is perhaps the most popular of the Agile methodologies: Scrum. Not to worry, all you Scrum aficionados out there; we’re not going to teach you how to do Scrum in this ZapFlash. Instead, this article is more about how to think about a broad set of problems in a particular way, starting with Scrum Buts.

Enter the Scrum But
The notion of a Scrum But arose when it became clear that thousands of organizations were attempting to follow Scrum for their software development projects, but many of them were having problems with one or another of its tenets. As a result, they would say things like:

“We use Scrum, but Retrospectives are a waste of time, so we don't do them.”

Or:

“We use Scrum, but we can't build a piece of functionality in a month, so our Sprints are 6 weeks long.”

Where Retrospectives and Sprints are well-known Scrum best practices. Note that both of these statements follow the same “We use Scrum, but X so Y” pattern, hence the term Scrum But.

The question with Scrum Buts, of course, is what to do with them—or more specifically, how to think about them. There are two schools of thought:

  1. Scrum Buts are simply excuses not to do Scrum properly, and if you’re not doing it properly, then you’re not really doing it at all.
  2. Resolving Scrum Buts when they come up in order to achieve the desired result (software that meets the stakeholders’ needs) is actually a part of the Scrum methodology. As a result, Scrum Buts are expected and even welcomed.

From ZapThink’s perspective, option #1 is an example of taking a dogmatic approach, which is inherently non-Agile. Option #2 basically says that you can modify the rules if necessary (one of the four principles of the Agile Manifesto), even the rules of Scrum itself.

In other words, principle #2 is self-referential—which may be more Agile to be sure, but people have problems with self-reference. It brings up uncomfortable visions of the liar’s paradox: “everything I say is a lie,” or in more modern terms, the first rule of Fight Club (“do not talk about Fight Club.”) How can we make sense of the world or anything in it if we have to deal with self-reference paradoxes? If a rule of Scrum is “you can change the rules of Scrum,” then couldn’t anything and everything be Scrum? What use is that?

“Meta” Thinking
Fortunately, such problems of self-reference have a straightforward, if subtle solution. Instead of thinking of such statements as referring to themselves, think of them as actually two separate but related statements, where one relates to the other. We call this “meta” thinking.

In the case of Scrum Buts, we have the Scrum methodology and the Scrum meta-methodology. A meta-methodology is a methodology for creating methodologies. Remember, the Scrum methodology is a methodology for creating software. The Scrum meta-methodology is a methodology for creating or improving the Scrum methodology. So when someone says:

“When you say, ‘We use Scrum, but we can't build a piece of functionality in a month, so our Sprints are 6 weeks long,’ my recommendation is to try three 30-day Sprints before extending the length of the Sprint.”

That entire statement is a Scrum meta-methodology statement. Furthermore, without such statements, your methodology wouldn’t be Agile. The obvious conclusion is that all Agile methodologies are actually meta-methodologies. Otherwise they wouldn’t be Agile!

Avoiding the Hall of Mirrors
I’m sure one of you wise guys out there is thinking, what about methodologies for creating meta-methodologies? We’d call meta-meta-methodologies, of course. And what about methodologies for creating those, ad infinitum? We call this the hall of mirrors problem, because you only need to have two mirrors facing each other to get the infinite tunnel effect.

Simple answer: we don’t need a methodology for creating meta-methodologies. Instead, an informal approach will do. In general, we only want to go to the meta-meta step if there’s a bona fide reason to do so, as with Model-Driven Architecture, when they talk about meta-meta-models. But even the brainiacs at the OMG don’t spend much time thinking about meta-meta-meta-models—at least, I hope not!

We often get this question in our Licensed ZapThink Architect SOA course when we discuss meta-policies. A meta-policy, of course, is a policy that applies to other policies. Because the automation of policy-related processes is a core enabler of SOA governance, polices for how an organization performs such automation are quite important—and such policies are meta-policies. An example of a useful meta-policy: “we’ll configure our registry/repository’s design-time workflow to conform to our existing Service lifecycle.” Such a configuration means configuring the policies in the tool as a way of automating the corresponding workflow, and thus that statement is a meta-policy.

In this example, the hall of mirrors question focuses more on automation then on the policies themselves. Does it make sense to automate meta-policy related processes? The answer is basically no. It’s best to create, communicate, and enforce meta-policies manually, through human communication activities. Yes, you could think of that last statement as a meta-meta-policy, but thinking about it that way doesn’t really help you, so don’t bother.

More Meta-Thinking
Meta-models are well known now, and other than those plus meta-methodologies and meta-policies, where does meta thinking come up? Way back in 2005, ZapThink discussed the notion of meta-processes. A meta-process, of course, is a process for dealing with processes, for example, the process of composing Services to implement a business process. While we pointed out in that article that such meta-processes should be handled manually, we also predicted increasing automation of Service composition processes—a capability that is still well in the future (although if you have any examples, please let us know.)

Another meta that is central to ZapThink’s thinking is the meta-requirement. In particular, we’ve been talking about the meta-requirement of business agility as a fundamental driver of SOA, and by extension, Agile Architecture in general. When the business asks for an agile system, they are asking for a system that can respond to changing requirements—which is what makes such agility a meta-requirement. And remember, it’s because the business agility meta-requirement is an emergent property of the enterprise that requires us to characterize the enterprise itself as a complex system.

Finally, at the risk of belaboring the point, let’s talk about meta-architecture: what does it mean to architect an architecture? Yes, ZapThink has been spending a lot of our brain cycles on meta-architecture as well. We’ve been putting our stamp on how best to do SOA for several years now. Our students may be architecting their organizations and their component systems, but ZapThink has been architecting SOA itself. And now that we can stick a fork in that, it’s time to work on architecting Cloud architecture and more broadly, Agile Architecture.

The ZapThink Take
Meta thinking about something means more than just thinking about the thing, it means thinking about how the thing might change. Meta-processes are processes for changing processes. The reason we need meta-policies is because policies might change—otherwise we’d just hard-code them into our software and be done with them. The business agility meta-requirement is fundamentally a requirement for change.

Complex systems are themselves self-adapting. They are always in a state of change. How would we ever expect to architect a complex system like an enterprise if we didn’t take an Agile Architecture approach that worked at the meta level to deal with change? Perhaps the lack of a complex systems approach to traditional Enterprise Architecture—architectural approaches that presume a final to-be state—is why all such approaches are inherently flawed.

Our Scrum But example is an illuminating illustration of what we mean by an Agile Architectural approach. You could look at a list of Scrum Buts and say, this team is doomed to failure. They’ve taken the good bits to Scrum and stripped those out, and now they’re screwed. Alternatively, you could look at the same list and say, with a few bits of advice about how to deal with the Scrum Buts (in other words, the right meta-methodology), this team might be successful after all.

This admittedly oversimplified scenario has two outcomes: team crashes and burns, or team is successful in spite of their Scrum Buts. In complex systems theory, these outcomes are called attractors: given a set of circumstances, the end result will usually follow one of a set of patterns, in spite of the fact that different people are involved, each with their own skills and preferences. The system is subject to perturbations (the Scrum Buts, in our example), as well as constraints (the advice that makes up the meta-methodology), and the various identities of the people.

Without the appropriate advice, the attractor that is most likely to describe the outcome is the failure attractor. Clearly, the more Scrum Buts you have, and the more serious they are, the more likely your project will fail (although failure is still not certain). But with the proper meta-methodology, you can steer the project toward the success attractor, in spite of all the Scrum Buts and the various people on the team, who may be disgruntled, incompetent, overworked, or whatever.

Note that the success attractor is not a final state in a traditional sense. Rather, it allows for the fact that perturbations, constraints, and identities are always subject to change. Generalize our Scrum meta-methodology example to the level of the enterprise, and you can get a sense of what we mean by Agile Architecture. Can we design the complex system of the enterprise, a system consisting of human and technology subsystems, to move toward desirable attractors through the introduction of appropriate meta-policies, meta-processes, and meta-methodologies? That’s the million dollar meta-architecture question.

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

Microservices Articles
When building large, cloud-based applications that operate at a high scale, it’s important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. “Fly two mistakes high” is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee A...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...