Welcome!

Microservices Expo Authors: Elizabeth White, APM Blog, Liz McMillan, Pat Romanski, XebiaLabs Blog

Related Topics: Microservices Expo, Java IoT, Agile Computing

Microservices Expo: Article

Agile - Waterfall: Global and Local Optimizations

Is NP=P and why should we really care?

Did you notice that when something happens to you, it seems to occur to other people as well? For example when you have toddlers, suddenly you see toddlers everywhere - and naturally they are all misbehaving compared to yours J, or when you're planning your wedding, all around you, people are planning theirs, and white dresses become scarce. Lately I had this feeling myself; I was reading a David Baldacci thriller, when half way through, he introduces NP problems and the consequences of finding that P=NP. In addition, I have lately been reading many articles about Critical chain project management. I feel compelled to contribute and provide my take and insights.

There are two kinds of problems, the easy ones which we solve at school, and the hard ones. We might remember the method to solve a quadratic equation by using the x = [-b/2 ± √((b/2)^2 - ac)] /a formula. We are less sure on advising a salesperson how to plan a sales trip to 20 cities in which he can visit every city once and has a limited budget (travelling salesperson problem). When it comes to packing for the holidays we are truly baffled. How are we supposed to fit all the stuff into one checked-in luggage? After continuous deliberation, we land in the exotic resort only to find our bathing suit missing...

Unfortunately and also luckily (more on the luck aspect - later) most of the problems around us are unsolvable, or should I say - not computable in an optimal manner in a reasonable amount of time.

How is this related to project management?

The difficulties we are experiencing managing projects successfully in a resource constrained environment stem from the fact that the Resource Constrained Multi-project Scheduling Problem is very difficult to solve in an optimal manner. Moreover it can't be optimized at all. Multi Project Resource Constrained Scheduling (acronym MPRCS) is known as a NP complete (or NP hard - until this day I am not sure if I understand the exact difference) complexity type problem. In a nutshell, NP problems are very complex mathematical problems. Actually, as mentioned above, most of the problems that we are facing are of NP complexity. For example, those of you who like to travel to Alaska and need to choose equipment from a long list of possibilities, with capacity constraints, are facing an NP problem (knapsack problem).

The theory claims that if we solve one NP problem we can solve all of them. As a byproduct of the solution we would receive a noble prize, render the entire world of encryption useless, and probably get kidnapped by the NSA and never heard of again.  So far this hasn't been the case and NP problems are not computable in an optimal manner in a reasonable amount of time. Hence - luckily for us - long encryption keys are at present undecipherable in a reasonable time, which enables secure global communication networks.  Conspiracy theorists maintain that governments have already developed computational abilities that can solve NP problems - rendering encryption as we know it useless. Dan Brown in his bestseller book Digital Fortress and David Baldacci in his book Simple Genius venture down this path.

Back to project management. Since scheduling is a NP problem, we can't optimally solve a multi project resource constrained system.

Wait just a minute - am I saying that there aren't computerized tools to optimally schedule a bunch of related projects who share resources??? Truth be told, there are tools offering nearly optimal solutions and granted, while they do not provide an optimal solution they calculate, using complex algorithms, a good enough solution. The time required for providing this solution can be very long, depending on the complexity of the problem. There are environments which benefit from these sophisticated optimization tools. Consider for example a medium-size factory with 100 machines and several thousand components. Running the scheduling software for the defined production floor is lengthy and can take up to 3 days. The specific output of the software is a production plan for the explicit setting. The drawback is that once implemented, any unforeseen change in production requires: either rerunning the optimization software which doesn't make sense duration wise, or applying quick fixes which render the nearly optimal production plan obsolete. If ever you watched an operating factory, you have seen the chaotic nature and constant changes with which it is plagued.

There are industries in which nearly optimal scheduling is vital and as the environment is complex, they must invest in a sophisticated software optimization tool (do remember that the solution is nearly optimal). One example is the airline industry. They can't manually allocate numerous aircrafts and particular crews across several hundred destinations worldwide. When Lufthansa allocates specific aircrafts, along with the constraint of having each aircraft operated by a predefined crew, it makes sense to purchase an advanced and pricey optimization tool. Indeed, once a month the scheduling team runs the optimization application which serves as the basis for aircraft and crew allocation, ticket pricing, and routing worldwide. However if one of the pilots is having the shrimp in coconut sauce for dinner and suffers from food poisoning the next day, entire segments of the optimized schedule collapse and emergency escalation protocols are initiated. The central scheduling team in Frankfurt must find a quick fix to the shrimp ordeal.

Ok - for most of our projects it doesn't make sense to invest so much in building the infrastructure to solve, in a near optimal manner, the multi project resource constrained problem.  Especially as the plan is useless once changes occur. All it takes is for a critical resource to be sick one day, for the entire plan to go bust. What is then the alternative? Heuristics - or as we better know them - rules of thumb. These enable local and semi global optimizations. The two famous heuristics to resource constrained project scheduling are: Agile and critical chain. One handles the local optimization and the other semi global optimization. While Goldratt who wrote about critical chain project management knew about heuristics and global optimization, Agile proponents don't really think of it as a local optimization mechanism, but we will leave it for future articles.

More on critical chain, and Goldratt's approach to solving the complexity of resource constrained scheduling, some other time.

A FREE voice over recording of this article is found here.

You can check out my book: The Agile PMO - leading the effective, value-driven, project management office. - - where I further discuss these concepts:

The Agile PMO

More Stories By Michael Nir

Michael Nir - President of Sapir Consulting - (M.Sc. Engineering) has been providing operational, organizational and management consulting and training for over 15 years. He is passionate about Gestalt theory and practice, which complements his engineering background and contributes to his understanding of individual and team dynamics in business. Michael authored 8 Bestsellers in the fields of Influencing, Agile, Teams, Leadership and others. Michael's experience includes significant expertise in the telecoms, hi-tech, software development, R&D environments and petrochemical & infrastructure industries. He develops creative and innovative solutions in project and product management, process improvement, leadership, and team building programs. Michael's professional background is analytical and technical; however, he has a keen interest in human interactions and behaviors. He holds two engineering degrees from the prestigious Technion Institute of Technology: a Bachelor of civil engineering and Masters of Industrial engineering. He has balanced his technical side with the extensive study and practice of Gestalt Therapy and "Instrumental Enrichment," a philosophy of mediated learning. In his consulting and training engagements, Michael combines both the analytical and technical world with his focus on people, delivering unique and meaningful solutions, and addressing whole systems.

@MicroservicesExpo Stories
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, will compare the Jevons Paradox to modern-day enterprise IT, e...
As applications are promoted from the development environment to the CI or the QA environment and then into the production environment, it is very common for the configuration settings to be changed as the code is promoted. For example, the settings for the database connection pools are typically lower in development environment than the QA/Load Testing environment. The primary reason for the existence of the configuration setting differences is to enhance application performance. However, occas...
More and more companies are looking to microservices as an architectural pattern for breaking apart applications into more manageable pieces so that agile teams can deliver new features quicker and more effectively. What this pattern has done more than anything to date is spark organizational transformations, setting the foundation for future application development. In practice, however, there are a number of considerations to make that go beyond simply “build, ship, and run,” which changes ho...
SYS-CON Events announced today the Kubernetes and Google Container Engine Workshop, being held November 3, 2016, in conjunction with @DevOpsSummit at 19th Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA. This workshop led by Sebastian Scheele introduces participants to Kubernetes and Google Container Engine (GKE). Through a combination of instructor-led presentations, demonstrations, and hands-on labs, students learn the key concepts and practices for deploying and maintainin...
In his general session at 18th Cloud Expo, Lee Atchison, Principal Cloud Architect and Advocate at New Relic, discussed cloud as a ‘better data center’ and how it adds new capacity (faster) and improves application availability (redundancy). The cloud is a ‘Dynamic Tool for Dynamic Apps’ and resource allocation is an integral part of your application architecture, so use only the resources you need and allocate /de-allocate resources on the fly.
While DevOps promises a better and tighter integration among an organization’s development and operation teams and transforms an application life cycle into a continual deployment, Chef and Azure together provides a speedy, cost-effective and highly scalable vehicle for realizing the business values of this transformation. In his session at @DevOpsSummit at 19th Cloud Expo, Yung Chou, a Technology Evangelist at Microsoft, will present a unique opportunity to witness how Chef and Azure work tog...
When scaling agile / Scrum, we invariable run into the alignment vs autonomy problem. In short: you cannot have autonomous self directing teams if they have no clue in what direction they should go, or even shorter: Alignment breeds autonomy. But how do we create alignment? and what tools can we use to quickly evaluate if what we want to do is part of the mission or better left out? Niel Nickolaisen created the Purpose Alignment model and I use it with innovation labs in large enterprises to de...
SYS-CON Events announced today that Numerex Corp, a leading provider of managed enterprise solutions enabling the Internet of Things (IoT), will exhibit at the 19th International Cloud Expo | @ThingsExpo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Numerex Corp. (NASDAQ:NMRX) is a leading provider of managed enterprise solutions enabling the Internet of Things (IoT). The Company's solutions produce new revenue streams or create operating...
If you’re responsible for an application that depends on the data or functionality of various IoT endpoints – either sensors or devices – your brand reputation depends on the security, reliability, and compliance of its many integrated parts. If your application fails to deliver the expected business results, your customers and partners won't care if that failure stems from the code you developed or from a component that you integrated. What can you do to ensure that the endpoints work as expect...
SYS-CON Events announced today that Tintri Inc., a leading producer of VM-aware storage (VAS) for virtualization and cloud environments, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Tintri VM-aware storage is the simplest for virtualized applications and cloud. Organizations including GE, Toyota, United Healthcare, NASA and 6 of the Fortune 15 have said “No to LUNs.” With Tintri they mana...
Analysis of 25,000 applications reveals 6.8% of packages/components used included known defects. Organizations standardizing on components between 2 - 3 years of age can decrease defect rates substantially. Open source and third-party packages/components live at the heart of high velocity software development organizations. Today, an average of 106 packages/components comprise 80 - 90% of a modern application, yet few organizations have visibility into what components are used where.
Throughout history, various leaders have risen up and tried to unify the world by conquest. Fortunately, none of their plans have succeeded. The world goes on just fine with each country ruling itself; no single ruler is necessary. That’s how it is with the container platform ecosystem, as well. There’s no need for one all-powerful, all-encompassing container platform. Think about any other technology sector out there – there are always multiple solutions in every space. The same goes for conta...
Let's recap what we learned from the previous chapters in the series: episode 1 and episode 2. We learned that a good rollback mechanism cannot be designed without having an intimate knowledge of the application architecture, the nature of your components and their dependencies. Now that we know what we have to restore and in which order, the question is how?
Digitization is driving a fundamental change in society that is transforming the way businesses work with their customers, their supply chains and their people. Digital transformation leverages DevOps best practices, such as Agile Parallel Development, Continuous Delivery and Agile Operations to capitalize on opportunities and create competitive differentiation in the application economy. However, information security has been notably absent from the DevOps movement. Speed doesn’t have to negat...
Your business relies on your applications and your employees to stay in business. Whether you develop apps or manage business critical apps that help fuel your business, what happens when users experience sluggish performance? You and all technical teams across the organization – application, network, operations, among others, as well as, those outside the organization, like ISPs and third-party providers – are called in to solve the problem.
Enterprise IT has been in the era of Hybrid Cloud for some time now. But it seems most conversations about Hybrid are focused on integrating AWS, Microsoft Azure, or Google ECM into existing on-premises systems. Where is all the Private Cloud? What do technology providers need to do to make their offerings more compelling? How should enterprise IT executives and buyers define their focus, needs, and roadmap, and communicate that clearly to the providers?
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management solutions, helping companies worldwide activate their data to drive more value and business insight and to transform moder...
Much of the value of DevOps comes from a (renewed) focus on measurement, sharing, and continuous feedback loops. In increasingly complex DevOps workflows and environments, and especially in larger, regulated, or more crystallized organizations, these core concepts become even more critical. In his session at @DevOpsSummit at 18th Cloud Expo, Andi Mann, Chief Technology Advocate at Splunk, showed how, by focusing on 'metrics that matter,' you can provide objective, transparent, and meaningful f...
There is little doubt that Big Data solutions will have an increasing role in the Enterprise IT mainstream over time. Big Data at Cloud Expo - to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA - has announced its Call for Papers is open. Cloud computing is being adopted in one form or another by 94% of enterprises today. Tens of billions of new devices are being connected to The Internet of Things. And Big Data is driving this bus. An exponential increase is...
The many IoT deployments around the world are busy integrating smart devices and sensors into their enterprise IT infrastructures. Yet all of this technology – and there are an amazing number of choices – is of no use without the software to gather, communicate, and analyze the new data flows. Without software, there is no IT. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the protocols that communicate data and the emerging data analy...