Welcome!

SOA & WOA Authors: Paul Diamond, Elizabeth White, Liz McMillan, Pat Romanski, Patrick Carey

Related Topics: Java, SOA & WOA, AJAX & REA

Java: Article

Agile Adoption – Crossing the Chasm

Why is it so difficult to switch to agile process from waterfall?

Everybody acknowledges that IT has to plays a key role in any new service or product design. Therefore, IT needs to align with business and be flexible to changing business needs. It is a question of how to be agile rather than should we be agile. Then, why is there such a gulf between the people who evangelize agile processes and those who look at them with great suspicion?

Any application is built to satisfy a business goal. The process has two major steps where the step one is to define the application behavior which will help in meeting the business goal and the step two is to translate the application behavior to a working application. The definition of the application behavior is the requirement specification and it is the link between business and IT.

Business - IT

In the waterfall methodology, development starts after the requirement is frozen. It is assumed that if business users can unambiguously and comprehensively specify the application behavior, IT can build the application satisfying the stated behavior. All the management practices based on waterfall methodology is build around this assumption. However, in most real life project we find that one of the following things have happen.

  • The stated behavior as understood by the business is different from the understanding of IT
  • The stated behavior is a sub-optimal solution for the given business goal
  • The business goal has changed

As a result the final application does not meet the business goal. Then it becomes a blame game between business and IT, where business claims that "this is not what I wanted" and IT claims that "I have delivered as per your signed-off requirement". Since there is a gap between desired application and what is stated in the requirement, it is concluded that the root cause of the failure is improper requirement definition. In the next project more rigorous process is applied to make the requirement definition more comprehensive and as early in the lifecycle as possible. In spite of all these rigor, one of the most frequently cited reason for project failure is improper requirement management.

In current business context, is it possible to freeze requirement?

If we go back in time, most IT applications were written to automate internal processes. The emphasis was on adopting industry best practices. Vast majority of the user were internal to the organization and they had to adopt to the application interface rather than the application interface being tailored for the user needs. Times have changed and many of the IT applications are an integral part of the product or service offering. Significant part of the users are external to the organization. The focus has shifted from following the best practices to having innovative offering. The new motto is usability and user centric design. Though there are guidelines and principle available on how to design such application, in practice it involves many iterations where many idea needs to be tried before a workable solution is arrived at. Sometimes, applications may have to be modified very late in the development cycle. In such cases, though the requirement is driven by a business goal, business users are not always sure about how the system should behave. It is necessary to try out the application before taking decision on its suitability. To this we can add the changing business goal arising out of change in the environment.

If all these points are added up, it becomes clear that requirements will change.

This has led to the realization that we need agile processes which works when requirements are constantly changing rather than spending time and energy on freezing requirement. As a result agile processes were born. The key principle behind any agile processes is to have a mechanism where the users can try out the application as it is being built and give feedback so that:

  1. Gap in understanding is reduced
  2. Sub-optimal solution can be improved
  3. Application can be realigned to any change in business goal

It is easier to adopt agile methodology in letter than in spirit. An iterative development process becomes ineffective, unless user feedback is used to change and fine tune the application behavior. This may involve introduction of new features, discarding features and reprioritization of features. It is not always easy to reorient an organization entrenched in waterfall development. Not only it is necessity to change mind-set of the people involved, it is also imperative to modify some the key management practices.

Funding: How Is the Project Sanctioned?
Normal process of budgeting involves calculating return on investment, which in turn, requires both investment and the return on it to be quantified. Even when the requirement is clearly defined, arriving at an accurate effort estimate is an error prone exercise. When requirement is expected to change during the life of the project, it becomes almost impossible to have an accurate estimate up front. This becomes a hazardous exercise and in practice the estimate gets tuned to make the ROI look good. Once sanctioned, the budget becomes frozen and it acts as a great disincentive to allow flexibility in changing requirement mid way. The way out of this situation is to have a method of incremental funding of project. Initially funds can be made available for first few iterations and this sanction can be based on a reasonable accurate estimation. The scope of the project and the fund required can get progressively fine tuned. With incremental funding, it also becomes easier to detect unviable project much earlier in the lifecycle.

Interaction: Do Stakeholders Trust Each Other?
Change in requirement involves rework and it implies discarding work already done. IT has to trust that a specific change suggested by business is needed and could not be specified earlier. Similarly, a feature may look simple from outside but may involve considerable amount of programming effort. Business has to trust the effort estimate made IT. Both IT and business has to believe that understanding is progressive and any gap in understanding is neither deliberate nor because of incompetence or carelessness of the other party. It can be achieved by having transparency from both side. IT needs to establish a method of estimation and make it transparent. Similarly, business needs to share the business plan and explain the need for any change. Any change will involve unplanned effort, more effort than what is feasible within a given time frame. Feature will have to be re-prioritized and and the revised plan should have joint ownership.

Defect: What Is a Defect?
Standard method of measuring defect is to:

  1. Define and freeze the requirement
  2. Derive test cases from the requirement
  3. Use the test cases to test the software
  4. Measure the number of test cases that has failed

Zero defect software means all the identified test cases are passed. This does not in any way indicate that the software meet the business needs. Since, test cases can only be identified for the explicitly documented requirement, we need to move beyond that and reorient how defect is defined. Therefore, regardless of presence of any explicit test case, a defect is when the application ...

  • ... crashes
  • ... behaves in an illogical manner
  • ... does not behave in the manner expected by the user

The responsibility of identifying and eliminating first and second category of defect is with developers. The third category can not be done by the developers alone and needs an active involvement of the user.

Success: How Is It Measured?
Traditionally, the goal of successful project management has been to the deliver the application on time, within budget and without defect. All project managers are brought up on the belief that the key to achieve this is to freeze requirement and minimize rework. However, when the planned application is a part of a major product or service release, the cost of failed application can be many time to cost of the application. Therefore it is necessary to shift the paradigm and measure success in terms of satisfying business needs. It can only be done by an agile methodology which interactively refines requirement by allowing the user to work with and feel the application on a regular basis. When we look at some of the most successful software company we find that it is better to deliver the right application even though it may be late, over budget or have some bugs than to deliver the wrong application within budget, on time and without defect. The likes of Microsoft, Apple and even Google has slipped on their promised delivery and the delivered software are also not free form problem. That has not stopped them from becoming the more successful than anybody else in what they choose to do.

Related Posts

More Stories By Udayan Banerjee

Udayan Banerjee is CTO at NIIT Technologies Ltd, an IT industry veteran with more than 30 years' experience. He blogs at http://setandbma.wordpress.com.
The blog focuses on emerging technologies like cloud computing, mobile computing, social media aka web 2.0 etc. It also contains stuff about agile methodology and trends in architecture. It is a world view seen through the lens of a software service provider based out of Bangalore and serving clients across the world. The focus is mostly on...

  • Keep the hype out and project a realistic picture
  • Uncover trends not very apparent
  • Draw conclusion from real life experience
  • Point out fallacy & discrepancy when I see them
  • Talk about trends which I find interesting
Google

@ThingsExpo Stories
The Internet of Things is a misnomer. That implies that everything is on the Internet, and that simply should not be - especially for things that are blurring the line between medical devices that stimulate like a pacemaker and quantified self-sensors like a pedometer or pulse tracker. The mesh of things that we manage must be segmented into zones of trust for sensing data, transmitting data, receiving command and control administrative changes, and peer-to-peer mesh messaging. In his session at @ThingsExpo, Ryan Bagnulo, Solution Architect / Software Engineer at SOA Software, focused on desi...
"At our booth we are showing how to provide trust in the Internet of Things. Trust is where everything starts to become secure and trustworthy. Now with the scaling of the Internet of Things it becomes an interesting question – I've heard numbers from 200 billion devices next year up to a trillion in the next 10 to 15 years," explained Johannes Lintzen, Vice President of Sales at Utimaco, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
"For over 25 years we have been working with a lot of enterprise customers and we have seen how companies create applications. And now that we have moved to cloud computing, mobile, social and the Internet of Things, we see that the market needs a new way of creating applications," stated Jesse Shiah, CEO, President and Co-Founder of AgilePoint Inc., in this SYS-CON.tv interview at 15th Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Gridstore™, the leader in hyper-converged infrastructure purpose-built to optimize Microsoft workloads, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Gridstore™ is the leader in hyper-converged infrastructure purpose-built for Microsoft workloads and designed to accelerate applications in virtualized environments. Gridstore’s hyper-converged infrastructure is the industry’s first all flash version of HyperConverged Appliances that include both compute and storag...
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Today’s enterprise is being driven by disruptive competitive and human capital requirements to provide enterprise application access through not only desktops, but also mobile devices. To retrofit existing programs across all these devices using traditional programming methods is very costly and time consuming – often prohibitively so. In his session at @ThingsExpo, Jesse Shiah, CEO, President, and Co-Founder of AgilePoint Inc., discussed how you can create applications that run on all mobile devices as well as laptops and desktops using a visual drag-and-drop application – and eForms-buildi...
We certainly live in interesting technological times. And no more interesting than the current competing IoT standards for connectivity. Various standards bodies, approaches, and ecosystems are vying for mindshare and positioning for a competitive edge. It is clear that when the dust settles, we will have new protocols, evolved protocols, that will change the way we interact with devices and infrastructure. We will also have evolved web protocols, like HTTP/2, that will be changing the very core of our infrastructures. At the same time, we have old approaches made new again like micro-services...
Code Halos - aka "digital fingerprints" - are the key organizing principle to understand a) how dumb things become smart and b) how to monetize this dynamic. In his session at @ThingsExpo, Robert Brown, AVP, Center for the Future of Work at Cognizant Technology Solutions, outlined research, analysis and recommendations from his recently published book on this phenomena on the way leading edge organizations like GE and Disney are unlocking the Internet of Things opportunity and what steps your organization should be taking to position itself for the next platform of digital competition.
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
As the Internet of Things unfolds, mobile and wearable devices are blurring the line between physical and digital, integrating ever more closely with our interests, our routines, our daily lives. Contextual computing and smart, sensor-equipped spaces bring the potential to walk through a world that recognizes us and responds accordingly. We become continuous transmitters and receivers of data. In his session at @ThingsExpo, Andrew Bolwell, Director of Innovation for HP's Printing and Personal Systems Group, discussed how key attributes of mobile technology – touch input, sensors, social, and ...
In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect at GE, and Ibrahim Gokcen, who leads GE's advanced IoT analytics, focused on the Internet of Things / Industrial Internet and how to make it operational for business end-users. Learn about the challenges posed by machine and sensor data and how to marry it with enterprise data. They also discussed the tips and tricks to provide the Industrial Internet as an end-user consumable service using Big Data Analytics and Industrial Cloud.
Building low-cost wearable devices can enhance the quality of our lives. In his session at Internet of @ThingsExpo, Sai Yamanoor, Embedded Software Engineer at Altschool, provided an example of putting together a small keychain within a $50 budget that educates the user about the air quality in their surroundings. He also provided examples such as building a wearable device that provides transit or recreational information. He then reviewed the resources available to build wearable devices at home including open source hardware, the raw materials required and the options available to power s...
Things are being built upon cloud foundations to transform organizations. This CEO Power Panel at 15th Cloud Expo, moderated by Roger Strukhoff, Cloud Expo and @ThingsExpo conference chair, addressed the big issues involving these technologies and, more important, the results they will achieve. Rodney Rogers, chairman and CEO of Virtustream; Brendan O'Brien, co-founder of Aria Systems, Bart Copeland, president and CEO of ActiveState Software; Jim Cowie, chief scientist at Dyn; Dave Wagstaff, VP and chief architect at BSQUARE Corporation; Seth Proctor, CTO of NuoDB, Inc.; and Andris Gailitis, C...
There's Big Data, then there's really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at Big Data Expo®, Hannah Smalltree, Director at Treasure Data, discussed how IoT, Big Data and deployments are processing massive data volumes from wearables, utilities and other machines...
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
In this Women in Technology Power Panel at 15th Cloud Expo, moderated by Anne Plese, Senior Consultant, Cloud Product Marketing at Verizon Enterprise, Esmeralda Swartz, CMO at MetraTech; Evelyn de Souza, Data Privacy and Compliance Strategy Leader at Cisco Systems; Seema Jethani, Director of Product Management at Basho Technologies; Victoria Livschitz, CEO of Qubell Inc.; Anne Hungate, Senior Director of Software Quality at DIRECTV, discussed what path they took to find their spot within the technology industry and how do they see opportunities for other women in their area of expertise.
While great strides have been made relative to the video aspects of remote collaboration, audio technology has basically stagnated. Typically all audio is mixed to a single monaural stream and emanates from a single point, such as a speakerphone or a speaker associated with a video monitor. This leads to confusion and lack of understanding among participants especially regarding who is actually speaking. Spatial teleconferencing introduces the concept of acoustic spatial separation between conference participants in three dimensional space. This has been shown to significantly improve comprehe...
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, data security and privacy.
Performance is the intersection of power, agility, control, and choice. If you value performance, and more specifically consistent performance, you need to look beyond simple virtualized compute. Many factors need to be considered to create a truly performant environment. In his General Session at 15th Cloud Expo, Harold Hannon, Sr. Software Architect at SoftLayer, discussed how to take advantage of a multitude of compute options and platform features to make cloud the cornerstone of your online presence.