|By Jason Bloomberg||
|August 7, 2011 09:00 AM EDT||
At ZapThink we’re proud to be iconoclasts. After all, building agile architectures requires critical appraisal—and frequent dismissal—of traditionally held beliefs. We know this role places us among the heretics who dare challenge established dogma.
In fact, the whole notion of agility has long suffered this battle between iconoclasm and dogmatism. As we discuss in our Licensed ZapThink Architect course, the Agile Manifesto embodies an iconoclastic approach to software development dogma – and yet, so many people have become dogmatic about the Agile Manifesto itself, entirely missing its point!
ZapThink once more jumped into this iconoclasm-masquerading-as-dogma fray with our recent ZapFlash, How I Became a REST “Convert.” We explained how you can implement SOA following REST principles, and how such an approach introduces important simplifications and efficiencies as compared to a Web Services-based approach.
Get out the torches and pitchforks! ZapThink is at it again! Not only are we flying in the face of established SOA dogma, we’re taking on REST dogma as well! Never mind that the original vision for the REST movement was inherently iconoclastic. In fact, it’s no wonder that RESTafarians use religious metaphors in their discussions (although Rastafarianism is a far cry indeed from the Catholic metaphors we’re leveraging in this ZapFlash).
To quote our beloved Agile Manifesto, we want to favor responding to change over following a plan—even if that plan is how SOA or even REST is “supposed to be done.” The goal is stuff that actually addresses the business problem, not some adherence to official dogma (another Agile principle, in case you didn’t notice). Therefore, it came as no surprise to us when an organization contacted us and let us know that they are taking the REST-based SOA approach we discussed in the ZapFlash, and that it actually works.
The Iconoclastic REST-based SOA of the US Coast Guard
We recently spoke with the US Coast Guard (USCG) about their SPEAR (Semper Paratus: Enterprise Architecture Realization) initiative. The SPEAR approach to SOA centers on document-centric, event-driven, loosely coupled, asynchronous, message-based Business Services. Now, there’s nothing particularly iconoclastic about event-driven SOA—after all, we debunked that notion back in 2004—but their story doesn’t end there. Another central characteristic of their SOA approach is their document-centricity. While it’s true that document style interfaces are the norm for Web Services, the USCG takes the notion to a new level.
In the Web Services world, when a WSDL file specifies a document style interface, then the constraints on the input and output messages fall into one or more schema definitions, instead of the SOAP message structure itself. As far as SOAP is concerned, the body of the SOAP message contains a payload consisting of whatever document you like. The advantage of the document style over the tightly coupled remote procedure call (RPC) style is it allows for flexible versioning of Services: many changes to a Service do not force a new contract version.
However, as anyone who has monkeyed with Web Services can attest, the operations of a Service still cause issues, even when it’s a document style interface. We can’t seem to get away from Web Services’ tightly coupled RPC heritage. And furthermore, XML schemas are strongly typed, which introduces a further undesirable level of tight coupling.
To resolve such issues, REST moves the operations out of any formal contract, instead relying upon the GET, POST, PUT and DELETE operations best known from HTTP. Even so, there’s no requirement in REST that resources are necessarily documents. True, URLs that point to documents are a common and familiar pattern, but URLs could as easily point to abstracted method calls as well.
SPEAR takes document-centricity to the next level. For SPEAR, the document is the interface. It has meaning both for human and machine consumption. It’s self-describing and removes the need for defining a specific, formal contract. Instead, the USCG provides a basic header/body document structure. The header contains elements like requester, type, ID, timestamps, and status (request, response, publication or error, for example). The body contains different parts depending upon the type of Service. Request/response Services, for example, include elements like request, response, publication, and exception. As a result, SPEAR’s Service contracts (if you even decide to call them that) consist of a simple document structure and core REST operations—nothing more.
Maintaining State the Service-Oriented Way
For example, take a straightforward request/response Service that executes a simple database query. The Service simply populates the response field in the request document and returns the entire document to the requester. As a result, the document still contains the timestamped request. The ESB can now publish the document or put it on a queue, and the document itself acts as its own cache.
This approach to state is a simple example of a new trend in REST-based state management: Hypermedia as the Engine of Application State (HATEOAS). With HATEOAS, documents and the hyperlinks they contain represent all the state information a distributed environment requires. Want to know the next step in a process? Simply follow the appropriate link.
In the case of SPEAR, the USCG has established an internal standard URI representation: domain://provider:context/resource. For example, service://uscg.mda.vesselCrew:test/sla. The domain represents a taxonomy of resource types, including system://, organization://, geography://, and service://. The provider component represents the address to the provider of the resource. The context (squeezed in where ports go in a URL), represents the business context like :test, :dev, :stage, etc.
The ESB then resolves URIs to the physical endpoint references, acting as a routing engine that delivers the Business Service abstraction. Any hyperlink to such a URI, therefore, points to a document that contains all the state information the system requires, and furthermore, the architecture offers late binding to Services as a key feature. There is no build time mapping to any particular resource. Instead of using strongly typed schemas, they rely upon dynamic, loose typing in the documents. Such an approach is decidedly not Web Services-friendly.
It’s also important to note that state maintenance in the Web Services world has always been problematic. There are basically three ways to main state information in interactions among inherently stateless Services: rely upon the Service consumer to maintain a correlation ID (either not broadly adopted or relies upon an underlying protocol like HTTP cookies); rely upon the underlying execution environment (vendor dependent); or place state information into the message. Unfortunately, Web Services offer no standard way of accomplishing the latter task, requiring SOA teams to customize their SOAP headers—which is usually a deal killer.
The SPEAR approach, however, includes state information in the message, because, of course, the message is the document. As a result, they are taking a fully Service-oriented approach to maintaining state. They are able to do so in spite of leveraging an ESB because they selected the Fiorano ESB, which has long maintained state in a fully message-centric manner, instead of spawning threads to keep track of state as in other vendors’ ESBs.
Asynchronicity as the General Case
For many years, Gartner believed that SOA interactions were inherently synchronous, request/response—casting SOA as little more than next-gen client/server—until TIBCO twisted their arm into admitting that yes, SOA could be event driven. Nevertheless, SOA has always allowed for asynchronous as well as synchronous Service exchange patterns. (The whole Gartner event-driven SOA kerfuffle led to the SOA 2.0 fiasco, lest we forget.) In fact, ZapThink pointed out the importance of asynchronous behavior as early as 2002.
In the SPEAR architecture, the listener acts as the handler, retaining messages at their last stop. The senders simply fire and forget. Messages are sent to the Fiorano bus, not the destination, allowing for dynamic routing of messages as well as publish/subscribe. In this way the ESB routes, filters, and enforces policy via the URIs themselves.
Publish/subscribe is a broadcast push exchange pattern: many consumers can subscribe to the same topic, and the ESB pushes messages to subscribers as appropriate. Even when a single consumer queries a resource, the ESB follows a push pattern, putting documents onto a queue for the consumer to pick up. As a result, SPEAR can support different consumer behaviors, depending upon the use case. If the consumer is accessing Services via unreliable network connections, or simply requires low message volumes and can tolerate some latency, then the consumer can simply poll the Service. In other cases, when latency is an issue or when there are higher message volumes, the ESB can perform a true push to the consumer via a JMS interface.
OK, hold on just one minute. JMS? I thought we were talking about REST, you know, over HTTP. How can JMS fit into this picture?
Yes, the USCG is being iconoclastic once again. The ESB is abstracting the endpoint, while at the same time providing a push-based messaging infrastructure. If the consumer wants the resource to push a real-time stream of large messages to it, then the URI should resolve to a JMS endpoint. If not, or if the consumer doesn’t support JMS (typically when it’s running over HTTP) then the resource will resolve to an HTTP endpoint.
The ZapThink Take
Is the USCG implementing “pure” SOA or “pure” REST? I’m not even going to dignify those questions with a response, because the whole notion of a “pure” architectural approach is inherently dogmatic. What the USCG has done is implement an architecture that actually works for them—that is, it delivers loosely coupled, abstracted Business Services with the flexibility and performance they require. And remember, actually working trumps following a pre-determined set of best practices every time—especially when those best practices don’t actually solve the problems at hand.
There are two important morals here. First, REST-based SOA is alive and well, and offers straightforward solutions to many of the knottier problems that Web Services-based SOA has suffered from. But even more importantly, the success the USCG has achieved shows that any architectural approach is nothing more than a loose collection of best practices. It’s up to the architect to know which best practices are best for solving their particular problems. Avoid a dogmatic approach, and select the right tool for the job, even if that brands you as an iconoclast.
You may have heard about the pets vs. cattle discussion – a reference to the way application servers are deployed in the cloud native world. If an application server goes down it can simply be dropped from the mix and a new server added in its place. The practice so far has mostly been applied to application deployments. Management software on the other hand is treated in a very special manner. Dedicated resources are set aside to run the management software components and several alerting syst...
Nov. 29, 2015 12:00 AM EST Reads: 193
It's been a busy time for tech's ongoing infatuation with containers. Amazon just announced EC2 Container Registry to simply container management. The new Azure container service taps into Microsoft's partnership with Docker and Mesosphere. You know when there's a standard for containers on the table there's money on the table, too. Everyone is talking containers because they reduce a ton of development-related challenges and make it much easier to move across production and testing environm...
Nov. 28, 2015 08:00 PM EST Reads: 615
The Internet of Things (IoT) is growing rapidly by extending current technologies, products and networks. By 2020, Cisco estimates there will be 50 billion connected devices. Gartner has forecast revenues of over $300 billion, just to IoT suppliers. Now is the time to figure out how you’ll make money – not just create innovative products. With hundreds of new products and companies jumping into the IoT fray every month, there’s no shortage of innovation. Despite this, McKinsey/VisionMobile data...
Nov. 28, 2015 01:00 PM EST Reads: 478
Hiring the wrong candidate can cost a company hundreds of thousands of dollars, and result in lost profit and productivity during the search for a replacement. In fact, the Harvard Business Review has found that as much as 80 percent of turnover is caused by bad hiring decisions. But when your organization has implemented DevOps, the job is about more than just technical chops. It’s also about core behaviors: how they work with others, how they make decisions, and how those decisions translate t...
Nov. 28, 2015 12:45 PM EST Reads: 186
Just over a week ago I received a long and loud sustained applause for a presentation I delivered at this year’s Cloud Expo in Santa Clara. I was extremely pleased with the turnout and had some very good conversations with many of the attendees. Over the next few days I had many more meaningful conversations and was not only happy with the results but also learned a few new things. Here is everything I learned in those three days distilled into three short points.
Nov. 28, 2015 12:00 PM EST Reads: 337
In his General Session at DevOps Summit, Asaf Yigal, Co-Founder & VP of Product at Logz.io, explored the value of Kibana 4 for log analysis and provided a hands-on tutorial on how to set up Kibana 4 and get the most out of Apache log files. He examined three use cases: IT operations, business intelligence, and security and compliance. Asaf Yigal is co-founder and VP of Product at log analytics software company Logz.io. In the past, he was co-founder of social-trading platform Currensee, which...
Nov. 28, 2015 12:00 PM EST Reads: 233
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York and Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty ...
Nov. 28, 2015 12:00 PM EST Reads: 553
One of the most important tenets of digital transformation is that it’s customer-driven. In fact, the only reason technology is involved at all is because today’s customers demand technology-based interactions with the companies they do business with. It’s no surprise, therefore, that we at Intellyx agree with Patrick Maes, CTO, ANZ Bank, when he said, “the fundamental element in digital transformation is extreme customer centricity.” So true – but note the insightful twist that Maes adde...
Nov. 28, 2015 12:00 PM EST Reads: 446
DevOps is about increasing efficiency, but nothing is more inefficient than building the same application twice. However, this is a routine occurrence with enterprise applications that need both a rich desktop web interface and strong mobile support. With recent technological advances from Isomorphic Software and others, rich desktop and tuned mobile experiences can now be created with a single codebase – without compromising functionality, performance or usability. In his session at DevOps Su...
Nov. 28, 2015 11:45 AM EST Reads: 406
Nov. 28, 2015 11:30 AM EST Reads: 224
People want to get going with DevOps or Continuous Delivery, but need a place to start. Others are already on their way, but need some validation of their choices. A few months ago, I published the first volume of DevOps and Continuous Delivery reference architectures which has now been viewed over 50,000 times on SlideShare (it's free to download...no registration required). Three things helped people in the deck: (1) the reference architectures, (2) links to the sources for each architectur...
Nov. 28, 2015 11:30 AM EST Reads: 243
As organizations realize the scope of the Internet of Things, gaining key insights from Big Data, through the use of advanced analytics, becomes crucial. However, IoT also creates the need for petabyte scale storage of data from millions of devices. A new type of Storage is required which seamlessly integrates robust data analytics with massive scale. These storage systems will act as “smart systems” provide in-place analytics that speed discovery and enable businesses to quickly derive meaningf...
Nov. 28, 2015 11:15 AM EST Reads: 416
Culture is the most important ingredient of DevOps. The challenge for most organizations is defining and communicating a vision of beneficial DevOps culture for their organizations, and then facilitating the changes needed to achieve that. Often this comes down to an ability to provide true leadership. As a CIO, are your direct reports IT managers or are they IT leaders? The hard truth is that many IT managers have risen through the ranks based on their technical skills, not their leadership ab...
Nov. 28, 2015 10:00 AM EST Reads: 400
Continuous processes around the development and deployment of applications are both impacted by -- and a benefit to -- the Internet of Things trend. To help better understand the relationship between DevOps and a plethora of new end-devices and data please welcome Gary Gruver, consultant, author and a former IT executive who has led many large-scale IT transformation projects, and John Jeremiah, Technology Evangelist at Hewlett Packard Enterprise (HPE), on Twitter at @j_jeremiah. The discussion...
Nov. 28, 2015 05:30 AM EST Reads: 735
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
Nov. 28, 2015 04:00 AM EST Reads: 543
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound...
Nov. 28, 2015 03:30 AM EST Reads: 476
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds. How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem"...
Nov. 28, 2015 03:00 AM EST Reads: 453
PubNub has announced the release of BLOCKS, a set of customizable microservices that give developers a simple way to add code and deploy features for realtime apps.PubNub BLOCKS executes business logic directly on the data streaming through PubNub’s network without splitting it off to an intermediary server controlled by the customer. This revolutionary approach streamlines app development, reduces endpoint-to-endpoint latency, and allows apps to better leverage the enormous scalability of PubNu...
Nov. 28, 2015 02:00 AM EST Reads: 331
Growth hacking is common for startups to make unheard-of progress in building their business. Career Hacks can help Geek Girls and those who support them (yes, that's you too, Dad!) to excel in this typically male-dominated world. Get ready to learn the facts: Is there a bias against women in the tech / developer communities? Why are women 50% of the workforce, but hold only 24% of the STEM or IT positions? Some beginnings of what to do about it! In her Day 2 Keynote at 17th Cloud Expo, San...
Nov. 28, 2015 02:00 AM EST Reads: 583
In today's enterprise, digital transformation represents organizational change even more so than technology change, as customer preferences and behavior drive end-to-end transformation across lines of business as well as IT. To capitalize on the ubiquitous disruption driving this transformation, companies must be able to innovate at an increasingly rapid pace. Traditional approaches for driving innovation are now woefully inadequate for keeping up with the breadth of disruption and change facin...
Nov. 28, 2015 02:00 AM EST Reads: 492