Microservices Expo Authors: Liz McMillan, Pat Romanski, Elizabeth White, Stackify Blog, Andreas Grabner

Related Topics: Microservices Expo

Microservices Expo: Article

BizTalk Server 2004 in an Investment Bank

The agile service-oriented architecture becomes a reality

This article describes recent work in a leading investment bank using Microsoft's BizTalk Server 2004 (BizTalk) as an integral component of a service-oriented architecture. I'll describe how BizTalk is used to implement lightweight workflow that builds new services from existing services and ties in tactical solutions to enable straight-through processing (STP) of service requests.

I'll look at the architectural vision for a service-oriented architecture and how BizTalk fits into this architecture as one possible implementation technology. I will also outline the progress that has been made to date in creating the base services that will enable this vision to flourish within the bank.

But first, some background on the unique IT requirements of investment banking.

Investment banking has unique information technology requirements. The profits to be made from a competitive advantage often dwarf the IT costs in gaining that advantage. For example, a new derivative product that fits with the mood of the market can generate huge turn over - if it's brought to the market in a timely fashion.

The natural consequence is that the business drives the IT more directly than in most other industries. Consequently, investment banks are often organized with IT functions aligned with business areas, i.e., an IT area supporting foreign exchange (FX) and an IT function supporting fixed income (FI), etc.

The advantage of this organizational structure is obvious - the IT functions respond to the business requirements in a timely and focused fashion. But the disadvantage is equally obvious - the endless development of tactical solutions, the continual patching of out-of-date systems, and the growth of inter-system complexity.

Of course, the banks recognize the drawbacks of their organizations and mitigate these problems by forming cross-functional IT areas with architecture boards at all levels. There is always a tension between the architectural desire for well-structured, loosely coupled strategic solutions and the business demand for functionality now. The architecture that offers most hope for squaring this particular circle is the service-oriented architecture (SOA).

An international investment bank has to respond to different regulatory and best-practice requirements in each country in which it operates. Moreover, an international bank is often organized as a set of separate business entities divided on a regional or country basis. One of the challenges in implementing an SOA is enabling agility in the tailoring/enhancement of services to cope with the differing international requirements and local processes.

This article discusses one of the architecture initiatives being driven by a leading investment bank; namely the "agile" service-oriented architecture. That is, an SOA that has rapid application development and enterprise application integration facilities, and facilitates straight-through processing with a capability to orchestrate disparate sub-services.

An Agile Service-Oriented Architecture
The traditional SOA (traditions are quickly established in Internet time!) replaces conventional layered applications with processes that make use of services that span business areas. The SOA makes services available on a network using widely accepted standards (i.e., XML, WSDL, UDDI, etc.) and focuses on defining contracts for consuming and producing business documents. One common misconception is that SOA is synonymous with Web services, whereas Web services are one, albeit attractive, option for implementing an SOA.

Many industries and investment banking in particular, have a need for a variation on the traditional SOA - the agile SOA. This can be thought of as a set of tools and implementation techniques for building an SOA that can quickly change to meet shifting business requirements and can handle lightweight workflow requirements in a heterogeneous enterprise environment.

Consider the simple example of reference data within a bank. There are myriad sources of reference data that need to be cross referenced. For example, financial instruments can be identified in a variety of ways (e.g,. exchange ticker, ISIN code, Reuters RIC code). Many trading, quotation, or back-office reconciliation applications may well refer to the same financial instrument using different identifiers. These facts give a clear requirement for a reference data service that can map one identifier to another. Such a service would fit well into the scope of a traditional SOA.

Now consider a more complex scenario where a number of business areas want to register new counter party (i.e., trading partner) details with the bank. Again, an obvious candidate for a service in the SOA but this time there are more complex factors to remember:

  • Checks of identity, address, etc. with reputable agencies
  • Credit checks
  • Fulfilling regional money laundering requirements
  • Data enrichment with credit ratings
The seemingly simple "Add Counter Party" service has become a complex, long-lived process that needs to correlate responses from sub-services (whether internal or external) and, in addition, may well need to interact with legacy systems.

In the complex and rapidly changing world of investment banking, the requirement to be able to fulfill the straight-through processing (STP) demands of services, such as the "Add Counter Party" service, are commonplace. These processes are sometimes referred to as edge processes as they appear in myriad forms around the core of the bank's business.

An implementation technology that can enable the creation of agile SOAs must not only have rapid application development (RAD) aspects, and orchestration technology for long-lived correlated processes, but must also enable enterprise application integration (EAI) to build services from existing infrastructure.

A Framework Implementation
To realize the promise of the agile SOA, the bank has been working on a framework in which to implement this architecture. The objective has been to create a secure, integrated, heterogeneous, and extensible framework that supports the aim of building an agile, service-oriented architecture.

The framework can be thought of as consisting of three components:

  • Infrastructure: Enterprise-level diagnostics, security components, directory services, etc.
  • Orchestration technology: To enable long-running business processes that interact with legacy data sources and applications. This facilitates the incremental delivery of the SOA rather than an all-or-nothing approach - something that would be unacceptable to the lines of business.
  • Base services: E.g., a noteworthy business event logging service
Microsoft's BizTalk Server 2004 is the choice for the orchestration technology for the .NET platform. BizTalk provides excellent facilities for the orchestration of processes that integrate legacy enterprise application and data sources. Moreover, its message-based subscription mechanism enables sets of loosely coupled services to interact in an efficient and scalable way. Furthermore, BizTalk's RAD tools (e.g., the orchestration designer and message mapping tool) allow new services to be added to the SOA in a timely fashion.

The infrastructure of the framework has been designed to provide a stable base for the development of services. Important components of the infrastructure are:

  • Security: The approach has been to leverage the emerging WS standards and to work closely with vendors to make sure that cross-platform Web service technology is a reality. In particular, the bank has developed standards and components that work with WS-Security and WS-Policy to enable end-to-end security of SOAP envelopes using a variety of authentication and encryption techniques: X509, Kerberos, etc. For the .NET platform, we use Microsoft's Web Services Enhancements to implement the Web service security stack and significant effort has been put into integrating this with BizTalk 2004.
  • Diagnostics: On the .NET platform, services make use of Microsoft's Enterprise Instrumentation application block. This provides a set of facilities for tracing requests through distributed systems and raising exceptions in a number of formats including Windows Management Instrumentation (WMI) events that can be monitored by Tivoli or similar systems.
  • UDDI: The bank is planning a global directory of the services that make up the SOA.
For the rest of this article I will briefly discuss two base services implemented in the framework: the business event logging service and the remote calculation service.

Tamper-Proof Logging
An investment bank works with a large number of counter parties and semi-autonomous business units. One of the attractions of a secure SOA is that it enables the business to interact with its complex web of partners in a more stream-lined fashion. An example of a service that promotes this sort of business activity is a tamper-proof logging service.

The framework has an implementation of an event logging service that uses hashing and encryption algorithms to make the log tamper-proof. The service exposes a Web service interface and is hosted by BizTalk orchestration that not only logs the business event but also publishes the event as a BizTalk message. This enables subscribing orchestrations to be triggered by business events of interest to them (see Figure 1).

As an example, consider the scenario where a bank client is using a bank-supplied application to obtain financial information. If the tamper-proof logging service exists, then both the sending and the receipt of the information can be recorded in the log. Each record will contain an encrypted hash of the message payload and the tamper-proof nature of the log guarantees that disputes can be settled with unambiguous information. Moreover, it would be possible to create an auditing service (written as a BizTalk orchestration) that subscribed to either or both of the send and receive events. In this way, BizTalk's efficient messaging subscription design allows services to be loosely coupled with each other.

It is easy to see that when a service such as tamper-proof logging exists within the bank, it can quickly become a component of many higher order services. One such service is the remote calculation service.

Remote Calculation Service
Spreadsheets are used extensively in an investment bank. Traders and back-office staff are very familiar with this sort of technology and build complicated models of trades, risk calculations, and so on. Spreadsheets have plug-ins for market data feeds from providers such as Reuters or Telerate and often make use of add-in analytic libraries that implement algorithms for calculating risk, curves, and prices.

Some of the algorithms run by traders, back office reconciliation, or business controlling staff take a surprisingly long time to run. Value-at-risk calculations tend to fall into this category and it is not unusual for a risk calculation to take a couple of hours to complete.

To see why spreadsheets are used so widely for these sort of calculations, consider the trader shown in Figure 2. Here, the trader has their risk model, which they have honed over the years. The spreadsheet model is fed with live data from the market data feeds and the trader uses their expert knowledge to input candidate prices, rates, or spreads into the model. At the end of the trading day the trader will run their value-at-risk calculation to inform their trading decisions for the next day.

Now consider the position of the business controller who is responsible for assessing the value at risk for a whole trading floor. They are in the invidious position of taking each of the trader's spreadsheets in turn, validating that the market data has been approved, and running each calculation. This sort of situation was one reason for developing the remote calculation service (see Figures 3 and 4).

The remote calculation service is one of the base services of the framework and can run a computation remotely on behalf of its client - one computation that is supported is spreadsheet calculations (including the driving of macros and add-ins, etc). The service allows its client to schedule a computation immediately or for some time in the future on a recurring basis. The computation can also be scheduled to run on receipt of a business event logged via the logging service described earlier.

The remote calculation service is implemented as a BizTalk orchestration and has support to return the calculation results using a custom output orchestration. This feature, again, relies on the message subscription design of BizTalk. In this way, the results of calculations might be transformed and routed back to requesting users using their preferred transport mechanism (i.e., SMTP, FTP, etc.)

Figure 4 illustrates, in a simple way, how the SOA can be built using the lower level services and how the BizTalk's flexibility can be utilized to add application, regional, or even user-specific processing into the architecture.

Investment banks have complicated businesses that spread a round the world with many different trading partners. They operate in an environment of constant change and need to react very quickly to keep their competitive edge. This article has shown how one investment bank has reacted to these challenges and how BizTalk Server can help make the agile service-oriented architecture a reality.

More Stories By David Regan

David Regan is a freelance systems programmer specializing in data- and rule-driven multitier systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

Microservices Articles
Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and co...
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, discussed why containers should be paired with new architectural practices such as microservices rathe...
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, will discuss how to use Kubernetes to setup a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace....
SYS-CON Events announced today the Kubernetes and Google Container Engine Workshop, being held November 3, 2016, in conjunction with @DevOpsSummit at 19th Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA. This workshop led by Sebastian Scheele introduces participants to Kubernetes and Google Container Engine (GKE). Through a combination of instructor-led presentations, demonstrations, and hands-on labs, students learn the key concepts and practices for deploying and maintainin...
Docker is sweeping across startups and enterprises alike, changing the way we build and ship applications. It's the most prominent and widely known software container platform, and it's particularly useful for eliminating common challenges when collaborating on code (like the "it works on my machine" phenomenon that most devs know all too well). With Docker, you can run and manage apps side-by-side - in isolated containers - resulting in better compute density. It's something that many developer...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
As software becomes more and more complex, we, as software developers, have been splitting up our code into smaller and smaller components. This is also true for the environment in which we run our code: going from bare metal, to VMs to the modern-day Cloud Native world of containers, schedulers and micro services. While we have figured out how to run containerized applications in the cloud using schedulers, we've yet to come up with a good solution to bridge the gap between getting your contain...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
DevOps is speeding towards the IT world like a freight train and the hype around it is deafening. There is no reason to be afraid of this change as it is the natural reaction to the agile movement that revolutionized development just a few years ago. By definition, DevOps is the natural alignment of IT performance to business profitability. The relevance of this has yet to be quantified but it has been suggested that the route to the CEO’s chair will come from the IT leaders that successfully ma...
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, will discuss why containers should be paired with new architectural practices such as microservices ra...