Microservices Expo Authors: Liz McMillan, Zakia Bouachraoui, Elizabeth White, Pat Romanski, Yeshim Deniz

Related Topics: Microservices Expo

Microservices Expo: Article

BizTalk Server 2004 in an Investment Bank

The agile service-oriented architecture becomes a reality

This article describes recent work in a leading investment bank using Microsoft's BizTalk Server 2004 (BizTalk) as an integral component of a service-oriented architecture. I'll describe how BizTalk is used to implement lightweight workflow that builds new services from existing services and ties in tactical solutions to enable straight-through processing (STP) of service requests.

I'll look at the architectural vision for a service-oriented architecture and how BizTalk fits into this architecture as one possible implementation technology. I will also outline the progress that has been made to date in creating the base services that will enable this vision to flourish within the bank.

But first, some background on the unique IT requirements of investment banking.

Investment banking has unique information technology requirements. The profits to be made from a competitive advantage often dwarf the IT costs in gaining that advantage. For example, a new derivative product that fits with the mood of the market can generate huge turn over - if it's brought to the market in a timely fashion.

The natural consequence is that the business drives the IT more directly than in most other industries. Consequently, investment banks are often organized with IT functions aligned with business areas, i.e., an IT area supporting foreign exchange (FX) and an IT function supporting fixed income (FI), etc.

The advantage of this organizational structure is obvious - the IT functions respond to the business requirements in a timely and focused fashion. But the disadvantage is equally obvious - the endless development of tactical solutions, the continual patching of out-of-date systems, and the growth of inter-system complexity.

Of course, the banks recognize the drawbacks of their organizations and mitigate these problems by forming cross-functional IT areas with architecture boards at all levels. There is always a tension between the architectural desire for well-structured, loosely coupled strategic solutions and the business demand for functionality now. The architecture that offers most hope for squaring this particular circle is the service-oriented architecture (SOA).

An international investment bank has to respond to different regulatory and best-practice requirements in each country in which it operates. Moreover, an international bank is often organized as a set of separate business entities divided on a regional or country basis. One of the challenges in implementing an SOA is enabling agility in the tailoring/enhancement of services to cope with the differing international requirements and local processes.

This article discusses one of the architecture initiatives being driven by a leading investment bank; namely the "agile" service-oriented architecture. That is, an SOA that has rapid application development and enterprise application integration facilities, and facilitates straight-through processing with a capability to orchestrate disparate sub-services.

An Agile Service-Oriented Architecture
The traditional SOA (traditions are quickly established in Internet time!) replaces conventional layered applications with processes that make use of services that span business areas. The SOA makes services available on a network using widely accepted standards (i.e., XML, WSDL, UDDI, etc.) and focuses on defining contracts for consuming and producing business documents. One common misconception is that SOA is synonymous with Web services, whereas Web services are one, albeit attractive, option for implementing an SOA.

Many industries and investment banking in particular, have a need for a variation on the traditional SOA - the agile SOA. This can be thought of as a set of tools and implementation techniques for building an SOA that can quickly change to meet shifting business requirements and can handle lightweight workflow requirements in a heterogeneous enterprise environment.

Consider the simple example of reference data within a bank. There are myriad sources of reference data that need to be cross referenced. For example, financial instruments can be identified in a variety of ways (e.g,. exchange ticker, ISIN code, Reuters RIC code). Many trading, quotation, or back-office reconciliation applications may well refer to the same financial instrument using different identifiers. These facts give a clear requirement for a reference data service that can map one identifier to another. Such a service would fit well into the scope of a traditional SOA.

Now consider a more complex scenario where a number of business areas want to register new counter party (i.e., trading partner) details with the bank. Again, an obvious candidate for a service in the SOA but this time there are more complex factors to remember:

  • Checks of identity, address, etc. with reputable agencies
  • Credit checks
  • Fulfilling regional money laundering requirements
  • Data enrichment with credit ratings
The seemingly simple "Add Counter Party" service has become a complex, long-lived process that needs to correlate responses from sub-services (whether internal or external) and, in addition, may well need to interact with legacy systems.

In the complex and rapidly changing world of investment banking, the requirement to be able to fulfill the straight-through processing (STP) demands of services, such as the "Add Counter Party" service, are commonplace. These processes are sometimes referred to as edge processes as they appear in myriad forms around the core of the bank's business.

An implementation technology that can enable the creation of agile SOAs must not only have rapid application development (RAD) aspects, and orchestration technology for long-lived correlated processes, but must also enable enterprise application integration (EAI) to build services from existing infrastructure.

A Framework Implementation
To realize the promise of the agile SOA, the bank has been working on a framework in which to implement this architecture. The objective has been to create a secure, integrated, heterogeneous, and extensible framework that supports the aim of building an agile, service-oriented architecture.

The framework can be thought of as consisting of three components:

  • Infrastructure: Enterprise-level diagnostics, security components, directory services, etc.
  • Orchestration technology: To enable long-running business processes that interact with legacy data sources and applications. This facilitates the incremental delivery of the SOA rather than an all-or-nothing approach - something that would be unacceptable to the lines of business.
  • Base services: E.g., a noteworthy business event logging service
Microsoft's BizTalk Server 2004 is the choice for the orchestration technology for the .NET platform. BizTalk provides excellent facilities for the orchestration of processes that integrate legacy enterprise application and data sources. Moreover, its message-based subscription mechanism enables sets of loosely coupled services to interact in an efficient and scalable way. Furthermore, BizTalk's RAD tools (e.g., the orchestration designer and message mapping tool) allow new services to be added to the SOA in a timely fashion.

The infrastructure of the framework has been designed to provide a stable base for the development of services. Important components of the infrastructure are:

  • Security: The approach has been to leverage the emerging WS standards and to work closely with vendors to make sure that cross-platform Web service technology is a reality. In particular, the bank has developed standards and components that work with WS-Security and WS-Policy to enable end-to-end security of SOAP envelopes using a variety of authentication and encryption techniques: X509, Kerberos, etc. For the .NET platform, we use Microsoft's Web Services Enhancements to implement the Web service security stack and significant effort has been put into integrating this with BizTalk 2004.
  • Diagnostics: On the .NET platform, services make use of Microsoft's Enterprise Instrumentation application block. This provides a set of facilities for tracing requests through distributed systems and raising exceptions in a number of formats including Windows Management Instrumentation (WMI) events that can be monitored by Tivoli or similar systems.
  • UDDI: The bank is planning a global directory of the services that make up the SOA.
For the rest of this article I will briefly discuss two base services implemented in the framework: the business event logging service and the remote calculation service.

Tamper-Proof Logging
An investment bank works with a large number of counter parties and semi-autonomous business units. One of the attractions of a secure SOA is that it enables the business to interact with its complex web of partners in a more stream-lined fashion. An example of a service that promotes this sort of business activity is a tamper-proof logging service.

The framework has an implementation of an event logging service that uses hashing and encryption algorithms to make the log tamper-proof. The service exposes a Web service interface and is hosted by BizTalk orchestration that not only logs the business event but also publishes the event as a BizTalk message. This enables subscribing orchestrations to be triggered by business events of interest to them (see Figure 1).

As an example, consider the scenario where a bank client is using a bank-supplied application to obtain financial information. If the tamper-proof logging service exists, then both the sending and the receipt of the information can be recorded in the log. Each record will contain an encrypted hash of the message payload and the tamper-proof nature of the log guarantees that disputes can be settled with unambiguous information. Moreover, it would be possible to create an auditing service (written as a BizTalk orchestration) that subscribed to either or both of the send and receive events. In this way, BizTalk's efficient messaging subscription design allows services to be loosely coupled with each other.

It is easy to see that when a service such as tamper-proof logging exists within the bank, it can quickly become a component of many higher order services. One such service is the remote calculation service.

Remote Calculation Service
Spreadsheets are used extensively in an investment bank. Traders and back-office staff are very familiar with this sort of technology and build complicated models of trades, risk calculations, and so on. Spreadsheets have plug-ins for market data feeds from providers such as Reuters or Telerate and often make use of add-in analytic libraries that implement algorithms for calculating risk, curves, and prices.

Some of the algorithms run by traders, back office reconciliation, or business controlling staff take a surprisingly long time to run. Value-at-risk calculations tend to fall into this category and it is not unusual for a risk calculation to take a couple of hours to complete.

To see why spreadsheets are used so widely for these sort of calculations, consider the trader shown in Figure 2. Here, the trader has their risk model, which they have honed over the years. The spreadsheet model is fed with live data from the market data feeds and the trader uses their expert knowledge to input candidate prices, rates, or spreads into the model. At the end of the trading day the trader will run their value-at-risk calculation to inform their trading decisions for the next day.

Now consider the position of the business controller who is responsible for assessing the value at risk for a whole trading floor. They are in the invidious position of taking each of the trader's spreadsheets in turn, validating that the market data has been approved, and running each calculation. This sort of situation was one reason for developing the remote calculation service (see Figures 3 and 4).

The remote calculation service is one of the base services of the framework and can run a computation remotely on behalf of its client - one computation that is supported is spreadsheet calculations (including the driving of macros and add-ins, etc). The service allows its client to schedule a computation immediately or for some time in the future on a recurring basis. The computation can also be scheduled to run on receipt of a business event logged via the logging service described earlier.

The remote calculation service is implemented as a BizTalk orchestration and has support to return the calculation results using a custom output orchestration. This feature, again, relies on the message subscription design of BizTalk. In this way, the results of calculations might be transformed and routed back to requesting users using their preferred transport mechanism (i.e., SMTP, FTP, etc.)

Figure 4 illustrates, in a simple way, how the SOA can be built using the lower level services and how the BizTalk's flexibility can be utilized to add application, regional, or even user-specific processing into the architecture.

Investment banks have complicated businesses that spread a round the world with many different trading partners. They operate in an environment of constant change and need to react very quickly to keep their competitive edge. This article has shown how one investment bank has reacted to these challenges and how BizTalk Server can help make the agile service-oriented architecture a reality.

More Stories By David Regan

David Regan is a freelance systems programmer specializing in data- and rule-driven multitier systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

Microservices Articles
The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
When building large, cloud-based applications that operate at a high scale, it’s important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. “Fly two mistakes high” is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee A...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...