Welcome!

Agile Computing Authors: Liz McMillan, Zakia Bouachraoui, Elizabeth White, Pat Romanski, Maria C. Horton

Related Topics: Microservices Expo

Microservices Expo: Article

SOA and the Art of Riding the Enterprise Service Bus, Part II

The Need for the ESB and Other Infrastructure Technologies

In the first article in this series (Vol. 4, issue 9), I described some of the middleware requirements necessary to support a full-function service-oriented architecture (SOA), and introduced the role of the enterprise service bus (ESB) in meeting those requirements.

In this article, I will go on to consider some specific situations in which ESB capabilities could be deployed. I will also describe how ESB capabilities relate to other architectural components in an SOA, and briefly discuss suitable technology options for implementing them.

The Capabilities of the Enterprise Service Bus
Now I'll summarize and categorize some of the ESB capabilities driven by the requirements we discussed. Whilst some of the capabilities are quite basic, some (such as autonomic or intelligent capabilities) represent significant steps towards an on- demand operating environment (see References). By analyzing which capabilities are applicable to specific SOA scenarios, we can refine our understanding of which technologies would be suitable candidates for implementing the ESB. In particular, we will go on to consider which of these capabilities are the minimum necessary to constitute a useful ESB.

Communication Capabilities
The ESB should support communication of service interactions through a variety of protocols. It may do this by providing a communication infrastructure, or by leveraging or integrating with existing infrastructures, such as HTTP or other middleware technologies, such as WebSphere MQ. The ESB should be able to route service interactions through a variety of protocols, and transform from one protocol to another where necessary.

Service Interaction Capabilities
The ESB should support the concepts of SOA, particularly the use of interfaces and policies to declare service operations and quality of service characteristics, respectively. The ESB should also support service-messaging models consistent with those interfaces, and capable of transmitting the required interaction context, such as security, transaction, or message correlation information. The ESB should also support the need to publish, discover, locate, and bind to services. Increasingly these capabilities will be based around Web services standards such as WSDL, SOAP, WS-PolicyFramework, and UDDI.

Integration Capabilities
To support SOA in a heterogeneous environment, the ESB needs to integrate with a variety of systems that do not directly support service-style interactions. These may include legacy systems, packaged applications, or other enterprise application integration technologies. This might require support for protocols (e.g., JDBC, FTP, EDI, etc.), adaptors (e.g., J2C or WebSphere Business Integration Adaptors), or for client APIs for various languages (e.g., Java, C+, C#) and platforms (J2EE, .NET, CORBA, etc.).

Quality of Service Capabilities
The ESB will support service interactions with different values to the business; it should therefore be capable of supporting various qualities of service. This might require support for atomic or compensated transactions, various levels of delivery assurance, error and exception handling processes, etc. These capabilities may need to be supported even when service interactions take place through unreliable protocols such as HTTP, and when a service is delivered to a consumer by aggregating several services from different providers. Ideally, these capabilities should be controlled by declarative policies associated with the services involved.

Security Capabilities
The ESB needs to both provide a security model to service consumers and integrate with the (potentially varied) security models of service providers. Both point-to-point (e.g., SSL encryption) and end-to-end security capabilities (e.g., identity pass-through, trust models, WS-Security etc.) will be required. The ESB can also support broader security concerns by providing centrally controlled auditing, logging, or systems management, etc.

Message Processing Capabilities
In a realistic integration scenario, there are likely to be multiple messaging and data models. The ESB will play a role transforming between these, whether between legacy data formats (e.g., COBOL copybooks) and XML, between basic XML formats and Web services messages, or between different XML formats (perhaps transforming an industry standard XML message to a proprietary or custom XML format). This will require the ESB to have some form of message processing or mediation capability, and the ability to manipulate the format and data of messages. In some cases, the ESB will need to make decisions (such as where to route service requests) based on the formats or content of messages associated with services.

Management and Autonomic Capabilities
As with any other middleware, the ESB will need to integrate with management and monitoring systems. Increasingly, management software will need to monitor business transactions that pass through several service interactions, so the ESB will become a key monitoring and management point. As organizations migrate towards an SOA and onwards to an on-demand operating environment, the infrastructure will need to evolve to perform more sophisticated function, such as billing, demand-based routing, or policy-based behavior (e.g., the ability to select service providers dynamically based on the quality of service they offer compared to the business value of individual transactions). All these imply that the ESB will have an increasingly sophisticated capability to monitor and alter its own behavior.

Minimum ESB Capabilities
Of course, not all the capabilities described above will be relevant to all the situations in which some form of ESB will deliver technical and business value. In that light, we need to consider what the minimum capabilities of an ESB should be in order to address the requirements we considered in the first part of this article, i.e., the ESB should support:

  • Decoupling the consumer view of services from their implementation
  • Decoupling technical aspects of service interactions
  • Integrating and managing services in the enterprise
The minimum capabilities required to achieve these goals are:
  • Communication capabilities: Support routing and addressing for at least one messaging style (such as request/response, or publish/subscribe), and at least one transport protocol that is or can be made widely available. This enables location transparency and service substitution, enabling the decoupling of the consumer view of services from their implementation.
  • Integration capabilities: Support several integration styles or adapters. It should enable services to be provided based on these integration capabilities, enabling the decoupling of technical aspects of service interactions and the integration of services in the enterprise.
  • Service interaction capabilities: Support an interface definition format and associated messaging model (such as WSDL and SOAP) to allow the decoupling of technical aspects of service interactions.
  • Management and autonomic capabilities: Provide a consistent administration model across a potentially distributed infrastructure, including control over naming, routing, addressing and transformation capabilities. This enables the management of services in the enterprise.
The ESB and Related Infrastructure Components
Now that we understand the requirements for an ESB and the capabilities it provides, we will look at some common situations in which an ESB, or similar infrastructure technologies, might be used, and how that affects the requirements and capabilities. (These uses have been described in more formal terms in the IBM Patterns for e-business [see References]).

The Enterprise Service Bus
The orthodox view of the ESB is of course its use to support SOA within a single organization. It supports the need to reuse function more widely between applications and across lines of business, and to source reusable function both internally and externally. The ESB allows service consumers to access services through an intermediary that presents a business view of available services, regardless of where they are implemented. Figure 1 shows an ESB acting as a service intermediary between applications within an organization.

We can relate this use of ESB technology to our original requirements:

  1. Decoupling the consumer view of services from their implementation: The ESB acts as an intermediary within the organization. All consumers source their function from the ESB, rather than from each other. The overall organization is therefore free to change implementations of applications or individual services, without affecting the consumers of the service.
  2. Decoupling technical aspects of service interactions: The ESB provides a decoupling point between consumers and providers. It can transform addresses, security models, assurance models, data formats, etc.
  3. Integrating and managing services in the enterprise: The ESB provides a management capability for interactions between applications and systems. It can provide consistent models for security, logging, and monitoring, or other capabilities, and where necessary map these to the individual models of participating systems.
The ESB Gateway
Intermediary technology such as the ESB can also be applied at a governance boundary - usually between one organization and the outside world, but also within large modular or distributed organizations. Figure 2 depicts this usage. ESB capabilities are required in this situation to fulfill two broad requirements:
  • To provide external service consumers with a consistent view of the services available to them, in terms of protocols, service namespace and models of security, transactionality, etc.
  • To provide internal service consumers with a consistent view of the external services with which they are permitted to interact, and again provide at least a manageably consistent view of protocols, service namespaces and models of security, transactionality, etc.
This use of ESB technology may be known as a "Service Gateway", a "B2B Gateway," or referred to in several other ways - but the underlying principles are the same as the ESB. In the IBM Patterns for e-business we use "ESB Gateway" as a unifying name that emphasizes the use of ESB technology at a governance boundary, but allows variations in usage such as the "Exposed ESB Gateway" where the boundary is between one organization and the wider world.

The ESB Gateway pattern will often emphasize security capabilities, and tends to concern basic routing of service interactions rather than aggregations or composition, but in reality these issues will vary from case to case. The ESB and ESB Gateway components will often be used together when both internal and external services need to be integrated.

We can relate this use of ESB technology to our original requirements:

  1. Decoupling the consumer view of services from their implementation: The gateway provides an intermediary that can publish service definitions to external consumers, and to which those consumers can bind. The organization operating the gateway is then free to change the implementation of services without affecting consumers - it can either change it's own implementation or subcontract services to an external provider.
  2. Decoupling technical aspects of service interactions: The gateway provides a decoupling point between consumers and providers. It can transform addresses, security models, assurance models, data formats etc. to support whatever level of decoupling is required.
  3. Integrating and managing services in the enterprise: The gateway provides a single management point for all external services - whether they are provided or consumed from the point of view of the gateway operator.
Other Infrastructure Requirements for SOA
Now that we have seen the role of the ESB and gateway components, we can consider a more complete infrastructure for SOA by adding components such as service directories and service choreographers. We don't have the space to describe all of these in detail, but by giving a brief overview we can at least place the ESB in a clearer context. Figure 3 illustrates an overall architecture for SOA. The roles of the other components in this architecture relative to the ESB are as follows:
  • The Internal and External Service Directories catalogue the services available to different groups of service consumers. Service providers will publish services to these directories to be consumed by intermediaries such as the ESB or ESB Gateway. Intermediaries will publish facade services to the directories, to be consumed by service consumers. Note that service directories should be understood to be separate from the service routing information required by the ESB in order to function, although as directory standards such as UDDI evolve, these two roles may become more closely entwined.
  • The Service Choreography Component aggregates, composes, and choreographs services into larger-grained services or processes. The service choreography component acts as a consumer of facade services provided by the ESB, and as a service provider to the ESB.
Implementation Technologies
The ESB components described above can be implemented either using proprietary technologies, by using open standards, or through custom development. The decisions as to which of these techniques should be used to implement which ESB capabilities are key decisions in implementing SOA today. As we undergo a rapid evolution of ESB technologies, Web services standards, and SOA concepts and techniques, organizations will need to consider a number of issues and how they will evolve over time:
  • Will an open standards approach offer specific interoperability benefits, and what level of work will be required to achieve that interoperability? For example, do all the technologies involved support the required open standards (e.g., perhaps supporting security standards such as WS-Security in addition to basic service interaction standards such as WSDL and SOAP)? Has interoperability been tested, perhaps through compliance to the Web Services Interoperability Organization profiles?
  • What degree of performance, scalability, integrity, or other service levels are required? How do they relate to the capabilities and maturities of standards-compliant versus proprietary technologies?
In this context, a number of implementation options should be considered for the ESB, including both standards-compliant and proprietary integration technologies. In reality, of course, any technology will provide a mixture of standards-compliant and proprietary features; the important task is to match the benefits of both aspects of the technology to individual cases. In this light, some of the implementation options for ESB components include:
  • Using specific features of enterprise application integration middleware
  • Using "Service Gateway" or "B2B Gateway" technology
  • Custom solutions using XML, message queuing and brokering technology, or application server communication and integration capabilities
We don't have the space here to go into the specifics of individual technologies, their features, or the choices to be made between them. However, the whitepapers and Redbooks in the references contain in-depth information on these topics, consistent with the description of the ESB provided here.

Acknowledgments
Some of this material draws on collaborative work; the following people contributed material to this article or participated in discussions that led to the ideas behind it: Jonathan Adams, Norbert Bieberstein, Bernhard Borges, Beth Hutchison, Rachel Reinitz, Keith Jones, Paul Verschueren, Olaf Zimmerman, Helen Wylie, Kyle Brown, Mark Colan, Paul Fremantle, Daniel Sturman, Dave Clarke, Kareem Yusuf, Chris Nott, and Alan Hopkins.

References

  • The IBM Redbook Patterns: Implementing an SOA using an Enterprise Service Bus: www.redbooks.ibm.com/abstracts/sg246346.html
  • The IBM developerWorks series "Understand Enterprise Service Bus Scenarios and Solutions in Service-Oriented Architecture" parts 1-3: www-106.ibm.com/developerworks/webservices/library/ws-esbscen, www-106.ibm.com/developerworks/webservices/library/ws-esbscen2.html, www-106.ibm.com/developerworks/webservices/library/ws-esbscen3
  • The on demand operating environment: www-306.ibm.com/e-business/ondemand/us/ evolvetech/operating_environment.html
  • Web Services Interoperability Organization: www.ws-i.org
  • More Stories By Rick Robinson

    Rick Robinson is an IT architect in IBM Software Services for WebSphere, based at the Hursley Laboratory in the United Kingdom. He has seven years of experience in the IT industry, and his roles have included solution architecture, design, and development. Rick holds a PhD in the physics of superconducting devices from the University of Birmingham. His areas of expertise include distributed systems design, the WebSphere platform, and service-oriented architecture.

    Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    IoT & Smart Cities Stories
    The platform combines the strengths of Singtel's extensive, intelligent network capabilities with Microsoft's cloud expertise to create a unique solution that sets new standards for IoT applications," said Mr Diomedes Kastanis, Head of IoT at Singtel. "Our solution provides speed, transparency and flexibility, paving the way for a more pervasive use of IoT to accelerate enterprises' digitalisation efforts. AI-powered intelligent connectivity over Microsoft Azure will be the fastest connected pat...
    There are many examples of disruption in consumer space – Uber disrupting the cab industry, Airbnb disrupting the hospitality industry and so on; but have you wondered who is disrupting support and operations? AISERA helps make businesses and customers successful by offering consumer-like user experience for support and operations. We have built the world’s first AI-driven IT / HR / Cloud / Customer Support and Operations solution.
    Codete accelerates their clients growth through technological expertise and experience. Codite team works with organizations to meet the challenges that digitalization presents. Their clients include digital start-ups as well as established enterprises in the IT industry. To stay competitive in a highly innovative IT industry, strong R&D departments and bold spin-off initiatives is a must. Codete Data Science and Software Architects teams help corporate clients to stay up to date with the mod...
    At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
    Druva is the global leader in Cloud Data Protection and Management, delivering the industry's first data management-as-a-service solution that aggregates data from endpoints, servers and cloud applications and leverages the public cloud to offer a single pane of glass to enable data protection, governance and intelligence-dramatically increasing the availability and visibility of business critical information, while reducing the risk, cost and complexity of managing and protecting it. Druva's...
    BMC has unmatched experience in IT management, supporting 92 of the Forbes Global 100, and earning recognition as an ITSM Gartner Magic Quadrant Leader for five years running. Our solutions offer speed, agility, and efficiency to tackle business challenges in the areas of service management, automation, operations, and the mainframe.
    The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
    With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading...
    DSR is a supplier of project management, consultancy services and IT solutions that increase effectiveness of a company's operations in the production sector. The company combines in-depth knowledge of international companies with expert knowledge utilising IT tools that support manufacturing and distribution processes. DSR ensures optimization and integration of internal processes which is necessary for companies to grow rapidly. The rapid growth is possible thanks, to specialized services an...
    At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...