Bick Group

David Linthicum

Subscribe to David Linthicum: eMailAlertsEmail Alerts
Get David Linthicum: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: SOA & WOA Magazine

SOA & WOA: Article

Service Archaeologist

The hot job for 2005

Want to leverage your enterprise's Web services? Chances are you'll be enabling or exposing existing application services and not building new. This should come as no surprise to anyone. However, while we've been focusing on the development of new services, how to do it, and what tools to use, most of the work that I see coming is learning how to translate and expose legacy services. So, my prediction is that one of the best paid jobs in 2005 will be engineers specializing in Web services enablement of legacy systems. I'll call them service archaeologists.

Indeed, most corporate data and business processing resides on mainframe computers; any attempt to move to an SOA has to include legacy/mainframe integration. Mainframes are not the mainframes of yesteryear. The good news is that most mainframes today are designed to work and play well with other systems within your IT infrastructure, and offer many points of integration, both at the information and the service levels. The bad news is that traditional mainframe-based applications were not designed to communicate outside of the processing space, and thus with some types of legacy systems Web service enablement is an unnatural act. However, there are always mechanisms to provide integration, it's just a matter of understanding what your requirements are, and selecting the correct technology.

Common Issues for "The Dig"
Legacy systems, such as mainframes, have three common issues that you must take into consideration: unstructured data, batch-oriented, binding of logic and data.

Mainframe information has a tendency to be unstructured, meaning that the information is contained in flat files or index sequential files, and not a database. This is challenging to those looking to expose that data into their SOA since a lot has to be understood pertaining to how the data is stored and retrieved. While this can make integration difficult, it's not impossible. In some instances changes must be made to mainframe programs to externalized unstructured data, and in some cases you can account for the unstructured data within the middleware layer through data abstraction facilities.

The fact that many legacy/mainframe systems are batch-oriented presents its own set of problems and opportunities. SOAs typically don't deal with batch systems; information moves between systems record by record, service by service. So, in many instances changes must be made to internal processing on the mainframe to support a more granular data exposure mechanism. However, in some cases it may make better sense to design your SOA in such a way as to provide the ability to accept and produce large amounts of batch data. Services can be written to accomplish this task.

Finally, many mainframe-based information systems do not have a clear separation of data and logic, such as those that leverage file-based information (e.g., ISAM/ Cobol). This is more of a challenge to understand than integration. Indeed you must understand how the code is designed to access the information, and then make a decision to expose the mainframe code as a service, sometimes requiring changes to the code. Or, access the information on its own, if possible, binding the data to services outside of the legacy system.

Mainframe Access Mechanisms
Your choices for enabling technology to access mainframe-based systems are numerous. Typically your hardware vendor will assist you in selecting the proper integration technology for your requirements, but you need to understand all of the potential solutions available in the marketplace which are becoming numerous.

To access mainframe-based processes you may use Logical Unit 6.2 (LU6.2). This is IBM's device-independent, process-to-process protocol that provides the facilities for peer-to-peer communications between two programs and also supports asynchronous networking. This mechanism allows you to leverage an internal mainframe process, and if needed, expose it as a service using the LU6.2 as the access point, "wrapping" it as a Web service. You can build services on top of LU6.2 yourself, or leverage a middleware layer that manages the internal process to Web Services translation for you (using LU6., or other interfaces).

For instance, as I'm writing this column Cape Clear Software and NEON Systems, Inc., announced that the two companies are working together to enable organizations to simplify the integration of their mainframe applications and data using Web Services and service-oriented architecture (SOA). The agreement includes tight integration between Cape Clear's Enterprise Service Bus (ESB) and NEON Systems' Shadow z/Services product, allowing organizations to integrate their mainframe applications and to provide those applications as services to the rest of the network.

To expose mainframe data as services to your SOA, you may leverage database gateways. These are also known as SQL gateways and are APIs that use a single interface to provide access to most databases that reside on many different types of platforms. They are similar to virtual database middleware products, providing developers with access to any number of databases residing in environments that are typically difficult to access, such as a mainframe. For example, using an ODBC, JDBC, or Web services interface and a database gateway, developers can access data that resides in a DB2 database on a mainframe, or in an Oracle database running on a minicomputer, and in a Sybase database running on a UNIX server. The developer simply makes an API call, and the database gateway does all the work, including distributing the query and assembling the data, perhaps exposed as a single virtual schema or view.

Database gateways translate the SQL calls into a standard format known as the Format and Protocol (FAP), the common connection between the client and the server. FAP is also the common link between very different databases and platforms. The gateway can translate the API call directly into FAP, moving the request to the target database, and translating the request so that the target database and platform can react.

Clearly, the use of legacy systems will continue, and as we move quickly to SOA, so will the enablement of these systems. The trick is to identify the services that are important to the enterprise now and create a plan for exposing those services including which enabling technology and standards to apply. Once you've done that prepare yourself for some complexity. None of these systems were designed to expose services, and it's not going to be an easy job no matter how good the translation and wrapping technology is. It could, in most cases, mean that you need to change and test code over 20 years old. Oh well, that's why service archaeologists will be making the big bucks.

More Stories By David Linthicum

Dave Linthicum is Sr. VP at Cloud Technology Partners, and an internationally known cloud computing and SOA expert. He is a sought-after consultant, speaker, and blogger. In his career, Dave has formed or enhanced many of the ideas behind modern distributed computing including EAI, B2B Application Integration, and SOA, approaches and technologies in wide use today. In addition, he is the Editor-in-Chief of SYS-CON's Virtualization Journal.

For the last 10 years, he has focused on the technology and strategies around cloud computing, including working with several cloud computing startups. His industry experience includes tenure as CTO and CEO of several successful software and cloud computing companies, and upper-level management positions in Fortune 500 companies. In addition, he was an associate professor of computer science for eight years, and continues to lecture at major technical colleges and universities, including University of Virginia and Arizona State University. He keynotes at many leading technology conferences, and has several well-read columns and blogs. Linthicum has authored 10 books, including the ground-breaking "Enterprise Application Integration" and "B2B Application Integration." You can reach him at david@bluemountainlabs.com. Or follow him on Twitter. Or view his profile on LinkedIn.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Stuart 11/01/04 03:33:30 PM EST

Excellent concept on Service Archaeology, however the specific scope based on legacy Mainframe misses the total mark. Another seriously important bank of legacy data is of course ... the database. What about stored procedures, Forms, Powerbuilder, etc, where the same symptoms lay waiting, even commanding that software engineers dig into their dark caves?

I believe that this second sector is even more prevalent in the SOA / Web Services legacy market.