IMHO

Integration Platforms: The Missing Link in
Enterprise Computing

By John Ousterhout
Web Techniques Magazine
May 1999
Volume 4, Issue 5

The most significant challenge in enterprise computing today is integration. The infrastructure of a modern business contains countless applications, devices, protocols, data sources, and frameworks. For an organization to run efficiently, all of its resources must be coordinated, and it must be easy to create new applications that leverage the capabilities of existing resources. Furthermore, a business that incorporates new technologies, such as those for the Web, must be able to integrate them with existing systems.

Unfortunately, many businesses today are struggling with this integration problem. Traditional system programming languages such as C, C++, and Java are poorly suited to the integration task. Approaches such as component frameworks and enterprise application integration (EAI) tools solve only part of the problem. As a result, companies are experiencing high development costs for integration applications. In many cases, the resulting applications are not flexible enough to handle the variety of resources that must be integrated or to evolve rapidly as business needs change.

In this article I'll describe the integration problem in more detail and explain why a new kind of software platform is emerging as the best solution. These integration platforms are based on powerful scripting languages such as Tcl, which stands for Tool Command Language, and can be used in a variety of ways to coordinate all of the resources of an enterprise. Over the next few years, integration platforms will rise to the same level of importance in an enterprise as other key software platforms such as the operating-system platform and the database platform.

The Integration Problem

The need for integration has grown out of the continual advance of business automation. In the 1980s, businesses moved their data online using general-purpose database systems. Then, starting in the late 1980s and accelerating in the 1990s, applications appeared that used the online data to automate individual business functions such as sales, customer service, and manufacturing. Now businesses are integrating the individual applications into coordinated systems. Companies that succeed in their integration activities will operate more efficiently, serve their customers better, and capitalize on opportunities more quickly.

The Internet and the World Wide Web have greatly increased the urgency of the integration problem. The growth of the Internet has made many new resources accessible, opening up opportunities for integration both within and between organizations. The Web has created totally new ways of communicating with customers and conducting business. To take advantage of the Web, businesses must integrate their existing applications with new Web technologies.

Several other factors have also contributed to the rise of integration applications. The proliferation of embedded processors in everything from factory equipment to cellular telephones has made it possible to automate physical processes and coordinate them with other resources. Corporate merger and acquisition activity requires the computing infrastructures of different companies to be joined together and coordinated. Component frameworks such as ActiveX and Enterprise JavaBeans provide large families of components ready for integration.

Old Tools Don't Work

The traditional approach to application development is based on system programming languages such as C, C++, and, more recently, Java. These languages have their roots in the computer systems of 35 years ago, when all applications were written from scratch and the primary problems addressed by programmers had to do with the design of data structures and algorithms. System programming languages are the right tools for building components and base applications, but they do not work as well for the integration task.

In integration applications, data structures and algorithms play a smaller role. They're typically encapsulated inside the resources being integrated. Instead, developers of integration applications face the following challenges:

System programming languages simply don't have the right properties for integration tasks. First, they're too complex and hard to learn for many integration application developers. Even worse, they're too inflexible for integration tasks. Their use of "strong typing" forces applications to be planned ahead of time. As a result, it's hard to accommodate new resources that weren't planned. Their use of compilation freezes the function of an application so that it can't evolve in the field. It's very difficult to build integration applications with C, C++, or even Java, and the resulting applications are static and brittle.

Integration Platforms

Integration problems are sufficiently different from traditional programming problems that they require a new software substrate. I call this an integration platform. An integration platform must have two fundamental properties: programmability and connectivity.

First and foremost, an integration platform must be programmable. Integration applications are complex enough to require some form of programming: purely declarative or graphical approaches aren't sufficient. In addition, the programming language must provide rapid development in the face of heterogeneity and constant change, which leads to three requirements:

The second fundamental property for an integration platform is connectivity. The platform must go everywhere and interoperate with everything. To meet the connectivity requirement, an integration platform must have three properties:

Portability: The platform must run on all of the world's important computing systems, including Windows, UNIX, and many others.

Extensibility: Although the platform should have built-in support for popular resources, it must also be easy to extend the platform to communicate with additional resources, such as new technologies or legacy applications.

Embeddability: It must be possible to embed the integration platform directly into applications and devices. This allows scripts in the integration language to be mixed with data of the application (such as small scripts intermixed with HTML in Web pages), and it allows scripts to be downloaded into a device or application to modify and extend its behavior.

Tcl

A leading example of an integration platform today is the Tcl scripting language (see "Online"). I created Tcl at U.C. Berkeley in the late 1980s for use as an application command language and also for creating graphical user interfaces. It has been available on the Internet as an open-source package since the early 1990s. Its usage has spread to include more than half a million developers and thousands of applications that span the integration space. For example, NBC uses Tcl to integrate corporate databases with video storage and transmission devices in order to automate its broadcast control systems. High-end Web sites such as CNET and AOL's Digital City use Tcl to integrate data from diverse sources to create dynamic Web pages.

Tcl embodies the call of the properties needed for an integration platform. The language is interpreted and very simple. Its typeless and string-oriented nature allows complex integration tasks to be implemented with small amounts of code. The Tcl interpreter is implemented as a C library package that is highly portable and can be embedded easily in a variety of devices and applications. For example, Vignette uses Tcl in its StoryServer application for Web content management. Tcl was designed from the start to be extensible and there now exists a rich collection of extensions that provide powerful facilities for GUIs, database access, and Web/Internet programming, among others. Additional extensions can be created easily by writing C, C++, or Java code that implements new Tcl commands. (For more information, see "Tcl and Perl: A Brief History.")

Tcl is an open-source package, which gives developers a great deal of flexibility in how they use it. Tcl is also supported commercially by Scriptics Corporation. Scriptics is creating commercial extensions to Tcl that increase its value for integration in a variety of domains such as the Web and finance.

Other Approaches to Integration

Although there are many other approaches to the integration problem, none of them provides the right combination of programmability and connectivity. For example, component frameworks such as ActiveX/DCOM and JavaBeans address the integration problem by casting all resources into a standard component framework. However, they assume that system programming languages will be used to integrate the components, which results in the problems described earlier. DCOM is also too Windows-centric to solve enterprise-scale integration problems. Other scripting languages such as Perl, JavaScript, and Visual Basic (see "Online") have many of the right programming properties for integration, but they lack connectivity: JavaScript is confined to Web browsers, Visual Basic to the Windows desktop, and Perl is not easily extensible or embeddable.

Recently several new companies have begun selling EAI products to address large-scale integration problems. Typically these products contain proprietary platforms with adapters to connect to a variety of other systems such as enterprise resource planning (ERP) applications. However, EAI applications fail to meet the programmability requirements for an integration platform. In most cases they attempt to provide solutions that eliminate all programming. Unfortunately, there is no such thing as an "integration solution." Every company has a different set of resources, processes, and business rules, so some programming will be required to customize the integration platform to the particular business. Without the right programming facilities it's hard for EAI products to reach their full potential. The proprietary nature of EAI products also limits their connectivity.

 ONLINE 

AOL's Digital City
www.digitalcity.com

CNET
www.cnet.com

JavaScript developer.netscape.com/tech/javascript

Perl
www.perl.com

Tcl scripting language
www.scriptics.com/resource

Vignette StoryServer application
www.vignette.com

Visual Basic
msdn.microsoft.com/vbasic

 

The Database Analogy

The rise of database platforms in the 1980s provides a model for how integration platforms will evolve over the next decade. By the late 1970s the demand for reliable shared access to online data was increasing rapidly, and it became clear that traditional programming techniques were poorly suited to the data-management task. As a result, a new group of languages for data management was born; these evolved into what is now SQL. Database platforms, which combined SQL with sophisticated facilities for managing data, came into existence as a base for building applications, and new companies such as Oracle, Sybase, and Informix were formed to sell them. This eventually led to a whole new industry consisting of platform vendors, tool vendors, and application vendors.

Integration platforms will evolve in a similar fashion. It is now clear that existing approaches don't work well for integration problems, and a new set of languages such as Tcl is filling the gap. The integration platform, combining a scripting language with facilities for communicating with and coordinating diverse resources, is just beginning to emerge. Over the next decade a variety of companies will come into existence to provide integration platforms, tools, and applications.

Today we're seeing the first attempts to create integration platforms; most of them lack essential properties of programmability or connectivity. Over time, either they will fill out these properties or they'll be supplanted by other platforms that provide them.

 


John is CEO of Scriptics Corporation, which is creating an integration platform around Tcl. He can be reached at john.ousterhout@scriptics.com.


Copyright © Web Techniques. All rights reserved.
Web Techniques Magazine