There is no disputing the fact that distributed object technology has come a long way since it's inception. However, before the distributed object industry can legitimately claim for itself the title 'ready for prime time,' there must first be an emergence of robust systems management tools that are engineered to support the complex needs of the distributed object environment. The maturity and success of distributed object technology in fact depends upon it's overall reliability and resistance to failure when operating under the harsh combat conditions of large scale production environments.
This article is the first in a two part series which addresses the subject of systems management for the distributed object environment. This first article identifies the systems management issues that were confronted by a systems development team at Wells Fargo Bank as it tackled the many challenges of controlling a large mission critical Common Object Request Broker Architecture (CORBA) system in the production environment. The various ways by which CORBA method servers can be managed and controlled are discussed. Part two profiles ORB systems management utility services.
Lack of Vendor Tools
A distributed object environment is one that utilizes an ORB to route messages from client requesters to method server processes residing across multiple, potentially disparate platforms in order to consummate object-oriented invocations to distributed objects. In order to manage a distributed object environment, most Object Request Broker (ORB) vendors provide users with a rudimentary and limited number of commands that can be used to start, stop, and monitor individual method server processes. However, these tools are insufficient to support the needs of large scale, around the clock, critical production systems that can span multiple platforms. Due to the lack of robust vendor tools in this domain, adopters of CORBA technology must provide their own software to fashion home-grown management mechanisms. As the saying goes, "necessity is the mother of invention". The ORB Systems Management (OSM) framework described in this article is based upon an actual system which, out of necessity, was conceived and constructed at Wells Fargo Bank, to provide high quality production support.
Production Survival Questions
After recognizing the limitations of the shrink wrapped tools that accompany ORB products, the conclusion was quickly drawn that an investment must be made to build a dynamic suite of tools that could ensure stability and limit failures in the production environment. Some basic questions were asked that stimulated thinking about what constituted an OSM capability. A partial list follows.
As this list suggests, managing a distributed object environment is an ambitious and challenging effort. Certainly, the power and competitive advantage that distributed objects present to a firm are enormous. At the same time, potential adopters need to be aware that there is an upfront price to be paid in building OSM-compliant tools as a prerequisite for enterprise-wide deployment of distributed objects. On the other hand, what is the price to be paid if a firm chooses not to build OSM tools and then experiences a major meltdown in production due to an inability to rapidly detect and resolve problems?
What is ORB Systems Management?
ORB Systems Management refers to the ability to control, monitor, configure and recover application resources that are essential to the proper functioning and stability of the distributed object environment. OSM is not intended to manage underlying low-level system resources per se. That task is still dedicated to products that typically utilize SNMP to manage and control devices and network resources. Instead, OSM focuses on managing the application specific method servers that are the essential work horses for conducting distributed object processing.
Managed Server Lifecycle Service
The Managed Server Lifecycle Service is a set of classes that are responsible for starting, naming, configuring, and stopping instances of managed servers. There are a number of architectural concepts that must first be defined in order to understand the managed server lifecycle service.
What is a Method Server?
A method server is an executable process that dispatches incoming messages received from clients to specific method routines within a distributed object. A method server may contain one or more CORBA class implementations. Typically, there is some form of commonality between the various class implementations bundled together within the method server. This may include a mutual superclass, or perhaps it may include a shared link to a backend system or external service provider. For example, a Customer method server could contain a Customer class implementation and a CustomerFactory class implementation.
What is a Managed Server?
What is missing in the typical method server is the inclusion of a special class that functions specifically to provide ORB systems management services. A server management class is used to communicate with other OSM components to gain control over the method server, and supply value-added management functions. Some of these functions include the ability to shutdown, quiesce and disable a method server, the ability to query a server's state, query the health of it's implementations, query it's current version, query it's performance statistics, dynamically alter the behavior of the managed server by resetting it's internal configuration attributes, etc. A server management class contains references to the implementations of each of its managed classes. A method server that utilizes the services of a server management class is called a managed server. A method server without a server management class is totally out of control, and will eventually be more of a hindrance to the organization than a benefit. Managed servers are thus essential prerequisites for properly gaining supervisory authority over method servers. To become managed, a method server must include a new subclass of OSM_ManagedServer. This class is in addition to the implementation classes that perform the business functionality. For example, the Customer managed server must contain the OSM_CustomerManagedServer class implementation along with the Customer and CustomerFactory class implementations. (Figure 1).
The managed server class is equipped to communicate with the OSM management system and can receive requests to perform operations on behalf of the implementations that it is responsible to manage. For example, if the managed server class receives a quiesce request, it will in turn trigger quiesce operations on each of it's supported implementations. In essence, the managed server implementation is the contact point for the OSM system to invoke operations that effect all of the other implementations within a method server. The managed server implementation acts as a management agent for all of the other implementations contained within the method server.
Each managed server may undergo a variety of state transitions during its lifecycle. OSM requires that the internal state transitions of the managed server be externally identifiable so that the management system can perform effective monitoring and control functions. The states include: available, in-flight, running, termination-pending, terminating, and disabled.
All managed servers follow a standard protocol during their startup phase. The OSM framework shields application developers from having to code the low level CORBA registration APIs. OSM
instead controls all internal CORBA initialization, registration of implementations, readying of implementations, as well as deactivation and termination logic.
Managed Server Identity
In a runtime environment, there may be many running instances of a managed server. Each instance of a managed server is given a unique name, called a server name. This allows the management system to individually identify each server for control purposes, by either operators or by automated means. For example, a Customer managed server might be known as CUST0001. A second instance might be named CUST0002. Server names are prerequisites for allowing the OSM management system to define and pre-configure a pool of managed servers for different classes. OSM establishes a tight coupling between a managed server name and a static object reference to the managed server class implementation. OSM is now able to directly access the managed server without excess performance overhead.
Managed Server Proxy Classes
A Managed Server Proxy is a class that represents the actual running executable process, but is in fact a persistent object residing inside a totally separate and independent process. In a sense the proxy class contractually has the 'power of attorney' to perform special transactions on behalf of the actual managed server, without having to intrude on the running managed server to perform the operation. For example, the managed server proxy contains attributes that represent the condition of the running managed server. This might include the managed server's state, the time the server started running, it's process id, the time the server process ended, the reason for its termination, a static object reference to the actual managed server, etc.. The proxy is created when the managed server is initially defined to the OSM control system. When the proxy receives a launch request from the OSM system, it spawns the actual managed server process it represents. When the managed server completes it's startup process, it pushes it's new state value to the proxy. The OSM control system can then ask the proxy for the correct state of the managed server. The proxy is also responsible for redirecting shutdown messages from the OSM control system to the managed server. (Figure 2).
Cascading Containment Relationships
It is essential that managed servers be controlled in groups. In order to provide effective production support, economies of scale must be leveraged. There are a number of containment relationships that exist between managed server proxy classes. These containment relationships basically represent a bill of materials structure, where one class contains another class, which contains another class recursively. At the lowest level is the actual managed server proxy. This represents a specific server such as CUST0001. At the next higher level is the ManagedServerGroup class. Individual managed server instances may be bundled together in groups within a common managed server class. Usually server groups include managed servers that are linked together into a set based upon mutual characteristics. For example, one group of Customer managed servers might have communication links to a North data center, while another group of Customer managed servers might have communication links to a South data center. Such redundancy might be needed in order to encapsulate a legacy system that is replicated in two data centers for business resumption reasons and for high availability. A CUST_North group and a CUST_South group could be created to contain individual managed servers. At the next higher level of containment is the actual Customer managed server class, which contains all Customer server groups. Above the managed server class level is the ManagedServerContainer class. This class is totally ad hoc and user definable. A managed server container class contains one or more managed server classes. For example, all managed server classes that access the North data center are collectively contained in a managed server container called North. For example, the North container would reference server groups for Customer, Account and perhaps Product, that mutually point to the North data center. A container class can also recursively contain other container classes. (Figure 3).
Based upon these containment relationships it is possible to invoke methods at any of the containment levels and expect the request to cascade down to the lowest level, that is, the level of the managed server proxy. The managed server proxy, will in turn, send the actual request to the running process that it represents. For example, a server 'shutdown' request that is directed at the North container would trigger a series of shutdown requests that are ultimately aimed at all running managed server processes that have internal links to the North data center.
Managed Server Properties
Each managed server can also utilize properties or attributes that are dynamically assigned in the runtime environment. For example, a managed server might be assigned properties like a unique IP address and port number in order to establish a socket connection to a particular external entity, like a CICS region. These properties are critical to the configuration of the managed server, and are therefore prerequisites for the proper functioning of the server. The managed server proxy is responsible for passing to it's respective managed server the object reference for it's properties. Properties can change during the lifecycle of a managed server. Once a property value changes, the behavior of the server might also change. For example, upon receiving a change to a communications port property, the managed server might dynamically reconnect from the previous port attribute to the newly specified port.
Class properties are attributes that are assigned at the managed server class level, such as OSM_CustomerManagedServer. Group properties are attributes for all managed server groups. Although group properties inherit their values from class properties, their values can be overridden. At the lowest level, the managed server proxy level, are instance properties. Instance properties are properties modeled after group properties. They can also have values that can be potentially different for each managed server proxy instance due to overriding. (Figure 4).
If a property changes at the class level, the change cascades down to all of the managed server groups, and then down to the managed server proxy instances. This approach provides for a powerful and highly flexible means to dynamically change the behavior of numbers of managed servers. Keep in mind that the changes to properties values only effects the managed server proxy. The actual managed server does not receive the altered properties until the OSM management system sends it a request to dynamically reset it's internal properties. For example, a running managed server might temporarily begin detailed trace logging of it's activities as a result of receiving an updated property value that controls it's trace logging levels. In another example, a managed server might dynamically switch a connection from one database to another database while the former is receiving corrective maintenance as a result of a change to a database connection property.
There can also be a properties capability at the managed server container level. This provides for a property to cascade across multiple container relationship levels and across multiple managed server classes.
Conclusion
Without the extensive toolset made available through OSM it is unlikely that the stability of the distributed object environment could be completely ensured. Thus, OSM-izing method servers is an essential prerequisite for production deployment.
Author
David Newman, architect of OSM, is President of Technium, Inc. a Walnut Creek, California, consulting firm specializing in distributed object technology and class-based business engineering. www.technium-inc.com.