Wednesday, May 2, 2018

Some OpenVMS PHP updates


Introduction

This post provides a brief overview of new PHP extensions provided by VMS Software Inc. for the Oracle RDB and Mimer relational databases and for the Redis in memory cache. The extensions are included in the PHP 5.6.10J kit for OpenVMS provided by VMS Software Inc., which may be installed on OpenVMS V8.4-1H1 or higher. The RDB and Mimer extensions have been developed specifically by VMS Software Inc. while Redis support is facilitated by the PhpRedis open source project (see https://github.com/phpredis/phpredis).

The following sections provide a brief overview of the three extensions, outlining the functions that are available and providing simple examples to illustrate their usage. It should be noted that the RDB and Mimer extensions have been kept deliberately simple. It is likely that additional functionality will be added over time; however the functionality that is currently provided should be more than sufficient for most cases.

RDB API

Oracle RDB is arguably the most commonly used relational database platform for OpenVMS. Over the years, various custom solutions have been developed to serve data stored in RDB databases via some form of web-based interface; however it is relatively straightforward to implement a generic RDB interface by creating a simple PHP extension that will allow developers to include queries and other database operations directly into PHP web pages.

With this in mind, VMS Software Inc. has developed a simple PHP extension for Oracle RDB that implements the set of functions listed in the table below.

Function
Description
rdb_fetch_row
Fetches a row of data using the specified cursor. The fetched data is returned as an array of strings.
rdb_ncol
Returns the number of columns (values) that would be returned by a fetch for the specified cursor.
rdb_attach
Attaches to the specified database.
rdb_close_cursor
Closes the specified cursor.
rdb_commit
Commits the current database transaction.
rdb_data
Returns the data value for the specified column for the last fetch operation.
rdb_declare_cursor
Declares a cursor.
rdb_detach
Disconnects from the database.
rdb_exec
Executes a previously prepared SQL statement.
rdb_execi
Executes the supplied SQL statement immediately.
rdb_fetch
Fetches a row of data for the specified cursor but does not explicitly return the fetched data.
rdb_free
Frees resources associated with the supplied cursor handle.
rdb_open_cursor
Opens a cursor using the supplied (previously declared) cursor handle.
rdb_prepare
Prepares an SQL statement and returns a handle for the prepared statement.
rdb_rollback
Rolls back the current database transaction.
rdb_set_readonly
Starts a read-only transaction.
rdb_error
Returns a description of the last error.
rdb_sqlcode
Returns the SQLCODE for the last database operation.

The interface has been kept deliberately simple, providing a small set of easy-to-use functions that provide sufficient functionality to address most requirements. Usage of the interface is for the most part self-explanatory and a detailed description of each individual function is therefore not required. Instead, usage is perhaps best illustrated by way of example.

The following example script illustrates the use of the extension to declare and use a database cursor to query the employees table in the sample database provided with Oracle RDB installations. The code beings by attaching to the database using the rdb_attach() function, which takes as input a single argument specifying the name of the database to attach to (in this case via the logical name sql$database). Note that the rdb_attach() function returns a completion status; it does not return any form of connection handle. For any real-world web application, the database attach would generally be performed only once as part of some initialization routine, as repeatedly connecting and disconnecting to and from the database when processing each PHP page request would incur significant overhead.

After connecting to the database, the script declares a cursor using the rdb_declare_cursor() function. The inputs to this function are the name of the cursor and the SQL statement for the cursor. Upon successful completion, the function returns a cursor handle that can then be used in subsequent calls to open and close the cursor and to fetch records. Note that it is up to the caller to specify a unique name for each cursor. Also note that cursor handles can be reused until they are explicitly freed via a call to the rdb_free() function, which frees allocated memory and cleans up any database resources associated with the handle that are no longer required.

Before opening the cursor, the script starts a new transaction using the rdb_set_readonly() function, which explicitly starts a read-only transaction on the database. It should be noted that it is not necessary to explicitly start a transaction; however it is generally good practice to do so, and if it is known that the transaction will be only reading data then it is generally beneficial to declare the transaction as read-only. This reduces resource usage helps to avoid contention issues. After starting the transaction, the script opens the cursor (using the previously allocated handle) and fetches rows until the end of the record stream is reached. Data values are displayed for each fetched record, and the cursor is closed once all records have been processed. Finally, the read-only transaction is rolled back, resources associated with the cursor handle are freed, and the script detaches from the database by calling rdb_detach(). Note that the transaction could equally be committed via a call to rdb_commit(), however strictly speaking there is no actual work to commit.

<?php
  if (! extension_loaded('rdb')) {
     if (! dl('rdb.exe')) {
        exit;
     }
  }

  if (rdb_attach('sql$database') == -1) {
     printf("%s\n", rdb_error());
     exit;
  }

  $cursor = 'C0001';
  $ch = rdb_declare_cursor($cursor, 'select employee_id,last_name,first_name from employees');

  if ($ch == NULL) {
     printf("%s\n", rdb_error());
     exit;
  }

  rdb_set_readonly();

  if (rdb_open_cursor($ch) == -1) {
     printf("%s\n", rdb_error());
     exit;
  }

  while (($row = rdb_fetch_row($ch)) != NULL) {
     print_r(array_values($row));
  }

  if (rdb_sqlcode() != 100) {
     printf("%s\n", rdb_error());
     exit;
  }

  if (rdb_close_cursor($ch) == -1) {
     printf("%s\n", rdb_error());
     exit;
  }

  rdb_rollback();

  rdb_free($ch);
  rdb_detach();
?>

Note that all functions return either an integer completion code, a cursor or statement handle, or a result set. In the event of an error, functions that return an integer completion code will return a value of -1 to indicate that an error has occurred, and other functions will return a NULL value in the event of an error. Whenever an error is returned, the function rdb_error() can be used to get a textual description of the last error and the function rdb_sqlcode() can be used to get the integer SQL error code. It should also be noted that the error handling in the above example could be improved by ensuring that any outstanding transaction is committed or rolled back and resources are properly cleaned up before existing following an error. Similar comments apply to other example code included in this document.

The above example uses the rdb_fetch_row() function to retrieve a row of data at a time, loading values for each column into an array. This is arguably the most convenient method of fetching data; however it is also possible to retrieve individual column values by using the rdb_fetch() and rdb_data() functions as illustrated below, where the call to rdb_fetch_row() in the above example has been replaced by a call to rdb_fetch() and a loop that uses rdb_data() to retrieve the values for each column for each row that is fetched. Using rdb_fetch_row() is simpler and slightly more efficient; however there may be situations where this alternative approach may be more appropriate.

while (rdb_fetch($ch) == 1) {
   for ($i = 0; $i < 3; $i++) {
      $val = rdb_data($ch, $i);
      echo "\t$val";
   }
   echo "\n";
}

The next example illustrates use of the rdb_execi() (execute immediate) function to execute a dynamic SQL update statement against the database. Note here that it would generally be good practice to explicitly set the transaction scope before performing the update, reserving only the employees table for write operations. By not explicitly setting the scope of the transaction, RDB will start a default read/write transaction reserving all tables in the database for read/write operations, which may cause contention problems if there are multiple database users, and will reserve more resources than necessary for the transaction in question. An explicit scope for the transaction may be established by calling the rdb_execi() function with an appropriate SQL “set transaction” statement.

<?php
   if (! extension_loaded('rdb')) {
      if (! dl('rdb.exe')) {
         exit;
      }
   }

   rdb_attach('sql$database');

   if (rdb_execi('update employees set city=\'Bolton\' where employee_id=\'00249\'') == -1) {
      printf("%s\n", rdb_error());
      exit;
   }

   rdb_commit();
   rdb_detach();
?>

Note that it is also possible to prepare SQL statements (using the rdb_prepare() function and execute them using rdb_exec()); however the RDB PHP API currently provides no mechanism to specify parameter markers and parameter values. It is intended that this functionality will be included in a subsequent release of the RDB API. From a PHP coding perspective, the lack of such functionality is perhaps of little consequence; however from a performance perspective it can be advantageous to prepare frequently used SQL statements and it is therefore desirable to have this functionality in the future.

Mimer API

The Mimer database (see http://www.mimer.com/) has existed on OpenVMS for almost longer than any other relational database and represents a viable alternative to some of the more commonly known databases for the OpenVMS operating system platform, providing excellent performance and stability with a small footprint and low administrative overhead.

The following table summarizes the functions provided by the Mimer PHP API. As for the RDB API, the set of functions provided is deliberately minimal, providing a basic set of functionality that should be sufficient for most application needs.

Function
Description
mimerdb_connect
Connects to the specified database.
mimerdb_disconnect
Disconnects from the database.
mimerdb_exec
Executes the supplied SQL statement.
mimerdb_fetch
Fetches a row of data using the specified cursor.
mimerdb_close
Closes the specified cursor.
mimerdb_error
Returns a description of the last error.
mimerdb_ncol
Returns the number of columns (values) that would be returned by a fetch for the specified cursor.
mimerdb_commit
Commits the current database transaction.
mimerdb_rollback
Rolls back the current database transaction.
mimerdb_set_readonly
Starts a read-only transaction.
mimerdb_sqlcode
Returns the SQLCODE for the last database operation.

The functions provided by the Mimer API are also very similar in terms of operation to those that are provided by the RDB API; however there are a few differences that are hopefully illustrated by the following example.

In particular, it should be noted that there are no specific functions to declare, open, and close database cursors. The mimerdb_exec() function determines from the provided SQL statement whether the operation will return any rows or not and acts accordingly, returning a value of 0 for success when the statement returns no result set, a positive integer sequence number when there is a result set, and a value of -1 to indicate that an error has occurred. If a positive integer is returned by mimerdb_exec() this implies that a cursor has been created and opened, and the result set may be retrieved using mimerdb_fetch(), which retrieves data into an array a row at a time. Note that the function mimerdb_close() should only be called when mimerdb_exec() returns a positive integer value (when there is a result set).

<?php
   if (! extension_loaded('mimerdb')) {
      if (! dl('mimerdb.exe')) {
         exit;
      }
   }

   $rv = mimerdb_connect('mydb', 'SYSADM', 'password');
   if ($rv != 1) {
      printf("%s\n", mimerdb_error());
      exit;
   }

   mimerdb_set_readonly();

   $ch = mimerdb_exec('select * from metric_groups');
   if ($ch == -1) {
      printf("%s\n", mimerdb_error());
      exit;
   }

   while (($row = mimerdb_fetch($ch)) != NULL) {
      print_r(array_values($row));
   }

   if (mimerdb_sqlcode() != 100) {
      printf("%s\n", mimerdb_error());
      exit;
   }

   mimerdb_close($ch);
   mimerdb_rollback();
   mimerdb_disconnect();
?>

With the exception of the mimerdb_fetch() and mimerdb_error() functions, all functions return an integer value that can be used to determine whether the operation in question was successful or not. A return value of -1 indicates that an error occurred, whereupon the function mimerdb_error() may be used to retrieve a textual description of the error and the function mimerdb_sqlcode() can be used to determine the integer SQL status code associated with the error in question. The function mimerdb_fetch() returns an array upon successful completion or NULL if an error occurred.

As with the Oracle RDB PHP interface, the Mimer PHP interface can only be connected to one database at a time.

Redis API

Redis (https://redis.io/) is a powerful, efficient, and functionally rich open source in-memory data structure store that can be used as used as an in-memory database, cache, and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, and geospatial indexes with radius queries. Redis is commonly used to improve the performance of websites by caching in memory frequently accessed static data; however it is also applicable to a wide range of other use-cases. Redis has recently been ported to OpenVMS, and the inclusion of Redis client functionality in the PHP distribution for OpenVMS provided by VMS Software Inc. makes it readily possible to leverage Redis functionality from OpenVMS-based PHP applications.

As noted above, the PHP Redis extension is provided by the PhpRedis open source project (https://github.com/phpredis/phpredis) and the reader should refer to the PhpRedis documentation  for full details regarding the use of the API; however it is perhaps useful to consider a few simple examples here.

The first example below illustrates the basics of connecting to the Redis cache and verifying that the cache is functioning by sending it a “PING” command, to which the cache should respond with the text “+PONG”. The example also calls the info() method to retrieve data about the cache including software version details and various operational metrics (for a details description of these data the reader should refer to the Redis documentation).

<?php
   if (! extension_loaded('redis')) {
      if (! dl('redis.exe')) {
         exit;
      }
   }

   $redis = new Redis();

   $redis->connect('127.0.0.1', 6379);
   echo print_r($redis->info(), true);
   echo $redis->ping();
   $redis->close();
?>


The next example illustrates adding a simple key/value pair to the cache and the retrieval and deletion of the key from the cache. The code also illustrates the use of exception handling that is provided by the API to catch and manage unexpected error conditions.


<?php
   if (! extension_loaded('redis')) {
      if (! dl('redis.exe')) {
        exit;
      }
    }

    $redis = new Redis();
    try {
        $redis->connect('127.0.0.1', 6379);
        $redis->set('greeting', 'Hello!');
        $reply = $redis->get('greeting');

        if ($reply){
            echo "Reply: '{$reply}'\n";
            if ($redis->delete('greeting')){
                echo "Key deleted\n";
            }
        }else{
            echo "Key not found\n";
        }
    } catch (RedisException $e){
        $exceptionMsg = $e->getMessage();
        echo "Houston we have a problem: {$exceptionMsg}\n";
    }
?>

The following final example illustrates some basic Redis list operations, such as populating a list and retrieving list elements. Once again, refer to https://github.com/phpredis/phpredis for additional information.

<?php
   if (! extension_loaded('redis')) {
      if (! dl('redis.exe')) {
         exit;
      }
   }

   // Connect to Redis on localhost
   $redis = new Redis();
   $redis->connect('127.0.0.1', 6379);
   echo "Connection to server successful";

   // Store list data
   $redis->lpush("language-list", "COBOL");
   $redis->lpush("language-list", "FORTRAN");
   $redis->lpush("language-list", "Pascal");

   // Get the stored data and print it
   $arList = $redis->lrange("language-list", 0 ,5);
   print_r($arList);
   $redis->delete("language-list");
?>


Sunday, April 3, 2016

Extending WSIT using RabbitMQ and AMQP

Introduction

The Web Services Integration Toolkit (WSIT) for HP OpenVMS[1] is an extensible API-level integration technology developed by HP for the OpenVMS operating system that facilitates the integration of new or existing code written in 3GL languages with Java. Based on a supplied XML-based interface definition or by analyzing debug records in object code, tools provided by WSIT generate Java beans and C language wrapper code that can be used to construct an interface to the underlying API, which can then incorporated into essentially any type of Java application. In addition to this powerful code generation facility, the toolkit includes a runtime component that can be used to facilitate and manage IPC-based communication between Java code and the backend API in a client-server fashion (out of-process deployment). The runtime environment provides security features (via hooks into SYSUAF or ACMS), and supports basic tuning functionality to allow applications to be scaled as appropriate. Alternatively, wrapped API code can be called directly from Java code via JNI (in-process deployment).

The choice of deployment model and the configuration options for out-of-process deployment depend upon many factors, such as whether the wrapped code is thread-safe and re-entrant, whether the wrapped code maintains any sort of state, and so on; however WSIT includes functionality to accommodate all of these nuances. WSIT understands all OpenVMS data types and can be used with essentially any OpenVMS 3GL, generating code that uses the correct argument passing mechanisms for the language in question. It can also be used to generate interfaces for ACMS applications.

Figure 1. API-level integration. Existing 3GL business logic is exposed as a set of API calls. WSIT provides capabilities to generate code to facilitate calling these API functions from Java. Existing business logic is not disrupted, but can now be accessed via new interface mechanisms.
All of this sounds very appealing as a means of integrating modern Java-based application environments with what will often be business-critical legacy application code; however there is a catch. For inter-process communication with the out-of-process deployment model, WSIT uses the ICC (Intra Cluster Communication) protocol[2]. While the ICC protocol is very efficient, it is totally proprietary to the OpenVMS operating system. This means that one way or another it is necessary to run at least some Java component on the OpenVMS system in order to communicate with the wrapped code or ACMS application; it is not possible to take the generated Java code and use it on another (non-OpenVMS) platform to communicate with the OpenVMS application environment; nor is it even possible to use the generated Java code on another OpenVMS system that is not part of the cluster on which the wrapped application resides[3]. Whilst there is no particular issue with running Java on OpenVMS, this ICC-dependent model for all but the simplest off situations essentially solves only part of the integration problem (albeit an important and often problematical one), and in order for non-OpenVMS-based Java applications to interact with the wrapped OpenVMS-based application environment it will in addition be necessary to implement some form of integration between the generated Java beans that must reside on OpenVMS and the non-OpenVMS-based Java environment. Possibly this could be done in a platform-agnostic manner by using WSIT to generate a Web Services-based interface that could be used with Apache Tomcat and AXIS2 on OpenVMS; however API-level interfaces generally do not map well to business services (a business service will typically be comprised of some sequence of API operations), or such an approach might not be sufficiently performant, or it may place unacceptable additional load on the OpenVMS environment. Some users have developed solutions using tools such as OpenAdaptor (https://www.openadaptor.org/) with the WSIT-generated Java beans on OpenVMS to integrate with external systems, leveraging the various protocol adapters provided by these tools; however such solutions do not mitigate the potential problem of placing additional (and potentially significant) processing load (CPU and memory) on the existing OpenVMS environment.

As noted above, one of the key features of WSIT is its powerful code generation capabilities. This code generation is template-driven, using Apache Velocity (https://velocity.apache.org/) to generate code using templates written in the Velocity Template Language (VTL). Templates are provided for application wrapper generation, Java bean generation, and for the generation of the various sample Java interface types (POJO, JSP, and AXIS2) that use the generated Java beans.

What is not at all well elucidated in the WSIT documentation is that this powerful code generation facility can be extensively customized[4]: it is possible to modify the existing Velocity templates, and it is possible to introduce new templates. This extensibility opens up a raft of interesting possibilities. Most obviously, it is possible to modify existing templates (or create new templates) to generate code that includes additional or modified functionality, such as the inclusion of logging functions, use of alternative authentication mechanisms, and so on. However, perhaps not so obvious is the fact that it is possible to modify the templates (or create a new set of templates) to generate code that does not rely upon the OpenVMS-proprietary ICC-based WSIT runtime, but instead uses another protocol for inter-process communication. In other words, it is possible to completely bypass the ICC-based runtime and use an alternative non-proprietary cross-platform integration technology for inter-process communication between the wrapped application code and the generated Java beans while still fully leveraging the powerful code generation capabilities and knowledge of OpenVMS data types provided by the WSIT software, whereupon it becomes possible to use the generated Java code to communicate with the wrapped application from any platform that supports a suitable version of the JVM and the integration technology in question.

This post illustrates how the approach described above can be used to extend WSIT to use the Advanced Message Queuing Protocol (AMQP) with RabbitMQ in place of the ICC protocol to provide an efficient, scalable, and fault-tolerant integration solution. Implementation details are described and simple examples are provided to illustrate how the approach can be used to integrate with both 3GL code and ACMS applications. Note that the solution leverages work done previously[5] to develop a generic consumer for RabbitMQ that is able to dynamically load shareable images (shared libraries) containing functions that can be mapped to specific queues and routing keys.

Details

As commented above, WSIT provides two deployment mechanisms for wrapped applications, namely in-process deployment and out-of-process deployment. With the in-process model, the WSIT runtime dynamically loads the wrapped application (a shareable image), and all code therefore runs in the same address space. This model is obviously likely to be more efficient from a performance perspective as it negates any inter-processes communication; however it is not necessarily as scalable or fault-tolerant, and it is likely to be inappropriate if the legacy application being wrapped is not reentrant or thread-safe. Java is inherently multi-threaded and uses threading to achieve scalability. Many legacy OpenVMS applications are not reentrant or thread-safe, and in contrast scalability would often have been achieved by running multiple instances of the program in question (which is a processing model that works very well on OpenVMS). It would be possible to introduce a mutex to single-thread access to the shareable image; however this would likely introduce a major bottleneck. Accordingly, for any serious piece of work, the out-of-process model will invariably need to be used, and indeed for ACMS applications the out-of-process model must be used to avoid potentially catastrophic interactions between POSIX threads and ASTs (Asynchronous System Traps), which are heavily used by the ACMS SI API. For the out-of-process model, the WSIT runtime can be configured to handle non-reentrant wrapped applications by starting a separate instance of the application for each client, and for reentrant applications it can be configured to start some specified minimum number of processes and to scale dynamically by starting additional processes, up to some prescribed threshold, and culling processes (down to the prescribed minimum) when they are no longer required. If a processes crashes, the WSIT runtime will create a new instance, within prescribed limits (maximum number of restarts, delay between restarts, and so on). It should be noted that the approach described in this post to customize WSIT to use protocols other than ICC completely bypasses all of this runtime functionality, and if such functionality is required then it would need to be provided by the middleware for which code is being generated, or it would need to be developed. The purpose of this post is to illustrate how to bypass ICC whilst still leveraging the code generation capabilities of WSIT, and the text does not go into any particular detail with regard to replicating the runtime capabilities of WSIT for any particular middleware technology.

The following diagram illustrates the basic architecture for the WSIT out-of-process model. Based on a supplied interface definition (or by examining object file debug records) WSIT is able to generate a server wrapper and a set of Java beans that map to wrapped API functions. The generated server wrapper (C code) is linked with the legacy application (or ACMS SI API) to create an OpenVMS shareable image that can be called directly via JNI or called indirectly via the ICC-based runtime as described above. The Java beans can be incorporated into any Java application, and WSIT provides facilities for generating several types of sample application (POJO, JSP, AXIS2-based web service) that can serve as a starting point for further development and integration with external client applications. However, as the diagram illustrates, because the generated Java beans are dependent upon the OpenVMS-proprietary WSIT ICC-based runtime, they cannot be used directly by external non-OpenVMS clients (or from OpenVMS clients that are not part of the cluster hosting the wrapped application).

Figure 2. WSIT out-of-process deployment model. The wrapped application is linked as a shareable image that is dynamically loaded by the WSIT runtime. Various options are provided to tune the runtime environment in terms of scalability and fault-tolerance, and to cater for various characteristics of the wrapped application such as reentrancy and multi-threading. Generated Java beans communicate with the wrapped application via the WSIT runtime, and can be used to create OpenVMS-based Java components that provide integration points for use by external clients. Because the generated beans rely on the ICC-based runtime, they cannot be incorporated into and used from external client code.
To overcome this limitation of WSIT and bypass the use of ICC for inter-process communication whilst still leveraging the powerful code generation facilities provided by the product, the supplied Velocity-based code generation templates can be modified, or alternative sets of Velocity templates can be used. The following sections describe how WSIT can be configured to use different templates and illustrate an alternative set of templates that use AMQP instead of ICC for communication between generated Java beans and server wrapper code for out-of-process deployment.

Specifying alternative templates

Velocity-based code generation is performed using the Java application com.hp.wsi.Generator, which resides in the wsi$root:[lib]idl2code.jar archive. For example, for a given interface definition file (in this case math.xml), code generation might be performed as follows:

$ define/job java$classpath wsi$root:[lib]velocity-dep-1_4.jar, -
wsi$root:[tools]idl2code.jar
$ java "com.hp.wsi.Generator" -i math.xml -a math -o [.generated]

The java command instructs the code generation facility to process the math.xml interface definition and to create generated output in the [.generated] directory. The “-a option essentially specifies a name for the application, which must be unique within the set of WSIT applications deployed on a particular OpenVMS node or cluster. If the specified output directory does not exist, it will be created, and if it does already exist then new versions of generated files will be created. Note that in this particular case we did not instruct the code generation facility to generate sample code for one of the supported application types (POJO, JSP, AXIS2-based web service) types, so only the server wrapper and Java bean code is generated; this code will be created in the directories [.generated.mathServer] and [.generated.math], respectively.

By default, the WSIT code generation facility uses Velocity templates that reside in various sub-directories under wsi$root:[tools.templates], and in order to change the structure of the generated code the simplest approach would arguably be to just modify these existing templates; however such an approach is not particularly flexible, as different sets of templates might be required for different projects, and by different developers. Fortunately WSIT provides a better way to allow developers to specify different sets of templates: The code generation facility determines the search path for Velocity templates from the value of the property file.resource.loader.path specified in the velocity.properties file. By default the generator uses the velocity.properties file found in wsi$root:[tools.templates]; however via the “-p” command line option for com.hp.wsi.Generator it is possible to specify an alternative directory for the location of this properties file, thereby allowing developers to use their own properties files and to specify in those properties files the locations of any customized Velocity templates. For example, the following command instructs the generator to look for velocity.properties in /librabbitmq$root/templates/ (note the requirement to use UNIX syntax to specify the path):

$ java "com.hp.wsi.Generator" -i math.xml -a math -o [.generated] -
-p /librabbitmq$root/templates/

An important point to note is that while it is possible to specify alternative locations for template files, the names of the template files used the code generation facility are fixed – it is not possible to add new types of templates or template files with different names from those expected by the tool. This is not a particular restriction, as the set of template files provided by WSIT and the functionality they are intended to provide will be sufficient to meet most integration project requirements.

Using AMQP with RabbitMQ instead of ICC

As mentioned above, the solution described in this post leverages work done previously to develop a generic consumer application for RabbitMQ that is able to dynamically load OpenVMS shareable images containing entry points (functions) that can be mapped to specific queues and routing keys. A key point to note here is that this is analogous to the way that WSIT works: the C code wrapper layer generated by WSIT is compiled and linked into a shareable image that is dynamically loaded by the WSIT runtime, and messages processed by the WSIT runtime are mapped by the generated shareable image wrapper code to the relevant underlying API function. By designing an alternative protocol solution that operates in an analogous manner to the ICC-based WSIT runtime it is possible to significantly reduce the effort required to implement the new AMQP and RabbitMQ-based solution. Whilst for other protocols and middleware technologies it might be necessary to implement something similar to the generic RabbitMQ consumer application, most middleware technologies will be readily amenable to this type of approach, and the level of effort required to implement some sort of generic server layer should typically not be significant. 

As a consequence of this design approach, nearly all of the work required to implement the RabbitMQ-based solution involves changes to the Velocity template file (or the creation of a new template file) associated with the generation of the Java bean code, which must be modified to use RabbitMQ Java client API calls in place of WSIT API calls. Specifically, the interfaceimpl-java.vm template file needs to be modified (or a new file created) as follows:

  •  Import definitions for various RabbitMQ Java API client classes
  •  Define new (or different) private objects for the connection factory, connections, channels, and exchange details
  • Modify the constructor to accept and use additional parameters including username and password, exchange name, and communication timeout[6]
  • Modification of the remove() method to ensure that connections to RabbitMQ are closed cleanly upon normal client termination
  • Modification of the loop that generates code for each method call to use the RabbitMQ Java API RpcClient() class instead of using mShell.invoke() to call into the WSI$JNISHR.EXE shareable image to communicate with the server process via ICC

A complete copy of the modified template file is included below in Appendix 1.

As can be seen, the changes are relatively minimal, and it is likely that changes to the templates would be similarly straightforward for other middleware technologies. Note that changes to other template files are not required for the RabbitMQ-based solution; however such changes may be necessary with other middleware technologies.

General observations and limitations

The solution has been tested using Java clients running on Windows and Linux and the RabbitMQ broker running on Linux and OpenVMS. Transaction rates of several hundred synchronous remote procedure calls (RPC’s) per second easily achieved for single-threaded clients using these configurations, and it is likely that considerable higher overall rates can be achieved by using asynchronous RPC's, where the client does not block waiting for each response. The solution can leverage clustered RabbitMQ brokers as illustrated below (for the case of an ACMS-based application) to provide excellent scalability and fault-tolerance.

Figure 3. Clustered RabbitMQ brokers can be used to enhance scalability and fault tolerance. Note that changes to the code generation template for the Java client will be required in order to take full advantage of the clustered configuration. Optionally, some form of load balancer (such as HAProxy) may be placed between the clients and the RabbitMQ cluster to provide connection balancing and to simplify client re-connection handling.
The solution is presently restricted to use with Java-based clients or other languages that use the Java virtual machine; however it would be possible with some effort to implement .NET-based equivalents of the relevant WSIT Java classes (primarily those classes that understand OpenVMS data types), whereupon it would be possible to use the RabbitMQ .NET client and implement Velocity templates for generation of the required C# code. Similarly it would be possible to support other popular languages, including scripting languages such as Ruby and Python.

Other enhancements might include: 
  • Authenticating and authorizing users by hooking into SYSUAF or via LDAP instead of using RabbitMQ’s default authentication and authorization mechanism
  • Modification of the generic AMQP server component to dynamically scale based on load (within given constraints), support restarts for failed processes, and so on (it is presently necessary to pre-start some static number of server processes)

Summary

This post describes how the template-driven WSIT code generation facilities can be customized to extend WSIT to use the Advanced Message Queuing Protocol (AMQP) with RabbitMQ in place of the OpenVMS-based ICC protocol to provide an efficient, secure, scalable, fault-tolerant, and fully cross-platform integration solution. The solution may be readily adapted to use with other Open Standard integration protocols such as Gearman, ZeroMQ, or STOMP, and the solution can potentially be extended to generate client code for languages other than Java. From a development perspective, generated client code is straightforward to incorporate into existing Java applications, and WSIT’s knowledge of OpenVMS datatypes and programming languages (and ACMS) makes it possible to wrap legacy application code with relative ease. The net result is a scalable and robust integration solution that permits legacy OpenVMS-based applications to interoperate and exchange data with external systems using Open Standards-based integration technologies, allowing OpenVMS users to both preserve and enhance their investment in the OpenVMS platform.




[2] To be somewhat more precise, ICC is the only protocol that can currently be used by the WSIT runtime. WSIT evolved from another HP integration technology for OpenVMS called BridgeWorks, which supported a number of protocols, including ICC, DCE, and COM. BridgeWorks provided a rich cross-platform integration solution that could be used to integrate both Java-based and Windows-based client applications with OpenVMS-based application environments. Much of the WSIT runtime was taken directly from BridgeWorks; however only the ICC protocol was left enabled. It would in theory be possible to enable the other protocols; however a better approach might be to incorporate into the runtime support for more modern and open standards-based protocols such as AMQP, ZeroMQ, STOMP, WebSockets, and so on.
[3] The fact that WSIT uses ICC for inter-process communication means that within an OpenVMS cluster it is possible to run the wrapped application and the generated Java code on different cluster nodes, thereby distributing the processing load and potentially enhancing availability. The fact that these components can be run on separate cluster nodes is not well-documented.
[4] With the latest versions of WSIT (V3.4-1) there an example is provided that illustrates how to add basic custom extensions to the code generation process; however this my no means illustrates the full extent of the customization that is possible.
[6] It may be desirable to further modify the constructor (or to provide additional methods) to allow specification of heartbeat frequency and a list of broker addresses to facilitate use in a clustered broker environment.



Appendix 1 – modified interfaceimpl-java.vm template file

#set( $wsibuffer = 'wsi$buffer' )
#set( $wsiacontext = 'wsi$acontext' )
#set( $wsioutbuffer = 'wsi$outbuffer' )
#set( $server = $application.Server )

package ${application.Name};

// Core java packages
import java.util.StringTokenizer;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.util.Calendar;

// Web Services based Holder classes
import javax.xml.rpc.holders.ByteHolder;
import javax.xml.rpc.holders.DoubleHolder;
import javax.xml.rpc.holders.BigDecimalHolder;
import javax.xml.rpc.holders.BigIntegerHolder;
import javax.xml.rpc.holders.CalendarHolder;
import javax.xml.rpc.holders.FloatHolder;
import javax.xml.rpc.holders.IntHolder;
import javax.xml.rpc.holders.LongHolder;
import javax.xml.rpc.holders.ObjectHolder;
import javax.xml.rpc.holders.ShortHolder;
import javax.xml.rpc.holders.StringHolder;

// WSI based classes
import com.hp.wsi.ServerConfig;
import com.hp.wsi.StringQualify;
import com.hp.wsi.WsiBuffer;
import com.hp.wsi.WsiDecimal;
import com.hp.wsi.WsiArray;
import com.hp.wsi.WsiStructure;
import com.hp.wsi.WsiV2Structure;
import com.hp.wsi.WsiException;
import com.hp.wsi.WsiIpcContext;

import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.RpcClient;

/**
 *  Implementation class for the I${interface.Name} interface.
 */
public class ${interface.Name}Impl implements I${interface.Name} {

    private WsiBuffer wsi$buffer        = new WsiBuffer(512);
    private WsiBuffer wsi$outbuffer     = new WsiBuffer();
    private byte []   wsi$retbuf;

    private int wsi$timeout = -1;
    private String wsi$exchange = "amq.direct";

    private ConnectionFactory cfconn;
    private Connection conn;
    private Channel ch;

#foreach( $routine in $server.Routines)
    private RpcClient wsi$svc_${routine.Methodid} = null;
#end
#foreach( $routine in $server.Acmss)
    private RpcClient wsi$svc_${routine.Methodid} = null;
#end


    /**
     * ${interface.Name}Impl
     *      intstantiates a default ${interface.Name}Impl object.
     *      This constructor is used to setup an in-process environment, where
     *      both the javabean and the server wrapper are in the same process space.
     *
     * @throws com.hp.wsi.WsiException
     *
     */
    public ${interface.Name}Impl(String host, int port, String username, String password, String exchange, int timeout) throws WsiEx
ception
    {
        try {
            cfconn = new ConnectionFactory();
            cfconn.setHost(host);
            cfconn.setPort(port);
            cfconn.setUsername(username);
            cfconn.setPassword(password);
            conn = cfconn.newConnection();
            ch = conn.createChannel();
            wsi$timeout = timeout;
            wsi$exchange = exchange;
        }
        catch (Exception e) {
            System.err.println("Main thread caught exception: " + e);
            e.printStackTrace();
            System.exit(1);
        }
    } // default constructor


    public ${interface.Name}Impl(String uri, String exchange, int timeout)
    {
        try {
            cfconn = new ConnectionFactory();
            cfconn.setUri(uri);
            conn = cfconn.newConnection();
            ch = conn.createChannel();
            wsi$timeout = timeout;
            wsi$exchange = exchange;
        }
        catch (Exception e) {
            System.err.println("Main thread caught exception: " + e);
            e.printStackTrace();
            System.exit(1);
        }
    }


    /**
     * remove()
     *    Cleans up all session context information being
     *    maintained for this JavaBean.   If an IPC is being
     *    used for inter-process communication, this channel
     *    will be closed in the process.   (Note that it is
     *    cleaner to explicitly call remove() in order to
     *    close the session.)
     *
     * @throws com.hp.wsi.WsiException
     *
     */
    public void remove () throws WsiException {
            try {
               if ( null != conn ) {
                   conn.close();
                   conn = null;
               }
            }
            catch( Exception e ) {
               throw new WsiException("WSI - Problem closing connection", e );
            }

    } // remove


#foreach( $method in $interface.Methods )
#set( $routine = $method.LinkedItem )

    /**
     *
     * ${method.Name}
     *      ${routine.Description}
     *
#foreach( $param in $routine.Parameters)
     * @param $param.Name   parameter has type ${param.JBFormalDefinition}( ${param.Typedef} )
#end
#if( $routine.Returnparam )
     * @return              returns type ${routine.Returnparam.JBFormalDefinition}( ${routine.Returnparam.Typedef} )
#end
     *
     * @throws com.hp.wsi.WsiException
     *
     */
#set( $paramformal = "")
#foreach( $param in $routine.Parameters)
#if( $velocityCount == 1 )
#set( $paramformal = "$param.JBFormalDefinition $param.Name" )
#else
#set( $paramformal = "$paramformal, $param.JBFormalDefinition $param.Name" )
#end
#end
    public#if($routine.Returnparam) $dtutility.getJBtype($routine.Returnparam.Datatype)#else void#end ${routine.WebServiceName} ($pa
ramformal)
                    throws WsiException
    {

#if( $routine.Returnparam )
       $dtutility.getJBtype($routine.Returnparam.Datatype) wsi$retval;
#end
       int wsi$valuesize;
       try
       {
           wsi$buffer.resetPosition();
#if( $routine.Argcount > 0 )
           ${wsibuffer}.putParamHeader($routine.Argcount);
#end

#foreach( $param in $routine.Parameters)
#if( $param.Arrayinfo )
#if( $param.Passingmechanism < 3 )
#set( $byref = "true" )
#else
#set( $byref = "false" )
#end
#if( $param.Nullterminated )
#set( $nullterm = 1 )
#else
#set( $nullterm = 0 )
#end
           ${wsibuffer}.putParamEntry( $param.Datatype, $param.ClassType,#if( $param.DynStringParam ) 0#else $param.Size#end, $param
.InverseScale );
           WsiArray wsi$${param.Name} = new WsiArray(${param.Name}#if( $param.Usage > 1 ).value#end, wsi$buffer, $param.Datatype,
                                    $param.ClassType, $param.Size, $param.InverseScale, $param.Arrayinfo.Rowbycolumn, $nullterm, "#i
f( $param.Passingmechanism < 3 )${param.Arrayinfo.DimString}#else${param.Arrayinfo.Dimcount}#end", $byref, true);
           wsi$${param.Name}.exportArray(wsi$buffer);
#else
           ${wsibuffer}.putParamEntry( $param.Datatype, $param.ClassType,#if( $param.Size == 0 ) ${param.Name}#if( $param.Usage > 1
).value#end.length()#else $param.Size#end, $param.InverseScale );
           ${wsibuffer}.put(#if( $param.Structure )(WsiV2Structure)#end${param.Name}#if( $param.Usage > 1 ).value#end, ${param.Datat
ype}${param.OptionalPutArgs});
#end
#end
           if (wsi$svc_${routine.Methodid} == null) {
              wsi$svc_${routine.Methodid} = new RpcClient(ch, wsi$exchange, "${method.Name}", wsi$timeout);
           }
           ${wsioutbuffer}.setBuffer(wsi$svc_${routine.Methodid}.primitiveCall(${wsibuffer}.getBuffer()));

#if( $routine.HasOutputBuffer )
           wsi$outbuffer.getParamHeader(0);
#end
#if( $routine.Returnparam )
           wsi$valuesize = ${wsioutbuffer}.getParamEntry();
           wsi$retval = ${wsioutbuffer}.get${routine.Returnparam.JBRetType}( $routine.Returnparam.Datatype );
#end

#foreach( $param in $routine.Parameters)
#if( $param.Usage > 1 )
           wsi$valuesize = wsi$outbuffer.getParamEntry();
#if( $param.Arrayinfo )
#if( $param.ResizableArray )
           wsi$${param.Name}.importNewByteArray(wsi$outbuffer, wsi$valuesize);
#else
           wsi$${param.Name}.importArray(wsi$outbuffer);
#end
           ${param.Name}.value = wsi$${param.Name}.getValue();
#else
#if( $param.Structure )
           ${wsioutbuffer}.getStructure((WsiV2Structure)${param.Name}.value);
#else
           ${param.Name}.value = ${wsioutbuffer}.get${param.JBRetType}( ${param.Datatype}${param.OptionalGetArgs});
#end
#end
#end
#end
       }
       catch (WsiException we)
       {
           throw we;
       }
       catch (Exception ex)
       {
           throw new WsiException(ex);
       }

       return#if( $routine.Returnparam ) wsi$retval#end;

    } // ${method.Name}
#end


#foreach( $method in $interface.AcmsMethods )
#set( $routine = $method.LinkedItem )
    /**
     *
     * ${method.Name}
     *      ${routine.Description}
     *
#foreach( $param in $routine.Parameters)
     * @param $param.Name    parameter has type ${param.JBFormalDefinition}
#end
     *
     * @throws com.hp.wsi.WsiException
     *
     */
#set( $paramformal = "")
#foreach( $param in $routine.Parameters)
#if( $velocityCount == 1 )
#set( $paramformal = "$param.JBFormalDefinition $param.Name" )
#else
#set( $paramformal = "$paramformal, $param.JBFormalDefinition $param.Name" )
#end
#end
    public void ${routine.WebServiceName} ($paramformal)
                    throws WsiException
    {

       int wsi$valuesize;
       try
       {
           ${wsibuffer}.resetPosition();
           ${wsibuffer}.putParamHeader(${routine.Argcount}+1);

           ${wsibuffer}.putParamEntry(WsiBuffer.DTYPE_T, WsiBuffer.CLASS_S, 0, 0);

#foreach( $param in $routine.Parameters)
           ${wsibuffer}.putParamEntry( $param.Datatype, $param.ClassType,#if( $param.Size == 0 ) ${param.Name}#if( $param.Usage > 1
).value#end.length()#else $param.Size#end, $param.InverseScale );
#if( $param.Arrayinfo )
#if( $param.Usage > 1 )
#set( $byref = "true" )
#else
#set( $byref = "false" )
#end
#if( $param.Nullterminated )
#set( $nullterm = 1 )
#else
#set( $nullterm = 0 )
#end
           WsiArray wsi$${param.Name} = new WsiArray(${param.Name}#if( $param.Usage > 1 ).value#end, wsi$buffer, $param.Datatype,
                                    $param.ClassType, $param.Size, $param.Scale, $param.Arrayinfo.Rowbycolumn, $nullterm, "#if( $par
am.Passingmechanism < 3 )${param.Arrayinfo.DimString}#else${param.Arrayinfo.Dimcount}#end", $byref, true);
           wsi$${param.Name}.exportArray(wsi$buffer);
#else
           ${wsibuffer}.put(#if( $param.Structure )(WsiV2Structure)#end${param.Name}#if( $param.Usage > 1 ).value#end, ${param.Datat
ype}${param.OptionalPutArgs});
#end
#end
           if (wsi$svc_${routine.Methodid} == null) {
              wsi$svc_${routine.Methodid} = new RpcClient(ch, wsi$exchange, "${method.Name}", wsi$timeout);
           }
           ${wsioutbuffer}.setBuffer(wsi$svc_${routine.Methodid}.primitiveCall(${wsibuffer}.getBuffer()));

           ${wsioutbuffer}.getParamHeader(0);

           wsi$valuesize = wsi$outbuffer.getParamEntry();
           String wsi$extstatus = ${wsioutbuffer}.getString(WsiBuffer.DTYPE_T,wsi$valuesize);

#foreach( $param in $routine.Parameters)
#if( $param.Usage > 1 )
           wsi$valuesize = wsi$outbuffer.getParamEntry();
#if( $param.Arrayinfo )
#if( $param.ResizableArray )
           wsi$${param.Name}.importNewByteArray(wsi$outbuffer, wsi$valuesize);
#else
           wsi$${param.Name}.importArray(wsi$outbuffer);
#end
           ${param.Name}.value = wsi$${param.Name}.getValue();
#else
#if( $param.Structure )
           ${wsioutbuffer}.getStructure((WsiV2Structure)${param.Name}.value);
#else
           ${param.Name}.value = ${wsioutbuffer}.get${param.JBRetType}( ${param.Datatype}${param.OptionalGetArgs});
#end
#end
#end
#end
       }
       catch (WsiException we)
       {
           throw we;
       }
       catch (Exception ex)
       {
           throw new WsiException(ex);
       }

       return;

    } // ${method.Name}
#end
} // class