PIPE IO Technical Bulletin

GT.M V5.3-002 Release

Legal Notice

December 15, 2008

Revision History
Revision 1.0December 15, 2008



                                            GT.M Group

                                            Fidelity National Information Services, Inc.

                                            2 West Liberty Boulevard, Suite 300

                                            Malvern, PA 19355

                                            United States of America

                                        



                                            GT.M Support: +1 (610) 578-4226

                                            Switchboard: +1 (610) 296-8877

                                            Fax: +1 (484) 595-5101

                                            Website: http://fis-gtm.com

                                            Email: gtmsupport@fnis.com

                                        

Table of Contents

Document Conventions
Bulletin Overview
Status Variables
$ZA
$DEVICE
$KEY
Using PIPE devices
Modes of PIPE device operation
READ Command
READ with no timeout
READ with a timeout
READ fixed-length
WRITE Command
WRITE /EOF
CLOSE Command
PIPE Device Examples
PIPE Deviceparameter Summary
OPEN Deviceparameters
COMMAND Applies to: PIPE
[NO]FIXED Applies to: SD FIFO PIPE
INDEPENDENT Applies to: PIPE
PARSE Applies to: PIPE
[NO]READONLY Applies to: FIFO PIPE SD
RECORDSIZE=intexpr Applies to: SD FIFO PIPE
SHELL Applies to: PIPE
STDERR Applies to: PIPE
VARIABLE Applies to: SD FIFO PIPE
[Z]WIDTH=intexpr Applies to: TRM SOC NULL SD FIFO PIPE
[Z][NO]WRAP Applies to: SD FIFO SOC PIPE
[NO]WRITEONLY Applies to: PIPE

Return to top

Document Conventions

In this document the lower case “pipe” refers to a UNIX pipe and the upper case “PIPE” to the GT.M device.

Return to top

Bulletin Overview

The V5.3-003 release extends the capabilities of the I/O status variables with support for a new PIPE device on the UNIX platforms as a mnemonic space device. An OPEN of the device starts a sub-process. Data written to the device by the M program is available to the process on its STDIN. The M program can read the STDOUT and STDERR of the sub-process. This facilitates output only applications, such as printing directly from a GT.M program to an lp command; input only applications, such as reading the output of a command such as ps; and co-processing applications, such as using iconv to convert data from one encoding to another. (C9H05-002859)

Return to top

Status Variables

GT.M provides several I/O status variables that convey information about the status of individual operations.

For PIPE:

0 Indicates normal termination or time out when using READ x:n, where n >0

9 Indicates failure of a READ x or READ x:n, where n>0

9 Indicates failure of a WRITE where the pipe is full and the WRITE would block

For PIPE:

0 indicates for READ with a zero (0) timeout that available data has been read.

"1,Resource temporarily unavailable" indicates no input available for a READ with a zero (0) timeout.

“1,<error signature>” indicates a read error

0 indicates for a WRITE that it was successful

"1,Resource temporarily unavailable" indicates a failure of a WRITE where the pipe is full and the WRITE would block. This condition also causes an exception.

“1,<error signature>” indicates a write error

For PIPE:

Contains the UNIX process id of the created process shell which executes the command connected to the PIPE device.

Return to top

Using PIPE devices

A PIPE is akin to a FIFO device. Both FIFO and PIPE map GT.M devices to UNIX pipes, the conceptual difference being that whereas a FIFO device specifies a named pipe, but does not specify the process on the other end of the pipe, a PIPE device specifies a process to communicate with, but the pipes are unnamed. Specifically, an OPEN of a PIPE creates a subprocess with which the GT.M process communicates.

A PIPE device is specified with a "PIPE" value for mnemonicspace on an OPEN command.

[Note]

GT.M ignores the mnemonicspace specification on an OPEN of a previously OPEN device and leaves the existing device with its original characteristics.

Return to top

Modes of PIPE device operation

The OPEN command for a PIPE provides a number of variations in the use of UNIX pipes shown below as Examples 1-4.

Example 1

       
            SET p="Printer"
            OPEN p:(COMMAND="lp":WRITEONLY)::"PIPE" 
        

Example 1

Example 1 shows the use of a PIPE device to spool data to the default printer by spooling to the lp command, opened via the default shell (the shell specified by the SHELL environment variable, and the shell used to start GT.M if SHELL is unspecified). The WRITEONLY device parameter specifies that the GT.M process will not read data back from the lpr command.

Example 2

            Set p="MyProcs"
            OPEN p:(COMMAND="ps -ef|grep $USER":READONLY)::"PIPE" 
        

Example 2

Example 2 shows the use of a PIPE device to identify processes belonging to the current userid. The READONLY device parameter specifies that the GT.M process will only read the output of the pipe, and will not provide it with any input. This example illustrates the fact that the command can be any shell command, can include environment variables and pipes within the command.

[Note]

On your UNIX, flags to the ps command may differ from this example.

Example 3:

            Set p="Convert"
            OPEN p:(SHELL="/bin/csh":COMMAND="iconv -f ISO_8859-1 -t WINDOWS-1252")::"PIPE"  
        

Example 3

Example 3 shows the use of a process to whose input the GT.M process writes to and whose output the GT.M process reads back in, in this example convering data from an ISO 8859-1 encoding to the Windows 1252 encoding. This example also shows the use of a different shell from the default. If the OPEN deviceparameters don't specify a SHELL, the PIPE device uses the shell specified by the environment variable SHELL; if it does not find a definition for SHELL, the device uses the system default /bin/sh.

Example 4:

            SET p="Files"
            SET e="Errors"
            OPEN p:(COMMAND="find /var/log -type d -print":READONLY:STDERR=e)::"PIPE"  
        

Example 4

Example 4, GT.M uses the standard system utility find to obtain a list of subdirectories of /var/log, which are read back via the device with handle "Files" with any errors (e.g, “Permission denied” messages for sub-directories that the find command cannot process) read back via the device with handle "Errors".

Return to top

READ Command

Return to top

READ with no timeout

A READ with no timeout reads whatever data is available to be read; if there is no data to be read, the process hangs until some data becomes available.

Return to top

READ with a timeout

A READ with a timeout reads whatever data is available to be read, and returns; if there is no data to be read, the process waits for a maximum of the timeout period, an integer number of seconds, for data to become available (if the timeout is zero, it returns immediately, whether or not any data was read). If the READ returns before the timeout expires $Test, it set $TEST to TRUE(1); if the timeout expires, it sets $TEST to FALSE (0). When the READ command does not specify a timeout, it does not change $TEST.

Return to top

READ fixed-length

READ specifying a maximum length reads (for example, READ X#10 for ten characters) until either the PIPE has supplied the specified number of characters, or a terminating delimiter. A fixed-length READ may include a timeout.

Return to top

WRITE Command

The PIPE device does non-blocking writes. If a process tries to WRITE to a full PIPE and the WRITE would block the device implicitly tries to complete the operation up to10 times. If the implicit retries don't succeed (remain blocked), the WRITE sets $DEVICE to "1,Resource temporarily unavailable", $ZA to 9, and triggers an error. If the GT.M process has defined an EXCEPTION, $ETRAP or $ZTRAP, it may choose to retry the WRITE after some action or delay might remove data from the pipe.

Return to top

WRITE /EOF

WRITE /EOF to a PIPE closes the underlying UNIX pipe to force the created process to flush data but does not CLOSE the PIPE device. When the GT.M process finishes READs from the PIPE device, it still should explicitly CLOSE the PIPE device. When the PIPE device's created process (“tr”, for example) buffers output written to a UNIX pipe, the GT.M process must do WRITE/EOF after its final WRITE to the PIPE device. To avoid an indefinite hang, the GT.M process must READ with timeout (typically 0) for any READs on the device prior to the WRITE /EOF. After the GT.M process does a WRITE /EOF on the PIPE device closing stdin and presenting EOF to the created process, forcing the created process to flush any buffered data to the UNIX pipe acting as input to the PIPE device, it no longer has to use timed READ, although it may.

Return to top

CLOSE Command

The CLOSE of a PIPE device prevents all subsequent access to the pipes associated with the device. Unless the OPEN that created the device specified INDEPENDENT, the process terminates. Note that any subsequent attempt by the created process to read from its stdin (which would be a closed pipe) returns an EOF and typical UNIX behavior would be to terminate on such an event.

Return to top

PIPE Device Examples

The following examples show the use of deviceparameters and status variables with PIPE devices.

Example :

            SET p1="test1" 
            OPEN p1:(shell="/bin/sh":comm="cat")::"PIPE"
            FOR i=1:1:10 DO
            . USE p1 
            . WRITE i,":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ",! 
            . READ x
            . USE $P 
            . WRITE x,! 
            CLOSE p1
            QUIT         
        

This WRITEs 10 lines of output to the cat command and reads the cat output back into the local variable x. The GT.M process WRITEs each line READ from the PIPE to the principal device. This example works since “cat” is not a buffering command. The example above would not work for a command such as tr that buffers its input.

Example :

            SET p1="test1"
            OPEN  p1:(SHELL="/bin/sh":COMMAND="tr -d e")::"PIPE"
            FOR i=1:1:1000 DO
            . USE p1
            . WRITE i,":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ",!
            . READ x:0
            . IF '+$DEVICE USE $P WRITE x,! 
            USE p1
            WRITE /EOF
            FOR  READ x QUIT:$ZEOF  USE $P WRITE x,! USE p1
            CLOSE p1
            QUIT   
        

This shows the use of “tr” (a buffering command) in the created process for the PIPE device. To see the buffering effect the GT.M process WRITEs 1000 lines to the PIPE device. Different operating systems may have different buffer sizes. Notice the use of the r x:0 and the check on $DEVICE in the loop. If $DEVICE is 0, WRITE x writes the data read to the principal device. No actual READs complete, however, until “tr” reaches its buffer size and writes to its stdout. The final few lines remain buffered by “tr” after the process finishes the first loop. The GT.M process then issues a WRITE /EOF to the PIPE causing “tr” to flush its buffered lines. In the final for loop the GT.M process uses the simple form of READ x from the PIPE followed by a WRITE of each line to the principal device until $zeof becomes TRUE.

Example :

 
            SET a="test"
            OPEN a:(COMMMAND="ntestin":INDEPENDENT)::"PIPE"
            USE a
            SET key=$KEY
            WRITE "Show ntestin still running after CLOSE of a",!
            WRITE "The parent process of 1 shows the parent shell has exited after CLOSE of a"
            READ line1,line2
            USE $p
            WRITE !,line1,!,line2,!,!
            SET k="ps -ef | grep -v grep | grep -v sh | grep -w '"_key_"' | awk '{print $2}'"
            SET b="getpid"
            OPEN b:(COMMMAND=k:READONLY)::"PIPE"
            USE b
            READ pid
            CLOSE a
            CLOSE b
            SET k2="ps -ef | grep -v grep | grep -v sh | grep -w '"_pid_"'"
            SET c="psout"
            OPEN c:(COMMMAND=k2:WRITEONLY)::"PIPE"
            CLOSE c
            QUIT  
        

This demonstrates that the created process “ntestin” keeps running as an INDEPENDENT process after the GT.M process CLOSEs the pipe. This GT.M process uses another PIPE device to return the process id of “ntestin” and READ it into “pid” so that it may be killed by this or another process, should that be appropriate.

Example :

            SET p1="test1"
            SET a=0
            OPEN p1:(SHELL="/bin/sh":COMMAND="cat":EXCEPTION="GOTO cont1")::"PIPE"
            SET c=":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz"
            FOR i=1:1:10000  DO
            . USE p1
            . WRITE i_c,!
            . USE $p WRITE i,!
            USE p1
            WRITE /EOF
            FOR  READ x QUIT:$ZEOF  USE $p WRITE x,! USE p1
            CLOSE p1
            QUIT
            cont1
            IF $ZEOF QUIT
            IF a=0 SET a=i/2
            SET z=$ZA
            ; use $device to make sure ztrap is caused by blocked write to pipe 
            SET d=$DEVICE 
            IF "1,Resource temporarily unavailable"=d DO 
            . USE $p
            . WRITE "pipe full, i= ",i," $ZA = ",z,!
            . SET i=i-1 
            . USE p1
            . FOR j=1:1:a  READ x USE $p WRITE j,"-",x,! USE p1
            QUIT  
        

This demonstrates WRITEs to a PIPE device with blocking. The WRITE loop has no READ to force the input pipe to fill up which blocks the cat output, causing cat to stop reading its input, letting the pipe acting as input on the PIPE device to fill up and creating the blocked condition. When the process takes the $ZTRAP to cont1 it tests $DEVICE to determine if the trap is caused by the full pipe. If so, it uses the for loop to read half the number of lines output by the main loop. It decrements “i” and returns to the original WRITE loop to retry the failed line and continue with the WRITEs to the pipe. Depending upon the configuration of the environment, it may trap several times before processing all lines.

Return to top

PIPE Deviceparameter Summary

DEVICE PARAMETER

CMD

DESCRIPTION

[NO]FIXED

O

Controls whether records have fixed length

RECORDSIZE=intexpr

O

Specifies the maximum record size.

VARIABLE

O

Controls whether records have variable length.

WIDTH=intexpr

O

Sets the device's logical record size and enables WRAP.

[NO]WRAP

O/U

Controls the handling of records longer than the device width.

Table 1. Pipe Format Deviceparameters

COMMAND=string

o

Specifies the command string to execut in a created process for the PIPE device. GT.M uses the default searching mechanism of the UNIX shell for creating the process and initiating its command(s).

SHELL=string

o

Specifies the path to a shell to be used instead of the default shell

STDERR=string

o

Specifies a device handle for a return pipe to which the created process writes any standard error. The GT.M process can “USE” ,“READ”, and “CLOSE” it, but cannot WRITE to it. When the GT.M process CLOSEs the PIPE device the PIPE device CLOSEs STDERR, if still OPEN.

WRITEONLY

o

Specifies that the GT.M process may only WRITE to the created process via the PIPE device.

READONLY

o

Specifies that the GT.M process may only READ from the created process via the PIPE device. It returns both stdout and stderr except when STDERR specifies a separate stderr.

PARSE

o

Specifies that GT.M parse the COMMAND and issue an OPEN exception for any invalid command.

INDEPENDENT

o

Specifies that the created process continues to execute after the PIPE device is CLOSEd.

Table 2. Pipe Access Deviceparameters

Return to top

OPEN Deviceparameters

Return to top

COMMAND Applies to: PIPE

The COMMAND deviceparameter value specifies the invocation the newly created shell process performs. An invalid command value triggers an error in the new process, not the process issuing the OPEN. This can make diagnosis difficult – see the PARSE deviceparameter for potential assistance.

Return to top

[NO]FIXED Applies to: SD FIFO PIPE

Selects a fixed record length format for the device. FIXED does not specify the actual length of a record. Use RECORDSIZE to specify the record length.

NOFIXED specifies a variable length record format for device. NOFIXED is a synonym for VARIABLE. FIXED is incompatible with STREAM and VARIABLE.

By default, records have VARIABLE length.

Return to top

INDEPENDENT Applies to: PIPE

The INDEPENDENT deviceparameter specifies the newly created process won't be terminated by the CLOSE of the device. By default, CLOSE terminates the process associated with the PIPE device.

Return to top

PARSE Applies to: PIPE

The PARSE deviceparameter invokes preliminary validation of the COMMAND value. When debugging, PARSE provides more accessible diagnosis for COMMAND values. By default, OPEN does not validate command values before passing them to the newly created process. PARSE has certain limitations, which may, or may not map to, those of the shell.

  • PARSE searches for the command in $PATH and in $gtm_dist and triggers an error if it isn't found.

  • PARSE does not resolve aliases, so they trigger an error.

  • PARSE does not resolve environment variables, except $gtm_dist (as mentioned above), so they trigger an error.

  • PARSE does not recognize built-in commands other than nohup and cd unless $PATH or $gtm_dist contain a version with the same name (as the built-in). In the case of nohup, PARSE looks for the next token in $PATH and $gtm_dist.  "When PARSE encounters cd it ignores what follows until the next "|" token (if one appears later in the COMMAND value).

  • PARSE rejects parentheses around commands.

    • The following example fails:

                                   OPEN p:(COMM="(cd; pwd)":WRITEONLY)::"PIPE"   
                                  
    • Which could be specified without a PARSE error as:

                                   OPEN p:(COMM="cd; pwd":WRITEONLY)::"pipe"   
                                  
    • This restriction does not include parentheses embedded in character strings as in:

                                    OPEN p:(COMM="echo ""(test)""":WRITEONLY)::"pipe"  
                                  
    • Or parameters to a command as in:

                                   OPEN p:(COMM="tr -d '()'":WRITEONLY)::"PIPE"  
                                  
    • The following are examples of valid OPEN commands using PARSE:

                                  OPEN a:(COMM="tr e j | echoback":STDERR=e:exception="g BADOPEN":PARSE)::"PIPE"
                                  OPEN a:(SHELL="/usr/local/bin/tcsh":COMM="/bin/cat |& nl":PARSE)::"PIPE"
                                  OPEN a:(COMM="mupip integ -file mumps.dat":PARSE)::"PIPE"
                                  OPEN a:(COMM="$gtm_dist/mupip integ -file mumps.dat":PARSE)::"PIPE"
                                  OPEN a:(COMM="nohup cat":PARSE)::"PIPE"   
                                  

Return to top

[NO]READONLY Applies to: FIFO PIPE SD

OPENs a device for reading only (READONLY) or reading and writing (NOREADONLY).

[Note]

To open a sequential file using the READONLY parameter, the file must exist on the disk. If it does not, GT.M issues a run-time error.

When GT.M encounters a WRITE directed to a device OPENed READONLY, GT.M issues a run-time error.

By default, OPEN accesses the device NOREADONLY (read-write).

For PIPE devices, also see the WRITEONLY deviceparameter.

Return to top

RECORDSIZE=intexpr Applies to: SD FIFO PIPE

Overrides the default record size for a device and specifies the new maximum record size in bytes.

Return to top

SHELL Applies to: PIPE

The SHELL deviceparameter specifies the shell for the new process. By default the newly created process uses the shell specified by the $SHELL environment variable, otherwise, if $SHELL is undefined the process uses /bin/sh.

Return to top

STDERR Applies to: PIPE

The STDERR deviceparameter specifies that the stderr output from the created process goes to a pipe with the name of the STDERR value. This pipe acts as a restricted device that can appears as the argument to USE, READ and CLOSE commands. It is implicitly READONLY and an attempt to WRITE to it triggers an error. If it has not previously acted as the argument to an explicit CLOSE command, the CLOSE of the PIPE device implicitly closes the the STDERR “device.”

Return to top

VARIABLE Applies to: SD FIFO PIPE

Specifies the VARIABLE record length format for the device.

By default, records have VARIABLE length.

Return to top

[Z]WIDTH=intexpr Applies to: TRM SOC NULL SD FIFO PIPE

Sets the device's logical record size and enables WRAP.

NOWRAP and WIDTH supersede each other. When WIDTH and NOWRAP appear together on the same USE command, the final one controls the device behavior. For a terminal, WIDTH=0 is equivalent to WIDTH=n:NOWRAP, where n is the default length of a logical record on that terminal.

Terminals inherit their default WIDTH in GT.M from the invoking shell environment. The default WIDTH for null and socket device is 255. For other devices WIDTH defaults to RECORDSIZE.

Return to top

[Z][NO]WRAP Applies to: SD FIFO SOC PIPE

Enables or disables automatic record termination. When the current record size ($X) reaches the maximum WIDTH and the device has WRAP enabled, GT.M starts a new record, as if the routine had issued a WRITE ! command.

NOWRAP causes GT.M to require a WRITE ! to terminate the record. NOWRAP allows $X to become greater than the device WIDTH for terminals and null devices.

The combination of STREAM and NOWRAP options on disk files allows you to write a record of arbitrary length. Without the STREAM option, the WRAP option determines the action taken when the record length exceeds the device WIDTH. NOWRAP causes GT.M to truncate the record, while WRAP causes GT.M to insert a format control character.

By default, records WRAP.

Return to top

[NO]WRITEONLY Applies to: PIPE

The WRITEONLY deviceparameter specifies that the PIPE acts only to send its output to the created process. Any attempt to READ from such a PIPE triggers an error. Note that when you open a PIPE with both STDERR and WRITEONLY you can still READ from the STDERR “device.”

For more information, see the GT.M web site.