Chapter 11: Advanced Tools

The QEF product's bin directory contains 214 different programs -- see Appendix D: Q-Tree Command Summaries. This chapter presents some of the more useful tools that are not directly associated with software construction, but are very useful in controlling and monitoring builds across the network.

The QEF subset is actually only about a third of the full Q-Tree package. For the most part the QEF tools are associated with software construction or the software process. Some tools have been added in response to user requests to serve various esoteric purposes in a way that behaves consistently across all platforms. The interested reader should review the complete list of QEF tools given in Appendix D.

This chapter discusses the use of programs often used to launch and monitor build processes. It also discusses two very useful programs ct and pa, and the oldest Q-Tree program of all, the program that some users refer to as "Our friend rpl".

11.1) Qremote, Cush, Lash, Josh, & Envset

In 1975, Tom Duff, Academy Award winner and inventor of Duff's device, said that the most interesting programs were those that wrote or ran other programs. He should find this section interesting indeed as qremote, cush, lash, and josh are all programs that run other programs, often each other, to facilitate launching programs with the correct environment and controls. envset does not fit into this class directly, except that its semantics are duplicated internally by the other programs.

The reader should read the brief descriptions in Appendix D for these programs. Briefly,

cush could (could not) use shell interface
cush runs its argument command in the specified directory with I/O redirection, environmental settings (possibly using a envset environment set) specified via options. Note that cush is invoked by qremote to actually run the latter program's argument command.

envset output environment setting commands
envset is the direct interface to the database specified in the user's /.qtree/envset.cf and the system <qtree>/{data,lib}/envset.cf which specifies sets of the setting and unsetting of environment variables, shell variables, aliases, and functions. envset sets are used by cush and qremote. A tree's root.vrs file's QremoteEnv setting is actually the name of an envset set to be used when doing a qremote in the tree.

josh a job shell
josh interprets command sets selected by argument names from the user's /.qtree/josh database, which becomes a convenient place to store commonly used commands that do not merit file of their own.

lash long argument list shell interface
lash runs its argument command with substitutions in that command from either lash's argument list or the standard input.

qremote run a command on a remote host (if necessary)
qremote runs rsh or some equivalent to execute a command on a remote or the local system. qremote can accept a qdsrv path number argument that specifies in what directory the command is to be run. Furthermore, by default it uses the BuildHost setting in the tree's root.vrs file as the name of the host on which to run.

Note: qremote needs to know the location of the Q-Tree on the remote host. This is done by sending a message to qhost which indicates the host is up and returns the Q-Tree path and the user's login directory See Chapter 6.5: The Qhost Server.

These programs are presented here as some build masters use them in combination to manage the builds of many products across many platforms using simple one line commands. The Cook's Tour: Envset, Cush, ... page illustrates their use to accomplish this in some detail.

Briefly, the user adds commands to his/her /.qtree/josh file similar to the following:

    paths = 585 588 613
    goproj6: ! run detached builds for proj6 
	lash -o 'qremote -g -%1 qef -d @D @*' @(paths)

When the user invokes the command:

    % josh -Ddir goproj6 - other args

josh will select from its database the goproj6 set, replace the "@D" by the D argument, the "@*" by other args, and the "@(paths)" by "585 588 613", and then run the resulting command:

    lash -o 'qremote -g -%1 qef -d dir other args' 585 588 613

lash will then run its argument command once for each argument, replacing the "%1" by the argument, thus lash executes:

    % qremote -g -585 qef -d dir other args
    % qremote -g -588 qef -d dir other args
    % qremote -g -613 qef -d dir other args

Considering the first of the above, qremote will chdir to the specified path (i.e., 585), and extract that directory's settings for BuildHost and QremoteEnv. It then retrieves the Q-Tree for the specified host from qdsrv, and given the host is not the current host, executes:

    % rsh host host-qtree/bin/cush -Q -E qremote-env
	-r pathdir go -d qef -d dir other args

If the command is to be run on the local host, the rsh host is not included.

Note:

The rsh used is whatever the command should be for the current host or a shell command called host in the user's /.qtree/qremote, or the system's <qtree>/data/qremote directories.

Note:

The Q flag specifies that cush should set $QTREE to host-qtree.

Note:

qremote translates the pathdir argument to the form used on the target host as might be necessary if the current host is a windows system whereas the target host is unix.

Note:

The go command will detach its argument command after redirecting the standard outputs to a diagnostic file as described in the next section.

So, josh begets lash, and lash begets qremotes, and qremotes beget rshs, and rshs beget cushs, and cushs beget gos, and gos beget qefs and qefs beget a multitude of other processes (often in parallel) and you can watch it all happening with the various go-file monitors.

I wonder if Tom finds that interesting.
Note: The other users on your network might find it offensive and you might consider added m1 to the qef arguments to suppress parallel execution.

11.2) Distributed Processing

This section duplicates in part the information presented in Page 19 of the Cook's Tour, thus it is limited to the technical aspects of distributed processing.

To indicate that a tool is to be run on another host or on one of a set of hosts, the qvrs variable _Servers_[service] is set to the list of hosts that are used to run _T_service tools, as in:

    set _Servers_[service] host1 host2 host3

This is all that needs to be done to cause service to be run on one of the named hosts. This feature can be used to distribute expensive processes such a C compiles to available cycle servers or to run a locally unavailable service on a host that can provide that service.

The balance of this section discusses how the magic happens.

The qef preprocessor qefpp will modify any _T_service, for which _Servers_[service] is defined, to insert the string "{!service!}" before the current value or "service" if _T_service was previously unset.
Note: This magic string is used, as versus "qremote ..." so that mimk can detect the string and ignore it in recipe comparisons.

mimk will detect the string "{!service!}" and replace it prior to command execution by "qremote DRSservice", hence a command "_T_service ..." will be executed as:

    qremote DRSservice service ...

Note:

The actual program service could be something other than service if _T_service was set prior to qefpp processing.

qremote, when invoked with the Sservice flag, retrieves the value of _Servers_[service]. It then sends messages to the qhost server running on each of the named hosts. A running qhost returns the pathname of the Q-Tree and the user's login directory on the host. These values are required for the command that qremote will eventually execute.

To choose from the list of active hosts, qremote sends the list as the argument to a "cycle" message to qdsrv, which will return the host in the list that was least recently used. See Chapter 6: Qdsrv and Qhost: The QEF Servers.

The qremote R flag specifies that qremote should remove any appearance of "qremote -DRS..." arguments in the argument command. This is necessary in situations where multiple tools within a recipe have _Servers_[service] settings such as might happen for the recipe:

    _T_purify _T_cc ...

One final note with respect to distributed processing. rsh has had an unfortunate deficiency since its creation: rsh does not exit with the exit status of the argument command. This creates problems for make-like processes such as mimk which uses the exit-status of its recipes to determine if the session should be terminated. To deal with this deficiency, qremote uses the qdsrv exstat* messages. It uses the "exstatid" message to retrieve the index of an exit-status slot. The returned value is passed the argument to a cush X flag. cush in turn will send the exit status of its argument command in an "exstatset" message to qdsrv, which qremote will retrieve using an "exstatget" message to qdsrv.

This is probably more information than you require, but may be of use to system administrators.

11.3) Go, G, Goes, Gomonitor, and the Build Monitor

The program go simply runs its argument command, after redirecting the new process' standard output and standard error to a go-file, described later. The original go process exits, thereby returning control to the executing program (e.g., the shell). A forked child go process actually executes the command and waits for its completion. To track go'd commands, lines are added to the user's /.qtree/goes/host/gomsgs file. The lines contain:

    pid    command  host key directory date&time

for example:

    00662     ls  matt - /m/guide/o/guide 2000/05/04 14:31:53
    00723     josh -d testing - these arguments are ignored
	 matt a /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.90.29
	/include 2000/05/04 14:42:56

Once the command has terminated, another line is added to the same file with the exit status and time as in:

00723:0   josh -d testin...  matt a /u*/l*/g*/i*/e*/include
	2000/05/04 14:43:01
00662:0   ls  matt - /m/guide/o/guide 2000/05/04 14:31:53

Note:

The command and directory elements may be abbreviated, as illustrated in the above example, to make the line fit into various displays.

The go files

By default go redirects the child's merged standard output and standard diagnostic output to a file called ,.g. Any previously existing ,.Og files, where O is zero or more `o's, are moved to a new name by effectively inserting an `o' immediately after the `.'. Thus ,.*g files are moved to ,.o*g.

Note:

This is an adoption of yet another Duff device, as Tom was the author of the original upd program which implemented a similar scheme.

If the go d flag is specified, the outputs are redirected to /.qtree/goes/host/99999key,g, where the 99999 is replaced by the pid of the go process and the key is replaced by a lower case alphabetic to ensure that the file does not have the same name as a previously created go-file.
Note: The key appears in the gomsgs line immediately after the host name. A minus sign (i.e., `') signifies that the output was redirected to a ,.g file.

josh supports similar I/O redirection. Its g flag simulates go and its d flag simulates go's d flag. qremote with a g flag also duplicates the basic functionality of go d as it builds the latter command into the command that will be executed on the remote host. The QEF GUI qefgui uses qremote g to dispatch all its builds and a variety of other commands. As such it is not uncommon for a user to use or simulate go directly or indirectly scores of times a day. To deal with the scores of go-files and the gomsgs audit trails, the Q-Tree provides the following facilities:

display, list, and/or delete go-files
Flags are provided to list (l), head (h), tail (t), tail continuously (f), cat and remove (P), remove and cat next (r), and remove (R) go

files. The go-file to which the operation is to be applied can be specified by one or more os (e.g., oo shows 3rd youngest) or using the age index. The d flag is used to specify that the .qtree/goes/*/*,g files are to be processed. The default is to process the ,.g files in the current directory. The q and Q flags show the statuses of qef processes in the go-files.

goes

display and purge gomsgs files
goes by default displays the active and completed job lines from the current host's gomsgs file. The hhost flag can be used to specify the host. The h flag will process all the existing gomsgs files. Flags are provided to remove lines for completed or day old jobs.

gomonitor

display list of active and inactive go'd jobs
gomonitor is usually run continuously in its own window. It displays the N (default 20) most recent gomsgs lines for completed or active jobs. Completed job lines appear in reverse video.

qefgui

the build monitor window
See Chapter 5.5: The Build Monitor. Facilities are provided to view, tail, and purge the gomsgs files and any of the go-files in the user's /.qtree/goes/* directories.

GODIR & HOME_QTREE

Sometimes it is impossible to have a user's /.qtree/goes/host directory in the same directory as those of other hosts. For example, a user's home directory on VMS cannot usually contain a directory called .qtree. Similarly a user's home directory on windows cannot be easily shared by other hosts. In order for the gomsgs and go-files directory to be accessible on other systems so that the go family of tools, $HOME_QTREE can be set to the name of the directory to be used as the user's /.qtree directory. If that directory is a container file system that is accessible to a unix system, a symbolic link on the latter system can be made to link the VMS system's $HOME_QTREE/goes/host into the unix side's /.qtree/goes/ directory. On a windows system, $GODIR can be set to the name of the user's unix-side /.qtree/goes/windows-host directory. Thus the go-files tools on a unix system can process go-files created on the VMS and windows system, thereby circumventing those systems' limitations.

11.4) Mkvernum and Vcc

Chapter 6 mentioned mkvernum and its link vcc as qdsrv client programs and illustrated their use. Another illustration of their use is given at the bottom of Page 12 of the Cook's Tour. mkvernum produces a version information file by translating a template with embedded `^' sequences such as `^T' for the time, `^r' for revision, and `^O' for operating system. A listing of all the special `^' strings is output by:

    % mkvernum -X
    Format special characters:

    ^(<string>)     evaluate <string> as qvrs expression
    ^C              Century (19-20)
    ^c              unique count -- retrieved from qdsrv
    ...

The template can be specified in one of four ways:

bbuilt-in	use named built-in format -- listed by the X flag as in: % mkvernum -b defl example char version_string[]={"V@(#)Example 9.1(42) 2000/05/07 15:09:55 Linux-2.0.34-i686"};
ffmtfile	use named file
llibfile	use named library file found in the directories named by @MkvernumPath or <qtree>/lib/verlib, as in: % mkvernum -l backup.vf example Example 9.1(43) 2000/05/07 15:03:37
sstring	use string as format as in: % mkvernum -s '^N (^c) ^P(^r) ^C^E ^T ^Z' example Example (50) Guide(9.1) 2000/05/07 15:19:01 EDT

vcc is a link to mkvernum that is used to create a version file and link it into a program. For example, for the following command:

    % vcc -mtest cc test.c +VFILE+

vcc will create a temporary C version file _vtest.c that contains something similar to:

    char version_string[]={"V@(#)Test 9.1(2) 2000/05/08
	03:49:38 Linux-2.0.34-i686"};

The "+VFILE+" is replaced by "_vtest.c" and the cc command is run, creating an executable with the embedded version string. The qsg program, library, and commands scripts described in Chapter 7.5: The Construction Script provide v flag to specify mkvernum/vcc to be used for the argument programs or libraries.

A number of qvrs variables are used within mkvernum:

Project: name of the project -- incorporated using ^P
Revision: Project revision -- incorporated using ^r or ^v for modules for which a matched @Revision[pat] does not exist
Revision[pat]: Revision for modules matched by pat, incorporated using ^R or ^v.
RevisionString[pat]: The ^X control is replaced by the format specified by @RevisionString[pat] for modules matched by pat or @RevisionString -- defaults to "^N^t^v(^c)", that is the capitalized module name, a tab, @Revision[pat] or @Revision and the module build count.
_F_mkvernum_cc[pat]: Specifies cc flags to be used compiling the version file for modules matched by pat (@_F_mkvernum_cc used if none).
MkvernumPath: Names of directories to be searched for f files.
MkvernumFormat[pat]: The bfls flag to be used to create version files for modules matched by pat -- @MkvernumFormat used if none.

11.5) Mnfdb: The Source Manifest Database Package

Originally the mnfdb package was not part of the QEF product but was in the full Q-Tree system, as it was primarily intended for use with SCCS maintained systems. However, some clients found it useful, even when using other version systems that supposedly provided similar mechanisms.

Briefly, mnfdb and its associated tools are used to maintain and query source manifest databases. A manifest, as used by this tool set, is a set of file and version tuples as in:

    README		9.23
    Sums.QEF		0.0
    arlo/README		9.1

A manifest should represent the source files and versions of those files that are required to produce a release of software. The mnfdb package provides tools that manage, modify, and query a database that is a collection of releases.

Note:

The qsg script newmnfdb can be used to add manifests to a database, provided SCCS is being used and sidsnap is available. However, the creation of a command to produce a manifest for other versioning systems is usually fairly simple.

A manifest database consists of release information lines consisting of a release name, date and time and description, and file-version entries which bind versions of the file to releases. A file version entry consists of three tab separated fields, which are the file name, the file version number, and space separated release names. Subsequent lines that have an empty first field are continuations for the file. For example:

    # This is a comment
    $V6.1   1995/01/18 01:43:41     this is an example
            release entry
    $V6.2   1995/01/20 12:43:42     this is a correction to V6.1
    $V7.1   1996/07/01 01:23:45     New Version (about time)
    file1   1.1     V6.1 V6.2
	    1.22    V7.1
    file2   1.1     V6.1
            1.2     V6.2
    file3   2.1     V7.1

The mnfdb package consists of:

mnfadd	add a new manifest to manifest database
mnfchk	check manifest database for syntax and consistency
mnfcmp	compare two manifests, as in: % mnfcmp -r V6.2 -R V7.1 file1 1.1 1.22 file2 1.1 ^ file3 ^ 2.1
mnfdb	output manifest database release information
mnfdel	delete releases from manifest database
mnfput	output a manifest or manifest database, as in: % mnfput -m -rV7.1 file1 1.22 file3 2.1

To use the manifest database system one does need tools to extract the version identifiers from the version system administration files and a tool that can extract the files named in a manifest, but both are relatively easy commands to implement. A very useful facility is a modification (if necessary) to the version system's check-out facility is to use the version number as specified by a manifest -- kdbm can help.

11.6) Ct and Pa

ct and its link and partner pa are not directly part of the construction system, although they are sometimes used in lieu of temporary files. They are described in this chapter as they are exceedingly useful, and sometimes special setups are required.

Briefly, ct cats its argument files or input into a file in /.qtree/ctdir. The actual file name used is ,NN, where NN is the lowest two digit number not already in use.

pa displays, removes, displays and removes, or lists files from the /.qtree/ctdir directory. The default is to process the youngest such file, but options are provided to specify explicit files, or files by age index.

The truly useful aspects of ct and pa are:

one doesn't need to name the temporary file
the files are accessible from anywhere on the network from any terminal and/or window
one can write to other user's cut directories to quickly and efficiently send data without fear of destroying that user's data.
one can direct input to a file and then use that file's name in commands that cannot manage two input streams as in:
```
    % ... | ct
    % ... | diff - `pa -n` # - -n outputs cutfile name
    % pa -R # remove temporary file
```
Note: The above depends on there being no intervening ct between the ct and pa n. To be safe one can use the ct n or e flags to retrieve the name or index of the created file.

Once again the limitations of VMS or windows can get in the way. However, one can set $CTDIR to name the directory to be used. Furthermore one can symbolically link another host's /.qtree/ctdir as a sub-directory of the local host's ctdir directory and then use the s subdir option to specify the sub-directory of the user's ctdir/ to be used.

Note:

The u option can be used to ct or pa files to or from another user's cut directory, provided permissions allow it. Sometimes one sets the mode of their ctdir to 777 to permit other users to create cut files. Alternatively, one can send the cut file's name to the other user which does not require write permission.

A final note. One use of ct/pa is so common and useful it's almost an idiom and that is:

    % pwd | ct # in some window
    % cd `pa -r` # in another window

One uses ct to save the current directory's pathname in a cut file and then "pastes" the contents of that file using pa into a chdir command in another window, possibly on another host and monitor, and then removes the cut file.

11.7) Rpl -- An Oldie But Goldie

rpl reads lines of the form:

    <file>:<address><action><text>

The named file is opened and the actions are performed. If the r (for replace) flag is specified, the modified file is written back into the original file. Otherwise the modified file is written to the standard output. The actions include line insertion, deletion, replacement, and output. The addresses can incorporate line numbers followed by plus or minus signs, and/or `/' enclosed regular expressions. The text can be extended over multiple lines by prefixing the continuation lines with a `\' -- unconventional but useful as so many `\'s at the end of lines must be treated literally.

rpl is described here for a variety of reasons:

a variety of programs (e.g., rep) provide a r flag that generates rpl input.
it's very useful, in fact it is referred to as "our friend rpl" by some users
it's the oldest Q-Tree program, being the first significant dt program on Unix, written in Feb., 1975 and was based on a dt program on an Xerox Sigma 7.

11.8) The -f Flag

Of the 2000 or so different flags in the complete Q-Tree, the ubiquitous x is the most common as it is provided by every program. The second most common flag is f, which usually means one of following:

name the file to be processed (e.g., qef fscript)
specify that the files to be processed are named in the standard input or the argument files
output the file name (e.g., rep f)
override the permissions (e.g., rm f)

The second use in the above list merits further elucidation as it is helpful when dealing with large software projects.

For over 15% of the Q-Tree programs, the f flag means that the files or arguments to be processed are named by the argument files or by the standard input. For example:

    % detox -r -f srclist._ # undosify files named by srclist._
    % cat srclist._ | detox -r -f # the same using stdin

processes the files named by srclist._ (i.e., the file produced by sls).

Note:

Some programs use F (e.g., rep) or l instead of f, as the latter was already being used.

Unfortunately the semantics of the f flags are not consistent, even when it does imply process argument files or its input as file lists due to historic uses of the f, however, the reader is encouraged to bear with these inconsistencies as the normal usefulness of the f flag compensates for the occasional irritation.

The following programs of the QEF subset use f to indicate files to be processed are named in the standard input or the argument files: bfsic, chstat, detox, dmpdeps, filelist, fnd, fndstr, gotstr (by default), incls, lash (use l), libs, lines, linked, lls, mkquete, rep (use F), rotf, sfsic, strfix, transl, vci, wmode, and wot.

c130.qh - 9.6 - 03/11/06