QEF home page

Chapter 6: Qdsrv and Qhost: The QEF Servers

The Guide's Table of Contents Previous Chapter Bottom Of Page Next Chapter

This chapter covers qdsrv (the QEF server), its client programs, and its uses, and the other QEF server qhost. All the provided services are described, but the emphasis is on the use of the path database to navigate the trees.


6.1) The QDSRV Services

The QEF server qdsrv provides five services to other programs in the QEF system: Path Database, Time Server, Version Number Database and Server, Variable Database Server, and User Path Identifier Server. Descriptions of these services follow.
Path Database Enterprise-wide software development encompasses a number of platforms, directories, developers, and files -- too many for most people to remember. qdsrv maintains a database of paths of project trees which can be queried by qds and its alias qd. This use of qdsrv is covered extensively below and in other chapters.
Time Server Given the common use of distributed file systems, it is not uncommon for builds to be done on a host other than the one that hosts the source file system. As such it is crucial that all the machines involved in the builds have approximately the same time. As part of its initialization, qef will retrieve the time of qdsrv's host. If the retrieved time is not within 10 seconds of the qef's host, qef will abort as in:
    % qef # being run on matt
    #{ @matt 2000/03/01 02:59:45
    qef: Time check failed
    	2000/03/01 02:59:59     gobo (qdsrv host)
    	2000/03/01 02:59:45     matt (local host)
    	14			difference
    #} E-1 matt@/m/qtree/linux2_0i/o9.1 2000/03/01 02:59:45(0)
    
    % qdmgt -t # check the time synchronization
    Time check failed
    	2000/03/01 03:06:20     gobo (qdsrv host)
    	2000/03/01 03:06:06     matt (local host)
    	14			difference
There is a qef flag (--t) that causes qef to suppress the time check, but this is inadvisable.
Version # Database and Server The program mkvernum and its link vcc are used to create version files such as <qtree>/relinfo or the version string in qef itself built at link time:
    % qtreeinf # show Q-Tree version info
    $QTREE: /m/qtree/linux2_0i/9.1
    Default $QTREE: /usr/qtree9.1

    0: /m/qtree/linux2_0i/9.1
      Real Dir: /m/qtree/linux2_0i/9.1
      # following is extracted from the relinfo file
      Qtree(full) 9.1.6(677) Linux-2.0.36-i686(linux2_0i)
		2000/03/24
	    dt on matt in /m/qtree/linux2_0i/o9.1 at
		02:27:24 EDT, 2000/03/24
      Using:
	    Tcl/tk 7.4(12) Linux-2.0.36-i686 2000/03/09
    
    % qef -V # get qef's version string
    Version: Qef (257) Qtree(9.1) 2000/02/27 07:02:20 
	Linux-2.0.34-i686
    
    % wot -vp qef # wot's is somewot like SCCS's what(1)
    Qef (179) Qtree(9.1) 2000/02/27 07:02:20 Linux-2.0.34-i686
    Libqdsrv (31) Qtree(9.1) 2000/01/02 05:49:37 Linux-2.0.34-i686
    Libdtree  9.1.1(233) Qtree(9.1) 2000/02/27 07:00:02 
	Linux-2.0.34-i686
The numbers in parentheses are unique for the module. mkvernum fetches the count for the module from the qdsrv which increments the count for the next request. qdsrv will also record the request and build information in the associated database <qtree>/mkvernum.db as in:
    qtree:qef 9.1(179) Linux-2.0.34-i686 gobo
	/p/qtree/linux2_0i/o9.1/qef dt 2000/02/27 07:02:20
Note:mkvernum type strings can usually be added to a program or a library using the --v flag on the corresponding commands, program, or library statement in the qeffile.
Variable Database Server The qdsrv also provides management and access to arbitrary variables, the values of which are then available throughout the network. Currently the only use of these variables is by qremote -- see Chapter 10.1. qremote retrieves variables of the form host.qtree which stores the directory name of the Q-Tree on the specific host. The list of such variables can be retrieved using the qdmgt message vlist as in:
    % qdmgt vlist m\*.qtree
    matt.qtree = /m/qtree/linux2_0i/9.1
    mokey.qtree = //C/qtree8_4
Variables can be deleted (using vdel), set (using vset), and retrieved (using vget).
User Path Identifier Server In Using qd in the third tutorial of Chapter 4, an example of the use of user identifiers is given. Using qdid --s or qds --I the user can bind a path to an qdsrv identifier. This path number can then be retrieved using qds --i. This facility allows the user to establish a pseudo home that can be easily reached on any host on which the path is available. For example:
    % qd -pguide -o # chdir to guide project's object tree
    Using 59 guide 9.1 object dt gobo linux2_0i /p/guide/o
	/p/guide/s

    % qdid -s # bind current tree to my identifier
    Id 2: 59 guide 9.1 object dt gobo linux2_0i /p/guide/o
	/p/guide/s

    % qdmgt listids # list current path ids in use
    bruce: 0 path: 32
    root: 1 path: -1 # -1 means no path assigned
    dt: 2 path: 59

    % qd -i # chdir to path bound to id
    Using 59 guide 9.1 object dt gobo linux2_0i /p/guide/o
	/p/guide/s
Note:If a user's path identifier is not used for 12 hours, it may be assigned to another user.
Godir Monitoring The programs qefgui and gomonitor both monitor go and josh jobs by scanning the user's ~/.qtree/goes directory. This is an expensive process (particularly on platforms that have slow file systems) which would consume about 2% of the CPU if the scan was done every five seconds. Rather than scan the goes directory every five seconds, all the processes that change the goes directory or its contents send a "godir user" message to qdsrv. qdsrv preserves the time of the last "godir" message for each user. gomonitor and qefgui sends "godirget user" messages to qdsrv every five seconds and qdsrv returns the time of the last "godir user" message. The client program can then compare that time against the last time it scanned the goes directory, thereby avoiding the scan if nothing has changed.
Cycle Server Choice qremote supports a --Sservice flag which causes it to select the remote host to run the job from those listed by the qvrs variable _Servers_[service]. To choose from the list, qremote sends a "cycle" message to qdsrv with the list of hosts as the arguments. qdsrv maintains an internal list of hosts named as "cycle" message arguments an index of the last time it was selected. qdsrv will check the "cycle" message arguments, find the host with the lowest selected index, sets its index and returns its name to qremote. This mechanism is provided so that qremote will select a different host than the previous instances of qdsrvs until all the hosts in the list have been cycled.
Exit Status Communi- cation Unfortunately rsh has had a deficiency since its creation: rsh does not exit with the exit status of the argument command. As such this creates problems for make-like processes such as mimk which use the exit-status of its recipes to determine if the session should be terminated. To deal with this deficiency, qdsrv provides the following exstat* messages:
MessageDescription                                                   
exstatidreserve and retrieve a exstat slot identifier
Used by qremote to reserve a slot which is passed to cush using cush's --X flag.
exstatset id statussend the exit status for slot id
Used by cush to send its argument command's exit status back to qremote.
exstatget idRetrieve the exit status for slot id
Used by qremote to retrieve the exit-status of a remote job run by cush.

See Chapter 11.2: Distributed Processing

6.2) QDSRV Startup

Ideally, qdsrv should be started automatically on a machine accessible from every development machine -- see Appendix A.4: QDSRV Setup. The qdsrv program handles a single database that documents all trees on all projects; therefore it should be located on some machine accessible by all development platforms.

To determine if qdsrv is running, issue the command:

    % qdmgt -v
    # if qdsrv is running ...
    Qdsrv   9.1(162) - 2000/01/19 01:08:00 - Linux-2.0.34-i686
	    pid=455; uid=0; host=gobo; port=6631
	    Database: <qtree>/data/qdsrv.db

    # If qdsrv is not running ...
    qdmgt: connect failed -- Connection refused
	    check that qdsrv is running -- See errQdsrv1(x-qmisc)

If you do see a response similar to the latter one above, try running qdsrv yourself:

    % qdsrv
    qdsrv started: errors will be logged in
	<qtree>/data/qdsrv.log
    # or
    qdsrv: Current host (matt) not Server host (gobo)
The second message indicates that you are not on the host for which qdsrv is configured. The host name is stored in the traits variable QDSRV_HOST in <qtree>/data/traits.ext. The following command will output the current settings:
    % traits QDSRV_HOST QDSRV_PORT
    QDSRV_HOST      gobo
    QDSRV_PORT      6631

The server qdsrv is run on some machine on the network attached to a TCP/IP port. The names of the host and the number of the port are specified by the above traits variables. An environment variable QDSRV_ID could be used to override the traits settings, but given the universal nature of the server, doing so is strictly a local temporary fix. The settings should be the same on all systems and arrangements should be made to ensure that qdsrv is always running on the named host.

6.3) QDSRV Client Programs

Most interaction with the server is through a few specialized commands.

mkvernum Creates release/version/time stamp files and uses qdsrv to record build records and to provide build numbers.
qdchk Check the qdsrv database paths for validity and consistency.
qdid Sets, displays, or assigns a user's path identifier.
qdmgt A generalized mechanism to send arbitrary qdsrv messages. qdmgt has some special purpose options:
--t
Check the current hosts time against that of the qdsrv host.
--f
Freeze the database to allow manual editing -- limited to the root user or the owner of the database.
--r
Reload the database after freezing it.
--v
Retrieve the qdsrv version information and output it.
qds Retrieves paths matched by arguments and default fields (used to reduce the number of matches) from the qdsrv path database. qds supports a --d flag to output a directory name that may be used as an argument to cd or pushd. See qd descriptions below.
qdupd Add the current tree or an argument-specified path to the database -- automatically invoked by rootvrs.
qef The primary QEF driver. It accesses qdsrv services to check the time.
qremote Runs commands, possibly on a remote host. Uses qdsrv to retrieve locations of the Q-Tree on the target host so as to be able to build a $PATH independent rsh command.
vcc A link to mkvernum that compiles the file that it builds -- see above.

6.4) The Path Database, Qds, and Qd

qdsrv loads the database <qtree>/data/qdsrv.db which consists of path records containing the following fields:
    <Index> <Project> <Release> <Type> <User>
	<Host> <Config> <RootPath> ...

as in:

    13 qtree 7.1 object dt sark linux /kent/u/qtree/o /u/qtree/s
Normal Uses

The main purpose of qds is to provide a directory name that can be used as an argument to chdir or to pushd. The arguments to qds (or the alias qd) are used to select the path. For example:

    % qds -pguide -a -S # all paths for the guide project
    52 guide 9.1  baseline dt  gobo  -       /p/guide/s
    59 guide 9.1  object  dt   gobo  linux2_0i /p/guide/o
	/p/guide/s
    63 guide 9.1  object  dt   matt  linux2_0i /m/guide/o
	/m/guide/s

    % qds -t object # guide object trees
    59 guide 9.1  object  dt   gobo  linux2_0i /p/guide/o
	/p/guide/s
        # query automatically assumes the current project, user,
	# and host if doing so reduces number of paths matched

    % qds -o -d # output first directory only
    /p/guide/o/guide
Note:--o is equivalent to --tobject, --b is --tbaseline.

If there are multiple matches, qds will apply the matches current project, current user, and current host to try to reduce the list to a single entry. These matches are equivalent to the flags --p, --u, and --h, each with the argument ".".

The --d flag forces the output to be a single directory. When --d is given and qds cannot reduce the list to a single directory, qds outputs the list to standard error, as in:

    % qds -d -o -pqtree
    The following multiple directories matched:
    32 qtree 9.1    object  dt	    gobo    linux2_0i.f
	    /p/qtree/linux2_0i/o9.1
	    /p/qtree/s9.1
    48 qtree 9.1    object  dt	    gobo    linux2_0i.p
	    /p/qtree/linux2_0i/o9.1p
	    /p/qtree/s9.1
    qds: Use the index in the first column to resolve ambiguity

Most of the time, a unique match can be found with just one or two flags. And after a few uses, you start to know the index values for the paths you commonly use -- you can also specify a path directly with the index, and this may be faster. For example, to go to the path with index 32:

    % cd $(qds -d -32)
The qd Alias/Function

Rather than frequently typing "cd $(qds -d ...)", we suggest you define a shell function or csh alias as follows:

    qd () {
	qdtmp=$(qds -d $*) && cd $qdtmp
    } # in sh, ksh or bash
    alias qd 'qdtmp=`qds -d \!*` && cd $qdtmp'# in csh

This text is also available in the qfunc database, thus can be used to set the function or alias:

    % eval $(qfunc qd) # in sh, bash, or ksh
    % eval `qfunc -c qd` # in csh

Once qd is defined, the command qd will cd to the appropriate directory in the selected path. If the selection does not result in a single path, qd will list the matching paths, including their indexes.

Special Cases

There are a number of special case qds flags in addition to the queries:

 

OptionDescription                                                   
-nchdir to next tree in the RootPath
-Nchdir to path with index N
-i[N]chdir to Nth (default 0th) root in the identifier's path
-I querychdir to matched path and bind the identifier to it

The --i may be followed by a decimal index indicating which root in the RootPath should be selected, thus typically yielding:

 

OptionDescription                                                   
-ichdir to the object tree
-i1chdir to the work area source tree
-i2chdir to the baseline tree

Often you will know the indices of the paths you use most commonly, but if not use qds as in:

    % qds -p <project name> -o

This command produces a list of the paths defined for the project. Then you can run:

    % qd -I -N

where N is the index of the path of interest. This changes directory to the project root and sets the identifier.

qds Query Arguments qds/qd supports a variety of flags to select paths:

 

OptionDescription                                                   
-C[a]show paths whose first directory contains the current directory
`a' option checks all directories of a path.
-R[a]show paths for the current directory's RootDir
`a' option checks all directories of a path
-asuppress default ambiguity resolution --p. --u. --h. queries
See below
-numselect path with index num
-c  configselect directories with given config field
`.' is equivalent to the @System[Name] or `sysnm`
-h  hostselect directories with given host field
`.' is equivalent to the name of the current host
-p  projectselect directories with given project field
`.' is equivalent to the value of qvrs @Project
-r  revisionselect directories with given revision field
`.' is equivalent to the value of qvrs @Revision
-t  typeselect directories with given type field
-u  userselect directories with given user field
`.' is equivalent to the name of the current user
-bequivalent to --tbaseline
-mequivalent to --tmaster
-oequivalent to --tobject
-sequivalent to --tsource
-vequivalent to --tversion
-wequivalent to --tworking
Note: The arguments for the --[chprtu] flags can employ shell wild card pattern matching as in qd -clinux*.

The above may seem formidable, however, in most instances it is sufficient to specify the --ttype argument (or one of the --[bmosvw] short cuts). This is because qds --d will try to reduce the number of matched patterns by applying --p., --u., and --h. queries in turn if they reduce the number of patterns matched (provided not all matches are eliminated).

qds -d argument

qds --d and qd take an optional argument that specifies the sub-directory to be appended to the root of the selected path.

By default, if the selected path has the same @Project setting as the current directory, qds --d will append the current displacement (i.e., @Branch) to the target root. This facilitates changing from a project tree (e.g., the object tree) to the corresponding location in another of the project's trees (e.g., baseline tree).

If the corresponding directory does not exist in the target tree, sub-directories are removed until one that does exist is found.

If the target tree does not have the same @Project as the current value, qds --d will use the root of the target tree.

If an argument is specified, it is interpreted as a sub-directory of the target directory, unless the argument begins with `/', in which case it is interpreted as a sub-directory of the target root. qds --d

Some Useful qdsrv Client Commands
CommandDescription                                                   
qdupdCreate database entry for the current directory
qdupd -h mattCreate database entry for the current directory for host matt
qdchkCheck validity of your paths on current host
qdmgt delete <path>Delete selected path from the database
Limited to paths owned by the user
qdmgt -XList valid messages to qdsrv
qdmgt list -PList all projects in the database
qdmgt list -P -udtList all projects in the database for user dt

6.5) The QHOST Server

qhost is a server that has to be run on any host that might be a target host for qremote.

Its primary purpose is to indicate to the client that its host is up and running. It also returns the system name, pathname of the Q-Tree, and the home directory for the user on the host. So:

    % qhost gobo
    gobo: fbsd4_9i /g/qtree/fbsd4_9i/9.1 /g/dt

qremote sends messages to check that hosts are available and to retrieve the Q-Tree path and the user's home directory.

Relevant qhost commands are:

    % qhost -S # Start the server on named host
    % qhost -R # Keep trying to start the server when
		# address was busy
    % qhost -q host # tell host's qhost to quit
    % qhost -v host # retrieve version of qhost run on host

c061.qh - 9.8 - 03/11/06 QEF Home The Guide's Table of Contents Previous Chapter Top Of Page Next Chapter