QEF home page

Chapter 12: QEF and Large Projects

The Guide's Table of Contents Previous Chapter Bottom Of Page Next Chapter

This chapter discusses a variety of issues with respect to the use of QEF when dealing with large scale or complicated projects.


The most important principle to keep in mind when dealing with large projects is divide and conquer. A large scale project should just be a combination of smaller related sub-projects. Partitioning a project into smaller projects facilitates and improves the overall management and development of the project.

The QEF system facilitates this partitioning through the provision of qefdirs and the ease with which projects can share configuration (e.g., qvrs files) and processing (e.g., qsg libraries) information and control.

By dividing the project into smaller more manageable chunks, development can be distributed effectively, without impeding their combination. The individual developer or development team need to consider their partition only, which will usually be a single directory or a small subset of the project's sub-directories. The project manager, Q.A. team, project librarian, and release engineers deal with the total project by having a single directory to manage all of the sub-projects.

Many of the problems of handling smaller projects have already been addressed in Chapter 7: Getting Started Really. This chapter discusses issues of project architecture and control that should be considered when re-engineering a project to be managed and constructed using the QEF system.
Note:One of the papers provided in the QEF web page http://qef.gts.org/html/additional.html Papers and Presentations is Washing Behind Your Ears: Principles of Software Hygiene. Much of the software philosophy behind this chapter is described in this paper, as is the application of the QEF system (albeit 10 years ago) to major products and projects within Nixdorf/Siemens.

12.1) Objectives

The major objectives of the application of QEF to a project should be to:
constantly maintain release-ready state of the project
It should be possible, on short notice, to produce the deliverables in the form it would take currently or has taken in the past and have confidence that the produced deliverables can be recreated flawlessly in the future. See Rolling Release Engineering below.
provide trivial commands to do incremental or bare-metal builds
The procedure to produce current or past releases, either from raw sources or incrementally, should be simple and well understood by most, if not all, of the development staff.
support for all of the project's processing
The project architecture and control system should provide support for distribution, packaging, testing, auditing, source reconciliation, etc. Procedures to perform nearly all development and administrative tasks should be simple commands and in many cases should be provided in a consistent manner across all of the organization's projects. In many cases these tasks can be encapsulated in qeffiles using generalized or locally developed qsg scripts.
provide a single point of control
All aspects of the project should be contained within a single directory, possibly with well documented and defined references to prerequisite or supporting projects. Typically this directory should contain the various baselines being maintained and appropriate links to the version management database, if these are not contained within the tree itself. All documentation, testing materials, source, and project archives and audit trails should be accessible from this directory. Sometimes distribution of information to other locations is necessary to deal with networking limitations, but in such a situation, procedures to prepare and possible implement the distribution, and to reconcile and validate changes should be readily available within this tree.
support for distributed development
The architecture of the project and its source should facilitate development either locally or remotely while ensuring that all phases of the development are isolated from changes not of current relevance. However, the architecture should ensure that remote development is consistent with the current approved state of the project.
These objectives would appear ambitious and do require some investment, however they are achievable.

12.2) The Software Process

Chapter 1.6: An Example Process briefly described a possible model of the development process and change control using QEF. The described approach is dependent on the Version/Revision control system being used. The QEF system does not require the organization to adhere to any specific software process, although the view pathing mechanism does encourage the separation of master source, source being changed, and the object tree. Furthermore the version system being used may have limitations that restrict those separations.

Basically the approach to development and change control should:

allow and encourage independent non-interfering development
developers should be able to make their changes in such a way that those changes do not interfere with the production or other developers' work. QEF supports this through the source view path mechanism that allows the developer to keep the source that is being changed in a separate tree.
limit authority for changes to master source
The master source tree should be read-only to all but a few so that changes are under control and approved. These people should also be responsible for maintaining production versions of the product that can be used for testing and for providing the headers and libraries that are currently produced from the master sources.
support the use of production directories to fill in the gaps
The developers should be able to build a sub-set of the project, using the production version of the product to provide the header files, libraries, and tools that are not being built by the developer.
The above objectives, coupled with the objectives of highly portable systems and reliable builds, were the major motivations in the development of the QEF system. The following sections discuss mechanisms that can be used to achieve these goals.

12.3) Overall Project Architecture

The overall project architecture does depend on the interdependence of the project's components. However, there are some guidelines that should be followed:
Top directory limited to directories
The root directory should be limited to sub-directories, the {root,tree}.vrs and qeffile files. This is to facilitate and encourage development of the sub-directories as separate sub-projects as much as is possible.
Topological ordering of directories
It should be able to topologically order, by dependency, the directories of the top level. It should be possible to build the entire product, other than documentation and post processing, by visiting each directory once and only once.
Note:Man and post processing passes often have to be done after the entire product is built as these passes may depend on the entire product. However this separation means that the processing involved in these passes can be skipped easily.
The documentation and post processing should also be done in single passes. Circular dependencies might arise which will have to be resolved by referencing the master source or minor reorganizations of the source, to ensure that header files, libraries, and tools that might be required are built in topologically lower directories.
Sub-projects should adhere to the principle of locality
See Section 4.
Provision of Administration Support
The root of a project's baseline tree should contain a directory that contains sub-directories that provide administrative support facilities and control files such as the FSIC package (see Chapter 10), source reconciliation, distribution, packaging, etc.
Support for Early Testing
The project should provide facilities to perform testing as soon as a sub-project is constructed. If possible, testing should not require the installation of the components to be tested (i.e., they can be tested within the object tree). Testing prerequisites should be limited to tools and files built or provided by the sub-project itself or sub-projects that are topologically lower than the current project (i.e., it should not be necessary to build entire project to test any sub-project).
The fundamental objective to be achieved through the architecture of the project is that development and testing of any sub-project can be done with minimal preparation and setup, while ensuring that one can be confident that the integrity of the development and testing of that sub-project is not jeopardized by doing a small subset of the containing project.

12.4) The Principle of Locality

The objective stated at the end of the previous section can largely be achieved through adherence to what we call the principle of locality.

Within any sub-product:

Prerequisites are limited to installed product or self references
One can build and test all the components of a sub-project without reference to source or uninstalled bi-products of other sub-projects. Any references to components not within the sub-project can be done via the installed directory (i.e., the product) as would exist if all sub-projects up to, but not including the sub-project being considered, had been installed. The only exception allowed is to have an include path that incorporates the root source directories to allow forward references to as yet uninstalled header files.
Dependencies within sub-project are limited to sibling directories
Within a directory of a sub-project, dependencies and prerequisites are limited to sibling directories.
Adherence to this principle goes a long way in supporting and encouraging partitioned development, incremental construction, and simple single pass construction.

12.5) Rolling Release Engineering

A major objective of the QEF system is to provide the mechanisms whereby a product is maintained in constant readiness to be released on demand.

An important attribute of release readiness is the assurance that the product, as built, released, and shipped today, can be reproduced exactly in the future by simply extracting the appropriate source files from the version system archives and rebuilding the product in its entirety.

The use of incremental builds, as supported by mimk and make, can be an impediment to achieving this goal, as the product as built today might make use of intermediate files that will not be built due to missed inconsistencies or changes to the build environment. While the QEF system does go a long way in ensuring the consistency of the product, further care and maintenance is required to guarantee release readiness.

Proper use of the separation of the baseline and working source trees, and diligence in ensuring that the product is regularly built from the baseline source will help to maintain a product in a state of readiness for release.

Occasionally a bare-metal-build (see Section 6) should be performed to ensure that circular-dependencies have not been introduced into the product.

A file system integrity check (see Chapter 10) should be performed on a regular basis, particularly after a bare-metal-build, to ensure that the completed product is well understood, that all the required sources exist and are registered.

12.6) Bare Metal Builds

Part of the process of rolling-release-engineering is the testing of the ability to rebuild the product from the baseline source and only that source.

Due to possible circular-dependencies or the existence of obsolete files in the product directory, the ability to reproduce the product incrementally does not guarantee the ability to produce it from the source. Many situations can arise whereby inconsistencies between the product as it currently exists and the product as it would be reproduced from source can be introduced during normal development.

The costs of creating a consistency mechanism that would check inter-directory circular dependencies are largely prohibitive, thus one has to resort to testing to ensure that the product can be reproduced.

The test to be performed is referred to as a bare-metal-build. That is, the product is produced on a raw machine, one on which the product and intermediate files either do not exist or are not visible, i.e., the machine is reduced to its bare-metal.

Most organizations cannot afford to have such machines available, thus bare-metal-builds will have to be simulated as follows:

  1. ensure no environment variables reference either directly or indirectly any files that are part of the product or its intermediate files;
  2. ensure that any changes that have been made to the host environment (such as running servers or configuration files) are benign and will not access any previously built part of the product.
  3. starting with an empty object tree and a source only baseline, rebuild the product. In case of failure, fix the source, remove everything thus far created (within reason) and iterate.
  4. Compare the resulting product with the model as maintained by file system integrity checks, correcting the model or construction process as required.
  5. Run the regression tests using the new product and any other testing that is deemed necessary.
  6. Repeat from step 1 until a clean flawless build is achieved.

    Using QEF:

        % qd -p<project> -t baseline -r <release>
        % mkqtree -o <new-object-tree>
        cd <new-object-tree>
        rootvrs -d <new-destination> -c <ConfigName>
        % qef All
        # let the testing begin
Note:The All construction should include the file system integrity checks. See Chapter 10.

12.7) Project Specific SysVrs files

In Chapter 8.5: The Qvrs Files and Their Processing Order, the sixth item refers to the optional project/platform specific qvrs file p_sysvrs.vrs, usually stored in the QefAdm/confvrs directory. These files are used to contain settings that might be required by the project on specific platforms. Typically in small projects there will be few such settings and they might be contained in the top level tree.vrs file using a switch statement, as in:
    switch @System[SysVrs]
    case sun*
	...
    case *
	...
    endswitch
However, for large projects that require many settings, such as the directories or tools of third party software or capability flags, one should probably set up project/system QefAdm/confvrs/p_sysvrs.vrs files. The actual value of sysvrs may be retrieved using:
    % system -v
or
    % qvrs @System\[SysVrs]
Note:One can specify the name of the file to be used by setting @PsysVrs to the basename of the file to be used.
Many settings in a project sysvrs file are usually dependent on the host and/or are subject to change. Similarly many settings may be required by other projects. As such volatile or shared variables should be set with the <qtree>/data/traits.ext file as described in Chapter 8.9: The Traits System, and imported into the project sysvrs files using the @(trait variable) function.
Note:Some organizations create a project called qefsupp specifically to hold organization/platform specific sysvrs files -- see Section 11.

12.8) Version Control Systems

The vci tool is an interface to the version system named by the @VCSys variable in the root.vrs file. vci is actually just a variant of qsg which uses the vci qsg library and the qsg library named by the @VCSys variable. The standard QEF product provides libraries for SCCS, RCS, CVS, and perforce. Others will be provided as created, either by QEF support or their clients.

The use of vci is optional. The QEF system does not require its use, however, it does provide a convenient mechanism whereby the complexities of using a working tree process is greatly simplified. The file arguments to vci interfaces are just the base names. Furthermore, the vci commands can be used in the object tree. The vci interface provides the directory names as required. Normally the vci interface just provides the basic version system interfaces -- see list below -- as would be needed by a developer. vci is not intended to replace the version systems administration tools, although some organizations have supplemented their vci libraries to provide administration functions.

To create a vci library, start by extracting the modules of the RCS library into an empty directory. To get a list of the rcs library modules use:

    % vci -s rcs -X
    changed	tell what files have been changed (RCS)
    ci		check in the argument files (RCS)
    co		check out the argument files (RCS)
    diff	run rcsdiff on the g-file and the v-file (RCS)
    lock	lock the argument files (RCS)
    new		create new administration files (RCS)
    novfiles	list source files without v-files (RCS)
    rlog	output the history of the argument file (RCS)
    tell	tell what files are being edited (RCS)
    unget	unget the argument files (RCS)
    vdiff	run qgdiff on the gfile and the vfile (RCS)
To extract the code, in an empty directory, run:
    % qsgdump -l-N -lrcs -E

Each script should then be modified to replace the RCS commands and their set-up by the appropriate code for the new version system. The qeffile to create and install the library will be:

    set Suffixes qsg
    Begin
    mkqsglib -n<name> -v 
The default installation directory for the above will be @_DestDir_/lib/qsg. To access this directory you will have to add:
    addpath QsgLib ProductDir/lib/qsg
to the tree.vrs or project/platform sysvrs file. Alternatively, you could install this library in <qtree>/lib/qsg, which would make the library accessible to all projects. If this approach is adopted, care should be taken to protect the library when or if the QEF product is re-installed -- see Appendix A: WARNING With Respect To Re-installation.
Note:Some QEF clients set up a qefsupp product specifically to hold organization specific qsg libraries such as the vci library and other files -- see Section 11.

12.9) Project Administration Support

Throughout this guide there have been references to the @QefAdm directory.
Note:This directory can be explicitly named by setting the variable in the root.vrs file using rootvrs "+QefAdm val".
The QefAdm directory is intended to hold administrative functions such as the FSIC package described in Chapter 10, the confvrs, treedirs, relinfo, and touchdir directories -- see Chapter 7.1: The Project Infrastructure.

In addition to the standard QEF functions it is recommended that directories to do source reconciliation, project distribution, product packaging, remote updates, project auditing, project backups, etc. be created in this directory to facilitate their invocation and to standardize their functionality. Examples of such functions can be found in the dtree qsg library. See x-qsg reconcile, updsite, and newmnfdb for examples or possible prototypes.

Note that all of these functions are usually run in the baseline or source tree itself. Normally qef will abort if run in a source or baseline tree. To permit qef to be run in such a tree, the qeffile should contain:

    set PermitQef some non-empty value
Normally these directories should not be created in the object tree or working trees. To indicate to treedirs that a directory is not to be created in object and working trees, create a suppress.vrs file in the directory to be suppressed. A prototype suppress.vrs file is provided by x-qefeg. To use this prototype, run:
    % x-qefeg -M suppress-eg > suppress.vrs
The file's contents are irrelevant except as documentation -- the file's existence is all that is checked.
Note:treedirs --X explains the suppress.vrs, noqfile.vrs, and nobuilds.vrs.

12.10) Documentation of Project Processes

Most readers have or want to have the ISO900[0-4] certification. This requires, in part, usable and accessible documentation that describes the day-to-day activities and frequently used procedures (i.e., how to check out a file). In fact having such documentation is a good idea even if ISO9000 certification is not sought.

To facilitate the creation of such documentation QEF provides a prototype x_db database called x-qproc to document such procedures and provide how-to notes. To get a list of the current x-qproc entries, use:

    % x-qproc -L
        x-qproc : software process procedures eXplanatory D.B.
    qproc-intro : introduction to the qproc database
      new-qproc : creating a company specific qproc database
        BMprocs : Build Master procedures
            bm1 : creating a new baseline tree
            bm2 : comparing the repository to current baseline
            bm3 : updating a baseline tree with changes in
			repository
    ...
           bm15 : creating/using/saving a source configuration file
        DVprocs : Developer procedures
            dv1 : creating a working tree
            dv2 : creating an object tree
            dv3 : configuring the object tree
            dv4 : creating working and object tree sub-trees
            dv5 : checking out files for modification
    ...
           dv16 : removing trees
Note the item new-qproc. To create an x_qproc database that is specific to your organization and version system, see the new-qproc item, as in:
    % x-qproc new-qproc
    new-qproc : creating a company specific qproc database

    This item describes creating a new project or company specific
    version of this database.
    ...
Basically one extracts the source for the current x-qproc database, saving it in a file called proc.xdb in a sub-directory of the qefsupp project's tree -- see Section 11. The name is changed to avoid conflicting with the <qtree> version. This file is modified as required to reflect the organization's procedures and then installed using the qeffile:
    set Suffixes xdb
    Begin
    commands @argv
When qef is run, <qefsupp>/bin/Xdb.dir/proc.xo will be installed and a copy of or link to <qtree>/bin/x_db called x-proc will be created in the <qefsupp>/bin.

12.11) Supplementary QEF Tools and Files

As mentioned in previous sections, some organizations set up a project, sometimes called qefsupp, to hold supplementary QEF files such as:
  • extended organization/platform/system specific sysvrs files -- see Section 7. One adds the qefsupp lib/sysvrs directory to @ConfVrsPath in individual project's tree.vrs file.
  • qsg libraries -- see Chapter 9.4 and this chapter's Section 8.
  • an approved bin/ directory for third party software. By creating a <qefsupp>/bin/ one can link or copy in all the third party tools that are needed, thereby eliminating the need for a horrific $PATH. One simply adds the qefsupp bin to the @BuildPath variable either by setting the traits variable or modifying @BuildPath in the tree.vrs or one of the sysvrs files.
  • howto, qfunc, and bp files.
  • Localized x-qproc database -- see Section 10.
It is highly recommended that the qefsupp product directory be produced using an standard QEF approach to a software project.

Given that the qefsupp directory constitutes important aspects of the build environment, it is crucial that it is managed using a version system and that appropriate measures are taken to record the revision of qefsupp used for any released product.

12.12) -30-

The primary author and creator of the QEF system has long appreciated the comments, questions, and suggestions of his users as they have often led to significant improvements to the system. QEF Inc. is therefore happy to receive comments, complaints, questions, and suggestions regarding this guide, other documentation, and the software. We are also happy to incorporate changes that you find necessary, such as new qsg scripts and vci interfaces, so that subsequent QEF product releases will contain your changes.

c135.qh - 9.3 - 03/10/22 QEF Home The Guide's Table of Contents Previous Chapter Top Of Page Next Chapter