Chapter 12: QEF and Large Projects

This chapter discusses a variety of issues with respect to the use of QEF when dealing with large scale or complicated projects.

The QEF system facilitates this partitioning through the provision of qefdirs and the ease with which projects can share configuration (e.g., qvrs files) and processing (e.g., qsg libraries) information and control.

By dividing the project into smaller more manageable chunks, development can be distributed effectively, without impeding their combination. The individual developer or development team need to consider their partition only, which will usually be a single directory or a small subset of the project's sub-directories. The project manager, Q.A. team, project librarian, and release engineers deal with the total project by having a single directory to manage all of the sub-projects.

Many of the problems of handling smaller projects have already been addressed in Chapter 7: Getting Started Really. This chapter discusses issues of project architecture and control that should be considered when re-engineering a project to be managed and constructed using the QEF system.

Note:

One of the papers provided in the QEF web page http://qef.gts.org/html/additional.html Papers and Presentations is Washing Behind Your Ears: Principles of Software Hygiene. Much of the software philosophy behind this chapter is described in this paper, as is the application of the QEF system (albeit 10 years ago) to major products and projects within Nixdorf/Siemens.

constantly maintain release-ready state of the project

It should be possible, on short notice, to produce the deliverables in the form it would take currently or has taken in the past and have confidence that the produced deliverables can be recreated flawlessly in the future. See Rolling Release Engineering below.

provide trivial commands to do incremental or bare-metal builds

The procedure to produce current or past releases, either from raw sources or incrementally, should be simple and well understood by most, if not all, of the development staff.

support for all of the project's processing

The project architecture and control system should provide support for distribution, packaging, testing, auditing, source reconciliation, etc. Procedures to perform nearly all development and administrative tasks should be simple commands and in many cases should be provided in a consistent manner across all of the organization's projects. In many cases these tasks can be encapsulated in qeffiles using generalized or locally developed qsg scripts.

provide a single point of control

All aspects of the project should be contained within a single directory, possibly with well documented and defined references to prerequisite or supporting projects. Typically this directory should contain the various baselines being maintained and appropriate links to the version management database, if these are not contained within the tree itself. All documentation, testing materials, source, and project archives and audit trails should be accessible from this directory. Sometimes distribution of information to other locations is necessary to deal with networking limitations, but in such a situation, procedures to prepare and possible implement the distribution, and to reconcile and validate changes should be readily available within this tree.

support for distributed development

The architecture of the project and its source should facilitate development either locally or remotely while ensuring that all phases of the development are isolated from changes not of current relevance. However, the architecture should ensure that remote development is consistent with the current approved state of the project.

Basically the approach to development and change control should:

allow and encourage independent non-interfering development

developers should be able to make their changes in such a way that those changes do not interfere with the production or other developers' work. QEF supports this through the source view path mechanism that allows the developer to keep the source that is being changed in a separate tree.

limit authority for changes to master source

The master source tree should be read-only to all but a few so that changes are under control and approved. These people should also be responsible for maintaining production versions of the product that can be used for testing and for providing the headers and libraries that are currently produced from the master sources.

support the use of production directories to fill in the gaps

The developers should be able to build a sub-set of the project, using the production version of the product to provide the header files, libraries, and tools that are not being built by the developer.

Top directory limited to directories

The root directory should be limited to sub-directories, the {root,tree}.vrs and qeffile files. This is to facilitate and encourage development of the sub-directories as separate sub-projects as much as is possible.

Topological ordering of directories

It should be able to topologically order, by dependency, the directories of the top level. It should be possible to build the entire product, other than documentation and post processing, by visiting each directory once and only once.

Note:

Man and post processing passes often have to be done after the entire product is built as these passes may depend on the entire product. However this separation means that the processing involved in these passes can be skipped easily.

The documentation and post processing should also be done in single passes. Circular dependencies might arise which will have to be resolved by referencing the master source or minor reorganizations of the source, to ensure that header files, libraries, and tools that might be required are built in topologically lower directories.

Sub-projects should adhere to the principle of locality

See Section 4.

Provision of Administration Support

The root of a project's baseline tree should contain a directory that contains sub-directories that provide administrative support facilities and control files such as the FSIC package (see Chapter 10), source reconciliation, distribution, packaging, etc.

Support for Early Testing

The project should provide facilities to perform testing as soon as a sub-project is constructed. If possible, testing should not require the installation of the components to be tested (i.e., they can be tested within the object tree). Testing prerequisites should be limited to tools and files built or provided by the sub-project itself or sub-projects that are topologically lower than the current project (i.e., it should not be necessary to build entire project to test any sub-project).

Within any sub-product:

Prerequisites are limited to installed product or self references

One can build and test all the components of a sub-project without reference to source or uninstalled bi-products of other sub-projects. Any references to components not within the sub-project can be done via the installed directory (i.e., the product) as would exist if all sub-projects up to, but not including the sub-project being considered, had been installed. The only exception allowed is to have an include path that incorporates the root source directories to allow forward references to as yet uninstalled header files.

Dependencies within sub-project are limited to sibling directories

Within a directory of a sub-project, dependencies and prerequisites are limited to sibling directories.

An important attribute of release readiness is the assurance that the product, as built, released, and shipped today, can be reproduced exactly in the future by simply extracting the appropriate source files from the version system archives and rebuilding the product in its entirety.

The use of incremental builds, as supported by mimk and make, can be an impediment to achieving this goal, as the product as built today might make use of intermediate files that will not be built due to missed inconsistencies or changes to the build environment. While the QEF system does go a long way in ensuring the consistency of the product, further care and maintenance is required to guarantee release readiness.

Proper use of the separation of the baseline and working source trees, and diligence in ensuring that the product is regularly built from the baseline source will help to maintain a product in a state of readiness for release.

Occasionally a bare-metal-build (see Section 6) should be performed to ensure that circular-dependencies have not been introduced into the product.

A file system integrity check (see Chapter 10) should be performed on a regular basis, particularly after a bare-metal-build, to ensure that the completed product is well understood, that all the required sources exist and are registered.

Due to possible circular-dependencies or the existence of obsolete files in the product directory, the ability to reproduce the product incrementally does not guarantee the ability to produce it from the source. Many situations can arise whereby inconsistencies between the product as it currently exists and the product as it would be reproduced from source can be introduced during normal development.

The costs of creating a consistency mechanism that would check inter-directory circular dependencies are largely prohibitive, thus one has to resort to testing to ensure that the product can be reproduced.

The test to be performed is referred to as a bare-metal-build. That is, the product is produced on a raw machine, one on which the product and intermediate files either do not exist or are not visible, i.e., the machine is reduced to its bare-metal.

Most organizations cannot afford to have such machines available, thus bare-metal-builds will have to be simulated as follows:

1. ensure no environment variables reference either directly or indirectly any files that are part of the product or its intermediate files;
2. ensure that any changes that have been made to the host environment (such as running servers or configuration files) are benign and will not access any previously built part of the product.
3. starting with an empty object tree and a source only baseline, rebuild the product. In case of failure, fix the source, remove everything thus far created (within reason) and iterate.
4. Compare the resulting product with the model as maintained by file system integrity checks, correcting the model or construction process as required.
5. Run the regression tests using the new product and any other testing that is deemed necessary.
6. Repeat from step 1 until a clean flawless build is achieved.

   Using QEF:

       % qd -p<project> -t baseline -r <release>
       % mkqtree -o <new-object-tree>
       cd <new-object-tree>
       rootvrs -d <new-destination> -c <ConfigName>
       % qef All
       # let the testing begin

    switch @System[SysVrs]
    case sun*
	...
    case *
	...
    endswitch

    % system -v
or
    % qvrs @System\[SysVrs]

The use of vci is optional. The QEF system does not require its use, however, it does provide a convenient mechanism whereby the complexities of using a working tree process is greatly simplified. The file arguments to vci interfaces are just the base names. Furthermore, the vci commands can be used in the object tree. The vci interface provides the directory names as required. Normally the vci interface just provides the basic version system interfaces -- see list below -- as would be needed by a developer. vci is not intended to replace the version systems administration tools, although some organizations have supplemented their vci libraries to provide administration functions.

To create a vci library, start by extracting the modules of the RCS library into an empty directory. To get a list of the rcs library modules use:

    % vci -s rcs -X
    changed	tell what files have been changed (RCS)
    ci		check in the argument files (RCS)
    co		check out the argument files (RCS)
    diff	run rcsdiff on the g-file and the v-file (RCS)
    lock	lock the argument files (RCS)
    new		create new administration files (RCS)
    novfiles	list source files without v-files (RCS)
    rlog	output the history of the argument file (RCS)
    tell	tell what files are being edited (RCS)
    unget	unget the argument files (RCS)
    vdiff	run qgdiff on the gfile and the vfile (RCS)

    % qsgdump -l-N -lrcs -E

Each script should then be modified to replace the RCS commands and their set-up by the appropriate code for the new version system. The qeffile to create and install the library will be:

    set Suffixes qsg
    Begin
    mkqsglib -n<name> -v 

    addpath QsgLib ProductDir/lib/qsg

In addition to the standard QEF functions it is recommended that directories to do source reconciliation, project distribution, product packaging, remote updates, project auditing, project backups, etc. be created in this directory to facilitate their invocation and to standardize their functionality. Examples of such functions can be found in the dtree qsg library. See x-qsg reconcile, updsite, and newmnfdb for examples or possible prototypes.

Note that all of these functions are usually run in the baseline or source tree itself. Normally qef will abort if run in a source or baseline tree. To permit qef to be run in such a tree, the qeffile should contain:

    set PermitQef some non-empty value

    % x-qefeg -M suppress-eg > suppress.vrs

Note:

treedirs X explains the suppress.vrs, noqfile.vrs, and nobuilds.vrs.

To facilitate the creation of such documentation QEF provides a prototype x_db database called x-qproc to document such procedures and provide how-to notes. To get a list of the current x-qproc entries, use:

    % x-qproc -L
        x-qproc : software process procedures eXplanatory D.B.
    qproc-intro : introduction to the qproc database
      new-qproc : creating a company specific qproc database
        BMprocs : Build Master procedures
            bm1 : creating a new baseline tree
            bm2 : comparing the repository to current baseline
            bm3 : updating a baseline tree with changes in
			repository
    ...
           bm15 : creating/using/saving a source configuration file
        DVprocs : Developer procedures
            dv1 : creating a working tree
            dv2 : creating an object tree
            dv3 : configuring the object tree
            dv4 : creating working and object tree sub-trees
            dv5 : checking out files for modification
    ...
           dv16 : removing trees

    % x-qproc new-qproc
    new-qproc : creating a company specific qproc database

    This item describes creating a new project or company specific
    version of this database.
    ...

    set Suffixes xdb
    Begin
    commands @argv

• extended organization/platform/system specific sysvrs files -- see Section 7. One adds the qefsupp lib/sysvrs directory to @ConfVrsPath in individual project's tree.vrs file.
• qsg libraries -- see Chapter 9.4 and this chapter's Section 8.
• an approved bin/ directory for third party software. By creating a <qefsupp>/bin/ one can link or copy in all the third party tools that are needed, thereby eliminating the need for a horrific $PATH. One simply adds the qefsupp bin to the @BuildPath variable either by setting the traits variable or modifying @BuildPath in the tree.vrs or one of the sysvrs files.
• howto, qfunc, and bp files.
• Localized x-qproc database -- see Section 10.

Given that the qefsupp directory constitutes important aspects of the build environment, it is crucial that it is managed using a version system and that appropriate measures are taken to record the revision of qefsupp used for any released product.