Next: , Previous: Editing, Up: Top   [Contents][Index]

7 Making a distribution

This section provides the instructions for publishing a Daikon distribution, a.k.a. making a release. If you only want to create the daikon.tar or daikon.tar.gz file in your own directory, then simply run make daikon.tar or make daikon.tar.gz.

Official releases have even version numbers (e.g., 4.6.4) and intermediate work has odd version numbers (e.g., 4.7.3). This means as you prepare for a release the current version number is probably odd. It will be updated as one of the first steps in the release process. After making the distribution, one of the final steps is to increment the version number again to prepare for subsequent development. This system has the useful side effect of allowing the build and test process to be repeated to fix a problem (steps 2–14 of Distribution steps below) without having to worry about updating or resetting the version number. Another advantage is to reinforce, to people who are working from the version control repository, that they are not using the released version, because the version numbers differ.

The Daikon distribution site is located at http://plse.cs.washington.edu/daikon/ and is served from the UW CSE file system at /cse/web/research/plse/daikon. In order to be able to write to the distribution site, your CSE user id must be a member of the ‘plse_www’ Unix group.

For each of the major steps below, an approximate elapsed time is listed. These timings are up to date as of November 2014. They were measured on a quad x86-64 based machine at 3.4GHz with 16GB of memory (buffalo.cs.washington.edu). Barring any difficulties, the entire process will take at least two hours — but could be much more depending on the number of different platforms on which you test the release.

Each of the steps below assumes that the current working directory is $DAIKONDIR and that you are using the Bash shell.


Next: , Up: Distribution   [Contents][Index]

7.1 Directory layout


Next: , Previous: Directory layout, Up: Distribution   [Contents][Index]

7.2 Getting the latest version of dependencies

  1. Update the Daikon source files to their most recent version.
    set -o pipefail
    (cd $DAIKONDIR && git pull && git log --branches --not --remotes && git status)
    (cd $DAIKONDIR/fjalar && git pull && git log --branches --not --remotes && git status)
    

    For each command, the last line printed should be ‘no changes found’. You also need to make sure that the "git log" command didn’t produce any output.

    [Time: moments]

  2. Download the latest version of the Checker Framework Java compiler from http://checkerframework.org/. Set the CHECKERFRAMEWORK environment variable to the location of the Checker Framework directory.

    [Time: moments]

  3. If the Checker Framework has been released since the last distribution, then you need to perform the following steps to update plume.jar.
    cd $DAIKONDIR/plume-lib/java
    git pull
    make check-all && make clean && make
    cp plume.jar ../../java/lib
    

    [Time: 5 min]

  4. Update the bibliography.
    (cd $BIBDIR; git pull)
    

    [Time: moments]

The Fjalar tool set (primarily, Kvasir) is built upon, or uses pieces from, two open source projects. The home page for the Valgrind instrumentation framework is http://www.valgrind.org. As of March 2015 we are using version 3.10.1, released in November, 2014. (You can verify this by inspecting the first line of the file $DAIKONDIR/fjalar/valgrind/NEWS.)

The home page for the GNU Binutils (a collection of binary tools, of which we only use readelf) is http://www.gnu.org/software/binutils/. As of April 2015 we are using version 2.25, released in December, 2014. [TODO: How can a reader confirm that this is correct or determine the version on his or her own?]

If a newer version of either of these projects is available, you should update the Fjalar source tree as soon as practical. For details, see the separate document Merging newer versions of Valgrind into Fjalar.


Previous: Updating dependencies, Up: Distribution   [Contents][Index]

7.3 Steps for making the distribution

  1. Set the version number in file $DAIKONDIR/doc/VERSION to an even number. You may do this manually, or if the value just needs to be incremented by 1 (to go from odd to even) you may use:
    cd $DAIKONDIR
    make update-dist-version-file
    

    Optionally, override the distribution date (default: today) by redefining the environment variable TODAY:

    export TODAY='November 1, 2013'
    

    Now, update the appropriate files with the date and version number of the release.

    make update-doc-dist-date-and-version
    

    [Time: moments]

  2. Do a clean rebuild of everything.
    make rebuild-everything-clean
    

    [Time: 20 min]

  3. Run the Nullness Checker. This is to verify that the Checker Framework Java compiler was correctly used to build Daikon. There should be no errors output.
    make -C $DAIKONDIR/java check-nullness
    

    [Time: 30 min]

  4. Run the Daikon regression tests. The last line printed should be "All tests succeeded."
    make -C $DAIKONDIR/tests clean diffs
    

    [Time: 25 min]

  5. Run the Kvasir regression tests. There should be no diff files marked FAILED.
    make -C $DAIKONDIR/tests/kvasir-tests clean-all nightly-summary-w-daikon
    

    [Time: 20 min]

  6. Run the Java parser tests.
    cd $DAIKONDIR/tests/parser-tests
    ./run-all
    

    Success is indicated by each of the diff comparisons being identical.

    [Time: moments]

  7. Check the release documents for spelling errors.
    cd $DAIKONDIR/doc
    ./texinfo-spellcheck daikon
    ./texinfo-spellcheck developer
    

    Any words that may be spelled incorrectly are output to standard out. There are three basic ways to remove items from the misspelled list:

    [Time: moments]

  8. Edit the doc/CHANGES file to indicate what has changed in this version. A good way to determine what has changed is to diff the Texinfo source files in the doc/ directory against the previous versions:
    cd $DAIKONDIR/doc
    diff -b -u -s --from-file /cse/web/research/plse/daikon/download/doc *.texinfo
    

    Another good source of information are the repository logs since the last release:

    git log v5.0.2..HEAD    (substitute last release number for 5.0.2)
    

    In addition, you should run the same command in your Fjalar clone:

    cd $DAIKONDIR/fjalar
    git log v5.0.2..HEAD
    

    [Time: a few minutes]

  9. Build everything to be distributed and copy it to the temporary staging area (http://plse.cs.washington.edu/daikon/staging-daikon/). The final output will be a list of files that were added/removed since the last release.
    make staging
    

    [Time: 4 min]

  10. Check the website to see if it has any broken links.
    make check-for-broken-doc-links
    

    Review check.log and make corrections as appropriate.

    In most cases (and for most 404 errors), you should fix the document with the incorrect link. Note that you must fix it in the original repository, not in the drop directories. This means correcting the file, most likely located beneath either $DAIKONDIR/doc or $BIBDIR and then committing the changes. You will then need to repeat the previous step (make staging) prior to re-running the broken link tool.

    Here is a workflow that may be used to deal with broken links:

    If the URL is correct but cannot be checked (for example, because the website prohibits spiders or because the URL redirects to another URL but you prefer to keep the first URL in your document), then you may need to add lines to file $DAIKONDIR/plume-lib/bin/checklink-args.txt.

    In some cases a "403 Forbidden" error is transient, due to the website being down. You can check such links by hand.

    If you made changes, re-run the tool.

    [Time: 3 min]

  11. Test the distribution on a 64-bit machine. Since the machine you are using to build the release is probably such a machine, you may simply run the command below. The last step in this test runs Chicory/Daikon on the StackAr example. Success is indicated by invariant output being written to the screen.
    make test-staged-dist
    

    [Time: 9 min]

  12. Test the distribution on a 32-bit machine. This is probably best done on a virtual machine running a current version of Fedora. The instructions below assume you are running on such a machine. They will download the staged daikon.tar.gz file, extract the files, set the appropriate environment variables and run the top level Makefile. Finally, a short verification test is run; success is indicated by invariant output being written to the screen.
    mkdir -p ~/tmp
    cd ~/tmp
    rm -rf daikon.tar.gz daikon
    wget http://plse.cs.washington.edu/daikon/staging-daikon/download/daikon.tar.gz
    tar xzf daikon.tar.gz
    export JAVA_HOME=/usr/lib/jvm/java
    export DAIKONDIR=`pwd`/daikon
    source daikon/scripts/daikon.bashrc
    cd daikon
    make
    make installation-test
    

    [Time: 10 min]

  13. Test the distribution on a 64-bit, Ubuntu based system. The 64-bit test run above is normally run on one of our local machines; therefore, it was probably done on a Fedora (Red Hat) based system. Ubuntu is another distributor of Linux based systems and we have run into some differences with Fedora in the past. This test is probably best done on a virtual machine running a current version. It is important to regularly check to see if a new major release of Ubuntu is available and upgrade your test bed as necessary.
    mkdir -p ~/tmp
    cd ~/tmp
    rm -rf daikon.tar.gz daikon
    wget http://plse.cs.washington.edu/daikon/staging-daikon/download/daikon.tar.gz
    tar xzf daikon.tar.gz
    export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
    export DAIKONDIR=`pwd`/daikon
    source daikon/scripts/daikon.bashrc
    cd daikon
    make
    make installation-test
    

    [Time: ?? min]

  14. Test the distribution on a Windows machine. On a Windows machine with Cygwin installed, as a guest user (or at least one without environment variables such as $DAIKONDIR defined!), download the staged daikon.tar.gz file, extract the files, set the appropriate environment variables and run the top level Makefile. The last line of output from the first make should indicate that Kvasir was not built because this is not a Linux platform. After running the second make command, success is indicated by invariant output being written to the screen.
    mkdir -p ~/tmp
    cd ~/tmp
    rm -rf daikon.tar.gz daikon
    wget http://plse.cs.washington.edu/daikon/staging-daikon/download/daikon.tar.gz
    tar xzf daikon.tar.gz
    # Convert OSTYPE from a shell variable to a system environment variable.
    export OSTYPE
    export JAVA_HOME="/cygdrive/c/Program Files/Java/jdk1.7.0_45"
    export DAIKONDIR=`pwd`/daikon
    source daikon/scripts/daikon.bashrc
    cd daikon
    make
    make installation-test
    

    [Time: 10 min]

  15. Test the distribution on a MacOSX machine. On a MacOSX machine with the development tools installed, download the staged daikon.tar.gz file, extract the files, set the appropriate environment variables and run the top level Makefile. The last line of output from the first make should indicate that Kvasir was not built because this is not a Linux platform. After running the second make command, success is indicated by invariant output being written to the screen.
    mkdir -p ~/tmp
    cd ~/tmp
    rm -rf daikon.tar.gz daikon
    curl -O http://plse.cs.washington.edu/daikon/staging-daikon/download/daikon.tar.gz
    # or: wget http://plse.cs.washington.edu/daikon/staging-daikon/download/daikon.tar.gz
    tar xzf daikon.tar.gz
    # Convert OSTYPE from a shell variable to a system environment variable.
    export OSTYPE
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home
    export DAIKONDIR=`pwd`/daikon
    source daikon/scripts/daikon.bashrc
    cd daikon
    make
    make installation-test
    

    [Time: 4 min]

  16. Save the previous release in the history archives.
    make save-current-release
    

    [Time: moments]

  17. Update the website.
    make staging-to-www
    

    [Time: 1 min]

  18. Commit changes back to the repository:

    Could this and the following step be done with cat $DAIKONDIR/doc/VERSION?

    git commit -a -m "Change version to X.Y.Z"
    git push
    cd fjalar
    git commit -a -m "Change version to X.Y.Z"
    git push
    

    [Time: moments]

  19. Add a version label to the repository:
    git tag -a vX.Y.Z
    git push --tags
    cd fjalar
    git tag -a vX.Y.Z
    git push --tags
    

    [Time: moments]

  20. At this point the distribution has been completed; however, we now need to bump up the version number in file $DAIKONDIR/doc/VERSION to an odd value for continuing development.
    make update-dist-version-file
    

    If you previously set the environment variable TODAY, you need to reset it:

    export -n TODAY
    

    Now, update the appropriate files with the date and version number for ongoing development. (We revert the change to doc/README as we want it to reflect the most recent actual release.)

    make update-doc-dist-date-and-version
    git checkout doc/README
    

    [Time: moments]

  21. Verify these changes by doing product builds (no need to clean first) and running the installation test.
    make rebuild-everything
    make -f Makefile-dist installation-test
    

    Success is indicated by invariant output being written to the screen.

    [Time: 5 min]

  22. Commit changes back to the repository:
    git commit -a -m "Bump version number for ongoing development."
    git push
    cd fjalar
    git commit -a -m "Bump version number for ongoing development."
    git push
    

    [Time: moments]

  23. Send mail to the ‘daikon-announce’ mailing list. This list is located at ‘daikon-announce@googlegroups.com’.

Previous: Updating dependencies, Up: Distribution   [Contents][Index]