Next: , Previous: , Up: Top   [Contents][Index]

7 Making a distribution

This section provides the instructions for publishing a Daikon distribution, a.k.a. making a release. If you only want to create the daikon.tar or daikon.tar.gz file in your own directory, then simply run make daikon.tar or make daikon.tar.gz.

Official releases have even version numbers (e.g., 4.6.4) and intermediate work has odd version numbers (e.g., 4.7.3). This means as you prepare for a release the current version number is probably odd. It will be updated as one of the steps in the release process. After making the distribution, one of the final steps is to increment the version number again to prepare for subsequent development. This system has the useful side effect of allowing the build and test process to be repeated to fix a problem (steps 1–7 of Distribution steps below) without having to worry about updating or resetting the version number. Another advantage is to reinforce, to people who are working from the version control repository, that they are not using the released version, because the version numbers differ.

The Daikon distribution site is located at http://plse.cs.washington.edu/daikon/ and is served from the UW CSE file system at /cse/web/research/plse/daikon. In order to be able to write to the distribution site, your CSE user id must be a member of the ‘plse_www’ Unix group.

For each of the major steps below, an approximate elapsed time is listed. These timings are up to date as of December 2015. They were measured on a quad x86-64 based machine at 3.4GHz with 16GB of memory (buffalo.cs.washington.edu). Barring any difficulties, the entire process will take at least two hours — but could be much more depending on the number of different platforms on which you test the release.

Each of the steps below assumes that you are using the Bash shell.


Next: , Up: Distribution   [Contents][Index]

7.1 Directory layout requirements


Next: , Previous: , Up: Distribution   [Contents][Index]

7.2 Distribution setup instructions

The default version of the Java JDK on the majority of the CSE machines is now JDK 8. However, for the time being, we wish to continue to use JDK 7 for our distribution process; the intention is to maximize compatibility with our existing code base. You will find a suitable version (for an AMD64 based machine) at /homes/gws/markro/jdk7.

The recommended setup for the distribution process:

This should fulfill most of the directory layout requirements noted above.


Next: , Previous: , Up: Distribution   [Contents][Index]

7.3 Getting the latest version of dependencies

  1. Update the Daikon source files to their most recent version.
    set -o pipefail
    (cd $DAIKONDIR && git pull && git log --branches --not --remotes && git status)
    (cd $DAIKONDIR/../fjalar && git pull && git log --branches --not --remotes && git status)
    

    Each of the two commands should print exactly 4 lines:

    Already up-to-date.
    On branch master
    Your branch is up-to-date with 'origin/master'.
    nothing to commit, working directory clean
    

    [Time: moments]

  2. Download the latest version of the Checker Framework Java compiler from http://checkerframework.org/. Set the CHECKERFRAMEWORK environment variable to the location of the Checker Framework directory.

    [Time: moments]

  3. Most likely a new Checker Framework has been released since the last distribution and/or there have been changes to plume-lib. You should always build an updated plume.jar just to be safe.
    make -C $DAIKONDIR plume-lib-update
    cd $DAIKONDIR/plume-lib/java
    make clean && make && make verify-plume-jar-classfile-version && \
        \cp -pf plume.jar ../../java/lib
    

    [Time: 9 min]

  4. Update the bibliography.
    (cd $BIBDIR; git pull)
    

    [Time: moments]

The Fjalar tool set (primarily, Kvasir) is built upon, or uses pieces from, two open source projects. The home page for the Valgrind instrumentation framework is http://www.valgrind.org. The home page for the GNU Binutils (a collection of binary tools, of which Kvasir uses only readelf) is http://www.gnu.org/software/binutils/.

File $DAIKONDIR/fjalar/valgrind/REVISION indicates the version of these tools that Kvasir uses. You can determine whether a newer version of these tools is available by comparing the REVISION file to http://valgrind.org/downloads/current.html and http://ftp.gnu.org/gnu/binutils/?C=M;O=D. If so, you should update the Fjalar source tree as soon as practical. For details, see the separate document “Merging newer versions of Valgrind into Fjalar”, which appears in the fjalar repository.


Previous: , Up: Distribution   [Contents][Index]

7.4 Steps for making the distribution

Up to this point (primarily the rebuilding of plume.jar), we have been using JDK 7 - and our release version will be built with it as well. However, we also need to ensure that our system runs on JDK 8. We do this now to reduce the risk of accidentally releasing any JDK 8 built components.

This means you will need to run the next three steps in a separate shell with export JAVA_HOME=/usr/lib/jvm/java.

Caution: Any problems encountered in the next 3 steps need to be corrected immediately and the release process restarted with step 1 below.

  1. Do a very clean rebuild of everything.
    make very-clean
    make rebuild-everything
    

    [Time: 20 min]

  2. Run the Daikon regression tests. The last line printed should be "All tests succeeded."
    make -C $DAIKONDIR/tests diffs
    

    [Time: 59 min]

  3. Run the Kvasir regression tests. The last line printed should be “All tests succeeded.”, indicating that there are no diff files marked FAILED.
    make -C $DAIKONDIR/tests/kvasir-tests regression-tests
    

    [Time: 23 min]

  4. Upon successful completion of the first three steps, you should exit your JDK 8 based Bash shell and return to your JDK 7 based shell. The remainder of the distribution process will use JDK 7.

    Caution: Any problems encountered in the next 7 steps need to be corrected immediately and the release process restarted with step 1 above (and going back to JDK 8).

  5. Do a very clean rebuild of everything.
    make very-clean
    make rebuild-everything
    

    [Time: 20 min]

  6. Run the Nullness Checker. This is to verify that the Checker Framework Java compiler was correctly used to build Daikon. There should be no errors output.
    make -C $DAIKONDIR/java check-nullness
    

    [Time: 30 min]

  7. Run the Daikon regression tests. The last line printed should be "All tests succeeded."
    make -C $DAIKONDIR/tests diffs
    

    [Time: 59 min]

  8. Run the Kvasir regression tests. The last line printed should be “All tests succeeded.”, indicating that there are no diff files marked FAILED.
    make -C $DAIKONDIR/tests/kvasir-tests regression-tests
    

    [Time: 23 min]

  9. Run the Java parser tests.
    cd $DAIKONDIR/tests/parser-tests
    ./run-all
    

    Success is indicated by each of the diff comparisons being identical.

    [Time: moments]

  10. Edit the doc/CHANGES file to indicate what has changed in this version. A good way to determine what has changed is to diff the Texinfo source files in the doc/ directory against the previous versions:
    cd $DAIKONDIR/doc
    diff -b -u -s --from-file /cse/web/research/plse/daikon/download/doc *.texinfo
    

    Another good source of information are the repository logs since the last release:

    cd $DAIKONDIR
    DAIKONVERSION=`wget -q http://plse.cs.washington.edu/daikon/download/doc/VERSION \
        -O - | xargs echo -n`
    git log v$DAIKONVERSION..HEAD
    

    In addition, you should run the same command in your Fjalar clone:

    cd $DAIKONDIR/fjalar
    DAIKONVERSION=`wget -q http://plse.cs.washington.edu/daikon/download/doc/VERSION \
        -O - | xargs echo -n`
    git log v$DAIKONVERSION..HEAD
    

    [Time: a few minutes]

  11. Check the release documents for spelling errors.
    cd $DAIKONDIR/doc
    make spell-check
    

    Any words that may be spelled incorrectly are output to standard out.

    [Time: moments]

  12. Once you have successfully reached this point, you are ready to make a release candidate and begin testing it. The following command will verify that doc/CHANGES is up-to-date and then list any uncommitted files.
    make check-repo
    

    The result of the command above should be:

    On branch master
    Your branch is up-to-date with 'origin/master'.
    nothing to commit (use -u to show untracked files)
    

    If the command output lists any files, they need to be committed to the repository and pushed now.

    [Time: moments]

  13. Set the version number in file $DAIKONDIR/doc/VERSION to an even number. You may do this manually, or if the value just needs to be incremented by 1 (to go from odd to even) you may use:
    make update-dist-version-file
    

    Optionally, override the distribution date (default: today) by redefining the environment variable TODAY:

    export TODAY='November 1, 2013'
    

    Now, update the appropriate files with the date and version number of the release and commit these changes back to the repository:

    make update-and-commit-version
    

    [Time: moments]

  14. It is important to build the release candidate from a clean enlistment. This is to ensure we are testing only the files intended for release and that those files are the same as a user would receive with a clone of the repository. This needs to be done in a separate, new Bash command shell. The following commands get a fresh clone of Daikon, set some environment variables and then build everything to be distributed and copy it to the temporary staging area (http://plse.cs.washington.edu/staging-daikon/).
    mkdir -p /tmp/$USER
    rm -rf /tmp/$USER/daikon
    git clone https://github.com/codespecs/daikon /tmp/$USER/daikon
    cd /tmp/$USER/daikon
    export DAIKONDIR=`pwd`
    export JAVA_HOME=/homes/gws/markro/jdk7
    source $DAIKONDIR/scripts/daikon.bashrc
    make staging
    

    The final output of this command will be a list of files that were added/removed since the last release. If any of these differences is unexpected, then investigate. If any corrections are required, do so back in the main repository, commit the changes and then repeat this step.

    [Time: 4 min]

  15. Next, check the staging website to see if it has any broken document links.
    make check-for-broken-doc-links
    

    Review check.log and make corrections as appropriate.

    In some cases a "403 Forbidden" error is transient, due to the website being down. You can check such links by hand.

    If the URL is correct but cannot be checked (for example, because the website prohibits spiders or because the URL redirects to another URL but you prefer to keep the first URL in your document), then you may need to add lines to an appropriate part of file $DAIKONDIR/plume-lib/bin/checklink-args.txt.

    In most cases (and for most 404 errors), you should fix the document with the incorrect link. Here is a workflow that may be used to deal with broken links:

    Note that you must fix problems in the original repository, not in the drop directories. This means:

    [Time: 8 min]

  16. Test the distribution on a 64-bit machine. Since the machine you are using to build the release is probably such a machine, you may simply run the command below. The last step in this test runs Chicory/Daikon on the StackAr example. Success is indicated by invariant output being written to the screen.
    make test-staged-dist
    

    [Time: 10 min; runs Checker Framework tests]

  17. Test the distribution on a 32-bit machine. This is probably best done on a virtual machine running a current version of Fedora. The instructions below assume you are running on such a machine. They will download the staged release archive, extract the files, set the appropriate environment variables and run the top level Makefile. Finally, a short verification test is run; success is indicated by invariant output being written to the screen.
    mkdir -p ~/tmp
    cd ~/tmp
    DAIKONVERSION=`wget -q http://plse.cs.washington.edu/staging-daikon/download/doc/VERSION \
        -O - | xargs echo -n`
    rm -rf daikon-$DAIKONVERSION.tar.gz daikon-$DAIKONVERSION
    wget http://plse.cs.washington.edu/staging-daikon/download/daikon-$DAIKONVERSION.tar.gz
    tar xzf daikon-$DAIKONVERSION.tar.gz
    cd daikon-$DAIKONVERSION
    export JAVA_HOME=/usr/lib/jvm/java
    export DAIKONDIR=`pwd`
    source scripts/daikon.bashrc
    make distribution-check
    

    [Time: 10 min]

  18. Test the distribution on a 64-bit, Ubuntu based system. The 64-bit test run above is normally run on one of our local machines; therefore, it was probably done on a Fedora (Red Hat) based system. Ubuntu is another distributor of Linux based systems and there have been some differences with Fedora in the past. This test is probably best done on a virtual machine running a current version. It is important to regularly check to see if a new major release of Ubuntu is available and upgrade your test bed as necessary.

    The instructions below download the staged release archive, extract the files, set the appropriate environment variables and run the top level Makefile. Finally, a short verification test is run; success is indicated by invariant output being written to the screen.

    mkdir -p ~/tmp
    cd ~/tmp
    DAIKONVERSION=`wget -q http://plse.cs.washington.edu/staging-daikon/download/doc/VERSION \
        -O - | xargs echo -n`
    rm -rf daikon-$DAIKONVERSION.tar.gz daikon-$DAIKONVERSION
    wget http://plse.cs.washington.edu/staging-daikon/download/daikon-$DAIKONVERSION.tar.gz
    tar xzf daikon-$DAIKONVERSION.tar.gz
    cd daikon-$DAIKONVERSION
    export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64  # or java-8 as appropriate
    export DAIKONDIR=`pwd`
    source scripts/daikon.bashrc
    make distribution-check
    

    [Time: 20 min]

  19. Test the distribution on a Windows machine with Cygwin installed, as a guest user (or at least one without environment variables such as $DAIKONDIR defined!).

    The output from the make command should show a number of lines beginning with "Processed" as dcomp_rt.jar is being built. Next it should indicate that Kvasir was not built because this is not a Linux platform. Then execution will continue with a test that runs Chicory/Daikon on the StackAr example. Success is indicated by invariant output being written to the screen.

    mkdir -p ~/tmp
    cd ~/tmp
    DAIKONVERSION=`wget -q http://plse.cs.washington.edu/staging-daikon/download/doc/VERSION \
        -O - | xargs echo -n`
    rm -rf daikon-$DAIKONVERSION.tar.gz daikon-$DAIKONVERSION
    wget http://plse.cs.washington.edu/staging-daikon/download/daikon-$DAIKONVERSION.tar.gz
    tar xzf daikon-$DAIKONVERSION.tar.gz
    cd daikon-$DAIKONVERSION
    # Convert OSTYPE from a shell variable to a system environment variable.
    export OSTYPE
    export JAVA_HOME="/cygdrive/c/Program\\\ Files/Java/jdk1.7.0_45"
    export DAIKONDIR=`pwd`
    source scripts/daikon.bashrc
    make distribution-check
    

    [Time: 10 min]

  20. Test the distribution on a MacOSX machine with the development tools installed.

    The output from the make command should show a number of lines beginning with "Processed" as dcomp_rt.jar is being built. Next it should indicate that Kvasir was not built because this is not a Linux platform. Then execution will continue with a test that runs Chicory/Daikon on the StackAr example. Success is indicated by invariant output being written to the screen.

    mkdir -p ~/tmp
    cd ~/tmp
    DAIKONVERSION=`curl -s http://plse.cs.washington.edu/staging-daikon/download/doc/VERSION \
        | xargs echo -n`
    rm -rf daikon-$DAIKONVERSION.tar.gz daikon-$DAIKONVERSION
    curl -O http://plse.cs.washington.edu/staging-daikon/download/daikon-$DAIKONVERSION.tar.gz
    # or: wget http://plse.cs.washington.edu/staging-daikon/download/daikon-$DAIKONVERSION.tar.gz
    tar xzf daikon-$DAIKONVERSION.tar.gz
    cd daikon-$DAIKONVERSION
    # Convert OSTYPE from a shell variable to a system environment variable.
    export OSTYPE
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home
    export DAIKONDIR=`pwd`
    source scripts/daikon.bashrc
    make distribution-check
    

    [Time: 4 min]

  21. Once you have successfully reached this point, you have a valid release candidate and are ready to make the actual release. Caution: If somewhere above you made a change to correct a problem you should have restarted the release process.

    At this point we are done building and testing the release candidate and you should exit the Bash command shell created in step 12 and return to your original Bash shell.

  22. Update the website by deleting the current release and then making the staged version the (new) current release.
    cd $DAIKONDIR
    make staging-to-www
    

    [Time: 1 min]

  23. Add a version label to the repository:
    cd $DAIKONDIR
    DAIKONVERSION=`cat $DAIKONDIR/doc/VERSION | xargs echo -n`
    git tag -a v$DAIKONVERSION -m $DAIKONVERSION
    git push --tags
    cd fjalar
    git tag -a v$DAIKONVERSION -m $DAIKONVERSION
    git push --tags
    

    [Time: moments]

  24. At this point the distribution has been completed. Bump the version number to an odd value for continuing development.
    cd $DAIKONDIR
    export -n TODAY
    make update-dist-version-file
    make update-doc-dist-date-and-version
    

    [Time: moments]

  25. Verify these changes by doing product builds (no need to clean first) and running a short verification test.
    make rebuild-everything quick-test
    

    Success is indicated by invariant output being written to the screen.

    [Time: 5 min]

  26. Commit changes back to the repository:
    cd $DAIKONDIR
    git commit -a -m "Bump version number for ongoing development."
    git push
    cd fjalar
    git commit -a -m "Bump version number for ongoing development."
    git push
    

    [Time: moments]

  27. Send mail to the ‘daikon-announce’ mailing list. Use the suggested template below, replacing <version number> with the actual number of the release.
    <to:>  daikon-announce@googlegroups.com
    <subject:>  Daikon version <version number> has been released
    
    Daikon version <version number> is now available.
    
    Please see the entry from the doc/CHANGES file that appears below for more details.
    
    You can download the latest version of Daikon at:
    http://plse.cs.washington.edu/daikon/download/
    
      <your name>
    
    <a copy of the current entry from the doc/CHANGES> file>
    

Previous: , Up: Distribution   [Contents][Index]