Next: Historical, Previous: Editing, Up: Top [Contents][Index]

## 7 Making a distribution (making a release)

This section provides the instructions for publishing a Daikon distribution, or making a release. If you only want to create the `daikon.tar` or `daikon.tar.gz` file in your own directory, then simply run `make daikon.tar` or `make daikon.tar.gz`.

Official releases have even version numbers (e.g., 4.6.4) and intermediate work has odd version numbers (e.g., 4.7.3). This means as you prepare for a release the current version number is probably odd. It will be updated as one of the steps in the release process. After making the distribution, one of the final steps is to increment the version number again to prepare for subsequent development. This system has the useful side effect of allowing the build and test process to be repeated to fix a problem without having to worry about updating or resetting the version number. Another advantage is to reinforce, to people who are working from the version control repository, that they are not using the released version, because the version numbers differ.

The Daikon distribution site is located at http://plse.cs.washington.edu/daikon/ and is served from the UW CSE file system at `/cse/web/research/plse/daikon`. In order to be able to write to the distribution site, your CSE user id must be a member of the ‘`plse_www`’ Unix group.

For each of the major steps below, an approximate elapsed time is listed. These timings are up to date as of December 2015. They were measured on a quad x86-64 based machine at 3.4GHz with 16GB of memory (buffalo.cs.washington.edu). Barring any difficulties, the entire process will take at least two hours — but could be much more depending on the number of different platforms on which you test the release.

Each of the steps below assumes that you are using the Bash shell.

Next: Distribution setup instructions, Up: Distribution [Contents][Index]

### 7.1 Directory layout requirements

- Environment variable `DAIKONDIR` must be set to a clone of https://github.com/codespecs/daikon.
- A clone of https://github.com/codespecs/fjalar must be a sibling of `${DAIKONDIR}`.
- Environment variable `JAVA_HOME` must be set to the appropriate JDK installation. For now, this should be a current version of JDK 8. If the default JDK on your system is not JDK 8, you will also need to add `${JAVA_HOME}/bin` to your `PATH`. See the example in the next section.
- Optionally, set environment variable `BIBDIR` to a clone of https://github.com/mernst/plume-bib.

Next: Updating dependencies, Previous: Directory layout requirements, Up: Distribution [Contents][Index]

### 7.2 Distribution setup instructions

Get a fresh copy of your Bash shell (e.g., log out and log back in), then run the following bash commands. Note that your system may no longer have JDK 8 installed at `/usr/lib/jvm-1.8.0`. If you do not have a local copy, one can be found at `/homes/gws/markro/jdk1.8.0_401`. In any case, adjust environment variables below as necessary.

    export DAIKONDIR=$inv/daikon
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0
    export PATH=${JAVA_HOME}/bin:${PATH}
    unset CHECKERFRAMEWORK

Next: The day before the release, Previous: Distribution setup instructions, Up: Distribution [Contents][Index]

### 7.3 Getting the latest version of dependencies

Update the Daikon source files to their most recent version.

    set -o pipefail
    (cd ${DAIKONDIR} && git pull && git log --branches --not --remotes && git status)
    (cd ${DAIKONDIR}/../fjalar && git pull && git log --branches --not --remotes && git status)

Each of the two commands should print exactly these lines:

    Already up-to-date.
    On branch master
    Your branch is up-to-date with 'origin/master'.

    nothing to commit, working directory clean

[Time: moments]

The Fjalar tool set (primarily, Kvasir) is built upon, or uses pieces from, two open source projects. The home page for the Valgrind instrumentation framework is https://valgrind.org. The home page for the GNU Binutils (a collection of binary tools, of which Kvasir uses only readelf) is http://www.gnu.org/software/binutils/.

File `${DAIKONDIR}/fjalar/valgrind/REVISION` indicates the version of these tools that Kvasir uses. You can determine whether a newer version of these tools is available by comparing the `REVISION` file to https://valgrind.org/downloads/current.html and http://ftp.gnu.org/gnu/binutils/?C=M;O=D. If so, you should update the Fjalar source tree as soon as practical. For details, see the separate document “Merging newer versions of Valgrind into Fjalar”, which appears in the `fjalar` repository.

Next: Distribution steps, Previous: Updating dependencies, Up: Distribution [Contents][Index]

### 7.4 The day before the release

Do these steps the day before the release, so that tests have time to complete overnight.

- Edit the `doc/CHANGELOG.md` file to indicate what has changed in this version. A good way to determine what has changed is to diff the Texinfo source files in the `doc/` directory against the previous versions:

      if [ -z "${DAIKONDIR}" ] && [ -d "front-end/" ] ; then DAIKONDIR=$(pwd) ; fi
      cd ${DAIKONDIR}/doc && \
      diff -b -u -s --from-file /cse/web/research/plse/daikon/download/doc *.texinfo

  Another good source of information are the repository logs since the last release:

      if [ -z "${DAIKONDIR}" ] && [ -d "front-end/" ] ; then DAIKONDIR=$(pwd) ; fi
      DAIKONVERSION=`wget -q http://plse.cs.washington.edu/daikon/download/doc/VERSION \
          -O - | xargs echo -n` && \
      cd ${DAIKONDIR} && \
      git log v${DAIKONVERSION}..HEAD

  In addition, you should run the same command in your Fjalar clone:

      if [ -z "${DAIKONDIR}" ] && [ -d "front-end/" ] ; then DAIKONDIR=$(pwd) ; fi
      DAIKONVERSION=`wget -q http://plse.cs.washington.edu/daikon/download/doc/VERSION \
          -O - | xargs echo -n` && \
      { cd ${DAIKONDIR}/fjalar || cd ${DAIKONDIR}/../fjalar ; } && \
      git log v${DAIKONVERSION}..HEAD

  When you have competed your updates, commit and push the changes.

  [Time: a few minutes]

- Check the release documents for spelling errors.

      cd ${DAIKONDIR}/doc
      make spell-check

  Any words that may be spelled incorrectly are output to standard out.

  - If the word is misspelled, correct it in the source (`.texinfo`) file.
  - If the word is spelled correctly and is a normal English word, add it to `daikon.dict`, keeping the list in alphabetical order.
  - If the word is spelled correctly and is a proper name, such as a person’s or company’s name, then either add it to `daikon.dict` or write `@nospellcheck` around it.
  - If the word is technical information, such as part of a computer command or filename, then mark it with a Texinfo command; see “Chapter 9: Marking Text, Words and Phrases” of the GNU Texinfo Manual (PDF, HTML).

  [Time: moments]

- Update dependencies in `daikon/java/lib/`. For details, see `daikon/java/lib/README`.

TODO: it would be good to build the staging release the day before, too, so that links can be fixed the day before the release or at least very early in the process.

Previous: The day before the release, Up: Distribution [Contents][Index]

### 7.5 Steps for making the distribution

1. Verify that Daikon passes its tests.

   All of the jobs at https://dev.azure.com/mernstdaikon/daikon/_build should pass (they should be green). Here is a list of them:

        https://dev.azure.com/mernstdaikon/daikon/_build/latest?definitionId=1&branchName=master

        https://dev.azure.com/mernstdaikon/daikon/_build/latest?definitionId=2&branchName=master

   If any of the jobs is not passing, then correct the problem and wait for the jobs to complete and pass. The delay to wait for this to happen is a reason that you should avoid making changes to Daikon on the release day. Instead, you should make changes the day before to permit the continuous integration jobs to run overnight.

2. Do a very clean rebuild of everything.

       cd ${DAIKONDIR} && make very-clean clean-kvasir && make rebuild-everything

   [Time: 20 min]

3. Once you have successfully reached this point, you are ready to make a release candidate and begin testing it. The following command will verify that `doc/CHANGELOG.md` is up-to-date and then list any uncommitted files.

       make check-repo

   The result of the command above should be:

       On branch master
       Your branch is up-to-date with 'origin/master'.
       nothing to commit (use -u to show untracked files)

   If the command output lists any files, they need to be committed to the repository and pushed now. In that case, you will need to return to step 1 and wait for the CI jobs to complete.

   [Time: moments]

4. Set the version number to an even number. You may:
   - Manually edit file `${DAIKONDIR}/doc/VERSION`, or
   - If the revision number just needs to be incremented by 1 (to go from odd to even) you may use:

         make update-dist-version-file

   Note that if a new feature has been added, or if some change has made the current version incompatible from the previous release (such as a change in the dtrace file format or revised names for tool options), then you should manually edit the VERSION file to increment the minor version number and reset the revision number to zero.

   Optionally, override the distribution date (default: today) by redefining the environment variable `TODAY`:

       export TODAY='November 1, 2013'

   Now, update the appropriate files with the date and version number of the release and commit these changes back to the repository: (WARNING: This step uses a tool that is no longer supported on JDK 8. You must switch to a different bash shell running at least JDK 17 for this step only. Switch back to your original bash shell afterwards.)

       make update-run-google-java-format && make update-and-commit-version

   [Time: moments]

5. It is important to build the release candidate from a clean enlistment. (TODO: In the future, perhaps do all release-related work on a branch. Otherwise, use of the below commands requires committing changes to master, and there might be many commits if there is trouble with the release.) This is to ensure we are testing only the files intended for release and that those files are the same as a user would receive with a clone of the repository. This needs to be done in a separate, new Bash command shell. The following commands get a fresh clone of Daikon, set some environment variables and then build everything to be distributed and copy it to the temporary staging area (http://plse.cs.washington.edu/staging-daikon/).

       # First, start a separate, new Bash command shell.  Don't kill your old one.

       rm -rf /tmp/${USER}/stage-daikon
       mkdir -p /tmp/${USER}/stage-daikon
       cd /tmp/${USER}/stage-daikon
       git clone --depth=1 https://github.com/codespecs/daikon
       cd daikon
       export DAIKONDIR=`pwd`
       export JAVA_HOME=/usr/lib/jvm/java-1.8.0
       export PATH=${JAVA_HOME}/bin:${PATH}
       unset CHECKERFRAMEWORK
       make compile && make staging

   The final output of this command will be a list of files that were added/removed since the last release. If any of these differences is unexpected, then investigate. If any corrections are required, do so back in the main repository, commit the changes, and then repeat this step.

   [Time: 23 min]

6. Next, check the staging website to see if it has any broken document links.

       make check-for-broken-doc-links

   Review `checklink.log` and make corrections as appropriate.

   Sometimes a website forbids robots, or artificially slows them. This can lead to link-checking failures. After you manually check a URL for correctness, you can add it to https://github.com/plume-lib/html-tools/blob/master/html-valid-urls so that no warning is issued in the future.

   In some cases a "403 Forbidden" error is transient, due to the website being down. You can check such links by hand.

   If the URL is correct but cannot be checked (for example, because the website prohibits spiders or because the URL redirects to another URL but you prefer to keep the first URL in your document), then you may need to add lines to an appropriate part of file `checklink-args.txt` in https://github.com/plume-lib/checklink.

   In most cases (and for most 404 errors), you should fix the document with the incorrect link. Here is a workflow that may be used to deal with broken links:

   - Use the ‘`Wayback Machine`’, http://archive.org/web/, and enter the URL.
   - You will get a timeline. Select a year and day, then you have to click on the time of day, and then you get a version of the page at that time.
   - Copy the URL from the address bar of your browser.
   - If the web search is not successful, email the maintainer of the old content.

   Note that you must fix problems in the original repository, not in the drop directories. This means:

   - Switch back to your original Bash shell.
   - Correct the file. (Most likely located beneath either `${DAIKONDIR}/doc` or `${BIBDIR}` (`~/bib/` if `${BIBDIR}` is not set).)
   - Commit the changes.
   - Repeat the previous step to rebuild the release candidate.
   - Repeat this step to verify your corrections.

   If `checklink.log` contains an error message of the form:

       List of broken links and other issues:
       http://plse.cs.washington.edu/daikon/download/doc/developer/Requirements-for-compiling-Kvasir.html
         Line: 185
         Code: 200 OK
        To do: Some of the links to this resource point to broken URI fragments
               (such as index.html#fragment).
       The following fragments need to be fixed:
               Requirements-for-compiling-Kvasir       Line: 185

   This is probably caused by the way `makeinfo` deals with chapter splits via indirect references through stub files. Two `suppress-fragment` entries in `checklink-args.txt` in https://github.com/plume-lib/checklink are required. For the example above, these would be:

       --suppress-fragment http://plse.cs.washington.edu/daikon/download/doc/developer/\
       Requirements-for-compiling-Kvasir.html#Requirements-for-compiling-Kvasir
       --suppress-fragment http://plse.cs.washington.edu/staging-daikon/download/doc/developer/\
       Requirements-for-compiling-Kvasir.html#Requirements-for-compiling-Kvasir

   but on two long lines without any `\` line continuation characters.

   [Time: 8 min]

7. Test the staged distribution on a 64-bit Rocky Linux machine. Since the machine you are using to build the release is probably such a machine, you may simply run the command below. Success is indicated by invariant output being written to the screen.

       make test-staged-dist

   [Time: 5 min]

8. Test the distribution on Ubuntu and Mac OS machines.

   To use Azure Pipelines, navigate to https://dev.azure.com/mernstdaikon/daikon/_build?definitionId=3&_a=summary and click “Run pipeline” then “Run”.

   If there is a problem, fix it and start over.

   [Time: 15 min, or longer if the CI service is busy running other jobs for the codespecs organization.]

9. Once you have successfully reached this point, you have a valid release candidate and are ready to make the actual release. Caution: If somewhere above you made a change to correct a problem you should have restarted the release process.

   At this point we are done building and testing the release candidate and you should exit the Bash command shell created in step 5 and return to your original Bash shell.

10. Update the website by deleting the current release and then making the staged version the (new) current release.

        cd ${DAIKONDIR}
        make staging-to-www

    [Time: 1 min]

11. Add a version label to the repository:

        cd ${DAIKONDIR}
        DAIKONVERSION=`cat ${DAIKONDIR}/doc/VERSION | xargs echo -n`
        git tag -a v${DAIKONVERSION} -m ${DAIKONVERSION}
        git push --tags
        cd fjalar
        git tag -a v${DAIKONVERSION} -m ${DAIKONVERSION}
        git push --tags

    [Time: moments]

12. At this point the distribution has been completed. Bump the version number to an odd value for continuing development.

        cd ${DAIKONDIR}
        export -n TODAY
        make update-dist-version-file
        make update-doc-dist-date-and-version

    [Time: moments]

13. Verify these changes by doing product builds (no need to clean first) and running a short verification test.

        cd ${DAIKONDIR}
        make rebuild-everything quick-test

    Success is indicated by invariant output being written to the screen.

    [Time: 5 min]

14. Commit changes back to the repository:

        cd ${DAIKONDIR}
        git commit -a -m "Bump version number for ongoing development."
        git push
        cd fjalar
        git commit -a -m "Bump version number for ongoing development."
        git push

    [Time: moments]

15. Send mail to the ‘`daikon-announce`’ mailing list. Use the suggested template below, replacing <version number> with the actual number of the release.

        <to:>  daikon-announce@googlegroups.com
        <subject:>  Daikon version <version number> has been released

        Daikon version <version number> is available at:
        http://plse.cs.washington.edu/daikon/download/

        Please see the entry from the doc/CHANGELOG.md file that appears below for more details.

          <your name>

        <a copy of the current entry from the doc/CHANGELOG.md file, with
        paragraphs refilled (remove unnecessary line breaks that trim lines
        to 80 columns for the CHANGELOG.md file but aren't desirable in email)>

    Note that if you use Microsoft Outlook 2010 or 2013 as your mailer, by default it will insert hard breaks in your outgoing message. See https://blog.techhit.com/551102-how-to-prevent-outlook-from-adding-line-breaks-to-sent-plain-text-messages for a work around. You must quit and restart Outlook to activate the change.

Previous: The day before the release, Up: Distribution [Contents][Index]