\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename valgrind-merge.info
@settitle Merging newer versions of Valgrind into Fjalar
@c %**end of header

@macro nospellcheck{text}
\text\
@end macro

@paragraphindent 0
@codequotebacktick on
@exampleindent 1

@finalout

@titlepage
@sp 10
@center @titlefont{Merging newer versions of Valgrind into Fjalar}
@sp 4
@center @today{}
@end titlepage

@everyfooting @| @| @today{}

@ifnottex
@node Top
@top Merging newer versions of Valgrind into Fjalar

This document describes how to update Fjalar, Kvasir, and DynComp
with newer versions of Valgrind and readelf.

@menu
* Introduction::
* Getting Started::
* Creating the diffs::
* Merging the changes::
* Updating Readelf from Binutils::
* Compiling and Testing::
* Committing::
* Valgrind modifications::
* Memcheck modifications::
* Supporting new VEX IR Instruction::
@end menu
@end ifnottex


@node Introduction
@chapter Introduction

Fjalar is built as a tool on top of the Valgrind instrumentation framework.
It incorporates code from the Valgrind tool Memcheck.  When new
versions of Valgrind and Memcheck are released, Fjalar (and Kvasir)
should be updated
to incorporate the Valgrind/Memcheck changes. This document tells you
how to do so.

Conceptually, this is two separate updates: (1) Updating our underlying
copy of Valgrind to the newer version of Valgrind, and (2) Updating the
Memcheck code contained in Fjalar to the newer version of Memcheck. Due
to very tight dependence of the Memcheck code on Valgrind, the changes
should be done simultaneously.

The update of the underlying copy of Valgrind can usually be done almost
automatically. Updating the Memcheck code often involves manual work,
due to substantial modifications to the Memcheck code that are incorporated
in Fjalar. The more frequent the merges are, the easier they will be to
do. This document gives instructions for merging only when an official
release is made.  This has the advantage of ensuring that the tool is more
stable, but the disadvantages of going longer between bug fixes and making
each merge larger and more difficult.

Fjalar also contains code derived from the GNU Binutils
tool @command{readelf}.  @xref{Updating Readelf from Binutils}
to update Fjalar when a new version of Binutils is released.

@menu
* Determining whether a merge is necessary::
@end menu


@node Determining whether a merge is necessary
@section Determining whether a merge is necessary

To determine whether a newer version of Valgrind is available, compare file
@url{https://raw.githubusercontent.com/codespecs/fjalar/master/valgrind/REVISION,
@file{fjalar/valgrind/REVISION}} to URL
@url{http://valgrind.org/downloads/current.html}.

To determine whether a newer version of Binutils is available, compare file
@url{https://raw.githubusercontent.com/codespecs/fjalar/master/valgrind/REVISION,
@file{fjalar/valgrind/REVISION}} to URL
@url{http://ftp.gnu.org/gnu/binutils/?C=M;O=D}.


@node Getting Started
@chapter Getting Started

The instructions in this document assume you use bash as your shell.
If you use a different shell, you will need to adjust the instructions
accordingly.  Also, we assume @env{$INV} points to a parent directory that
contains both the Daikon and Fjalar source trees.

@menu
* Obtain a copy of Daikon and Fjalar::
* Ensure your source is up to date::
* Verify current sources pass the regression tests::
* Get Valgrind version tags::
* Obtain a copy of the base Valgrind source tree::
* Obtain a copy of the target Valgrind source tree::
* Remove files that have been removed upstream::
@end menu


@node Obtain a copy of Daikon and Fjalar
@section Obtain a copy of Daikon and Fjalar

(You may skip this step if you already have local clones of these repositories.)

Clone a copy of Daikon and Fjalar from our GitHub
repositories:

@example
export INV=`pwd`
git clone https://github.com/codespecs/daikon.git
git clone https://github.com/codespecs/fjalar.git
@end example

If your operating system is Ubuntu, run
@example
sudo apt -yqq install binutils-dev
@end example


@node Ensure your source is up to date
@section Ensure your source is up to date

(If you just made fresh clones of the repositories, you may skip this step.)

Update your enlistment to the latest versions:

@example
cd $INV/daikon
git pull
cd $INV/fjalar
git pull
@end example

Ensure there are no local changes:

@example
cd $INV/daikon
git diff
cd $INV/fjalar
git diff
@end example

The diff commands should produce no output. If there are any differences,
commit and push them, then start over from the beginning.

Ensure there are no outgoing changes:

@example
cd $INV/daikon
git log --branches --not --remotes
cd $INV/fjalar
git log --branches --not --remotes
@end example

The log commands should produce no output. If there are outgoing changes,
push them, then start over from the beginning.


@node Verify current sources pass the regression tests
@section Verify current sources pass the regression tests

Re-compile Fjalar, then run Daikon's Kvasir tests. (The tests take about 20 minutes.)

@example
# Optional, to verify Daikon
make -C $DAIKONDIR compile test

make -C $INV/fjalar build
make -C $INV/daikon/tests/kvasir-tests clean-all regression-tests
@end example

The tests should pass. If any tests fail, fix them, then start over from
the beginning.
As of March 22, 2020, all tests pass on a standard UW/CSE machine running CentOS 7.7.1908.
They also pass on Fedora 25, 28, 30 and Ubuntu 16.04, 17.10 and 18.04.
(We use Azure pipelines and CircleCI workflows to run these configurations.)

At this point, the source trees are ready for the merge.
However, the Fjalar repository contains
@command{automake/autoconf} generated files to simplify things for the end user;
also, there are many other build and test generated files that
need to be omitted to
simplify the output of the diff commands below.

@example
cd $INV/fjalar
make very-clean
@end example

In a later step, we will add any new files found to the Git repository.
We do not want to add any temporary files still present in the current
directories. The following command will get a list of
all unknown (not tracked) files.  Review the list and in most cases
you will want to delete the file(s).

@example
cd $INV/fjalar
git ls-files --others --exclude-standard
@end example


@node Get Valgrind version tags
@section Get Valgrind version tags

Execute the following Git command:

@example
git ls-remote --tags --refs http://sourceware.org/git/valgrind.git | grep -v svn | sort -k2V
@end example

@noindent
The output will list the tags of the most recent releases of Valgrind.  It will
look something like:

@example
4b818ef7a776a9a003b5ddaf6f4285caa90e6e47        refs/tags/VALGRIND_3_14_0
223b0c579b4c31d1cf3c1975a1ded9821bbf7279        refs/tags/VALGRIND_3_15_0
@end example

Assuming that the Valgrind merge process has been done on a regular basis,
the last @samp{tags} line will show the next (target) version number and the
second to last @samp{tags} line will show the current version number.  You should
verify this by comparing with version number from the file @file{valgrind/REVISION}.
Using this information, set two environment variables to be used in the
steps below.

@example
export VOLDVER=VALGRIND_3_14_0
export VNEWVER=VALGRIND_3_15_0
@end example


@node Obtain a copy of the base Valgrind source tree
@section Obtain a copy of the base Valgrind source tree

First, make the directory @file{valgrind-old} that contains a copy of the Valgrind
source tree that was used the last time a Valgrind merge was performed.

If you performed the last merge and saved the
@file{valgrind-old} and @file{valgrind-new} directories, then just run:

@example
cd $INV
rm -rf valgrind-old
mv valgrind-new valgrind-old
@end example

Otherwise, run the following commands.

@example
cd $INV
git clone -b $VOLDVER --depth 1 http://sourceware.org/git/valgrind.git valgrind-old
@end example


@node Obtain a copy of the target Valgrind source tree
@section Obtain a copy of the target Valgrind source tree

Next, make the directory @file{valgrind-new} that contains a copy of the
latest Valgrind source tree.

@example
cd $INV
git clone -b $VNEWVER --depth 1 http://sourceware.org/git/valgrind.git valgrind-new
@end example

@c For a  full repro of the current Valgrind source:
@c git clone http://sourceware.org/git/valgrind.git valgrind-git

@node Remove files that have been removed upstream
@section Remove files that have been removed upstream

The following command lists
files that have been removed by the Valgrind developers since the last
merge:

@example
cd $INV
diff -qrb -x.git valgrind-old valgrind-new |grep "Only in valgrind-old"
@end example

Remove these files.

This step could be done later, but we do it now to reduce the size
of the patch and diff files.


@node Creating the diffs
@chapter Creating the diffs

Run:

@example
$INV/fjalar/generate-patch-files.sh
@end example

This creates patch files showing what the Valgrind maintainers have
changed since
the last time Valgrind code was merged into Fjalar.
File @file{$INV/memcheck.patch} is for Memcheck, and
file @file{$INV/coregrind.patch} is for everything else (which
we'll call "Coregrind").

It also creates files
@file{memcheck-Fjalar.diff} and @file{coregrind-Fjalar.diff}
that show the Fjalar changes to Coregrind and Memcheck.
These will not be used for any automated patching, but for visual
verification later.


@node Merging the changes
@chapter Merging the changes

@menu
* Coregrind Merge::
* Coregrind Conflicts::
* Memcheck Merge::
* Memcheck Conflicts::
* Updating Fjalar/Kvasir::
* Get New Valgrind Regtest Baseline Results::
@end menu


@node Coregrind Merge
@section Coregrind Merge

Now we can merge the changes from the created diffs. The Coregrind patch
should apply with very little conflicts.

It can be difficult to undo a patch operation, so you should first attempt
a dry run.

@example
cd $INV/fjalar/valgrind
patch -p1 -lu < $INV/coregrind.patch --dry-run
@end example

There may be a few "failed hunks" (conflicts) --- this is all right, we
will explain how to deal with these shortly.  If there are massive
numbers of conflicts or some other indication of failure,
there might have been a problem in the previous diffing step.

When you are satisfied with the results of the dry run,
continue with the actual merge:

@example
cd $INV/fjalar/valgrind
patch -p1 -lu < $INV/coregrind.patch
@end example

@strong{IMPORTANT NOTE} @*
This process fails for two different kinds of files,
both in the Valgrind test directories. First, new zero-length goal files
(@file{*.exp}) are not copied from valgrind-new to fjalar.  Second,
executable scripts (typically named @file{filter_<test>}) are copied or updated,
but lose the execute file mode bit.

While this is documented @command{patch} behavior, we need to add additional
steps to correct these problems.

@example
cd $INV/valgrind-new
find -size 0 -exec touch $INV'/fjalar/valgrind/@{@}' \;
find -not -type d -executable -exec chmod +x $INV'/fjalar/valgrind/@{@}' \;
@end example

@c The 'find' commands above didn't work for me this time as the names contained a '.'
@c that file system did not like.  I had to do:
@c find -size 0 -exec sh -c "echo $INV'/fjalar/valgrind{}' | sed 's|\./|/|'" \;|xargs -p touch

@node Coregrind Conflicts
@section Coregrind Conflicts

Handle any conflicts. They are listed in the patch output, or run this
command:

@example
find -name '*.rej'
@end example

For every change in the file you will need to examine both the changed
code as well as our original code and determine if it needs to be hand
merged or if the change is not relevant. It may help to refer to
@code{coregrind-Fjalar.diff} during this process. Remove the @code{.rej}
and @code{.orig} files as you go, so that finally the @code{find}
command produces no output.


@node Memcheck Merge
@section Memcheck Merge

We use the @file{memcheck.patch} file to update Fjalar as its code is
derived from Valgrind/Memcheck.

@example
cd $INV/fjalar/valgrind/fjalar
patch -p2 -lu < $INV/memcheck.patch --dry-run
@end example

There may be a few "failed hunks" (conflicts) - this is all right, we
will explain how to deal with these shortly.  If there are massive
numbers of conflicts or some other indication of failure,
there might have been a problem in the previous diffing step.

You may wish to experiment with the fuzz factor option, the default is @option{-F2}.  I find
that @option{-F3} often increases the success rate (reduces the
number of conflicts), but this can be
dangerous as it ignores more of the context in the patch file, which
increases the odds of a faulty patch.

When you are satisfied with the results of the dry run,
continue with the actual merge:

@example
cd $INV/fjalar/valgrind/fjalar
patch -p2 -lu < $INV/memcheck.patch
@end example


@node Memcheck Conflicts
@section Memcheck Conflicts

Handle any conflicts. They are listed in the patch output, or run this
command:

@example
find -name '*.rej'
@end example

For every change in the file you will need to examine both the changed
code as well as our original code and determine if it needs to be hand
merged or if the change is not relevant.

Our changes to Memcheck are much more substantial than our changes to
Coregrind.

One bit of confusion is that a portion of the original @file{mc_translate.c}
has been relocated to @file{fjalar/mc_translate.h} so it can be shared.

The largest modification to Memcheck is in @file{mc_translate.c} and special
care should be made to ensure it is present and up to date.
The function @code{MC_(instrument)} handles the instrumentation of calls for
DynComp.
It is primarily contained in a switch statement; each case handles one
instruction type, and the body contains both the original code for
Memcheck and also the code for DynComp. After the update, you should
double-check that each clause contains corresponding code: any changes
to the Memcheck versions are reflected in the DynComp versions, and
any new clause has a DynComp version.


@node Updating Fjalar/Kvasir
@section Updating Fjalar/Kvasir

Like @file{fjalar/mc_translate.c},
@file{fjalar/kvasir/dyncomp_translate.c}
is derived from
@file{valgrind/memcheck/mc_translate.c}.  However, due to extensive
modifications and appending @quoteleft{}@code{_DC}@quoteright{} to most
function names to avoid
conflicts, using the patch program with the @file{memcheck.patch} file
is of little use.
Instead, you will need to use @file{memcheck.patch} as a template for
editing @file{fjalar/kvasir/dyncomp_translate.c} by hand.

One bit of confusion is that a portion of the original @file{mc_translate.c}
has been relocated to @file{fjalar/mc_translate.h} so it can be shared.

Additionally Fjalar/Kvasir need to be updated to properly handle any
changes in Valgrind API/functionality. For the most part Valgrind
maintains a relatively stable interface to its tools. Any tool-visible
changes should be noted in the change logs.

A somewhat more problematic area are changes in the VEX IR. DynComp
makes heavy use of the VEX IR, so any changes in it need to be
reflected. Most of the changes to the public VEX IR can be discovered
by running the following command:

@example
cd $inv
diff -b valgrind-old/VEX/pub/libvex_ir.h valgrind-new/VEX/pub/libvex_ir.h
@end example

It is also worth noting that almost all the changes will be for machines
that we do not support.

The files with all the VEX IR code for DynComp are located in
@file{dyncomp_translate.[ch]}. @file{dyncomp_translate.c} is structured primarily
into functions of the form @code{expr2tags_[EXPRESSION_TYPE]()}.
These functions will contain a switch for all VEX IR instructions
corresponding to the expression type and some call to a DynComp tag
function. Most often VEX IR changes will be syntactical in nature and
will only involve changing the names of the instructions. If the log
indicates that new VEX IR instructions have been added, they will need
to be explicitly supported by DynComp. Please see Appendix C for
guidelines on supporting new VEX instructions.


@node Get New Valgrind Regtest Baseline Results
@section Get New Valgrind Regtest Baseline Results

Valgrind has a set of verification tests known as @command{regtest}.  We want
to run these tests on a clean version of Valgrind to establish a
baseline set of results for partial verification of the merge process.
Since running these tests generate a lot of additional files, we wait
until after we have done the source merge update to run this test.
This keeps the files used during the source merge update to a minimum.

You will need to make sure @command{gdb} is installed on your machine
to run these tests. Also, they are very sensitive
to the system software on the host machine.  In most cases, there will be
some failures. Rather, it is important to capture the results
as they should be similar after the source merge. They may not be
identical due to Valgrind improvements or the addition of new test cases.

To run the @command{regtest} verification tests,
return to the root of your @file{valgrind-new} directory.  Then run
the following commands:

@example
./configure --prefix=`pwd`/inst
make all install
make regtest > test.log 2>&1
@end example

@noindent
This will take about 10-15 minutes.  A summary of the @command{regtest}
results will be at the end of the log file.  We will use these results in
@ref{Testing Valgrind}.

Here is a copy of the last few lines from the latest run of these tests (01/21/2026)
running Valgrind 3.23 on a UW/CSE Rocky Linux (9.7) based system:

@example
== 809 tests, 0 stderr failures, 0 stdout failures, 2 stderrB failures, 16 stdoutB failures, 0 post failures ==
gdbserver_tests/hginfo                   (stdoutB)
gdbserver_tests/hgtls                    (stdoutB)
gdbserver_tests/mcblocklistsearch        (stderrB)
gdbserver_tests/mcbreak                  (stdoutB)
gdbserver_tests/mcclean_after_fork       (stdoutB)
gdbserver_tests/mcinfcallWSRU            (stderrB)
gdbserver_tests/mcleak                   (stdoutB)
gdbserver_tests/mcmain_pic               (stdoutB)
gdbserver_tests/mcsignopass              (stdoutB)
gdbserver_tests/mcsigpass                (stdoutB)
gdbserver_tests/mcvabits                 (stdoutB)
gdbserver_tests/mcwatchpoints            (stdoutB)
gdbserver_tests/mssnapshot               (stdoutB)
gdbserver_tests/nlcontrolc               (stdoutB)
gdbserver_tests/nlgone_abrt              (stdoutB)
gdbserver_tests/nlgone_return            (stdoutB)
gdbserver_tests/nlpasssigalrm            (stdoutB)
gdbserver_tests/nlvgdbsigqueue           (stdoutB)

...checking makefile consistency
...checking header files and include directives
Unknown directory 'inst'. Please update check_headers_and_includes
make: *** [Makefile:1440: regtest] Error 1
@end example

(Note that the failure of all the gdbserver tests indicates that the system does not
have the proper glibc-debuginfo installed.)
(Note the final include file failure - this is expected behavior.)
(Note that the unknown directory failure is expected behavior.)

Here is a another example from the latest run of these tests (01/21/2026)
running Valgrind 3.23 on a Ubuntu Linux (25.10) based system:

@example
== 895 tests, 9 stderr failures, 2 stdout failures, 1 stderrB failure, 2 stdoutB failures, 0 post failures ==
gdbserver_tests/mcinfcallWSRU            (stderrB)
gdbserver_tests/nlcontrolc               (stdoutB)
gdbserver_tests/nlvgdbsigqueue           (stdoutB)
memcheck/tests/linux/sys-execveat        (stdout)
memcheck/tests/linux/sys-execveat        (stderr)
memcheck/tests/sendmsg                   (stderr)
memcheck/tests/x86-linux/scalar          (stderr)
helgrind/tests/tc22_exit_w_lock          (stderr)
drd/tests/std_thread2                    (stderr)
none/tests/fdleak_ipv4                   (stderr)
none/tests/file_dclose                   (stderr)
none/tests/scripts/bug231357             (stdout)
none/tests/scripts/bug231357             (stderr)
none/tests/socket_close                  (stderr)

...checking makefile consistency
...checking header files and include directives
Unknown directory 'inst'. Please update check_headers_and_includes
make: *** [Makefile:1445: regtest] Error 1
@end example

As you can see, the results vary depending on the host system.  The important thing to verify
is that the results are the same before and after any Binutils changes.

@node Updating Readelf from Binutils
@chapter Updating Readelf from Binutils

@menu
* Overview::
* Get Current Valgrind Regtest Baseline Results::
* Creating the binutils patch and diff files::
* Merging the binutils changes::
* Hints to modifing dwarf.c and readelf.c::
@end menu


@node Overview
@section Overview

The GNU Binutils are completely independent of Valgrind and have their
own home page (@uref{http://www.gnu.org/software/binutils/})
and release site (@uref{http://ftp.gnu.org/gnu/binutils/}).
While the majority of this document discusses how to update Valgrind,
this chapter describes how to update the @command{readelf} code from Binutils.
For ease of verification, it is recommended that you do not update Valgrind
and Binutils at the same time.


GNU Binutils is a collection of approximately twenty binary tools.
We are interested only in @command{readelf}, a tool that displays
information from an ELF format object file.  Fjalar does not use this
binary directly, rather a portion of the source code for Fjalar contains
a modified version of the source used to build @command{readelf}.


@node Get Current Valgrind Regtest Baseline Results
@section Get Current Valgrind Regtest Baseline Results

Just as is done for a Valgrind source update
(@ref{Get New Valgrind Regtest Baseline Results}),
we want to run the Valgrind @command{regtest} regression tests to
establish a baseline set of results for partial verification of the
@command{readelf} update process.  However, in this case we want
to run the tests prior to the source update as we will use the
current Fjalar sources.

@example
cd $INV/fjalar
make clean build && make test 2>&1 | tee test.log
@end example

@noindent
As noted in @ref{Get New Valgrind Regtest Baseline Results},
we do not expect all the tests to pass.  We will use the results in
@ref{Testing Valgrind}.

@noindent
Note: If there have been no operating system updates, we would expect
the results to be identical to the ones obtained the last time a
full Valgrind update was done.


@node Creating the binutils patch and diff files
@section Creating the binutils patch and diff files

First, create patch files showing what the Binutils maintainers have
changed since the last time the readelf code was merged into Fjalar.
There are two patch files: one for the include files and one for the
actual readelf files.  Note that you will need to modify the export
for @env{BUNEWVER}.  The first line of the example below shows how
to find the correct value.

@c Alternately, could work from the binutils git repository:
@c   git clone git://sourceware.org/git/binutils-gdb.git

@example
# Get new version number from @uref{http://ftp.gnu.org/gnu/binutils/?C=M;O=D}
export BUNEWVER=2.34

source $INV/fjalar/valgrind/REVISION
export BUOLDVER=$BINUTILS_VERSION

export BUOLD=binutils-$BUOLDVER
export BUNEW=binutils-$BUNEWVER
mkdir -p ~/tmp
cd ~/tmp
wget http://ftp.gnu.org/gnu/binutils/binutils-$BUOLDVER.tar.gz
tar xzf binutils-$BUOLDVER.tar.gz
wget http://ftp.gnu.org/gnu/binutils/binutils-$BUNEWVER.tar.gz
tar xzf binutils-$BUNEWVER.tar.gz
diff -ur $BUOLD/include $BUNEW/include > include.patch
diff -u  $BUOLD/libiberty/dwarfnames.c \
         $BUNEW/libiberty/dwarfnames.c >> include.patch
diff -u -xMakefile* $BUOLD/binutils $BUNEW/binutils > binutils.patch
@end example

Next, create diff files comparing the old Binutils to the current
versions in Fjalar.  These diffs will not be used for automated
patching, but for visual verification assistance.  As the majority of
the files should patch quite well, we prepare diffs for only the two
largest and most complicated files.

@example
cd $INV/fjalar/valgrind/fjalar
diff -ub ~/tmp/$BUOLD/binutils/dwarf.c dwarf.c > dwarf-binutils-vs-fjalar.diff
diff -ub ~/tmp/$BUOLD/binutils/readelf.c readelf.c > readelf-binutils-vs-fjalar.diff
@end example


@node Merging the binutils changes
@section Merging the binutils changes

Before applying any patches, run Daikon's Kvasir tests:

@example
  make -C $INV/daikon/tests/kvasir-tests clean-all regression-tests
@end example

@noindent
@xref{Verify current sources pass the regression tests} for the expected
results of running these tests (but don't follow any of the other
instructions in that section).

Now we can merge the changes by applying the patch files.
The patch files probably include files that Fjalar does not use.
Running @command{patch}
will produce a warning message including the line ``can't find file to
patch''.  It is safe to ignore these messages.
The @command{lines-notbetween} command used below will prevent these
messages from appearing.  These unused files will also produce a message
of the form ``<number> out of <number> hunks ignored''.  These may also
be ignored.  A patch failure is indicated by the message ``Hunk #<number>
FAILED at <number>''.  These lines identify patch conflicts and will
need to be corrected as described below.
@file{include.patch} should apply with few, if any, conflicts.
@file{binutils.patch} will probably work on all but @file{dwarf.c} and
@file{readelf.c}.
For these two, a visual merge tool (such as @command{meld}) will be
helpful.
If you plan to use such a tool, you should make copies of these
two files prior to applying the patch. Alternatively, you could add the
"-b" option to the @file{binutils} patch command to backup all the files
modified by the patch.

@example
cp -pf dwarf.c dwarf.c.orig
cp -pf readelf.c readelf.c.orig
@end example

It can be difficult to undo a patch operation, so you should first attempt
a dry run.  It is recommended that you only do one of the patches at a
time in order to make review of the output easier.
(The @command{lines-notbetween} command is from
@uref{http://mernst.github.io/plume-lib/, plume-lib}.)

@smallexample
cd $INV/fjalar/valgrind/fjalar
patch -p1 -lu -f < ~/tmp/include.patch --dry-run \
 | lines-notbetween --inclusive "can't find file to patch" "No file to patch.  Skipping patch."

cd $INV/fjalar/valgrind/fjalar
patch -p2 -lu -f -F3 < ~/tmp/binutils.patch --dry-run \
 | lines-notbetween --inclusive "can't find file to patch" "No file to patch.  Skipping patch."
@end smallexample

There may be a few "failed hunks" (conflicts) --- this is all right, we
will explain how to deal with these shortly.  If there are massive
numbers of conflicts or some other indication of failure,
there might have been a problem in the previous diffing step.

When you are satisfied with the results of the dry run,
continue with the actual merge.  Here are two ways to do it.

@itemize
@item
This command:

@example
cd $INV/fjalar/valgrind/fjalar
patch -p1 -lu -f < ~/tmp/include.patch
patch -p2 -lu -f -F3 < ~/tmp/binutils.patch
@end example

@noindent
puts conflicts in @file{.rej} files.  The conflicts are listed in the patch
output, or run this command to find them:

@example
find -name '*.rej'
@end example

@item
Alternately, this command:

@example
cd $INV/fjalar/valgrind/fjalar
patch -p1 -lu -f --merge=diff3 < ~/tmp/include.patch
patch -p2 -lu -f -F3 --merge=diff3 < ~/tmp/binutils.patch
@end example

@noindent
puts conflicts inline in the file; @command{git status} will list them.

@end itemize

Now, handle the conflicts.

For every change in the file you will need to examine both the changed
code as well as our original code and determine if it needs to be hand
merged or if the change is not relevant.

As noted above, @file{dwarf.c} and @file{readelf.c} will probably
have several conflicts.  As you edit the new files, you may wish to
review the diffs in the previous versions as the new diffs should be similar.
@xref{Hints to modifing dwarf.c and readelf.c} for more source details
that may help you with the merge process.

Again, a visual merge tool can make this process easier. Here is one
method for doing so.  As noted above, you will need to have made copies
of @file{dwarf.c} and @file{readelf.c} prior to applying the patch. We
will do a 3-way merge with the new Binutils file on the left, the original
fjalar version on the right and our new fjalar version in the center.
Look at @file{dwarf.c.rej} and note the line numbers of the failed
patches.

@example
meld $BUNEW/binutils/dwarf.c dwarf.c dwarf.c.orig
@end example

Using the line numbers noted above, locate the problem areas and modify
the target (center) version of dwarf.c as needed to correct the merge
issues.  Repeat this process for @file{readelf.c}.

After you have completed the source merge,
update the ``@code{export BINUTILS_VERSION=}'' line in @file{fjalar/valgrind/REVISION}.
Then proceed to @ref{Compiling and Testing}.


@need 2000
@node Hints to modifing dwarf.c and readelf.c
@section Hints to modifing dwarf.c and readelf.c

Many sections of code in @file{dwarf.c} and @code{readelf.c}, some
of them quite large, are not needed for Fjalar and have been
disabled with C preprocessor commands.  The general format is:

@example
#if 0 // This code is not needed for Fjalar.

...  <unused code> ...

#endif // This code is not needed for Fjalar.
@end example

@noindent
Keeping the unused code instead of deleting it makes the
@command{diff/patch} process go much smoother with fewer
conflicts.  In is unnecessary (and inadvisable) to make any
modifications to these disabled sections of code.

One complication is that Valgrind is compiled with many more
compiler warning options that Binutils.  Currently, this list is:
@example
-Wcast-align
-Wcast-qual
-Wempty-body
-Wformat
-Wformat-security
-Wformat-signedness
-Wignored-qualifiers
-Wimplicit-fallthrough=2
-Wlogical-op
-Wmissing-declarations
-Wmissing-parameter-type
-Wno-cast-qual
-Wold-style-declaration
-Wpointer-arith
-Wwrite-strings
@end example
This can cause a very large number of warnings from Binutils code
that has been merged into Valgrind.  Most are benign, but you will have to
inspect them all.  It is a good idea to modify the code to remove as
many of these new warnings as possible to make subsequent merges easier.

Programs written to run under the Valgrind tool must obey certain
requirements with respect to the use of the C runtime library.
In particular, most of those routines have been replaced
with Valgrind equivalents. Thus, a call to @code{assert} must be replaced
with a call to @code{tl_assert}.  The most common changes you will see
are for the memory allocation, string manipulation and file output
routines.  The name of the routine must be wrapped in @code{VG_(..)}.
For example: @code{VG_(malloc)} or @code{VG_(strlen)}.
It is not
critical to find and correct all of these changes in your first
pass over the modified code.  They will be identified via an
"implicit declaration" warning from the compiler and an
"undefined reference" error from the linker.

The other major area of difficulty when doing the source merge
is with the @code{printf} calls
found in the source.  Remember that the original @command{readelf}
is a tool to print out information about binary object files.
Fjalar uses this code to disassemble the object files and harvest
information needed for analysis of the user's program. Fjalar does
not want to print out this information (unless a debugging
option has been selected).  Thus, almost all the
calls to @code{printf} or @code{putchar} need to be guarded by
a test of one of the Fjalar debug options.  In particular,
@code{fjalar_debug} for the output of debugging information
and @code{fjalar_debug_dump} for the output of the contents of
the binary file being processed.

@need 2000
Please note that these @code{if} guards on the @code{printf} statements
lead to a
subtle problem; if the operands of the @code{printf} call have
side effects, then they must be removed from the @code{printf}
and the value assigned to a temp.
@noindent
For example:
@example
printf ("DW_OP_bit_piece: ");
printf (_("size: %s "),
          dwarf_vmatoa ("u", read_uleb128 (data, &bytes_read, end)));
data += bytes_read;
@end example
@noindent
needs to be rewritten as:
@example
uconst_data = read_uleb128 (data, &bytes_read, end);
if (fjalar_debug_dump)
printf ("DW_OP_bit_piece: ");
if (fjalar_debug_dump)
printf (_("size: %s "),
          dwarf_vmatoa ("u", uconst_data));
data += bytes_read;
@end example

A couple of preprocessor short cuts have been used to help with the process.
@example
if (fjalar_debug) printf (<print data>);
@end example
@noindent
may be replaced with:
@example
FJALAR_DPRINTF(<print data>);
@end example
@noindent
This shortcut is used extensively in @file{readelf.c}.

Within @file{dwarf.c} there is a large section of code from about
line 1100 to line 2560 that uses the following:
@example
#define printf(...) do @{if(ok_to_print) printf(__VA_ARGS__);@} while (0)
...
#undef printf
@end example
@noindent
This allows the @code{printf} statements within this section of code
to be used without modification.

TODO:  This trick could be applied elsewhere.  For instance,  
the routine @code{display_loc_list} in @file{dwarf.c} could use the #define trick (for 3 output
routines, not just @code{printf}).

A missed @code{VG_(..)} wrapper or a missing @code{printf}
that needs to be protected should not be a big problem.  Typically, the
Fjalar tool will operate as expected but contain superfluous information
in its output.  This will likely be caught as a diff in a @file{.out}
file when running the kvasir-tests regressions.


@node Compiling and Testing
@chapter Compiling and Testing

We must now ensure that the merged code compiles correctly and passes
the regression test suite.

@menu
* Compiling Fjalar::
* Testing Valgrind::
* Testing Fjalar::
* Documenting Changes::
@end menu


@node Compiling Fjalar
@section Compiling Fjalar

The first step is to build Fjalar/Kvasir.

@example
cd $INV/fjalar
make build
@end example

The build target should @command{autogen} the @file{config} and @file{Makefiles}
and compile Fjalar. Fix any compilation errors.

Once you have run @command{make build} once, thereafter you can just run
@command{make -C $INV/fjalar/valgrind/fjalar}.


@node Testing Valgrind
@section Testing Valgrind

Before we test Fjalar, it is a good idea to verify we haven't
broken the underlying Valgrind:

@example
cd $INV/fjalar
make test
@end example

@noindent
While a few tests may not pass, the results should be identical to
the results obtained when running these tests
as described in @ref{Get New Valgrind Regtest Baseline Results}
(for a Valgrind update)
or @ref{Get Current Valgrind Regtest Baseline Results}
(for a Binutils update).


@node Testing Fjalar
@section Testing Fjalar

The regression suite is located in the tests/kvasir-tests directory
of the Daikon tree.
It can be run using the following commands (takes about 20 minutes):

@example
cd $INV/daikon/tests/kvasir-tests
make clean-all
make regression-tests
@end example

@noindent
@xref{Verify current sources pass the regression tests} for the expected
results of running these tests.

If tests fail with differences that contain "<STACK_ADDR>", "<STATIC_ADDR>" or
"<HEAP_ADDR>" on the goal file side and hexadecimal values on the new file
side, it might be due to compiler changes or platform changes that
caused code or data to reside at different memory addresses.  Compare the
address ranges in @file{$DAIKONDIR/tests/kvasir-tests/clean-output.pl}
with those in the new binaries and see if changes to
@file{clean-output.pl} are required.


@node Documenting Changes
@section Documenting Changes

Any user-visible changes should be documented in
@file{$INV/daikon/docs/CHANGES}.
Additionally, the first line in @file{fjalar/valgrind/REVISION} should be updated
with the Valgrind version used for this merge.

If new Valgrind tests were added, their names should be added to
the @file{.gitignore} file in the "Ignore generated test drivers"
section.


@node Committing
@chapter Committing

Once the test suite passes without errors and all changes are documented,
the changes should be committed to the version control repository.

@menu
* Tell Git about files that were added or removed::
* Commit commands::
@end menu


@node Tell Git about files that were added or removed
@section Tell Git about files that were added or removed

The @samp{git-update} target below searches through the two patch files
looking for files that were added or deleted.  It then does a
@code{git add} or @code{git remove} as appropriate.

@example
cd $INV/fjalar
make git-update > git.script
# Double-check that it looks OK, then run it:
bash -v ./git.script
@end example


@node Commit commands
@section Commit commands

Next, we do a final visual verification of the changes and commit them
to the repository.

@example
cd $INV/fjalar
# Double-check the diffs - standard practice before committing
git diff -b | more
# If looks OK, proceed with commit.
git commit -a -m "Valgrind merge from revision $@{VALGRIND_REVISION@}\
   to $@{VALGRIND_REVISION_NEW@}."
@end example

Note: We recommend saving the patch and diff files to further document
the process.


@node Valgrind modifications
@appendix Valgrind modifications

In an effort to aid in determining the appropriate measures to take
when merging conflicted files, this section briefly explains our
modifications to Valgrind files.

The most important set of changes is the addition of extra shadow
state in Coregrind and VEX. The shadow area is an area of memory that
Valgrind provides for tools to use. Unfortunately, it is of a fixed
size, and Memcheck uses all of it. We've had to increase the size of
the shadow area.
Additionally, we had to modify some of the VEX architecture files to
include additional information specific to the amd64 and x86 platforms.
Finally, we had to extend Valgrind's implementation of the C library
with a few extra functions.

(Remember that we assume @env{$INV} points to a parent directory that
contains both the Daikon and Fjalar source trees.)

@table @asis
@item $INV/fjalar/valgrind/configure.ac
Change id to Fjalar; add fjalar/Makefile to list of targets
@item $INV/fjalar/valgrind/coregrind/m_libcbase.c
Add VG_ version of strnlen
@item $INV/fjalar/valgrind/coregrind/m_libcfile.c
Add VG_ version of mkdir
@item $INV/fjalar/valgrind/coregrind/m_machine.c
Add get and set functions for additional machine registers
@item $INV/fjalar/valgrind/coregrind/m_scheduler/scheduler.c
Modify an assertion that triggers based on shadow size
@item $INV/fjalar/valgrind/coregrind/pub_core_machine.h
Add additional general purpose registers to the guest state
@item $INV/fjalar/valgrind/coregrind/pub_core_threadstate.h
The declaration of the extra shadow space
@item $INV/fjalar/valgrind/include/pub_tool_libcbase.h
Add VG_ version of strnlen
@item $INV/fjalar/valgrind/include/pub_tool_libcfile.h
Add VG_ version of mkdir and fcntl
@item $INV/fjalar/valgrind/include/pub_tool_machine.h
Add get and set functions for additional machine registers
@item $INV/fjalar/valgrind/Makefile.am
Add fjalar to the list of tools; add files to clean-local target
@item $INV/fjalar/valgrind/none/tests/Makefile.am
Add @nospellcheck{coregrind/libcoregrind-amd64-linux.a to libvex} tests
@item fjalar/valgrind/tests/check_headers_and_includes
Add fjalar to @nospellcheck{tool_dirs and inst to dirs_to_ignore}
@item $INV/fjalar/valgrind/VEX/priv/guest_amd64_helpers.c
Add additional general purpose registers to the guest state
@item $INV/fjalar/valgrind/VEX/priv/guest_x86_helpers.c
Add additional general purpose registers to the guest state
@item fjalar/valgrind/VEX/priv/host_amd64_defs.c
Add function name to debug dump of call instruction
@item $INV/fjalar/valgrind/VEX/priv/host-generic/reg_alloc2.c
Modify memory calculation using the shadow size
@item $INV/fjalar/valgrind/VEX/pub/libvex.h
Add additional general purpose registers to the guest state
@item $INV/fjalar/valgrind/VEX/pub/libvex_ir.h
Add additional general purpose registers to the guest state
@end table


@node Memcheck modifications
@appendix Memcheck modifications

The modifications made to Memcheck are more organizational in nature.
A few functions from @file{mc_main.c} and @file{mc_translate.c} have been made
non-static. An extra header has also been created and filled with their
signatures for use by Fjalar.  Several changes to remove compile warnings
for use of function name as variable.

Other modifications include:

@table @asis
@item $INV/fjalar/valgrind/fjalar/mc_main.c
Run-time arguments and versioning information for Kvasir to replace
the default Memcheck ones
@item $INV/fjalar/valgrind/fjalar/mc_translate.c
In Memcheck's instrumentation block, duplicate each case for DynComp.
@end table


@node Supporting new VEX IR Instruction
@appendix Supporting new VEX IR Instruction

It is recommended that you acquaint yourself with the VEX IR by reading
through:

@table @asis
@item $INV/fjalar/valgrind/VEX/pub/libvex.h
@item $INV/fjalar/valgrind/VEX/pub/libvex_ir.h
@end table

In addition to being the primary headers for the VEX library, the above
files represent the majority of the public documentation on VEX.
Valgrind's translation pipeline consists of the following:

@smallexample
Native assembly -> Pre-instrumented VEX IR -> Post-Instrumented VEX IR -> Final assembly
@end smallexample

Valgrind begins by translating the entirety of an assembly basic block
into VEX IR. Valgrind then allows tools to examine the
translated basic block and insert their own instrumentation. Valgrind
finishes by translating the instrumented IR back into the machine's native
assembly.

In order to keep track of comparabilities, DynComp instruments almost
every VEX IR instruction type. Any added instructions will likely need
to be supported by DynComp. In general DynComp's functionality parallels
Memcheck's, so the best starting point for implementing support for a
new instruction would be to mirror Memcheck's implementation. DynComp's
layout is very similar to Memcheck's, so mirroring functionality should
be fairly straightforward. It is, however, very unlikely that a new
instruction type will be added as the VEX IR is a relatively mature
instruction set and has been in use for almost 9 years at the time of
this writing.

Another type of addition that will need to be supported are added
"IR Expressions." Most VEX IR instructions are implemented as a set
of IR Expressions - Take the following IR instructions for example:

@example
t5 = Add32(t12,0x8:I32)
t10 = CmpLE32S(t2,0x21:I32)
@end example

The above IR Instructions are of the type PUT and they store the
result of an IR Expression into a temporary. These instructions consist
of the destination temporary, an IR Expression, which conceptually is
the operation to be formed, and the arguments to that expression. Most
IR Instructions will have a similar format. DynComp is particularly
interested in analyzing all possible IR Expressions.

@example
$INV/fjalar/valgrind/fjalar/kvasir/dyncomp_translate.c
@end example

contains the following set of functions for processing IR Expressions.

@table @code
@item @strong{IRExpr* expr2tags_DC ( DCEnv* dce, IRExpr* e )}
The main expression handling function. Contains a switch which is
responsible for delegating expression handling to one of the other functions
@item @strong{IRExpr* expr2tags_Qop_DC(...)}
Responsible for the handling of expressions which take 4 arguments.
@item @strong{IRExpr* expr2tags_Triop_DC(...)}
Responsible for the handling of expressions which take 3 arguments.
@item @strong{IRExpr* expr2tags_Binop_DC(...)}
Responsible for the handling of expressions which take 2 arguments.
@item @strong{IRExpr* expr2tags_Unop_DC(...)}
Responsible for the handling of expressions which take 1 argument.
@item @strong{IRExpr* expr2tags_Ldle_DC(...)}
Responsible for the handling of load expressions
@item @strong{IRExpr* expr2tags_CCall_DC(...)}
Responsible for the handling of calls to pure functions.
@item @strong{IRExpr* expr2tags_Mux0X_DC(...)}
Responsible for the handling of multiplexing expressions
@end table

If a new IR Expression is added, it will need to be handled by one of
the above functions. The easiest way to implement it will be to base it
on the implementation of an existing instruction. Alternatively, it
should be straightforward to mimic Memcheck's handling of the expression.

@iftex
@contents
@end iftex

@bye
