How to upgrade to a new version of Boehm Garbage Collector in Mercury.

WARNING: This process is difficult and should not be undertaken lightly. Before attempting to upgrade Boehm GC, you should definitely discuss it on the reviews mailing list first.

The setup

This is the first attempt to update Boehm since Mercury switched from CVS to Git. Therefore, I have taken the opportunity to setup this process in a more git-ish way (Boehm GC also uses git). I set this up for version 7.4.2 of the collector and libatomic_ops.

Over time we have made some changes to the collector, some of which have not been pushed upstream. The changes that have not been pushed upstream must be managed by us. I've forked the bdwgc and libatomic_opts repositories. Our forks are currently located here:

	Web	git	branch
BDW GC	https://github.com/Mercury-Language/bdwgc	`https://github.com/Mercury-Language/bdwgc.git`	release-7_4-mercury
libatomic_ops	https://github.com/Mercury-Language/libatomic_ops	`https://github.com/Mercury-Language/libatomic_ops.git`	release-7_4-mercury

On a clean checkout of the Mercury repository, I created a branch off of the master branch.

$ git branch upgrade_boehm master
$ git checkout upgrade_boehm

Then, on this branch I deleted the existing boehm_gc directory from the repository.

$ rm -rf boehm_gc
$ git commit -a

Next we add the bdwgc and libatomic_ops repositories as git submodules. This basically creates a reference from the Mercury repository to these other repositories without importing their history into the Mercury repository.

The references to submodules are relative. So if the remote named origin has the url https://www.github.com/Mercury-Language/mercury.git then we can have git look for the bdwgc repository at the relative path ../bdwgc.git or https://www.github.com/Mercury-Language/bdwgc.git.

$ git submodule add -b release-7_4-mercury ../bdwgc.git boehm_gc
$ git submodule add -b release-7_4-mercury ../libatomic_ops.git libatomic_ops

I've written a script named prepare.sh and committed it, it can be used to initialize and checkout the submodules.

Mercury's customisations to the Boehm GC

I've created a branch named mercury7_2 based on the gc7_2 tag in the bdwgc repository. This branch contains the Mercury customisations to boehm_gc as a series of patches. Then to upgrade to 7.4.2 I created a new branch release-7_4-mercury (from mercury7_2), switched to it, and rebased it onto the point in the boehm_gc tree that represents the BDWGC 7.4.2 release, that is the tag gc7_4_2:

$ git branch release-7_4-mercury mercury7_2
$ git checkout release-7_4-mercury
$ git rebase --onto gc7_4_2 gc7_2

I needed to solve several merge conflicts to complete the rebase.

Pulling changes from upstream

The final step is how to update Mercury's copy of the collector when there are changes upstream. At the time of writing there are some important patches on the collector's release-7_4 branch (The TSX bug).

$ git remote -v
github-ivan https://github.com/ivmai/bdwgc.git (fetch)
github-ivan https://github.com/ivmai/bdwgc.git (push)
github-mercury https://github.com/Mercury-Language/bdwgc.git (fetch)
github-mercury git@github.com:Mercury-Language/bdwgc.git (push)
origin  ../bdwgc.git (fetch)
origin  ../bdwgc.git (push)

Starting with libatomic_ops I update the release-7_4 branch to the latest changes and then rebase Mercury's customisations (in the release-7_4-mercury branch) on top of those.

$ git checkout release-7_4
$ git pull github-ivan release-7_4
$ git push github-mercury release-7_4          # Optional
$ git checkout release-7_4-mercury
$ git pull github-mercury release-7_4-mercury
$ git rebase release-7_4

There are only two Mercury-specific change so this went smoothly. However, the second change commits some autogenerated files, this makes it easier to build libatomic_ops inside the Mercury tree. We need to regenerate those files.

$ ./autogen.sh
$ git commit --amend -a

Double check that you got everything:

$ git status --ignored

Now, publish this change with --force because we are changing history.

$ git tag release-7_4-mercury-20160915
$ git push --force github-mercury release-7_4-mercury
$ git push github-mercury --tags

Now do the same for boehm_gc, however I've documented two examples depending on what you're trying to achieve.

Upgrade to a newer patch within the 7.4 release.

$ git checkout release-7_4
$ git pull github-ivan release-7_4
$ git push github-mercury release-7_4          # Optional
$ git checkout release-7_4-mercury
$ git pull github-mercury release-7_4-mercury
$ git rebase release-7_4

This rebase hard two merge conflicts however they were simple.

Upgrade from the 7.4 release to the 7.6 release

$ git checkout -b release-7_6 github-ivan/release-7.6
$ git push github-mercury release-7_6          # Optional
$ git checkout -b release-7_6-mercury release-7_4-mercury
$ git rebase release-7_6

There were a number of conflicts and some patches have since been merged upstream so I was able to drop them entirely.

Now I need to update the version of libatomic_ops we include with as a submodule in the boehm repository. If the branch name in libatomic_ops was changing I would need to check the .gitmodules file, but in this example it isn't.

$ git submodule update --remote --checkout
$ git add libatomic_ops
$ git commit

Depending on which step we chose above, tag and push these changes.

$ git tag release-7_4-mercury-YYYYMMDD
$ git push --force github-mercury release-7_4-mercury
$ git push github-mercury --tags

$ git tag release-7_6-mercury-20160916
$ git push github-mercury release-7_6-mercury
$ git push github-mercury --tags

Back in the mercury repository I needed to point the boehm_gc submodule to a different branch, Depending on which option we chose above we may need to update the branch name that the Mercury repository refers to.

The branch name has not changed
The branch name was release-7_4_mercury and is now release-7_6-mercury. The branch is adjusted by editing .gitmodules
$ vim .gitmodules

In any case you will need to tell git that the branches have been updated, and these submodules should now refer to different git IDs.

$ git submodule update --remote --checkout
$ git add .gitmodules boehm_gc
$ git commit

Once done, bootcheck the compiler in at least the asm_fast.gc and hlc.gc grades. Then use the new system to do an install (with non-empty LIBGRADES) and test that the installed version can compile some test programs. This is because the update may have added some new files which may not be copied into the install directories. Some build scripts may also need to be updated (in particular tools/bootcheck and scripts/prepare_install_dir.in).

Finally update .README.in (in the root directory) and bindist/bindist.README to reflect the current version of the collector being used. Then commit these changes and have the changes reviewed before pushing them into the public repository.

Modifying Mercury's customisations to the Boehm GC

It is sometimes necessary to modify the files in the boehm_gc directory that we have customised while not updating the collector. For example, the set of options accepted by the mgnuc script may change and boehm_gc/Mmakefile or boehm_gc/Makefile.direct may need to be updated accordingly.

The procedure for doing this is:

Clone the Mercury bdwgc repository.
Checkout the release-M_N-mercury branch of the collector, where M and N are respectively the major and minor release numbers of version of the collector that Mercury is currently using.
Make your changes and commit them.
Add an annotated tag (git tag -a) named release-M_N-mercury-YYYYMMDD where YYYYMMDD is the date.
Push the changes and the tag.
Checkout the above tag in the mercury/boehm_gc submodule.
The output of the git status command will say that boehm_gc has been modified. You can git add and commit that change.
Remind other developers to run git submodule update --recursive in their workspaces.