Git Repositories

The main DAOS repository is hosted on GitHub and is also used to track the status of each patch as it is built, tested and finally merged to a branch. Developers submit patches to GitHub using git, and there is a rich browser interface for performing code reviews. Reviewers can also pull the change directly to their own repository and review the change locally.

To upload changes to GitHub, you must first register and add an SSH key to your account. For details on setting up your GitHub account and sending patches, please see Working with GitHub.

GitHub can host many git repositories, and of course each of those repositories can contain many branches.

The IOR DAOS driver, the MPI-IO DAOS driver and the go bindings for the DAOS API are hosted on GitHub. Please find below a summary of all the repositories available today:

Contributions to the GitHub repositories should be submitted through GitHub pull requests.

Managing Changes in Git

Whole books could be written about this topic, and there plenty of online tutorials on the web that explain this in more detail and suggest other methods of managing changes. However, this distilled version is (hopefully) enough to get started.

To make an initial checkout of the master DAOS Git repository:

$ git clone git@github.com:daos-stack/daos.git
$ cd daos_m

This will create the master branch in your repository, which should never be modified. Typically in git, one creates a new local branch for each change being made. The branch is created based on one of the local, pristine branches follow the main development branches, usually the master. A new local branch is usually create using checkout:

$ git checkout -b my-brilliant-idea master

This creates a new branch based on master called my-brilliant-idea, and also makes it the current branch.

In order ensure that the commit description contains the correct name and email address for you, it is possible to specify this directly to Git creating or modifying the $HOME/.gitconfig file in your home directory (preferred), or in the .git/config file in the local repository. This only needs to be done one time, or once per checkout, if done in the local repository.

[user]
        name = Random J. Developer
        email = random@developer.example.org
[remote "review"]
        url = ssh://random@review.hpdd.intel.com:29418/daos/daos_m 

Now all git commit commands in any repository will use this name/email regardless of which user account is used to do the modifications, if using $HOME/.gitconfig file. If you want to specify a different name or email for a specific repository, it is possible to add this information to the .git/config file in that specific repository. See commit comment details below.  The optional [remote "review"] block adds an alias for the remote Gerrit repository to simplify pulling updates and pushing patches, and should only be added in the local .git/config file.

For DAOS patches, the code needs to follow specific DAOS Code Style Guidelines, as do the commit comments for each patch.

Once you've made the change, and you want to save the code for testing and reviewed by others, you commit the change. Before you commit in Git, you need to identify which new or changed file(s) you want to commit using git add <filename> for a specific filename, or git add -u for all modified files. If you haven't added any new files, and you want to commit all the files that you have changed (which is the usual case), then you can use the -a option to commit:

$ git commit -avs

This will put you into an editor to update your commit comment, and when you save it will commit the change locally. The -v option appends the diff to the edit buffer so you can review what is about to be committed. This is very useful to verify that you are committing what you want to commit, and not changes or files that are unrelated to the current change.  The -s option adds a Signed-off-by: line to the commit message, which is required for all patches submitted to DAOS.

It is best to commit relatively frequently to keep your changes limited to a single fix. If you notice other, unrelated, issues that need to be fixed, then it's best to put them in a separate commit on a separate branch, so they can be submitted to Gerrit independently for review, testing, and landing. The stash command is handy for this:

# While fixing a bug you notice something evil that must be fixed.
# First set your current work aside:
$ git stash

# Next go create a new branch and purge the ugliness you just discovered:
$ git checkout -b my-eyes-are-bleeding master
\{ fix ugliness \}
$ git commit -av

# Now go back to what you were working on:
$ git checkout my-brilliant-idea
$ git stash pop

Formatting Git Commit Comments

Having good commit comments helps everyone who is working on the code. See Commit Messages.

Sample Commit message:

DAOS-nnnn component: short description of change under 62 columns

A more detailed explanation of the change being made.  This can be as
detailed as you'd like, possibly several paragraphs in length.

Please provide a detailed explanation of what problem was solved, a
good high-level description of how it was solved, and which parts of
the code were affected (including function names as needed, for easier
searching).  Including specific error messages, stack traces, and
similar is also good.

Wrap lines at 70 columns or less.

Signed-off-by: Random J Developer <random@developer.example.org>

Submitting Patches for Review, Testing, and Landing

A change request is created by pushing a patch to a special branch on the Gerrit repository. To create an change request for a patch in your local branch the local commit (or series of commits) needs to be pushed to the Gerrit review repository.

It is possible to push a series of dependent commits from a local branch in this same way, and each one will create a separate change in Gerrit. Patch series should be used to split complex code changes that are implementing a single larger feature into chunks that are easier to understand, review, test, and land. Changes in a patch series will be dependent on each other, so if one patch needs to be refreshed after a review or test failure it will cause all of the later patches to be refreshed also. That will clear all of the review and test results and restart testing on all of the patches.

Patch series should not be used for code that was written and then had a series of bug fixes applied to it after local testing. The later (local) bug fixes should be squashed into the original change before submission. Note that each patch in a series must build and test properly before it can be landed, so compile warnings or test failures in the middle of a patch series are not allowed.

If you have a series of independent commits to be reviewed, each one should be in a separate local branch and pushed separately to Gerrit. This allows the patches to be reviewed, tested, and landed independently, and will avoid the overhead and delay of repeatedly reviewing, building, and testing patches that are only refreshed because of an earlier (unrelated) patch in a series being modified.

git push ssh://USER@review.hpdd.intel.com:29418/daos/daos_m HEAD:refs/for/master

for the master branch of the daos/daos_m repo.

For convenience, you can add this to your ~/.ssh/config file:

Host review
  Hostname review.hpdd.intel.com
  Port 29418
  User \{YourUserid\}
  IdentityFile ~/.ssh/my-key-id_rsa

Creating a review request for a change against master (assuming the remote alias has been added to ssh config):

git push ssh://review/daos/daos_m HEAD:refs/for/master

Add the Gerrit server as a remote to your git repository:

$ git remote add review ssh://review/daos/daos_m

Automatically Building and Testing a Patch

When the patch has been pushed to the Gerrit daos/daos_m repository, it will print a URL for that change set that should be added to the Jira ticket for tracking.  This should now happen automatically.

All patches submitted to Gerrit will be built automatically by Jenkins for a number of different distro and kernel versions. Currently for the master branch this includes RHEL6 i686/x86_64 client/server, RHEL7 x86_64 client, and SLES11 x86_64 client/server. The success/failure of these builds will be posted to the change in Gerrit as a link to the Jenkins build artifacts.

After a successful build on all of the configurations, one or more of the configurations will be regression tested automatically, and a link to the test results in Maloo will be posted to Gerrit.

Automated regression testing for each patch should normally pass.  In some cases it might fail for a variety of reasons, such as a bug in the patch, an intermittent test failure that is present in the existing code, problems with the testing system, etc.  In all failure cases, the Maloo test failure(s) should be investigated as to the root cause.  Once the test failure is identified in Maloo, the failed test result should either be linked (Associate bug) to an existing DAOS ticket in Jira, or if necessary a new Jira ticket with details of the failure should be filed (Raise bug), and a comment added to Gerrit with the JIRA DAOS ticket number.  At this point the test could be resubmitted, if there are no other problems with the patch, or it can be refreshed if there are review comments that need to be addressed (per below).

Requesting Patch Review

In order to actually get a patch landed, you need to request at least two reviews in Gerrit for the patch. This is done by visiting the Gerrit web page for the change (at the URL returned when the commit was pushed, or it can be found from your home page after logging in). Enter the name or email address for the reviewer and click the Add... button in the top right corner.

In conjunction with the two review requests, the patch has to successfully build on all supported architecture/distro combinations, and pass the automatic testing (as reported by Maloo). Once the patch has passed two reviews (+1 or +2 Review from reviewers that are not the original developer), built correctly (+1 Verified from Jenkins), and has passed autotest (+1 Verified from Maloo) the patch will automatically appear in the Gerrit Gatekeeper's list of patches to land.

It is the responsibility of the patch submitter to request the reviewers to look at the patch or to request patch landing, if this is not happening in a timely manner, by adding a comment into the patch.

Updating Patches After Review

Gerrit makes it easy to update a patch after review, and doing this allows reviewers to see differences between the patches so they only need to review the changes between the patches instead of having to review the entire patch again. Also, the original inline comments are maintained and moved to the new patches. The key to keeping updated review requests linked to the original patch is the Change-Id: field in the commit comment - this is what Gerrit uses to find the original patch to update.

A critical thing to point out is that you must submit a new version of the entire patch - not just an update to the patch.

The easiest way to update the most recent commit (which is often the one you want to update), is to use "git commit --amend -a".  This will "add" any modifications in the current repository and merge them into the last commit.  If there are no changes, or "-a" is not used, this will just allow you to edit the most recent commit message.  This is useful if you don't have a Change-Id: line in your commit message (because you didn't install the commit-msg hook, see Commit Comments above), but one was added automatically by Gerrit.  You can copy the Change-Id:line from the Gerrit web page for that change and paste it into the amended commit message.  Typically, the updated patch should also be rebased to the tip of the current branch before pushing it again, to ensure that it does not conflict with other changes that may have been landed since the patch was first developed.

Another way of modifying a series of existing changes is to use "git rebase -i <parent-branch>". Interactive rebase will display a list of patches in an editor, and you can reorder the patches, "edit" the patch, "reword" the commit comments, and "squash" two patches into a single patch. See the help text in the commit message editor window for more information. 

Once the change has been updated, push your branch to Gerrit again, and the original request will be updated with the new version of the patch.

(Re-)Basing One Change on Another Change

It is possible to base a new change, or rebase an existing change, on an uncommitted patch in Gerrit. This might be useful if both changes are impacting the same code, and one change is clearly dependent on another for some reason, or if they will cause conflicts when merged and a decision is made to land them in a specific order. It is desirable to break a patch into multiple functionally-separate commits. If the patches are truly independent (i.e. no overlapping changes to the same files, no dependent functions, etc), then those patches should be submitted in separate branches, and this section does not apply.

If both patches are being developed by the same person, the easiest way to have a series of dependent changes is to commit them into separate patches order on the same branch. Then, when a change is made to any patch on that branch, all of the dependent changes will also be resubmitted to Gerrit so that they will be ready to land when the earlier patches are merged.

If updates or fixes need to be made to one of the patches, these updates should be merged into the original commit where the code was added, rather than being an additional patch at the end of the series. This can be done by running "git rebase -i <parent-branch>", then marking a particular patch for edit. That will cause the earlier patches in the series up to the to be applied, then stop for interactive editing/testing of that patch. Once the patch has been updated, run git add -u to include the new updates into this patch, and git rebase --continue to merge the updates into the existing patch and continue the rebase process.

In the case where changes need to be based on a patch from another developer, it is possible to check out the desired patch from Gerrit using one of the supplied Git URLs in the Download section under the patchset. Select the checkout and Anonymous HTTP or SSH options, then copy the URL and paste it at the command prompt of a previously checked-out DAOS tree.  For example to fetch patchset 42 from https://review.hpdd.intel.com/31188/, create a branch for it, and then create a new branch for your local changes based on that change, use:

$ git fetch http://review.hpdd.intel.com/p/daos/daos_m refs/changes/88/31188/42 && git checkout FETCH_HEAD
$ git checkout -b b_31188
$ git checkout -b b_31188_my_changes
{edit local tree to make changes as usual}
$ git commit -a

Any changes in the b_31188_my_changes will be based on top of those of b_31188. When b_1264_my_changes is pushed to Gerrit, it will have a dependency on change 31188, so it will not be able to land until change 31188 itself is landed.  It is typically not desirable rebase the b_31188 branch to a new version of master or modify it, or it will cause the other developer's patch to be updated in Gerrit, and lose its existing review and test results.  However, in some cases this is desirable, and Gerrit allows a user other than the original author to push an updated version of the patch (using the same Change-Id: label) in case there is a problem with the original patch (e.g. no longer applies, has a bug, etc).

If you already have the changes in a local branch (e.g. b_my_changes) and want to rebase that branch on top of another uncommitted patch from Gerrit, the process is similar:

$ git fetch http://review.hpdd.intel.com/p/daos/daos_m refs/changes/88/31188/42 && git checkout FETCH_HEAD
$ git checkout -b b_31188
$ git checkout b_my_changes
$ git rebase --onto b_31188

That will rebase the patch(es) in the current branch onto the other patch (or branch) checked out from Gerrit.

Pushing a Git Tag

This is normally done by the branch maintainer, but is recorded here for future reference.

Switch to the branch you want to create a branch in.

Create the tag with:

git tag -a TAG_NAME

then push it upstream with

git push TAG_NAME

Note, this requires certain privileges on the server. If review is a secondary server for your repository, use:

git push review TAG_NAME