Skip to content

SYNC_UPSTREAM.md

Latest commit

 

History

History
319 lines (255 loc) · 10.8 KB

SYNC_UPSTREAM.md

File metadata and controls

319 lines (255 loc) · 10.8 KB

How to synchronize HCC with upstream

This section shows the step to synchronize HCC with upstream projects, such as Clang, LLVM, and LLD. The process is currently carried out manually but it could and should be automated.

Upstream Clang, LLVM, and LLD all sit in different git repositories and they are almost changed daily. Sometimes a change upstream may affect several projects, but it is usually not easy to figure it out from the commit log.

HCC has been configured to use git submodules to track these dependencies, but there are users who prefer using repo tool. This document has been revised to cope with both git submodules and repo. Maintaining repo manifest is an optional step for now.

HCC depends on amd-common LLVM / LLD / Clang, which is a fork maintained by AMD, and may contain patches not yet upstreamed. amd-common LLVM / LLD / Clang is automatically synchronized with upstream LLVM / LLD / Clang every 4 hours, so they are very close to the latest codes upstream.

There are two roles in the HCC merge process. The one who conducts the upstream sync process through getting all the amd-common code, merging it into amd-hcc, and filing the pull requests. And the one who reviews all of your sync pull requests. The HCC merge maintainer will take on the first role, and is required to interact, modify, and satisfy the changes requested by the reviewer before the pull requests can be accepted.

Detailed step-by-step instructions are in following sections.

Useful github remote repositories (For reference)

git locations of repositories used in the merge process are:

Set own forks for HCC modules

Go to these links and fork onto your personal account. If you have old forks, please remove them, or create another fork which tracks the tip of corresponding repositories.

  1. https://github.com/RadeonOpenCompute/hcc
  2. https://github.com/RadeonOpenCompute/hcc-clang-upgrade
  3. https://github.com/RadeonOpenCompute/llvm
  4. https://github.com/RadeonOpenCompute/lld
  5. https://github.com/llvm-mirror/compiler-rt
  6. https://github.com/llvm-mirror/clang-tools-extra

Note, you'd won't need to do this for rocdl, since we will use its master branch.

Step-by-step Merge Process

Assuming that we have created forks for all the repositories above, follow this:

From here on, replace your_name with your own github id

  • git clone git@github.com:your_name/hcc.git hcc_fork
  • cd hcc_fork
  • git checkout clang_tot_upgrade
  • git submodule update --init

This directory would be referred as HCC directory hereafter.

Add all remotes and forks for submodules

  • change to HCC directory
  • cd lld
  • git remote add lld_fork git@github.com:your_name/lld.git
  • cd ../compiler
  • git remote add llvm_fork git@github.com:your_name/llvm.git
  • cd ../compiler-rt
  • git remote add compiler-rt_fork git@github.com:your_name/compiler-rt.git
  • git remote add compiler-rt https://github.com/llvm-mirror/compiler-rt
  • cd ../clang-tools-extra
  • git remote add clang-tools-extra_fork git@github.com:your_name/clang-tools-extra.git
  • git remote add clang-tools-extra https://github.com/llvm-mirror/clang-tools-extra
  • cd ../clang
  • git remote add clang_fork git@github.com:your_name/hcc-clang-upgrade.git
  • git remote add clang https://github.com/RadeonOpenCompute/clang

Use git remote -v to verify the new remotes are added.

Checkout latest master ROCm-Device-Libs

  • change to HCC directory
  • cd rocdl
  • git fetch --all
  • git checkout origin/master

Merge amd-common LLD commits

From here, replace YYYYMMDD with your date.

  • change to HCC directory
  • cd lld
  • git fetch --all
  • git checkout -b merge_YYYYMMDD origin/amd-hcc
  • git merge origin/amd-common --no-edit
  • Resolve any merge conflicts encountered here

Merge amd-common LLVM commits

  • change to HCC directory
  • cd compiler
  • git fetch --all
  • git checkout -b merge_YYYYMMDD origin/amd-hcc
  • git merge origin/amd-common --no-edit
  • Resolve any merge conflicts encountered here

Merge amd-common COMPILER-RT commits from llvm-mirror

  • change to HCC directory
  • cd compiler-rt
  • git fetch --all
  • git checkout -b merge_YYYYMMDD origin/amd-hcc
  • git merge --no-ff compiler-rt/master --no-edit
  • Resolve any merge conflicts encountered here

Merge amd-common CLANG-TOOLS-EXTRA commits from llvm-mirror

  • change to HCC directory
  • cd clang-tools-extra
  • git fetch --all
  • git checkout -b merge_YYYYMMDD origin/amd-hcc
  • git merge clang-tools-extra/master --no-edit
  • Resolve any merge conflicts encountered here

Merge amd-common Clang commits

  • change to HCC directory
  • cd clang
  • git fetch --all
  • git checkout -b merge_YYYYMMDD origin/clang_tot_upgrade
  • git merge --no-ff clang/amd-common --no-edit
  • Resolve any merge conflicts encountered here

Build merged HCC

  • change to HCC directory
  • cd ..; mkdir -p build; cd build
  • cmake -DCMAKE_BUILD_TYPE=Release ../hcc_fork
  • make -j$(nproc)

Fix any compilation failures if there is any. Repeat this step until HCC can be built.

Quick sanity tests on merged HCC

Assume commands below are carried out in HCC build directory. And HCC directory is at ~/hcc_fork.

Test with one C++AMP FP math unit test.

compiler/bin/hcc `bin/clamp-config --build --cxxflags --ldflags` -lm \
  ~/hcc_fork/tests/Unit/AmpMath/amp_math_cos.cpp
./a.out ; echo $?

Test with one grid_launch unit test with AM library usage.

compiler/bin/hcc `bin/hcc-config --build --cxxflags --ldflags` -lhc_am \
  ~/hcc_fork/tests/Unit/GridLaunch/glp_const.cpp
./a.out ; echo $?

Test with one HC unit test with atomic function and 64-bit arithmetic.

compiler/bin/hcc `bin/hcc-config --build --cxxflags --ldflags` \
  ~/hcc_fork/tests/Unit/HC/hc_atomic_add_global.cpp
./a.out ; echo $?

Full sanity tests on merged HCC

Test all unit tests.

  • change to HCC build directory
  • make test

Resolve any issues encountered here.

Push and Create Pull Request for HCC Clang submodule

Push and Create Pull Request for amd-hcc LLD submodule

Push and Create Pull Request for amd-hcc LLVM submodule

Push and Create Pull Request for amd-hcc COMPILER-RT submodule

  • change to HCC directory
  • cd compiler-rt
  • git push compiler-rt_fork merge_YYYYMMDD:merge_YYYYMMDD Create a pull request to merge merge_YYYYMMDD from your compiler-rt fork into https://github.com/RadeonOpenCompute/compiler-rt on branch amd-hcc.

Push and Create Pull Request for amd-hcc CLANG-TOOLS-EXTRA submodule

*** Wait until all Pull Requests are approved and merged. *** On github when making a pull request, the repository you are trying to update will have a list of reviewers who have the authority to approve pull requests and merge them into the repositories. During the review period, you are required to interact with the reviewers.

For the pull requests to be approved, you must wait for a reviewer to accept the changes you are making. The reviewer will request changes if the code is not ready for merge. It is your responsibility to interact and resolve all conflicts so that the PR is ready for merge.

Update HCC git submodules configuration

  • change to HCC directory

  • git checkout origin/clang_tot_upgrade

  • cd clang

  • git fetch --all

  • git checkout origin/clang_tot_upgrade

  • cd ../compiler

  • git fetch --all

  • git checkout origin/amd-hcc

  • cd ../compiler-rt

  • git fetch --all

  • git checkout origin/amd-hcc

  • cd ../clang-tools-extra

  • git fetch --all

  • git checkout origin/amd-hcc

  • cd ../lld

  • git fetch --all

  • git checkout origin/amd-hcc

  • cd ../rocdl

  • git fetch --all

  • git checkout origin/master

  • change to HCC directory

  • git checkout -b merge_YYYYMMDD clang_tot_upgrade

  • git add clang clang-tools-extra compiler compiler-rt lld rocdl

  • git commit -m "[Config] revise submodule configuration", or provide custom commit log

  • git push origin merge_YYYYMMDD:merge_YYYYMMDD Create a pull request to merge merge_YYYYMMDD from your hcc fork into https://github.com/RadeonOpenCompute/hcc on branch clang_tot_upgrade.

Upon reaching here, the merge process is completed.

(Optional) Update HCC repo manifest

Clone HCC with repo

  • Create a directory called 'repo'. It would be called as HCC Repo directory hereafter.
  • repo init -u https://github.com/RadeonOpenCompute/HCC-Native-GCN-ISA.git
  • repo sync

Clone HCC repo manifest

A manifest directory is created to track working commits for all components.

  • git clone https://github.com/RadeonOpenCompute/HCC-Native-GCN-ISA.git manifest

This directory would be referred as HCC manifest directory hereafter.

Initialize git submodules

  • change to HCC Repo directory
  • cd hcc
  • git checkout clang_tot_upgrade
  • git pull
  • git submodule update --init

Update HCC repo manifest

  • repo manifest -r > default.xml
  • copy default.xml to HCC manifest directory
  • change too HCC manifest directory
  • git add default.xml
  • git commit -m "[Config] revise manifest", or provide custom commit log
  • git push to push HCC repo manifest online