phantom/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2025-01-25 02:33:24 -03:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge #20619: guix: Quality of life improvements

Browse source 
570e43fe72 guix: Print build params inside/outside of container (Carl Dong)
2f9d1fdde6 guix: Move DISTSRC determination to guix-build.sh (Carl Dong)
0b7cd07bb5 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong)
d27ff8b86a guix: Add more sanity checks to guix-build.sh (Carl Dong)
57f9533146 guix: Add section headings to guix-build.sh (Carl Dong)
38b7b2ed72 genbuild: Specify rev-parse length (Carl Dong)
036dc740da docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong)
34f0fda2d3 guix: Small updates to README wording (Carl Dong)
402e3a5b1e guix: Update HOSTS README entry for new architectures (Carl Dong)
cfa7ceb21b guix: Remove README development environment section (Carl Dong)
93b6a8544a guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong)
0f31e24703 guix: Add SUBSTITUTE_URLS option (Carl Dong)
444fcfca90 guix: Make guix honor MAX_JOBS setting (Carl Dong)

Pull request description:

  After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them.

  Notable changes:
  1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment
  2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules
  3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process.
  4. README cleanup

ACKs for top commit:
  fanquake:
    ACK 570e43fe72 - lets move this forward.

Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
This commit is contained in:
fanquake 2021-01-12 18:52:42 +08:00
commit 18017152c2
No known key found for this signature in database
GPG key ID: 2EEB9F5CC09526C1
5 changed files with 210 additions and 56 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

76
contrib/guix/README.md
View file

@ -40,25 +40,27 @@ Otherwise, follow the [Guix installation guide][guix/bin-install].
Guix allows us to achieve better binary security by using our CPU time to build
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
this: users can decide whether or not to bootstrap and to use substitutes.
this: users can decide whether or not to bootstrap and to use substitutes
(pre-built packages).
After installation, you may want to consider [adding substitute
servers](#speeding-up-builds-with-substitute-servers) to speed up your build if
that fits your security model (say, if you're just testing that this works).
This is skippable if you're using the [Dockerfile][fanquake/guix-docker].
servers](#speeding-up-builds-with-substitute-servers) from which to download
pre-built packages to speed up your build if that fits your security model (say,
if you're just testing that this works). Substitute servers are set up by
default if you're using the [Dockerfile][fanquake/guix-docker].
If you prefer not to use any substitutes, make sure to set
`ADDITIONAL_GUIX_ENVIRONMENT_FLAGS` like the following snippet. The first build
will take a while, but the resulting packages will be cached for future builds.
If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
like in the following snippet. The first build will take a while, but the
resulting packages will be cached for future builds.
```sh
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--no-substitutes'
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
```
Likewise, to perform a bootstrapped build (takes even longer):
```sh
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap --no-substitutes'
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap'
```
### Using a version of Guix with `guix time-machine` capabilities
@ -82,17 +84,6 @@ export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
## Usage
### As a Development Environment
For a Bitcoin Core depends development environment, simply invoke
```sh
guix environment --manifest=contrib/guix/manifest.scm
```
And you'll land back in your shell with all the build dependencies required for
a `depends` build injected into your environment.
### As a Tool for Deterministic Builds
From the top of a clean Bitcoin Core repository:
@ -113,10 +104,8 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
* _**HOSTS**_
Override the space-separated list of platform triples for which to perform a
bootstrappable build. _(defaults to "x86\_64-linux-gnu
arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu")_
> Windows and OS X platform triplet support are WIP.
bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf
aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32")_
* _**SOURCES_PATH**_
@ -147,13 +136,29 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
string) is interpreted the same way as not setting `V` at all, and that `V=0`
has the same effect as `V=1`.
* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
* _**SUBSTITUTE_URLS**_
Additional flags to be passed to `guix environment`. For a fully-bootstrapped
A whitespace-delimited list of URLs from which to download pre-built packages.
A URL is only used if its signing key is authorized (refer to the [substitute
servers section](#speeding-up-builds-with-substitute-servers) for more
details).
* _**ADDITIONAL_GUIX_COMMON_FLAGS**_
Additional flags to be passed to all `guix` commands. For a fully-bootstrapped
build, set this to `--bootstrap --no-substitutes` (refer to the [security
model section](#choosing-your-security-model) for more details). Note that a
fully-bootstrapped build will take quite a long time on the first run.
* _**ADDITIONAL_GUIX_TIMEMACHINE_FLAGS**_
Additional flags to be passed to `guix time-machine`.
* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
Additional flags to be passed to the invocation of `guix environment` inside
`guix time-machine`.
## Tips and Tricks
### Speeding up builds with substitute servers
@ -161,14 +166,15 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
_This whole section is automatically done in the convenience
[Dockerfiles][fanquake/guix-docker]_
For those who are used to life in the fast _(and trustful)_ lane, you can use
[substitute servers][guix/substitutes] to enable binary downloads of packages.
For those who are used to life in the fast _(and trustful)_ lane, you can
specify [substitute servers][guix/substitutes] from which to download pre-built
packages.
> For those who only want to use substitutes from the official Guix build farm
> and have authorized the build farm's signing key during Guix's installation,
> you don't need to do anything.
#### Authorize the signing keys
#### Step 1: Authorize the signing keys
For the official Guix build farm at https://ci.guix.gnu.org, run as root:
@ -182,7 +188,7 @@ For dongcarl's substitute server at https://guix.carldong.io, run as root:
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
```
#### Use the substitute servers
#### Step 2: Specify the substitute servers
The official Guix build farm at https://ci.guix.gnu.org is automatically used
unless the `--no-substitutes` flag is supplied.
@ -196,7 +202,7 @@ To use dongcarl's substitute server for Bitcoin Core builds after having
[authorized his signing key](#authorize-the-signing-keys):
```
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"'
export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
```
## FAQ
@ -212,9 +218,9 @@ As mentioned at the bottom of [this manual page][guix/bin-install]:
### When will Guix be packaged in debian?
Vagrant Cascadian has been making good progress on this
[here][debian/guix-package]. We have all the pieces needed to put up an APT
repository and will likely put one up soon.
Thanks to Vagrant Cascadian's diligent work, Guix is now [in debian
experimental][debian/guix-experimental]! Hopefully it will make its way into a
release soon.
[b17e]: http://bootstrappable.org/
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
@ -226,5 +232,5 @@ repository and will likely put one up soon.
[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
[guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html
[debian/guix-package]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644
[debian/guix-experimental]: https://packages.debian.org/experimental/guix
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix

159
contrib/guix/guix-build.sh
View file

@ -2,6 +2,116 @@
export LC_ALL=C
set -e -o pipefail
###################
## Sanity Checks ##
###################
################
# Check 1: Make sure that we can invoke required tools
################
for cmd in git make guix cat mkdir; do
if ! command -v "$cmd" > /dev/null 2>&1; then
echo "ERR: This script requires that '$cmd' is installed and available in your \$PATH"
exit 1
fi
done
################
# Check 2: Make sure GUIX_BUILD_OPTIONS is empty
################
#
# GUIX_BUILD_OPTIONS is an environment variable recognized by guix commands that
# can perform builds. This seems like what we want instead of
# ADDITIONAL_GUIX_COMMON_FLAGS, but the value of GUIX_BUILD_OPTIONS is actually
# _appended_ to normal command-line options. Meaning that they will take
# precedence over the command-specific ADDITIONAL_GUIX_<CMD>_FLAGS.
#
# This seems like a poor user experience. Thus we check for GUIX_BUILD_OPTIONS's
# existence here and direct users of this script to use our (more flexible)
# custom environment variables.
if [ -n "$GUIX_BUILD_OPTIONS" ]; then
cat << EOF
Error: Environment variable GUIX_BUILD_OPTIONS is not empty:
'$GUIX_BUILD_OPTIONS'
Unfortunately this script is incompatible with GUIX_BUILD_OPTIONS, please unset
GUIX_BUILD_OPTIONS and use ADDITIONAL_GUIX_COMMON_FLAGS to set build options
across guix commands or ADDITIONAL_GUIX_<CMD>_FLAGS to set build options for a
specific guix command.
See contrib/guix/README.md for more details.
EOF
exit 1
fi
################
# Check 3: Make sure that we're not in a dirty worktree
################
if ! git diff-index --quiet HEAD -- && [ -z "$FORCE_DIRTY_WORKTREE" ]; then
cat << EOF
ERR: The current git worktree is dirty, which may lead to broken builds.
Aborting...
Hint: To make your git worktree clean, You may want to:
1. Commit your changes,
2. Stash your changes, or
3. Set the 'FORCE_DIRTY_WORKTREE' environment variable if you insist on
using a dirty worktree
EOF
exit 1
else
GIT_COMMIT=$(git rev-parse --short=12 HEAD)
fi
################
# Check 4: Make sure that build directories do no exist
################
# Default to building for all supported HOSTs (overridable by environment)
export HOSTS="${HOSTS:-x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu
x86_64-w64-mingw32}"
DISTSRC_BASE="${DISTSRC_BASE:-${PWD}}"
# Usage: distsrc_for_host HOST
#
# HOST: The current platform triple we're building for
#
distsrc_for_host() {
echo "${DISTSRC_BASE}/distsrc-${GIT_COMMIT}-${1}"
}
# Accumulate a list of build directories that already exist...
hosts_distsrc_exists=""
for host in $HOSTS; do
if [ -e "$(distsrc_for_host "$host")" ]; then
hosts_distsrc_exists+=" ${host}"
fi
done
if [ -n "$hosts_distsrc_exists" ]; then
# ...so that we can print them out nicely in an error message
cat << EOF
ERR: Build directories for this commit already exist for the following platform
triples you're attempting to build, probably because of previous builds.
Please remove, or otherwise deal with them prior to starting another build.
Aborting...
EOF
for host in $hosts_distsrc_exists; do
echo " ${host} '$(distsrc_for_host "$host")'"
done
exit 1
else
mkdir -p "$DISTSRC_BASE"
fi
#########
# Setup #
#########
# Determine the maximum number of jobs to run simultaneously (overridable by
# environment)
MAX_JOBS="${MAX_JOBS:-$(nproc)}"
@ -16,11 +126,23 @@ SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
# Execute "$@" in a pinned, possibly older version of Guix, for reproducibility
# across time.
time-machine() {
# shellcheck disable=SC2086
guix time-machine --url=https://github.com/dongcarl/guix.git \
--commit=b066c25026f21fb57677aa34692a5034338e7ee3 \
--max-jobs="$MAX_JOBS" \
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
-- "$@"
}
# Make sure an output directory exists for our builds
OUTDIR="${OUTDIR:-${PWD}/output}"
[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR"
#########
# Build #
#########
# Function to be called when building for host ${1} and the user interrupts the
# build
int_trap() {
@ -38,9 +160,9 @@ and untracked files and directories will be wiped, allowing you to start anew.
EOF
}
# Deterministically build Bitcoin Core for HOSTs (overridable by environment)
# Deterministically build Bitcoin Core
# shellcheck disable=SC2153
for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32}; do
for host in $HOSTS; do
# Display proper warning when the user interrupts the build
trap 'int_trap ${host}' INT
@ -50,6 +172,19 @@ for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv
# for the particular $HOST we're building for
export HOST="$host"
# shellcheck disable=SC2030
cat << EOF
INFO: Building commit ${GIT_COMMIT:?not set} for platform triple ${HOST:?not set}:
...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
...running at most ${MAX_JOBS:?not set} jobs
...from worktree directory: '${PWD}'
...bind-mounted in container to: '/bitcoin'
...in build directory: '$(distsrc_for_host "$HOST")'
...bind-mounted in container to: '$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")'
...outputting in: '${OUTDIR:?not set}'
...bind-mounted in container to: '/outdir'
EOF
# Run the build script 'contrib/guix/libexec/build.sh' in the build
# container specified by 'contrib/guix/manifest.scm'.
#
@ -99,20 +234,36 @@ for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv
# make the downloaded depends sources available to it. The sources
# should have been downloaded prior to this invocation.
#
# shellcheck disable=SC2086
# ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"}
#
# fetch substitute from SUBSTITUTE_URLS if they are
# authorized
#
# Depending on the user's security model, it may be desirable to use
# substitutes (pre-built packages) from servers that the user trusts.
# Please read the README.md in the same directory as this file for
# more information.
#
# shellcheck disable=SC2086,SC2031
time-machine environment --manifest="${PWD}/contrib/guix/manifest.scm" \
--container \
--pure \
--no-cwd \
--share="$PWD"=/bitcoin \
--share="$DISTSRC_BASE"=/distsrc-base \
--share="$OUTDIR"=/outdir \
--expose="$(git rev-parse --git-common-dir)" \
${SOURCES_PATH:+--share="$SOURCES_PATH"} \
${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
--max-jobs="$MAX_JOBS" \
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
-- env HOST="$host" \
MAX_JOBS="$MAX_JOBS" \
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
${V:+V=1} \
${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
DISTSRC="$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")" \
OUTDIR=/outdir \
bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh"
)

26
contrib/guix/libexec/build.sh
View file

@ -11,9 +11,15 @@ if [ -n "$V" ]; then
export VERBOSE="$V"
fi
# Check that environment variables assumed to be set by the environment are set
echo "Building for platform triple ${HOST:?not set} with reference timestamp ${SOURCE_DATE_EPOCH:?not set}..."
echo "At most ${MAX_JOBS:?not set} jobs will run at once..."
# Check that required environment variables are set
cat << EOF
Required environment variables as seen inside the container:
HOST: ${HOST:?not set}
SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH:?not set}
MAX_JOBS: ${MAX_JOBS:?not set}
DISTSRC: ${DISTSRC:?not set}
OUTDIR: ${OUTDIR:?not set}
EOF
#####################
# Environment Setup #
@ -23,19 +29,6 @@ echo "At most ${MAX_JOBS:?not set} jobs will run at once..."
# $HOSTs after successfully building.
BASEPREFIX="${PWD}/depends"
# Setup an output directory for our build
OUTDIR="${OUTDIR:-${PWD}/output}"
[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR"
# Setup the directory where our Bitcoin Core build for HOST will occur
DISTSRC="${DISTSRC:-${PWD}/distsrc-${HOST}}"
if [ -e "$DISTSRC" ]; then
echo "DISTSRC directory '${DISTSRC}' exists, probably because of previous builds... Aborting..."
exit 1
else
mkdir -p "$DISTSRC"
fi
# Given a package name and an output name, return the path of that output in our
# current guix environment
store_path() {
@ -189,6 +182,7 @@ esac
# Make $HOST-specific native binaries from depends available in $PATH
export PATH="${BASEPREFIX}/${HOST}/native/bin:${PATH}"
mkdir -p "$DISTSRC"
(
cd "$DISTSRC"

3
doc/guix.md Normal file
View file

@ -0,0 +1,3 @@
# Bootstrappable Bitcoin Core Builds
See [contrib/guix/README.md](../contrib/guix/README.md)

2
share/genbuild.sh
View file

@ -31,7 +31,7 @@ if [ "${BITCOIN_GENBUILD_NO_GIT}" != "1" ] && [ -e "$(command -v git)" ] && [ "$
fi
# otherwise generate suffix from git, i.e. string like "59887e8-dirty"
GIT_COMMIT=$(git rev-parse --short HEAD)
GIT_COMMIT=$(git rev-parse --short=12 HEAD)
git diff-index --quiet HEAD -- || GIT_COMMIT="$GIT_COMMIT-dirty"
fi