======================================================================
ERROR: test_revocation_mode_soft (tests.test_validate.ValidateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/test_validate.py", line 85, in test_revocation_mode_soft
validate_path(context, path)
File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/../certvalidator/validate.py", line 50, in validate_path
return _validate_path(validation_context, path)
File "/tmp/guix-build-python-certvalidator-0.1-1.e5bdb4b.drv-0/source/tests/../certvalidator/validate.py", line 358, in _validate_path
raise PathValidationError(pretty_message(
certvalidator.errors.PathValidationError: The path could not be validated because the end-entity certificate expired 2022-01-14 12:00:00Z
2f356a0ca8 scripted-diff: Drop Darwin version for better maintainability (Hennadii Stepanov)
Pull request description:
After this PR, any macOS tools version bumping in the future will touch fewer files in the repo.
Pointing a Darwin version for the `--host` system does not matter for the following reasons:
- in terms of the resulted binaries, we should only care about the minimum supported macOS version which is a separated parameter in our build system.
- in terms of the build system itself, the usage of the `$(host)` variable is self-consistent enough. Btw `$(host_os)` value already has the version dropped:
```
$ make -C depends --no-print-directory print-host_os HOST=x86_64-apple-darwin19
host_os=darwin
```
ACKs for top commit:
gruve-p:
ACK 2f356a0ca8
promag:
ACK 2f356a0ca8.
fanquake:
ACK 2f356a0ca8
Tree-SHA512: 374896ab0ba02b0d8b4b21431fe963bd213b0d09586e0898c13a4c5fa294c1b693f1b2c92880c245c4157c14217b4825b36522f461930477f4d2a727086ebb2a
a3f61676e8 test: Make more shell scripts verifiable by the `shellcheck` tool (Hennadii Stepanov)
Pull request description:
Some shell scripts from `contrib/guix` and `contrib/shell` are not verifiable by the `shellcheck` tool for the following reasons:
- they have no extension (see 4eccf063b2 from bitcoin/bitcoin#21375)
- they have the `.bash` extension while `.sh` is expected
This PR adds these scripts to the input for the `shellcheck` tool, and it fixes discovered `shellcheck` warnings.
ACKs for top commit:
dongcarl:
Code Review ACK a3f61676e8, this is a good robustness improvement for our shell scripts.
jamesob:
crACK a3f61676e8
Tree-SHA512: 6703f5369d9c04c1a174491f381afa5ec2cc4d37321c1b93615abcdde4dfd3caae82868b699c25b72132d8c8c6f2e9cf24d38eb180ed4d0f0584d8c282e58935
From what I can see the only platform this drops support for is CentOS
7. CentOS 7 reached the end of it's "full update" support at the end of
2020. It does receive maintenance updates until 2024, however I don't
think supporting glibc 2.17 until 2024 is realistic. Note that anyone
wanting to self-compile and target a glibc 2.17 runtime could build with
--disable-threadlocal.
glibc 2.18 was released in August 2013.
https://sourceware.org/legacy-ml/libc-alpha/2013-08/msg00160.html
0f95247246 Integrate univalue into our buildsystem (Cory Fields)
9b49ed656f Squashed 'src/univalue/' changes from 98fadc0909..a44caf65fe (fanquake)
Pull request description:
This PR more tightly integrates building Univalue into our build system. This follows the same approach we use for [LevelDB](https://github.com/bitcoin-core/leveldb/), ([`Makefile.leveldb.include`](https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.leveldb.include)), and [CRC32C](https://github.com/bitcoin-core/crc32c) ([`Makefile.crc32c.include`](https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.crc32c.include)), and will be the same approach we use for [minisketch](https://github.com/sipa/minisketch); see #23114.
This approach yields a number of benefits, including:
* Faster configuration due to one less subconfigure being run during `./configure` i.e 22s with this PR vs 26s
* Faster autoconf i.e 13s with this PR vs 17s
* Improved caching
* No more issues with compiler flags i.e https://github.com/bitcoin/bitcoin/pull/12467
* More direct control means we can build exactly the objects we want
There might be one argument against making this change, which is that builders should have the option to use "proper shared/system libraries". However, I think that falls down for a few reasons. The first being that we already don't support building with a number of system libraries (secp256k1, leveldb, crc32c); some for good reason. Univalue is really the odd one out at the moment.
Note that the only fork of Core I'm aware of, that actively patches in support for using system libs, also explicitly marks them as ["DANGEROUS"](a886811721/configure.ac (L1430)) and ["NOT SUPPORTED"](a886811721/configure.ac (L1312)). So it would seem they exist more to satisfy a distro requirement, as opposed to something that anyone should, or would actually use in practice.
PRs like #22412 highlight the "issue" with us operating with our own Univalue fork, where we actively fix bugs, and make improvements, when upstream (https://github.com/jgarzik/univalue) may not be taking those improvements, and by all accounts, is not currently actively maintained. Bitcoin Core should not be hamstrung into not being able to fix bugs in a library, and/or have to litter our source with "workarounds", i.e #22412, for bugs we've already fixed, based on the fact that an upstream project is not actively being maintained. Allowing builders to use system libs is really only exacerbating this problem, with little benefit to our project. Bitcoin Core is not quite like your average piece of distro packaged software.
There is the potential for us to give the same treatment to libsecp256k1, however it seems doing that is currently less straightforward.
ACKs for top commit:
dongcarl:
ACK 0f95247246 less my comment above, always nice to have an include-able `sources.mk` which makes integration easier.
theuni:
ACK 0f95247246. Thanks fanquake for keeping this going.
Tree-SHA512: a7f2e41ee7cba06ae72388638e86b264eca1b9a8b81c15d1d7b45df960c88c3b91578b4ade020f8cc61d75cf8d16914575f9a78fa4cef9c12be63504ed804b99
I used Guix's values for the powerpc64(le) dynamic linkers, and the
/lib-prefix seems to be a Guix-ism rather than standard. The standard
path for the linker-loaders start with /lib64.
I've taken the new loader values from SYSDEP_KNOWN_INTERPRETER_NAMES in
glibc's sysdeps/unix/sysv/linux/powerpc/ldconfig.h file.
For future reference, loader path values can also be found on glibc's
website: https://sourceware.org/glibc/wiki/ABIList?action=recall&rev=16
This addresses issues like the one in #12467, where some of our compiler flags
end up being dropped during the subconfigure of Univalue. Specifically, we're
still using the compiler-default c++ version rather than forcing c++17.
We can drop the need subconfigure completely in favor of a tighter build
integration, where the sources are listed separately from the build recipes,
so that they may be included directly by upstream projects. This is
similar to the way leveldb build integration works in Core.
Core benefits of this approach include:
- Better caching (for ex. ccache and autoconf)
- No need for a slow subconfigure
- Faster autoconf
- No more missing compile flags
- Compile only the objects needed
There are no benefits to Univalue itself that I can think of. These changes
should be a no-op there, and to downstreams as well until they take advantage
of the new sources.mk.
This also removes the option to use an external univalue to avoid similar ABI
issues with mystery binaries.
Co-authored-by: fanquake <fanquake@gmail.com>
a43b8e9555 build: set OSX_MIN_VERSION to 10.15 (fanquake)
Pull request description:
Taken out of #20744, as splitting up some of the build changes was mentioned [here](https://github.com/bitcoin/bitcoin/pull/22937#discussion_r707303172).
This is required to use `std::filesystem` on macOS, as support for it only landed in the libc++.dylib shipped with 10.15. So if we want to move to using `std::filesystem` for `23.0`, this bump is required.
See also: https://developer.apple.com/documentation/xcode-release-notes/xcode-11-release-notes
> Clang now supports the C++17 \<filesystem\> library for iOS 13, macOS 10.15, watchOS 6, and tvOS 13.
macOS 10.15 was released in October 2019. macOS OS's seem to have a life of about 3 years, so it's possible that 10.14 will become officially unsupported by the end of 2021 and prior to the release of 23.0.
Guix builds:
```bash
bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
abc8b749be65f1339dcdf44bd1ed6ade2533b8e3b5030ad1dde0ae0cede78136 guix-build-a43b8e955558/output/dist-archive/bitcoin-a43b8e955558.tar.gz
1edcc301eb4c02f3baa379beb8d4c78e661abc24a293813bc9d900cf7255b790 guix-build-a43b8e955558/output/x86_64-apple-darwin19/SHA256SUMS.part
e9dbb5594a664519da778dde9ed861c3f0f631525672e17a67eeda599f16ff44 guix-build-a43b8e955558/output/x86_64-apple-darwin19/bitcoin-a43b8e955558-osx-unsigned.dmg
11b23a17c630dddc7594c25625eea3de42db50f355733b9ce9ade2d8eba3a8f3 guix-build-a43b8e955558/output/x86_64-apple-darwin19/bitcoin-a43b8e955558-osx-unsigned.tar.gz
257ba64a327927f94d9aa0a68da3a2695cf880b3ed1a0113c5a966dcc426eb5e guix-build-a43b8e955558/output/x86_64-apple-darwin19/bitcoin-a43b8e955558-osx64.tar.gz
```
ACKs for top commit:
hebasto:
ACK a43b8e9555
jarolrod:
ACK a43b8e9
Tree-SHA512: 9ac77be7cb56c068578860a3b2b8b7487c9e18b71b14aedd77a9c663f5d4bb19756d551770c02ddd12f1797beea5757b261588e7b67fb53509bb998ee8022369
Previously, if the builder exported $VERSION in their environment (as
past Gitian-building docs told them to), but their HEAD does not
actually point to v$VERSION, their build outputs will differ from those
of other builders.
This is because the contrib/guix/guix-* scripts only ever act on the
current git worktree, and does not try to check out $VERSION if $VERSION
is set in the environment.
Setting $VERSION only makes the scripts pretend like the current
worktree is $VERSION.
This problem was seen in jonatack's attestation for all.SHA256SUMS,
where only his bitcoin-22.0rc3-osx-signed.dmg differed from everyone
else's.
Here is my deduced sequence of events:
1. Aug 27th: He guix-builds 22.0rc3 and uploads his attestations up to
guix.sigs
2. Aug 30th, sometime after POSIX time 1630310848: he pulls the latest
changes from master in the same worktree where he guix-built 22.0rc3
and ends up at 7be143a960
3. Aug 30th, sometime before POSIX time 1630315907: With his worktree
still on 7be143a960, he guix-codesigns. Normally, this would result
in outputs going in guix-build-7be143a960e2, but he had
VERSION=22.0rc3 in his environment, so the guix-* scripts pretended
like he was building 22.0rc3, and used 22.0rc3's guix-build directory
to locate un-codesigned outputs and dump codesigned ones.
However, our SOURCE_DATE_EPOCH defaults to the POSIX time of HEAD
(7be143a960), which made all timestamps in the resulting codesigned
DMG 1630310848, 7be143a960e2's POSIX timestamp. This differs from the
POSIX timestamp of 22.0rc3, which is 1630348517. Note that the
windows codesigning procedure does not consider SOURCE_DATE_EPOCH.
We resolve this by only allowing VERSION overrides via the FORCE_VERSION
environment variable.
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.
Co-authored-by: Carl Dong <contact@carldong.me>
90b3e482e9 release: Release with separate SHA256SUMS and sig files (Carl Dong)
Pull request description:
This allows us to:
- remove the rfc4880 EOL hacks, and
- release with a SHA256SUMS.asc file that's a combination of all signer signatures
ACKs for top commit:
achow101:
ACK 90b3e482e9
laanwj:
Concept and code review ACK 90b3e482e9
Tree-SHA512: 5d5086063d303aa0cbd590e5fdf2ae8f555e25f4e43bf67545e33384449b990e94834c711622530ad0eb3dcc83f52746884a5081dadb0acff8dd799cfadafac7
9b313dfef1 guix: Ensure EPOCH_SOURCE_DATE does not include GPG information (Andrew Chow)
43225f0a2a guix: Remove extra \r from all.SHA256SUMS line ending (Andrew Chow)
d080c27066 guix, doc: Add a note that codesigners need to rebuild after tagging (Andrew Chow)
4a466388a0 guix: Allow changing the base manifest in guix-verify (Andrew Chow)
33455c7696 guix: Make all.SHA256SUMS rather than codesigned.SHA256SUMS (Andrew Chow)
Pull request description:
`guix-verify` expects `all.SHA256SUMS` but `guix-attest` produces `codesigned.SHA256SUMS`. Since `all.SHA256SUMS` makes more sense (as the file contains all the sha256sums, not just the codesigned ones), `guix-attest` has been changed to output a file of that name.
As a quality of life improvement, `guix-verify` can take `SIGNER` and use the signer's manifest as the base to compare against. This makes it easier to compare a single person's attestations with everyone else's and can make it more obvious when one builder is clearly mismatching with everyone else.
Lastly `release-process.md` is updated with a note about a gotcha that can cause a mismatch in the codesigned attestation.
ACKs for top commit:
fanquake:
ACK 9b313dfef1
Tree-SHA512: 0d60627def38288dbd3059ad1e72cad224f9205da11b1a561c082ef28250a074df5cc5f2797c91a7be027bc486a3fda3319c2e496a8724e5b539337236c6f990
If the user has set log.showSignature=true in their git config, then the
git log will always output GPG signature information. Since git log is
used to set EPOCH_SOURCE_DATE, this will mistakenly have GPG signature
information in it which causes issues for the build. To avoid this
issue, we override the config and force log.showSignature=false.
guix-attest mistakenly added an extra \r to the line endings in
all.SHA256SUMS, causing guix-verify to erroneously fail.
Co-Authored-By: Carl Dong <contact@carldong.me>
9f01feda0a guix/build: Remove vestigial SKIPATTEST.TAG (Carl Dong)
Pull request description:
No longer needed or referenced by anything. A relic from prior to the great hierarchy overhaul of #22182
ACKs for top commit:
achow101:
ACK 9f01feda0a
fanquake:
ACK 9f01feda0a
Tree-SHA512: a94cf63f0c5cb8dbacf1025b6c0e81b219c2a3c93b3cbcefc239ccde29e602ecd4b717b1d93dbe53cb791a5017236fb09823c034aec42b0c31894fc9e0ab8b21
e6a94d4446 guix: Bump to version-1.3.0 from upstream (Carl Dong)
90fd13b954 guix: Pin kernel header version (Carl Dong)
Pull request description:
```
- Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support)
- Use 4.9 for all others (second-oldest LTS release, released in
combination with glibc glibc 2.24 in Debian stretch)
```
```
The chosen commit is the HEAD of Guix's version-1.3.0 branch as of July
15th, 2021.
Also fix visual indenting.
```
-----
This + the documentation PR should make our Guix system ready for release!
ACKs for top commit:
MarcoFalke:
review ACK e6a94d4446 to change to vanilla guix. Did not review the kernel change.
laanwj:
ACK e6a94d4446
fanquake:
ACK e6a94d4446
Tree-SHA512: a175e4ddb3ee786a39f5e800ce336932ad2f6797a3a28400a6f723875d0f19833fd36cedc41b3580e4604110517211bd9f557be36adf7265fd8e591c434ae032
fac4814106 doc/release-process: Add torrent creation details (Carl Dong)
5d24cc3d82 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong)
5da2ee49d5 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong)
318c60700b guix: Adapt release-process.md to new Guix process (Carl Dong)
fcab35b229 guix-attest: Produce and sign normalized documents (Carl Dong)
c2541fd0ca guix: Overhaul README (Carl Dong)
46ce6ce378 tree-wide: Rename gitian-keys to builder-keys (Carl Dong)
fc4f8449f3 guix: Update various check_tools lists (Carl Dong)
263220a85c guix: Check for a sane services database (Carl Dong)
Pull request description:
Based on: #21462
Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier.
Current TODOs:
- [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up
- [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login
- [x] Fanquake docker option: complete section
- [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section
- [x] Building from source option: Insert dependency tree diagram that I made
- [x] Building from source option: redo sectioning, kind of a mess right now
- [x] Optional setup: make clear which parts are only needed if building from source
- [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key
- [x] Overall (after everything): Make the links work.
Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option.
ACKs for top commit:
fanquake:
ACK fac4814106 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release.
Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
That way we can easily combine the document and detached signature to
produce cleartext signature files for upload during the release process.
See subsequent commits which modify doc/release-process.md for more
details.
1edddf5de4 Avoid GCC 7.1 ABI change warning in guix build (Pieter Wuille)
Pull request description:
The arm-linux-gnueabihf guix build output is littered with warnings like:
```
/gnu/store/7a96hdqdb2qi8a39f09n84xjy2hr23rs-gcc-cross-arm-linux-gnueabihf-8.4.0/include/c++/bits/stl_vector.h:1085:4: note:
parameter passing for argument of type '__gnu_cxx::__normal_iterator<CRecipient*, std::vector<CRecipient> >' changed in GCC 7.1
```
These are irrelevant for us. Disable them using `-Wno-psabi`.
ACKs for top commit:
laanwj:
ACK 1edddf5de4
hebasto:
ACK 1edddf5de4, after thorough reading related materials, I agree this change can be merged. As I mentioned above, I have been compiling my arm-32bit binaries with `-Wno-psabi` flag for two years, and no related flaws were observed.
Tree-SHA512: 485c7500547ac5da567ad23847341c18ff832607f5a1002676404cc647e437cf3445b6894ecff5b52929ca52bea946c06bd90eace1997c895e56204e787065e4
- Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support)
- Use 4.9 for all others (second-oldest LTS release, released in
combination with glibc glibc 2.24 in Debian stretch)
On bare systems, it is possible to be lacking a services database. Check
for basic entries before attempting a build.
See the error message in the diff for more context.
We use these flags in our test-security-check make target, but they are
only available because debian patches them in.
We can patch them in for our Guix builds so that we can check the sanity
of our security/symbol checking suite before running them.
Now that our Guix builds are performed on glibc 2.24 and 2.27 (RISCV),
we no-longer need to pass the --enable-glibc-back-compat option.
Replace it with --disable-threadlocal, to prevent the usage of symbols
from glibc 2.18.
None of the binaries produced required symbols later than 2.17, and 2.27
(RISCV).
Our 'bitcoin-linux-g++' definition better integrates with our depends
system than the stock linux-g++-64 definition.
This fixes a bug whereby Guix builds on x86_64 for x86_64 did not
produce a QMinimalIntegrationPlugin and led to bitcoin-qt not being
built.
Support for riscv64 in glibc landed in 2.27 so it's unavoidable that we
use 2.27.
Running a Bitcoin build with toolchains based on 2.24 for platforms
other than riscv64 seem to produce binaries which do not have 2.17
symbols. So use 2.24 since it's more recent and maintained by Debian
Stretch.
aa80b5759d scripts: check macOS SDK version is set (fanquake)
c972345bac scripts: check minimum required Windows version is set (fanquake)
29615aef52 scripts: check minimum required macOS vesion is set (fanquake)
8732f7b6c9 scripts: LIEF 0.11.5 (fanquake)
Pull request description:
macOS:
We use a compile flag ([-mmacosx-version-min=10.14](https://github.com/bitcoin/bitcoin/blob/master/depends/hosts/darwin.mk#L96)) to set the minimum required version of macOS needed to run our binaries. This adds a sanity check that the version is being set as expected.
Clangs Darwin driver should infer the SDK version used during compilation, and forward that through to the linker. Add a check that this has been done, and the expected SDK version is set. Should help prevent issues like #21771 in future.
Windows:
We use linker flags ([-Wl,--major/minor-subsystem-version](https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L683)) to set the minimum required version of Windows needed to run our binaries. This adds a sanity check that the version is being set as expected.
Gitian builds:
```bash
# macOS:
8b6fcd61d75001c37b2af3fceb5ae09f5d2fe85e97d361f684214bd91c27954a bitcoin-f015e1c2cac9-osx-unsigned.dmg
3c1e412bc7f5a7a5d0f78e2cd84b7096831414e1304c1307211aa3e135d89bbf bitcoin-f015e1c2cac9-osx-unsigned.tar.gz
50b7b2804e8481f63c69c78e3e8a71c0d811bf2db8895dd6d3edae9c46a738ae bitcoin-f015e1c2cac9-osx64.tar.gz
fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz
8a20f21b20673dfc8c23e22b20ae0839bcaf65bf0e02f62381cdf5e7922936f0 bitcoin-core-osx-22-res.yml
# Windows:
b01fcdc2a5673387050d6c6c4f96f1d350976a121155fde3f76c2af309111f9d bitcoin-f015e1c2cac9-win-unsigned.tar.gz
b95bdcbef638804030671d2332d58011f8c4ed4c1db87d6ffd211515c32c9d02 bitcoin-f015e1c2cac9-win64-debug.zip
350bf180252d24a3d40f05e22398fec7bb00e06d812204eb5a421100a8e10638 bitcoin-f015e1c2cac9-win64-setup-unsigned.exe
2730ddabe246d99913c9a779e97edcadb2d55309933d46f1dffd0d23ecf9aae5 bitcoin-f015e1c2cac9-win64.zip
fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz
aa60d7a753e8cb2d4323cfbbf4d964ad3645e74c918cccd66862888f8646d80f bitcoin-core-win-22-res.yml
```
ACKs for top commit:
hebasto:
ACK aa80b5759d, tested by breaking tests:
Tree-SHA512: 10150219910e8131715fbfe20edaa15778387616ef3bfe1a5152c7acd3958fe8f88c74961c3d3641074eb72824680c22764bb1dc01a19e92e946c2d4962a8d2c
e2c40a4ed5 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong)
4cc35daed5 Rewrite guix-{attest,verify} for new hier (Carl Dong)
28a9c9b839 Make SHA256SUMS fragment right after build (Carl Dong)
Pull request description:
Based on: #22075
Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff
The following changes resolve many usability improvements which were pointed out to me:
1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents
2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var)
4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server.
For those who sign on an separate machine than the one you do builds on, the following steps will work:
1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest`
2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine
3. Sign the `SHA256SUMS` files:
```bash
for i in "<path-to-achow101>/*.SHA256SUMS"; do
gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
done
```
5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs`
-----
After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build:
```
output
└── x86_64-w64-mingw32
├── bitcoin-4e069f7589da-win64-debug.zip
├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
├── bitcoin-4e069f7589da-win64.zip
├── bitcoin-4e069f7589da-win-unsigned.tar.gz
└── SHA256SUMS.part
```
These `SHA256SUMS.part` fragments look something like:
```
3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825 dist-archive/bitcoin-4e069f7589da.tar.gz
def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288 x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip
```
Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level
When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create:
1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and
3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs
Then both files are signed, resulting in the following `guix.sigs` hierarchy:
```
4e069f7589da/
└── dongcarl
├── all.SHA256SUMS
├── all.SHA256SUMS.asc
├── noncodesigned.SHA256SUMS
└── noncodesigned.SHA256SUMS.asc
```
ACKs for top commit:
achow101:
ACK e2c40a4ed5
hebasto:
ACK e2c40a4ed5, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only.
Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
683d197970 Use latest signapple commit (Andrew Chow)
Pull request description:
Update gitian and guix to use the same latest signapple commit.
Also changed guix to use the actual repo. The changes from the fork were incorporated upstream.
ACKs for top commit:
fanquake:
ACK 683d197970 - sanity checked that the updated package is built:
Tree-SHA512: a4981f8bbe33e6c5654632bc9b9f6f2f1e675741a19ac7296205e370f1e64a747101ecb632e0cc82a0134e4c2e9ce47b3f7b4d8c8f75f0f06dd069c078303759
108a6be92a guix: Check for disk space availability before building (Carl Dong)
d7dec89091 guix: Remove dest if OUTDIR mv fails (Carl Dong)
Pull request description:
There seems to be some corner cases that can be hit when guix scripts unexpectedly fail in the middle of operation, see: https://gnusha.org/bitcoin-builds/2021-05-24.log
- Perform an early disk space check for `guix-build`
- Overwrite existing output directory after a successful build (the existing one might be malformed), and cleanup output directory if the `mv` somehow fails
ACKs for top commit:
laanwj:
Tested ACK 108a6be92a
achow101:
ACK 108a6be92a
Tree-SHA512: cf6438317da40bf55714cd2d8cce859b3d435cc66cabefe8d4a53552d7880966acfe84ffe8fadf1c80e368ae6b037992258a6d409df85ffc6ce8bf780e98e2e5
a58868d201 build: Makes rcc output always deterministic (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2.
The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732).
This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless.
---
Also it improves interaction with ccache:
On master (f6c44e999b):
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:45:43 2021
stats zeroed Sun Apr 11 15:45:05 2021
cache hit (direct) 638
cache hit (preprocessed) 0
cache miss 1
cache hit rate 99.84 %
called for link 10
cleanups performed 0
files in cache 20023
cache size 13.2 GB
max cache size 15.0 GB
```
The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`.
With this PR:
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:28:46 2021
stats zeroed Sun Apr 11 15:28:21 2021
cache hit (direct) 639
cache hit (preprocessed) 0
cache miss 0
cache hit rate 100.00 %
called for link 10
cleanups performed 0
files in cache 20012
cache size 13.2 GB
max cache size 15.0 GB
```
ACKs for top commit:
fanquake:
ACK a58868d201
Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
d420e5c1c0 guix-attest: Avoid incomplete sigdirs with ERR traps (Carl Dong)
feda2c8e31 guix: Skip attesting to dist-archive (Carl Dong)
d522d8006b guix: Attest to inputs in inputs.SHA256SUMS (Carl Dong)
f9e2960c01 guix: Construct $OUTDIR in ${DISTSRC}/output (Carl Dong)
022abc85fc guix: Minor quoting fix in libexec/build.sh (Carl Dong)
c83c4fa5b7 guix-attest: Allow skipping GPG signing with NO_SIGN (Carl Dong)
0e1c2e448c guix-attest: Use ascii-armor signatures (Carl Dong)
b5fd89c4c8 guix-attest: Only use cross-platform flags for find+xargs (Carl Dong)
5926432ba6 guix: Add guix-verify script (Carl Dong)
30daf76a97 guix: Add guix-attest script (Carl Dong)
Pull request description:
Adds replacements for `gsign` and `gverify`.
Personally I'm not a big fan of using the word "sign" as it's been used to refer to both codesigning and GPG signing.
ACKs for top commit:
laanwj:
Code review and tested ACK d420e5c1c0
Tree-SHA512: 93d82d201f4596eaea0e3825aa55b013dfb91790e6ccee79893833d37921513d7b4e735f0641103e1e2ea8308abe4cb6218b73160924708802f2e0e3f7f6caf1
The Qt Resource Compiler (rcc) has a command-line option
`--format-version` which has the default value 2.
The only difference from `--format-version 1` is adding a last modified
timestamp to the output file. That, in turn, forces us to use
`QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds.
This change makes rcc output always deterministic by using
`--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless. Also it improves interaction
with ccache.
Co-authored-by: fanquake <fanquake@gmail.com>
46b025e00d test: add new python linter to check file names and permissions (windsok)
6f6bb3ebc7 test: fix file permissions on various scripts (windsok)
Pull request description:
Adds a new python linter test which tests for correct filenames and file permissions in the repository.
Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050
Summary of tests:
* Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames.
* Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test)
* Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line.
* Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test)
* Checks every file that contains a shebang line to ensure it has an executable permission
Additionally updates the permissions on various files to comply with the new tests.
Fixes#21729
ACKs for top commit:
practicalswift:
cr re-ACK 46b025e00d: patch still looks correct
kiminuo:
code review ACK 46b025e00d if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK.
laanwj:
Code review ACK 46b025e00d
Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
7fc5e865b9 test: install lief in CI (fanquake)
955140b326 contrib: consolidate PIE and NX security checks (fanquake)
2aa1631822 contrib: use LIEF in PE symbol checks (fanquake)
e93ac26b85 contrib: use LIEF in macOS symbol checks (fanquake)
a632cbcee5 contrib: use f strings in symbol-check.py (fanquake)
0f5d77c8e4 contrib: add PE PIE check to security checks (fanquake)
8e1f40dd9a contrib: use LIEF for PE security checks (fanquake)
a25b2e965c contrib: use LIEF for macOS security checks (fanquake)
7e7eae7aa8 contrib: use f strings in security-check.py (fanquake)
2e7a9f7ade guix: install LIEF in Guix container (fanquake)
465967b5ef gitian: install LIEF in gitian container (fanquake)
Pull request description:
This PR is a proof of concept for using [LIEF](https://github.com/lief-project/LIEF) for the PE and MACHO symbol and security checks. It replaces our current approach of manually parsing the output of `objdump` & `otool`. If the consensus is that using LIEF is ok, then I also plan on replacing [pixie.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/pixie.py), and using LIEF for all checks. LIEF for Linux is also currently blocked (on the next release, unless we want to build master) on one change for RISC-V that I [sent upstream](https://github.com/lief-project/LIEF/pull/562).
LIEF is seemingly well maintained, and is the basis for a number of other tools. It also has some very nice documentation; i.e the [Python API for ELF](https://lief.quarkslab.com/doc/latest/api/python/elf.html). It also has many builtins we can take advantage of. i.e [`is_pie`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.is_pie), [`has_nx`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.has_nx) etc. This means we can [consolidate some of our checks](9c5eeb5484). If/when end up using LIEF for lightning then we can consolidate further, and cleanup these scripts. i.e to not parse the binary inside the checks, but once at the start of the script.
Guix builds:
```bash
# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
963a08638c46f9a3d75cd4b0c155d1ca091bbeba27167291adcd3dca03fd4c3d guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
a3ce927c46b103789a010c41a6ebfafe4548d90ee7d88f2a735c9183b775da5c guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
2503ac8901068805d5e7251fd5cfeb7c1f8ba3528bdfcf3aa1e0c40bfd5c1cbc guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
5798697e58e1788df85aa9e2e4d33fef0456169fcbd2521f13b3b5806ac0d84d guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 guix-build-f51237d94d98/output/dist-archive/bitcoin-f51237d94d98.tar.gz
9b4b8756c5c84295eb6b61b6b32a07a8d07723fb38aaa8f519b6133935061bda guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
cbd821aa464a9c16f7979dbec1a5e66939e777a567f55f7081499a8d528d42c5 guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
abed530a82e97e3cf621c90a13c0881b0e39ccce2a6f42a3ff80de76e2abc5f7 guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
8b6d2bdd8b58ff1f6072bf8693abe3ce773ff3a7d8d2b7218207e69945b9d31b guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
d99cc705032d22ae819975992216899ed960ba25871a05c8789d00b80418511f guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
5240ca4f4ef7c62088185224ac319ad9a4a9b40075df10af18d8a6355bca32fb guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
adc16eaee4b51e8615ce8b3be9f6c018698237df4ad6e0886cf0d4ab6bc9e5c4 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.dmg
b188af0572ee682d74cc82c7e6e464115205fc130a457cfe19d42ac9ddd267f8 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.tar.gz
e764062fde144e6fb5d6dd776c10fc2daa8d775831f7e43247d17a6c6e060c97 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx64.tar.gz
dab3d26ac94c669140f7329d14e57ef02b0fe92b8a8f9d96c32a416adea0da0f guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
ca59d4379fbe2b9a52deebeaf88508e0eda4215f28d319aff0781289dd159712 guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
52b7c35321a85c4f6c95bf0e687574454b71ede9bec1c9cf17f37c578c888a94 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win-unsigned.tar.gz
a543895a00f8ffb3ba50ca68396d52ad5a18dd8efe38730e0049dd70d283a092 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-debug.zip
aec050d03c65268a986148500f7341cceb8c5f85287e0e3cde8933ce4b4dee32 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-setup-unsigned.exe
57ba33ed6ee8d3a885e342471359301473e83037d5442895beb686921a4c50e9 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64.zip
```
Gitian builds:
```bash
# macOS:
2f066e852bdd30ac46e5ecdf7619d19d408035c318a3edf0f1893ec2e25efb69 bitcoin-41a1b3d1b130-osx-unsigned.dmg
8cf8ac4d21740f490262453c330b5f4a5c5b8139dfc1b322efefce3f3b93d1b2 bitcoin-41a1b3d1b130-osx-unsigned.tar.gz
cf1b84efdd9d2588a1ce9513580fb56b38bfafe60e18f8adbeedf03521c6c2b2 bitcoin-41a1b3d1b130-osx64.tar.gz
14995244b0bb3e80e7b79975c9c70fdfb3ee3c04fda3efd5358ce1c4efa3a312 src/bitcoin-41a1b3d1b130.tar.gz
93881069d5e1dc385c08895a7b035a94eb010325afc2776c99b6aafa21096eb8 bitcoin-core-osx-22-res.yml
# Windows:
4d56dd7713121684b7eaa448679c65df2fd0aa5319bf8d12fb6cfa9f0b005cf7 bitcoin-f51237d94d98-win-unsigned.tar.gz
4558f4173152b084bcba25aa1a53c605208a70fe20392141b63cefb476528c85 bitcoin-f51237d94d98-win64-debug.zip
b63feaca010e86d514cfe38d716e3c8a8b8058e4f969b868aaaeb8a8a3d3dc81 bitcoin-f51237d94d98-win64-setup-unsigned.exe
de7d8586cc91ba391fe911853a99d9fd15fc6f9a60f9b91a0447940173aac67a bitcoin-f51237d94d98-win64.zip
4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 src/bitcoin-f51237d94d98.tar.gz
45efaca35b5fad0a04dfd06e44f7c00b990aa91c7bf2faea57e020d3491a6cf0 bitcoin-core-win-22-res.yml
# Linux:
055d646c5f8cf4708008374546176012ff758566a2645a3a01e1a33eab1002fe bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
bfc8b0efc36b0474c88546b12d2723c04b4dc629ae311082025c7e0b8f0d1aa9 bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
9dfaa5acfffadad8942b32996458013a155d12ed07be76601f232233627b5cb9 bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
54eb57905ff8513b9f628707b61aa4659c362fb2f6d17e0ee240b4da3674907d bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
ad98d876616eff578ad8cfd17dfbabe48ed14200823579687d66694bae3d2fe3 bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
fe1b421dd1cb6e04d5dc5d341459dc15fa6e15b80906e5d8e0405cf43495e0f7 bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
9001d95cc7d2722d9d7dd83d9da8e5adf575fddf91b615b76b9bcfece30ecf6f bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
9e0650ad2aba70c0fd1608a077e95f335dc1bb4a79eab9b0b56ac87427a4fd4f bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
fbfde0134944d3dbd32991455b0a8abdd334853ab8a4c1a1a4c060d9de071c50 bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
2fa2cfddce98c44c65305326fc623a7f065129208337503d813a08d51580cb8a bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
b2d6caeee0e3c350a43165c39876ebed8e588958007af0d06996e341c7060683 bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
bfdb827e75d43d61462513c9a843620b93c9160d9d246cad13278baaa07f64ea bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 src/bitcoin-f51237d94d98.tar.gz
34820a093916fa35b0fd98806a50092f46b20271af7422f43e2a4223ef6f9bb7 bitcoin-core-linux-22-res.yml
```
ACKs for top commit:
laanwj:
re-ACK 7fc5e865b9
Tree-SHA512: 0c30838413448ecfcf55e6273f607fdb01cb1acafa1d2762afad59360fca7d8efa78ec55064f50cba56cb2c9e98741e13665cba8e9b4b8e5b62b8a53f9bf8990
We already attest to the relevant dist-archive in inputs.SHA256SUMS,
which is recorded at build-time.
We use a SKIPATTEST.TAG file to indicate output directories which do not
require attestation (much like the CACHEDIR.TAG specification).
Generally, it's better to have build scripts declare properties of
directories instead of introducing name-based special cases in attest
scripts since build scripts have a more detailed context of what is
going on.
At build/codesigning-time, hash build inputs and output the digest to
${OUTDIR}/inputs.SHA256SUMS, which gets included in the final SHA256SUMS
constructed by guix-attest.
Example final SHA256SUMS:
ee832d2a35b7701bff581dea05a536118b118e3ad0a587a2855b6ee8cd6fba20 inputs/bitcoin-78199266af7b.tar.gz
ca765e70a0c12866dd63c0be228b675278a26329e5f8f5b5c52fd09200fedf21 bitcoin-78199266af7b-powerpc64le-linux-gnu-debug.tar.gz
dae95327d7f2c324e2728c4b73627be6cb2c0d2f2e5bea940d1d5e6463939327 bitcoin-78199266af7b-powerpc64le-linux-gnu.tar.gz
While files are being output to $OUTDIR, it will be under
${DISTSRC}/output, and only when everything is done, will
${DISTSRC}/output be moved to the actual $OUTDIR.
This makes it so that a Ctrl-C in the middle of a build is less likely
to result in a partially-constructed $OUTDIR. In fact, if I understand
correctly, if $OUTDIR and $DISTSRC reside on the same filesystem, the
move (rename) is likely atomic.
Also, since the "working $OUTDIR" is under ${DISTSRC}/output, it will be
cleaned properly by the guix-clean script.
c799a19b4b build, qt: No longer need to set QT_RCC_TEST=1 for determinism (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler (rcc) output order relies on [`QHash`](https://doc.qt.io/qt-5/qhash.html):
> This randomization of `QHash` is enabled by default. Even though programs should never depend on a particular `QHash` ordering, there may be situations where you temporarily need deterministic behavior, for example for debugging or regression testing. To disable the randomization, define the environment variable `QT_HASH_SEED` to have the value 0.
Since #3620 we use `QT_RCC_TEST=1` to achieve a deterministic output.
Since Qt 5.3.1 hash seeding is disabled for all of the bootstrapped tools, including rcc. Therefore, `QT_RCC_TEST=1` is no longer needed.
See commit [5283a6c87beac5a43f612786fefd6e43f2c70bf6](5283a6c87b).
ACKs for top commit:
fanquake:
ACK c799a19b4b
Tree-SHA512: 9d116ac1e8c605ee3e8ed7f618586f0de85d8b06bbbb70fe8c298939ce203d2a7e97264a9afac037179993ab54c5f69a65ebb9ab27ca7f45acb963011bd45743
Passing ADDITIONAL_GUIX_COMMON_FLAGS="--no-substitutes --bootstrap" as
suggested doesn't work:
```bash
...outputting in: '/bitcoin/guix-build-a1f0b8b62eb8/output/x86_64-linux-gnu'
...bind-mounted in container to: '/outdir-base/x86_64-linux-gnu'
guix time-machine: error: bootstrap: unrecognized option
```
and I think bootstrapping is more than covered in the preceding "Choose
your security model" section.
867a5e172a guix: Register garbage collector root for containers (Carl Dong)
8f8b96fb54 guix: Update hint messages to mention guix-clean (Carl Dong)
44f6d4f56b guix: Record precious directories and add guix-clean (Carl Dong)
84912d4b24 build: Remove spaces from variable-printing rules (Carl Dong)
Pull request description:
```
guix: Record precious directories and add guix-clean
Many users have reported problems that stem from having an unclean
working tree. To that end, I've written a guix-clean script which should
help reset the working tree while respecting user-specified precious
directories.
Precious directories, such as:
- SOURCES_PATH
- BASE_CACHE
- SDK_PATH
- OUTDIR
Should be preserved when cleaning the working tree, and are thus
recorded in ./contrib/guix/var/precious_dirs.
The ./contrib/guix/guix-clean script is able to parse that file and make
sure to avoid them when cleaning out the working tree.
```
ACKs for top commit:
laanwj:
ACK 867a5e172a
Tree-SHA512: c498fad781ff5e6406639df2b91b687fc528273fdf266bcdba8f6eec3b3b37ecce544b6da0252f0b9c6717f9d88e844e4c7b72d1877bdbabfc6871ddd0172af5
By registering the container profiles as garbage collector roots, it
will prevent `guix gc` from garbage collecting derivations which our
container needs and inconvieniencing the user with a rebuild.
Many users have reported problems that stem from having an unclean
working tree. To that end, I've written a guix-clean script which should
help reset the working tree while respecting user-specified precious
directories.
Precious directories, such as:
- SOURCES_PATH
- BASE_CACHE
- SDK_PATH
- OUTDIR
Should be preserved when cleaning the working tree, and are thus
recorded in ./contrib/guix/var/precious_dirs.
The ./contrib/guix/guix-clean script is able to parse that file and make
sure to avoid them when cleaning out the working tree.
This relatively easy change eliminates all runtime dependencies (except
for the kernel) for dmg, which is the only native build tool that gets
put in our output tarballs.
This allows much more flexibility when constructing the codesigning
environment, and is much more robust.
./windeploy is a "working directory", and therefore belongs inside
distsrc-*. Many people have noticed their Guix builds failing after
hours simply because they did not remove windeploy (but did remove the
distsrc-* directories).
In Guix, there are two flags for controlling parallelism:
Note: When I say "derivation," think "package"
--cores=n
- controls the number of CPU cores to build each derivation. This is
the value passed to `make`'s `--jobs=` flag.
- defaults to 0: as many cores as is available
--max-jobs=n
- controls how many derivations can be built in parallel
- defaults to 1
Therefore, if set --max-jobs=$MAX_JOBS and don't set --cores, Guix could
theoretically spin up $MAX_JOBS * $(nproc) number of threads, and that's
no good.
So we could either default to --cores=1, --max-jobs=$MAX_JOBS
- Pro: --cores=1 means that `make` will be invoked with `-j1`,
avoiding problems with package whose build systems and test
suites break when running multi-threaded.
- Con: There will be times when only 1 or 2 derivations can be built
at a time, because the rest of the dependency graph all depend
on those 1 or 2 derivations. During these times, the machine
will be severely under-utilized.
or --cores=$MAX_JOBS, --max-jobs=1
- Pro: We don't encounter prolonged periods of
severe under-utilization mentioned above.
- Con: Many packages' build systems and test suites break when running
multi-threaded.
or --cores=1, --max-jobs=1 and let the user override with
$ADDITIONAL_GUIX_COMMON_FLAGS
fbbb2d4fc1 lint: Fix spelling errors in comments (fyquah)
Pull request description:
Found some spelling errors while running spelling linter https://github.com/bitcoin/bitcoin/pull/21245
This PR fixes them.
ACKs for top commit:
fanquake:
ACK fbbb2d4fc1 - I thought we just fixed all of these.
Tree-SHA512: 95525040001f94e899b778c616cb66ebafb679dff88835b66fccf6349d8eb942d6b7374c536a44e393f13156bce9a32ed57e6a82bb02074d2b3cddb2696addb2
c33b199456 guix: Bump glibc and linux-headers (Carl Dong)
65363a1bd8 guix: Rebase on 95aca2991b (1.2.0-12.dffc918) (Carl Dong)
Pull request description:
On bumping the time-machine:
```
A few changes which are useful for us:
1. 'gnu: cross-gcc-arguments: Enable 128 bit long double for POWER9.' is
now merged into master.
2. gnutls is bumped to 3.6.15 and the temporal test failure in
status-request-revoked is fixed. Note that this does not fix the case
where one has installed Guix v1.2.0 and is running a substitute-less
bootstrap build, since the `guix time-machine` command itself has a
dependency on gnutls v3.6.12 (the one with the broken test) and will
thus try to build it before attempting to jump forwards in time. This
does however, mean that those who build a version of Guix that also
contains this fix will not go backwards in time to build the broken
gnutls v3.6.12.
```
On bumping the rest:
```
Bump glibc and linux-headers to match those of our Gitian counterparts.
We also require a glibc >= 2.28 for the test-symbol-check scripts to
work properly.
The default BASE-GCC-FOR-LIBC also has to be bumped since glibc 2.31
requires a gcc >= 6.2
```
This is a prerequisite for #20980
ACKs for top commit:
fanquake:
ACK c33b199456 - I think going ahead with this now and to sycn back up to gitian is fine. It will also unblock #20980. Potential code signing related issues can be sorted out in #21239 and later PRs.
Tree-SHA512: 31f022aadb93ba44813b0da005b1f2e5d67d76e8cdcdb53368924d1ea6cb076a21218c26831a6b0dcdcfe33507f54934330489ba557371d740f5587b7d727b95
a0a7a4337d guix, doc: Update default HOSTS value (Hennadii Stepanov)
Pull request description:
This is a #21089 follow up.
ACKs for top commit:
fanquake:
ACK a0a7a4337d
Tree-SHA512: c1813cc2b9212a79fd34d4e25cd0816b58264e1890daf777cd59411bd20fcc9affe312871d06fab1308b8f55c1a78ac1101e631882c18360a4709ecef4529f05
13a9fd11a5 guix: Passthrough SDK_PATH into container (Carl Dong)
Pull request description:
This is a usability improvement for Guix builders so that they don't have to extract the Xcode tarball into `depends/SDKs` every time.
Inspiration: https://github.com/bitcoin/bitcoin/pull/21089#issuecomment-778639698
ACKs for top commit:
laanwj:
Tested ACK 13a9fd11a5
Tree-SHA512: 63392d537e48a0da9f0ee04a929613b139bef1ac5643187871c9ea5376afd2a3d95df0f5e0950ae0eccd2813b166667be98401e5a248ae9c187fe4e84e54d427
Bump glibc and linux-headers to match those of our Gitian counterparts.
We also require a glibc >= 2.28 for the test-symbol-check scripts to
work properly.
The default BASE-GCC-FOR-LIBC also has to be bumped since glibc 2.31
requires a gcc >= 6.2
A few changes which are useful for us:
1. 'gnu: cross-gcc-arguments: Enable 128 bit long double for POWER9.' is
now merged into master.
2. gnutls is bumped to 3.6.15 and the temporal test failure in
status-request-revoked is fixed. Note that this does not fix the case
where one has installed Guix v1.2.0 and is running a substitute-less
bootstrap build, since the `guix time-machine` command itself has a
dependency on gnutls v3.6.12 (the one with the broken test) and will
thus try to build it before attempting to jump forwards in time. This
does however, mean that those who build a version of Guix that also
contains this fix will not go backwards in time to build the broken
gnutls v3.6.12.
a6a1b106dc guix: only download sources for hosts being built (fanquake)
Pull request description:
For example, if a user is only interested in building for Linux, this saves downloading the macOS compiler and additional dependencies, which is meaningful on a slow/poor connection. This will result in a few additional `make` invocations, for the Linux hosts, however this is low overhead, and time-wise irrelevant in terms of the overall build.
ACKs for top commit:
laanwj:
Code review ACK a6a1b106dc
Tree-SHA512: 34c916ae6f69fed0d5845690b39111a8bee37208fd727176f375cf5eb4860f512abe12bde2680d697c859b4d50a3bc5688ddca7c2f28f9968fcf358753cf3f6d
If a user is only interested in building for Linux, this saves
downloading the macOS compiler and additional dependencies.
This will result in a few additional `make` invocations, for the Linux
hosts, however this is quite low overhead.
Co-authored-by: Carl Dong <contact@carldong.me>
The new time-machine commit is Guix v1.2.0 with a yet-unupstreamed patch
for NSIS.
A few important changes:
1. Guix switched back from using CPATH to C{,PLUS}_INCLUDE_PATH as the
way to indicate #include search paths.
2. GCC's library is now split into a separate output, whereas before it
was included in the default output. This means that our gcc toolchain
packages need to propagate that output.
3. A few package versions were bumped
When building nsis, if VERSION is not specified, it defaults to
cvs_version which is non-deterministic as it includes the current date.
This patches nsis to default to SOURCE_DATE_EPOCH if it exists so that
nsis is reproducible.
Upstream change: https://github.com/kichik/nsis/pull/13
When using worktrees or submodules, you'll see a `.git' plain text file
at the root of your working tree instead of the usual `.git' directory.
This plain text file will point to the real GIT_DIR, under the
GIT_COMMON_DIR. From experimentation, the full GIT_COMMON_DIR is
required to exist for operations such as git-archive(1), so we expose it
as readonly inside the container.
Any -O argument will enable optimizations in GNU ld. We can use -O2
here, as this matches our compile flags. Note that this would also
enable additional optimizations if using the lld or gold linkers,
when compared to -O0.
The libtool unsorted 'find' determinism issue seemed to have been solved
in gcc-9's git: d41cd173e23ebea7c758644d6ad6e0fde1c2e3a6 or SVN: r262451
Furthermore, it seems that Ubuntu Focal 20.04 LTS is going to ship with
gcc 9 and mingw-w64 7, which will match what we have now.
-----
A note on this:
Careful observers will see that previously I stated that all released
versions of gcc were bootstrapped with a libtool 2.2.7a, meaning that
they all had the unsorted 'find' determinism issue first resolved in
libtool 2.2.7b.
However, I was mistaken, gcc's ltmain.sh CLAIMS it was generated by
libtool 2.2.7a, but it was in fact edited manually. It seems that gcc
maintains their own versions of ltmain.sh and libtool.m4, and only
sometimes backports patches from upstream.
Quite confusing.
This is no longer needed after 3bef7c22 in the mingw-w64 git repository,
which is first included in mingw-w64 v7.0.0.
As of the previous bump to our Guix time machine, we now use mingw-w64
v7.0.0.
Most of the mingw-w64 toolchain changes have now been upstreamed, we can
point to a commit that exists upstream.
NOTE: I'm not changing the URL yet until we see that Guix upstream will
accept all my patches for macOS.
-----
The Guix tree that's referred to by this commit contains the following
changes relevant to our mingw-w64 build:
b066c25026
Adds a PACKAGES-WITH-*PATCHES procedure which we can use in the future
to apply patches to packages if those patches are not considered
appropriate to upstream Guix
4719b71572
Adds mingw-w64 (the libc itself) reproducibility patches, taken from
debian.
79825bee07 + 401d28e433 + c1c50cb5b0
Add mingw-w64 specific binutils patches, taken from debian.
Specifically, the "Make DLL import libraries reproducible" patch made
libbitcoinconsensus.dll.a build reproducibly. The followup commits
were hotfixes for my mistakes.
0f864175dc
Bumps mingw-w64 to v7.0.0. This is the first release that enables
secure APIs by default (which we need), and gains _FORTIFY_SOURCE
support. This will also be what Ubuntu Focal 20.04 LTS releases with.
cdf00cf75d
Bumps NSIS to v3.05. This is the first release that includes a fix for
a reproducibility bug found by some of the electrum developers. See
details here: https://sourceforge.net/p/nsis/bugs/1230/
This bump will includes a couple of commits which improve the
reproducibility of the mingw-w64 toolchain. Most of which came from
debian. They will be upstreamed as upstream Guix release timeline
allows.
- Add "--no-insert-timestamp" LDFLAG for x86_64-w64-mingw32 builds
"The option --no-insert-timestamp can be used to insert a zero value for
the timestamp, this ensuring that binaries produced from identical
sources will compare identically." - ld(1)
- Set "SetDateSave off" in NSIS script
From https://nsis.sourceforge.io/Docs/Chapter4.html#flags
"This command sets the file date/time saving flag which is used by the
File command to determine whether or not to save the last write date and
time of the file, so that it can be restored on installation. Valid
flags are 'on' and 'off'. 'on' is the default."
- Add commented out NSIS options for reproducibility debugging in NSIS
script
- Make ZIPs deterministic by reseting file modification times to
SOURCE_DATE_EPOCH using touch(1) (Reference:
https://reproducible-builds.org/docs/archives/)
Previously, Guix would produce a gcc which did not know to use the SSP
function from glibc, and required a gcc make flag for it to do so, in my
attempt to fix it upstream I realized that this is no longer the case.
This can be verified by performing a Guix build and doing
readelf -s ... | grep __stack_chk
to check that symbols are coming from glibc, and doing
readelf -d ... | grep NEEDED | grep ssp
to see that libssp.so is not being depended on
- store_path() previously only worked for cross compilation packages, we
remove this assumption here
- Add CROSS_GCC_LIB variable which points to where gcc libs/headers are
located
- Add gcc libs/headers to our CROSS_*_PATH environment variables
- Check that all directories in CROSS_*_PATH are sane
- Clearer and more accurate prose
- Pin `guix pull' to commit rather than branch
- Just use `use-module' instead of `define-module'
- Use `bash-minimal' instead of `bash'
- Remove unneeded `tcsh' from manifest
- Explicitly use `python-3.7'
- Add comments about how {native,cross}-toolchains are produced and
why
