- Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support)
- Use 4.9 for all others (second-oldest LTS release, released in
combination with glibc glibc 2.24 in Debian stretch)
On bare systems, it is possible to be lacking a services database. Check
for basic entries before attempting a build.
See the error message in the diff for more context.
Now that our release binaries are build in a glibc 2.24 and 2.27
environment, we can't use a symbol from glibc 2.28 to test our checks.
Replace renameat2() with nextup(), which was introduced in 2.24.
Note that this also means re-disabling the test for RISC-V, however
RISC-V is built in a glibc 2.27 environment, and our minimum required
glibc for that binary is 2.27.
We use these flags in our test-security-check make target, but they are
only available because debian patches them in.
We can patch them in for our Guix builds so that we can check the sanity
of our security/symbol checking suite before running them.
This is important to make sure that we're not testing tools different
from the one we're building with.
Introduce determine_wellknown_cmd, which encapsulates how we
should handle well-known tools specification (IFS splitting, env
override, etc.).
Now that our Guix builds are performed on glibc 2.24 and 2.27 (RISCV),
we no-longer need to pass the --enable-glibc-back-compat option.
Replace it with --disable-threadlocal, to prevent the usage of symbols
from glibc 2.18.
None of the binaries produced required symbols later than 2.17, and 2.27
(RISCV).
Our 'bitcoin-linux-g++' definition better integrates with our depends
system than the stock linux-g++-64 definition.
This fixes a bug whereby Guix builds on x86_64 for x86_64 did not
produce a QMinimalIntegrationPlugin and led to bitcoin-qt not being
built.
Support for riscv64 in glibc landed in 2.27 so it's unavoidable that we
use 2.27.
Running a Bitcoin build with toolchains based on 2.24 for platforms
other than riscv64 seem to produce binaries which do not have 2.17
symbols. So use 2.24 since it's more recent and maintained by Debian
Stretch.
bdb8b9a347 test: doc: improve doc for `from_hex` helper (mention `to_hex` alternative) (Sebastian Falbesoner)
1914054208 scripted-diff: test: rename `FromHex` to `from_hex` (Sebastian Falbesoner)
a79396fe5f test: remove `ToHex` helper, use .serialize().hex() instead (Sebastian Falbesoner)
2ce7b47958 test: introduce `tx_from_hex` helper for tx deserialization (Sebastian Falbesoner)
Pull request description:
There are still many functional tests that perform conversions from a hex-string to a message object (deserialization) manually. This PR identifies all those instances and replaces them with a newly introduced helper `tx_from_hex`.
Instances were found via
* `git grep "deserialize.*BytesIO"`
and some of them manually, when it were not one-liners.
Further, the helper `ToHex` was removed and simply replaced by `.serialize().hex()`, since now both variants are in use (sometimes even within the same test) and using the helper doesn't really have an advantage in readability. (see discussion https://github.com/bitcoin/bitcoin/pull/22257#discussion_r652404782)
ACKs for top commit:
MarcoFalke:
review re-ACK bdb8b9a347😁
Tree-SHA512: e25d7dc85918de1d6755a5cea65471b07a743204c20ad1c2f71ff07ef48cc1b9ad3fe5f515c1efaba2b2e3d89384e7980380c5d81895f9826e2046808cd3266e
e8cd3700ee devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in symbol-check.py (W. J. van der Laan)
a33381acf5 devtools: Add xkb version to symbol-check (W. J. van der Laan)
19e598bab0 devtools: Fix verneed section parsing in pixie (W. J. van der Laan)
Pull request description:
I misunderstood the ELF specification for version symbols (verneed): The `vn_aux` pointer is relative to the main verneed record, not the start of the section.
This caused many symbols to not be versioned properly in the return value of `elf.dyn_symbols`. This was discovered in #21454.
Fix it by correcting the offset computation.
- xkb versions symbols (using the prefix `V`), as this library is used by bitcoin-qt, add it to the valid versions in `symbol-check.py`
This unfortunately brings to light some symbols that have been introduced since and weren't caught (from a gitian compile of master):
```
bitcoin-cli: symbol getrandom from unsupported version GLIBC_2.25
bitcoin-cli: failed IMPORTED_SYMBOLS
bitcoind: symbol getrandom from unsupported version GLIBC_2.25
bitcoind: symbol log from unsupported version GLIBC_2.29
bitcoind: symbol fcntl64 from unsupported version GLIBC_2.28
bitcoind: symbol pow from unsupported version GLIBC_2.29
bitcoind: symbol exp from unsupported version GLIBC_2.29
bitcoind: failed IMPORTED_SYMBOLS
bitcoin-qt: symbol exp from unsupported version GLIBC_2.29
bitcoin-qt: symbol fcntl64 from unsupported version GLIBC_2.28
bitcoin-qt: symbol log from unsupported version GLIBC_2.29
bitcoin-qt: symbol pow from unsupported version GLIBC_2.29
bitcoin-qt: symbol statx from unsupported version GLIBC_2.28
bitcoin-qt: symbol getrandom from unsupported version GLIBC_2.25
bitcoin-qt: symbol renameat2 from unsupported version GLIBC_2.28
bitcoin-qt: symbol getentropy from unsupported version GLIBC_2.25
bitcoin-qt: failed IMPORTED_SYMBOLS
bitcoin-wallet: symbol exp from unsupported version GLIBC_2.29
bitcoin-wallet: symbol log from unsupported version GLIBC_2.29
bitcoin-wallet: symbol fcntl64 from unsupported version GLIBC_2.28
bitcoin-wallet: failed IMPORTED_SYMBOLS
test_bitcoin: symbol getrandom from unsupported version GLIBC_2.25
test_bitcoin: symbol log from unsupported version GLIBC_2.29
test_bitcoin: symbol fcntl64 from unsupported version GLIBC_2.28
test_bitcoin: symbol pow from unsupported version GLIBC_2.29
test_bitcoin: symbol exp from unsupported version GLIBC_2.29
test_bitcoin: failed IMPORTED_SYMBOLS
```
ACKs for top commit:
hebasto:
ACK e8cd3700ee
Tree-SHA512: 8c15e3478eb642f01a1ddaadef03f80583f088f9fa8e3bf171ce16b0ec05ffb4675ec147d7ffc6a4360637ed47fca517c6ca2bac7bb30d794c03783cfb964b79
The (ancient) versions specified here were deceptive. Entries older than
MAX_VERSIONS['GLIBC'], which is 2.17, are ignored here. So reorganize
the code to avoid confusion for other people reading this code.
aa80b5759d scripts: check macOS SDK version is set (fanquake)
c972345bac scripts: check minimum required Windows version is set (fanquake)
29615aef52 scripts: check minimum required macOS vesion is set (fanquake)
8732f7b6c9 scripts: LIEF 0.11.5 (fanquake)
Pull request description:
macOS:
We use a compile flag ([-mmacosx-version-min=10.14](https://github.com/bitcoin/bitcoin/blob/master/depends/hosts/darwin.mk#L96)) to set the minimum required version of macOS needed to run our binaries. This adds a sanity check that the version is being set as expected.
Clangs Darwin driver should infer the SDK version used during compilation, and forward that through to the linker. Add a check that this has been done, and the expected SDK version is set. Should help prevent issues like #21771 in future.
Windows:
We use linker flags ([-Wl,--major/minor-subsystem-version](https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L683)) to set the minimum required version of Windows needed to run our binaries. This adds a sanity check that the version is being set as expected.
Gitian builds:
```bash
# macOS:
8b6fcd61d75001c37b2af3fceb5ae09f5d2fe85e97d361f684214bd91c27954a bitcoin-f015e1c2cac9-osx-unsigned.dmg
3c1e412bc7f5a7a5d0f78e2cd84b7096831414e1304c1307211aa3e135d89bbf bitcoin-f015e1c2cac9-osx-unsigned.tar.gz
50b7b2804e8481f63c69c78e3e8a71c0d811bf2db8895dd6d3edae9c46a738ae bitcoin-f015e1c2cac9-osx64.tar.gz
fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz
8a20f21b20673dfc8c23e22b20ae0839bcaf65bf0e02f62381cdf5e7922936f0 bitcoin-core-osx-22-res.yml
# Windows:
b01fcdc2a5673387050d6c6c4f96f1d350976a121155fde3f76c2af309111f9d bitcoin-f015e1c2cac9-win-unsigned.tar.gz
b95bdcbef638804030671d2332d58011f8c4ed4c1db87d6ffd211515c32c9d02 bitcoin-f015e1c2cac9-win64-debug.zip
350bf180252d24a3d40f05e22398fec7bb00e06d812204eb5a421100a8e10638 bitcoin-f015e1c2cac9-win64-setup-unsigned.exe
2730ddabe246d99913c9a779e97edcadb2d55309933d46f1dffd0d23ecf9aae5 bitcoin-f015e1c2cac9-win64.zip
fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz
aa60d7a753e8cb2d4323cfbbf4d964ad3645e74c918cccd66862888f8646d80f bitcoin-core-win-22-res.yml
```
ACKs for top commit:
hebasto:
ACK aa80b5759d, tested by breaking tests:
Tree-SHA512: 10150219910e8131715fbfe20edaa15778387616ef3bfe1a5152c7acd3958fe8f88c74961c3d3641074eb72824680c22764bb1dc01a19e92e946c2d4962a8d2c
e2c40a4ed5 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong)
4cc35daed5 Rewrite guix-{attest,verify} for new hier (Carl Dong)
28a9c9b839 Make SHA256SUMS fragment right after build (Carl Dong)
Pull request description:
Based on: #22075
Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff
The following changes resolve many usability improvements which were pointed out to me:
1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents
2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var)
4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server.
For those who sign on an separate machine than the one you do builds on, the following steps will work:
1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest`
2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine
3. Sign the `SHA256SUMS` files:
```bash
for i in "<path-to-achow101>/*.SHA256SUMS"; do
gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
done
```
5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs`
-----
After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build:
```
output
└── x86_64-w64-mingw32
├── bitcoin-4e069f7589da-win64-debug.zip
├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
├── bitcoin-4e069f7589da-win64.zip
├── bitcoin-4e069f7589da-win-unsigned.tar.gz
└── SHA256SUMS.part
```
These `SHA256SUMS.part` fragments look something like:
```
3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825 dist-archive/bitcoin-4e069f7589da.tar.gz
def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288 x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip
```
Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level
When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create:
1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and
3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs
Then both files are signed, resulting in the following `guix.sigs` hierarchy:
```
4e069f7589da/
└── dongcarl
├── all.SHA256SUMS
├── all.SHA256SUMS.asc
├── noncodesigned.SHA256SUMS
└── noncodesigned.SHA256SUMS.asc
```
ACKs for top commit:
achow101:
ACK e2c40a4ed5
hebasto:
ACK e2c40a4ed5, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only.
Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
I misunderstood the ELF specification for version symbols (verneed):
The `vn_aux` pointer is relative to the main verneed record, not the
start of the section.
This caused many symbols to not be versioned properly in the return
value of `elf.dyn_symbols`. This was discovered in #21454.
Fix it by correcting the offset computation.
683d197970 Use latest signapple commit (Andrew Chow)
Pull request description:
Update gitian and guix to use the same latest signapple commit.
Also changed guix to use the actual repo. The changes from the fork were incorporated upstream.
ACKs for top commit:
fanquake:
ACK 683d197970 - sanity checked that the updated package is built:
Tree-SHA512: a4981f8bbe33e6c5654632bc9b9f6f2f1e675741a19ac7296205e370f1e64a747101ecb632e0cc82a0134e4c2e9ce47b3f7b4d8c8f75f0f06dd069c078303759
Clangs Darwin driver should infer the SDK version used during compilation, and
forward that through to the linker. Add a check that this has been done, and the
expected SDK version is set.
Should help prevent issues like #21771 in future.
We use linker flags (-Wl,--major/minor-subsystem-version) to set the
minimum required version of Windows needed to run our binaries. This
adds a sanity check that the version is being set as expected.
We use a compile flag (-mmacosx-version-min) to set the minimum required
version of macOS needed to run our binaries. This adds a sanity check
that the version is being set as expected.
108a6be92a guix: Check for disk space availability before building (Carl Dong)
d7dec89091 guix: Remove dest if OUTDIR mv fails (Carl Dong)
Pull request description:
There seems to be some corner cases that can be hit when guix scripts unexpectedly fail in the middle of operation, see: https://gnusha.org/bitcoin-builds/2021-05-24.log
- Perform an early disk space check for `guix-build`
- Overwrite existing output directory after a successful build (the existing one might be malformed), and cleanup output directory if the `mv` somehow fails
ACKs for top commit:
laanwj:
Tested ACK 108a6be92a
achow101:
ACK 108a6be92a
Tree-SHA512: cf6438317da40bf55714cd2d8cce859b3d435cc66cabefe8d4a53552d7880966acfe84ffe8fadf1c80e368ae6b037992258a6d409df85ffc6ce8bf780e98e2e5
5d82a57db4 contrib: remove torv2 seed nodes (Jon Atack)
5f7e086dac contrib: update generate-seeds.py to ignore torv2 addresses (Jon Atack)
8be56f0f8e p2p, refactor: extract OnionToString() from CNetAddr::ToStringIp() (Jon Atack)
5f9d3c09b4 p2p: remove torv2 from CNetAddr::ToStringIP() (Jon Atack)
3d39042144 p2p: remove torv2 in SetIP() and ADDR_TORV2_SIZE constant (Jon Atack)
cff5ec477a p2p: remove pre-addrv2 onions from SerializeV1Array() (Jon Atack)
4192a74413 p2p: ignore torv2-in-ipv6 addresses in SetLegacyIPv6() (Jon Atack)
1d631e956f p2p: remove BIP155Network::TORV2 from GetBIP155Network() (Jon Atack)
7d1769bc45 p2p: remove torv2 from SetNetFromBIP155Network() (Jon Atack)
eba9a94b9f fuzz: rename CNetAddr/CService deserialize targets (Jon Atack)
c56a1c9b18 p2p: drop onions from IsAddrV1Compatible(), no longer relay torv2 (Jon Atack)
f8e94002fc p2p: remove torv2/ADDR_TORV2_SIZE from SetTor() (Jon Atack)
0f1c58ae87 test: update feature_proxy to torv3 (Jon Atack)
Pull request description:
This patch removes support in Bitcoin Core for Tor v2 onions, which are already removed from the release of Tor 0.4.6.
- no longer serialize/deserialize and relay Tor v2 addresses
- ignore incoming Tor v2 addresses
- remove Tor v2 addresses from the addrman and peers.dat on node launch
- update generate-seeds.py to ignore Tor v2 addresses
- remove Tor v2 hard-coded seeds
Tested with tor-0.4.6.1-alpha (no v2 support) and 0.4.5.7 (v2 support). With the latest Tor (no v2 support), this removes all the warnings like those reported with current master in https://github.com/bitcoin/bitcoin/issues/21351
```
<bitcoind debug log>
Socks5() connect to […].onion:8333 failed: general failure
<tor log>
Invalid hostname [scrubbed]; rejecting
```
and the addrman no longer has Tor v2 addresses on launching bitcoind.
```rake
$ ./src/bitcoin-cli -addrinfo
{
"addresses_known": {
"ipv4": 44483,
"ipv6": 8467,
"torv2": 0,
"torv3": 2296,
"i2p": 6,
"total": 55252
}
}
```
After recompiling back to current master and restarting with either of the two Tor versions (0.4.5.7 or 0.4.6.1), -addrinfo initially returns 0 Tor v2 addresses and then begins finding them again.
Ran nodes on this patch over the past week on mainnet/testnet/signet/regtest after building with DEBUG_ADDRMAN.
Verified that this patch bootstraps an onlynet=onion node from the Tor v3 hardcoded fixed seeds on mainnet and testnet and connects to blocks and v3 onion peers: `rm ~/.bitcoin/testnet3/peers.dat ; ./src/bitcoind -testnet -dnsseed=0 -onlynet=onion`
Tested using `addnode`, `getaddednodeinfo`,`addpeeraddress`, `disconnectnode` and `-addrinfo` that a currently valid, connectable Tor v2 peer can no longer be added:
Thanks to Vasil Dimov, Carl Dong, and Wladimir J. van der Laan for their work on BIP155 and Tor v3 that got us here.
ACKs for top commit:
laanwj:
Code review ACK 5d82a57db4
Tree-SHA512: 590ff3d2f6ef682608596facb4b01f44fef69716d2ab3552ae1655aa225f4bf104f9ee08d6769abb9982a8031de93340df553279ce1f5023771f9f2b651178bb
a58868d201 build: Makes rcc output always deterministic (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2.
The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732).
This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless.
---
Also it improves interaction with ccache:
On master (f6c44e999b):
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:45:43 2021
stats zeroed Sun Apr 11 15:45:05 2021
cache hit (direct) 638
cache hit (preprocessed) 0
cache miss 1
cache hit rate 99.84 %
called for link 10
cleanups performed 0
files in cache 20023
cache size 13.2 GB
max cache size 15.0 GB
```
The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`.
With this PR:
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:28:46 2021
stats zeroed Sun Apr 11 15:28:21 2021
cache hit (direct) 639
cache hit (preprocessed) 0
cache miss 0
cache hit rate 100.00 %
called for link 10
cleanups performed 0
files in cache 20012
cache size 13.2 GB
max cache size 15.0 GB
```
ACKs for top commit:
fanquake:
ACK a58868d201
Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
