7fc5e865b9 test: install lief in CI (fanquake)
955140b326 contrib: consolidate PIE and NX security checks (fanquake)
2aa1631822 contrib: use LIEF in PE symbol checks (fanquake)
e93ac26b85 contrib: use LIEF in macOS symbol checks (fanquake)
a632cbcee5 contrib: use f strings in symbol-check.py (fanquake)
0f5d77c8e4 contrib: add PE PIE check to security checks (fanquake)
8e1f40dd9a contrib: use LIEF for PE security checks (fanquake)
a25b2e965c contrib: use LIEF for macOS security checks (fanquake)
7e7eae7aa8 contrib: use f strings in security-check.py (fanquake)
2e7a9f7ade guix: install LIEF in Guix container (fanquake)
465967b5ef gitian: install LIEF in gitian container (fanquake)
Pull request description:
This PR is a proof of concept for using [LIEF](https://github.com/lief-project/LIEF) for the PE and MACHO symbol and security checks. It replaces our current approach of manually parsing the output of `objdump` & `otool`. If the consensus is that using LIEF is ok, then I also plan on replacing [pixie.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/pixie.py), and using LIEF for all checks. LIEF for Linux is also currently blocked (on the next release, unless we want to build master) on one change for RISC-V that I [sent upstream](https://github.com/lief-project/LIEF/pull/562).
LIEF is seemingly well maintained, and is the basis for a number of other tools. It also has some very nice documentation; i.e the [Python API for ELF](https://lief.quarkslab.com/doc/latest/api/python/elf.html). It also has many builtins we can take advantage of. i.e [`is_pie`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.is_pie), [`has_nx`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.has_nx) etc. This means we can [consolidate some of our checks](9c5eeb5484). If/when end up using LIEF for lightning then we can consolidate further, and cleanup these scripts. i.e to not parse the binary inside the checks, but once at the start of the script.
Guix builds:
```bash
# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
963a08638c46f9a3d75cd4b0c155d1ca091bbeba27167291adcd3dca03fd4c3d guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
a3ce927c46b103789a010c41a6ebfafe4548d90ee7d88f2a735c9183b775da5c guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
2503ac8901068805d5e7251fd5cfeb7c1f8ba3528bdfcf3aa1e0c40bfd5c1cbc guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
5798697e58e1788df85aa9e2e4d33fef0456169fcbd2521f13b3b5806ac0d84d guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 guix-build-f51237d94d98/output/dist-archive/bitcoin-f51237d94d98.tar.gz
9b4b8756c5c84295eb6b61b6b32a07a8d07723fb38aaa8f519b6133935061bda guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
cbd821aa464a9c16f7979dbec1a5e66939e777a567f55f7081499a8d528d42c5 guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
abed530a82e97e3cf621c90a13c0881b0e39ccce2a6f42a3ff80de76e2abc5f7 guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
8b6d2bdd8b58ff1f6072bf8693abe3ce773ff3a7d8d2b7218207e69945b9d31b guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
d99cc705032d22ae819975992216899ed960ba25871a05c8789d00b80418511f guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
5240ca4f4ef7c62088185224ac319ad9a4a9b40075df10af18d8a6355bca32fb guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
adc16eaee4b51e8615ce8b3be9f6c018698237df4ad6e0886cf0d4ab6bc9e5c4 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.dmg
b188af0572ee682d74cc82c7e6e464115205fc130a457cfe19d42ac9ddd267f8 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.tar.gz
e764062fde144e6fb5d6dd776c10fc2daa8d775831f7e43247d17a6c6e060c97 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx64.tar.gz
dab3d26ac94c669140f7329d14e57ef02b0fe92b8a8f9d96c32a416adea0da0f guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
ca59d4379fbe2b9a52deebeaf88508e0eda4215f28d319aff0781289dd159712 guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
52b7c35321a85c4f6c95bf0e687574454b71ede9bec1c9cf17f37c578c888a94 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win-unsigned.tar.gz
a543895a00f8ffb3ba50ca68396d52ad5a18dd8efe38730e0049dd70d283a092 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-debug.zip
aec050d03c65268a986148500f7341cceb8c5f85287e0e3cde8933ce4b4dee32 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-setup-unsigned.exe
57ba33ed6ee8d3a885e342471359301473e83037d5442895beb686921a4c50e9 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64.zip
```
Gitian builds:
```bash
# macOS:
2f066e852bdd30ac46e5ecdf7619d19d408035c318a3edf0f1893ec2e25efb69 bitcoin-41a1b3d1b130-osx-unsigned.dmg
8cf8ac4d21740f490262453c330b5f4a5c5b8139dfc1b322efefce3f3b93d1b2 bitcoin-41a1b3d1b130-osx-unsigned.tar.gz
cf1b84efdd9d2588a1ce9513580fb56b38bfafe60e18f8adbeedf03521c6c2b2 bitcoin-41a1b3d1b130-osx64.tar.gz
14995244b0bb3e80e7b79975c9c70fdfb3ee3c04fda3efd5358ce1c4efa3a312 src/bitcoin-41a1b3d1b130.tar.gz
93881069d5e1dc385c08895a7b035a94eb010325afc2776c99b6aafa21096eb8 bitcoin-core-osx-22-res.yml
# Windows:
4d56dd7713121684b7eaa448679c65df2fd0aa5319bf8d12fb6cfa9f0b005cf7 bitcoin-f51237d94d98-win-unsigned.tar.gz
4558f4173152b084bcba25aa1a53c605208a70fe20392141b63cefb476528c85 bitcoin-f51237d94d98-win64-debug.zip
b63feaca010e86d514cfe38d716e3c8a8b8058e4f969b868aaaeb8a8a3d3dc81 bitcoin-f51237d94d98-win64-setup-unsigned.exe
de7d8586cc91ba391fe911853a99d9fd15fc6f9a60f9b91a0447940173aac67a bitcoin-f51237d94d98-win64.zip
4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 src/bitcoin-f51237d94d98.tar.gz
45efaca35b5fad0a04dfd06e44f7c00b990aa91c7bf2faea57e020d3491a6cf0 bitcoin-core-win-22-res.yml
# Linux:
055d646c5f8cf4708008374546176012ff758566a2645a3a01e1a33eab1002fe bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
bfc8b0efc36b0474c88546b12d2723c04b4dc629ae311082025c7e0b8f0d1aa9 bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
9dfaa5acfffadad8942b32996458013a155d12ed07be76601f232233627b5cb9 bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
54eb57905ff8513b9f628707b61aa4659c362fb2f6d17e0ee240b4da3674907d bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
ad98d876616eff578ad8cfd17dfbabe48ed14200823579687d66694bae3d2fe3 bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
fe1b421dd1cb6e04d5dc5d341459dc15fa6e15b80906e5d8e0405cf43495e0f7 bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
9001d95cc7d2722d9d7dd83d9da8e5adf575fddf91b615b76b9bcfece30ecf6f bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
9e0650ad2aba70c0fd1608a077e95f335dc1bb4a79eab9b0b56ac87427a4fd4f bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
fbfde0134944d3dbd32991455b0a8abdd334853ab8a4c1a1a4c060d9de071c50 bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
2fa2cfddce98c44c65305326fc623a7f065129208337503d813a08d51580cb8a bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
b2d6caeee0e3c350a43165c39876ebed8e588958007af0d06996e341c7060683 bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
bfdb827e75d43d61462513c9a843620b93c9160d9d246cad13278baaa07f64ea bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 src/bitcoin-f51237d94d98.tar.gz
34820a093916fa35b0fd98806a50092f46b20271af7422f43e2a4223ef6f9bb7 bitcoin-core-linux-22-res.yml
```
ACKs for top commit:
laanwj:
re-ACK 7fc5e865b9
Tree-SHA512: 0c30838413448ecfcf55e6273f607fdb01cb1acafa1d2762afad59360fca7d8efa78ec55064f50cba56cb2c9e98741e13665cba8e9b4b8e5b62b8a53f9bf8990
fa0422c251 ci: Add msan fuzz config (MarcoFalke)
fa399a76c6 ci: Use clang-12 in msan task (MarcoFalke)
fab30174af ci: Set BASE_SCRATCH_DIR early, so that it can be used in test configs (MarcoFalke)
Pull request description:
Similar to the valgrind config, this config is not run by any ci task in this repo, but it can be used by other repos or self-hosted infrastructure.
ACKs for top commit:
practicalswift:
cr ACK fa0422c251: patch looks correct
Tree-SHA512: 2122ac0948978a7b952efc80d4aa3674b27d48c6166e0ce917c61ac4ee6b68d701a83e5f71ee6868c208885ee45aae409ca022ebcb23ccbe37819a8c36e34872
fa44f5119a ci: Clarify that previous_releases task is using DEBUG (MarcoFalke)
fad0f21c3c ci: Use clang in multiprocess task to avoid OOM (MarcoFalke)
faeabef4f3 ci: Enable D_GLIBCXX_DEBUG for multiprocess task (MarcoFalke)
Pull request description:
Enable `-D_GLIBCXX_DEBUG` via the depends `DEBUG` flag. Also `--enable-debug` to get debug symbols in traces.
ACKs for top commit:
hebasto:
ACK fa44f5119a, I have reviewed the code and it looks OK, I agree it can be merged, and CI is green.
Tree-SHA512: ab2a216bb44ee462f9dd181ec9025962502bd4201a1118ff52b6a193398e7ea3ca465a45a5eb341e308758fc3ef34ea3521f8a1f85ed64478ef3c1f6c1b8b141
88d4d5ff2f rpc: add help for enumeratesigners and walletdisplayaddress (Sjors Provoost)
b0db187e5b ci: use --enable-external-signer instead of --with-boost-process (Sjors Provoost)
b54b2e7b1a Move external signer out of wallet module (Sjors Provoost)
Pull request description:
In addition, this PR enables external signer testing on CI.
This PR moves the ExternalSigner class and RPC methods out of the wallet module.
The `enumeratesigners` RPC can be used without a wallet since #21417. With additional modifications external signers could be used without a wallet in general, e.g. via `signrawtransaction`.
The `signerdisplayaddress` RPC is ranamed to `walletdisplayaddress` because it requires wallet context. A future `displayaddress` RPC call without wallet context could take a descriptor argument.
This commit fixes a `rpc_help.py` failure when configured with `--disable-wallet`.
ACKs for top commit:
ryanofsky:
Code review ACK 88d4d5ff2f
fanquake:
ACK 88d4d5ff2f
Tree-SHA512: 3242a24e22313aed97eee32a520bfcb1c17495ba32a2b8e06a5e151e2611320e2da5ef35b572d84623af0a49a210d2f9377a2531250868d1a0ccf3e144352a97
An earlier version of #16546 used both --with-boost-process and --enable-external-signer, which was simplified to only use the latter. However I forgot to update CI, so the external signer tests were not run.
The depends dir can not be overwritten by a FILE_ENV file. Also, a FILE_ENV file
might depend on the DEPENDS_DIR value. Thus, set it before reading FILE_ENV.
This commit does not change behavior, but is required for later commits.
Can be reviewed with --color-moved=dimmed-zebra
56ace907b9 Fix fuzz binary compilation under windows (Dan Benjamin)
Pull request description:
Small change to allow the fuzz binary to compile under windows. Also removed --disable-fuzz-binary from the windows CI test. This fixes#21212.
ACKs for top commit:
MarcoFalke:
review ACK 56ace907b9 the best bugfixes are the ones removing code
Tree-SHA512: 6088fd955a5e511b5ca1b3eaa8469a889eb6d994c2827acac7695dac6e4e320a344b45f4015a2f279b16df0d4b23ec4df13304ae6315395ad2fe8c5b526cada4
This package is currently installed as a side-effect of installing our
other libboost-*-dev packages. However as those continue to dissapear,
it makes sense to install boost dev explicitly.
3f8776a139 Re-add dead code detection (flack)
Pull request description:
This re-adds unreachable code detection for Python based on `vulture`.
Effectively, this reverts f4beb4996d. The difference to the previous version is that this runs with the `--min-confidence 100` setting. From https://pypi.org/project/vulture/:
> Use `--min-confidence 100` to only report code that is guaranteed to be unused within the analyzed files.
So this should avoid the previous issues where static analysis had wrong positives due to the dynamic nature of Python code by only reporting things that are unambiguous (such as code after a `return` statement). As such, there is not suppressions list.
My motivation was mainly #21081 which would have been caught by this (as can be seen by the CI run failing). This is still marked as draft because #21081 is needed to get the linter to pass. Also, there is a second problem that this found (see https://github.com/bitcoin/bitcoin/pull/19509/files#r571454691). From what I can tell, this is a spurious type comment that could just be removed (or if that line has no side effects it could also be deleted altogether?). I could add a commit here to fix it, but I wanted to see if there is interest in having this linter again in the first place
ACKs for top commit:
practicalswift:
ACK 3f8776a139
Tree-SHA512: 52314ad4f627d969de1eb15375ca677ed86a2e816fe773756a1ce22421214ba407b5a09a4bf701a3aab1a10c7b336f548e4cef3327edf154acba55e987db21f6
060a2a64d4 ci: remove boost thread installation (fanquake)
06e1d7d81d build: don't build or use Boost Thread (fanquake)
7097add83c refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef8 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d4
vasild:
ACK 060a2a64d4
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
2ecaf21433 gitian: remove execstack workaround for ricv64 & powerpc64le (fanquake)
5baff2b318 build: use focal in gitian descriptors (fanquake)
Pull request description:
This PR changes the gitian descriptors to use Ubuntu Focal (20.04), over Bionic (18.04), moving from GCC 7.5 to GCC 8.4 for native Linux builds, mingw-w64 GCC 7.3 to mingw-w64 GCC 9.3 for Windows builds, while continuing to use GCC 8.4 for all cross builds and Clang 8.0.0 for macOS builds.
It also drops the `-Wl,-z,noexecstack` workaround we've been using for the riscv64 and powerpc64le hosts, as it's no-longer needed. One new package is installed in the osx build, `libtinfo5`, as libtinfo5.so is required by our downloaded Clang 8.
A bump to Focal will at least be required if we want to update to a newer Qt (5.15, #19716) for 22.0, as we need a newer version of [`g++-mingw-w64`](https://packages.ubuntu.com/focal/g++-mingw-w64-x86-64) and the [`mingw-w64`](https://mingw-w64.org/doku.php) headers. This can still be done while continuing to use GCC 8.4 for Linux builds (see below), however the newer `g++-mingw-w64` will be based off of GCC 9.3.
**Some considerations**
GCC 9 is affected by #20005 "memcmp with constants that contain zero bytes are broken in GCC", and the newer `g++-mingw-w64` will be based off of GCC 9.3.
The `--no-*` variants of the Windows linker flags (i.e `--no-dynamicbase`) we use to [test our `security-check.py` script](16b784d953/contrib/devtools/test-security-check.py (L53)) are not patched into the mingw binutils in Focal (they have been re-added in Groovy (20.10)). This isn't currently an issue, however, we might add a call to `test-security-check` for Guix (#20980), and if we wanted to do the same for gitian, it would not work. Note how it's quite "easy" for us to apply the `--no-*` variant patch to our Guix build; it would be quite a bit harder to do in Gitian.
Gitian Builds @ 2ecaf21433
#### Linux
```bash
8882ea78486fbae4fac574b9089eb1107c6372d0dd7dfcda4f0f930576f9d6c1 bitcoin-2ecaf214331b-aarch64-linux-gnu-debug.tar.gz
50a9e30943b4eee5163edff3331241e745ff32a2c4463c21a6fdc5986e2d0383 bitcoin-2ecaf214331b-aarch64-linux-gnu.tar.gz
ec4e55a447fddf033fee33cd5f22bfeda3c3612f059194bcf6238859f7989d7a bitcoin-2ecaf214331b-arm-linux-gnueabihf-debug.tar.gz
444fe1b3b933c00bcbd4a9d86888cff3b61c1215b1debccd2843e842d1224777 bitcoin-2ecaf214331b-arm-linux-gnueabihf.tar.gz
88e486ff465980dc1a4aab9687d142ec6f727ed2c52cf539f69db2877dee83b2 bitcoin-2ecaf214331b-powerpc64-linux-gnu-debug.tar.gz
66144ac264c65cada9d86446e6026c85b04fb88198b8f41b42840f6031db3e6c bitcoin-2ecaf214331b-powerpc64-linux-gnu.tar.gz
34bcc13d78d929d575e34e77a6672f23ca7ea23230b28ec2eed563889352ba86 bitcoin-2ecaf214331b-powerpc64le-linux-gnu-debug.tar.gz
b4c5f959664f3063df4330edfe343c17120eb6b556ee1c15c4aeb2c1c54ffd49 bitcoin-2ecaf214331b-powerpc64le-linux-gnu.tar.gz
918fa72ab6f6ebce4e9663c93f72fe26651c260477cbb54749f7eb61438b5cc1 bitcoin-2ecaf214331b-riscv64-linux-gnu-debug.tar.gz
f704f9f8c053ffe37d854e2e81e0f4c0614c435dad7f5d82518c681b73a76ae6 bitcoin-2ecaf214331b-riscv64-linux-gnu.tar.gz
b59e3a62f1df9d79f30e916b3c9655f654036fe3a420040c53acc8dd9f4162c5 bitcoin-2ecaf214331b-x86_64-linux-gnu-debug.tar.gz
a4dc9ca877cc97544e65db11be38406d16f15d74fcdcd2318bb92474729bc60d bitcoin-2ecaf214331b-x86_64-linux-gnu.tar.gz
b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz
af6ebc91147778e4e6705eade62608dde4d6e60522d79087fa9129bdb7c01199 bitcoin-core-linux-22-res.yml
```
#### Windows
```bash
121a3970a6911cb8c453b2ce37d03f6cbb43333e29db8fa516c68563fb367f43 bitcoin-2ecaf214331b-win-unsigned.tar.gz
6294e9efebe935092f9ba119dc60ad4094f18b51c4181324e54d3057524d6101 bitcoin-2ecaf214331b-win64-debug.zip
5b5a236b63e67f5f6c07ad9aa716aa7b72fb63722c96798b332c6d164738f9cf bitcoin-2ecaf214331b-win64-setup-unsigned.exe
c1fa5894c5e02a201637567c80b9bde9024f44673dcd06fd4d489c1709179279 bitcoin-2ecaf214331b-win64.zip
b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz
665fd7eb61aed368150db58a254f15fb5efb51a4efa5abcc52571cb7a1a5de22 bitcoin-core-win-22-res.yml
```
#### macOS
```bash
6a1deae7662aa782baa82a42590f862c6bcdc4f4e38daa9b8c2a9eed1fbb5397 bitcoin-2ecaf214331b-osx-unsigned.dmg
1ee843266e84928a4323fa255c833528c2617a2c9fd2f98fb26ba19bbfc1227b bitcoin-2ecaf214331b-osx-unsigned.tar.gz
097b64dadc167d8e5b733421bf1541a40760ad952990f7cf3f35adc6ae2616d0 bitcoin-2ecaf214331b-osx64.tar.gz
b40ba2d5da498330ade92a4ccebcceb1452b94c8ffeacb336f87e93b5c88d8af src/bitcoin-2ecaf214331b.tar.gz
6e378fb543928e40c7119b96be6ff773d38506a9a888f8b02c7f1b8a0801a80e bitcoin-core-osx-22-res.yml
```
ACKs for top commit:
laanwj:
Build script changes review ACK 2ecaf21433
Tree-SHA512: 975d5830b787d2e08988f43cbc6e839294171c1d94c8219636308b05f9b77041421612ae67be24a631674670cfc9c2d96d8177f2b3158a78fc3deea19631febf
This fixes issue #19388. The changes are as follows:
- Add a new flag to configure, --enable-fuzz-binary, which allows building test/fuzz/fuzz regardless of whether we are building to do actual fuzzing
- Set -DPROVIDE_MAIN_FUNCTION whenever --enable-fuzz is no
- Add the following libraries to FUZZ_SUITE_LD_COMMON:
- LIBBITCOIN_WALLET
- SQLLITE_LIBS
- BDB_LIBS
- if necessary, some or all of:
- NATPMP_LIBS
- MINIUPNPC_LIBS
- LIBBITCOIN_ZMQ / ZMQ_LIBS
Compilers used change as follows:
Linux native GCC 7.5 -> GCC 8.4
Linux cross GCC 8.4 -> GCC 8.4
Windows mingw-w64 7.3 -> mingw-w64 9.3
macOS Clang 8.0.0 -> Clang 8.0.0
The macOS and Win cross builds in the CI are updated to use Focal, and
per the op, running the security tests is disabled in the Windows
build.
faff3991a9 ci: Fuzz with integer sanitizer (MarcoFalke)
Pull request description:
Otherwise the suppressions file will go out of sync
ACKs for top commit:
practicalswift:
cr ACK faff3991a9: patch looks correct
Tree-SHA512: 349216d071a2c5ccf24565fe0c52d7a570ec148d515d085616a284f1ab9992ce10ff82eb17962dddbcda765bbd3a9b15e8b25f34bdbed99fc36922d4161d307c
4045a6722c ci: Use cpu=1 for linter (Dhruv Mehta)
739d39022d ci: Move linter task to cirrus (Dhruv Mehta)
Pull request description:
Solves #20467: Move linter to Cirrus-CI as Travis-CI.org is shutting down
ACKs for top commit:
MarcoFalke:
ACK 4045a6722c
Tree-SHA512: 9aa7487ac86c91fc68bb584d29134e304dbd46702514a5d47d1ef0de6b877d96d42b7589870fc67ad9a31f5d3a789728446da4418688f336111a9ba0f8de5feb
xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
more maintained, and has the right toggles for us to achieve output
determinism without using blunt tools like faketime.
In this commit, we use xorrisofs from the build environment rather than
building it ourselves using depends. This is not necessary and can be
changed in the future.
From https://wiki.debian.org/genisoimage?action=recall&rev=11 :
> The classical command line interface for production of ISO 9660
> filesystem images is the option set established by program mkisofs.
> For reasons of licensing and other problems with its author, Debian
> ships a fork of mkisofs, called genisoimage, which was split off in
> 2006 and then developed independently.
>
> Meanwhile, genisoimage gets no new features and not even bug fixes. It
> is first choice only if its options -udf or -hfs are needed.
>
> Replacement in most uses cases, especially for bootable ISO 9660
> filesystems, archiving, and backup, is xorrisofs which starts the -as
> mkisofs emulation mode of program xorriso.
This adds a brief help text to `git-subtree-check.sh` and adds and an
option to do a full remote check instead of having two different code
paths with a successful exit status. Also make it explicit that the CI
is not doing this.
c82d15b6d1 depends: Do not force Precompiled Headers (PCH) for building Qt on Linux (Hennadii Stepanov)
Pull request description:
On CentOS 8 (Cirrus CI job) the forced `-pch` option breaks Qt build.
Removing `-pch` option does not affect build time for other systems:
- master (e2ff5e7b35):
```
$ time make -j 9 -C depends/ qt
...
Caching qt...
make: Leaving directory '/home/hebasto/guix/GitHub/bitcoin/depends'
real 4m22,359s
user 18m3,719s
sys 1m24,769s
```
- this PR:
```
$ time make -j 9 -C depends/ qt
...
Caching qt...
make: Leaving directory '/home/hebasto/guix/GitHub/bitcoin/depends'
real 4m14,862s
user 18m3,355s
sys 1m24,506s
```
Qt docs: https://doc.qt.io/qt-5/qmake-precompiledheaders.htmlFixes#20423
ACKs for top commit:
MarcoFalke:
review ACK c82d15b6d1
Tree-SHA512: 0f2a3712e90de881d00f8e56c363edde33dd4f5c117df5744ab4e51d0a8146331de7236bc8329d68ddd91535cd853e68ee80ef4cceb6a909786abfd8881b01e8
a52ecc936a build: set minimum supported macOS to 10.14 (fanquake)
Pull request description:
This is a requirement for C++17 support. See my comments [here](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-643722538):
> You cannot use std::get with std::variant on macOS < 10.14, because Apples libc++ doesn't support the std::bad_variant_access exception. [Relevant comment](https://github.com/bitcoin/bitcoin/pull/19183#discussion_r439794318) in #19183.
> While we could work around this in our own code, using std::get_if, this would still be a problem for 3rd-party dependencies.
> I've been testing Qt 5.15LTS (we'll have to enable C++17 in qt, and may upgrade to a newer version at the same time), and you can't enable -std c++17, while targeting a macOS deployment version < 10.14, configuring will fail. They are making use of std::get with std::variant throughout their cocoa code.
We would have to had to have bumped to at least 10.13 in any case, as Qt 5.15 (#19716) [requires 10.13+](https://doc.qt.io/qt-5/supported-platforms.html).
ACKs for top commit:
hebasto:
ACK a52ecc936a, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: f669b2fc777aeea1e9afdbbc7bd9afe3997418211db6ba53c934cae0e62a9b999603da539518c229f34961d275c9e2f315c7b022cf5fb97bd201a69c85d470cc
97c738ff1b [tests] Recommend f-strings for formatting, update feature_block to use them (Anthony Towns)
8ae9d314e9 Bump minimum python version to 3.6 (Anthony Towns)
Pull request description:
Python 3.5 has reached [end-of-life](https://devguide.python.org/#status-of-python-branches) as of September 2020, and 3.6 has some moderately nice [features](https://docs.python.org/3/whatsnew/3.6.html):
- `f'x = {x}'` as an alternative to `'x = {}'.format(x)` format strings (cf https://github.com/bitcoin/bitcoin/pull/13718#issuecomment-406591027)
- underscore separators for large numbers, like `1_234_567`
- improvements to async
- improvements to typing module
Note that 3.6 is not available in xenial (16.04), but is available in bionic (18.04), while focal (20.04) has 3.8. CentOS 7 and 8 have 3.6.8, Debian stable has 3.7.3, and [gentoo and arch already had 3.6 and 3.7 in 2018](https://github.com/bitcoin/bitcoin/pull/14954#issuecomment-447118707).
ACKs for top commit:
MarcoFalke:
re-ACK 97c738ff1b
Tree-SHA512: ec7fce68845edde4d61a42de12c065fd49e5217311a6fda1323206f091a0afd50f293645dffc27d420127e4e5deb864e953f1b67eff735a0dfbbedd7899a9d60
Without the fix, the tests immediately abort:
terminate called after throwing an instance of 'std::runtime_error'
what(): locale::facet::_S_create_c_locale name not valid
./qt/test/test_bitcoin-qt: line 2: 116150 Aborted
This avoids timeouts when signing a large raw transaction
https://cirrus-ci.com/task/5009228131729408?command=ci#L4981
test_framework.authproxy.JSONRPCException: 'signrawtransactionwithwallet' RPC took longer than 120.000000 seconds. Consider using larger timeout for calls that take longer to return. (-344)
0e2a5e448f tests: dumping and minimizing of script assets data (Pieter Wuille)
4567ba034c tests: add generic qa-asset-based script verification unit test (Pieter Wuille)
f06e6d0345 tests: functional tests for Schnorr/Taproot/Tapscript (Pieter Wuille)
3c226639eb tests: add BIP340 Schnorr signature support to test framework (Pieter Wuille)
206fb180ec --- [TAPROOT] Tests --- (Pieter Wuille)
d7ff237f29 Activate Taproot/Tapscript on regtest (BIP 341, BIP 342) (Pieter Wuille)
e9a021d7e6 Make Taproot spends standard + policy limits (Pieter Wuille)
865d2c37e2 --- [TAPROOT] Regtest activation and policy --- (Pieter Wuille)
72422ce396 Implement Tapscript script validation rules (BIP 342) (Johnson Lau)
330de894a9 Use ScriptExecutionData to pass through annex hash (Pieter Wuille)
8bbed4b7ac Implement Taproot validation (BIP 341) (Pieter Wuille)
0664f5fe1f Support for Schnorr signatures and integration in SignatureCheckers (BIP 340) (Pieter Wuille)
5de246ca81 Implement Taproot signature hashing (BIP 341) (Johnson Lau)
9eb590894f Add TaggedHash function (BIP 340) (Pieter Wuille)
450d2b2371 --- [TAPROOT] BIP340/341/342 consensus rules --- (Pieter Wuille)
5d62e3a68b refactor: keep spent outputs in PrecomputedTransactionData (Pieter Wuille)
8bd2b4e784 refactor: rename scriptPubKey in VerifyWitnessProgram to exec_script (Pieter Wuille)
107b57df9f scripted-diff: put ECDSA in name of signature functions (Pieter Wuille)
f8c099e220 --- [TAPROOT] Refactors --- (Pieter Wuille)
Pull request description:
This is an implementation of the Schnorr/taproot consensus rules proposed by BIPs [340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), [341](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), and [342](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
See the list of commits [below](https://github.com/bitcoin/bitcoin/pull/19953#issuecomment-691815830). No signing or wallet support of any kind is included, as testing is done entirely through the Python test framework.
This is a successor to https://github.com/bitcoin/bitcoin/pull/17977 (see discussion following [this comment](https://github.com/bitcoin/bitcoin/pull/17977#issuecomment-682285983)), and will have further changes squashed/rebased. The history of this PR can be found in #19997.
ACKs for top commit:
instagibbs:
reACK 0e2a5e448f
benthecarman:
reACK 0e2a5e4
kallewoof:
reACK 0e2a5e448f
jonasnick:
ACK 0e2a5e448f almost only looked at bip340/libsecp related code
jonatack:
ACK 0e2a5e448f modulo the last four commits (tests) that I plan to finish reviewing tomorrow
fjahr:
reACK 0e2a5e448f
achow101:
ACK 0e2a5e448f
Tree-SHA512: 1b00314450a2938a22bccbb4e177230cf08bd365d72055f9d526891f334b364c997e260c10bc19ca78440b6767712c9feea7faad9a1045dd51a5b96f7ca8146e
This adds a unit test that does generic script verification tests,
with positive/negative witnesses/scriptsigs, under various flags.
The test data is large (several MB) so it's stored in the qa-assets
repo.
33df8d46bb ci: Build Arm64 on Travis without functional tests (Fabian Jahr)
Pull request description:
The Travis Arm64 env is a pretty big PITA for quite a while now. It simply seems to time out due to a lack of resources as far as I can tell from my research into the matter.
I have reported the issue in August and received no response: https://travis-ci.community/t/arm64-failing-without-message/9775. Others have complained about similar issues with Arm64 over the past year. The explanation for this is probably that this env is still considered to be in beta: https://docs.travis-ci.com/user/multi-cpu-architectures.
Considering the high frequency of failures (I didn't run any numbers, just based on my personal observations) and the beta status I would suggest that we shorten the runtime by removing part of the test suite until the env is more stable. I would suggest removing functional tests since it should help for sure but maybe there are other ideas.
ACKs for top commit:
MarcoFalke:
ACK 33df8d46bb
Tree-SHA512: 55b0c658b526611d16a26e9508611ddeecbfbd842ec064ada61d188a8be2f473e080c2a35746a9aae8006b11b0c2ab9dd4639b0f3004ecf821506e26c7345972
fdf697fe75 ci: Increase CCACHE_SIZE in some builds on Travis (Hennadii Stepanov)
Pull request description:
This PR will decrease build time for the longest Travis builds.
Up to ~20 min improvement with warmed up cache is expected.
Top commit has no ACKs.
Tree-SHA512: ee3a22162d03537be11e60b53de043247f6a65e35e630e0807b758a846b7e05ef2059d18846644aafa3cd5dc7d3e3f56a8ccb94cc71b95227debc9201d3142a2
870f0cd2a0 build: Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory (practicalswift)
Pull request description:
Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory.
First UBSan, then ASan followed by TSan... and now: yes, the wait is over -- **MSan is finally here!** :)
Some historical context:
* 2017: Continuous compilation with Clang Thread Safety analysis enabled (#10866, #10923)
* 2018: Continuous testing with trapping on signed integer overflows (`-ftrapv`) (#12686)
* 2018: Continuous testing of use of locale dependent functions (#13041)
* 2018: Continuous testing of format strings (#13705)
* 2018: Continuous compilation with MSVC `TreatWarningAsError` (#14151)
* 2018: Continuous testing under UndefinedBehaviorSanitizer – UBSan (#14252, #14673, #17006)
* 2018: Continuous testing under AddressSanitizer – ASan (#14794, #17205, #17674)
* 2018: Continuous testing under ThreadSanitizer – TSan (#14829)
* 2019: Continuous testing in an unsigned char environment (`-funsigned-char`) (#15134)
* 2019: Continuous compile-time testing of assumptions we're making (#15391)
* 2019: Continuous testing of fuzz test cases under Valgrind (#17633, #18159, #18166)
* 2020: Finally... MemorySanitizer – MSAN! :)
What is the next step? What tools should we add to CI to keep bugs from entering `master`? :)
ACKs for top commit:
MarcoFalke:
ACK 870f0cd2a0
Tree-SHA512: 38327c8b75679d97d469fe42e704cacd1217447a5a603701dd8a58ee50b3be2c10248f8d68a479ed081c0c4b254589d3081c9183f991640b06ef689061f75578
faca73000f ci: Install fixed version of clang-format for linters (MarcoFalke)
fa4695da4c build: Sort Makefile.am after renaming file (MarcoFalke)
cccc2784a3 scripted-diff: Move ui_interface to the node lib (MarcoFalke)
fa72ca6a9d qt: Remove unused includes (MarcoFalke)
fac96e6450 wallet: Do not include server symbols (MarcoFalke)
fa0f6c58c1 Revert "Fix link error with --enable-debug" (MarcoFalke)
Pull request description:
This reverts a hacky workaround from commit b83cc0f, which only happens to work due to compiler optimizations. Then, it actually fixes the linker error.
The underlying problem is that the wallet includes symbols from the server (ui_interface), which usually results in linker failures. Though, in this specific case the linker failures have not been observed (unless `-O0`) because our compilers were smart enough to strip unused symbols.
Fix the underlying problem by creating a new header-only with the needed symbol and move ui_interface to node to clarify that this is part of libbitcoin_server.
ACKs for top commit:
Sjors:
ACK faca730
laanwj:
ACK faca73000f
hebasto:
re-ACK faca73000f, since the [previous](https://github.com/bitcoin/bitcoin/pull/19331#pullrequestreview-434420539) review:
Tree-SHA512: e9731f249425aaea50b6db5fc7622e10078cf006721bb87989cac190a2ff224412f6f8a7dd83efd018835302337611f5839e29e15bef366047ed591cef58dfb4
a92e48b02d test: move TEST_RUNNER_EXTRA into native tsan setup (fanquake)
Pull request description:
`feature_block.py` is being run in the tsan job, i.e [here](https://travis-ci.org/github/bitcoin/bitcoin/jobs/703122309), even though it should be excluded. My hasty assumption is that this will fix it. In any case, all other instances of `TEST_RUNNER_EXTRA` seem to have moved out of `.travis.yml` and into the different CI configurations.
ACKs for top commit:
MarcoFalke:
ACK a92e48b02d
practicalswift:
ACK a92e48b02d -- patch looks correct
hebasto:
ACK a92e48b02d, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 86057bef2cc87c6acdbbf94f8cd7a5147510448c3e67aacde8daf247e3ccf649cfc5afbbd10693e084f426042d98150616c0e49bfa5f32b949dff9cebd2fd95d
