fa340b8794 refactor: Avoid magic value of all-zeros in assumeutxo base_blockhash (MarcoFalke)
fae33f98e6 Fix assumeutxo crash due to invalid base_blockhash (MarcoFalke)
fa5668bfb3 refactor: Use type-safe assumeutxo hash (MarcoFalke)
0000007709 refactor: Remove unused code (MarcoFalke)
faa921f787 move-only: Add util/hash_type (MarcoFalke)
Pull request description:
Starting with commit d6af06d68a, a block hash of all-zeros is invalid and will lead to a crash of the node. Can be tested by cherry-picking the test changes without the other changes.
Stack trace (copied from https://github.com/bitcoin/bitcoin/pull/21584#discussion_r612673879):
```
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff583c8b1 in __GI_abort () at abort.c:79
#2 0x00007ffff582c42a in __assert_fail_base (fmt=0x7ffff59b3a38 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x555556c8b450 "!hashBlock.IsNull()", file=file@entry=0x555556c8b464 "txdb.cpp", line=line@entry=89,
function=function@entry=0x555556c8b46d "virtual bool CCoinsViewDB::BatchWrite(CCoinsMap &, const uint256 &)") at assert.c:92
#3 0x00007ffff582c4a2 in __GI___assert_fail (assertion=0x555556c8b450 "!hashBlock.IsNull()", file=0x555556c8b464 "txdb.cpp", line=89,
function=0x555556c8b46d "virtual bool CCoinsViewDB::BatchWrite(CCoinsMap &, const uint256 &)") at assert.c:101
#4 0x000055555636738b in CCoinsViewDB::BatchWrite (this=0x5555577975c0, mapCoins=std::unordered_map with 110 elements = {...}, hashBlock=...) at txdb.cpp:89
#5 0x00005555564a2e80 in CCoinsViewBacked::BatchWrite (this=0x5555577975f8, mapCoins=std::unordered_map with 110 elements = {...}, hashBlock=...) at coins.cpp:30
#6 0x00005555564a43de in CCoinsViewCache::Flush (this=0x55555778eaf0) at coins.cpp:223
#7 0x00005555563fc11d in ChainstateManager::PopulateAndValidateSnapshot (this=0x55555740b038 <g_chainman>, snapshot_chainstate=..., coins_file=..., metadata=...)
at validation.cpp:5422
#8 0x00005555563fab3d in ChainstateManager::ActivateSnapshot (this=0x55555740b038 <g_chainman>, coins_file=..., metadata=..., in_memory=true) at validation.cpp:5299
#9 0x0000555555e8c893 in validation_chainstatemanager_tests::CreateAndActivateUTXOSnapshot<validation_chainstatemanager_tests::chainstatemanager_activate_snapshot::test_method()::$_12>(NodeContext&, boost::filesystem::path, validation_chainstatemanager_tests::chainstatemanager_activate_snapshot::test_method()::$_12) (node=...,
root=..., malleation=...) at test/validation_chainstatemanager_tests.cpp:199
#10 0x0000555555e8877a in validation_chainstatemanager_tests::chainstatemanager_activate_snapshot::test_method (this=0x7fffffffc8d0)
at test/validation_chainstatemanager_tests.cpp:262
ACKs for top commit:
laanwj:
Code review re-ACK fa340b8794
jamesob:
ACK fa340b8794 ([`jamesob/ackr/21584.1.MarcoFalke.fix_assumeutxo_crash_due`](https://github.com/jamesob/bitcoin/tree/ackr/21584.1.MarcoFalke.fix_assumeutxo_crash_due))
Tree-SHA512: c2c4e66c1abfd400ef18a04f22fec1f302f1ff4d27a18050f492f688319deb4ccdd165ff792eee0a1f816e7b69fb64080662b79517ab669e3d26b9eb77802851
792be53d3e refactor: Replace std::bind with lambdas (Hennadii Stepanov)
a508f718f3 refactor: Use appropriate thread constructor (Hennadii Stepanov)
30e4448215 refactor: Make TraceThread a non-template free function (Hennadii Stepanov)
Pull request description:
This PR does not change behavior.
Its goal is to improve readability and maintainability of the code.
ACKs for top commit:
jnewbery:
utACK 792be53d3e
jonatack:
tACK 792be53d3e
MarcoFalke:
cr ACK 792be53d3e
Tree-SHA512: a03142f04f370f6bc02bd3ddfa870819b51740fcd028772241d68c84087f95a2d78207cbd5edb3f7c636fcf2d76192d9c59873f8f0af451d3b05c0cf9cf234df
5f96d7d22d rpc: gettxoutsetinfo rejects hash_serialized_2 for specific height (Fabian Jahr)
23fe50436b test: Add test for coinstatsindex behavior in reorgs (Fabian Jahr)
90c966b0f3 rpc: Allow gettxoutsetinfo and getblockstats for stale blocks (Fabian Jahr)
b9362392ae index, rpc: Add use_index option for gettxoutsetinfo (Fabian Jahr)
bb7788b121 test: Test coinstatsindex robustness across restarts (Fabian Jahr)
e0938c2909 test: Add tests for block_info in gettxoutsetinfo (Fabian Jahr)
2501576ecc rpc, index: Add verbose amounts tracking to Coinstats index (Fabian Jahr)
655d929836 test: add coinstatsindex getindexinfo coverage, improve current tests (Jon Atack)
ca01bb8d68 rpc: Add Coinstats index to getindexinfo (Fabian Jahr)
57a026c30f test: Add unit test for Coinstats index (Fabian Jahr)
6a4c0c09ab test: Add functional test for Coinstats index (Fabian Jahr)
3f166ecc12 rpc: gettxoutsetinfo can be requested for specific blockheights (Fabian Jahr)
3c914d58ff index: Coinstats index can be activated with command line flag (Fabian Jahr)
dd58a4de21 index: Add Coinstats index (Fabian Jahr)
a8a46c4b3c refactor: Simplify ApplyStats and ApplyHash (Fabian Jahr)
9c8a265fd2 refactor: Pass hash_type to CoinsStats in stats object (Fabian Jahr)
2e2648a902 crypto: Make MuHash Remove method efficient (Fabian Jahr)
Pull request description:
This is part of the coinstats index project tracked in #18000
While the review of the new UTXO set hash algorithm (MuHash) takes longer recently #19328 was merged which added the possibility to run `gettxoutsetinfo` with a specific hash type. As the first type it added `hash_type=none` which skips the hashing of the UTXO set altogether. This alone did not make `gettxoutsetinfo` much faster but it allows the use of an index for the remaining coin statistics even before a new hashing algorithm has been added. Credit to Sjors for the idea to take this intermediate step.
Features summary:
- Users can start their node with the option `-coinstatsindex` which syncs the index in the background
- After the index is synced the user can use `gettxoutsetinfo` with `hash_type=none` or `hash_type=muhash` and will get the response instantly out of the index
- The user can specify a height or block hash when calling `gettxoutsetinfo` to see coin statistics at a specific block height
ACKs for top commit:
Sjors:
re-tACK 5f96d7d22d
jonatack:
Code review re-ACK 5f96d7d22d per `git range-diff 13d27b4 07201d3 5f96d7d`
promag:
Tested ACK 5f96d7d22d. Light code review ACK 5f96d7d22d.
Tree-SHA512: cbca78bee8e9605c19da4fbcd184625fb280200718396c694a56c7daab6f44ad23ca9fb5456d09f245d8b8d9659fdc2b3f3ce5e953c1c6cf4003dbc74c0463c2
Add simple interfaces::Echo IPC interface with one method that just takes and
returns a string, to test multiprocess framework and provide an example of how
it can be used to spawn and call between processes.
615965cfd1 Move common package version code to init/common (Russell Yanofsky)
5bed2ab42c Move common logging start code to init/common (Russell Yanofsky)
1fb7fcfa52 Move common logging GetArgs code to init/common (Russell Yanofsky)
90469c1690 Move common logging AddArg code to init/common (Russell Yanofsky)
387c4cf588 Move common sanity check code to init/common (Russell Yanofsky)
a67b54855b Move common global init code to init/common (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
This change is move-only and can be easily reviewed with `--color-moved=dimmed_zebra`. The moves are needed to avoid duplicating common init code between different binaries (`bitcoin-node`, `bitcoin-wallet`, etc) in #10102. In #10102, each binary has it's own init file (`src/init/bitcoin-node.cpp`, `src/init/bitcoin-wallet.cpp`) so this PR moves the common code to `src/init/common.cpp`.
ACKs for top commit:
MarcoFalke:
review ACK 615965cfd1 🖱
practicalswift:
cr ACK 615965cfd1: dimmed zebra looks correct
Tree-SHA512: 859e1d86aee17eb50a49d806cf62d30d12f6b15018e41c096da41d7e535a9d2d088481cb340fee59e6c68e512a74b61c7146f2683465f553dc4953bf32f2a7b4
fadcd3f78e doc: Remove irrelevant link to GitHub (MarcoFalke)
fa121b628d blockstorage: [refactor] Use chainman reference where possible (MarcoFalke)
fa0c7d9ad2 move-only: Move *Disk functions to blockstorage (MarcoFalke)
fa91b2b2b3 move-only: Move AbortNode to shutdown (MarcoFalke)
fa413f07a1 move-only: Move ThreadImport to blockstorage (MarcoFalke)
faf843c07f refactor: Move load block thread into ChainstateManager (MarcoFalke)
Pull request description:
This picks up the closed pull request #21030 and is the first step toward fixing #21220.
The basic idea is to move all disk access into a separate module with benefits:
* Breaking down the massive files init.cpp and validation.cpp into logical units
* Creating a standalone-module to reduce the mental complexity
* Pave the way to fix validation related circular dependencies
* Pave the way to mock disk access for testing, especially where it is performance critical (like fuzzing)
ACKs for top commit:
promag:
Code review ACK fadcd3f78e, checked (almost) moved only changes. This is a nice tidy up change and doesn't change behavior. Easily reviewed commit by commit.
jamesob:
ACK fadcd3f78e ([`jamesob/ackr/21575.1.MarcoFalke.refactor_create_blocksto`](https://github.com/jamesob/bitcoin/tree/ackr/21575.1.MarcoFalke.refactor_create_blocksto))
ryanofsky:
Code review ACK fadcd3f78e. New organization makes sense, moves extraneous things outside of validation.cpp. PR is also easy to review with helpfully split up moveonly commits.
Tree-SHA512: 917996592b6d8f9998289d8cb2b1b78b23d1fdb3b07216c9caec1380df33baa09dc2c1e706da669d440b497e79c9c62a01ca20dc202df5ad974a75f3ef7a143b
88d4d5ff2f rpc: add help for enumeratesigners and walletdisplayaddress (Sjors Provoost)
b0db187e5b ci: use --enable-external-signer instead of --with-boost-process (Sjors Provoost)
b54b2e7b1a Move external signer out of wallet module (Sjors Provoost)
Pull request description:
In addition, this PR enables external signer testing on CI.
This PR moves the ExternalSigner class and RPC methods out of the wallet module.
The `enumeratesigners` RPC can be used without a wallet since #21417. With additional modifications external signers could be used without a wallet in general, e.g. via `signrawtransaction`.
The `signerdisplayaddress` RPC is ranamed to `walletdisplayaddress` because it requires wallet context. A future `displayaddress` RPC call without wallet context could take a descriptor argument.
This commit fixes a `rpc_help.py` failure when configured with `--disable-wallet`.
ACKs for top commit:
ryanofsky:
Code review ACK 88d4d5ff2f
fanquake:
ACK 88d4d5ff2f
Tree-SHA512: 3242a24e22313aed97eee32a520bfcb1c17495ba32a2b8e06a5e151e2611320e2da5ef35b572d84623af0a49a210d2f9377a2531250868d1a0ccf3e144352a97
This commit moves the ExternalSigner class and RPC methods out of the wallet module.
The enumeratesigners RPC can be used without a wallet since #21417.
With additional modifications external signers could be used without a wallet in general, e.g. via signrawtransaction.
The signerdisplayaddress RPC is ranamed to walletdisplayaddress because it requires wallet context.
A future displayaddress RPC call without wallet context could take a descriptor argument.
This commit fixes a rpc_help.py failure when configured with --disable-wallet.
ebc4ab721b refactor: post Optional<> removal cleanups (fanquake)
57e980d13c scripted-diff: remove Optional & nullopt (fanquake)
Pull request description:
Same rationale & motivation as #21404, which turned out to be quite low in the number of potential conflicts. Lets see what the bot has to say here.
ACKs for top commit:
practicalswift:
cr ACK ebc4ab721b: patch looks correct
jnewbery:
utACK ebc4ab721b
laanwj:
Code review ACK ebc4ab721b
Tree-SHA512: 550fbeef09b9d35ddefaa805d1755c18c8fd499c4b0f77ebfece8c20296a7abd1cf6c699e2261f92fe3552deeb7555ec2a2287ffe3ab9e98bb9f8612a4d43be3
52f0be3a93 compat: remove memcpy -> memmove backwards compatibility alias (fanquake)
Pull request description:
In glib 2.13 memcpy was changed such that the way it copied bytes was reversed.
This caused all sorts of issues for existing software, which depended on the
existing behavior (when they should have been using memmove). See:
https://sourceware.org/bugzilla/show_bug.cgi?id=12518https://bugzilla.redhat.com/show_bug.cgi?id=638477
Now that we require glibc 2.17+ (#17538), we should be well clear of having to
maintain our memcpy -> memmove aliasing, which was introduced in #4339.
Gitian builds:
```bash
# Linux:
52dee59c8c7d5620ac9b140b79fcaf3d2f15a219293140190f9283ba871f5391 bitcoin-52f0be3a9332-aarch64-linux-gnu-debug.tar.gz
8963473b8791c5c6033a992d7dd761832fe1fb5732be790a6e9f8c11d67ad8ae bitcoin-52f0be3a9332-aarch64-linux-gnu.tar.gz
1fb3365c1ef60ecd1eb2d18f671f8f1e8cde0585de7de74aa0c5121093100c26 bitcoin-52f0be3a9332-arm-linux-gnueabihf-debug.tar.gz
305c5b032d51ba97459715211112204a09d119edd6ec2a12b796559ad3fde761 bitcoin-52f0be3a9332-arm-linux-gnueabihf.tar.gz
1f950a3e3979a4e1a67696b3fddc3090a0489a43b49e2b58a348d4b02ada2aa8 bitcoin-52f0be3a9332-powerpc64-linux-gnu-debug.tar.gz
0b9731dba768b30c91dadec4cd7a98c86e06fbf6354555f798b46b7c4fab7b5f bitcoin-52f0be3a9332-powerpc64-linux-gnu.tar.gz
c4a37aae56cc023964f8d9e82d1b66913079cab559cbfc1c9127969aa968a06f bitcoin-52f0be3a9332-powerpc64le-linux-gnu-debug.tar.gz
dfbaa4f3bf12988a0a7f82c4b10162e5e7a63382a7e29d0170bc32ce344c97c3 bitcoin-52f0be3a9332-powerpc64le-linux-gnu.tar.gz
3a0280d2c06516e50b0841d6f42d9589355dc9a1f8bb9a0b123554cd91b08004 bitcoin-52f0be3a9332-riscv64-linux-gnu-debug.tar.gz
cc199a0f254b2366e80a6a884120ec3ea442983990ba1a5eb993c36060686eba bitcoin-52f0be3a9332-riscv64-linux-gnu.tar.gz
eb8e7ca673cc06c167ab082fe457a41f73758ecd5b34941300e3cd378c29b197 bitcoin-52f0be3a9332-x86_64-linux-gnu-debug.tar.gz
dad19226c0e4c54b78ca2fa85fc28c5bfd1e1178e3f765472bd2f895a1d57145 bitcoin-52f0be3a9332-x86_64-linux-gnu.tar.gz
ef89be95b84bb7c6fef055634cd20caf2fa5b42441502918dbfbf758bb2daab6 src/bitcoin-52f0be3a9332.tar.gz
dc61f5ca33330c1609bc56b23f39fef3c1ff5ec6a1799d5b7a18f3c3b3acc9f9 bitcoin-core-linux-22-res.yml
```
Guix builds:
```bash
b50d6399cb59e5e4a9247b12a3eda61de6e51bd87ef1f27b388b75b71dfccf92 output/bitcoin-52f0be3a9332-aarch64-linux-gnu-debug.tar.gz
23d845dc13e60a581ebdfbaa6063f559a56cce06734e1b50790d2fc13e257793 output/bitcoin-52f0be3a9332-aarch64-linux-gnu.tar.gz
79094406fe00939bbce17a6d65de5a2686625e871432350c69e674cc80b1491c output/bitcoin-52f0be3a9332-arm-linux-gnueabihf-debug.tar.gz
65a91913249a743015eceea5a56c497d606af17270cb7e8a3df10cf729b757ec output/bitcoin-52f0be3a9332-arm-linux-gnueabihf.tar.gz
5e75ca5e8cf6934ba5a5a1b4d26c1b361b118e10ef34b73845d038035ddb9b85 output/bitcoin-52f0be3a9332-osx-unsigned.dmg
774b372696cde8ceab40f6909dadea3fc87b375b495fcfb4ee8a963afd7fbd3a output/bitcoin-52f0be3a9332-osx-unsigned.tar.gz
dc4bdfb7b32dcc0b6e876d6d7ab3cb8d1472f21f66546ab70515f96262292e21 output/bitcoin-52f0be3a9332-osx64.tar.gz
ea178ff9e28439f80129445cf260215c74eea2e610f62ff045061f287675d3ff output/bitcoin-52f0be3a9332-powerpc64-linux-gnu-debug.tar.gz
0390687a7aaa3f0a8a78be2deab21116599e5b332f00a2d1fdce97a5bd30e3eb output/bitcoin-52f0be3a9332-powerpc64-linux-gnu.tar.gz
52c948719a27f252f5969558abc2718c1e365ea85496322cb4ec97eab8a234cc output/bitcoin-52f0be3a9332-powerpc64le-linux-gnu-debug.tar.gz
5a4a8748dffe7e6a5bd07f3f564b1f2052440c4199fe25aaa41675bfb69e61db output/bitcoin-52f0be3a9332-powerpc64le-linux-gnu.tar.gz
ba521bd2b4e73aea317821a9e08da9a326c0be3b38d923b35ba14bc68ee6c814 output/bitcoin-52f0be3a9332-riscv64-linux-gnu-debug.tar.gz
783ea81ab2f6b642b13ebf7882aa822d12f95936574a8848a74b1b8978e6801d output/bitcoin-52f0be3a9332-riscv64-linux-gnu.tar.gz
376706fc12e58d7d559a87e1ce64be22eaac3fc32d95c60d603ad893d9128cc1 output/bitcoin-52f0be3a9332-win-unsigned.tar.gz
7aa48242fb71e29b00992b2be8677f1ea49f2ca82c5355bf0c1d4c8d14635596 output/bitcoin-52f0be3a9332-win64-debug.zip
41e6461ab573fa8f6ac0f198193e72a4a047bb7a4193f743b937e81739c929cc output/bitcoin-52f0be3a9332-win64-setup-unsigned.exe
e2c4ecb05f24577da12f722d848bf6ac89f3f549d6d2bfd30d65676099c0725b output/bitcoin-52f0be3a9332-win64.zip
60ed63b3b562fa2141f18f1556a03c2474b75797088cd68fdb3e7d057a6983a3 output/bitcoin-52f0be3a9332-x86_64-linux-gnu-debug.tar.gz
adb0bb62dc8b99d025a863e921b8e670f4c8f4b5600cd6d79eb552ede10bc8b8 output/bitcoin-52f0be3a9332-x86_64-linux-gnu.tar.gz
ef89be95b84bb7c6fef055634cd20caf2fa5b42441502918dbfbf758bb2daab6 output/src/bitcoin-52f0be3a9332.tar.gz
```
ACKs for top commit:
laanwj:
Concept and code review ACK 52f0be3a93
Tree-SHA512: 851634a633cc7d27b10f11436768f3695a7615d5850166c3718028c36d3a7dd56baa2dd1028f47802891703e9f5a1d382f559e388ecef2249e2004edc62d97bf
1a6323bdbe doc: update developer notes for removal of MakeUnique (fanquake)
3ba2840e7e scripted-diff: remove MakeUnique<T>() (fanquake)
Pull request description:
Since requiring C++17, this is just pointless abstraction. I think we should just "tear the band-aid off" and remove it. Similar to the changes happening in #21366.
Also, having a comment saying this is deprecated doesn't prevent it's usage in new code. i.e : https://github.com/bitcoin/bitcoin/pull/20946#discussion_r561949731.
The repository is fairly quiet at the moment, so any potential complaints about having to rebase should be minimal. Might as well get this over and done with.
ACKs for top commit:
jnewbery:
utACK 1a6323bdbe
practicalswift:
cr ACK 1a6323bdbe: patch looks correct
ajtowns:
ACK 1a6323bdbe -- code review only
glozow:
ACK 1a6323bdbe looks correct
Tree-SHA512: 4a14b9611b60b9b3026b54d6f5a2dce4c5d9b63a7b93d7de1307512df736503ed84bac66e7b93372c76e3117f49bf9f29cd473d3a47cb41fb2775bc10234736f
e017a913d0 bitcoind: Add -daemonwait option to wait for initialization (Wladimir J. van der Laan)
c3e6fdee6d shutdown: Use RAII TokenPipe in shutdown (Wladimir J. van der Laan)
612f746a8f util: Add RAII TokenPipe (Wladimir J. van der Laan)
Pull request description:
This adds a `-daemonwait` flag that does the same as `-daemon` except that it, from a user perspective, backgrounds the process only after initialization is complete. This is similar to the behaviour of some other software such as c-lightning.
This can be useful when the process launching bitcoind wants to guarantee that either the RPC server is running, or that initialization failed, before continuing. The exit code indicates the initialization result.
The use of the libc function `daemon()` is replaced by a custom implementation which is inspired by the [glibc implementation](https://github.com/lattera/glibc/blob/master/misc/daemon.c#L44), but which also creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is that no MACOS-specific pragmas are needed anymore to silence a deprecation warning.
TODO:
- [x] Factor out `token_read` and `token_write` to an utility, and use them in `shutdown.cpp` as well—this is exactly the same kind of communication mechanism.
- [x] RAII-ify pipe endpoints.
- [x] Improve granularity of the `configure.ac` checks. This currently still checks for the function `daemon()` which makes no sense as it's not used. It should check for individual functions such as
`fork()` and `setsid()` etc—the former being required, the second optional.
- [-] ~~Signal propagation during initialization: if say, pressing Ctrl-C during `-daemonwait` it would be good to pass this SIGINT on to the child process instead of detaching the parent process and letting the child run free.~~ This is not necessary, see https://github.com/bitcoin/bitcoin/pull/21007#issuecomment-769007341.
Future:
- Consider if it makes sense to use this in the RPC tests (there would be no more need for "is RPC ready" polling loops). I think this is out of scope for this PR.
ACKs for top commit:
jonatack:
Tested ACK e017a913d0 checked change since previous review is move-only
Tree-SHA512: 53369b8ca2247e4cf3af8cb2cfd5b3399e8e0e3296423d64be987004758162a7ddc1287b01a92d7692328edcb2da4cf05d279b1b4ef61a665b71440ab6a6dbe2
In glib 2.13 memcpy was changed such that the way it copied bytes was reversed.
This caused all sorts of issues for existing software, which depended on the
existing behavior (when they should have been using memmove). See:
https://sourceware.org/bugzilla/show_bug.cgi?id=12518https://bugzilla.redhat.com/show_bug.cgi?id=638477
Now that we require glibc 2.17+ (#17538), we should be well clear of having to
maintain our memcpy -> memmove aliasing, which was introduced in #4339.
7af25024e9 build: compile libnatpmp with -DNATPMP_STATICLIB on Windows (fanquake)
ee35745754 build: use newer source for libnatpmp (fanquake)
Pull request description:
The source we are currently using is from 2015. The upstream repo has
received a small number of bug fixes and improvements since then.
Including one that fixes an issue for Windows users: https://github.com/miniupnp/libnatpmp/pull/13.
The source we are currently using is the most recent "official" release,
however I don't think it's worth waiting for a new one. The maintainer
was prompted to do so in Oct 2020, then again in Jan of this year, and
no release has eventuated. Given libnatpmp is a new inclusion into our
repository, I think we should be using this newer source.
This also cleans up a few warnings we currently see in Windows depends builds:
```bash
Extracting libnatpmp...
/home/ubuntu/bitcoin/depends/sources/libnatpmp-20150609.tar.gz: OK
Preprocessing libnatpmp...
Configuring libnatpmp...
Building libnatpmp...
make[1]: Entering directory '/home/ubuntu/bitcoin/depends/work/build/x86_64-w64-mingw32/libnatpmp/20150609-13efa1beb87'
x86_64-w64-mingw32-gcc -Os -fPIC -Wall -DENABLE_STRNATPMPERR -c -o natpmp.o natpmp.c
x86_64-w64-mingw32-gcc -Os -fPIC -Wall -DENABLE_STRNATPMPERR -c -o getgateway.o getgateway.c
natpmp.c:42: warning: "EWOULDBLOCK" redefined
42 | #define EWOULDBLOCK WSAEWOULDBLOCK
|
In file included from natpmp.c:38:
/usr/share/mingw-w64/include/errno.h:166: note: this is the location of the previous definition
166 | #define EWOULDBLOCK 140
|
natpmp.c:43: warning: "ECONNREFUSED" redefined
43 | #define ECONNREFUSED WSAECONNREFUSED
|
In file included from natpmp.c:38:
/usr/share/mingw-w64/include/errno.h:110: note: this is the location of the previous definition
110 | #define ECONNREFUSED 107
|
natpmp.c:271:5: warning: ‘readnatpmpresponseorretry’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
271 | int readnatpmpresponseorretry(natpmp_t * p, natpmpresp_t * response)
| ^~~~~~~~~~~~~~~~~~~~~~~~~
ar crs libnatpmp.a natpmp.o getgateway.o
make[1]: Leaving directory '/home/ubuntu/bitcoin/depends/work/build/x86_64-w64-mingw32/libnatpmp/20150609-13efa1beb87'
Staging libnatpmp...
Postprocessing libnatpmp...
Caching libnatpmp...
```
ACKs for top commit:
hebasto:
ACK 7af25024e9
Tree-SHA512: 6939014ea986149a5bfdd42b516d563a65ae643516e234579d3f28e7c2f877b0270cc4305ae7c7cb131d6d946a6e0aedc84b4cc880a412612a878a333398b9d7
Implement the following commands from the I2P SAM protocol:
* HELLO: needed for all of the remaining ones
* DEST GENERATE: to generate our private key and destination
* NAMING LOOKUP: to convert .i2p addresses to destinations
* SESSION CREATE: needed for STREAM CONNECT and STREAM ACCEPT
* STREAM CONNECT: to make outgoing connections
* STREAM ACCEPT: to accept incoming connections
Extract `ReadBinaryFile()` and `WriteBinaryFile()` from `torcontrol.cpp`
to its own `readwritefile.{h,cpp}` files, so that it can be reused from
other modules.
fd6580e405 [refactor] txmempool: split epoch logic into class (Anthony Towns)
Pull request description:
Splits the epoch logic introduced in #17925 into a separate class.
Uses clang's thread safety annotations and encapsulates the data more strongly to reduce chances of bugs from API misuse.
ACKs for top commit:
jonatack:
ACK fd6580e405 using clang thread safety annotations looks like a very good idea, and the encapsulation this change adds should improve robustness (and possible unit test-ability) of the code. Verified that changing some of the locking duly provoked build-time warnings with Clang 9 on Debian and that small changes in the new `Epoch` class were covered by failing functional test assertions in `mempool_updatefromblock.py`, `mempool_resurrect.py`, and `mempool_reorg.py`
hebasto:
re-ACK fd6580e405, since my [previous](https://github.com/bitcoin/bitcoin/pull/18017#pullrequestreview-569619362) review:
Tree-SHA512: 7004623faa02b56639aa05ab7a078320a6d8d54ec62d8022876221e33f350f47df51ddff056c0de5be798f8eb39b5c03c2d3f035698555d70abc218e950f2f8c
5200929bfe depends: Include GUIX_ENVIRONMENT in id string (Carl Dong)
4c7d418588 depends: Improve id string robustness (Carl Dong)
b3bdff42b5 build: Proper quoting for var printing targets (Carl Dong)
Pull request description:
```
Environment variables and search paths can drastically effect the
operation of build tools.
Include these in our id string to mitigate against false cache hits.
```
Note to builders: This will invalidate all depends output caches in `BASE_CACHE`
ACKs for top commit:
laanwj:
re-ACK 5200929bfe
Tree-SHA512: e70c98da89cde90dc54bc3be89b925787cf94bbf246e27cc9345816b312073d78a02215448f731f21d8cf033c455234a2377ff1d66c00e1f3db69c9c9687d027
060a2a64d4 ci: remove boost thread installation (fanquake)
06e1d7d81d build: don't build or use Boost Thread (fanquake)
7097add83c refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef8 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d4
vasild:
ACK 060a2a64d4
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
615ba0eb96 test: add Sock unit tests (Vasil Dimov)
7bd21ce1ef style: rename hSocket to sock (Vasil Dimov)
04ae846904 net: use Sock in InterruptibleRecv() and Socks5() (Vasil Dimov)
ba9d73268f net: add RAII socket and use it instead of bare SOCKET (Vasil Dimov)
dec9b5e850 net: move CloseSocket() from netbase to util/sock (Vasil Dimov)
aa17a44551 net: move MillisToTimeval() from netbase to util/time (Vasil Dimov)
Pull request description:
Introduce a class to manage the lifetime of a socket - when the object
that contains the socket goes out of scope, the underlying socket will
be closed.
In addition, the new `Sock` class has a `Send()`, `Recv()` and `Wait()`
methods that can be overridden by unit tests to mock the socket
operations.
The `Wait()` method also hides the
`#ifdef USE_POLL poll() #else select() #endif` technique from higher
level code.
ACKs for top commit:
laanwj:
Re-ACK 615ba0eb96
jonatack:
re-ACK 615ba0eb96
Tree-SHA512: 3003e6bc0259295ca0265ccdeb1522ee25b4abe66d32e6ceaa51b55e0a999df7ddee765f86ce558a788c1953ee2009bfa149b09d494593f7d799c0d7d930bee8
Move `CloseSocket()` (and `NetworkErrorString()` which it uses) from
`netbase.{h,cpp}` to newly added `src/util/sock.{h,cpp}`.
This is necessary in order to use `CloseSocket()` from a newly
introduced Sock class (which will live in `src/util/sock.{h,cpp}`).
`sock.{h,cpp}` cannot depend on netbase because netbase will depend
on it.
22eb7930a6 tracing: add tracing framework (William Casarin)
933ab8a720 build: detect sys/sdt.h for eBPF tracing (William Casarin)
Pull request description:
Instead of writing ad-hoc logging everywhere (eg: #19509), we can take advantage of linux user static defined traces, aka. USDTs ( not the stablecoin 😅 )
The linux kernel can hook into these tracepoints at runtime, but otherwise they have little to no performance impact. Traces can pass data which can be printed externally via tools such as bpftrace. For example, here's one that prints incoming and outgoing network messages:
# Examples
## Network Messages
```
#!/usr/bin/env bpftrace
BEGIN
{
printf("bitcoin net msgs\n");
@start = nsecs;
}
usdt:./src/bitcoind:net:push_message
{
$ip = str(arg0);
$peer_id = (int64)arg1;
$command = str(arg2);
$data_len = arg3;
$data = buf(arg3,arg4);
$t = (nsecs - @start) / 100000;
printf("%zu outbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);
@outbound[$command]++;
}
usdt:./src/bitcoind:net:process_message
{
$ip = str(arg0);
$peer_id = (int64)arg1;
$command = str(arg2);
$data_len = arg3;
$data = buf(arg3,arg4);
$t = (nsecs - @start) / 100000;
printf("%zu inbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);
@inbound[$ip, $command]++;
}
```
$ sudo bpftrace netmsg.bt
output: https://jb55.com/s/b11312484b601fb3.txt
if you look at the bottom of the output you can see a histogram of all the messages grouped by message type and IP. nice!
## IBD Benchmarking
```
#!/usr/bin/env bpftrace
BEGIN
{
printf("IBD to 500,000 bench\n");
}
usdt:./src/bitcoind:CChainState:ConnectBlock
{
$height = (uint32)arg0;
if ($height == 1) {
printf("block 1 found, starting benchmark\n");
@start = nsecs;
}
if ($height >= 500000) {
@end = nsecs;
@duration = @end - @start;
exit();
}
}
END {
printf("duration %d ms\n", @duration / 1000000)
}
```
This one hooks into ConnectBlock and prints the IBD time to height 500,000 starting from the first call to ConnectBlock
Userspace static tracepoints give lots of flexibility without invasive logging code. It's also more flexible than ad-hoc logging code, allowing you to instrument many different aspects of the system without having to enable per-subsystem logging.
Other ideas: tracepoints for lock contention, threads, what else?
Let me know what ya'll think and if this is worth adding to bitcoin.
## TODO
- [ ] docs?
- [x] Integrate systemtap-std-dev/libsystemtap into build (provides the <sys/sdt.h> header)
- [x] ~dtrace macos support? (is this still a thing?)~ going to focus on linux for now
ACKs for top commit:
laanwj:
Tested ACK 22eb7930a6
0xB10C:
Tested ACK 22eb7930a6
Tree-SHA512: 69242242112b679c8a12a22b3bc50252c305894fb3055ae6e13d5f56221d858e58af1d698af55e23b69bdb7abedb5565ac6b45fa5144087b77a17acd04646a75
