bitcoin/src
merge-script ede388d03d
Merge bitcoin/bitcoin#30911: build: simplify by flattening the dependency graph
12fa9511b5 build: simplify dependency graph (Cory Fields)
c4e498300c build: avoid unnecessary dependencies on generated headers (Cory Fields)

Pull request description:

  These changes speed up my build (default config/options/targets) by roughly 10%. I suspect the difference may be more significant in other build configs.

  Before:
  > $ time cmake --build build -j24
  > real3m26.932s

  After:
  > $ time cmake --build build -j24
  > real3m7.556s

  Generally they allow for jobservers (either `make -jX` or `ninja`) to be better utilized. This can be verified using `top` while building and looking at the number of compiles running at any given time before/after these changes. Before, it's easy to observe periods of stalling when only one or two compiles are happening. After these changes, the compiler process count should mostly match the number of jobs given (`-jX`) until it falls off at the end.

  ---

  The first commit sets [DEPENDS_EXPLICIT_ONLY](https://cmake.org/cmake/help/latest/command/add_custom_command.html#command:add_custom_command) for commands which generate our test header files. Without this option, `test_bitcoin`'s generated headers won't be built until all of its other dependencies have been built. This introduces a significant stall in the build, though currently only Ninja benefits from this being set, and only CMake >= 3.27 understands it.

  Example from a generated `build.ninja`:

  Before:

  > \# Custom command for src/test/data/base58_encode_decode.json.h
  >
  > build src/test/data/base58_encode_decode.json.h | ${cmake_ninja_workdir}src/test/data/base58_encode_decode.json.h: CUSTOM_COMMAND /home/cory/dev/bitcoin/src/test/data/base58_encode_decode.json /home/cory/dev/bitcoin/cmake/script/GenerateHeaderFromJson.cmake || libcrc32c.a libcrc32c_sse42.a libleveldb.a libminisketch.a minisketch_clmul src/bitcoin_clientversion src/crypto/libbitcoin_crypto.a src/crypto/libbitcoin_crypto_avx2.a src/crypto/libbitcoin_crypto_sse41.a src/crypto/libbitcoin_crypto_x86_shani.a src/generate_build_info src/libbitcoin_cli.a src/libbitcoin_common.a src/libbitcoin_consensus.a src/libbitcoin_node.a src/secp256k1/src/libsecp256k1.a src/secp256k1/src/secp256k1_precomputed src/test/util/libtest_util.a src/univalue/libunivalue.a src/util/libbitcoin_util.a src/wallet/libbitcoin_wallet.a src/zmq/libbitcoin_zmq.a

  After:

  > \# Custom command for src/test/data/base58_encode_decode.json.h
  >
  > build src/test/data/base58_encode_decode.json.h | ${cmake_ninja_workdir}src/test/data/base58_encode_decode.json.h: CUSTOM_COMMAND /home/cory/dev/bitcoin/src/test/data/base58_encode_decode.json /home/cory/dev/bitcoin/cmake/script/GenerateHeaderFromJson.cmake

  ---

  The second commit is more significant. It sets [CMAKE_OPTIMIZE_DEPENDENCIES](https://cmake.org/cmake/help/latest/prop_tgt/OPTIMIZE_DEPENDENCIES.html) globally, which allows the objects of static libs to be built in parallel when one lib depends on the other. This can be set as a per-lib property, ~but I don't see any need for that as we don't currently have any edge-cases where this wouldn't be ok. If those should arise, we could always disable on a per-lib basis~.

  Edit: turns out this triggers an [upstream bug](https://gitlab.kitware.com/cmake/cmake/-/issues/24058), which I guess can be considered an edge-case until fixed in CMake. I've added 2 per-lib opt-outs as a result.

  Example:

  Before:

  > \# Link the static library src/libbitcoin_cli.a
  >
  > build src/libbitcoin_cli.a: CXX_STATIC_LIBRARY_LINKER__bitcoin_cli_RelWithDebInfo src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o src/CMakeFiles/bitcoin_cli.dir/rpc/client.cpp.o || src/univalue/libunivalue.a

  After:

  > \# Link the static library src/libbitcoin_cli.a
  >
  > build src/libbitcoin_cli.a: CXX_STATIC_LIBRARY_LINKER__bitcoin_cli_RelWithDebInfo src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o src/CMakeFiles/bitcoin_cli.dir/rpc/client.cpp.o
  >

ACKs for top commit:
  l0rinc:
    utACK 12fa9511b5
  hebasto:
    ACK 12fa9511b5.

Tree-SHA512: f85f507e70cdc06acd07542161d9f9b8edf9ba866f08c8ef17aaaed770fa11530a27521c4413456d863463a6e77d4d6983fa623a64e17bbd602c2bc70aacc112
2025-02-12 16:02:57 +01:00
..
bench Merge bitcoin/bitcoin#21590: Safegcd-based modular inverses in MuHash3072 2025-01-27 16:50:16 -05:00
common net: Use mockable steady clock in PCP implementation 2025-01-13 21:53:56 +01:00
compat netbase: refactor CreateSock() to accept sa_family_t 2024-03-01 13:13:07 -05:00
consensus Check leaves size maximum in MerkleComputation 2024-12-17 10:12:31 +07:00
crc32c Update crc32c subtree to latest upstream master 2024-02-27 18:28:19 +00:00
crypto Merge bitcoin/bitcoin#21590: Safegcd-based modular inverses in MuHash3072 2025-01-27 16:50:16 -05:00
index scripted-diff: rename block and undo functions for consistency 2025-01-09 15:17:02 +01:00
init scripted-diff: Replace strprintf(Untranslated) with Untranslated(strprintf) 2024-12-04 15:09:05 -04:00
interfaces interfaces: Add helper function for wallet on pruning 2025-01-05 17:28:19 +01:00
ipc Merge bitcoin/bitcoin#31384: mining: bugfix: Fix duplicate coinbase tx weight reservation 2025-02-10 08:26:01 -05:00
kernel kernel: Move block tree db open to block manager 2025-01-20 21:19:39 +01:00
leveldb Update leveldb subtree to latest upstream 2025-01-16 11:09:56 +00:00
logging scripted-diff: LogPrint -> LogDebug 2024-08-29 13:49:57 +02:00
minisketch Update minisketch subtree to latest master 2024-06-12 14:38:39 +01:00
node Merge bitcoin/bitcoin#31384: mining: bugfix: Fix duplicate coinbase tx weight reservation 2025-02-10 08:26:01 -05:00
policy miner: init: add -blockreservedweight startup option 2025-02-04 11:53:11 -05:00
primitives tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00
qt qt: Update the src/qt/locale/bitcoin_en.xlf translation source file 2025-02-06 10:30:30 +00:00
rpc Merge bitcoin/bitcoin#31384: mining: bugfix: Fix duplicate coinbase tx weight reservation 2025-02-10 08:26:01 -05:00
script miniscript: Make NodeRef a unique_ptr 2025-01-21 13:17:20 -05:00
secp256k1 Update secp256k1 subtree to v0.6.0 2024-11-04 14:59:46 -05:00
support util: Add missing types in make_secure_unique 2024-12-10 21:51:48 +01:00
test test: Remove stale gettime test 2025-02-12 12:16:20 +01:00
univalue test: clang-format -i src/univalue/test/unitester.cpp 2024-12-20 15:01:44 +01:00
util Merge bitcoin/bitcoin#30911: build: simplify by flattening the dependency graph 2025-02-12 16:02:57 +01:00
wallet Merge bitcoin/bitcoin#30909: wallet, assumeutxo: Don't Assume m_chain_tx_count, Improve wallet RPC errors 2025-01-31 15:45:14 -05:00
zmq cmake: Add FindZeroMQ module 2024-10-25 18:09:36 +01:00
.clang-format refactor: Print verbose serialize compiler error messages 2023-12-15 15:20:54 +01:00
.clang-tidy tidy: add clang-tidy modernize-use-starts-ends-with check 2024-09-14 20:33:32 +03:00
addrdb.cpp scripted-diff: Rename PACKAGE_* variables to CLIENT_* 2024-10-28 12:36:19 +00:00
addrdb.h Use serialization parameters for CAddress serialization 2023-09-05 10:13:25 +02:00
addresstype.cpp policy: Add OP_1 <0x4e73> as a standard output type 2024-07-30 14:06:58 -04:00
addresstype.h policy: Add OP_1 <0x4e73> as a standard output type 2024-07-30 14:06:58 -04:00
addrman.cpp addrman: cap the max_pct to not exceed the maximum number of addresses 2024-11-11 12:47:53 -03:00
addrman.h addrman: cap the max_pct to not exceed the maximum number of addresses 2024-11-11 12:47:53 -03:00
addrman_impl.h Merge bitcoin/bitcoin#30568: addrman: change internal id counting to int64_t 2024-09-20 12:55:22 -04:00
arith_uint256.cpp Add a fuzz test for Num3072 multiplication and inversion 2025-01-09 10:11:46 -05:00
arith_uint256.h doc: Clarify comments about endianness after #30526 2025-01-03 09:19:53 -05:00
attributes.h
banman.cpp refactor: Delay translation of _() literals 2025-01-14 19:21:37 +01:00
banman.h net_processing: remove Misbehavior score and increments 2024-05-30 08:35:18 -04:00
base58.cpp refactor: Avoid needless, unsafe c-style cast 2024-12-19 13:46:31 +01:00
base58.h
bech32.cpp Split out bech32 separator char to header 2024-10-19 18:49:53 +02:00
bech32.h Split out bech32 separator char to header 2024-10-19 18:49:53 +02:00
bip324.cpp net: merge V2Transport constructors, move key gen 2023-09-10 16:11:52 -04:00
bip324.h net: merge V2Transport constructors, move key gen 2023-09-10 16:11:52 -04:00
bitcoin-chainstate.cpp kernel: Move block tree db open to block manager 2025-01-20 21:19:39 +01:00
bitcoin-cli-res.rc scripted-diff: Rename PACKAGE_* variables to CLIENT_* 2024-10-28 12:36:19 +00:00
bitcoin-cli.cpp refactor: Use TranslateFn type consistently 2025-01-15 12:15:40 +01:00
bitcoin-tx-res.rc scripted-diff: Rename PACKAGE_* variables to CLIENT_* 2024-10-28 12:36:19 +00:00
bitcoin-tx.cpp refactor: Use TranslateFn type consistently 2025-01-15 12:15:40 +01:00
bitcoin-util-res.rc scripted-diff: Rename PACKAGE_* variables to CLIENT_* 2024-10-28 12:36:19 +00:00
bitcoin-util.cpp refactor: Use TranslateFn type consistently 2025-01-15 12:15:40 +01:00
bitcoin-wallet-res.rc scripted-diff: Rename PACKAGE_* variables to CLIENT_* 2024-10-28 12:36:19 +00:00
bitcoin-wallet.cpp refactor: Use TranslateFn type consistently 2025-01-15 12:15:40 +01:00
bitcoind-res.rc scripted-diff: Rename PACKAGE_* variables to CLIENT_* 2024-10-28 12:36:19 +00:00
bitcoind.cpp refactor: introduce a more general LockDirectories for init 2025-01-16 21:06:21 +00:00
blockencodings.cpp scripted-diff: LogPrint -> LogDebug 2024-08-29 13:49:57 +02:00
blockencodings.h tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00
blockfilter.cpp util: Move util/string.h functions to util namespace 2024-05-16 10:16:08 -05:00
blockfilter.h Remove unused includes from blockfilter.h 2023-08-17 18:28:15 +02:00
chain.cpp
chain.h doc: update NeedsRedownload() comment 2024-09-17 09:54:18 +02:00
chainparams.cpp test: Don't enforce BIP94 on regtest unless specified by arg 2024-10-28 11:38:38 -04:00
chainparams.h
chainparamsbase.cpp net, init: derive default onion port if a user specified a -port 2024-11-14 13:41:02 -05:00
chainparamsbase.h net, init: derive default onion port if a user specified a -port 2024-11-14 13:41:02 -05:00
chainparamsseeds.h seeds: Regenerate mainnet seeds 2024-08-27 07:00:27 +02:00
checkqueue.h Merge bitcoin/bitcoin#31313: refactor: Clamp worker threads in ChainstateManager constructor 2024-12-03 18:02:37 -05:00
clientversion.cpp refactor: Delay translation of _() literals 2025-01-14 19:21:37 +01:00
clientversion.h scripted-diff: Clarify "user agent" variable name 2024-10-28 12:35:49 +00:00
cluster_linearize.h doc: correct typos 2024-11-11 14:14:39 +00:00
CMakeLists.txt build: disable bitcoin-node if daemon is not built 2025-02-10 15:01:05 +01:00
coins.cpp Merge bitcoin/bitcoin#30906: refactor: prohibit direct flags access in CCoinsCacheEntry and remove invalid tests 2024-12-04 14:09:05 -05:00
coins.h coins, refactor: Remove direct GetFlags access 2024-12-02 13:52:34 +01:00
compressor.cpp Clean up things that include script/standard.h 2023-08-14 17:38:27 -04:00
compressor.h refactor: Rename CTransaction::nVersion to version 2024-06-07 13:55:23 -04:00
core_io.h rpc: add getdescriptoractivity 2024-11-26 20:47:08 -05:00
core_memusage.h
core_read.cpp ci: Update Clang in "tidy" job 2024-12-05 14:37:47 +00:00
core_write.cpp refactor: Rename CTransaction::nVersion to version 2024-06-07 13:55:23 -04:00
cuckoocache.h validation: Don't error if maxsigcachesize exceeds uint32::max 2024-07-04 22:35:29 +02:00
dbwrapper.cpp dbwrapper: Bump max file size to 32 MiB 2024-11-30 20:19:08 +01:00
dbwrapper.h dbwrapper: Bump max file size to 32 MiB 2024-11-30 20:19:08 +01:00
deploymentinfo.cpp
deploymentinfo.h
deploymentstatus.cpp
deploymentstatus.h
dummywallet.cpp wallet, test: Be able to always swap BDB endianness 2024-05-16 15:03:13 -04:00
external_signer.cpp refactor: Remove Span operator==, Use std::ranges::equal 2024-08-13 07:44:31 +02:00
external_signer.h
flatfile.cpp scripted-diff: LogPrint -> LogDebug 2024-08-29 13:49:57 +02:00
flatfile.h refactor: Add FlatFileSeq member variables in BlockManager 2024-07-24 09:39:35 +02:00
hash.cpp crypto, hash: replace custom rotl32 with std::rotl 2024-01-05 17:12:38 +01:00
hash.h Squashed 'src/secp256k1/' changes from 2f2ccc46954..0cdc758a563 2024-11-04 14:59:46 -05:00
headerssync.cpp scripted-diff: LogPrint -> LogDebug 2024-08-29 13:49:57 +02:00
headerssync.h tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00
httprpc.cpp args: Support -norpccookiefile for bitcoind and bitcoin-cli 2024-12-03 10:38:21 +01:00
httprpc.h
httpserver.cpp scripted-diff: Replace strprintf(Untranslated) with Untranslated(strprintf) 2024-12-04 15:09:05 -04:00
httpserver.h rpc: increase the defaults for -rpcthreads and -rpcworkqueue 2024-11-04 17:08:21 +01:00
i2p.cpp Merge bitcoin/bitcoin#29833: i2p: fix and improve logs 2024-06-26 15:28:26 -04:00
i2p.h i2p: log errors properly according to their severity 2024-06-12 16:19:50 -03:00
indirectmap.h
init.cpp Merge bitcoin/bitcoin#31384: mining: bugfix: Fix duplicate coinbase tx weight reservation 2025-02-10 08:26:01 -05:00
init.h refactor: introduce a more general LockDirectories for init 2025-01-16 21:06:21 +00:00
key.cpp refactor: remove un-tested early returns 2024-08-04 08:52:22 +02:00
key.h refactor: move SignSchnorr to KeyPair 2024-08-04 08:51:36 +02:00
key_io.cpp key: clear out secret data in DecodeExtKey 2024-10-27 15:38:54 +01:00
key_io.h Clean up things that include script/standard.h 2023-08-14 17:38:27 -04:00
logging.cpp log: Enforce trailing newline, Remove redundant m_started_new_line 2024-10-01 11:31:39 +02:00
logging.h log: Enforce trailing newline, Remove redundant m_started_new_line 2024-10-01 11:31:39 +02:00
mapport.cpp mapport: remove dead code in DispatchMapPort 2024-10-25 15:02:07 -04:00
mapport.h interfaces: remove now unused 'use_upnp' arg from 'mapPort' 2024-10-24 18:23:30 +02:00
memusage.h memusage: Add DynamicUsage for std::string 2024-11-04 18:46:40 +01:00
merkleblock.cpp [clang-tidy] Enable the misc-no-recursion check 2024-04-07 14:04:45 +01:00
merkleblock.h tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00
net.cpp Merge bitcoin/bitcoin#31022: test: Add mockable steady clock, tests for PCP and NATPMP implementations 2025-02-11 11:04:39 -08:00
net.h net: add LogIP() helper, use in net_processing 2024-11-26 13:22:55 +01:00
net_permissions.cpp util: move fees.h and error.h to common/messages.h 2024-05-16 10:16:08 -05:00
net_permissions.h Accept "in" and "out" flags to -whitelist to allow whitelisting manual connections 2024-02-28 10:05:56 -03:00
net_processing.cpp tracing: add misbehaving conn tracepoint 2025-02-04 10:25:22 +01:00
net_processing.h net: add GetOrphanTransactions() to PeerManager 2024-10-01 21:55:18 -04:00
net_types.cpp rpc: avoid copying into UniValue 2024-05-20 16:48:19 +00:00
net_types.h tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00
netaddress.cpp net: Add optional length checking to CService::SetSockAddr 2025-01-13 21:53:56 +01:00
netaddress.h net: Add optional length checking to CService::SetSockAddr 2025-01-13 21:53:56 +01:00
netbase.cpp scripted-diff: drop config/ subdir for bitcoin-config.h, rename to bitcoin-build-config.h 2024-10-10 12:22:12 +02:00
netbase.h net: add All() in ReachableNets 2024-09-10 11:20:40 -03:00
netgroup.cpp Merge bitcoin/bitcoin#27581: net: Continuous ASMap health check 2023-12-06 11:22:42 -05:00
netgroup.h net: Add continuous ASMap health check logging 2023-12-02 22:03:08 +01:00
netmessagemaker.h refactor: NetMsg::Make() without nVersion 2023-11-20 14:02:27 +01:00
noui.cpp node: Use log levels in noui_ThreadSafeMessageBox 2024-03-21 16:41:16 +01:00
noui.h
outputtype.cpp fix incorrect multisig redeem script size limit for segwit 2024-05-03 14:20:44 -03:00
outputtype.h fix incorrect multisig redeem script size limit for segwit 2024-05-03 14:20:44 -03:00
pow.cpp consensus: add DeriveTarget() to pow.h 2025-01-22 11:29:05 +01:00
pow.h consensus: add DeriveTarget() to pow.h 2025-01-22 11:29:05 +01:00
prevector.h prevector: avoid GCC bogus warnings in insert method 2024-09-11 17:41:26 +02:00
protocol.cpp scripted-diff: get rid of remaining "command" terminology in protocol.{h,cpp} 2024-10-26 23:44:15 +02:00
protocol.h scripted-diff: get rid of remaining "command" terminology in protocol.{h,cpp} 2024-10-26 23:44:15 +02:00
psbt.cpp util: add TransactionError includes and namespace declarations 2024-05-16 10:16:08 -05:00
psbt.h Merge bitcoin/bitcoin#30406: refactor: modernize-use-equals-default 2024-07-11 19:08:46 +01:00
pubkey.cpp refactor: Use immediate lambda to work around GCC bug 117966 2024-12-16 10:39:28 +01:00
pubkey.h refactor: Make XOnlyPubKey tolerate constexpr std::arrays 2024-08-28 19:09:51 +02:00
random.cpp fuzz: Abort when using global PRNG without re-seed 2024-12-16 15:23:56 +01:00
random.h refactor: Drop unused UCharCast 2025-01-14 19:01:53 +01:00
randomenv.cpp util: Remove RandAddSeedPerfmon 2024-10-21 23:24:17 +02:00
randomenv.h
rest.cpp Merge bitcoin/bitcoin#31583: rpc: add target to getmininginfo field and show next block info 2025-01-22 15:01:23 -05:00
rest.h
scheduler.cpp scripted-diff: Rename SingleThreadedSchedulerClient to SerialTaskRunner 2024-02-15 14:43:14 +01:00
scheduler.h kernel: Remove dependency on CScheduler 2024-02-16 17:12:52 +01:00
serialize.h Allow std::span in stream serialization 2024-12-19 14:41:21 +01:00
signet.cpp scripted-diff: LogPrint -> LogDebug 2024-08-29 13:49:57 +02:00
signet.h
span.h refactor: Simplify SpanPopBack 2024-12-19 13:46:52 +01:00
streams.cpp streams: add DataStream::GetMemoryUsage 2024-11-04 18:46:40 +01:00
streams.h refactor: Avoid passing span iterators when data pointers are expected 2024-12-19 14:39:55 +01:00
sync.cpp util: avoid using thread_local variable that has a destructor 2024-05-16 18:16:46 +02:00
sync.h tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00
threadsafety.h tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00
tinyformat.h refactor: Introduce struct to hold a runtime format string 2025-01-15 12:16:08 +01:00
torcontrol.cpp Merge bitcoin/bitcoin#31223: net, init: derive default onion port if a user specified a -port 2024-12-13 18:56:37 -05:00
torcontrol.h net, init: derive default onion port if a user specified a -port 2024-11-14 13:41:02 -05:00
txdb.cpp refactor: Rely on returned value of GetCoin instead of parameter 2024-09-18 20:03:47 +02:00
txdb.h kernel: Move default cache constants to caches 2025-01-15 15:44:55 +01:00
txmempool.cpp txmempool: fix typos in comments 2024-12-31 00:04:20 -03:00
txmempool.h txmempool: fix typos in comments 2024-12-31 00:04:20 -03:00
txorphanage.cpp [fuzz] TxOrphanage::SanityCheck accounting 2025-02-07 13:55:57 -05:00
txorphanage.h [fuzz] TxOrphanage::SanityCheck accounting 2025-02-07 13:55:57 -05:00
txrequest.cpp [refactor] make GetCandidatePeers take uint256 and in-out vector 2025-01-29 18:05:16 -05:00
txrequest.h [refactor] make GetCandidatePeers take uint256 and in-out vector 2025-01-29 18:05:16 -05:00
uint256.cpp scripted-diff: Rename SetHex to SetHexDeprecated 2024-07-24 09:15:34 +02:00
uint256.h doc: Clarify comments about endianness after #30526 2025-01-03 09:19:53 -05:00
undo.h Include version.h in fewer places 2023-11-16 11:36:22 +10:00
validation.cpp Merge bitcoin/bitcoin#30909: wallet, assumeutxo: Don't Assume m_chain_tx_count, Improve wallet RPC errors 2025-01-31 15:45:14 -05:00
validation.h Merge bitcoin/bitcoin#31483: kernel: Move kernel-related cache constants to kernel cache 2025-01-16 15:04:58 +00:00
validationinterface.cpp refactor: include the proper header rather than forward-declaring RemovalReasonToString 2024-10-08 15:25:47 +00:00
validationinterface.h [refactor] change ActiveTipChange to use CBlockIndex ref instead of ptr 2024-07-25 11:01:22 +01:00
versionbits.cpp
versionbits.h
walletinitinterface.h tidy: modernize-use-equals-default 2024-07-08 11:12:01 +02:00