fa3761d19d ci: Reduce tsan CPU and memory for faster scheduling (MarcoFalke)
aaaa07bc84 ci: Use credits for ARM task (MarcoFalke)
Pull request description:
After https://github.com/bitcoin/bitcoin/pull/27562 the task should finish in less than 10 minutes, so also using credits for it will be cheap and improve dev experience.
ACKs for top commit:
fanquake:
ACK fa3761d19d
Tree-SHA512: 98ba76eaa63bcbab674076bb9877b3a20d1b90e6d51e2fd7b97ae245414e984d734006b15ea60228a1f4f9bc72fbe2a1a8910fd05e7c52dd2a8b223bfaa25b50
fa199ee614 ci: Drop NO_WERROR=1 for clang-10 build (MarcoFalke)
fad2c200f4 build: Bump minimum Clang to clang-10 (MarcoFalke)
fad7cfee8d doc: Remove outdated CentOS comment (MarcoFalke)
Pull request description:
It doesn't make sense to support a minimum clang version that is difficult to install on all supported operating systems, which generally ship a later version:
* Ubuntu Focal 20.04: https://packages.ubuntu.com/focal/clang-10 and https://packages.ubuntu.com/focal/clang-12
* Debian Bullseye: https://packages.debian.org/bullseye/clang-13
* CentOS 8 Stream: All Clang versions from 11.0 to 15.0
Also, it allows to drop build code, which means it won't waste review when rolling over into cmake (`cmake/module/CheckStdFilesystem.cmake`).
ACKs for top commit:
hebasto:
ACK fa199ee614
fanquake:
ACK fa199ee614
Tree-SHA512: c1a0e8f191a6db866b8be3c9d254dc3f576fa021e2eaaeb68f3354554a8b38eaa90bbf9871ff92351b715e62a6b7b98cf94eba6dc53d7c951bddb6ad49ba7716
97844d9268 index: Enable reindex-chainstate with active indexes (Martin Zumsande)
60bec3c82d index: Use first block from locator instead of looking for fork point (Martin Zumsande)
Pull request description:
This makes two improvements to the index init phase:
**1) Prevent index corruption in case a reorg happens when the index was switched off**:
This is done by reading in the top block stored in the locator instead of looking for a fork point already in `BaseIndex::Init()`.
Before, we'd just go back to the fork point by calling `FindForkInGlobalIndex()`, which would have corrupted the coinstatsindex because its saved muhash needs to be reverted step by step by un-applying all blocks in between, which wasn't done before. This is now being done a bit later in `ThreadSync()`, which has existing logic to call the custom `Rewind()` method when going back along the chain to the forking point (thanks ryanofsky for pointing this out to me!).
**2) Allow using the `-reindex-chainstate` option without needing to disabling indexes**:
With `BaseIndex::Init()` not calling `FindForkInGlobalIndex()` anymore, we can allow `reindex-chainstate` with active indexes. `reindex-chainstate` deletes the chain and rebuilds it later in `ThreadImport`, so there is no chain available during `BaseIndex::Init()`, which would lead to problems (see #24789).
But now we'll only need the chain a bit later in `BaseIndex::ThreadSync`, which will wait for the reindex-chainstate in `ThreadImport` to finish and will continue syncing after that.
ACKs for top commit:
ryanofsky:
Code review ACK 97844d9268. Just simple rebase since last review
Tree-SHA512: e24973fc22e0b87a49026f4820aecb0a4e415f4d381bade9969dd31cf97afecfea0449dce7fcc797343b792199cc8287276d1f5ffa4433dcb54fb24a808db6fb
This is achieved by letting the index sync thread wait until
reindex-chainstate is finished.
This also disables the pruning check when reindexing the chainstate (which is
incompatible with prune mode) because there would be no chain at this point
in init.
The index sync code has logic to go back the chain to the forking point, while
also updating index-specific state, which is necessary to prevent
possible corruption of the coinstatsindex.
Also add a test for this (a reorg happens while the index is deactivated)
that would not pass before this change.
a09269a146 guix: document when certain guix patches can be dropped (fanquake)
Pull request description:
Additional notes for when patches can be dropped.
ACKs for top commit:
hebasto:
ACK a09269a146, I have reviewed the changes and they look OK.
jarolrod:
ACK a09269a146
Tree-SHA512: c1876b9a4e3cf73645d25c9077cef19a9b6b7fe2eda5dc9d82fd3ca3f9105453406c1b197e6635035b6ce19c9f255c070bebed5563f68913033d04627202155a
ddddf4957b ci: Run iwyu on all src files (MarcoFalke)
Pull request description:
This makes it easier to look at the CI output of a file without having to manually add it first to the list.
ACKs for top commit:
hebasto:
ACK ddddf4957b
Tree-SHA512: 342b52838ae45ea343731c30058cdd5595d5ea5601a1f396de4466ccdd63f7ab07b3a193df3669e4dca7cb535557dcc98f866b3cf986b98176b20ecead123868
We no-longer run any security/syymbol checks in the CI, and doubt we
will in future (if we do, it'll be via Guix, where this var would be
redundant in any case). The CI environment doesn't (exactly) match the
release build environment (and is semi-regularly changing), and the
binaries produced in the CI don't match how we build release binaries,
so there is no point trying to run these checks, especially as we add
more involved tests, i.e #26953.
This partially reverts commit 71383f2fad.
This should be fine, because if warnings are issues again in the future,
it can be disabled again, along with a list of the false warnings.
fad09b703f ci: Remove unused errtrace trap ERR (MarcoFalke)
Pull request description:
This was added in commit 069752b726, presumably at a time when the functional tests wouldn't capture stderr.
Now that all tests capture and print stderr on failure, it can be removed. Reference:
* Unit tests capture via `2>&1`:
d7700d3a26/src/Makefile.test.include (L421)
* Functional tests capture as well:
d7700d3a26/test/functional/test_framework/test_node.py (L356)
ACKs for top commit:
fanquake:
ACK fad09b703f
hebasto:
ACK fad09b703f, tested on Ubuntu 22.04: I can still see warnings from the sanitizers in both unit and functional tests.
Tree-SHA512: 1e786eee432a7a50eb9f78b06b2b157321cc16f91b613e3b476e9e51572592fe4bcf4dc15df176e5f019f24497ac68cf332d2037b55b57498c93f4e19613163c
These should only be relevant for a glibc that is built as part of a
Guix system, and should not be required for a glibc that is just being
built to compile our binaries against. A x86_64 linux bitcoind produced
with Guix using master vs this change has no difference. i.e:
```diff
@@ -20311,15 +20311,15 @@
This is experimental software.
The source code is available from %s.
Please contribute if you find %s useful. Visit %s for further information about the software.
The %s developers
The Bitcoin Core developers
<https://bitcoincore.org/>
Copyright (C) %i-%i
-v25.99.0-gda0bf1d07639b0490791bbd6aec71bbea8aa2aThe %s developer<https://github.com/bitcoin/bitcDistributed under the MIT software license, see the accompanyingThis is experimeThe source code is available froPlease contribute if you find %s useful. Visit %s for further information about Copyright (C) %ibool BCLog::Logger::StartLogging()
+v25.99.0-gd7700d3a26478d9b1648463c188648c7047b1cThe %s developer<https://github.com/bitcoin/bitcDistributed under the MIT software license, see the accompanyingThis is experimeThe source code is available froPlease contribute if you find %s useful. Visit %s for further information about Copyright (C) %ibool BCLog::Logger::StartLogging()
std::string BCLog::Logger::LogLevelToStr(BCLog::Level) const
std::string LogCategoryToStr(BCLog::LogFlags)
void BCLog::Logger::LogPrintStr(const string&, const string&, const string&, int, BCLog::LogFlags, BCLog::Level)
void BCLog::Logger::ShrinkDebugFile()
Failed to shrink debug log file: fseek(...) failed
logging.cpp
m_buffering
```
```diff
@@ -1505889,15 +1505889,15 @@
call aa3380 <malloc@plt+0xa4edb0>
mov (%rsp),%rdx
movdqa 0x465540(%rip),%xmm0
mov %rax,0x7a0559(%rip)
lea 0x7a0552(%rip),%rsi
lea 0x3957bb(%rip),%rdi
mov %rdx,0x7a0554(%rip)
- mov $0x3038,%edx
+ mov $0x3036,%edx
movups %xmm0,(%rax)
movdqa 0x465524(%rip),%xmm0
mov %dx,0x30(%rax)
mov 0x7a0529(%rip),%rdx
movups %xmm0,0x10(%rax)
movdqa 0x46551d(%rip),%xmm0
movups %xmm0,0x20(%rax)
```
```diff
@@ -37238,17 +37238,17 @@
0x00b73730 65202573 20646576 656c6f70 65727300 e %s developers.
0x00b73740 54686520 42697463 6f696e20 436f7265 The Bitcoin Core
0x00b73750 20646576 656c6f70 65727300 434f5059 developers.COPY
0x00b73760 494e4700 3c687474 70733a2f 2f626974 ING.<https://bit
0x00b73770 636f696e 636f7265 2e6f7267 2f3e0043 coincore.org/>.C
0x00b73780 6f707972 69676874 20284329 2025692d opyright (C) %i-
0x00b73790 25690053 61746f73 68690000 00000000 %i.Satoshi......
- 0x00b737a0 7632352e 39392e30 2d676461 30626631 v25.99.0-gda0bf1
- 0x00b737b0 64303736 33396230 34393037 39316262 d07639b0490791bb
- 0x00b737c0 64366165 63373162 62656138 61613261 d6aec71bbea8aa2a
+ 0x00b737a0 7632352e 39392e30 2d676437 37303064 v25.99.0-gd7700d
+ 0x00b737b0 33613236 34373864 39623136 34383436 3a26478d9b164846
+ 0x00b737c0 33633138 38363438 63373034 37623163 3c188648c7047b1c
0x00b737d0 54686520 25732064 6576656c 6f706572 The %s developer
0x00b737e0 3c687474 70733a2f 2f676974 6875622e <https://github.
0x00b737f0 636f6d2f 62697463 6f696e2f 62697463 com/bitcoin/bitc
0x00b73800 44697374 72696275 74656420 756e6465 Distributed unde
0x00b73810 72207468 65204d49 5420736f 66747761 r the MIT softwa
0x00b73820 7265206c 6963656e 73652c20 73656520 re license, see
0x00b73830 74686520 6163636f 6d70616e 79696e67 the accompanying
```
```diff
@@ -1,5 +1,5 @@
Hex dump of section '.gnu_debuglink':
0x00000000 62697463 6f696e64 2e646267 00000000 bitcoind.dbg....
- 0x00000010 6b6e8eda kn..
+ 0x00000010 345cb865 4\.e
```
1b1ffbd014 Build: Log when test -f fails in Makefile (TheCharlatan)
541012e621 Build: Use AM_V_GEN in Makefiles where appropriate (TheCharlatan)
Pull request description:
This PR triages some behavior around Makefile recipe echoing suppression.
When generating new files as part of the Makefile the recipe is sometimes suppressed with $(AM_V_GEN) and sometimes with `@`. We should prefer $(AM_V_GEN), since this also prints the lines in silent mode. This is arguably more in style with the current recipe echoing.
Before:
`Generated test/data/script_tests.json.h`
Now:
` GEN test/data/script_tests.json.h`
A side effect of this change is that the recipe for generating build.h is now echoed on each make run. Arguably this makes its generation more transparent.
Sometimes the error emitted by `test -f` is currently thrown without any logging. This makes it a bit harder to debug. Instead, print a helpful log message to point the developer in the right direction.
Alternatively this could have been implemented by just removing the recipe echo suppression (@), but the subsequent make output became too noisy.
ACKs for top commit:
fanquake:
ACK 1b1ffbd014
Tree-SHA512: e31869fab25e72802b692ce6735f9561912caea903c1577101b64c9cb115c98de01a59300e8ffe7b05b998345c1b64a79226231d7d1453236ac338c62dc9fbb3
e9dcac1ec7 add `lief` to `spelling.ignore-words` (brunoerg)
258f93000b test: fix spelling in `interface_usdt_utxocache` (brunoerg)
Pull request description:
Add `lief` to `spelling.ignore-words` since it's the name of a Python lib and fix spelling error in `interface_usdt_utxocache` (s/eariler/earlier)
ACKs for top commit:
fanquake:
ACK e9dcac1ec7
Tree-SHA512: f42cdbb74144c5d289d70bb9bac1179650bb32594678e15f4e18e4b2f68009999d60ff69494d4e68650d82fc1838e67515ed2f922ee84db98735ef906911ec40
0282b2126d walletdb: Remove unused CreateMockWalletDatabase (Andrew Chow)
Pull request description:
This has been superseded by the MockableDatabase. Remove to avoid confusion as to which type of mock database to use for testing.
I thought this was included in #26715, maybe it got lost in a rebase.
ACKs for top commit:
furszy:
utACK 0282b212
brunoerg:
crACK 0282b2126d
Tree-SHA512: 18445c4d8a4e2609ef7471c6d75297f43694b79e768f6c993a5addb1dc0e88bd12bac263c9d8e903d828ddf6bf50434f9e2f72048f4fc528e98fed8ee65123ca
so we erase all the records atomically or abort the entire
procedure.
and, at the same time, we can share the same db txn context
for the db cursor and the erase functionality.
extra note from the Db.cursor doc:
"If transaction protection is enabled, cursors must be
opened and closed within the context of a transaction"
thus why added a `CloseCursor` call before calling to
`TxnAbort/TxnCommit`.
If the batch ptr is not passed, the cursor will not use the db active
txn context which could lead to a deadlock if the code tries to modify
the db while it is traversing it.
E.g. the 'EraseRecords()' function.
33e2b82a4f wallet, bench: Remove unused database options from WalletBenchLoading (Andrew Chow)
80ace042d8 tests: Modify records directly in wallet ckey loading test (Andrew Chow)
b3bb17d5d0 tests: Update DuplicateMockDatabase for MockableDatabase (Andrew Chow)
f0eecf5e40 scripted-diff: Replace CreateMockWalletDB with CreateMockableWalletDB (Andrew Chow)
075962bc25 wallet, tests: Include wallet/test/util.h (Andrew Chow)
14aa4cb1e4 wallet: Move DummyDatabase to salvage (Andrew Chow)
f67a385556 wallet, tests: Replace usage of dummy db with mockable db (Andrew Chow)
33c6245ac1 Introduce MockableDatabase for wallet unit tests (Andrew Chow)
Pull request description:
For the wallet's unit tests, we currently use either `DummyDatabase` or memory-only versions of either BDB or SQLite. The tests that use `DummyDatabase` just need a `WalletDatabase` so that the `CWallet` can be constructed, while the tests using the memory-only databases just need a backing data store. There is also a `FailDatabase` that is similar to `DummyDatabase` except it fails be default or can have a configured return value. Having all of these different database types can make it difficult to write tests, particularly tests that work when either BDB or SQLite is disabled.
This PR unifies all of these different unit test database classes into a single `MockableDatabase`. Like `DummyDatabase`, most functions do nothing and just return true. Like `FailDatabase`, the return value of some functions can be configured on the fly to test various failure cases. Like the memory-only databases, records can actually be written to the `MockableDatabase` and be retrieved later, but all of this is still held in memory. Using `MockableDatabase` completely removes the need for having BDB or SQLite backed wallets in the unit tests for the tests that are not actually testing specific database behaviors.
Because `MockableDatabase`s can be created by each unit test, we can also control what records are stored in the database. Records can be added and removed externally from the typical database modification functions. This will give us greater ability to test failure conditions, particularly those involving corrupted records.
Possible alternative to #26644
ACKs for top commit:
furszy:
ACK 33e2b82
TheCharlatan:
ACK 33e2b82a4f
Tree-SHA512: c2b09eff9728d063d2d4aea28a0f0e64e40b76483e75dc53f08667df23bd25834d52656cd4eafb02e552db0b9e619cfdb1b1c65b26b5436ee2c971d804768bcc
3ece0ebf62 build, doc: Adjust comment after PR27254 (Hennadii Stepanov)
Pull request description:
This is a follow up for https://github.com/bitcoin/bitcoin/pull/27254.
ACKs for top commit:
TheCharlatan:
ACK 3ece0ebf62
Tree-SHA512: 36c627524304f0ea964383488acb9b16d0b553cc9cf3bce7a12aec65a7905b13e4582b7b7ec6d1efa32cd3c623969bc00f805ff31d9e6eec86a614e75796c8ff
fa01c3c59c ci: Remove CI_EXEC bloat (MarcoFalke)
fa8a428c92 move-only: Move almost all CI_EXEC code to 06_script_b.sh (MarcoFalke)
Pull request description:
`CI_EXEC` has many issues:
* It is roughly equivalent to `bash -c "$*"`, meaning that the full command will be treated as a single string, ignoring tokens.
* It must be put in front of (almost) every command, making it easy to forget, hard to debug the resulting failure, and the code verbose.
Fix all issues by removing it almost completely.
ACKs for top commit:
TheCharlatan:
ACK fa01c3c59c
Tree-SHA512: 4a65d61f5c35ca945d31f270dba3e96305fd83333a7713f0452c67f02a78e1901113e9f18d21e1dc016403c0033eb32038a9308d0a0ded7ee6b970d18381a1c2
9ae854da19 depends: no-longer nuke libc++abi.so* in native_clang package (fanquake)
Pull request description:
We weren't copying it over in any case.
ACKs for top commit:
hebasto:
ACK 9ae854da19
theuni:
Sure. utACK no-op 9ae854da19.
Tree-SHA512: 3cc7f18f27c1b498f930bc1a09283aa04ba673d3c1a5220d8462213f0a06b74bc34989f23404402850de518cba35ddab900a54f7f0fac112fc86664e4155f8cb
67aacc73ea build: cleanup comments after adding yet another libtool hack (Cory Fields)
283d95516a build: Fix shared lib linking for darwin with lld (Cory Fields)
Pull request description:
Solves one of the last remaining blockers for #21778. Fixes lld linking shared libs for macos via libtool.
lld fails one of libtool's earliest checks [because it happens to output a warning that contains a specific string](https://git.savannah.gnu.org/cgit/libtool.git/tree/m4/libtool.m4#n999):
> # If there is a non-empty error log, and "single_module"
> # appears in it, assume the flag caused a linker warning
And here is the test being run:
> x86_64-apple-darwin-ld: warning: Option `-single_module' is deprecated in ld64:
> x86_64-apple-darwin-ld: warning: Unnecessary option: this is already the default
Because the warning is printed the test fails. So libtool falls back to a very primitive and broken link-line for shared libs.
Arguably this should be worked-around in upstream lld by changing the warning string, as otherwise every libtool project will fail to link with it.
Like many other libtool hacks, the solution is to simply disable the check and hard-code the answer we know to be correct.
ACKs for top commit:
hebasto:
re-ACK 67aacc73ea
Tree-SHA512: 792e4d208a3a4921edb5f267f43ecd052b5b650df0db5cb2788ee1e4f3c4087413f354b22e407ff5fa2f99a22a16154ec6826d14c6654a57c00aae3b3e744bca
libtool gets a false-positive from the warning produced by lld -single_module
because it is already the default and unneeded.
Skip the check unconditionally for Darwin linkers.
