This is required for the next commit. Also, drop CI_RETRY_EXE before
"dnf install", because it requires getopt, which will only be installed
later on via util-linux
fa199ee614 ci: Drop NO_WERROR=1 for clang-10 build (MarcoFalke)
fad2c200f4 build: Bump minimum Clang to clang-10 (MarcoFalke)
fad7cfee8d doc: Remove outdated CentOS comment (MarcoFalke)
Pull request description:
It doesn't make sense to support a minimum clang version that is difficult to install on all supported operating systems, which generally ship a later version:
* Ubuntu Focal 20.04: https://packages.ubuntu.com/focal/clang-10 and https://packages.ubuntu.com/focal/clang-12
* Debian Bullseye: https://packages.debian.org/bullseye/clang-13
* CentOS 8 Stream: All Clang versions from 11.0 to 15.0
Also, it allows to drop build code, which means it won't waste review when rolling over into cmake (`cmake/module/CheckStdFilesystem.cmake`).
ACKs for top commit:
hebasto:
ACK fa199ee614
fanquake:
ACK fa199ee614
Tree-SHA512: c1a0e8f191a6db866b8be3c9d254dc3f576fa021e2eaaeb68f3354554a8b38eaa90bbf9871ff92351b715e62a6b7b98cf94eba6dc53d7c951bddb6ad49ba7716
ddddf4957b ci: Run iwyu on all src files (MarcoFalke)
Pull request description:
This makes it easier to look at the CI output of a file without having to manually add it first to the list.
ACKs for top commit:
hebasto:
ACK ddddf4957b
Tree-SHA512: 342b52838ae45ea343731c30058cdd5595d5ea5601a1f396de4466ccdd63f7ab07b3a193df3669e4dca7cb535557dcc98f866b3cf986b98176b20ecead123868
We no-longer run any security/syymbol checks in the CI, and doubt we
will in future (if we do, it'll be via Guix, where this var would be
redundant in any case). The CI environment doesn't (exactly) match the
release build environment (and is semi-regularly changing), and the
binaries produced in the CI don't match how we build release binaries,
so there is no point trying to run these checks, especially as we add
more involved tests, i.e #26953.
This partially reverts commit 71383f2fad.
This should be fine, because if warnings are issues again in the future,
it can be disabled again, along with a list of the false warnings.
fad09b703f ci: Remove unused errtrace trap ERR (MarcoFalke)
Pull request description:
This was added in commit 069752b726, presumably at a time when the functional tests wouldn't capture stderr.
Now that all tests capture and print stderr on failure, it can be removed. Reference:
* Unit tests capture via `2>&1`:
d7700d3a26/src/Makefile.test.include (L421)
* Functional tests capture as well:
d7700d3a26/test/functional/test_framework/test_node.py (L356)
ACKs for top commit:
fanquake:
ACK fad09b703f
hebasto:
ACK fad09b703f, tested on Ubuntu 22.04: I can still see warnings from the sanitizers in both unit and functional tests.
Tree-SHA512: 1e786eee432a7a50eb9f78b06b2b157321cc16f91b613e3b476e9e51572592fe4bcf4dc15df176e5f019f24497ac68cf332d2037b55b57498c93f4e19613163c
fa01c3c59c ci: Remove CI_EXEC bloat (MarcoFalke)
fa8a428c92 move-only: Move almost all CI_EXEC code to 06_script_b.sh (MarcoFalke)
Pull request description:
`CI_EXEC` has many issues:
* It is roughly equivalent to `bash -c "$*"`, meaning that the full command will be treated as a single string, ignoring tokens.
* It must be put in front of (almost) every command, making it easy to forget, hard to debug the resulting failure, and the code verbose.
Fix all issues by removing it almost completely.
ACKs for top commit:
TheCharlatan:
ACK fa01c3c59c
Tree-SHA512: 4a65d61f5c35ca945d31f270dba3e96305fd83333a7713f0452c67f02a78e1901113e9f18d21e1dc016403c0033eb32038a9308d0a0ded7ee6b970d18381a1c2
[WARN] The commit is obviously broken and will not run the CI system. In
the rare case this is hit in a git bisect, just skip the commit.
The goal here was to make it trivial to review with the git option:
--color-moved=dimmed-zebra
It is required to move everything into one file because "exit 0" will
otherwise stop working as intended when the containing bash script is no
longer executed with "source ...".
If there is desire to split up 06_script_b.sh into logical chunks in the
future, it will also be easier after the following commit.
Instead of enumerating each passed env var, just pass all. This avoids
the risk of missing to enumerate one. Also, it is less code.
The risk could be that an env var causes non-deterministic behavior, but
this can be fixed by explicitly excluding it once the issue is known.
Values with newlines can not be stored in the file and parsed by
docker/podman, so they are excluded.
fac395e5eb ci: Bump ci/lint/Dockerfile (MarcoFalke)
fa6eb65167 test: Use python3.8 pow() (MarcoFalke)
88881cf7ac Bump python minimum version to 3.8 (MarcoFalke)
Pull request description:
There is no pressing reason to drop support for 3.7, however there are several maintenance issues:
* There is no supported operating system that ships 3.7 by default. (debian:buster is EOL and unmaintained to the extent that it doesn't run in the CI environment. See https://github.com/bitcoin/bitcoin/pull/27340#issuecomment-1484988445)
* Compiling python 3.7 from source is also unsupported on at least macos, according to https://github.com/bitcoin/bitcoin/pull/24017#issuecomment-1107820790
* Recent versions of lief require 3.8, see https://github.com/bitcoin/bitcoin/pull/27507#issuecomment-1517561645
Fix all maintenance issues by bumping the minimum.
ACKs for top commit:
RandyMcMillan:
ACK fac395e
fjahr:
ACK fac395e5eb
fanquake:
ACK fac395e5eb
Tree-SHA512: c198decdbbe29d186d73ea3f6549d8a38479383495d14a965a2f9211ce39637b43f13a4c2a5d3bf56e2d468be4bbe49b4ee8e8e19ec69936ff43ddf2b714c712
be55f545d5 move-only: Extract common/args and common/config.cpp from util/system (TheCharlatan)
Pull request description:
This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/24303https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". It is part of a series of patches splitting up the `util/system` files. Its preceding pull request is https://github.com/bitcoin/bitcoin/pull/27254.
The pull request contains an extraction of ArgsManager related functions from util/system into their own common/ file.
The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager. The ArgsManager belongs into the common library, since the kernel library should not depend on it. See [doc/design/libraries.md](https://github.com/bitcoin/bitcoin/blob/master/doc/design/libraries.md) for more information on this rationale.
ACKs for top commit:
MarcoFalke:
re-ACK be55f545d5🚲
ryanofsky:
Code review ACK be55f545d5. Just small cleanups since the last review.
hebasto:
ACK be55f545d5, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 90eb03334af0155b823030b4f2ecf286d35058d700ee2ddbbaa445be19e31eb0fe982656f35bd14ecee3ad2c3d0db3746855cb8f3777eff7253713e42873e111
This is an extraction of ArgsManager related functions from util/system
into their own common file.
Config file related functions are moved to common/config.cpp.
The background of this commit is an ongoing effort to decouple the
libbitcoinkernel library from the ArgsManager. The ArgsManager belongs
into the common library, since the kernel library should not depend on
it. See doc/design/libraries.md for more information on this rationale.
fa4a46de0b ci: Bump nowallet_libbitcoinkernel task to ubuntu:focal (MarcoFalke)
fabc7d90a9 ci: Use credits in more tasks (MarcoFalke)
facae3b149 ci: Use Cirrus CI dockerfile env (MarcoFalke)
Pull request description:
Currently the CI env has many intermittent issues:
* The Ubuntu package servers are frequently down
* Occasionally other stuff is down, such as dnf, pip, or the android sdk
* Installing packages is slower than downloading them, at least on Cirrus, which has a fast download speed
Fix all issues by using the Cirrus CI dockerfile env.
ACKs for top commit:
josibake:
code review ACK fa4a46de0b
Tree-SHA512: fea5663f7b6dc1c4ea9f87188026ec542b9269bac8ee3398cd58d4df6c86a0af9d275f1876e03f92fb1f6166ec49b817d9e588e6fe1ed54b77592502c2eccd9d
This fixes some cases, i.e under --no-install-recommends, where
libclang-rt-dev wouldn't be installed, and configuring would then fail.
Followup to #27444.
This should avoid errors when running it twice. For example, network
errors on the second invocation of 'apt update'; or unguarded
modifications such as APPEND_APT_SOURCES_LIST, which will append the
same string repeatedly.
The base install may be run twice in Cirrus CI with dockerfiles, or
locally when running twice with DANGER_RUN_CI_ON_HOST specified.
ed4a8339b8 ci: fix git dubious permissions error (josibake)
Pull request description:
fixes https://github.com/bitcoin/bitcoin/pull/27376#issuecomment-1496449588
this appears to be caused by a more recent version of git being sensitive to mismatched permissions on directories. we didn't notice this before because we were using two separate user accounts to fix up dir permissions in the container , but the second account was removed in #27376
there might be a more elegant way to do this, but this does the trick and seems to be the way others are fixing this issue around the internets.
ACKs for top commit:
RandyMcMillan:
ACK ed4a833
hebasto:
re-ACK ed4a8339b8
Tree-SHA512: dad467deca101a24f3ed34b3e26a1db5099a5bd5c3e9c9a22771c59848f7d7e7843c7386348e6fdf86d5a556e4706e5e20005d7a6637193e1c8aef7a5ff7fb19
b5ef1419ec ci: cleanup of CI_EXEC & CI_EXEC_ROOT (refs #27321) (Vasil Stoyanov)
Pull request description:
Basically it removes the above-mentioned env-vars as per MarcoFalke's instructions. The only deviation from the plan laid out there was that I double-quoted the last instance of $ANDROID_HOME for the sake of consistency and future-proofing and the rest of the non-quoted vars due to lint failing the build.
Fixes#27321.
ACKs for top commit:
josibake:
ACK b5ef1419ec
hernanmarino:
untested ACK b5ef1419ec. LGTM
Tree-SHA512: a79776bf64a2fa8b38195cc84445e171fd689f156aac5a1e5d39040300567eb9f4c2ebd00fbf3fa0e55b68793f8f752d94f7d817f6097ed9dd3a8ea57651b981
This is a minimal extraction of a single function, but also the only use
of std::exception in util/system.
The background of this commit is an ongoing effort to decouple the
libbitcoinkernel library from the ArgsManager defined in system.h.
Moving the function out of system.h allows including it from a separate
source file without including the ArgsManager definitions from system.h.
fa1b4e5c32 Use steady clock in FlushStateToDisk (MarcoFalke)
1111e2f8b4 Use steady clock in SeedStrengthen and FindBestImplementation (MarcoFalke)
Pull request description:
There may be a theoretical deadlock for the duration of the offset when the system clock is adjusted into a past time while executing `SeedStrengthen`.
Fix this by using steady clock.
Do the same in `FindBestImplementation`, which shouldn't be affected, because it discards outlier measurements. However, doing the same there for consistency seems fine.
Do the same in `FlushStateToDisk`, which should make the flushes more steady, if the system clock is adjusted by a large offset.
ACKs for top commit:
john-moffett:
ACK fa1b4e5c32
willcl-ark:
ACK fa1b4e5c3
Tree-SHA512: cc625e796b186accd53222bd64eb57d0512bc7e588312d254349b542bbc5e5daac348ff2b3b3f7dc5ae0bbbae2ec11fdbf3022cf2164211633765a4b0108e83e
802cc1ef53 Deduplicate bitcoind and bitcoin-qt init code (Ryan Ofsky)
d172b5c671 Add InitError(error, details) overload (Ryan Ofsky)
3db2874bd7 Extend bilingual_str support for tinyformat (Ryan Ofsky)
c361df90b9 scripted-diff: Remove double newlines after some init errors (Ryan Ofsky)
Pull request description:
Add common InitConfig function to deduplicate bitcoind and bitcoin-qt code reading config files and creating the datadir.
Noticed the duplicate code while reviewing #27073 and want to remove it because difference in bitcoin-qt and bitcoind behavior make it hard to evaluate changes like #27073
There are a few minor changes in behavior:
- In bitcoin-qt, when there is a problem reading the configuration file, the GUI error text has changed from "Error: Cannot parse configuration file:" to "Error reading configuration file:" to be consistent with bitcoind.
- In bitcoind, when there is a problem reading the settings.json file, the error text has changed from "Failed loading settings file" to "Settings file could not be read" to be consistent with bitcoin-qt.
- In bitcoind, when there is a problem writing the settings.json file, the error text has changed from "Failed saving settings file" to "Settings file could not be written" to be consistent with bitcoin-qt.
- In bitcoin-qt, if there datadir is not accessible (e.g. no permission to read), there is an normal error dialog showing "Error: filesystem error: status: Permission denied [.../settings.json]", instead of an uncaught exception.
ACKs for top commit:
Sjors:
Light review ACK 802cc1ef53
TheCharlatan:
ACK 802cc1ef53
achow101:
ACK 802cc1ef53
Tree-SHA512: 9c78d277e9ed595fa8ce286b97d2806e1ec06ddbbe7bd3434bd9dd7b456faf8d989f71231e97311f36edb9caaec645a50c730bd7514b8e0fe6e6f7741b13d981
Add common InitConfig function to deduplicate bitcoind and bitcoin-qt code
reading config files and creating the datadir.
There are a few minor changes in behavior:
- In bitcoin-qt, when there is a problem reading the configuration file, the
GUI error text has changed from "Error: Cannot parse configuration file:" to
"Error reading configuration file:" to be consistent with bitcoind.
- In bitcoind, when there is a problem reading the settings.json file, the
error text has changed from "Failed loading settings file" to "Settings
file could not be read" to be consistent with bitcoin-qt.
- In bitcoind, when there is a problem writing the settings.json file, the
error text has changed from "Failed saving settings file" to "Settings file
could not be written" to be consistent with bitcoin-qt.
- In bitcoin-qt, if there datadir is not accessible (e.g. no permission to read),
there is an normal error dialog showing "Error: filesystem error: status:
Permission denied [.../settings.json]", instead of an uncaught exception
4275195606 De-duplicate add_coin methods to a test util helper (Jon Atack)
9d92c3d7f4 Create InsecureRandMoneyAmount() test util helper (Jon Atack)
81f5ade2a3 Move random test util code from setup_common to random (Jon Atack)
Pull request description:
- Move random test utilities from `setup_common` to a new `random` file, as many tests don't use this code.
- Create a helper to generate semi-random CAmounts up to `MONEY_RANGE` rather than only uint32, and use the helper in the unit tests.
- De-duplicate a shared `add_coin` method by extracting it to a `coins` test utility.
ACKs for top commit:
pinheadmz:
ACK 4275195606
achow101:
ACK 4275195606
john-moffett:
ACK 4275195606
Tree-SHA512: 3ed974251149c7417f935ef2f8865aa0dcc33b281b47522b0f96f1979dff94bb8527957f098fe4d210f40d715c00f29512f2ffe189097102229023b7284a3a27
fa8e92c022 doc: Update ci docs (721217.xyz)
5fffff54e9 ci: Cache stuff in volumes, not host folders (MarcoFalke)
Pull request description:
Storing cached stuff in host system folders may lead to unexpected issues when the ci-built stuff is used for a non-ci build or a ci task leaks into another ci task.
ACKs for top commit:
john-moffett:
ACK fa8e92c022
Tree-SHA512: 8b0c9019452fbe507a272c1037c3dce3c178c21f85ab1096ed3372ad9d4b3c7aa27d89e5bf80c9a6260ea652e0268be0cbe61d6a4fcb3add569fa38076d32287
935acdcc79 refactor: modernize the implementation of uint256.* (pasta)
Pull request description:
- Constructors of uint256 to utilize Span instead of requiring a std::vector
- converts m_data into a std::array
- Prefers using `WIDTH` instead of `sizeof(m_data)`
- make all the things constexpr
- replace C style functions with c++ equivalents
- memset -> std::fill
This may also be replaced by std::memset, but I think that std::fill is more idiomatic of modern c++ and readable.
- memcpy -> std::copy
Note: In practice, implementations of std::copy avoid multiple assignments and use bulk copy functions such as std::memmove if the value type is TriviallyCopyable and the iterator types satisfy LegacyContiguousIterator. (https://en.cppreference.com/w/cpp/algorithm/copy)
This could also likely be replaced by std::memcpy, but as said above, I believe the using std::copy is the more c++ way to do anything and is almost guaranteed to compile to the same asm
- memcmp -> std::memcmp
ACKs for top commit:
achow101:
ACK 935acdcc79
hebasto:
Approach ACK 935acdcc79.
aureleoules:
reACK 935acdcc79
john-moffett:
ACK 935acdcc79
stickies-v:
Approach ACK 935acdcc7
Tree-SHA512: 4f1ba54ff2198eea0e505d41e73d552c84c60f6878d5c85a94a8ab57f39afc94ef8d79258e7afd01fa84ec2a99f4404bb877eecd671f65e1ee9273f3129fc650
fa6986a66b ci: Print iwyu patch in git diff format (MarcoFalke)
Pull request description:
Seems more dev friendly to also have a patch to copy-paste
ACKs for top commit:
hebasto:
ACK fa6986a66b, tested on Ubuntu 22.04 locally.
fanquake:
ACK fa6986a66b - did not test but example CI output looks ok.
stickies-v:
utACK fa6986a66b
Tree-SHA512: 7cfd8584bf12e03c28af23f4712c6bcafd648d87ddb92788b9cd35455b2db49f4bd4aef8ad4711f75c7f11ad2bb2492c2eb6044007086c20e36016575c060603
fa486de212 ci: Cache package manager install step (MarcoFalke)
Pull request description:
Use the local podman or docker image cache to skip the slow `apt` step
ACKs for top commit:
jamesob:
ACK fa486de212 ([`jamesob/ackr/26976.1.MarcoFalke.ci_cache_package_manager`](https://github.com/jamesob/bitcoin/tree/ackr/26976.1.MarcoFalke.ci_cache_package_manager))
Tree-SHA512: 3495346c6c862b63296d2691cc492bf52a0a99ee7fae798887c792609904546013eba788045cd508a5f669f2c52e3479c122c18a5275c87af38237a1b5c9da17
Don't enable `-Werror` (in the CI) for compilers at least older than
our current release compiler (GCC 10). It provides little-to-no value,
other than turning compiler bugs & false positives into build failures,
and we aren't going to mutate perfectly fine/working code, for the sake
of avoid a warning that shouldn't even exist.
I also do not see the point of playing whack-a-mole and turning off various
warnings/trying to further work around the broken compiler, just to
acheive warningless builds for the sake of warningless builds.
One anecdote from "How SQLite Is Tested":
> Static analysis has found a few bugs in SQLite, but those are the
> exceptions. More bugs have been introduced into SQLite while trying
> to get it to compile without warnings than have been found by static
> analysis.
https://www.sqlite.org/testing.html.
6d58117a31 build: Build minisketch test in `make check`, not in `make` (Hennadii Stepanov)
Pull request description:
On master (d1e42659bb):
```
$ ./autogen.sh && ./configure --without-gui --disable-wallet && make clean
$ make 2>&1 | grep LD | grep -v .la
CXXLD bitcoind
CXXLD bitcoin-cli
CXXLD bitcoin-tx
CXXLD bitcoin-util
CXXLD test/test_bitcoin
CXXLD bench/bench_bitcoin
CXXLD minisketch/test
CXXLD test/fuzz/fuzz
CXXLD univalue/test/object
CXXLD univalue/test/unitester
$ make check 2>&1 | grep LD
CCLD exhaustive_tests
CCLD tests
```
With this PR:
```
$ ./autogen.sh && ./configure --without-gui --disable-wallet && make clean
$ make 2>&1 | grep LD | grep -v .la
CXXLD bitcoind
CXXLD bitcoin-cli
CXXLD bitcoin-tx
CXXLD bitcoin-util
CXXLD test/test_bitcoin
CXXLD bench/bench_bitcoin
CXXLD test/fuzz/fuzz
CXXLD univalue/test/object
CXXLD univalue/test/unitester
$ make check 2>&1 | grep LD
CXXLD minisketch/test
CCLD exhaustive_tests
CCLD tests
```
In fact, this PR restores behavior that was before bitcoin/bitcoin#22646, and that behavior looks more optimal.
As an outcome, the `contrib/guix/libexec/build.sh` does not spend resources to build binaries which are not a part of the release package.
ACKs for top commit:
TheCharlatan:
ACK 6d58117a31
Tree-SHA512: 4957c8f88a01aca005813bf4c1c26f433756bf68ea0c958481c638ead229fa8e23ecae3a8ac31ea555876ba6f2cc10ecd91caf2e2f664de5cb529ec05fb38fa7
faba08b5b4 refactor: Remove stray cs_main redundant declaration (MarcoFalke)
fa02591edf doc: Export threadsafety.h from sync.h (MarcoFalke)
Pull request description:
Looks like this was forgotten when introducing kernel/cs_main ?
Also, there is a commit to export threadsafety.h from sync.h.
ACKs for top commit:
hebasto:
ACK faba08b5b4
Tree-SHA512: 0aa58e7693b6fcd504f9da7339f8baa463a6407f67b27f68002db705f4642321ac3765f16c3d906c925ee24085591b79160a62fa5f4aaf6f2e5dcc788411800d
a3a2bd9e8a ci: Drop no longer needed package-specific flags (Hennadii Stepanov)
071eef1e97 build: Propagate user-defined flags to host packages (Hennadii Stepanov)
Pull request description:
On master (4f8b1f8759) `{CPP,C,CXX,LD}FLAGS` that are specified in the command line are not propagated to packages:
```
$ make --no-print-directory -C depends print-libevent_cxxflags CXXFLAGS=-some-fancy-flag
libevent_cxxflags=-pipe -O2
```
This PR:
- propagates `{CPP,C,CXX,LD}FLAGS` to host packages:
```
$ make --no-print-directory -C depends print-libevent_cxxflags CXXFLAGS=-some-fancy-flag
libevent_cxxflags= -some-fancy-flag
```
- does not propagate `{CPP,C,CXX,LD}FLAGS` to native packages:
```
$ make --no-print-directory -C depends print-native_b2_cxxflags CXXFLAGS=-some-fancy-flag
native_b2_cxxflags=
```
- actually addresses the https://github.com/bitcoin/bitcoin/pull/23551#issuecomment-973896518
ACKs for top commit:
TheCharlatan:
Code review ACK a3a2bd9e8a
Tree-SHA512: 243d6b1b0e9c5de46debc36de62a77b6b4d6f638940fd530040c219956ec624e321b0c25290fed164e3a8c88befa7b97b20f765d7b9a428c269b3720f21da099
b68e5a7fef lint: specify the right commit range when running locally (James O'Beirne)
dff7ed5732 test: add an easy way to run linters locally (James O'Beirne)
Pull request description:
Adds a Dockerfile configuration ~~(originally written mostly by fanquake)~~ that allows straightforward running of linters with compatible versions locally. This removes a ton of annoyance when trying to appease CI, because many of the linter versions are quite old and difficult to maintain locally.
I realize that people may not be thrilled to add more ancillary tooling to the repo, but I think this makes a lot of sense given the linter versions listed in this container configuration are dictated by this repo (within the CI configuration), so having these things live in two separate places is a recipe for version mismatches.
Eventually we can likely just use this container on CI directly to avoid any chance of inconsistencies between local dev experience and CI.
ACKs for top commit:
aureleoules:
ACK b68e5a7fef
stickies-v:
ACK b68e5a7fe
john-moffett:
ACK b68e5a7fef
Tree-SHA512: 7ef7a5dae023d81fdb6296d5d92dfa074ee321c7993e607c9f014d0f21c91558611aa00fc3ce1edc7b5e68371aea0d27fa1931291a79bb867a6c783bb536775f
Adds a Dockerfile configuration
that allows straightforward running of linters with compatible versions
locally. This removes a ton of annoyance when trying to appease CI,
because many of the linter versions are quite old and difficult to
maintain locally.
I realize that people may not be thrilled to more ancillary tooling to
the repo, but I think this makes a lot of sense given the linter
versions listed in this container configuration are dictated by this
repo (within the CI configuration), so having these things live in
two separate places is a recipe for version mismatches.
Eventually we can likely just use this container on CI directly to avoid
any chance of inconsistencies between local dev experience and CI.
fa87d71872 ci: Add missing lint dependency (MarcoFalke)
Pull request description:
Also, document each dependency.
Adding `gpg` avoids errors when running a release or dev branch in the CI:
01ec5308bf/ci/lint/06_script.sh (L30-L42)
```
bash: line 1: gpg: command not found
```
https://cirrus-ci.com/task/4582854860996608?logs=lint#L185
ACKs for top commit:
hebasto:
ACK fa87d71872
Tree-SHA512: 869a3d2feab764b2c8d47d481359680a1d2c54a33b13ca26c5f8ce56cf2f368d4c74637dcbc53fdbf323f10940965c1c0e592e2fb4ce725d5cd467e77e62b6e5
123043e99c ci: Bump lint task image to Ubuntu Jammy (Hennadii Stepanov)
9b86114058 ci: Use pyenv's `python-build` to install Python in lint task (Hennadii Stepanov)
Pull request description:
This PR:
- is an alternative to bitcoin/bitcoin#26581 and bitcoin/bitcoin#26637
- closesbitcoin/bitcoin#26548
Key advantages of this PR over others:
- it uses pyenv's `python-build` [standalone](https://github.com/pyenv/pyenv/tree/master/plugins/python-build#using-python-build-standalone)
- requires no additional computational resources
Note for testing. The lint task must success regardless of whether the `python_cache` is populated or invalidated.
ACKs for top commit:
MarcoFalke:
ACK 123043e99c
fanquake:
ACK 123043e99c
Tree-SHA512: ba0fcdd4f2939a59692b173dcd1f5704444cfcfbb8111538c6f8160056d0536bba250e4f9b0f8c66f8b454e52034bf36ffe6afae76cdc0f7cc5b58b576d790ba
DOCKER in names is confusingly used as synonym for "image", "container",
and "ci". Fix the confusion by picking the term that fits the context.
-BEGIN VERIFY SCRIPT-
ren() { sed -i "s:$1:$2:g" $( git grep -l "$1" ) ; }
ren DOCKER_PACKAGES CI_BASE_PACKAGES
# This better reflects that they are the common base for all CI
# containers.
ren DOCKER_ID CI_CONTAINER_ID
# This is according to the documentation of "--detach , -d: Run
# container in background and print container ID".
ren DOCKER_NAME_TAG CI_IMAGE_NAME_TAG
# This avoids confusing with CONTAINER_NAME and clarifies that it is an
# image.
ren DOCKER_ADMIN CI_CONTAINER_CAP
# This clarifies that it is a capability added to the container.
ren DOCKER_CI_CMD_PREFIX CI_EXEC_CMD_PREFIX
# This brings it in line with the CI_EXEC naming.
-END VERIFY SCRIPT-
fa5cbf2290 ci: Properly set COMMIT_RANGE in lint task (MarcoFalke)
Pull request description:
Currently the variable holds (apart from the commits in the pull request) all commits to master since the pull was opened.
This is problematic, because already merged commits are linted in unrelated pulls, leading to:
* Wasted resources. For example, currently the lint task may take 9 minutes, when it should take 1. See https://cirrus-ci.com/task/6032782770569216?logs=lint#L1449
* False failures. For example, when a "wrong" commit is in master it can lead to some pulls failing unrelatedly, and others not.
Now that the CI has the `/merge` commit (since commit fad7281d78), `COMMIT_RANGE` can simply be set to `HEAD~..HEAD` to only hold the changes in the pull.
ACKs for top commit:
fanquake:
ACK fa5cbf2290
Tree-SHA512: e85fca4ca9d2615ddd2544403485e06885769a3f70bca297e23eefda2a1d28f47c5271f6adfa6ce0e5e972335c78098b76e0db4b109f59d0986bf508cef7528f
fabb6af850 ci: Remove duplicate CC and CXX from tsan task (MarcoFalke)
fa5d9a0e24 Revert "ci: Use clang-15 in tsan task" (MarcoFalke)
faa835e7e5 Revert "test: Drop no longer needed `race:epoll_ctl` TSan suppression" (MarcoFalke)
Pull request description:
Looks like there are still bugs in clang-15, so we need to roll back all the way to the previously used version (clang-13).
ACKs for top commit:
hebasto:
ACK fabb6af850, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: d62203049847ab9095ee3fc89e18bdd721d1d9d5a7ef7a9f524c80e6be58d1d9f6aa2f14533df1ea77eb59597fba6fa9b987b17eb03b2c3f7cb577ab59cd59c0
- Constructors of uint256 to utilize Span instead of requiring a std::vector
- converts m_data into a std::array
- Prefers using `WIDTH` instead of `sizeof(m_data)`
- make all the things constexpr
- replace C style functions with c++ equivalents
- memset -> std::fill
- memcpy -> std::copy
Note: In practice, implementations of std::copy avoid multiple assignments and use bulk copy functions such as std::memmove if the value type is TriviallyCopyable and the iterator types satisfy LegacyContiguousIterator. (https://en.cppreference.com/w/cpp/algorithm/copy)
- memcmp -> std::memcmp
2811f40f30 ci: only run USDT interface tests on CirrusCI (0xb10c)
Pull request description:
As mentioned in #26571, the task running the USDT interface tests fail when run in docker. cc7335edc8 in #25528 added that the tests are run in a **VM** in Cirrus CI. Running them locally in docker containers might not work:
- We use [bcc] as tracing toolkit which requires the kernel headers to compile the BPF bytecode. As docker containers use the hosts kernel and don't run their own, there is a potential for mismatches between kernel headers available in the container and the host kernel. This results in a failure loading the BPF byte code.
- Privilges are required to load the BPF byte code into the kernel. Normally, the docker containers aren't run with these.
- We currently use an untrusted third-party PPA to install the bpfcc-tools package on Ubuntu 22.04. Using this on a local dev system could be a security risk.
To not hinder the ASan + LSan + UBSan part of the CI task, the USDT tests are disabled on non-CirrusCI runs.
[bcc]: https://github.com/iovisor/bcc
Top commit has no ACKs.
Tree-SHA512: 7c159b59630b36d5ed927b501fa0ad6f54ff3d98d0902f975cc8122b4147a7f828175807d40a470a85f0f3b6eeb79307d3465a287dbd2f3d75b803769ad0b6e7
As mentioned in #26571, the task running the USDT interface tests
fail when run in docker. cc7335edc8
in #25528 added that the tests are run in a **VM** in Cirrus CI.
Running them locally in docker containers might not work:
- We use [bcc] as tracing toolkit which requires the kernel headers
to compile the BPF bytecode. As docker containers use the hosts
kernel and don't run their own, there is a potential for mismatches
between kernel headers available in the container and the host
kernel. This results in a failure loading the BPF byte code.
- Privilges are required to load the BPF byte code into the kernel.
Normally, the docker containers aren't run with these.
- We currently use an untrusted third-party PPA to install the
bpfcc-tools package on Ubuntu 22.04. Using this on a local dev
system could be a security risk.
To not hinder the ASan + LSan + UBSan part of the CI task, the USDT
tests are disabled on non-CirrusCI runs.
[bcc]: https://github.com/iovisor/bcc
fad1c55301 lint: Skip COMMIT_RANGE if no CIRRUS_PR (MarcoFalke)
Pull request description:
It doesn't make sense to run this for non-PRs, because:
* There are known whitespace "violations" in previous commits, so the lint may fail
* Once the changes are merged, it is too late to fix them up (force pushes are illegal)
* It isn't possible to determine which commits to run on if there is no reference branch (target branch of the pull request)
Moreover, the test fails on non-master:
* https://github.com/bitcoin/bitcoin/runs/8664441400
Fix all issues by skipping it.
ACKs for top commit:
hebasto:
ACK fad1c55301, also tested in my personal Cirrus account.
Tree-SHA512: be15f00e2b2a9069583833545883e0e5968a33d2455dad59e6fb47c1102b4dd16ef932e9ba945e29e9d941e6c17bd531a02c66b0491097801be6bda476875537
54dd8f51ce ci: use ci_exec_root for clang install (josibake)
Pull request description:
fixes a bug introduced in #25900 ; see https://github.com/bitcoin/bitcoin/pull/25900#issuecomment-1327311069
the general idea of #25900 was to use a non-root user as much as possible to avoid modifying the user's local filesystem. however, it appears the root user is needed to correctly install clang.
ACKs for top commit:
hebasto:
ACK 54dd8f51ce, tested on Ubuntu 22.04.
Tree-SHA512: beb01d4b6127fbba3c8d18e85cf7ec7d1b2ec93ea05c475ab51bcaa04ef1b0591d886f1a7e0732c5ae86806013f022c0b44027380d2b0cfb1bfdc843e40f99b4
849f20a6d3 ci: create and use non-root user for docker image (josibake)
Pull request description:
Previously, everything in the ci docker image ran as the root user. This would lead to certain directories (`ci/scratch`, `depends`) being owned by `root` after running the ci locally which would lead to annoying behavior such as subsequent guix builds failing due to `depends/` being owned by root.
This PR adds a non-root user in the container and chowns the mounted working directory. All the `docker exec` commands now run as the non-root user, except for the few that still need to run as root (mainly, installing packages).
To test this I checked out a fresh copy of the repo, applied my changes, ran the CI, and verified all the local file permissions were unchanged after the CI was finished running.
ACKs for top commit:
hebasto:
ACK 849f20a6d3, tested on Ubuntu 22.04 by running commands as follows:
Tree-SHA512: 734dca0f36157fce5fab243b4ff657fc17ba980e8e4e4644305f41002ff21bd5cef02c306ea1e0b5c841d4c07c095e8e4be16722e6a38c890717c60a3f5ec62a
b89530483d util: move threadinterrupt into util (fanquake)
Pull request description:
Alongside thread and threadnames. It's part of libbitcoin_util.
ACKs for top commit:
ryanofsky:
Code review ACK b89530483d. No changes since last review other than rebase
theuni:
ACK b89530483d.
Tree-SHA512: 0421f4d1881ec295272446804b27d16bf63e6b62b272f8bb52bfecde9ae6605e8109ed16294690d3e3ce4b15cc5e7c4046f99442df73adb10bdf069d3fb165aa
Running all commands as the root user in the docker image
will change local file permissions in the ci and depends directory.
Add a non-root user to the container and use this user whenever
possible when running docker exec commands.
da16893474 ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task (Hennadii Stepanov)
702836530f ci: Make `getopt` path architecture agnostic (Hennadii Stepanov)
Pull request description:
The "macOS native" CI task always uses the recent OS image.
This PR updates it up to the recent macOS release.
Cirrus Labs [stopped](https://github.com/bitcoin/bitcoin/pull/25160#issuecomment-1162829773) updating macOS images for `x86_64`, therefore, an `arm64` image been used.
Also `make test-security-check` has been dropped as it ["isn't even expected to pass"](https://github.com/bitcoin/bitcoin/issues/26386#issuecomment-1290318628) on `arm64` in CI.
ACKs for top commit:
Sjors:
utACK da16893
Tree-SHA512: 36785d33b7f11b3cdbc53bcfbf97d88bf821fad248c825982dd9f8e3413809a4ef11190eaf950e60fdf479b62ff66920c35d9ea42d534723f015742eec7e19b6
fa6054e952 ci: Allow PIP_PACKAGES on centos (MacroFake)
fac085a05c ci: Remove unused package (MacroFake)
Pull request description:
This was added in 7fc5e865b9 but I can't see a reason why this should be forbidden.
This is also needed for other changes (bumping the minimum python version).
ACKs for top commit:
hebasto:
re-ACK fa6054e952
Tree-SHA512: e8ead9ee00079024eb1e8c6e7b31c78cf2a3392159b444765c2ea9a58bed2a7550bf71083210692a45bb8ed7896cb882b72bf70baa13a2384864b2b510b73005
21a9e94dbb ci: remove hardcoded tag list from ci scripts (josibake)
d530ba390e doc: update test/README.md (josibake)
614d4682ba script: default to necessary tags in get_previous_releases.py (josibake)
Pull request description:
Almost every time I need to use this script, I forget the tag list is needed and that a specific set of tags is needed for the backwards compatibility tests to work. I end up wasting time reading through the script and googling to find the tag list before remembering it is in `test/README.md`
I assume (hope) I'm not the only one this happens to, so I figured it would make more sense to have the script default to downloading/building the necessary tags. This has the added benefit of making the script the source of truth: the script already needs to be updated with the SHA256_SUM of the binary for every new tag that is added, so it makes sense to use `SHA256_SUMS` list as the necessary tag list. This means there is less risk of the README and the script drifting (i.e updating the readme with a new tag and forgetting to update the script, or updating the script and forgetting to update the README). Now all that needs to happen is to update the `SHA256_SUMS` list in the script and everything Just Works (TM)
ACKs for top commit:
Sjors:
re-tACK 21a9e94dbb
Tree-SHA512: 97b488227a89a6827584edd251820a7074fad75dfd7f26f1aa5f858e2521d2e02effd0f11e6dc4676e1155d3d5aba6ff94a4b58ffef80dc201376afd5927deb9
Also:
- Make DEFAULT_MAX_SIG_CACHE_SIZE into constexpr
DEFAULT_MAX_SIG_CACHE_BYTES to utilize the compile-time integer
arithmetic overflow checking available to constexpr.
- Fix comment (MiB instead of MB) for DEFAULT_MAX_SIG_CACHE_BYTES.
- Pass in max_size_bytes parameter to InitS*Cache(), modify log line to
no longer allude to maxsigcachesize being split evenly between the two
validation caches.
- Fix possible integer truncation and add a comment.
[META] I've kept the integer types as int64_t in order to not introduce
unintended behaviour changes, in the next commit we will make
them size_t.
It is part of the node library. Also, it won't be moved to the kernel
lib, as it will be pruned of ArgsManager.
-BEGIN VERIFY SCRIPT-
# Move module
git mv src/mempool_args.cpp src/node/
git mv src/mempool_args.h src/node/
# Replacements
sed -i 's:mempool_args\.h:node/mempool_args.h:g' $(git grep -l mempool_args)
sed -i 's:mempool_args\.cpp:node/mempool_args.cpp:g' $(git grep -l mempool_args)
sed -i 's:MEMPOOL_ARGS_H:NODE_MEMPOOL_ARGS_H:g' $(git grep -l MEMPOOL_ARGS_H)
-END VERIFY SCRIPT-
cc7335edc8 ci: run USDT interface test in a VM (0xb10c)
dba6f82342 test: adopt USDT utxocache interface tests (0xb10c)
220a5a2841 test: hook into PID in tracing tests (0xb10c)
Pull request description:
Changes a CI task that runs test the previously not run `test/functional/interface_usdt_*.py` functional tests (added in https://github.com/bitcoin/bitcoin/pull/24358).
This task is run as CirussCI `compute_engine_instance` VM as hooking into the tracepoints is not possible in CirrusCI docker containers (https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845). We use an unoffical PPA and untrusted `bpfcc-tools` package in the CI as the Ubuntu jammy and Debian bullseye packages are outdated. We hope use an official package when new Ubuntu/Debian releases are available for the use with Google Compute Engine.
We make sure to hook into `bitcoind` binaries in USDT interface tests via their PID, instead of their path. This makes sure multiple functional tests running in parallel don't interfere with each other.
The utxocache USDT interface tests is adopted to a change of the functional test framework that wasn't detected as the tests weren't run in the CI. As the tracepoints expose internals, it can happen that we need to adopt the interface test when internals change. This is a bit awkward, and if it happens to frequently, we should consider generalizing the tests a bit more. For now it's fine, I think.
See the individual commit messages for more details on the changes.
Fixes https://github.com/bitcoin/bitcoin/issues/24782
Fixes https://github.com/bitcoin/bitcoin/issues/23296
I'd like to hear from reviewers:
- Are we OK with using the [`hadret/bpfcc`](https://launchpad.net/~hadret/+archive/ubuntu/bpfcc) PPA for now? There is a clear plan when to drop it and as is currently, it could only impact the newly added VM task.
- ~~Adding a new task increases CI runtime and costs. Should an existing `container` CI task be ported to a VM and reused instead?~~ Yes, see https://github.com/bitcoin/bitcoin/pull/25528#issuecomment-1179509525
ACKs for top commit:
MarcoFalke:
cr ACK cc7335edc8
Tree-SHA512: b7fddccc0a77d82371229d048abe0bf2c4ccaa45906497ef3040cf99e7f05561890aef4c253c40e4afc96bb838c9787fae81c8454c6fd9db583276e005a4ccb3
fa74e726c4 refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono) (MacroFake)
fa3b3cb9b5 Expose underlying clock in CThreadInterrupt (MacroFake)
Pull request description:
This gets rid of the `value*1000` manual conversion.
ACKs for top commit:
naumenkogs:
utACK fa74e726c4
dergoegge:
Code review ACK fa74e726c4
Tree-SHA512: 90409c05c25f0dd2f1c4dead78f707ebfd78b7d84ea4db9fcefd9c4958a1a3338ac657cd9e99eb8b47d52d4485fa3c947dce4ee1559fb56ae65878685e1ed9a3
cb3e9a1e3f Move {Load,Dump}Mempool to kernel namespace (Carl Dong)
aa30676541 Move DEFAULT_PERSIST_MEMPOOL out of libbitcoinkernel (Carl Dong)
06b88ffb8a LoadMempool: Pass in load_path, stop using gArgs (Carl Dong)
b857ac60d9 test/fuzz: Invoke LoadMempool via CChainState (Carl Dong)
b3267258b0 Move FopenFn to fsbridge namespace (Carl Dong)
ae1e8e3756 mempool: Use NodeClock+friends for LoadMempool (Carl Dong)
f9e8e5719f mempool: Improve comments for [GS]etLoadTried (Carl Dong)
813962da0b scripted-diff: Rename m_is_loaded -> m_load_tried (Carl Dong)
413f4bb52b DumpMempool: Pass in dump_path, stop using gArgs (Carl Dong)
bd4407817e DumpMempool: Use std::chrono instead of weird int64_t arthmetics (Carl Dong)
c84390b741 test/mempool_persist: Test manual savemempool when -persistmempool=0 (Carl Dong)
Pull request description:
This is part of the `libbitcoinkernel` project: #24303, https://github.com/bitcoin/bitcoin/projects/18
-----
This PR moves `{Dump,Load}Mempool` into its own `kernel/mempool_persist` module and introduces `ArgsManager` `node::` helpers in `node/mempool_persist_args`to remove the scattered calls to `GetBoolArg("-persistmempool", DEFAULT_PERSIST_MEMPOOL)`.
More context can be gleaned from the commit messages.
-----
One thing I was reflecting on as I wrote this was that in the long run, I think we should probably invert the validation <-> mempool relationship. Instead of mempool not depending on validation, it might make more sense to have validation not depend on mempool. Not super urgent since `libbitcoinkernel` will include both validation and mempool, but perhaps something for the future.
ACKs for top commit:
glozow:
re ACK cb3e9a1e3f via `git range-diff 7ae032e...cb3e9a1`
MarcoFalke:
ACK cb3e9a1e3f🔒
ryanofsky:
Code review ACK cb3e9a1e3f
Tree-SHA512: 979d7237c3abb5a1dd9b5ad3dbf3b954f906a6d8320ed7b923557f41a4472deccae3e8a6bca0018c8e7a3c4a93afecc502acd1e26756f2054f157f1c0edd939d
Overloading sleep_for is not needed, as
* seconds and minutes can be converted to milliseconds by the compiler,
not needing a duration_cast
* std::condition_variable::wait_for will convert milliseconds to the
duration type of the underlying clock
So simply expose the clock.
Our CI tasks are run by CirrusCI in Docker containers in a Google
Compute Engine based Kubernetes environment. These containers have
limited capabilities - especially CAP_SYS_ADMIN is missing. See
https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845
We need elevated privileges to hook into the USDT tracepoints. We use a
CirrusCI "compute_engine_instance" (a VM, not a container) where we have
the required privileges. The ubunut-mininmal-2204-lts was choosen with
debian-11 being an alternative. Both pack an outdated 'bpfcc-tools'
package (v0.18.0) from 2020. This version prints warnings to stderr
during BPF bytecode compilation, which causes our functional test runner
to fail. This is fixed in newer verison.
Until debian-12 or a newer Ubuntu release is avaliable as image in GCE
(https://cloud.google.com/compute/docs/images/os-details), we use a
third-party and untrusted PPA that releases up-to-date versions of the
package.
The official iovisor (authors of BCC) PPA is outdated too. An
alternative would be to compile BCC from source in the CI.
Co-authored-by: MacroFake <falke.marco@gmail.com>
These files change infrequently, and not much header shuffling is required.
We don't add everything in src/util/ yet, because IWYU makes some
dubious suggestions, which I'm going to follow up with upstream.
