Commit graph

790 commits

Author SHA1 Message Date
MarcoFalke
fa8409e760
build: Bump g++ minimum supported version to 11 2024-03-14 12:15:22 +01:00
MarcoFalke
fae70ba00d
ci: Better tidy errors 2024-03-14 09:14:06 +01:00
fanquake
ce54330cf6
ci: use Debian Bookworm (GCC 12) for ARM ci job
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
2024-03-12 16:26:27 +00:00
fanquake
0da6451c58
ci: use Debian Bookworm (GCC 12) for win64 job
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
2024-03-12 16:26:27 +00:00
fanquake
bd55b7a528
Merge bitcoin/bitcoin#29610: ci: Fix "macOS native" job
acc06bc91f ci, macos: Use `--break-system-packages` with Homebrew's python (Hennadii Stepanov)
ae5f72027f ci: Add workaround for Homebrew's python link error (Hennadii Stepanov)

Pull request description:

  Homebrew [promoted](https://github.com/Homebrew/homebrew-core/pull/150390) `python@3.12` to the default `python3`. Now, our "macOS native" CI job is facing the following issues:

  1. Installing `qt@5` [requires](https://github.com/bitcoin/bitcoin/actions/runs/8216848118/job/22471875454#step:4:51) re-installing `python@3.12`:
  ```
  ==> Fetching dependencies for qt@5: readline, python@3.12 and gettext
  ```
  2. Re-installing `python@3.12` [fails](https://github.com/bitcoin/bitcoin/actions/runs/8216848118/job/22471875454#step:4:127) due to symbolic link conflicts on macOS `x86_64`:
  ```
  ==> Pouring python@3.12--3.12.2_1.ventura.bottle.tar.gz
  Error: The `brew link` step did not complete successfully
  ```
  3. Homebrew's `python@3.12` is marked as externally managed (according to PEP 668), necessitating different approaches for installing Python packages.

  This pull request resolves all the issues mentioned above.

ACKs for top commit:
  m3dwards:
    reACK acc06bc91f to get the CI passing again.

Tree-SHA512: 82cf72bff328f1e0725342431ac14ad4bae7a758186d97db6c7a558e4b661dcbf3fabe536978e26e709c5f6f7f5c11aac46642634c6685f1291592d8d825ad87
2024-03-12 11:43:33 +00:00
Hennadii Stepanov
acc06bc91f
ci, macos: Use --break-system-packages with Homebrew's python
Homebrew's python@3.12 is marked as externally managed (PEP 668),
necessitating different approaches for installing Python packages.

For more details, please refer to https://github.com/orgs/Homebrew/discussions/3404.
2024-03-11 19:57:56 +00:00
Martin Zumsande
ecc036c5d6 ci: add --v2transport to an existing CI job 2024-02-29 13:50:30 -05:00
fanquake
eaede27655
Merge bitcoin/bitcoin#29408: lint: Check for missing bitcoin-config.h includes
fa58ae74ea refactor: Add missing include for USE_BDB, USE_SQLITE to bench/wallet_ismine.cpp (MarcoFalke)
fa31908ea8 lint: Check for missing or redundant bitcoin-config.h includes (MarcoFalke)
fa63b0e351 lint: Make lint error easier to spot in output (MarcoFalke)
fa770fd368 doc: Add missing RUST_BACKTRACE=1 (MarcoFalke)
fa10051267 lint: Add get_subtrees() helper (MarcoFalke)

Pull request description:

  Missing `bitcoin-config.h` includes are problematic, because the build could silently pass, but produce an unintended result. For example, a slower fallback algorithm could be picked, even though `bitcoin-config.h` indicates that a faster feature is available and should be used.

  As the build succeeds silently, this problem is not possible to detect with iwyu.

  Thus, fix this by using a linter based on grepping the source code.

ACKs for top commit:
  theuni:
    Weak ACK fa58ae74ea.
  TheCharlatan:
    ACK fa58ae74ea
  hebasto:
    ACK fa58ae74ea, tested on Ubuntu 23.10 -- it catches bugs properly. I didn't review rust code changes.

Tree-SHA512: cf4346f81ea5b8c215da6004cb2403d1aaf569589613c305d8ba00329b82b3841da94fe1a69815ce15f2edecbef9b031758ec9b6433564976190e3cf91ec8181
2024-02-26 10:32:28 +00:00
Ryan Ofsky
84388c942c ci: avoid running git diff after patching
Drop `git diff` command so it is easier to run CI locally if git checkout is a
worktree. Currently it fails because the directory is not recognized as a git
repository.

The `git diff` command was added recently in #28359 commit
fa07ac48d8 and can be avoided just by teeing the
patch to stdout
2024-02-21 11:16:56 -05:00
fanquake
46d261631d
Merge bitcoin/bitcoin#29456: docs: ci multi-arch requires qemu
540282905d docs: ci multi-arch requires qemu (Max Edwards)

Pull request description:

  On a fresh Debian system qemu isn't installed and therefore the multi-architecture CI system doesn't run.

  This documentation notes that qemu is required and how to install it.

ACKs for top commit:
  maflcko:
    lgtm ACK 540282905d
  willcl-ark:
    utACK 540282905d

Tree-SHA512: 903c12cf7b16f2146b99a952577c5550d60faf65f9e72b9f4d9479b52228118ab46349e5130de5281b39da05d3bc0b4ae8a8165601e62ce145647a98ef197131
2024-02-21 14:52:36 +00:00
MarcoFalke
fa31908ea8
lint: Check for missing or redundant bitcoin-config.h includes 2024-02-20 15:03:23 +01:00
Max Edwards
540282905d docs: ci multi-arch requires qemu 2024-02-20 10:55:33 +00:00
fanquake
bdddf364c9
Merge bitcoin/bitcoin#29441: ci: Avoid CI failures from temp env file reuse
fa91bf2559 ci: Skip git install if it is already installed (MarcoFalke)
c65fde4831 ci: vary /tmp/env (Sjors Provoost)

Pull request description:

  * Currently, running separate CI tasks at the same time may intermittently fail, because they race to read/write `/tmp/env`. Fix this by adding `$CONTAINER_NAME` to the file name.

  * Also, add `$USER`, while touching the line, to allow different users to run the same CI task at the same time.

  * Also, skip the git install if there is no need.

  Ref: https://github.com/bitcoin/bitcoin/pull/29274

ACKs for top commit:
  Sjors:
    ACK fa91bf2559
  BrandonOdiwuor:
    ACK fa91bf2559
  hebasto:
    ACK fa91bf2559.

Tree-SHA512: 9a8479255a2afb6618f9d0796488d9430ba95266b90ce39536a9817c1974ca4049beeaab5355a38b25171f76fc386dbec06b1919aaa079f08a5a0c0a146232c8
2024-02-20 10:21:44 +00:00
fanquake
8531e1e731
ci: Use DEBUG=1 in depends for MSAN jobs
Followup to #27448, which was deffered, as it produces #27448 and
another similar issue in sqlite, see comment here:
https://github.com/bitcoin/bitcoin/pull/27448#issuecomment-1514902450.
2024-01-26 13:39:05 +00:00
fanquake
800ddef6b9
ci: use LLVM 17.0.6 in MSAN jobs 2024-01-26 13:38:52 +00:00
Sjors Provoost
c65fde4831
ci: vary /tmp/env 2024-01-18 18:18:08 +01:00
fanquake
1dc2c9b385
ci: cleanup C*FLAG usage in Valgrind jobs
This was being used to avoid a missing -O2. After the previous commits,
this is no-longer an issue.
2024-01-16 09:51:19 +00:00
fanquake
17e33fb578
Merge bitcoin/bitcoin#29237: depends: Allow PATH with spaces in directory names.
4756114e50 [depends] Allow PATH with spaces in directory names. (Mark Friedenbach)

Pull request description:

  The goal of this PR is to help close https://github.com/bitcoin/bitcoin/pull/28733. I reverted the change on `depends/config.guess` based on the feedback provided in the previous PR. I've also incorporated the test mentioned by maflcko

ACKs for top commit:
  maflcko:
    lgtm ACK 4756114e50
  hebasto:
    ACK 4756114e50, successfully built depends on Ubuntu 22.04.
  TheCharlatan:
    ACK 4756114e50

Tree-SHA512: ee257f6efd235839156bc236384f08d77b91debc3c257168368a71e70742639f28a3289572b8693609c1109062dc9968e461103d1f4f5679906506e94b54e649
2024-01-15 13:14:32 +00:00
fanquake
8c5e4f42d5
Merge bitcoin/bitcoin#29208: build: Bump clang minimum supported version to 14
aaaace2fd1 fuzz: Assume presence of __builtin_*_overflow, without checks (MarcoFalke)
fa223ba5eb Revert "build: Fix undefined reference to __mulodi4" (MarcoFalke)
fa7c751bd9 build: Bump clang minimum supported version to 14 (MarcoFalke)

Pull request description:

  Most supported operating systems ship with clang-14 (or later), so bump the minimum to that and allow new code to drop workarounds for previous clang bugs.

  For reference:
  * https://packages.debian.org/bookworm/clang (`clang-14`)
  * https://packages.ubuntu.com/jammy/clang (`clang-14`)
  * CentOS-like 8/9 Stream: All Clang versions from 15 to 17
  * FreeBSD 12/13: All Clang versions from 15 to 16
  * OpenSuse Tumbleweed ships with https://software.opensuse.org/package/clang (`clang17`); No idea about OpenSuse Leap

  On operating systems where the clang version is not shipped by default, the user would have to use GCC, or install clang in a different way. For example:

  * https://packages.debian.org/bullseye/g++ (g++-10)
  * https://packages.ubuntu.com/focal/g++-10
  * https://apt.llvm.org/, or nix, or guix, or compile clang from source, ...

ACKs for top commit:
  fanquake:
    ACK aaaace2fd1

Tree-SHA512: 81d066b14cc568d27312f1cc814b09540b038a10a0a8e9d71fc9745b024fb6c32a959af673e6819b817ea7cef98da4abfa63dff16cffb7821b40083016b0291f
2024-01-12 10:03:22 +00:00
Mark Friedenbach
4756114e50 [depends] Allow PATH with spaces in directory names.
added test and update code based on feedback
2024-01-11 15:36:18 -03:00
MarcoFalke
fa0c594b33
ci: Rename tasks (previous releases, macOS cross) 2024-01-11 17:32:43 +01:00
fanquake
f3ca6db8d3
ci: move CMake into base packages
This is already used in multiple CIs, and will soon become a requirement
for most CIs, i.e when we migrate depends packages to use CMake, for
example:
https://github.com/bitcoin/bitcoin/pull/21778#issuecomment-1885576324.

Some of the CIs in 21778 are failing because CMake isn't available, so
just break this out and make CMake globally available.
2024-01-11 11:30:04 +00:00
MarcoFalke
fa7c751bd9
build: Bump clang minimum supported version to 14 2024-01-09 14:17:55 +01:00
Hennadii Stepanov
080763a058
build: Drop ALLOW_HOST_PACKAGES support in depends
The `ALLOW_HOST_PACKAGES` variable was introduced in bitcoin#10508 "to
speed up build and avoid timeout".

It is no longer the case for our CI infrastructure, which uses self-
hosted persistent workers and depends caching.

In the current circumstances, it does not seem worth porting this
feature to the upcoming CMake-based build system.
2024-01-08 15:56:25 +00:00
Hennadii Stepanov
5ddd7c65b4
build: Bump native_clang up to 17.0.6 2023-12-21 09:37:32 +00:00
fanquake
f0e829022a
Merge bitcoin/bitcoin#28967: build: disable external-signer for Windows
308aec3e56 build: disable external-signer for Windows (fanquake)
35537318a1 ci: remove --enable-external-signer from win64 job (fanquake)

Pull request description:

  It's come to light that Boost ASIO (a Boost Process sub dep) has in some
  instances, been quietly  initialising our network stack on Windows (see
  PR https://github.com/bitcoin/bitcoin/pull/28486 and discussion in https://github.com/bitcoin/bitcoin/issues/28940).

  This has been shielding a bug in our own code, but the larger issue
  is that Boost Process/ASIO is running code before main, and doing things
  like setting up networking. This undermines our own assumptions about
  how our binary works, happens before we run any sanity checks,
  and before we call our own code to setup networking. Note that ASIO also
  calls WSAStartup with version `2.0`, whereas we call with `2.2`.

  It's also not clear why a feature like external signer would have a
  dependency that would be doing anything network/socket related,
  given it only exists to spawn a local process.

  See also the discussion in https://github.com/bitcoin/bitcoin/issues/24907. Note that the maintaince of Boost Process in general,
  has not really improved. For example, rather than fixing bugs like https://github.com/boostorg/process/issues/111,
  i.e, https://github.com/boostorg/process/pull/317, the maintainer chooses to just wrap exception causing overflows
  in try-catch blocks: 0c42a58eac. These changes get merged in large,
  unreviewed PRs, i.e https://github.com/boostorg/process/pull/319.

  This PR disables external-signer on Windows for now. If, in future, someone
  changes how Boost Process works, or replaces it entirely with some
  properly reviewed and maintained code, we could reenable this feature on
  Windows.

ACKs for top commit:
  hebasto:
    re-ACK 308aec3e56.
  TheCharlatan:
    ACK 308aec3e56

Tree-SHA512: 7405f7fc9833eeaacd6836c4e5b1c1a7845a40c1fdd55c1060152f8d8189e4777464fde650e11eb1539556a75dddf49667105987078b1457493ee772945da66e
2023-12-13 11:55:21 +00:00
fanquake
dce1dfbc47
Merge bitcoin/bitcoin#28622: build: use macOS 14 SDK (Xcode 15.0)
8ea45e626e build: use macOS 14 SDK (Xcode 15.0) (fanquake)
51c97ffb69 build: patch boost process for macOS 14 SDK (fanquake)
423949a13b depends: add -platform_version to macOS build flags (fanquake)

Pull request description:

  This fixes: https://github.com/bitcoin/bitcoin/pull/28349#issuecomment-1748515277 (cross-compiling with C++20 for macOS). See https://developer.apple.com/xcode/cpp/#c++20 for C++20 support in Apples libc++, some features landed with Xcode 14.3, although many more landed with Xcode 15.0.

ACKs for top commit:
  hebasto:
    ACK 8ea45e626e.
  TheCharlatan:
    ACK 8ea45e626e

Tree-SHA512: 274ce2c9b9f8e4d755c07b8d0d4897a7f92708ac64e6afb7a3f75bdb485e863fc7f40badf3a88b129ce36f6cca72f768dc2ed7fba2bdf0bb6da2bf0c8fedee10
2023-12-07 10:00:47 +00:00
fanquake
d854914043
Merge bitcoin/bitcoin#28992: ci: Use Ubuntu 24.04 Noble for asan,tsan,tidy,fuzz
fad2392c58 ci: Use Ubuntu 24.04 Noble for asan (MarcoFalke)
fa83b65ef8 ci: Use Ubuntu 24.04 Noble for tsan,tidy,fuzz (MarcoFalke)

Pull request description:

  23.10 will be EOL mid next year, so a bump is needed before then for the `master` branch (and possibly the `26.x` branch).

  Doing the bump now is fine, because the clang version is pinned to 17 inside the CI tasks. So a default clang version change in the system image should not affect the tasks. Once clang-18 is available and the default in April next year (https://discourse.ubuntu.com/t/noble-numbat-release-schedule/35649#planned-and-potentially-disruptive-archive-wide-activities-2), the pinned version could be bumped (for CI tasks that require a pin, like tidy), or the pin can be removed (for CI tasks that usually do not require a pin, like fuzz or the sanitizers).

ACKs for top commit:
  fanquake:
    ACK fad2392c58

Tree-SHA512: c40aede4e2281a5d539d5f65d2c08a57bf92e4a00b4f45a4260b57b7443a63d1a0603115da4a3bbd100ac5f6ade3f2eda0916e4b565573741162a76294ec0ac5
2023-12-06 14:13:42 +00:00
fanquake
8ea45e626e
build: use macOS 14 SDK (Xcode 15.0) 2023-12-05 09:55:01 +00:00
MarcoFalke
fad2392c58
ci: Use Ubuntu 24.04 Noble for asan 2023-12-04 12:29:02 +01:00
MarcoFalke
fa83b65ef8
ci: Use Ubuntu 24.04 Noble for tsan,tidy,fuzz 2023-12-04 09:38:42 +01:00
fanquake
35537318a1
ci: remove --enable-external-signer from win64 job
This is redundant in any case, because --enable-external-signer is
already in `BITCOIN_CONFIG_ALL`.
2023-12-01 10:45:57 +00:00
fanquake
97d53e90ba
ci: remove libz-dev from macOS build deps
Remove no-longer used libz-dev (from DMG compression).
Followup to #28432.
Missed this in #28932.
2023-11-30 15:17:19 +00:00
MarcoFalke
fad82fea2b
ci: Reduce use of bash -c
It is confusing to treat commands as a single string. This change is
also required to support paths and strings with spaces in them in the
future.

This requires replacing TEST_RUNNER_ENV with a global export, because it
no longer works. See:

```bash
$ export ENV="A=1" && $ENV ls
bash: A=1: command not found...
```

Or in the CI task:

+ DIR_UNIT_TEST_DATA=/ci_container_base/ci/scratch/qa-assets/unit_test_data/
+ LD_LIBRARY_PATH=/ci_container_base/depends/i686-pc-linux-gnu/lib
+ BITCOIND=bitcoin-node make -j10 check VERBOSE=1
/ci_container_base/ci/test/03_test_script.sh: line 166: BITCOIND=bitcoin-node: command not found

https://github.com/bitcoin/bitcoin/pull/28954/checks?check_run_id=19096858944
https://cirrus-ci.com/task/6718317604372480
2023-11-29 15:13:16 +01:00
MarcoFalke
fafcee4874
ci: Rename test script to 03_test_script.sh 2023-11-28 13:25:40 +01:00
fanquake
0ffcc5b680
ci: remove python3-setuptools from mac build deps 2023-11-23 17:52:38 +00:00
fanquake
172cd92620
Merge bitcoin/bitcoin#28862: lint: Report all lint errors instead of early exit
fa01f884d3 ci: Add missing COPY for ./test/lint/test_runner (MarcoFalke)
faff3e3b46 lint: Report all lint errors instead of early exit (MarcoFalke)

Pull request description:

  `all-lint.py` currently collects all failures. However, the `06_script.sh` does not, since July this year (https://github.com/bitcoin/bitcoin/pull/28103#discussion_r1268115806).

  Fix this by printing all failures before exiting.

  Can be tested by modifying (for example) two subtrees in the same commit and then running the linters.

ACKs for top commit:
  kevkevinpal:
    ACK [fa01f88](fa01f884d3)
  TheCharlatan:
    lgtm ACK fa01f884d3

Tree-SHA512: c0f3110f2907d87e29c755e3b77a67dfae1f8a25833fe6ef8f2f2c58cfecf1aa46f1a20881576b62252b04930140a9e416c78b4edba0780d3c4fa7aaebabba81
2023-11-22 17:00:45 +00:00
Ryan Ofsky
21bfee0720 depends: bump libmultiprocess to fix capnproto deprecation warnings
This incorporates PR https://github.com/chaincodelabs/libmultiprocess/pull/88
"Fix current deprecation warnings as of capnproto-1.0.1" and reverts the
NO_WERROR CI workaround added in https://github.com/bitcoin/bitcoin/pull/28735
2023-11-17 15:27:19 -05:00
fanquake
22025d06e5
Merge bitcoin/bitcoin#28605: Fix typos
43de4d3630 doc: fix typos (Sjors Provoost)

Pull request description:

  This PR fixes typos found by lint-spelling.py using codespell 2.2.6.

  Our CI linter job uses codespell 2.2.5 and found fewer typos that I did locally. In any case it's happy now.

ACKs for top commit:
  pablomartin4btc:
    re ACK 43de4d3630

Tree-SHA512: c032fe86cb49c924a468385653b31f309a9db68c478d70335bba3e65a1ff3826abe80284fe00a090ab5a509e1edbf17e476f6922fb15d055e50f1103dad2ccb0
2023-11-16 10:35:49 +00:00
MarcoFalke
fa01f884d3
ci: Add missing COPY for ./test/lint/test_runner 2023-11-13 18:10:51 +01:00
MarcoFalke
faff3e3b46
lint: Report all lint errors instead of early exit 2023-11-13 18:08:10 +01:00
fanquake
6342348072
Merge bitcoin/bitcoin#28076: util: Replace std::filesystem with util/fs.h
bbbbdb0cd5 ci: Add filesystem lint check (MarcoFalke)
fada2f9110 refactor: Replace <filesystem> with <util/fs.h> (MarcoFalke)

Pull request description:

  Using `std::filesystem` is problematic:

  * There is a `fs` namespace wrapper for it. So having two ways to achieve the same is confusing.
  * Not using the `fs` wrapper is dangerous and buggy, because it disables known bugs by deleting problematic functions.

  Fix all issues by removing use of it and adding a linter to avoid using it again in the future.

ACKs for top commit:
  TheCharlatan:
    ACK  bbbbdb0cd5
  fanquake:
    ACK bbbbdb0cd5 🦀

Tree-SHA512: 0e2d49742b08eb2635e6fce41485277cb9c40fe20b81017c391d3472a43787db1278a236825714ca1e41c9d2f59913865cfb0c649e3c8ab1fb598c849f80c660
2023-11-13 14:10:54 +00:00
fanquake
e862bceb17
Merge bitcoin/bitcoin#27935: fuzz: call lookup functions before calling Ban
fca0a8938e ci: remove "--exclude banman" for fuzzing in mac (brunoerg)
f9b286353f fuzz: call lookup functions before calling `Ban` (brunoerg)

Pull request description:

  Fixes #27924

  To not have any discrepancy, it's required to call lookup functions before calling `Ban`. If we don't do it, the assertion `assert(banmap == banmap_read);` may fail because `BanMapFromJson` will call `LookupSubNet` and cause the discrepancy between the banned and the loaded one. It happens especially in MacOS (#27924).

  Also, calling lookup functions before banning is what RPC `setban` does.

ACKs for top commit:
  maflcko:
    lgtm ACK fca0a8938e
  dergoegge:
    ACK fca0a8938e

Tree-SHA512: a3d635088a556df4507e65542157f10b41d4f87dce42927b58c3b812f262f4544b6b57f3384eef1097ffdd7c32b8dd1556aae201254960cbfbf48d45551200f7
2023-11-13 10:57:01 +00:00
brunoerg
fca0a8938e ci: remove "--exclude banman" for fuzzing in mac 2023-11-09 10:11:59 -03:00
fanquake
b3898e946c
Merge bitcoin/bitcoin#28826: ci: Switch IWYU to clang_17 branch
9f208c0171 ci: Switch IWYU to `clang_17` branch (Hennadii Stepanov)

Pull request description:

  The IWYU version [0.21](https://github.com/include-what-you-use/include-what-you-use/releases/tag/0.21) has been tagged, and the `clang_17` branch is available now.

ACKs for top commit:
  maflcko:
    lgtm ACK 9f208c0171

Tree-SHA512: 8b8f8743d1c2719b6383b5a6a48356ac02a301d1ce9cee77f93cc04c12de22e9ac6b59e23550a589540e68292cfac0d85bacedc9ca26f6b589011d36ee1d38cf
2023-11-09 13:07:05 +00:00
Hennadii Stepanov
9f208c0171
ci: Switch IWYU to clang_17 branch 2023-11-09 12:04:17 +00:00
fanquake
49d953281d
fuzz: explicitly specify llvm-symbolizer path in runner
It's not completely clear to me why this needs to be explicitly
specified in some environments, and not in others, while at the same time
that `llvm-symbolizer` is already in PATH, but this has fixed the 2 issues
outlined in #28147.

Use `LLVM_SYMBOLIZER_PATH` as the env var, as that is somewhat also used
inside LLVM, but not consistently, i.e it's checked for in the asan_symbolize
script, but not in in the ubsan_symbolize script, or from in compiler-rt.
2023-11-07 16:57:23 +00:00
Sjors Provoost
43de4d3630
doc: fix typos
As found by lint-spelling.py using codespell 2.2.6.
2023-11-07 10:21:51 +09:00
MarcoFalke
3333f14efa
depends: Bump to capnproto-c++-1.0.1 2023-11-03 12:48:42 +01:00
MarcoFalke
faea58eee4
Bump g++ minimum supported version to 10
Also, enable -Werror=maybe-uninitialized in
ci/test/00_setup_env_native_qt5.sh
2023-10-30 15:12:26 +01:00
MarcoFalke
fae379b6b1
build: Bump minimum supported Clang to clang-13 2023-10-24 18:52:00 +02:00
fanquake
43704827b4
Merge bitcoin/bitcoin#28211: Bump python minimum supported version to 3.9
fa25e8b0a1 doc: Recommend lint image build on every call (MarcoFalke)
faf70c1f33 Bump python minimum version to 3.9 (MarcoFalke)
fa8996b930 ci: Bump i686_multiprocess.sh to latest Ubuntu LTS (MarcoFalke)

Pull request description:

  All supported operating systems ship with python 3.9 (or later), so bumping the minimum should not cause any issues. A bump will allow new code to use new python 3.9 features.

  For reference:
  * https://packages.debian.org/bullseye/python3
  * https://packages.ubuntu.com/focal/python3.9
  * FreeBSD 12/13 also ships with 3.9
  * CentOS-like 8/9 also ships with 3.9 (and 3.11)
  * OpenSuse Leap also ships with 3.9 (and 3.11) https://software.opensuse.org/package/python311-base

  This is for Bitcoin Core 27.0 in 2024 (next year), not the soon upcoming 26.0 next month.

ACKs for top commit:
  Sjors:
    ACK fa25e8b0a1
  jamesob:
    ACK fa25e8b0a1 ([`jamesob/ackr/28211.1.MarcoFalke.bump_python_minimum_supp`](https://github.com/jamesob/bitcoin/tree/ackr/28211.1.MarcoFalke.bump_python_minimum_supp))

Tree-SHA512: 86c9f6ac4b5ba94a62ee6a6062dd48a8295d8611a39cdb5829f4f0dbc77aaa1a51edccc7a99275bf699143ad3a6fe826de426d413e5a465e3b0e82b86d10c32e
2023-10-24 17:24:30 +01:00
MarcoFalke
fa65887377
ci: Add missing --external to podman image prune 2023-10-23 14:10:12 +02:00
fanquake
fc1073bb45
Merge bitcoin/bitcoin#28643: ci: Add missing CI_RETRY_EXE before git clone
6889a80766 ci: Add missing CI_RETRY_EXE before git clone (MarcoFalke)
b705bade44 ci: Export `IN_GETOPT_BIN` on macOS (Hennadii Stepanov)
1c2132ddd9 Revert "ci: Upgrading pip version in macos environment" (Hennadii Stepanov)

Pull request description:

  This PR is a resurrection of https://github.com/bitcoin/bitcoin/pull/28623:

  > This should fix [bitcoin/bitcoin/actions/runs/6457002476/job/17527598426#step:7:240](https://github.com/bitcoin/bitcoin/actions/runs/6457002476/job/17527598426#step:7:240):
  >
  > ```
  > + git clone --depth=1 https://github.com/bitcoin-core/qa-assets /Users/runner/work/bitcoin/bitcoin/ci/scratch/qa-assets
  > Cloning into '/Users/runner/work/bitcoin/bitcoin/ci/scratch/qa-assets'...
  > error: RPC failed; curl 18 HTTP/2 stream 5 was reset
  > error: 54975 bytes of body are still expected
  > fetch-pack: unexpected disconnect while reading sideband packet
  > fatal: early EOF
  > fatal: fetch-pack: invalid index-pack output
  > Error: Process completed with exit code 128.
  > ```

ACKs for top commit:
  maflcko:
    lgtm ACK 6889a80766

Tree-SHA512: cd50102061a5c57fcf53ecbf552e7eae09995395e91651d00037e3f101c4906c0a994b2f6b1ccf984f2fc902ca59f0db66206ace56a2b6e2810dfa4167f3e115
2023-10-20 11:23:04 +01:00
stickies-v
e44c574650
ci: always prune all dangling bitcoin-ci-test images
Since all bitcoin-ci-test images are now labeled, we can always
prune all dangling images, regardless of whether we are in
RESTART_CI_DOCKER_BEFORE_RUN.

To be safe, still prune all images if RESTART_CI_DOCKER_BEFORE_RUN
in case the filtering doesn't work, or if images were created on
an earlier version that did not assign labels.
2023-10-13 11:23:30 +01:00
MarcoFalke
6889a80766
ci: Add missing CI_RETRY_EXE before git clone 2023-10-13 10:37:34 +01:00
Hennadii Stepanov
b705bade44
ci: Export IN_GETOPT_BIN on macOS
This variable is required for the `retry` script.
2023-10-13 10:37:34 +01:00
Hennadii Stepanov
1c2132ddd9
Revert "ci: Upgrading pip version in macos environment"
This reverts commit 057750c09d.

It is not needed anymore in the GHA CI.
This change will make the code much simpler in the following commit.
2023-10-13 10:37:28 +01:00
stickies-v
ce1699706e
ci: add label to docker images
This allows us or the user to perform batch operations on all
images produced by the ci, e.g. to prune all dangling images,
without affecting non-ci images.
2023-10-13 10:17:21 +01:00
fanquake
504f1f30ed
Merge bitcoin/bitcoin#28547: ci: Work around podman stop intermittent failure
fa2c894cbb ci: move-only CI_CONTAINER_ID to 02_run_container.sh (MarcoFalke)
fa695b4df0 ci: Work around podman stop bug (MarcoFalke)
fa09a031c1 ci: Add set -ex to 02_run_container.sh (MarcoFalke)
fac9abbf47 ci: Rename 04_install to 02_run_container (MarcoFalke)

Pull request description:

  Sometimes, it seems that `podman stop` does not work. Presumably, it falls back to `podman kill`, which is async.

  Try to work around this intermittent issue by using the `rm --force` over `stop`.

  Example failing log https://cirrus-ci.com/task/4549784611061760?logs=ci#L238:

  ```
  Restart docker before run to stop and clear all containers started with --rm
  ++ podman container stop --all
  e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009
  ++ echo 'Prune all dangling images'
  Prune all dangling images
  ++ docker image prune --force
  Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  +++ docker run --cap-add LINUX_IMMUTABLE --rm --interactive --detach --tty --mount type=bind,src=/tmp/cirrus-build-1970593815,dst=/tmp/cirrus-build-1970593815,readonly --mount type=volume,src=ci_macos_cross_ccache,dst=/tmp/ccache_dir --mount type=volume,src=ci_macos_cross_depends,dst=/ci_container_base/depends --mount type=volume,src=ci_macos_cross_previous_releases,dst=/ci_container_base/prev_releases --env-file /tmp/env --name ci_macos_cross ci_macos_cross
  Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  time="2023-09-27T20:55:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
  Error: creating container storage: the container name "ci_macos_cross" is already in use by e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009. You have to remove that container to be able to reuse that name: that name is already in use

ACKs for top commit:
  hebasto:
    ACK fa2c894cbb, I have reviewed the code and tested it locally.

Tree-SHA512: 31fca340c6bedaadf4dd51fa745d9b3969042cebc0c7c904ef18af3f2f986039ec4354ccdff1422fbf77cf223e4423857368dce53cfa67ef15c76b78d007eace
2023-10-13 10:24:07 +02:00
MarcoFalke
fa2843eba4
ci: Bump asan 2023-10-12 13:07:06 +02:00
fanquake
ccf7895e17
Merge bitcoin/bitcoin#28633: ci: Install Qt's default Android API platform
78d3062b68 ci: Install Android API 31 platform as Qt expects (Hennadii Stepanov)

Pull request description:

  When building the `qt` package, it expects that the default (in Qt's view) Android API platform is installed.

  During the recent Qt version [update](https://github.com/bitcoin/bitcoin/pull/28561), it has been changed:
  ```diff
  --- a/mkspecs/features/android/sdk.prf
  +++ b/mkspecs/features/android/sdk.prf
  @@ -1,6 +1,6 @@
   API_VERSION_TO_USE = $$(ANDROID_API_VERSION)
   isEmpty(API_VERSION_TO_USE): API_VERSION_TO_USE = $$API_VERSION
  -isEmpty(API_VERSION_TO_USE): API_VERSION_TO_USE = android-28
  +isEmpty(API_VERSION_TO_USE): API_VERSION_TO_USE = android-31

   ANDROID_JAR_FILE = $$ANDROID_SDK_ROOT/platforms/$$API_VERSION_TO_USE/android.jar
   !exists($$ANDROID_JAR_FILE) {
  ```

  This PR fixes the CI for the Android task and addresses https://github.com/bitcoin/bitcoin/pull/28561#issuecomment-1749180177.

  Qt [docs](https://doc.qt.io/qt-5/android.html) still claim that Android API Level 21 and up are supported, however, I did not test every possible configuration.

  NOTE: https://github.com/bitcoin/bitcoin/pull/28611 is still valid.

ACKs for top commit:
  maflcko:
    lgtm ACK 78d3062b68
  jarolrod:
    tACK 78d3062b68

Tree-SHA512: 781fba6d80aae7e6500854de14af0d30169c258e395b9e482a5430a7b4a2211a6181f8c9ee58543c896b431abf09e3e7c5573b9672ed128658f11f98a2006e7e
2023-10-12 09:11:49 +02:00
Hennadii Stepanov
78d3062b68
ci: Install Android API 31 platform as Qt expects 2023-10-10 19:29:18 +01:00
fanquake
8735e2c136
ci: use LLVM/Clang 17 in tidy job 2023-10-10 11:02:10 +01:00
fanquake
ce46b68941
ci: use LLVM 17.0.2 in MSAN jobs 2023-10-10 10:23:18 +01:00
MarcoFalke
fa2c894cbb
ci: move-only CI_CONTAINER_ID to 02_run_container.sh
This limits the scope of the CI_CONTAINER_ID symbol.

Can be reviewed with --color-moved=dimmed-zebra
2023-10-09 16:17:04 +02:00
MarcoFalke
fa695b4df0
ci: Work around podman stop bug
Force remove any containers, pontentially leaving dangling processes,
which should be fine.
2023-10-09 16:16:55 +02:00
MarcoFalke
fa09a031c1
ci: Add set -ex to 02_run_container.sh
The same is done by the 06 script.
2023-10-09 16:16:41 +02:00
MarcoFalke
fac9abbf47
ci: Rename 04_install to 02_run_container
This reflects what the script does (docker run ...).
2023-10-09 16:16:22 +02:00
MarcoFalke
fac88a874f
ci: Avoid cache depends/build 2023-10-05 13:45:04 +02:00
MarcoFalke
fac054d24c
ci: Print Linux kernel info 2023-10-03 16:37:43 +02:00
fanquake
99ce8366ed
Merge bitcoin/bitcoin#28504: ci: Use nproc over MAKEJOBS in 01_base_install
fa3b5e5e57 ci: Use nproc over MAKEJOBS in 01_base_install (MarcoFalke)

Pull request description:

  Currently `$MAKEJOBS` is the default value in `01_base_install.sh` when building the container image.

  This problem can't be fixed (see below), so just use `nproc` for now.

  Other solutions would be bad:

  * Passing in the `MAKEJOBS` as a dockerfile env would create a new image if the number of tasks are changed, seems verbose and confusing.
  * Leaving `master` as-is would leave CPUs unused if there are more than `4`.

ACKs for top commit:
  fanquake:
    ACK - I think this is fine as-is, it's an improvement over the current state fa3b5e5e57

Tree-SHA512: 15014eb7b548945737b0fed5cd46f0cd4b72b1d54d044f60fce628f914053a2de6518878428a54822d236d08d6f257d8c464d3fb589400f4be56022d2ec8676d
2023-09-20 16:14:30 +00:00
fanquake
e9a4793b82
Merge bitcoin/bitcoin#28432: build: Produce a .zip for macOS distribution
b5790c35f7 build: remove dmg dependencies (fanquake)
33ae0bd1e4 macdeploy: remove DMG generation from deploy script (fanquake)
a128111c29 build: produce a .zip for macOS distribution (Hennadii Stepanov)
c38561d6b1 build: add -zip option to macdeployqtplus (fanquake)

Pull request description:

  It is https://github.com/bitcoin/bitcoin/pull/27099 revived with addressed [comments](https://github.com/bitcoin/bitcoin/pull/27099#issuecomment-1708705686).

  From https://github.com/bitcoin/bitcoin/pull/27099#issue-1584429885:
  > Reviving the discussion around using a `.zip` for the distributed macOS binaries, as opposed to a `.dmg`.
  >
  > Given we only had a single report of the "no finder window" issue (#26176), I wonder if that means macOS users were able to figure it out, they gave up/didn't report, or, we just have very few macOS users.
  >
  > Related to #18128.

  That's how it looks on macOS:

  ![image](https://github.com/bitcoin/bitcoin/assets/32963518/baa637bb-256b-4b24-8645-8c2754c2ae64)

ACKs for top commit:
  Sjors:
    tACK b5790c35f7
  jarolrod:
    ACK b5790c35f7
  TheCharlatan:
    utACK b5790c35f7

Tree-SHA512: 6e9cb3ab0f60f8a92bfec50577e8d096c5b23ec09ebbb334826415609140ddc96d470aea37379495c1c6bb1beec0d306b09460f62e1543bb0f4396c10a1dfbe2
2023-09-20 11:40:47 +00:00
MarcoFalke
fa3b5e5e57
ci: Use nproc over MAKEJOBS in 01_base_install 2023-09-19 16:13:27 +00:00
fanquake
b5790c35f7
build: remove dmg dependencies 2023-09-15 13:47:50 +01:00
Hennadii Stepanov
a128111c29
build: produce a .zip for macOS distribution
Instead of a .dmg.

Co-authored-by: fanquake <fanquake@gmail.com>
2023-09-15 13:47:50 +01:00
fanquake
717a4d8944
Merge bitcoin/bitcoin#28476: ci: LLVM 17 for MSAN jobs
a241d6069c ci: use LLVM 17.0.0 in MSAN jobs (fanquake)

Pull request description:

  See https://libcxx.llvm.org/Hardening.html as well as https://discourse.llvm.org/t/rfc-removing-the-legacy-debug-mode-from-libc/71026.

ACKs for top commit:
  MarcoFalke:
    review ACK a241d6069c

Tree-SHA512: c374dabf307fe762be0da96f63695a150f6018c1468fe9414fad23f74f5818bbf7a5a699e109084e31467482a900cfebf1d5835821e4da94aa310b2c9570749c
2023-09-15 11:49:29 +01:00
Andrew Chow
541976b42e
Merge bitcoin/bitcoin#27850: test: Add unit & functional test coverage for blockstore
de8f9123af test: cover read-only blockstore (Matthew Zipkin)
5c2185b3b6 ci: enable chattr +i capability inside containers (Matthew Zipkin)
e573f24202 unit test: add coverage for BlockManager (Matthew Zipkin)

Pull request description:

  This PR adds unit and functional tests to cover the behavior described in #2039. In particular, that bitcoind will crash on startup if a reindex is requested but the `blk` files are read-only. Eventually this behavior can be updated with https://github.com/bitcoin/bitcoin/pull/27039. This PR just commits the test coverage from #27039 as suggested in https://github.com/bitcoin/bitcoin/pull/27039#issuecomment-1584915782

ACKs for top commit:
  jonatack:
    ACK de8f9123af modulo suggestions in https://github.com/bitcoin/bitcoin/pull/27850#discussion_r1319010039, tested on macOS, but not on Linux for the Linux-related change in the last push
  achow101:
    ACK de8f9123af
  MarcoFalke:
    lgtm ACK de8f9123af 📶

Tree-SHA512: b9bd684035dcea11c901b649fc39f397a2155a9a8459f3348e67947e387e45312fddeccb52981aef486f8a31deebb5356a7901c1bb94b78f82c24192a369af73
2023-09-14 13:21:14 -04:00
MarcoFalke
bbbbdb0cd5
ci: Add filesystem lint check 2023-09-14 18:58:44 +02:00
fanquake
a241d6069c
ci: use LLVM 17.0.0 in MSAN jobs
See https://libcxx.llvm.org/Hardening.html as well as
https://discourse.llvm.org/t/rfc-removing-the-legacy-debug-mode-from-libc/71026.
2023-09-14 11:12:24 +01:00
MarcoFalke
fa23c9aa7c
ci: clang-17 for fuzz and tsan 2023-09-13 13:35:39 +02:00
Matthew Zipkin
5c2185b3b6
ci: enable chattr +i capability inside containers 2023-09-06 10:40:09 -04:00
fanquake
260445bee8
Merge bitcoin/bitcoin#28359: ci: Asan with -ftrivial-auto-var-init=pattern
fa07ac48d8 ci: Asan with -ftrivial-auto-var-init=pattern (MarcoFalke)

Pull request description:

  This makes memory bugs deterministic. `-ftrivial-auto-var-init=pattern` is incompatible with other memory sanitizers (like valgrind and msan), but that is irrelevant here, because the address sanitizer in this fuzz CI config is already incompatible with them.

  `-ftrivial-auto-var-init=pattern` goes well with `-fsanitize=bool` and `-fsanitize=enum`, but those are already enabled via `-fsanitize=undefined`. See https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks

ACKs for top commit:
  fanquake:
    ACK fa07ac48d8 - going to get back to fixing up the cxxflags usage in CI, but not a blocker here:

Tree-SHA512: 2ea6c5128a9cd262bdd1b1475c7e1be23ce2c520fad05f6c2aace6c29e203573323c0564d272e25da35ad5ff46fde488a5eae1ed14d3349e793d14a5e2533fb1
2023-09-05 10:07:50 +01:00
fanquake
df5af114df
Merge bitcoin/bitcoin#28353: ci: Remove /ro_base bind mount, Remove LC_ALL=C from s390x task
fa70cbd969 ci: Remove unused TEST_RUNNER_ENV="LC_ALL=C" from s390x task (MarcoFalke)
fa33354dcb ci: Remove /ro_base bind mount (MarcoFalke)
fa0df9d4c4 doc: Remove sudo from command that is already run as root (MarcoFalke)

Pull request description:

  Remove some CI stuff no longer needed.

ACKs for top commit:
  fanquake:
    ACK  fa70cbd969 - did not test the s390x job.

Tree-SHA512: 3a6ed0cfc855a92c2f834e59494c0a19a5647510247aece5e40a1aa78074894fe7454e684a1ea1f8f0662c50ac1caf2e390398b0fcfbf81544e6488fa9b8915e
2023-08-30 16:23:39 +01:00
MarcoFalke
fa07ac48d8
ci: Asan with -ftrivial-auto-var-init=pattern 2023-08-30 10:52:27 +02:00
MarcoFalke
faf70c1f33
Bump python minimum version to 3.9 2023-08-29 11:54:29 +02:00
MarcoFalke
fa8996b930
ci: Bump i686_multiprocess.sh to latest Ubuntu LTS
There is no need to have it stuck on the previous one.
This is needed for the next commit.
2023-08-29 11:54:22 +02:00
fanquake
db57574aa8
Merge bitcoin/bitcoin#28173: ci: Run Windows native task on GitHub Actions
6a43372980 ci: Run "Win64 native" job on GitHub Actions (Hennadii Stepanov)

Pull request description:

  From https://github.com/bitcoin/bitcoin/issues/28098:
  > Thus, someone would have to sponsor an amount of roughly 5kUSD/mo for those two tasks.

  > If the goal is to stay on a free plan, I think the only option is GitHub Actions CI.

  Historical context:
  - https://github.com/bitcoin/bitcoin/pull/17697
  - https://github.com/bitcoin/bitcoin/issues/17803
  - https://github.com/bitcoin/bitcoin/pull/18031

  Security concerns:
  - https://github.com/bitcoin/bitcoin/issues/28098#issuecomment-1651432106
  - https://github.com/bitcoin/bitcoin/issues/28098#issuecomment-1651688197

  `GITHUB_TOKEN` permissions (from the build log in my personal repo):
  ```
  2023-07-27T07:30:17.8313534Z ##[group]GITHUB_TOKEN Permissions
  2023-07-27T07:30:17.8314113Z Contents: read
  2023-07-27T07:30:17.8314608Z Metadata: read
  2023-07-27T07:30:17.8314957Z Packages: read
  2023-07-27T07:30:17.8315233Z ##[endgroup]
  ```

  Comparison of resources:

  | Resource | Current, Cirrus CI | Suggested, GitHub Actions |
  |---|:-:|:-:|
  | CPU | 6 | 2 |
  | RAM, GB | 12 | 7 |

  The `TEST_RUNNER_TIMEOUT_FACTOR` variable is set to the current default value for all CI tasks: 64440bb733/ci/test/00_setup_env.sh (L48)

Top commit has no ACKs.

Tree-SHA512: ddfdaf7a1e4793a64ac0cd20f116b29608dd06f15b062769ac70b3ea2fb82775aa96aa79c7b768efefec4338aaa5b57d267b592f62d0e8d5d94ecc11001a165d
2023-08-28 11:27:29 +01:00
MarcoFalke
fa70cbd969
ci: Remove unused TEST_RUNNER_ENV="LC_ALL=C" from s390x task 2023-08-27 22:50:42 +02:00
MarcoFalke
fa33354dcb
ci: Remove /ro_base bind mount
Just set the bind mount to BASE_READ_ONLY_DIR, which allows to drop one
line of code and makes the code easier to understand.
2023-08-27 22:50:07 +02:00
fanquake
94955b4b1d
depends: use LLVM/Clang 15.0.6 for macOS cross-compile
There is no x86_64 binaries for 15.0.7.
2023-08-27 10:02:55 +01:00
Hennadii Stepanov
6a43372980
ci: Run "Win64 native" job on GitHub Actions 2023-08-25 13:50:04 +01:00
fanquake
27101d0f0c
Merge bitcoin/bitcoin#28330: ci: Add missing docker.io prefix for native tasks to CI_IMAGE_NAME_TAG
fab7f5c01d ci: Add missing docker.io prefix to CI_IMAGE_NAME_TAG (MarcoFalke)

Pull request description:

  Currently, the CI system may pick the wrong (non-native) architecture due to the missing prefix.

  For example, assuming the CI_IMAGE_NAME_TAG is `debian:bookworm` and the user has previously pulled an s390x image:

  ```
  $ podman run --rm 'docker.io/s390x/debian:bookworm' dpkg --print-architecture
  exec /usr/bin/dpkg: exec format error
  ```

  Now, `debian:bookworm` will refer to the same image:

  ```
  $ podman run --rm 'debian:bookworm' dpkg --print-architecture
  exec /usr/bin/dpkg: exec format error
  ```

  However, `docker.io/debian:bookworm` works fine:

  ```
   $ podman run --rm 'docker.io/debian:bookworm' dpkg --print-architecture
  arm64

ACKs for top commit:
  hebasto:
    ACK fab7f5c01d.

Tree-SHA512: c423c5cd454a95fa3e67081411ca08d316b8c680a5bba49196c514b91df65d9cc46a47700cc00d9579327842615f98146d0ac50abb016616a9b17d042598dab6
2023-08-24 10:56:03 +01:00
MarcoFalke
fab7f5c01d
ci: Add missing docker.io prefix to CI_IMAGE_NAME_TAG 2023-08-23 22:33:02 +02:00
MarcoFalke
fa15f7e082
ci: Remove no longer applicable section
This fails with:

"Error: determining starting point for build: no FROM statement found"
2023-08-23 15:40:21 +02:00
MarcoFalke
fa378bed56
ci: Start with clean env
This should help to avoid non-determinism.
2023-08-23 15:40:05 +02:00
MarcoFalke
fa8c250c2f
ci: Limit scope of some env vars
No need to have a larger scope than needed.

Can be reviewed with --color-moved=dimmed-zebra
2023-08-23 15:39:45 +02:00
fanquake
9b066da8af
Merge bitcoin/bitcoin#28295: ci: Add missing amd64 to win64-cross task
fa56d17a4b ci: Add missing amd64 to win64-cross task (MarcoFalke)

Pull request description:

  Currently the task will fail if run on non-`x86_64`.

  Fix this by adding the missing `amd64`, similar to

  7bf078f2b7/ci/test/00_setup_env_i686_multiprocess.sh (L11)

ACKs for top commit:
  hebasto:
    ACK fa56d17a4b

Tree-SHA512: faab1c5b945283b7e8d080bbcc8e9379c480cf6973506149ace5990cb4d04673f83f4bc36d08d5b4e9cb17a86fdbe23ac97ef4eab0e842616b367b8138229c58
2023-08-18 14:42:01 +01:00
MarcoFalke
fa56d17a4b
ci: Add missing amd64 to win64-cross task
Also, do the same for android, which also fails.
2023-08-18 14:23:17 +02:00
MarcoFalke
fa968ef6a3
ci: Add missing ${CI_RETRY_EXE} before curl 2023-08-18 14:11:40 +02:00