b2ee8b207d net: Deserialize hardcoded seeds from BIP155 blob (W. J. van der Laan)
9b29d5df7f contrib: Add explicit port numbers for testnet seeds (W. J. van der Laan)
2a257de113 contrib: Add a few TorV3 seed nodes (W. J. van der Laan)
06030f7a42 contrib: generate-seeds.py generates output in BIP155 format (W. J. van der Laan)
Pull request description:
Closes#20239 and mitigates my node's problem in #21351.
- Add a few hardcoded seeds for TorV3
- As the [bitcoin-seeder](https://github.com/sipa/bitcoin-seeder) doesn't collect TorV3 addresses yet, I have extracted these from my own node using [a script](https://gist.github.com/laanwj/b3d7b01ef61ce07c2eff0a72a6b90183) and added them manually. This is intended to be a temporary stop gap until 22.0's seeds update.
- Change hardcoded seeds to variable length BIP155 binary format.
- It is stored as a single serialized blob in a byte array, instead of pseudo-IPv6 address slots. This is more flexible and, assuming most of the list is IPv4, more compact.
- Only the (networkID, addr, port) subset (CService). Services and time are construed on the fly as before.
- Change input format for `nodes_*.txt`.
- Drop legacy `0xAABBCCDD` format for IPv4. It is never generated by `makeseeds.py`.
- Stop interpreting lack of port as default port, interpret it as 'no port', to accomodate I2P and other port-less protocols (not handled in this PR). An explicit port is always generated by `makeseeds.py` so in practice this makes no difference right now.
A follow-up to this PR could do the same for I2P.
ACKs for top commit:
jonatack:
ACK b2ee8b207d
Tree-SHA512: 11a6b54f9fb0192560f2bd7b218f798f86c1abe01d1bf37f734cb88b91848124beb2de801ca4e6f856e9946aea5dc3ee16b0dbb9863799e42eec1b239d40d59d
Many users have reported problems that stem from having an unclean
working tree. To that end, I've written a guix-clean script which should
help reset the working tree while respecting user-specified precious
directories.
Precious directories, such as:
- SOURCES_PATH
- BASE_CACHE
- SDK_PATH
- OUTDIR
Should be preserved when cleaning the working tree, and are thus
recorded in ./contrib/guix/var/precious_dirs.
The ./contrib/guix/guix-clean script is able to parse that file and make
sure to avoid them when cleaning out the working tree.
7476b46f18 guix: Build dmg as a static binary (Carl Dong)
06d6cf6784 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong)
65176ab573 guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong)
ca85679eb4 guix: Use clang-toolchain instead of clang (Carl Dong)
1aec0eda8f guix: Fallback to local build for substitute-enabled Guix users (Carl Dong)
1742f8e12d guix: Add early health check for guix-daemon (Carl Dong)
c1ae726a13 guix: More thoroughly control native toolchain (Carl Dong)
39741128d3 guix: Supply --link-profile (Carl Dong)
d55a1056ee guix: Add troubleshooting documentation entries (Carl Dong)
7f401c953f guix: Adapt guix-build to prelude, restructure hier (Carl Dong)
4eccf063b2 guix: Remove guix-build.sh filename extension (Carl Dong)
7753357a7b guix: Add source-able bash prelude and utils (Carl Dong)
e5b49a01f5 guix: Create windeploy inside distsrc-* (Carl Dong)
3e9982ab38 contrib: Silence git-describe when looking for tag (Carl Dong)
d5a71e9785 guix: Use --cores instead of --max-jobs (Carl Dong)
Pull request description:
This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts:
- Resolves confusion between `--cores=` and `--max-jobs=`
- `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var
- `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that
- `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people)
- Add troubleshooting documentation to `README.md`
- Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon`
- Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally
- `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple`
A few robustness changes are also included:
- We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile`
- We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug
- We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages.
Finally, we restructure the guix building hierarchy such that it looks something like:
```
guix-build-<short-hash-or-version-tag>
├── distsrc-<short-hash-or-version-tag>-${HOST}
│ ├── contrib
│ ├── depends
│ ├── src
│ └── ...
├── distsrc-<short-hash-or-version-tag>-...
└── output
├── dist-archive
│ └── bitcoin-<short-hash-or-version-tag>.tar.gz
├── *-linux-*
│ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz
│ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz
├── x86_64-apple-darwin18
│ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz
│ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg
│ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz
└── x86_64-w64-mingw32
├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip
├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe
├── bitcoin-<short-hash-or-version-tag>-win64.zip
└── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz
```
Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts.
ACKs for top commit:
sipa:
ACK 7476b46f18
laanwj:
ACK 7476b46f18
Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
This relatively easy change eliminates all runtime dependencies (except
for the kernel) for dmg, which is the only native build tool that gets
put in our output tarballs.
This allows much more flexibility when constructing the codesigning
environment, and is much more robust.
./windeploy is a "working directory", and therefore belongs inside
distsrc-*. Many people have noticed their Guix builds failing after
hours simply because they did not remove windeploy (but did remove the
distsrc-* directories).
In Guix, there are two flags for controlling parallelism:
Note: When I say "derivation," think "package"
--cores=n
- controls the number of CPU cores to build each derivation. This is
the value passed to `make`'s `--jobs=` flag.
- defaults to 0: as many cores as is available
--max-jobs=n
- controls how many derivations can be built in parallel
- defaults to 1
Therefore, if set --max-jobs=$MAX_JOBS and don't set --cores, Guix could
theoretically spin up $MAX_JOBS * $(nproc) number of threads, and that's
no good.
So we could either default to --cores=1, --max-jobs=$MAX_JOBS
- Pro: --cores=1 means that `make` will be invoked with `-j1`,
avoiding problems with package whose build systems and test
suites break when running multi-threaded.
- Con: There will be times when only 1 or 2 derivations can be built
at a time, because the rest of the dependency graph all depend
on those 1 or 2 derivations. During these times, the machine
will be severely under-utilized.
or --cores=$MAX_JOBS, --max-jobs=1
- Pro: We don't encounter prolonged periods of
severe under-utilization mentioned above.
- Con: Many packages' build systems and test suites break when running
multi-threaded.
or --cores=1, --max-jobs=1 and let the user override with
$ADDITIONAL_GUIX_COMMON_FLAGS
3a0446fad4 script: Add explanatory comment to tc.sh (dscotese)
Pull request description:
This is a replacement for #21289
tc.sh is used to limit bandwidth. I ran it and it is limiting my bandwidth. When I ran it, I got one error. I have not found an explanation anywhere of what the error means, but my best guess is consistent with the result, so I propose the explanatory comment to save others time when they use it and also get the error.
ACKs for top commit:
laanwj:
that said, LGTM ACK 3a0446fad4
Tree-SHA512: 5403a2a0fec3724625c20402a96334c3c7a620324a930c5fd828017da8911d2867aecb7a2ad94a23d1f189009d3eb197a67eb59c8e4531fd215d9b1edb600440
663f6cd9dd contrib: Use -daemonwait in systemd init script (Wladimir J. van der Laan)
Pull request description:
Make systemd invoke dependencies only when ready by using `-daemonwait` in the service file instead of `-daemon`.
Closes#21322 by making bitcoind conform to behavior specified for `type=forking`.
This may need some tuning of timeouts.
ACKs for top commit:
darosior:
ACK 663f6cd
hebasto:
re-ACK 663f6cd9dd
Tree-SHA512: 890005852b632a202caa578e6c796ebdc9da0b2379a9157a4f56f7db9d193c0ffbb78d120bbf112ab2f273855f2a08c3da000b1f7a9fb5222a3b94dcdb16b878
Because only macOS wasy mentioned, I was unsure if this would be a macOS specific tool. I guess Linux is more used than Mac, so Linux guide should be there, too.
0fc0c00f7a test: Drop unused get_machine function (Hennadii Stepanov)
61a0f8f9cc test: Cleanup test files in test-{security,symbol}-check.py (Hennadii Stepanov)
Pull request description:
1) Test source and executable files are neither ignored by `.gitignore` nor removed by `make clean` and `make distclean`.
2) The `get_machine` function is no longer used since #21255.
ACKs for top commit:
fanquake:
ACK 0fc0c00f7a
Tree-SHA512: ef3fcf22d4a04b6e4f37f748bd4be57e09696d2a77982e26292843cb2a1297789c8325f5c4bdad37d8094fce7765c4cc9ab19809e07471487943361b2b1a252c
e4c0cada79 ci, gitian: Drop unneeded python3-dev package for macOS builds (Hennadii Stepanov)
Pull request description:
ACKs for top commit:
fanquake:
ACK e4c0cada79 - gitian builds match and I checked that this doesn't end up installed as a side-effect of another package.
Tree-SHA512: 520a3909b106a0e005b195c5395691edf62b76ee2df43b6971b7aa193648d68e6dac69cb4f1dc474f594b015a2fc2074061865e571d89365174beb5c1780356f
95f97111dd contrib/init: (OpenRC) quote some unquoted variables. (parazyd)
737feadff7 contrib/init: (OpenRC) Do not fail if both rpcuser and rpcpassword are unset. (parazyd)
Pull request description:
This pull request improves the available OpenRC initscripts in
`contrib/init`.
The first commit (737feadff7) reworks
`checkconfig()` to not fail if **both** `rpcuser` and `rpcpassword`
are unset, because this implies that bitcoind shall use the `.cookie`
file for RPC authentication. Currently, the initscript does not allow
starting bitcoind without a set `rpcuser` and `rpcpassword`.
The second commit (95f97111dd) simply
quotes some unquoted variables.
ACKs for top commit:
kristapsk:
ACK 95f97111dd
Tree-SHA512: 62bebcd07143c147e349c0cfc17b54ef21bd4684377b444f58c6bd1f509a4d3e1af58746fa7215f18e33021f691bbbc5e42f4df497458322b055e545b7f30d46
remove fix_configure_mac.patch
Fixed upstream: https://bugreports.qt.io/browse/QTBUG-67286
remove fix_riscv64_arch.patch
Was fixed upstream in 6a39e49a6cdeb28a04a3657bb6a22f848d5dfa9d
remove fix_rcc_determinism.patch
Fixed upstream in https://bugreports.qt.io/browse/QTBUG-62511
remove freetype_back_compat.patch
By the time we ship a release with Qt 5.12, we'll certainly no-longer be
supporting Ubuntu 14.04 and Ubuntu 16.04 ships with FreeType 2.6.1,
which is new enough that using the symbol is no-longer an issue.
The renaming of FT_Get_X11_Font_Format() happened in FreeType 2.6
remove xkb-default.patch
This was removed upstream in d5abf545971da717014d316127045fc19edbcd65
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
f7f3829a68 build, doc: Drop libbz2-dev from macOS cross-compiling dependencies (Hennadii Stepanov)
d8239362e2 build, doc: Drop libcap-dev from macOS cross-compiling dependencies (Hennadii Stepanov)
Pull request description:
The `libcap-dev` and `libbz2-dev` packages are no longer required when cross-compiling for macOS.
ACKs for top commit:
fanquake:
ACK f7f3829a68
Tree-SHA512: 820cdc2724f3346c0942d4d4115fc7206f7bf02889d9fa6cbdbd1d9e3afa03a067c1c3fa64dff596aefdc74898178b7c7d64027a6501486e3b606f4760de04ae
fbbb2d4fc1 lint: Fix spelling errors in comments (fyquah)
Pull request description:
Found some spelling errors while running spelling linter https://github.com/bitcoin/bitcoin/pull/21245
This PR fixes them.
ACKs for top commit:
fanquake:
ACK fbbb2d4fc1 - I thought we just fixed all of these.
Tree-SHA512: 95525040001f94e899b778c616cb66ebafb679dff88835b66fccf6349d8eb942d6b7374c536a44e393f13156bce9a32ed57e6a82bb02074d2b3cddb2696addb2
c33b199456 guix: Bump glibc and linux-headers (Carl Dong)
65363a1bd8 guix: Rebase on 95aca2991b (1.2.0-12.dffc918) (Carl Dong)
Pull request description:
On bumping the time-machine:
```
A few changes which are useful for us:
1. 'gnu: cross-gcc-arguments: Enable 128 bit long double for POWER9.' is
now merged into master.
2. gnutls is bumped to 3.6.15 and the temporal test failure in
status-request-revoked is fixed. Note that this does not fix the case
where one has installed Guix v1.2.0 and is running a substitute-less
bootstrap build, since the `guix time-machine` command itself has a
dependency on gnutls v3.6.12 (the one with the broken test) and will
thus try to build it before attempting to jump forwards in time. This
does however, mean that those who build a version of Guix that also
contains this fix will not go backwards in time to build the broken
gnutls v3.6.12.
```
On bumping the rest:
```
Bump glibc and linux-headers to match those of our Gitian counterparts.
We also require a glibc >= 2.28 for the test-symbol-check scripts to
work properly.
The default BASE-GCC-FOR-LIBC also has to be bumped since glibc 2.31
requires a gcc >= 6.2
```
This is a prerequisite for #20980
ACKs for top commit:
fanquake:
ACK c33b199456 - I think going ahead with this now and to sycn back up to gitian is fine. It will also unblock #20980. Potential code signing related issues can be sorted out in #21239 and later PRs.
Tree-SHA512: 31f022aadb93ba44813b0da005b1f2e5d67d76e8cdcdb53368924d1ea6cb076a21218c26831a6b0dcdcfe33507f54934330489ba557371d740f5587b7d727b95
a0a7a4337d guix, doc: Update default HOSTS value (Hennadii Stepanov)
Pull request description:
This is a #21089 follow up.
ACKs for top commit:
fanquake:
ACK a0a7a4337d
Tree-SHA512: c1813cc2b9212a79fd34d4e25cd0816b58264e1890daf777cd59411bd20fcc9affe312871d06fab1308b8f55c1a78ac1101e631882c18360a4709ecef4529f05
13a9fd11a5 guix: Passthrough SDK_PATH into container (Carl Dong)
Pull request description:
This is a usability improvement for Guix builders so that they don't have to extract the Xcode tarball into `depends/SDKs` every time.
Inspiration: https://github.com/bitcoin/bitcoin/pull/21089#issuecomment-778639698
ACKs for top commit:
laanwj:
Tested ACK 13a9fd11a5
Tree-SHA512: 63392d537e48a0da9f0ee04a929613b139bef1ac5643187871c9ea5376afd2a3d95df0f5e0950ae0eccd2813b166667be98401e5a248ae9c187fe4e84e54d427
Bump glibc and linux-headers to match those of our Gitian counterparts.
We also require a glibc >= 2.28 for the test-symbol-check scripts to
work properly.
The default BASE-GCC-FOR-LIBC also has to be bumped since glibc 2.31
requires a gcc >= 6.2
A few changes which are useful for us:
1. 'gnu: cross-gcc-arguments: Enable 128 bit long double for POWER9.' is
now merged into master.
2. gnutls is bumped to 3.6.15 and the temporal test failure in
status-request-revoked is fixed. Note that this does not fix the case
where one has installed Guix v1.2.0 and is running a substitute-less
bootstrap build, since the `guix time-machine` command itself has a
dependency on gnutls v3.6.12 (the one with the broken test) and will
thus try to build it before attempting to jump forwards in time. This
does however, mean that those who build a version of Guix that also
contains this fix will not go backwards in time to build the broken
gnutls v3.6.12.
a6a1b106dc guix: only download sources for hosts being built (fanquake)
Pull request description:
For example, if a user is only interested in building for Linux, this saves downloading the macOS compiler and additional dependencies, which is meaningful on a slow/poor connection. This will result in a few additional `make` invocations, for the Linux hosts, however this is low overhead, and time-wise irrelevant in terms of the overall build.
ACKs for top commit:
laanwj:
Code review ACK a6a1b106dc
Tree-SHA512: 34c916ae6f69fed0d5845690b39111a8bee37208fd727176f375cf5eb4860f512abe12bde2680d697c859b4d50a3bc5688ddca7c2f28f9968fcf358753cf3f6d
95990b9f32 guix: Update conservative space requirements (Carl Dong)
5e6df11326 guix: Add support for powerpc64{,le} (Carl Dong)
Pull request description:
```
The new time-machine commit contains a few small changes that make the
powerpc cross-toolchain work.
```
See this compare to review my custom patches to Guix: 7d6bd44da5...6c9d16db96
ACKs for top commit:
fanquake:
ACK 95990b9f32
Tree-SHA512: 464b0fb93d65962d8c27499293edb618d13d18f40d44e3eed96935e86d430666dfb1c5b8a30f99ffdfd17b44514ad88e358977390b689a2e3831d521f6f7b86a
If a user is only interested in building for Linux, this saves
downloading the macOS compiler and additional dependencies.
This will result in a few additional `make` invocations, for the Linux
hosts, however this is quite low overhead.
Co-authored-by: Carl Dong <contact@carldong.me>
The new time-machine commit is Guix v1.2.0 with a yet-unupstreamed patch
for NSIS.
A few important changes:
1. Guix switched back from using CPATH to C{,PLUS}_INCLUDE_PATH as the
way to indicate #include search paths.
2. GCC's library is now split into a separate output, whereas before it
was included in the default output. This means that our gcc toolchain
packages need to propagate that output.
3. A few package versions were bumped
When building with g++-10 (or 8) on Focal, binaries are being produced
with noexecstack by default, so we can remove the workaround of
explicitly passing "-Wl,-z,noexecstack" for risvc46 and powerpc64le.
When building for powerpc64 this is still required.
Compilers used change as follows:
Linux native GCC 7.5 -> GCC 8.4
Linux cross GCC 8.4 -> GCC 8.4
Windows mingw-w64 7.3 -> mingw-w64 9.3
macOS Clang 8.0.0 -> Clang 8.0.0
The macOS and Win cross builds in the CI are updated to use Focal, and
per the op, running the security tests is disabled in the Windows
build.
bff7c66e67 Add documentation to contrib folder (Troy Giorshev)
381f77be85 Add Message Capture Test (Troy Giorshev)
e4f378a505 Add capture parser (Troy Giorshev)
4d1a582549 Call CaptureMessage at appropriate locations (Troy Giorshev)
f2a77ff97b Add CaptureMessage (Troy Giorshev)
dbf779d5de Clean PushMessage and ProcessMessages (Troy Giorshev)
Pull request description:
This PR introduces per-peer message capture into Bitcoin Core. 📓
## Purpose
The purpose and scope of this feature is intentionally limited. It answers a question anyone new to Bitcoin's P2P protocol has had: "Can I see what messages my node is sending and receiving?".
## Functionality
When a new debug-only command line argument `capturemessages` is set, any message that the node receives or sends is captured. The capture occurs in the MessageHandler thread. When receiving a message, it is captured as soon as the MessageHandler thread takes the message off of the vProcessMsg queue. When sending, the message is captured just before the message is pushed onto the vSendMsg queue.
The message capture is as minimal as possible to reduce the performance impact on the node. Messages are captured to a new `message_capture` folder in the datadir. Each node has their own subfolder named with their IP address and port. Inside, received and sent messages are captured into two binary files, msgs_recv.dat and msgs_sent.dat, like so:
```
message_capture/203.0.113.7:56072/msgs_recv.dat
message_capture/203.0.113.7:56072/msgs_sent.dat
```
Because the messages are raw binary dumps, included in this PR is a Python parsing tool to convert the binary files into human-readable JSON. This script has been placed on its own and out of the way in the new `contrib/message-capture` folder. Its usage is simple and easily discovered by the autogenerated `-h` option.
## Future Maintenance
I sympathize greatly with anyone who says "the best code is no code".
The future maintenance of this feature will be minimal. The logic to deserialize the payload of the p2p messages exists in our testing framework. As long as our testing framework works, so will this tool.
Additionally, I hope that the simplicity of this tool will mean that it gets used frequently, so that problems will be discovered and solved when they are small.
## FAQ
"Why not just use Wireshark"
Yes, Wireshark has the ability to filter and decode Bitcoin messages. However, the purpose of the message capture added in this PR is to assist with debugging, primarily for new developers looking to improve their knowledge of the Bitcoin Protocol. This drives the design in a different direction than Wireshark, in two different ways. First, this tool must be convenient and simple to use. Using an external tool, like Wireshark, requires setup and interpretation of the results. To a new user who doesn't necessarily know what to expect, this is unnecessary difficulty. This tool, on the other hand, "just works". Turn on the command line flag, run your node, run the script, read the JSON. Second, because this tool is being used for debugging, we want it to be as close to the true behavior of the node as possible. A lot can happen in the SocketHandler thread that would be missed by Wireshark.
Additionally, if we are to use Wireshark, we are at the mercy of whoever it maintaining the protocol in Wireshark, both as to it being accurate and recent. As can be seen by the **many** previous attempts to include Bitcoin in Wireshark (google "bitcoin dissector") this is easier said than done.
Lastly, I truly believe that this tool will be used significantly more by being included in the codebase. It's just that much more discoverable.
ACKs for top commit:
MarcoFalke:
re-ACK bff7c66e67 only some minor changes: 👚
jnewbery:
utACK bff7c66e67
theStack:
re-ACK bff7c66e67
Tree-SHA512: e59e3160422269221f70f98720b47842775781c247c064071d546c24fa7a35a0e5534e8baa4b4591a750d7eb16de6b4ecf54cbee6d193b261f4f104e28c15f47
c86b9a65eb contrib: remove verify.sh (Sebastian Falbesoner)
c84838e7af contrib: binary verification script verify.sh rewritten in python (Sebastian Falbesoner)
Pull request description:
The rationale for the PR is the same as for #18132:
> Most of our test scripts are written in python. We don't have enough reviewers for bash scripts and they tend to be clumsy anyway. Especially when it comes to argument parsing.
Note that there are still a lot of things that could be improved in this replacement (e.g. using regexps for version string parsing, adding type annotations, dividing up into more functions, getting a pylint score closer to 10, etc.), but I found the original shell script quite hard to read, so it's possibly still a good first step for an improvement.
~Not sure though if it's worth the reviewers time, and if it's even continued to be used long-term (maybe there are plans to merge it with `get_previous_releases.py`, which partly does the same?), so chasing for Concept ACKs right now.~
ACKs for top commit:
laanwj:
Tested and code review ACK c86b9a65eb
Tree-SHA512: f7949eead4ef7e5913fe273923ae5c5299408db485146cf996cdf6f8ad8c0ee4f4b30bb6b08a5964000d97b2ae2e7a1bdc88d11c613c16d2d135d80b444e3b16
543bf745d3 gitian-linux: Extend noexec-stack workaround to powerpc (Wladimir J. van der Laan)
00f67c8aa1 gitian-linux: Build binaries for 64-bit POWER (Luke Dashjr)
63fc2b1782 gitian: Properly quote arguments in wrappers (Luke Dashjr)
798bc0b29a Support glibc-back-compat on 64-bit POWER (Luke Dashjr)
Pull request description:
Rebase of #14066 by luke-jr.
Let's try to get PowerPC support in in the beginning of the 22.0 cycle so that it gets some testing, and is not a last-minute decision this time, like for last … 2 or 3 major versions.
The symbol/security tooling-related changes have been dropped since they were part of #20434.
Top commit has no ACKs.
Tree-SHA512: df0f8cd320c90f359f8b512c5cb8b59bb277516b57a05482cc8923c656106513b7428e315aaa8ab53e0bd6f80556b07d3639c47f6d9913bcfbfe388b39ef47c4
This commit adds brief documentation for this feature. Included in the
justification is the purpose of this feature as well as usage and
functionality tips.
This commit adds contrib/message-capture/message-capture-parser.py, a python
script to be used alongside -capturemessages to parse the captured
messages.
It is complete with arguments and will parse any file given, sorting the
messages in the files when creating the output. If an output file is
specified with -o or --output, it will dump the messages in json format
to that file, otherwise it will print to stdout.
The small change to the unused msg_generic is to bring it in line with
the other message classes, purely to avoid a bug in the future.
1fca9811e1 lint: Skip whitespace lint for guix patches (Carl Dong)
a91c46c57d guix: Make nsis reproducible by respecting SOURCE-DATE-EPOCH (Carl Dong)
Pull request description:
```
When building nsis, if VERSION is not specified, it defaults to
cvs_version which is non-deterministic as it includes the current date.
This patches nsis to default to SOURCE_DATE_EPOCH if it exists so that
nsis is reproducible.
Upstream change: https://github.com/kichik/nsis/pull/13
```
Sidenote: also a good demonstration of how Guix allows us to flexibly patch our tools!
Note to reviewers: if you want to compare hashes, please build after Jan 16th 2021 without my substitute server enabled!
ACKs for top commit:
fanquake:
ACK 1fca9811e1
Tree-SHA512: b800e0ce5f73827ad353739effb9167ec3a6bdb362c725ae20dd3f025ce78660f85c70ce1d75cd0896facf1e8fe38a9e058459ed13dec71ab3a2fe41e20eaa5d
92370033a2 contrib: embed C++11 patch in install_db4.sh (jackielove4u)
Pull request description:
This is a continuation of https://github.com/bitcoin/bitcoin/pull/20665.
Closes#20722.
ACKs for top commit:
laanwj:
ACK 92370033a2
fanquake:
ACK 92370033a2.
Tree-SHA512: ebfd16f5301158de1acc1b8eeca43b3d94f0a6d438832133a30648e5e8a88268b4af983be0bb57f3018e3af8459f32f0de676c1b4e8942e199a4497c776631c5
2c403279e2 gitian: Remove codesign_allocate and pagestuff from MacOS build (Andrew Chow)
f55eed2514 gitian: use signapple to create the MacOS code signature (Andrew Chow)
95b06d2185 gitian: use signapple to apply the MacOS code signature (Andrew Chow)
42bb1ea363 gitian: install signapple in gitian-osx-signer.yml (Andrew Chow)
Pull request description:
The MacOS code signing issues that were encountered during the 0.21.0 release cycle have shown that it is necessary for us to use a code signing tool for which the source code is available and modifiable by us. Given that there appears to not be such a tool available, I have written such a tool, [signapple](https://github.com/achow101/signapple), that we can use. This tool is able to create a valid MacOS code signature, detach it in a way that we were doing previously, and attach it to the unsigned binary. This tool can also verify that the signature is correct.
This PR implements the usage of that tool in the gitian build for the code signed MacOS binary. The code signer will use this tool to create the detached signature. Gitian builders will use this tool to apply the detached signature. The `gitian-osx-signer.yml` descriptor has been modified to install this tool so that the `detached-sig-apply.sh` script can use it. Additionally, the `codesign_allocate` and `pagestuff` tools are no longer necessary so they are no longer added to the tarball used in code signing. Lastly, both the `detached-sig-create.sh` and `detached-sig-apply.sh` scripts are made to be significantly less complex and to not do unexpected things such as unpacking an already unpacked tarball.
The detached code signature that signapple creates is almost identical to that which we were previously creating. The only difference is that the cpu architecture name is included in the extension (e.g. we have `bitcoin-qt.x86_64sign` instead of `bitcoin-qt.sign`). This was done in order to support signing universal binaries which we may want to do in the future. However signapple can still apply existing code signatures as it will accept the `.sign` extension. If it is desired, it can be modified to produce signatures with just the `.sign` extension. However I do not think it is necessary to maintain compatibility with the old process.
ACKs for top commit:
laanwj:
Code review ACK 2c403279e2
Tree-SHA512: 2a0e01e9133f8859b9de26e7e8fe1d2610d2cbdee2845e6008b12c083c7e3622cbb2d9b83c50a269e2c3074ab95914a8225d3cd4108017f58b77a62bf10951e0
9d02654677 doc: Fix systemd spelling and link to doc/init.md (Hennadii Stepanov)
601778c310 script: Add Documentation key to bitcoind.service (Hennadii Stepanov)
d9392b724c script: Improve robustness of bitcoind.service on startup (Hennadii Stepanov)
Pull request description:
If network interfaces are not properly up the following happens:
```
...
2021-01-08T10:17:11Z scheduler thread start
2021-01-08T10:17:11Z libevent: getaddrinfo: address family for nodename not supported
2021-01-08T10:17:11Z Binding RPC on address 127.0.0.1 port 8332 failed.
2021-01-08T10:17:11Z HTTP: creating work queue of depth 16
2021-01-08T10:17:11Z Using random cookie authentication.
2021-01-08T10:17:11Z Generated RPC authentication cookie /var/lib/bitcoind/.cookie
2021-01-08T10:17:11Z HTTP: starting 2 worker threads
2021-01-08T10:17:11Z init message: Loading banlist...
2021-01-08T10:17:11Z SetNetworkActive: true
2021-01-08T10:17:11Z Error: Cannot resolve -externalip address: <EDITED>
2021-01-08T10:17:11Z Shutdown: In progress...
2021-01-08T10:17:11Z scheduler thread exit
2021-01-08T10:17:11Z Shutdown: done
```
This PR improves robustness on startup in such cases in documented way:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
Also minor doc improvements are added.
ACKs for top commit:
Sjors:
ACK 9d02654
practicalswift:
ACK 9d02654677: patch looks correct
darosior:
ACK 9d02654677 -- been using the first patch too
Tree-SHA512: 38294f5682c09e6ea9008de7d7459098c920cf1b98ad8ef8a5d2ca01f2f781c0fec5591dc40ef36eeb19d94991b0c7fb7cb38c4e716bc7219875c9bcd0a55e1b
When building nsis, if VERSION is not specified, it defaults to
cvs_version which is non-deterministic as it includes the current date.
This patches nsis to default to SOURCE_DATE_EPOCH if it exists so that
nsis is reproducible.
Upstream change: https://github.com/kichik/nsis/pull/13
595a34dbea contrib/signet: Document miner script in README.md (Anthony Towns)
ff7dbdc08a contrib/signet: Add script for generating a signet chain (Anthony Towns)
13762bcc96 Add bitcoin-util command line utility (Anthony Towns)
95d5d5e625 rpc: allow getblocktemplate for test chains when unconnected or in IBD (Anthony Towns)
81c54dec20 rpc: update getblocktemplate with signet rule, include signet_challenge (Anthony Towns)
Pull request description:
Adds `contrib/signet/miner` for mining signet blocks.
Adds `bitcoin-util` cli utility, with the idea being it can provide bitcoin related functionality that does not rely on the ability to access a running node. Only subcommand currently is "grind" which takes a hex-encoded header and grinds its nonce until its nBits is satisfied.
Updates `getblocktemplate` to include `signet_challenge` field, and makes `getblocktemplate` require the signet rule when invoked on the signet change. Removes connectivity and IBD checks from `getblocktemplate` when applied to a test chain (regtest, testnet, signet).
ACKs for top commit:
laanwj:
code review ACK 595a34dbea
Tree-SHA512: 8d43297710fdc1edc58acd9b53e1bd1671e5724f7097b40ab73653715dc8becc70534c4496cbba9290f4dd6538a7a3d5830eb85f83391ea31a3bb5b9d3378cc3
