00897d0677 script: actually trigger the optimization in BuildScript (Antoine Poinsot)
Pull request description:
The counter is an optimization over calling `ret.empty()`. It was
suggested that the compiler would realize `cnt` is only `0` on the first
iteration, and not actually emit the check and conditional.
This optimization was actually not triggered at all, since we
incremented `cnt` at the beginning of the first iteration. Fix it by
incrementing at the end instead.
This was reported by Github user "Janus".
Fixes#25682. Note this does *not* change semantics. It only allows the optimization of moving instead of copying on first `CScript` element to actually be reachable.
ACKs for top commit:
sipa:
utACK 00897d0677
MarcoFalke:
review ACK 00897d0677
Tree-SHA512: b575bd444b0cd2fe754ec5f3e2f3f53d2696d5dcebedcace1e38be372c82365e75938dfe185429ed5a83efe1a395e204bfb33efe56c10defc5811eaee50580e3
c99a1ecc52 guix: enable hardening options in GCC Build (fanquake)
aa87879a77 guix: pass enable-bind-now to glibc (fanquake)
3897a131d0 guix: enable SSP for RISC-V glibc (2.27) (fanquake)
Pull request description:
The GCC (10.3.0) and glibcs (2.24 and 2.27) we build both support configuration option for turning on hardening features by default.
For example, [our GCC provides](https://gcc.gnu.org/install/configure.html) `--enable-default-pie`:
> Turn on -fPIE and -pie by default.
`--enable-default-ssp`:
> Turn on -fstack-protector-strong by default.
and `--enable-cet` options:
> Enable building target run-time libraries with control-flow instrumentation, see -fcf-protection option.
It also provides `--enable-standard-branch-protection`, but we don't do that here, because we don't support building with it yet (#24123).
You could verify the that the on-by-default pie flags are working by Guix building master + this change:
```diff
--- a/configure.ac
+++ b/configure.ac
@@ -971,7 +971,6 @@ if test "$use_hardening" != "no"; then
AX_CHECK_LINK_FLAG([-Wl,-z,relro], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-z,relro"], [], [$LDFLAG_WERROR])
AX_CHECK_LINK_FLAG([-Wl,-z,now], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-z,now"], [], [$LDFLAG_WERROR])
AX_CHECK_LINK_FLAG([-Wl,-z,separate-code], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-z,separate-code"], [], [$LDFLAG_WERROR])
- AX_CHECK_LINK_FLAG([-fPIE -pie], [PIE_FLAGS="-fPIE"; HARDENED_LDFLAGS="$HARDENED_LDFLAGS -pie"], [], [$CXXFLAG_WERROR])
```
and verifying that the `PIE` security checks fail. Then, build this PR branch, + the same change, and checking that they still pass.
A similar thing can be done with the stack-protector, i.e perform a Guix build, and observe the security checks failing after applying this diff to master:
```diff
--- a/configure.ac
+++ b/configure.ac
@@ -936,8 +936,6 @@ dnl -fstack-reuse=none for all gcc builds. (Only gcc understands this flag)
AX_CHECK_COMPILE_FLAG([-fstack-reuse=none], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -fstack-reuse=none"])
if test "$use_hardening" != "no"; then
use_hardening=yes
- AX_CHECK_COMPILE_FLAG([-Wstack-protector], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -Wstack-protector"])
- AX_CHECK_COMPILE_FLAG([-fstack-protector-all], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -fstack-protector-all"])
```
Then check that a build doesn't fail when building this PR + that change. Although it should be noted that the security checks will pass for this + that change, even though the GCC option is for stack-protector-strong, rather than stack-protector-all. This is because our stack protector check is currently just for the presencse of the canary, and not a check that every function is instrumented.
[For glibc](https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html), we enable `--enable-stack-protector=all` (RISC-V only):
> Compile the C library and all other parts of the glibc package using the GCC -fstack-protector, -fstack-protector-strong or -fstack-protector-all options to detect stack overruns. Only the dynamic linker and a small number of routines called directly from assembler are excluded from this protection.
and `--enable-bind-now`:
> Disable lazy binding for installed shared objects and programs. This provides additional security hardening because it enables full RELRO and a read-only global offset table (GOT), at the cost of slightly increased program load times.
You could check that the stack-protector option is being used for the RISC-V builds, by comparing the contents of a function that comes from glibc, i.e `atexit`, in a build of master:
```bash
riscv64-linux-gnu/src/bitcoind: file format elf64-littleriscv
00000000007aa078 <atexit>:
7aa078:003a5617 auipca2,0x3a5
7aa07c:f8863603 lda2,-120(a2) # b4f000 <__dso_handle>
7aa080:4581 lia1,0
7aa082:ff8b3317 auipct1,0xff8b3
7aa086:41e30067 jr1054(t1) # 5d4a0 <__cxa_atexit@plt>
```
vs this PR:
```bash
riscv64-linux-gnu/src/bitcoind: file format elf64-littleriscv
00000000007aa078 <atexit>:
7aa078:003aa797 auipca5,0x3aa
7aa07c:3c87b783 lda5,968(a5) # b54440 <__stack_chk_guard@GLIBC_2.27>
7aa080:6398 lda4,0(a5)
7aa082:1101 addisp,sp,-32
7aa084:ec06 sdra,24(sp)
7aa086:e43a sda4,8(sp)
7aa088:6722 lda4,8(sp)
7aa08a:639c lda5,0(a5)
7aa08c:00f71d63 bnea4,a5,7aa0a6 <atexit+0x2e>
7aa090:60e2 ldra,24(sp)
7aa092:003a5617 auipca2,0x3a5
7aa096:f6e63603 lda2,-146(a2) # b4f000 <__dso_handle>
7aa09a:4581 lia1,0
7aa09c:6105 addisp,sp,32
7aa09e:ff8b3317 auipct1,0xff8b3
7aa0a2:40230067 jr1026(t1) # 5d4a0 <__cxa_atexit@plt>
7aa0a6:ff8b3097 auipcra,0xff8b3
7aa0aa:2ba080e7 jalr698(ra) # 5d360 <__stack_chk_fail@plt>
```
Note that none of the above means we would actually remove the use of hardening flags from our configure.
Guix Build (x86_64):
```bash
8de8ceac0f34729f17c64cd3b788d8e73e8a29cf51ec88ae33e04b1002f07162 guix-build-c99a1ecc52d8/output/aarch64-linux-gnu/SHA256SUMS.part
d638d329d2d23324aa8cb491b5fa9cfc59e7998cc95f6c47540ae34767316764 guix-build-c99a1ecc52d8/output/aarch64-linux-gnu/bitcoin-c99a1ecc52d8-aarch64-linux-gnu-debug.tar.gz
ce57cfd97109e2cebc91936653e291073230e9da1197d60edd6703c2c8e4961a guix-build-c99a1ecc52d8/output/aarch64-linux-gnu/bitcoin-c99a1ecc52d8-aarch64-linux-gnu.tar.gz
917770f42ca696048c11ce3e7a100b9cc59cbe482878bccf11c1d84e327e61a7 guix-build-c99a1ecc52d8/output/arm-linux-gnueabihf/SHA256SUMS.part
a5e6ea54cb58941b2dceaa036495c65d83e3ae65b806af7124718df428206b38 guix-build-c99a1ecc52d8/output/arm-linux-gnueabihf/bitcoin-c99a1ecc52d8-arm-linux-gnueabihf-debug.tar.gz
c035aa6599aeab74445bcf15966886fafb1e4397d6f4e66e4e5ff05770f3af94 guix-build-c99a1ecc52d8/output/arm-linux-gnueabihf/bitcoin-c99a1ecc52d8-arm-linux-gnueabihf.tar.gz
a48654be85a540b393fefa87f75f10fcb1652cfb824eb5cb32da9aeffdbe9843 guix-build-c99a1ecc52d8/output/arm64-apple-darwin/SHA256SUMS.part
8cf48b00d6cbe7bc203043dde34ca51a82e25bc3b4e91802730209a90637a8ed guix-build-c99a1ecc52d8/output/arm64-apple-darwin/bitcoin-c99a1ecc52d8-arm64-apple-darwin-unsigned.dmg
6ff1c1f0fbf64303421f71a91c14020554ab96673f2461aae80ef2249a846ebd guix-build-c99a1ecc52d8/output/arm64-apple-darwin/bitcoin-c99a1ecc52d8-arm64-apple-darwin-unsigned.tar.gz
0df1d3d95759b26a9cc448dba29291c5d940e9faf9a79c7658775285498809eb guix-build-c99a1ecc52d8/output/arm64-apple-darwin/bitcoin-c99a1ecc52d8-arm64-apple-darwin.tar.gz
3556666828f68205b8b82771a7046e10e10cf31bd894c6ed389bbaa2397b917c guix-build-c99a1ecc52d8/output/dist-archive/bitcoin-c99a1ecc52d8.tar.gz
970390a724f2b9e40731942a427a5893a489fdac9c970a5a2f52cd684c4e2bcb guix-build-c99a1ecc52d8/output/powerpc64-linux-gnu/SHA256SUMS.part
c281257c8f9466aca2d68971ff8cd219288f62a601396d4f8f1497a4404fac11 guix-build-c99a1ecc52d8/output/powerpc64-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64-linux-gnu-debug.tar.gz
79e68965a50907f4c3382143f7c58dd71b927f87fe80a62c06b434232d764b93 guix-build-c99a1ecc52d8/output/powerpc64-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64-linux-gnu.tar.gz
b65be16861b1d11225f5497c58adbc585bb1b192096018f006ae11c851235d65 guix-build-c99a1ecc52d8/output/powerpc64le-linux-gnu/SHA256SUMS.part
5edb31e2d6702ab3e24189db1a1151bb40dc009a2d6f196eca19124947400a24 guix-build-c99a1ecc52d8/output/powerpc64le-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64le-linux-gnu-debug.tar.gz
d6e0414082f91a443bcfee9647f8cf9ad09d13fdf6acd6070866505b420db8eb guix-build-c99a1ecc52d8/output/powerpc64le-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64le-linux-gnu.tar.gz
89edc84604ea960dff7598999cabb14e2dbd7d585021acfd3065e0e8ebb77786 guix-build-c99a1ecc52d8/output/riscv64-linux-gnu/SHA256SUMS.part
091d582c7797792ab62653e61aa2192db768fb624615a2393284d7fad2a643bd guix-build-c99a1ecc52d8/output/riscv64-linux-gnu/bitcoin-c99a1ecc52d8-riscv64-linux-gnu-debug.tar.gz
fcc20f8f7e2889f544e10d77e714496fd44e3dfdb2d1919b12ec5d41aeb9a8ac guix-build-c99a1ecc52d8/output/riscv64-linux-gnu/bitcoin-c99a1ecc52d8-riscv64-linux-gnu.tar.gz
4b736dbfca1c0eb37390d791a9cdfe12aa3111f65a0c92775cd68044696f5b17 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/SHA256SUMS.part
dc51605e5c0f25e25aa1672471c2096e2c95f59d9c7adbee81714ad33da559a0 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/bitcoin-c99a1ecc52d8-x86_64-apple-darwin-unsigned.dmg
96a7b7b0144049215a4e51a01c4c90dcbf8469590a380fe2b1faca652f80c545 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/bitcoin-c99a1ecc52d8-x86_64-apple-darwin-unsigned.tar.gz
8d0a9e33e02db7c234d3cff2cf8489a93ae83a0efb9c02dd0a4a43b1615d5f75 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/bitcoin-c99a1ecc52d8-x86_64-apple-darwin.tar.gz
60e21c7d8eb8422bf3280d63fca7e3983b8d62949b46f582e483bfadf42d9838 guix-build-c99a1ecc52d8/output/x86_64-linux-gnu/SHA256SUMS.part
93cce61cbd237e8d63a7b60fd7c0611834d2587899f241c80ad3e7c31ce9f5c6 guix-build-c99a1ecc52d8/output/x86_64-linux-gnu/bitcoin-c99a1ecc52d8-x86_64-linux-gnu-debug.tar.gz
86e6d35ced80385dbebc9d0b4e443a86d9b5dfecff4928fccb4331fc37b7c8bc guix-build-c99a1ecc52d8/output/x86_64-linux-gnu/bitcoin-c99a1ecc52d8-x86_64-linux-gnu.tar.gz
cdf1045063b8ad18735d623fa45867a3b6fbcabefac6ef763ad4d04e956ef2b7 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/SHA256SUMS.part
e032c517396d818f2a5f7a2f8453966de37a1734f2f2d95ad0e39358647f5068 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64-debug.zip
b09cc098672215e810b4a11df0ebce760f716546d76745367898bb1850a6a8b4 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64-setup-unsigned.exe
a27108b306be7099a426bf2e02009b7271c8c04394bf5c5aa4f592b69be77fb5 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64-unsigned.tar.gz
a682fe68b09de24e1bdef49836d4fc5080e779fac66a73c9dcafb8fc6126af3a guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64.zip
```
Guix Build (arm64):
```bash
917770f42ca696048c11ce3e7a100b9cc59cbe482878bccf11c1d84e327e61a7 guix-build-c99a1ecc52d8/output/arm-linux-gnueabihf/SHA256SUMS.part
a5e6ea54cb58941b2dceaa036495c65d83e3ae65b806af7124718df428206b38 guix-build-c99a1ecc52d8/output/arm-linux-gnueabihf/bitcoin-c99a1ecc52d8-arm-linux-gnueabihf-debug.tar.gz
c035aa6599aeab74445bcf15966886fafb1e4397d6f4e66e4e5ff05770f3af94 guix-build-c99a1ecc52d8/output/arm-linux-gnueabihf/bitcoin-c99a1ecc52d8-arm-linux-gnueabihf.tar.gz
1a306a6dc68183f210aa56c6eb07785654e1c2e21ac9e2bd866d8fdec34a527c guix-build-c99a1ecc52d8/output/arm64-apple-darwin/SHA256SUMS.part
7da1d43adabf4725b6244df9625b683f47669949ffbcf37184619e431151138f guix-build-c99a1ecc52d8/output/arm64-apple-darwin/bitcoin-c99a1ecc52d8-arm64-apple-darwin-unsigned.dmg
ac38ae4188927e2e0b0d3bdaae9d314424e4f7e3ab2a90c6cbedc8a985ae237e guix-build-c99a1ecc52d8/output/arm64-apple-darwin/bitcoin-c99a1ecc52d8-arm64-apple-darwin-unsigned.tar.gz
1b1653f3b3dff1bf5737223a4e5c2b674b700baba4ef594e3c7a040b5e81f3f6 guix-build-c99a1ecc52d8/output/arm64-apple-darwin/bitcoin-c99a1ecc52d8-arm64-apple-darwin.tar.gz
3556666828f68205b8b82771a7046e10e10cf31bd894c6ed389bbaa2397b917c guix-build-c99a1ecc52d8/output/dist-archive/bitcoin-c99a1ecc52d8.tar.gz
970390a724f2b9e40731942a427a5893a489fdac9c970a5a2f52cd684c4e2bcb guix-build-c99a1ecc52d8/output/powerpc64-linux-gnu/SHA256SUMS.part
c281257c8f9466aca2d68971ff8cd219288f62a601396d4f8f1497a4404fac11 guix-build-c99a1ecc52d8/output/powerpc64-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64-linux-gnu-debug.tar.gz
79e68965a50907f4c3382143f7c58dd71b927f87fe80a62c06b434232d764b93 guix-build-c99a1ecc52d8/output/powerpc64-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64-linux-gnu.tar.gz
b65be16861b1d11225f5497c58adbc585bb1b192096018f006ae11c851235d65 guix-build-c99a1ecc52d8/output/powerpc64le-linux-gnu/SHA256SUMS.part
5edb31e2d6702ab3e24189db1a1151bb40dc009a2d6f196eca19124947400a24 guix-build-c99a1ecc52d8/output/powerpc64le-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64le-linux-gnu-debug.tar.gz
d6e0414082f91a443bcfee9647f8cf9ad09d13fdf6acd6070866505b420db8eb guix-build-c99a1ecc52d8/output/powerpc64le-linux-gnu/bitcoin-c99a1ecc52d8-powerpc64le-linux-gnu.tar.gz
89edc84604ea960dff7598999cabb14e2dbd7d585021acfd3065e0e8ebb77786 guix-build-c99a1ecc52d8/output/riscv64-linux-gnu/SHA256SUMS.part
091d582c7797792ab62653e61aa2192db768fb624615a2393284d7fad2a643bd guix-build-c99a1ecc52d8/output/riscv64-linux-gnu/bitcoin-c99a1ecc52d8-riscv64-linux-gnu-debug.tar.gz
fcc20f8f7e2889f544e10d77e714496fd44e3dfdb2d1919b12ec5d41aeb9a8ac guix-build-c99a1ecc52d8/output/riscv64-linux-gnu/bitcoin-c99a1ecc52d8-riscv64-linux-gnu.tar.gz
4b736dbfca1c0eb37390d791a9cdfe12aa3111f65a0c92775cd68044696f5b17 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/SHA256SUMS.part
dc51605e5c0f25e25aa1672471c2096e2c95f59d9c7adbee81714ad33da559a0 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/bitcoin-c99a1ecc52d8-x86_64-apple-darwin-unsigned.dmg
96a7b7b0144049215a4e51a01c4c90dcbf8469590a380fe2b1faca652f80c545 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/bitcoin-c99a1ecc52d8-x86_64-apple-darwin-unsigned.tar.gz
8d0a9e33e02db7c234d3cff2cf8489a93ae83a0efb9c02dd0a4a43b1615d5f75 guix-build-c99a1ecc52d8/output/x86_64-apple-darwin/bitcoin-c99a1ecc52d8-x86_64-apple-darwin.tar.gz
60e21c7d8eb8422bf3280d63fca7e3983b8d62949b46f582e483bfadf42d9838 guix-build-c99a1ecc52d8/output/x86_64-linux-gnu/SHA256SUMS.part
93cce61cbd237e8d63a7b60fd7c0611834d2587899f241c80ad3e7c31ce9f5c6 guix-build-c99a1ecc52d8/output/x86_64-linux-gnu/bitcoin-c99a1ecc52d8-x86_64-linux-gnu-debug.tar.gz
86e6d35ced80385dbebc9d0b4e443a86d9b5dfecff4928fccb4331fc37b7c8bc guix-build-c99a1ecc52d8/output/x86_64-linux-gnu/bitcoin-c99a1ecc52d8-x86_64-linux-gnu.tar.gz
cdf1045063b8ad18735d623fa45867a3b6fbcabefac6ef763ad4d04e956ef2b7 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/SHA256SUMS.part
e032c517396d818f2a5f7a2f8453966de37a1734f2f2d95ad0e39358647f5068 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64-debug.zip
b09cc098672215e810b4a11df0ebce760f716546d76745367898bb1850a6a8b4 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64-setup-unsigned.exe
a27108b306be7099a426bf2e02009b7271c8c04394bf5c5aa4f592b69be77fb5 guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64-unsigned.tar.gz
a682fe68b09de24e1bdef49836d4fc5080e779fac66a73c9dcafb8fc6126af3a guix-build-c99a1ecc52d8/output/x86_64-w64-mingw32/bitcoin-c99a1ecc52d8-win64.zip
```
ACKs for top commit:
hebasto:
ACK c99a1ecc52
Tree-SHA512: bb895544c06bab741b965d853e60151a2e38e058c5c373f8e1a01a652220d47c8a0227a31fa96d6e81a48126d677fdec1cb47c20c104c89953d4ca051924b402
8f1ff487b3 libxcb: use a patch instead of sed (fanquake)
Pull request description:
To remove the unneeded pthread-stubs requirement.
Should almost be enough to close#16838.
seds dead (mostly). The usage left in `qt.mk` are for substituting runtime values.
ACKs for top commit:
hebasto:
ACK 8f1ff487b3.
Tree-SHA512: 2b6ebbe98a838d8e08e54737292b02176ff4c85a541ae1ec0c590c75e33ba92289628b88ca3144f2e214f4327515f7fd22c39687312f44183b759815c092b24f
3fe58a98d3 tidy: run clang-tidy in quiet mode (fanquake)
Pull request description:
This should cut the amount of output by atleast half.
```bash
/opt/homebrew/opt/llvm/bin/clang-tidy --use-color -p=bitcoin bitcoin/src/wallet/test/fuzz/coinselection.cpp
278 warnings generated.
Suppressed 278 warnings (278 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
```
becomes
```bash
/opt/homebrew/opt/llvm/bin/clang-tidy --use-color -p=/bitcoin -quiet bitcoin/src/wallet/test/fuzz/coinselection.cpp
278 warnings generated.
```
See https://clang.llvm.org/extra/clang-tidy/.
> --quiet
> Run clang-tidy in quiet mode. This suppresses
> printing statistics about ignored warnings and
> warnings treated as errors if the respective
> options are specified.
However the `run-clang-tidy` option is `-quiet`. See f28c006a58/clang-tools-extra/clang-tidy/tool/run-clang-tidy.py (L244).
ACKs for top commit:
jamesob:
ACK 3fe58a98d3
Tree-SHA512: 936cfab09a78e71a4a2a2b3539087c1a95fe279923cc8c3f950c70924d6920dcd54b0da8b5493a1337c23a3b25834ab0db5a6b0c660c6e46d750f0fc966e7625
- make the code easier to read and understand
- improve performance by avoiding unnecessary move operations
- the cleaner, simpler, and easier to read the code is, the
better chance the compiler has at implementing it well
as the classes themselves are private, and to be consistent within all the
*Impl classes in src/node/interfaces.cpp and src/wallet/interfaces.cpp
following this order:
public:
// ... virtual methods ...
// ... nonvirtual helper methods ...
// ... data members ...
and add documentation in src/node/interfaces.cpp and src/wallet/interfaces.cpp
to help future reviewers and contributors.
In AvailableCoins, we need to know whether we can solve for an output.
This was done by using IsSolvable, which just calls ProduceSignature and
produces a dummy signature. However, we already do that in order to get
the size of the input by using CalculateMaximumSignedInputSize. As this
function returns -1 if ProduceSignature fails, we can just remove the
use of IsSolvable and check that input_bytes is not -1 to determine
the solvability of an output.
dd065dae9f refactor: Make mapBlocksUnknownParent local, and rename it (Hennadii Stepanov)
Pull request description:
This PR is a second attempt at #19594. This PR has two motivations:
- Improve code hygiene by eliminating a global variable, `mapBlocksUnknownParent`
- Fix fuzz test OOM when running too long ([see #19594 comment](https://github.com/bitcoin/bitcoin/pull/19594#issuecomment-958801638))
A minor added advantage is to release `mapBlocksUnknownParent` memory when the reindexing phase is done. The current situation is somewhat similar to a memory leak because this map exists unused for the remaining lifetime of the process. It's true that this map should be empty of data elements after use, but its internal metadata (indexing structures, etc.) can have non-trivial size because there can be many thousands of simultaneous elements in this map.
This PR helps our efforts to reduce the use of global variables. This variable isn't just global, it's hidden inside a function (it looks like a local variable but has the `static` attribute).
This global variable exists because the `-reindex` processing code calls `LoadExternalBlockFile()` multiple times (once for each block file), but that function must preserve some state between calls (the `mapBlocksUnknownParent` map). This PR fixes this by allocating this map as a local variable in the caller's scope and passing it in on each call. When reindexing completes, the map goes out of scope and is deallocated.
I tested this manually by reindexing on mainnet and signet. Also, the existing `feature_reindex.py` functional test passes.
ACKs for top commit:
mzumsande:
re-ACK dd065dae9f
theStack:
re-ACK dd065dae9f
shaavan:
reACK dd065dae9f
Tree-SHA512: 9cd20e44d2fa1096dd405bc107bc065ea8f904f5b3f63080341b08d8cf57b790df565f58815c2f331377d044d5306708b4bf6bdfc5ef8d0ed85d8e97d744732c
b4b657ba57 refactor: log `nEvicted` message in `LimitOrphans` then return void (chinggg)
Pull request description:
Fix https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49347
LimitOrphans() can log expired tx and it should log evicted tx as well instead of returning the `nEvicted` number for caller to print the message.
Since `LimitOrphans()` now returns void, the redundant assertion check in fuzz test is also removed.
Top commit has no ACKs.
Tree-SHA512: 18c41702321b0e59812590cd389f3163831d431f4ebdc3b3e1e0698496a6bdbac52288f28f779237a58813c6717da1a35e8933d509822978ff726c1b13cfc778
22f1e4a02f Merge bitcoin-core/leveldb-subtree#32: fix macro HAVE_O_CLOEXEC when O_CLOEXEC not found
1eeb1cb879 fix macro HAVE_O_CLOEXEC when O_CLOEXEC not found
git-subtree-dir: src/leveldb
git-subtree-split: 22f1e4a02fd8e96090bb699a04c95c784aa88e74
Use lto.h from clang+llvm not libtapi. The later is older,
and comes bundled with the libtapi repo.
Copy libLTO.so when building with FORCE_USE_SYSTEM_CLANG.
71d1d13627 test: add unit test for AvailableCoins (josibake)
da03cb41a4 test: functional test for new coin selection logic (josibake)
438e04845b wallet: run coin selection by `OutputType` (josibake)
77b0707206 refactor: use CoinsResult struct in SelectCoins (josibake)
2e67291ca3 refactor: store by OutputType in CoinsResult (josibake)
Pull request description:
# Concept
Following https://github.com/bitcoin/bitcoin/pull/23789, Bitcoin Core wallet will now generate a change address that matches the payment address type. This improves privacy by not revealing which of the outputs is the change at the time of the transaction in scenarios where the input address types differ from the payment address type. However, information about the change can be leaked in a later transaction. This proposal attempts to address that concern.
## Leaking information in a later transaction
Consider the following scenario:
1. Alice has a wallet with bech32 type UTXOs and pays Bob, who gives her a P2SH address
2. Alice's wallet generates a P2SH change output, preserving her privacy in `txid: a`
3. Alice then pays Carol, who gives her a bech32 address
4. Alice's wallet combines the P2SH UTXO with a bech32 UTXO and `txid: b` has two bech32 outputs
From a chain analysis perspective, it is reasonable to infer that the P2SH input in `txid: b` was the change from `txid: a`. To avoid leaking information in this scenario, Alice's wallet should avoid picking the P2SH output and instead fund the transaction with only bech32 Outputs. If the payment to Carol can be funded with just the P2SH output, it should be preferred over the bech32 outputs as this will convert the P2SH UTXO to bech32 UTXOs via the payment and change outputs of the new transaction.
**TLDR;** Avoid mixing output types, spend non-default `OutputTypes` when it is economical to do so.
# Approach
`AvailableCoins` now populates a struct, which makes it easier to access coins by `OutputType`. Coin selection tries to find a funding solution by each output type and chooses the most economical by waste metric. If a solution can't be found without mixing, coin selection runs over the entire wallet, allowing mixing, which is the same as the current behavior.
I've also added a functional test (`test/functional/wallet_avoid_mixing_output_types.py`) and unit test (`src/wallet/test/availablecoins_tests.cpp`.
ACKs for top commit:
achow101:
re-ACK 71d1d13627
aureleoules:
ACK 71d1d13627.
Xekyo:
reACK 71d1d13627 via `git range-diff master 6530d19 71d1d13`
LarryRuane:
ACK 71d1d13627
Tree-SHA512: 2e0716efdae5adf5479446fabc731ae81d595131d3b8bade98b64ba323d0e0c6d964a67f8c14c89c428998bda47993fa924f3cfca1529e2bd49eaa4e31b7e426
4e616d20c9 test: check that combining PSBTs with different txs fails (Sebastian Falbesoner)
2a428c7989 test: support passing PSBTMaps directly to PSBT ctor (Sebastian Falbesoner)
Pull request description:
This PR adds missing test coverage for the `combinepsbt` RPC, in the case of combining two PSBTs with different transactions:
b8067cd435/src/psbt.cpp (L24-L27)
The calling function `CombinePSBTs` checks for the false return value and then returns the transaction error string `PSBT_MISMATCH`:
b8067cd435/src/psbt.cpp (L433-L435)b8067cd435/src/util/error.cpp (L30-L31)
ACKs for top commit:
instagibbs:
reACK 4e616d20c9
achow101:
ACK 4e616d20c9
Tree-SHA512: 45b2b224b13b44ad69ae62e4bc20f74cab32770cf8127b026ec47a7520f7253148fdbf1fad612afece59e45a6738bef9a351ae87ea98dc83d095cc78f6db0318
Pass `--enable-default-pie` and `--enable-default-ssp` when configuring
our GCCs. This achieves the following:
--enable-default-pie
Turn on -fPIE and -pie by default.
--enable-default-ssp
Turn on -fstack-protector-strong by default.
Note that this isn't a replacement for passing hardneing flags
ourselves, but introduces some redundency, and there isn't really a
reason to not build a more "hardenings enabled" toolchain by default.
See also:
https://gcc.gnu.org/install/configure.html
Both glibcs we build support `--enable-bind-now`:
Disable lazy binding for installed shared objects and programs.
This provides additional security hardening because it enables full RELRO
and a read-only global offset table (GOT), at the cost of slightly
increased program load times.
See:
https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
Pass `--enable-stack-protector=all` when building the glibc used for the
RISC-V toolchain, to enable stack smashing protection on all functions,
in the glibc code.
Test each component of the RBF policy in isolation. Unlike the RBF
functional tests, these do not rely on things like RPC results, mempool
submission, etc.
`LimitOrphans()` can log expired tx and it should log evicted tx as well
instead of returning the number for caller to print the message.
Since `LimitOrphans()` now return void, the redundant assertion check in
fuzz test is also removed.
e838a98475 depends: re-enable using -flto when building expat (fanquake)
304452558c depends: expat 2.4.8 (fanquake)
Pull request description:
Currently, when building the expat package in depends, using `-flto` (`LTO=1`), the configure check can fail, because it cannot determine the system endianess:
```bash
configure:18718: result: unknown
configure:18733: error: unknown endianness
presetting ac_cv_c_bigendian=no (or yes) will help
```
Fix that by defining `_DEFAULT_SOURCE`, which in turn defines `__USE_MISC` (`features.h`):
```c
#if defined _DEFAULT_SOURCE
# define __USE_MISC1
#endif
```
which exposes additional definitions in `endian.h`:
```c
#include <features.h>
/* Get the definitions of __*_ENDIAN, __BYTE_ORDER, and __FLOAT_WORD_ORDER. */
#include <bits/endian.h>
#ifdef __USE_MISC
# define LITTLE_ENDIAN__LITTLE_ENDIAN
# define BIG_ENDIAN__BIG_ENDIAN
# define PDP_ENDIAN__PDP_ENDIAN
# define BYTE_ORDER__BYTE_ORDER
#endif
```
and gives us a working configure.
You could test building this change with Guix + LTO with [this branch](https://github.com/fanquake/bitcoin/tree/lto_in_guix). Note that that build may fail for other reasons (on x86_64), unrelated to this change.
Some related upstream discussion:
https://bugs.gentoo.org/757681https://forums.gentoo.org/viewtopic-t-1013786.html
ACKs for top commit:
hebasto:
re-ACK e838a98475, only [suggested](https://github.com/bitcoin/bitcoin/pull/25697#discussion_r929735675) changes since my recent [review](https://github.com/bitcoin/bitcoin/pull/25697#pullrequestreview-1050657421).
jarolrod:
code review ACK e838a98475
Tree-SHA512: 9dbf64c9bd1fd995a4d1addc011ffeff83d50df736030012346c97605e63aed4b5bac390a81abe646c1be28ad6fd600f64560dcb26bbc2edf5d513ca3b180bfa
fa74e726c4 refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono) (MacroFake)
fa3b3cb9b5 Expose underlying clock in CThreadInterrupt (MacroFake)
Pull request description:
This gets rid of the `value*1000` manual conversion.
ACKs for top commit:
naumenkogs:
utACK fa74e726c4
dergoegge:
Code review ACK fa74e726c4
Tree-SHA512: 90409c05c25f0dd2f1c4dead78f707ebfd78b7d84ea4db9fcefd9c4958a1a3338ac657cd9e99eb8b47d52d4485fa3c947dce4ee1559fb56ae65878685e1ed9a3
fa2247a9f9 refactor: Make CTransaction constructor explicit (MacroFake)
Pull request description:
It involves calculating two hashes, so the performance impact should be
made explicit.
Also, add the module to iwyu.
ACKs for top commit:
aureleoules:
ACK fa2247a9f9.
hebasto:
ACK fa2247a9f9, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: e236c352a472c7edfd4f0319a5a16a59f627b0ab7eb8531b53c75d730a3fa3e990a939978dcd952cd73e647925fc79bfa6d9fd87624bbc3ef180f40f95acef19
The counter is an optimization over calling `ret.empty()`. It was
suggested that the compiler would realize `cnt` is only `0` on the first
iteration, and not actually emit the check and conditional.
This optimization was actually not triggered at all, since we
incremented `cnt` at the beginning of the first iteration. Fix it by
incrementing at the end instead.
This was reported by Github user "Janus".
fa57c449cf fuzz: Remove no-op SetMempoolConstraints (MacroFake)
Pull request description:
Now that the mempool no longer uses the args manager (after commit e4e201dfd9), there is no point setting the mempool limits after it is constructed.
Fix that by setting them once right before the mempool is constructed.
ACKs for top commit:
dongcarl:
utACK fa57c449cf
glozow:
utACK fa57c449cf
Tree-SHA512: d236f9cdcee8c2076272b82c97f8a5942f1ecf119ab36edafd42088ef97554592348a61e1fbe504fd52b30301ef0177813042599ad12e8cb95b4a20586c85bb0
fa28d0f3c3 scripted-diff: Replace NullUniValue with UniValue::VNULL (MacroFake)
fa962103e8 fuzz: refactor: Replace NullUniValue with UniValue{} (MacroFake)
Pull request description:
This refactor is needed to disable the (potentially expensive for large json) UniValue copy constructors.
ACKs for top commit:
fanquake:
ACK fa28d0f3c3
Tree-SHA512: 7d4204cce0a6fc4ecda96973de77d15b7e4c7caa3e0e890e1f5b9a4b9ace8b240b1f7565d6ab586e168a5fa1201b6c60a924868ef34d6abfbfd8ab7f0f99fbc7
850b0850cc fix comment spellings from the codespell lint (Greg Weber)
Pull request description:
test/lint/all-lint.py includes the codespell lint
ACKs for top commit:
aureleoules:
ACK 850b0850cc.
Tree-SHA512: bf63690da2652886e705d6594903bab67ff0f35a0e5a5505f063827f5148ebce47681e541cbe0e52396baf1addb25d9fe50e5faa9176456f579a7cd2f1321c44
