ff6549c3c8 fix: update rest info on block size and json (Chris Abrams)
Pull request description:
Addressing the ambiguous block size text in rest docs: https://github.com/bitcoin/bitcoin/issues/18703
Also makes sure to let developers know there is `.json` option for the rest output format.
ACKs for top commit:
MarcoFalke:
ACK ff6549c3c8
promag:
ACK ff6549c3c8.
Tree-SHA512: 9ef93c1432d650b1f9599778ba092c1ca5b084a537af257078e1c713c76c5d3a4cc4b1ede8a2489964be8ed0303ad8bea58c1cb4759bbb9b24dbdebfec8001d3
c31cbe7cfe Add C++17 test to Travis (Pieter Wuille)
7829685e27 Add configure option for c++17 (Pieter Wuille)
0fbde488b2 Support conversion between Spans of compatible types (Pieter Wuille)
7cbfebbf3d Update ax_cxx_compile_stdcxx.m4 (Pieter Wuille)
Pull request description:
This adds a `--enable-c++17` option to the configure script, fixes the only C++17 incompatibility (with a commit taken from #18468), and adds a Travis test for it.
This is all off by default, and release builds remain C++11.
It implements the first step of the plan in https://github.com/bitcoin/bitcoin/issues/16684.
ACKs for top commit:
elichai:
tACK c31cbe7cfe
practicalswift:
Tested ACK c31cbe7cfe
hebasto:
ACK c31cbe7cfe, tested on Linux Mint 19.3 both C++11 and C++17 modes. Compiled and passed tests locally.
Tree-SHA512: a4b00776dbceef9c12abbb404c6bcd48f7916ce24c8c7a14116355f64e817578b7fcddbedd5ce435322319d1e4de43429b68553f4d96d970c308fe3e3e59b9d1
182dbdf0f4 util: Detect posix_fallocate() instead of assuming (Vasil Dimov)
Pull request description:
Don't assume that `posix_fallocate()` is available on Linux and not
available on other operating systems. At least FreeBSD has it and we
are not using it.
Properly check whether `posix_fallocate()` is present and use it if it
is.
ACKs for top commit:
laanwj:
ACK 182dbdf0f4
Tree-SHA512: f9ed4bd661f33ff6b2b1150591e860b3c1f44e12b87c35e870d06a7013c4e841ed2bf17b41ad6b18fe471b0b23a4b5e42cf1400637180888e0bc56c254fe0766
cd543d9193 test: check misbehavior more independently in p2p_filter.py (Danny Lee)
Pull request description:
This expands on #18672 in two ways:
- Check positive cases (`filterload` accepted, `filteradd` accepted) in addition to the negative cases added in #18672
- Address MarcoFalke 's [suggestion](https://github.com/bitcoin/bitcoin/pull/18672#discussion_r412101752) to successfully load a filter before testing `filteradd`
ACKs for top commit:
theStack:
re-ACK cd543d9193
Tree-SHA512: f82402f6287ccddf08b38b6432d5e2b2b2ef528802a981d04c24bac459022f732d9090d4849d72d3d1eb2c757161dcb18c4c036b6e11dc80114e9cd49f21c3bd
32b6b386a5 tests: Sort fuzzing harnesses (practicalswift)
e1e181fad1 tests: Add fuzzing coverage for JSONRPCTransactionError(...) and RPCErrorFromTransactionError(...) (practicalswift)
103b6ecce0 tests: Add fuzzing coverage for TransactionErrorString(...) (practicalswift)
dde508b8b0 tests: Add fuzzing coverage for ParseFixedPoint(...) (practicalswift)
1532259fca tests: Add fuzzing coverage for FormatHDKeypath(...) and WriteHDKeypath(...) (practicalswift)
90b635e84e tests: Add fuzzing coverage for CHECK_NONFATAL(...) (practicalswift)
a4e3d13df6 tests: Add fuzzing coverage for StringForFeeReason(...) (practicalswift)
a19598cf98 tests: Add fuzzing harness for functions in system.h (ArgsManager) (practicalswift)
Pull request description:
Add fuzzing harnesses for various classes/functions in `util/`.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
Top commit has no ACKs.
Tree-SHA512: d27947220850c2a202c7740f44140c17545f45522596912452ccab0c2f5379abeb07cc769982c7855cb465059425206371a2b75ee1c285b03984161c9619d0b0
7918c1b019 test: Add CreateWalletFromFile test (Russell Yanofsky)
Pull request description:
Add unit test calling CreateWalletFromFile, which isn't currently called from other unit tests, with some basic checks to make sure it rescans and registers for notifications correctly.
Motivation for this change was to try to write a test that would fail without the early `handleNotifications` call in ef8c6ca60767cac589d98ca57ee33179608ccda8 from https://github.com/bitcoin/bitcoin/pull/16426, but succeed with it:
ef8c6ca607/src/wallet/wallet.cpp (L3978-L3986)
However, writing a full test for the race condition that call prevents isn't possible without the locking changes from #16426. So this PR just adds as much test coverage as is possible now.
This new test is also useful for https://github.com/bitcoin/bitcoin/pull/15719, since it detects the stale notifications.transactionAddedToMempool notifications that PR eliminates.
ACKs for top commit:
MarcoFalke:
ACK 7918c1b019
jonatack:
ACK 7918c1b019
Tree-SHA512: 44035aee698ecb722c6039d061d8fac2011e9da0b314e4aff19be1d610b53cacff99016b34d6b84669bb3b61041b2318d9d8e3363658f087802ae4aa36ca17b8
8098dea069 test: Add mempool_updatefromblock.py (Hennadii Stepanov)
Pull request description:
This PR adds a new test for mempool update of transaction descendants/ancestors information (count, size) when transactions have been re-added from a disconnected block to the mempool.
It could be helpful for working on PRs like #17925, #18191.
ACKs for top commit:
ariard:
ACK 8098dea
Tree-SHA512: 7e808fa8df8d7d7a7dbdc3f79361049b49c7bce9b58fd5539b28c9636bedac747695537e500d7ed68dc8bdb80167ad3f1c01086f7551691405d2ba2e38ef1d06
45615de26c ci: Fix default retry script usage (Hennadii Stepanov)
Pull request description:
On master (5352d14b37) `CI_RETRY_EXE=${CI_RETRY_EXE:retry}` works as a [Substring Expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html), and that is wrong.
If `CI_RETRY_EXE` variable was unset initially, its new value becomes an empty string, but not "retry" as one could expect. Consequently, the `${CI_RETRY_EXE} ...` command does _not_ use `ci/retry/retry` script.
This PR makes for `CI_RETRY_EXE` variable a usual parameter expansion, i.e., `${parameter:-word}`.
Reference: https://github.com/bitcoin/bitcoin/pull/18735#issuecomment-620095489
Top commit has no ACKs.
Tree-SHA512: 108173f6b2677979b9ddf2f9b9df4a6c56f5efa81c36543a1816bb3b984e42984bf3c83fe413ea3a5ca1e2317c4efb02fea7180a6b44863af7cfe6202e9cf94d
66fe7b1a98 test: added test for upgradewallet RPC (Harris)
Pull request description:
This PR adds tests for the newly merged *upgradewallet* RPC.
Additionally, it expands `test_framework/util.py` by adding the function `adjust_bitcoin_conf_for_pre_17` to support nodes that don't parse configuration sections.
This test uses two older node versions, v0.15.2 and v0.16.3, to create older wallet versions to be used by `upgradewallet`.
Fixes https://github.com/bitcoin/bitcoin/issues/18767
Top commit has no ACKs.
Tree-SHA512: bb72ff1e829e2c3954386cc308842820ef0828a4fbb754202b225a8748f92d4dcc5ec77fb146bfd5484a5c2f29ce95adf9f3fb4483437088ff3ea4a8d2c442c1
fabe44e815 bench: Start nodes with -nodebuglogfile (MarcoFalke)
Pull request description:
For benchmarking we don't want to depend on the speed of the disk or the amount of debug logging
ACKs for top commit:
fanquake:
ACK fabe44e815 - This makes some of these benchmarks significantly faster to run. MempoolEviction total runtime is down from ~46s to 11s on my machine:
Tree-SHA512: d99700901650325896b9115d20b84a27042152f46266f595bf7ea1414528c0b346f4e707a12ee8b8ba99c35cf155e645e67971c1b2a679c4e609c400ff8b08ae
fcb7261625 Prevent valgrind false positive in rest_blockhash_by_height (Russell Yanofsky)
Pull request description:
A bad interaction between valgrind and clang 6.0.0-1ubuntu2 with -O2 optimizations makes valgrind misleadingly imply C++ code is reading an uninitialized blockheight value in `rest_blockhash_by_height` just because that's what clang optimized code is doing. The C++ code looks like:
```c++
int32_t blockheight;
if (!ParseInt32(height_str, &blockheight) || blockheight < 0) {
```
while the optimized code looks like:
```
0x00000000000f97ab <+123>: callq 0x4f8860 <ParseInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*)>
0x00000000000f97b0 <+128>: mov 0xc(%rsp),%ebx
0x00000000000f97b4 <+132>: test %ebx,%ebx
0x00000000000f97b6 <+134>: js 0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
0x00000000000f97bc <+140>: xor $0x1,%al
0x00000000000f97be <+142>: jne 0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
```
During the rest_interface.py test:
eef90c14ed/test/functional/interface_rest.py (L266)
when `height_str` is empty, `ParseInt32` returns false and `blockheight` value is never assigned. The optimized code reads the uninitialized `blockheight` value in `0xc(%rsp)` before the checking the `ParseInt32` return value in `%al`, which is harmless, but triggers the following error from valgrind:
```
==30660== Thread 13 b-httpworker.2:
==30660== Conditional jump or move depends on uninitialised value(s)
==30660== at 0x2017B6: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:614)
==30660== by 0x2041B9: operator() (rest.cpp:670)
==30660== by 0x2041B9: std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartREST(util::Ref const&)::$_1>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (std_function.h:301)
==30660== by 0x3EC994: operator() (std_function.h:706)
==30660== by 0x3EC994: HTTPWorkItem::operator()() (httpserver.cpp:55)
==30660== by 0x3ED16D: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:114)
==30660== by 0x3E9168: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:342)
==30660== by 0x3EDAAA: __invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:60)
==30660== by 0x3EDAAA: __invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:95)
==30660== by 0x3EDAAA: _M_invoke<0, 1, 2> (thread:234)
==30660== by 0x3EDAAA: operator() (thread:243)
==30660== by 0x3EDAAA: std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run() (thread:186)
==30660== by 0x64256DE: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
==30660== by 0x54876DA: start_thread (pthread_create.c:463)
==30660== by 0x6DC888E: clone (clone.S:95)
==30660== Uninitialised value was created by a stack allocation
==30660== at 0x20173A: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:608)
==30660==
{
<insert_a_suppression_name_here>
Memcheck:Cond
fun:_ZL24rest_blockhash_by_heightRKN4util3RefEP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
fun:operator()
fun:_ZNSt17_Function_handlerIFbP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEZ9StartRESTRKN4util3RefEE3$_1E9_M_invokeERKSt9_Any_dataOS1_S9_
fun:operator()
fun:_ZN12HTTPWorkItemclEv
fun:_ZN9WorkQueueI11HTTPClosureE3RunEv
fun:_ZL16HTTPWorkQueueRunP9WorkQueueI11HTTPClosureEi
fun:__invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
fun:__invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
fun:_M_invoke<0, 1, 2>
fun:operator()
fun:_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvP9WorkQueueI11HTTPClosureEiES6_iEEEEE6_M_runEv
obj:/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
fun:start_thread
fun:clone
}
```
This is a known bad interaction between clang and valgrind. The clang optimized code is correct but valgrind has no way of knowing that accessing the uninitialized value isn't a problem. Issue has been reported previously:
- https://bugs.llvm.org/show_bug.cgi?id=32604#c4
- https://github.com/Z3Prover/z3/issues/972
This commit just sets blockheight to -1 as a workaround.
This change was originally made in 41d5d651594c6c939add7a58b7e30c97dccdf24a from #18740 to fix the travis error there (https://travis-ci.org/github/bitcoin/bitcoin/jobs/678453061#L7157) but MarcoFalke suggested https://github.com/bitcoin/bitcoin/pull/18740#discussion_r414772851 moving to a new PR, since apparently the error's been seen on travis previously
ACKs for top commit:
MarcoFalke:
ACK fcb7261625
practicalswift:
ACK fcb7261625
Tree-SHA512: ec8abf45bd3d6c6e0e7e404d0b2a749efd43910619b84b0b5fe7dab22881598d1011a0f3ff2e146bf46320b63eb152bf63c62c06f1ab84c35dd640abc468f18f
50fc4df6c4 [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a178536 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502472 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48dc5 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f20a [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eecce3 [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a333 [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)
Pull request description:
This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.
The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.
This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.
For privacy improvements around # 1, please see #16698.
Thank you to gmaxwell for the idea of how to break out this subset of functionality (https://github.com/bitcoin/bitcoin/pull/16698#issuecomment-571399346)
ACKs for top commit:
fjahr:
Code review ACK 50fc4df6c4
MarcoFalke:
ACK 50fc4df6c4, I think this is ready for merge now 👻
amitiuttarwar:
The current tip `50fc4df` currently has 6 ACKs on it, so I've opened #18807 to address the last bits.
jnewbery:
utACK 50fc4df6c4.
ariard:
Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
robot-visions:
ACK 50fc4df6c4
sipa:
utACK 50fc4df6c4
naumenkogs:
utACK 50fc4df
Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
As per BIP 141, if there is more than 1 pubkey that matches the witness
commitment structure, the one with the highest output index should be
chosen. This adds a sanity check that we are doing that, which will fail
if anyone trys to "optimise" GetWitnessCommitmentIndex() be returning
early.
Per BIP 141, the witness commitment structure is atleast 38 bytes,
OP_RETURN (0x6a) + 36 (0x24) + 4 byte header (0xaa21a9ed) + 32 byte
SHA256 hash. It can be longer, however any additional data has no
consensus meaning.
7b2b06dfe3 tests: Add missing sync_all to wallet_importdescriptors.py (Andrew Chow)
Pull request description:
node1 will sometimes do sendtoaddress before it has received a funding transaction which will cause the test to fail. sync_all to ensure it gets the transaction first.
Fixes#18800
ACKs for top commit:
instagibbs:
ACK 7b2b06d The wallet endpoint right after is indeed node 1.
Tree-SHA512: b610a771d062b5f955cd70b34337577a1ab8dacbf4be20aa74e1e8234495b0be9faff138eb1713f29decb5574446e0583e221bc2c9a6eea13611b422ea3a296a
fa301fec96 test: Fix wallet_bumpfee intermittent error (MarcoFalke)
Pull request description:
Remove incorrect and undocumented `connect_nodes(self.nodes[0], 1)`.
Issue is that transactions are re-relayed (going full circle) between the two nodes, that have two connections between each other.
https://travis-ci.org/github/bitcoin/bitcoin/jobs/679201559#L6992
Also fix some pep8 while touching the file
This bug has been introduced by accident in c1dde3a949
ACKs for top commit:
achow101:
ACK fa301fec96
Tree-SHA512: a6565ca30dbe44b02e3f58f159d2515c2ea4a74030118fafc1a3391ce980a4b6d4505dcf51315fda24843f72550a7dea7407b877b3b796883dd73d3b6f009e6f
node1 will sometimes do sendtoaddress before it has received a funding
transaction which will cause the test to fail. sync_all to ensure it
gets the transaction first.
fae49f6e42 ci: Add and document BASE_BUILD_DIR (MarcoFalke)
Pull request description:
Also fixes#18768
ACKs for top commit:
hebasto:
re-ACK fae49f6e42, which is essentially the same as the previously [reviewed changes](https://github.com/bitcoin/bitcoin/pull/18735#pullrequestreview-400581536).
Tree-SHA512: 216565a05ccd513dd9f114b2333d3c283fd71914d32f9b05f145cb7c70633b083ff8ef60798d6f22f4be6a4d652b03806551fd74b5b596c92968501a4d9726d2
Previously, we would specify the makensis output file path twice:
1. At the top of Makefile.am as BITCOIN_WIN_INSTALLER, and
2. In share/setup.nsi.ini
This commit uses the -X flag of makensis to eliminate the need for the
second instance mentioned above, referring makensis directly to the
value of BITCOIN_WIN_INSTALLER
When using worktrees or submodules, you'll see a `.git' plain text file
at the root of your working tree instead of the usual `.git' directory.
This plain text file will point to the real GIT_DIR, under the
GIT_COMMON_DIR. From experimentation, the full GIT_COMMON_DIR is
required to exist for operations such as git-archive(1), so we expose it
as readonly inside the container.
Previously, the sourced script would create the source tarball. Now, it
only assigns variables and the source-ing script has more flexibility in
determining what to do with these variables.
See later commit showing how this flexibility is useful in our Guix
builds.
bda62e87e6 Adding build instructions to Bitcoin Core, fixes#18658 (Saahil Shangle)
Pull request description:
Making the instructions for building Bitcoin Core more clear in the main `README.md` will reduce confusion between the `build_msvc` and `doc` folders.
ACKs for top commit:
laanwj:
ACK bda62e87e6
Tree-SHA512: ee4c394661eba48d4229e3d1e9ddb67ccb79589429bfa9986cb0242cd615d1f3cc5332063562c1e89c0cdd9ae2e609f61e8bfb209926d8363d35d3da6d94ae9c
fa72a75102 ci: Document why tests can not be run on mac (MarcoFalke)
Pull request description:
Fixes#18794
Top commit has no ACKs.
Tree-SHA512: 297652eda412aa8cf7255e20a6f294d22773dad8637a3d7b5204f3b638e911ce5b2e40e85f81395a34c1b5a5b497665944c2d6ea17c70c30c0c9e0ab553f956e
cd24f37ea9 doc: Better explain GNU ld's dislike of ld64's options (fanquake)
Pull request description:
There's also now more than a single option being special cased for
darwin. If we didn't special case these options they would still end
up on the link line and the binaries produced would just segfault.
I'm going to plug #17874 here as well, because adding
`-fatal-warnings` to our `AX_CHECK_LINK_FLAG` calls would
mostly prevent this sort of option mangling from happening.
An example of the warning behaviour:
```bash
echo "int main() {}" | g++ -x c++ -std=c++11 -Wl,-dead_strip -
/usr/bin/ld: warning: cannot find entry symbol ad_strip; defaulting to 0000000000001040
nm -C a.out
0000000000001000 t _init
0000000000001040 T _start
U ad_strip
```
ACKs for top commit:
dongcarl:
ACK cd24f37ea9
Tree-SHA512: 8c5ff11b647e7d44dbb3f509a07caf8606a6b481c114403f0de72b3ad65395dbe9a3436e731ae1b46a823431ed23c3c6aacab8942d78629d59cd8c258c5dbf02
2aa48edec0 refactor: Drop unused ${WRAP_DIR}/${HOST} directory (Hennadii Stepanov)
1362be0447 build: Drop make dist in gitian builds (Hennadii Stepanov)
Pull request description:
After the merge of #18331, the packaged source tarball is created by `git archive`, but the binaries are built from another one which is made by `make dist`.
With this PR the only source tarball, created by `git archive`, is used both for binaries building and for packaging to users.
Close#16588.
Close#18547.
As a good side-effect, #18349 becomes redundant.
**Change in behavior**
The following variables 1b151e3ffc/configure.ac (L2-L6)
are no longer used for naming of directories and tarballs.
Instead of them the gitian descriptors use a git tag (if available) or a commit hash.
---
Also a small refactor commit picked from #18404.
ACKs for top commit:
dongcarl:
ACK 2aa48edec0
MarcoFalke:
ACK 2aa48edec0
fanquake:
ACK 2aa48edec0 - I've had a quick look over this, and don't want to block merging if this actually gets as closer to finally having this all sorted out. Obviously we've still got #18741, and after speaking to Carl this morning, there will likely be even more changes after that (not Guix specific).
Tree-SHA512: d3b16f87e48d1790a3264940c28acd5d881bfd10f3ce94fb0c8a6af76d8039289d01e0cd4972adac49ae24362857251f6c1e5e09e3e9fbf636c10708b4015a7c
3e38023af7 scripts: add PE .reloc section check to security-check.py (fanquake)
Pull request description:
The `ld` in binutils has historically had a few issues with PE binaries, there's a good summary in this [thread](https://sourceware.org/bugzilla/show_bug.cgi?id=19011).
One issue in particular was `ld` stripping the `.reloc` section out of PE binaries, even though it's required for functioning ASLR. This was [reported by a Tor developer in 2014](https://sourceware.org/bugzilla/show_bug.cgi?id=17321) and they have been patching their [own binutils](https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/binutils) ever since. However their patch only made it into binutils at the [start of this year](https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=dc9bd8c92af67947db44b3cb428c050259b15cd0). It adds an `--enable-reloc-section` flag, which is turned on by default if you are using `--dynamic-base`. In the mean time this issue has also been worked around by other projects, such as FFmpeg, see [this commit](91b668acd6).
I have checked our recent supported Windows release binaries, and they do contain a `.reloc` section. From what I understand, we are using all the right compile/linker flags, including `-pie` & `-fPIE`, and have never run into the crashing/entrypoint issues that other projects might have seen.
One other thing worth noting here, it how Debian/Ubuntu patch the binutils that they distribute, because that's what we end up using in our gitian builds.
In the binutils-mingw-w64 in Bionic (18.04), which we currently use in gitian, PE hardening options/security flags are enabled by default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8ubuntu1/changelog) and the [relevant commit](452b3013b8).
However in Focal (20.04), this has now been reversed. PE hardening options are no-longer the default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8.8/changelog) and [relevant commit](7bd8b2fbc2), which cites same .reloc issue mentioned here.
Given that we explicitly specify/opt-in to everything that we want to use, the defaults aren't necessarily an issue for us. However I think it highlights the importance of continuing to be explicit about what we want, and not falling-back or relying on upstream.
This was also prompted by the possibility of us doing link time garbage collection, see #18579 & #18605. It seemed some sanity checks would be worthwhile in-case the linker goes haywire while garbage collecting.
I think Guix is going to bring great benefits when dealing with these kinds of issues. Carl you might have something to say in that regard.
ACKs for top commit:
dongcarl:
ACK 3e38023af7
Tree-SHA512: af14d63bdb334bde548dd7de3e0946556b7e2598d817b56eb4e75b3f56c705c26aa85dd9783134c4b6a7aeb7cb4de567eed996e94d533d31511f57ed332287da
fac24dea00 ci: Run functional tests on mac again (MarcoFalke)
Pull request description:
ACKs for top commit:
hebasto:
ACK fac24dea00, verified travis build log.
Tree-SHA512: 406282a7ac03e5c193830b727366c7b1350639f1850aff951bf7ddd4b0c3e3ffb396b950ccb3a64ddc59500fa2739766f3c34806b4d144bc4535bb2bd765b959
fa501700e9 wallet: Recommned absolute path for dumpwallet (MarcoFalke)
Pull request description:
Avoids misunderstandings such as #9564
ACKs for top commit:
kristapsk:
utACK fa501700e9
Tree-SHA512: f675ef607992857ffeb556a2945b5436a70b39c5d83f05a8be15a6fccc84cbe9d03e52f8239e28d159e41ed7c6f119b7a38e8ab327029f04609f63c559c12c49
223588b1bb Add a --descriptors option to various tests (Andrew Chow)
869f7ab30a tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow)
cf06062859 Correctly check for default wallet (Andrew Chow)
886e0d75f5 Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow)
3c19fdd2a2 Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow)
388ba94231 Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow)
1346e14831 Functional tests for descriptor wallets (Andrew Chow)
f193ea889d add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen)
ce24a94494 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow)
1cb42b22b1 Generate new descriptors when encrypting (Andrew Chow)
82ae02b165 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow)
b713baa75a Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow)
8b9603bd0b Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow)
72a9540df9 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow)
84b4978c02 Implement SignMessage for descriptor wallets (Andrew Chow)
bde7c9fa38 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow)
d50c8ddd41 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow)
f1ca5feb4a Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow)
586b57a9a6 Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow)
f866957979 Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow)
a775f7c7fd Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow)
bfdd073486 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow)
58c7651821 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow)
e014886a34 Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow)
46dfb99768 Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow)
4cb9b69be0 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow)
d1ec3e4f19 Add IsSingleType to Descriptors (Andrew Chow)
953feb3d27 Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow)
2363e9fcaa Load the descriptor cache from the wallet file (Andrew Chow)
46c46aebb7 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow)
ec2f9e1178 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow)
741122d4c1 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow)
2db7ca765c Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow)
db7177af8c Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow)
78f8a92910 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow)
834de0300c Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow)
d8132669e1 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow)
3194a7f88a Introduce WalletDescriptor class (Andrew Chow)
6b13cd3fa8 Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow)
aeac157c9d Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow)
96accc73f0 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow)
6b8119af53 Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow)
06620302c7 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow)
Pull request description:
Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`.
Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each.
Descriptors can also be imported with a new `importdescriptors` RPC.
Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things.
A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`).
ACKs for top commit:
Sjors:
utACK 223588b1bb (rebased, nits addressed)
jonatack:
Code review re-ACK 223588b1bb.
fjahr:
re-ACK 223588b1bb
instagibbs:
light re-ACK 223588b
meshcollider:
Code review ACK 223588b1bb
Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986
