7918c1b019 test: Add CreateWalletFromFile test (Russell Yanofsky)
Pull request description:
Add unit test calling CreateWalletFromFile, which isn't currently called from other unit tests, with some basic checks to make sure it rescans and registers for notifications correctly.
Motivation for this change was to try to write a test that would fail without the early `handleNotifications` call in ef8c6ca60767cac589d98ca57ee33179608ccda8 from https://github.com/bitcoin/bitcoin/pull/16426, but succeed with it:
ef8c6ca607/src/wallet/wallet.cpp (L3978-L3986)
However, writing a full test for the race condition that call prevents isn't possible without the locking changes from #16426. So this PR just adds as much test coverage as is possible now.
This new test is also useful for https://github.com/bitcoin/bitcoin/pull/15719, since it detects the stale notifications.transactionAddedToMempool notifications that PR eliminates.
ACKs for top commit:
MarcoFalke:
ACK 7918c1b019
jonatack:
ACK 7918c1b019
Tree-SHA512: 44035aee698ecb722c6039d061d8fac2011e9da0b314e4aff19be1d610b53cacff99016b34d6b84669bb3b61041b2318d9d8e3363658f087802ae4aa36ca17b8
8098dea069 test: Add mempool_updatefromblock.py (Hennadii Stepanov)
Pull request description:
This PR adds a new test for mempool update of transaction descendants/ancestors information (count, size) when transactions have been re-added from a disconnected block to the mempool.
It could be helpful for working on PRs like #17925, #18191.
ACKs for top commit:
ariard:
ACK 8098dea
Tree-SHA512: 7e808fa8df8d7d7a7dbdc3f79361049b49c7bce9b58fd5539b28c9636bedac747695537e500d7ed68dc8bdb80167ad3f1c01086f7551691405d2ba2e38ef1d06
45615de26c ci: Fix default retry script usage (Hennadii Stepanov)
Pull request description:
On master (5352d14b37) `CI_RETRY_EXE=${CI_RETRY_EXE:retry}` works as a [Substring Expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html), and that is wrong.
If `CI_RETRY_EXE` variable was unset initially, its new value becomes an empty string, but not "retry" as one could expect. Consequently, the `${CI_RETRY_EXE} ...` command does _not_ use `ci/retry/retry` script.
This PR makes for `CI_RETRY_EXE` variable a usual parameter expansion, i.e., `${parameter:-word}`.
Reference: https://github.com/bitcoin/bitcoin/pull/18735#issuecomment-620095489
Top commit has no ACKs.
Tree-SHA512: 108173f6b2677979b9ddf2f9b9df4a6c56f5efa81c36543a1816bb3b984e42984bf3c83fe413ea3a5ca1e2317c4efb02fea7180a6b44863af7cfe6202e9cf94d
66fe7b1a98 test: added test for upgradewallet RPC (Harris)
Pull request description:
This PR adds tests for the newly merged *upgradewallet* RPC.
Additionally, it expands `test_framework/util.py` by adding the function `adjust_bitcoin_conf_for_pre_17` to support nodes that don't parse configuration sections.
This test uses two older node versions, v0.15.2 and v0.16.3, to create older wallet versions to be used by `upgradewallet`.
Fixes https://github.com/bitcoin/bitcoin/issues/18767
Top commit has no ACKs.
Tree-SHA512: bb72ff1e829e2c3954386cc308842820ef0828a4fbb754202b225a8748f92d4dcc5ec77fb146bfd5484a5c2f29ce95adf9f3fb4483437088ff3ea4a8d2c442c1
fabe44e815 bench: Start nodes with -nodebuglogfile (MarcoFalke)
Pull request description:
For benchmarking we don't want to depend on the speed of the disk or the amount of debug logging
ACKs for top commit:
fanquake:
ACK fabe44e815 - This makes some of these benchmarks significantly faster to run. MempoolEviction total runtime is down from ~46s to 11s on my machine:
Tree-SHA512: d99700901650325896b9115d20b84a27042152f46266f595bf7ea1414528c0b346f4e707a12ee8b8ba99c35cf155e645e67971c1b2a679c4e609c400ff8b08ae
fcb7261625 Prevent valgrind false positive in rest_blockhash_by_height (Russell Yanofsky)
Pull request description:
A bad interaction between valgrind and clang 6.0.0-1ubuntu2 with -O2 optimizations makes valgrind misleadingly imply C++ code is reading an uninitialized blockheight value in `rest_blockhash_by_height` just because that's what clang optimized code is doing. The C++ code looks like:
```c++
int32_t blockheight;
if (!ParseInt32(height_str, &blockheight) || blockheight < 0) {
```
while the optimized code looks like:
```
0x00000000000f97ab <+123>: callq 0x4f8860 <ParseInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*)>
0x00000000000f97b0 <+128>: mov 0xc(%rsp),%ebx
0x00000000000f97b4 <+132>: test %ebx,%ebx
0x00000000000f97b6 <+134>: js 0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
0x00000000000f97bc <+140>: xor $0x1,%al
0x00000000000f97be <+142>: jne 0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
```
During the rest_interface.py test:
eef90c14ed/test/functional/interface_rest.py (L266)
when `height_str` is empty, `ParseInt32` returns false and `blockheight` value is never assigned. The optimized code reads the uninitialized `blockheight` value in `0xc(%rsp)` before the checking the `ParseInt32` return value in `%al`, which is harmless, but triggers the following error from valgrind:
```
==30660== Thread 13 b-httpworker.2:
==30660== Conditional jump or move depends on uninitialised value(s)
==30660== at 0x2017B6: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:614)
==30660== by 0x2041B9: operator() (rest.cpp:670)
==30660== by 0x2041B9: std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartREST(util::Ref const&)::$_1>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (std_function.h:301)
==30660== by 0x3EC994: operator() (std_function.h:706)
==30660== by 0x3EC994: HTTPWorkItem::operator()() (httpserver.cpp:55)
==30660== by 0x3ED16D: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:114)
==30660== by 0x3E9168: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:342)
==30660== by 0x3EDAAA: __invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:60)
==30660== by 0x3EDAAA: __invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:95)
==30660== by 0x3EDAAA: _M_invoke<0, 1, 2> (thread:234)
==30660== by 0x3EDAAA: operator() (thread:243)
==30660== by 0x3EDAAA: std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run() (thread:186)
==30660== by 0x64256DE: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
==30660== by 0x54876DA: start_thread (pthread_create.c:463)
==30660== by 0x6DC888E: clone (clone.S:95)
==30660== Uninitialised value was created by a stack allocation
==30660== at 0x20173A: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:608)
==30660==
{
<insert_a_suppression_name_here>
Memcheck:Cond
fun:_ZL24rest_blockhash_by_heightRKN4util3RefEP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
fun:operator()
fun:_ZNSt17_Function_handlerIFbP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEZ9StartRESTRKN4util3RefEE3$_1E9_M_invokeERKSt9_Any_dataOS1_S9_
fun:operator()
fun:_ZN12HTTPWorkItemclEv
fun:_ZN9WorkQueueI11HTTPClosureE3RunEv
fun:_ZL16HTTPWorkQueueRunP9WorkQueueI11HTTPClosureEi
fun:__invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
fun:__invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
fun:_M_invoke<0, 1, 2>
fun:operator()
fun:_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvP9WorkQueueI11HTTPClosureEiES6_iEEEEE6_M_runEv
obj:/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
fun:start_thread
fun:clone
}
```
This is a known bad interaction between clang and valgrind. The clang optimized code is correct but valgrind has no way of knowing that accessing the uninitialized value isn't a problem. Issue has been reported previously:
- https://bugs.llvm.org/show_bug.cgi?id=32604#c4
- https://github.com/Z3Prover/z3/issues/972
This commit just sets blockheight to -1 as a workaround.
This change was originally made in 41d5d651594c6c939add7a58b7e30c97dccdf24a from #18740 to fix the travis error there (https://travis-ci.org/github/bitcoin/bitcoin/jobs/678453061#L7157) but MarcoFalke suggested https://github.com/bitcoin/bitcoin/pull/18740#discussion_r414772851 moving to a new PR, since apparently the error's been seen on travis previously
ACKs for top commit:
MarcoFalke:
ACK fcb7261625
practicalswift:
ACK fcb7261625
Tree-SHA512: ec8abf45bd3d6c6e0e7e404d0b2a749efd43910619b84b0b5fe7dab22881598d1011a0f3ff2e146bf46320b63eb152bf63c62c06f1ab84c35dd640abc468f18f
50fc4df6c4 [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a178536 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502472 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48dc5 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f20a [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eecce3 [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a333 [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)
Pull request description:
This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.
The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.
This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.
For privacy improvements around # 1, please see #16698.
Thank you to gmaxwell for the idea of how to break out this subset of functionality (https://github.com/bitcoin/bitcoin/pull/16698#issuecomment-571399346)
ACKs for top commit:
fjahr:
Code review ACK 50fc4df6c4
MarcoFalke:
ACK 50fc4df6c4, I think this is ready for merge now 👻
amitiuttarwar:
The current tip `50fc4df` currently has 6 ACKs on it, so I've opened #18807 to address the last bits.
jnewbery:
utACK 50fc4df6c4.
ariard:
Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
robot-visions:
ACK 50fc4df6c4
sipa:
utACK 50fc4df6c4
naumenkogs:
utACK 50fc4df
Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
7b2b06dfe3 tests: Add missing sync_all to wallet_importdescriptors.py (Andrew Chow)
Pull request description:
node1 will sometimes do sendtoaddress before it has received a funding transaction which will cause the test to fail. sync_all to ensure it gets the transaction first.
Fixes#18800
ACKs for top commit:
instagibbs:
ACK 7b2b06d The wallet endpoint right after is indeed node 1.
Tree-SHA512: b610a771d062b5f955cd70b34337577a1ab8dacbf4be20aa74e1e8234495b0be9faff138eb1713f29decb5574446e0583e221bc2c9a6eea13611b422ea3a296a
fa301fec96 test: Fix wallet_bumpfee intermittent error (MarcoFalke)
Pull request description:
Remove incorrect and undocumented `connect_nodes(self.nodes[0], 1)`.
Issue is that transactions are re-relayed (going full circle) between the two nodes, that have two connections between each other.
https://travis-ci.org/github/bitcoin/bitcoin/jobs/679201559#L6992
Also fix some pep8 while touching the file
This bug has been introduced by accident in c1dde3a949
ACKs for top commit:
achow101:
ACK fa301fec96
Tree-SHA512: a6565ca30dbe44b02e3f58f159d2515c2ea4a74030118fafc1a3391ce980a4b6d4505dcf51315fda24843f72550a7dea7407b877b3b796883dd73d3b6f009e6f
node1 will sometimes do sendtoaddress before it has received a funding
transaction which will cause the test to fail. sync_all to ensure it
gets the transaction first.
fae49f6e42 ci: Add and document BASE_BUILD_DIR (MarcoFalke)
Pull request description:
Also fixes#18768
ACKs for top commit:
hebasto:
re-ACK fae49f6e42, which is essentially the same as the previously [reviewed changes](https://github.com/bitcoin/bitcoin/pull/18735#pullrequestreview-400581536).
Tree-SHA512: 216565a05ccd513dd9f114b2333d3c283fd71914d32f9b05f145cb7c70633b083ff8ef60798d6f22f4be6a4d652b03806551fd74b5b596c92968501a4d9726d2
bda62e87e6 Adding build instructions to Bitcoin Core, fixes#18658 (Saahil Shangle)
Pull request description:
Making the instructions for building Bitcoin Core more clear in the main `README.md` will reduce confusion between the `build_msvc` and `doc` folders.
ACKs for top commit:
laanwj:
ACK bda62e87e6
Tree-SHA512: ee4c394661eba48d4229e3d1e9ddb67ccb79589429bfa9986cb0242cd615d1f3cc5332063562c1e89c0cdd9ae2e609f61e8bfb209926d8363d35d3da6d94ae9c
fa72a75102 ci: Document why tests can not be run on mac (MarcoFalke)
Pull request description:
Fixes#18794
Top commit has no ACKs.
Tree-SHA512: 297652eda412aa8cf7255e20a6f294d22773dad8637a3d7b5204f3b638e911ce5b2e40e85f81395a34c1b5a5b497665944c2d6ea17c70c30c0c9e0ab553f956e
cd24f37ea9 doc: Better explain GNU ld's dislike of ld64's options (fanquake)
Pull request description:
There's also now more than a single option being special cased for
darwin. If we didn't special case these options they would still end
up on the link line and the binaries produced would just segfault.
I'm going to plug #17874 here as well, because adding
`-fatal-warnings` to our `AX_CHECK_LINK_FLAG` calls would
mostly prevent this sort of option mangling from happening.
An example of the warning behaviour:
```bash
echo "int main() {}" | g++ -x c++ -std=c++11 -Wl,-dead_strip -
/usr/bin/ld: warning: cannot find entry symbol ad_strip; defaulting to 0000000000001040
nm -C a.out
0000000000001000 t _init
0000000000001040 T _start
U ad_strip
```
ACKs for top commit:
dongcarl:
ACK cd24f37ea9
Tree-SHA512: 8c5ff11b647e7d44dbb3f509a07caf8606a6b481c114403f0de72b3ad65395dbe9a3436e731ae1b46a823431ed23c3c6aacab8942d78629d59cd8c258c5dbf02
2aa48edec0 refactor: Drop unused ${WRAP_DIR}/${HOST} directory (Hennadii Stepanov)
1362be0447 build: Drop make dist in gitian builds (Hennadii Stepanov)
Pull request description:
After the merge of #18331, the packaged source tarball is created by `git archive`, but the binaries are built from another one which is made by `make dist`.
With this PR the only source tarball, created by `git archive`, is used both for binaries building and for packaging to users.
Close#16588.
Close#18547.
As a good side-effect, #18349 becomes redundant.
**Change in behavior**
The following variables 1b151e3ffc/configure.ac (L2-L6)
are no longer used for naming of directories and tarballs.
Instead of them the gitian descriptors use a git tag (if available) or a commit hash.
---
Also a small refactor commit picked from #18404.
ACKs for top commit:
dongcarl:
ACK 2aa48edec0
MarcoFalke:
ACK 2aa48edec0
fanquake:
ACK 2aa48edec0 - I've had a quick look over this, and don't want to block merging if this actually gets as closer to finally having this all sorted out. Obviously we've still got #18741, and after speaking to Carl this morning, there will likely be even more changes after that (not Guix specific).
Tree-SHA512: d3b16f87e48d1790a3264940c28acd5d881bfd10f3ce94fb0c8a6af76d8039289d01e0cd4972adac49ae24362857251f6c1e5e09e3e9fbf636c10708b4015a7c
3e38023af7 scripts: add PE .reloc section check to security-check.py (fanquake)
Pull request description:
The `ld` in binutils has historically had a few issues with PE binaries, there's a good summary in this [thread](https://sourceware.org/bugzilla/show_bug.cgi?id=19011).
One issue in particular was `ld` stripping the `.reloc` section out of PE binaries, even though it's required for functioning ASLR. This was [reported by a Tor developer in 2014](https://sourceware.org/bugzilla/show_bug.cgi?id=17321) and they have been patching their [own binutils](https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/binutils) ever since. However their patch only made it into binutils at the [start of this year](https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=dc9bd8c92af67947db44b3cb428c050259b15cd0). It adds an `--enable-reloc-section` flag, which is turned on by default if you are using `--dynamic-base`. In the mean time this issue has also been worked around by other projects, such as FFmpeg, see [this commit](91b668acd6).
I have checked our recent supported Windows release binaries, and they do contain a `.reloc` section. From what I understand, we are using all the right compile/linker flags, including `-pie` & `-fPIE`, and have never run into the crashing/entrypoint issues that other projects might have seen.
One other thing worth noting here, it how Debian/Ubuntu patch the binutils that they distribute, because that's what we end up using in our gitian builds.
In the binutils-mingw-w64 in Bionic (18.04), which we currently use in gitian, PE hardening options/security flags are enabled by default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8ubuntu1/changelog) and the [relevant commit](452b3013b8).
However in Focal (20.04), this has now been reversed. PE hardening options are no-longer the default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8.8/changelog) and [relevant commit](7bd8b2fbc2), which cites same .reloc issue mentioned here.
Given that we explicitly specify/opt-in to everything that we want to use, the defaults aren't necessarily an issue for us. However I think it highlights the importance of continuing to be explicit about what we want, and not falling-back or relying on upstream.
This was also prompted by the possibility of us doing link time garbage collection, see #18579 & #18605. It seemed some sanity checks would be worthwhile in-case the linker goes haywire while garbage collecting.
I think Guix is going to bring great benefits when dealing with these kinds of issues. Carl you might have something to say in that regard.
ACKs for top commit:
dongcarl:
ACK 3e38023af7
Tree-SHA512: af14d63bdb334bde548dd7de3e0946556b7e2598d817b56eb4e75b3f56c705c26aa85dd9783134c4b6a7aeb7cb4de567eed996e94d533d31511f57ed332287da
fac24dea00 ci: Run functional tests on mac again (MarcoFalke)
Pull request description:
ACKs for top commit:
hebasto:
ACK fac24dea00, verified travis build log.
Tree-SHA512: 406282a7ac03e5c193830b727366c7b1350639f1850aff951bf7ddd4b0c3e3ffb396b950ccb3a64ddc59500fa2739766f3c34806b4d144bc4535bb2bd765b959
fa501700e9 wallet: Recommned absolute path for dumpwallet (MarcoFalke)
Pull request description:
Avoids misunderstandings such as #9564
ACKs for top commit:
kristapsk:
utACK fa501700e9
Tree-SHA512: f675ef607992857ffeb556a2945b5436a70b39c5d83f05a8be15a6fccc84cbe9d03e52f8239e28d159e41ed7c6f119b7a38e8ab327029f04609f63c559c12c49
Add unit test calling CreateWalletFromFile, which isn't currently called from
other unit tests, with some basic checks to make sure it rescans and registers
for notifications correctly.
Motivation for this change was to try to write a test that would fail without
the early `handleNotifications` call in ef8c6ca60767cac589d98ca57ee33179608ccda8
from https://github.com/bitcoin/bitcoin/pull/16426, but succeed with it:
ef8c6ca607/src/wallet/wallet.cpp (L3978-L3986)
However, writing a full test for the race condition that call prevents isn't
possible without the locking changes from #16426. So this PR just adds as much
test coverage as is possible now.
This new test is also useful for https://github.com/bitcoin/bitcoin/pull/15719,
since it detects the stale notifications.transactionAddedToMempool notifications
that PR eliminates.
223588b1bb Add a --descriptors option to various tests (Andrew Chow)
869f7ab30a tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow)
cf06062859 Correctly check for default wallet (Andrew Chow)
886e0d75f5 Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow)
3c19fdd2a2 Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow)
388ba94231 Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow)
1346e14831 Functional tests for descriptor wallets (Andrew Chow)
f193ea889d add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen)
ce24a94494 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow)
1cb42b22b1 Generate new descriptors when encrypting (Andrew Chow)
82ae02b165 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow)
b713baa75a Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow)
8b9603bd0b Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow)
72a9540df9 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow)
84b4978c02 Implement SignMessage for descriptor wallets (Andrew Chow)
bde7c9fa38 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow)
d50c8ddd41 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow)
f1ca5feb4a Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow)
586b57a9a6 Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow)
f866957979 Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow)
a775f7c7fd Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow)
bfdd073486 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow)
58c7651821 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow)
e014886a34 Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow)
46dfb99768 Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow)
4cb9b69be0 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow)
d1ec3e4f19 Add IsSingleType to Descriptors (Andrew Chow)
953feb3d27 Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow)
2363e9fcaa Load the descriptor cache from the wallet file (Andrew Chow)
46c46aebb7 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow)
ec2f9e1178 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow)
741122d4c1 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow)
2db7ca765c Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow)
db7177af8c Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow)
78f8a92910 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow)
834de0300c Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow)
d8132669e1 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow)
3194a7f88a Introduce WalletDescriptor class (Andrew Chow)
6b13cd3fa8 Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow)
aeac157c9d Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow)
96accc73f0 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow)
6b8119af53 Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow)
06620302c7 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow)
Pull request description:
Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`.
Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each.
Descriptors can also be imported with a new `importdescriptors` RPC.
Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things.
A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`).
ACKs for top commit:
Sjors:
utACK 223588b1bb (rebased, nits addressed)
jonatack:
Code review re-ACK 223588b1bb.
fjahr:
re-ACK 223588b1bb
instagibbs:
light re-ACK 223588b
meshcollider:
Code review ACK 223588b1bb
Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986
A bad interaction between valgrind and clang 6.0.0-1ubuntu2 with -O2
optimizations makes valgrind misleadingly imply C++ code is reading an
uninitialized blockheight value in rest_blockhash_by_height just because that's
what clang optimized code is doing. The C++ code looks like:
int32_t blockheight;
if (!ParseInt32(height_str, &blockheight) || blockheight < 0) {
while the optimized code looks like:
0x00000000000f97ab <+123>: callq 0x4f8860 <ParseInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*)>
0x00000000000f97b0 <+128>: mov 0xc(%rsp),%ebx
0x00000000000f97b4 <+132>: test %ebx,%ebx
0x00000000000f97b6 <+134>: js 0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
0x00000000000f97bc <+140>: xor $0x1,%al
0x00000000000f97be <+142>: jne 0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
During the rest_interface.py test:
self.test_rest_request("/blockhashbyheight/", ret_type=RetType.OBJ, status=400)
when height_str is empty, ParseInt32 returns false and blockheight value is
never assigned. The optimized code reads the uninitialized blockheight value
in 0xc(%rsp) before the checking the ParseInt32 return value in %al, which is
harmless, but triggers the following error from valgrind:
==30660== Thread 13 b-httpworker.2:
==30660== Conditional jump or move depends on uninitialised value(s)
==30660== at 0x2017B6: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:614)
==30660== by 0x2041B9: operator() (rest.cpp:670)
==30660== by 0x2041B9: std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartREST(util::Ref const&)::$_1>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (std_function.h:301)
==30660== by 0x3EC994: operator() (std_function.h:706)
==30660== by 0x3EC994: HTTPWorkItem::operator()() (httpserver.cpp:55)
==30660== by 0x3ED16D: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:114)
==30660== by 0x3E9168: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:342)
==30660== by 0x3EDAAA: __invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:60)
==30660== by 0x3EDAAA: __invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:95)
==30660== by 0x3EDAAA: _M_invoke<0, 1, 2> (thread:234)
==30660== by 0x3EDAAA: operator() (thread:243)
==30660== by 0x3EDAAA: std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run() (thread:186)
==30660== by 0x64256DE: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
==30660== by 0x54876DA: start_thread (pthread_create.c:463)
==30660== by 0x6DC888E: clone (clone.S:95)
==30660== Uninitialised value was created by a stack allocation
==30660== at 0x20173A: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:608)
==30660==
{
<insert_a_suppression_name_here>
Memcheck:Cond
fun:_ZL24rest_blockhash_by_heightRKN4util3RefEP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
fun:operator()
fun:_ZNSt17_Function_handlerIFbP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEZ9StartRESTRKN4util3RefEE3$_1E9_M_invokeERKSt9_Any_dataOS1_S9_
fun:operator()
fun:_ZN12HTTPWorkItemclEv
fun:_ZN9WorkQueueI11HTTPClosureE3RunEv
fun:_ZL16HTTPWorkQueueRunP9WorkQueueI11HTTPClosureEi
fun:__invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
fun:__invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
fun:_M_invoke<0, 1, 2>
fun:operator()
fun:_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvP9WorkQueueI11HTTPClosureEiES6_iEEEEE6_M_runEv
obj:/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
fun:start_thread
fun:clone
}
This is a known bad interaction between clang and valgrind. The clang optimized
code is correct but valgrind has no way of knowing that accessing the
uninitialized value isn't a problem. Issue has been reported previously:
https://bugs.llvm.org/show_bug.cgi?id=32604#c4https://github.com/Z3Prover/z3/issues/972
This commit just sets blockheight to 0 as a workaround.
faec063887 log: Use Join() helper when listing log categories (MarcoFalke)
Pull request description:
This removes the global `ListLogCategories` and replaces it with a one-line member function `LogCategoriesString`, which just calls `Join`.
Should be a straightforward refactor to get rid of a few LOC.
ACKs for top commit:
laanwj:
ACK faec063887
promag:
ACK faec063887, I also think it's fine as it is (re https://github.com/bitcoin/bitcoin/pull/18669#discussion_r412944724).
Tree-SHA512: 2f51f9ce1246eda5630015f3a869e36953c7eb34f311baad576b92d7829e4e88051c6189436271cd0a13732a49698506345b446b98fd28e58edfb5b62169f1c9
fa8b9b5d1f test: Fix intermittent failure in wallet_importmulti (MarcoFalke)
Pull request description:
The wallet is async, so after generating a block, we must call `syncwithvalidationinterfacequeue`. Otherwise the timestamp will be of the previous block.
https://travis-ci.org/github/bitcoin/bitcoin/jobs/677685073#L2648
ACKs for top commit:
promag:
ACK fa8b9b5d1f.
Tree-SHA512: c21f9912aabbe22019d4ac9d0da06d6e46ef7f2a84d2781110e04c9836eb0ecf90a22cf2bae7f608be611670d17b20600135d1c5e5404aa1e762839816285fb4
fd8e99da57 tests: Add fuzzing harness for functions in primitives/transaction.h (practicalswift)
d5a31b7cb4 tests: Add fuzzing harness for functions in primitives/block.h (practicalswift)
Pull request description:
Add fuzzing harnesses for various classes/functions in `primitives/`.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
Top commit has no ACKs.
Tree-SHA512: ed54bd5b37ff5e40cfa8d3cd8c65d91a2f64fca87b6a5c3b8ddd6becd876ed172735fb53da4d00a86f318fb94517afd179e07cb28a43edf301ffe4dad703cca4
fa66280396 fuzz: Run in parallel (MarcoFalke)
Pull request description:
Can be reviewed with `--ignore-all-space`
ACKs for top commit:
practicalswift:
ACK fa66280396 -- patch looks correct and the live demo in https://github.com/bitcoin-core/qa-assets/pull/11#issuecomment-615439226 convinced me: this is good!
fanquake:
ACK fa66280396 - this seems sane, and clearly provides some speedup. I'd post benchmarks but can't seem to get through a full run of `./test/fuzz/test_runner.py` without hitting at least a few crashes; see #18763, 18762.
Tree-SHA512: d3b545ca90c75bed27f08fe712399d0ed1ac36b13fb289c83e5606eee8dec4c19f5f5cf91758f0a6b1606d8d6b455fbe46df2588faffe7462b185bd34dc2baaf
bb1ec36fb1 doc: Document how to fuzz Bitcoin Core using honggfuzz (practicalswift)
Pull request description:
Document how to fuzz Bitcoin Core using Honggfuzz.
ACKs for top commit:
fanquake:
ACK bb1ec36fb1 - did a couple quick runs on a severely under powered VM.
Tree-SHA512: 117944c52763a5672f988c62fecb01b85f19f3827fad5582a51464aefdaac4d9a9cd81e2118199f6ea1bb3ab0893c8459ca3d1df7f67bfcf215d5e305225f210
0956e46bff test: use zero-argument super() shortcut (Python 3.0+) (Sebastian Falbesoner)
Pull request description:
This mini-PR replaces all calls to `super(...)` with arguments with the zero-argument shortcut `super()` where applicable. See [PEP 3135](https://www.python.org/dev/peps/pep-3135/#specification):
> The new syntax:
>
> super()
>
> is equivalent to:
>
> super(__class__, <firstarg>)
>
> where __class__ is the class that the method was defined in, and <firstarg> is
> the first parameter of the method (normally self for instance methods, and cls
> for class methods).
ACKs for top commit:
fanquake:
ACK 0956e46bff
Tree-SHA512: 4ac66fe7ab2be2e8a514e5fcfc41dbb298f21b23ebb7b7b0310d704b0b3cef8adf287a8d80346d1ea9418998c597b4f0ff1f66148d0d806bb43db6607e0fe1cf
e9ea95a30d [net processing] Move all const declarations to top of net_processing.cpp (John Newbery)
507b36dd1b [validation] Move all const declarations to top of validation.h (John Newbery)
0109622b08 [validation] Move validation-only consts to validation.cpp (John Newbery)
b8580cacc7 [net processing] Move net processing consts to net_processing.cpp (John Newbery)
Pull request description:
Following the main.cpp split, there are still some constants in the wrong places, eg net_processing constants in validation.h. Move them all to their rightful homes. At the same time, make them constexpr.
Also move all const declarations to the top of their files, and ensure that they all have doxygen comments.
ACKs for top commit:
practicalswift:
ACK e9ea95a30d -- patch looks correct
MarcoFalke:
ACK e9ea95a30d🚉
Tree-SHA512: 44d81da73c7be01e1d36b939789d793f297d3b94f84ea4e7ac853c621cc7054b5a05c7c9e7b83db506db44c16f344541be8f240d955694211e53a84c32b0d2c5
as defined in PEP 3135:
"The new syntax:
super()
is equivalent to:
super(__class__, <firstarg>)
where __class__ is the class that the method was defined in, and <firstarg> is
the first parameter of the method (normally self for instance methods, and cls
for class methods)."
2495110012 test: add coverage for -rpcwallet cli option (Jon Atack)
Pull request description:
The bitcoin-cli `-rpcwallet=` option is an essential RPC/CLI option when more than one wallet is loaded (see `bitcoin-cli -help | grep -A5 rpcwallet` or `src/bitcoin-cli.cpp::L61`) and it currently has no test coverage.
It is not only used by users, but also by the test framework and ~10 test files via `get_wallet_rpc()`.
This PR adds coverage, while simultaneously improving the `-getinfo` coverage when multiple wallets are loaded. This is similar to the test coverage that would be added in #18594.
ACKs for top commit:
robot-visions:
ACK 2495110012
Tree-SHA512: caaa8b99fb8fa481ab2c6b2a287ed29720bb4553c3f66657462c44fa2990acaaf36cabeaaf81408678e5fdce4e105d729dd94b5ed8588dd1a6f2cb03fc25acf3
c4027e7350 refactor: test: use wait_for_getdata() in p2p_compactblocks.py (Sebastian Falbesoner)
Pull request description:
The method `wait_for_getdata()` was recently changed to be more precise by waiting for a specified list of hashes, instead of only matching _any_ `getdata` message (see Issue #18614 and PR #18690). This PR replaces the remaining occurences of manual inspection of `last_messages` with this call.
ACKs for top commit:
robot-visions:
ACK c4027e7350
Tree-SHA512: e10b346742f235b6ee2ef1f32f7fd74406c1a277389f020fb9913a93e94cc9530e1e9414872b83c9d2ae652ebce2b09b2c8c8372260c1afb4e0e54fbf7a935b0
fa262712ca test: Check submitblock return values (MarcoFalke)
Pull request description:
Add `assert_equal` in some tests to check the `submitblock` return value
ACKs for top commit:
robot-visions:
ACK fa262712ca
Tree-SHA512: 25d9effe82a4f6852184b9ac848f96336cc2cafb0bb07edb2792f00cd363f0759575bc9c164dd62f64425d3754028b4acd0675600c07d51277aa80bf66c6f960
