Commit graph

4390 commits

Author SHA1 Message Date
Andrew Chow
cbcad79eef
Merge bitcoin/bitcoin#21576: rpc, gui: bumpfee signer support
2c07cfacd1 gui: bumpfee signer support (Sjors Provoost)
7e02a33297 rpc: bumpfee signer support (Sjors Provoost)
304ece9945 rpc: document bools in FillPSBT() calls (Sjors Provoost)

Pull request description:

  The `bumpfee` RPC call and GUI fee bump interface now work with an external signer.

ACKs for top commit:
  achow101:
    ACK 2c07cfacd1
  furszy:
    code review ACK 2c07cfac
  jarolrod:
    tACK 2c07cfa

Tree-SHA512: 0c7b931f76fac67c9e33b9b935f29af6f69ac67a5ffcc586ed2f1676feac427735b1d971723b29ef332bb6fb5762949598ebbf728587e8f0ded95a9bfbb3e7a4
2022-12-20 15:30:17 -05:00
MarcoFalke
8ab19237e1
Merge bitcoin/bitcoin#26723: test: call keypoolrefill with priv key disabled should throw an error
ec63a4892e test: call `keypoolrefill` with private keys disabled should throw an error (brunoerg)

Pull request description:

  This PR adds test coverage for the following error:
  cb32328d1b/src/wallet/rpc/addresses.cpp (L332-L334)

ACKs for top commit:
  aureleoules:
    ACK ec63a4892e

Tree-SHA512: b5deda8981ff472f290e6e16c8723a58e02cbe099afd1f672c099f4add0a1d9b192b11a2c3f0e11b96794671f6b9efa75812b7a174248d7c58d7fd7d3310e7b9
2022-12-19 15:17:44 +01:00
MarcoFalke
bd13d6b369
Merge bitcoin/bitcoin#26656: tests: Improve runtime of some tests when --enable-debug
1647a11f39 tests: Reorder longer running tests in test_runner (Andrew Chow)
ff6c9fe027 tests: Whitelist test p2p connection in rpc_packages (Andrew Chow)
8c20796aac tests: Use waitfornewblock for work queue test in interface_rpc (Andrew Chow)
6c872d5e65 tests: Initialize sigops draining script with bytes in feature_taproot (Andrew Chow)
544cbf776c tests: Use batched RPC in feature_fee_estimation (Andrew Chow)
4ad7272f8b tests: reduce number of generated blocks for wallet_import_rescan (Andrew Chow)

Pull request description:

  When configured with `--enable-debug`, many tests become dramatically slower. These slow downs are particularly noticed in tests that generate a lot of blocks in separate calls, make a lot of RPC calls, or send a lot of data from the test framework's P2P connection. This PR aims to improve the runtime of some of the slower tests and improve the overall runtime of the test runner. This has improved the runtime of the test runner from ~400s to ~140s on my computer.

  The slowest test by far was `wallet_import_rescan.py`. This was taking ~320s. Most of that time was spent waiting for blocks to be mined and then synced to the other nodes. It was generating a new block for every new transaction it was creating in a setup loop. However it is not necessary to have one tx per block. By mining a block only every 10 txs, the runtime is improved to ~61s.

  The second slowest test was `feature_fee_estimation.py`. This test spends most of its time waiting for RPCs to respond. I was able to improve its runtime by batching RPC requests. This has improved the runtime from ~201s to ~140s.

  In `feature_taproot.py`, the test was constructing a Python `CScript` using a very large list of `OP_CHECKSIG`s. The constructor for the Python implementation of `CScript` was iterating this list in order to create a `bytes` from it even though a `bytes` could be created from it without iterating. By making the `bytes` before passing it into the constructor, we are able to improve this test's runtime from ~131s to ~106s.

  Although `interface_rpc.py` was not typically a slow test, I found that it would occasionally have a super long runtime. It typically takes ~7s, but I have observed it taking >400s to run on occasion. This longer runtime occurs more often when `--enable-debug`. This long runtime was caused by the "exceeding work queue" test which is really just trying to trigger a race condition. In this test, it would create a few threads and try an RPC in a loop in the hopes that eventually one of the RPCs would be added to the work queue while another was processing. It used `getrpcinfo` for this, but this function is fairly fast. I believe what was happening was that with `--enable-debug`, all of the code for receiving the RPC would often take longer to run than the RPC itself, so the majority of the requests would succeed, until we got lucky after 10's of thousands of requests. By changing this to use a slow RPC, the race condition can be triggered more reliably, and much sooner as well. I've used `waitfornewblock` with a 500ms timeout. This improves the runtime to ~3s consistently.

  The last test I've changed was `rpc_packages.py`. This test was one of the higher runtime variability tests. The main source of this variation appears to be waiting for the test node to relay a transaction to the test framework's P2P connection. By whitelisting that peer, the variability is reduced to nearly 0.

  Lastly, I've reordered the tests in `test_runner.py` to account for the slower runtimes when configured with `--enable-debug`. Some of the slow tests I've looked at were listed as being fast which was causing overall `test_runner.py` runtime to be extended. This change makes the test runner's runtime be bounded by the slowest test (currently `feature_fee_estimation.py` with my usual config (`-j 60`).

ACKs for top commit:
  willcl-ark:
    ACK 1647a11

Tree-SHA512: 529e0da4bc51f12c78a40d6d70b3a492b97723c96a3526148c46943d923c118737b32d2aec23d246392e50ab48013891ef19fe6205bf538b61b70d4f16a203eb
2022-12-19 10:14:35 +01:00
brunoerg
ec63a4892e test: call keypoolrefill with private keys disabled should throw an error 2022-12-18 22:00:38 -03:00
Andrew Chow
66c08e741d
Merge bitcoin/bitcoin#24865: rpc: Enable wallet import on pruned nodes and add test
564b580bf0 test: Introduce MIN_BLOCKS_TO_KEEP constant (Aurèle Oulès)
71d9a7c03b test: Wallet imports on pruned nodes (Aurèle Oulès)
e6906fcf9e rpc: Enable wallet import on pruned nodes (Aurèle Oulès)

Pull request description:

  Reopens #16037

  I have rebased the PR, addressed the comments of the original PR and added a functional test.

  > Before this change importwallet fails if any block is pruned. This PR makes it possible to importwallet if all required blocks aren't pruned. This is possible because the dump format includes key timestamps.

  For reviewers:
  `python test/functional/wallet_pruning.py --nocleanup` will generate a large blockchain (~700MB) that can be used to manually test wallet imports on a pruned node. Node0 is not pruned, while node1 is.

ACKs for top commit:
  kouloumos:
    ACK 564b580bf0
  achow101:
    reACK 564b580bf0
  furszy:
    ACK 564b580
  w0xlt:
    ACK 564b580bf0

Tree-SHA512: b345a6c455fcb6581cdaa5f7a55d79e763a55cb08c81d66be5b12794985d79cd51b9b39bdcd0f7ba0a2a2643e9b2ddc49310ff03d16b430df2f74e990800eabf
2022-12-16 17:30:57 -05:00
MarcoFalke
9a72119e7e
Merge bitcoin/bitcoin#26651: test: Avoid intermittent timeout in feature_assumevalid.py
fa34e5f3d3 test: Avoid intermittent timeout in feature_assumevalid.py (MarcoFalke)

Pull request description:

  Currently the test will spin up p2p connections in the beginning, then announce the headers to all nodes, but only send the blocks sequentially. This takes a long time, so when getting to the last node, it will have already timed out, while node1 is busy eating blocks. Example:

  ```
   node2 2022-12-06T19:31:35.419291Z [msghand] [net_processing.cpp:5783] [SendMessages] [net] Requesting block 2cfdb317b3b901f79e2d4f96339d0c0dffd8ef2685d324f62ab0e2fa3402430e (1) peer=0
   node2 2022-12-06T19:31:35.424784Z [msghand] [net.cpp:2776] [PushMessage] [net] sending getdata (577 bytes) peer=0
  [...]
   node2 2022-12-06T19:41:35.423257Z [msghand] [net_processing.cpp:5729] [SendMessages] Timeout downloading block 2cfdb317b3b901f79e2d4f96339d0c0dffd8ef2685d324f62ab0e2fa3402430e from peer=0, disconnecting
   node1 2022-12-06T19:41:35.438706Z [msghand] [net_processing.cpp:5783] [SendMessages] [net] Requesting block 6575919043306ed309014d0bd79814b4fab8afaa281e026d8cc3a1c4c2336fbc (1748) peer=0
   node2 2022-12-06T19:41:35.521253Z [net] [net.cpp:573] [CloseSocketDisconnect] [net] disconnecting peer=0
   node2 2022-12-06T19:41:35.630417Z [net] [net_processing.cpp:1532] [FinalizeNode] [net] Cleared nodestate for peer=0
  ```

  Fix this by only spinning up the p2p connection right before they are needed.

ACKs for top commit:
  jamesob:
    ACK fa34e5f3d3 ([`jamesob/ackr/26651.1.MarcoFalke.test_avoid_intermittent`](https://github.com/jamesob/bitcoin/tree/ackr/26651.1.MarcoFalke.test_avoid_intermittent))

Tree-SHA512: 7a1b114c07dfa30237c8cd8637dd6646c5c2dc2530c9de61db231738fddc800b620c31dc664237e33d35e951cf161f015fda593162efc9d85c5f68c6e37217d4
2022-12-15 13:09:06 +01:00
Aurèle Oulès
564b580bf0
test: Introduce MIN_BLOCKS_TO_KEEP constant 2022-12-15 09:53:51 +01:00
Aurèle Oulès
71d9a7c03b
test: Wallet imports on pruned nodes
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
Co-authored-by: Andreas Kouloumos <kouloumosa@gmail.com>
2022-12-15 09:53:50 +01:00
MarcoFalke
678889e6c6
Merge bitcoin/bitcoin#26689: test: add add_wallet_options to TestShell
bcb7123406 test: add add_wallet_options to TestShell (josibake)

Pull request description:

  following 555519d082, `TestShell` now always runs with `-disablewallet`. simple fix is to add `add_wallet_options` to `add_options`; looks like testshell was overlooked when adding in the `add_wallet_options` call to the functional tests in #26480

ACKs for top commit:
  amitiuttarwar:
    ACK bcb7123406

Tree-SHA512: db554a8b3c8ff5bd10cab9592b316035a92f86a0a0ae8ff914de9687bbbb6fc2235bdf82c4cd40e4071782f8b6edf91aad372e82ed3b826c9d8ae39dbe3dbf57
2022-12-14 09:16:02 +01:00
Andrew Chow
daf881de9d
Merge bitcoin/bitcoin#23319: rpc: Return fee and prevout (utxos) to getrawtransaction
f86697163e rpc: Return fee and prevout(s) to getrawtransaction (Douglas Chimento)

Pull request description:

  Add fee response in BTC to getrawtransaction #23264

  ### For Reviewers

  * Verbose arg is now an int
  * Verbose = 2 includes a `fee` field and `prevout`
  * [./test/functional/rpc_rawtransaction.py](./test/functional/rpc_rawtransaction.py) contains a new test to validate fields of new verbosity 2 (not the values)

  ```
  bitcoin-cli -chain=test getrawtransaction 9ae533f7da9be4a34997db78343a8d8d6d6186b6bba3959e56f416a5c70e7de4 2 000000000000001d442e556146d5f2841d85150c200e8d8b8a4b5005b13878f6
  ```
  ```
    "in_active_chain": true,
    "txid": "9ae533f7da9be4a34997db78343a8d8d6d6186b6bba3959e56f416a5c70e7de4",
    "hash": "7f23e3f3a0a256ddea1d35ffd43e9afdd67cc68389ef1a804bb20c76abd6863e",
   ....
    "vin": [
      {
        "txid": "23fc75d6d74f6f97e225839af69ff36a612fe04db58a4414ec4828d1749a05a0",
        "vout": 0,
        "scriptSig": {
          "asm": "",
          "hex": ""
        },
        "prevout": {
          "generated": false,
          "height": 2099486,
          "value": 0.00017764,
          "scriptPubKey": {
            "asm": "0 7846ce1ced3253d8bd43008db2ca364cc722f5a2",
            "hex": "00147846ce1ced3253d8bd43008db2ca364cc722f5a2",
            "address": "tb1q0prvu88dxffa302rqzxm9j3kfnrj9adzk49mlp",
            "type": "witness_v0_keyhash"
          }
        },
        "sequence": 4294967295
      },
  ...
   "fee": 0.00000762
  }
  ```

ACKs for top commit:
  achow101:
    ACK f86697163e
  aureleoules:
    ACK f86697163e
  hernanmarino:
    re ACK f86697163e
  pablomartin4btc:
    re-tACK f86697163e

Tree-SHA512: 591fdc285d74fa7803e04ad01c7b70bc20fac6b1369e7bd5b8e2cde9b750ea52d6c70d79225b74bef4f4bbc0fb960877778017184e146119da4a55f9593d1224
2022-12-13 18:09:09 -05:00
MarcoFalke
a4baf3f177
Merge bitcoin/bitcoin#26628: RPC: Reject RPC requests with same named parameter specified multiple times
8c3ff7d52a test: Suggested cleanups for rpc_namedparams test (Ryan Ofsky)
d1ca563825 bitcoin-cli: Make it an error to specify the "args" parameter two different ways (Ryan Ofsky)
6bd1d20b8c rpc: Make it an error server-side to specify same named parameter multiple times (Ryan Ofsky)
e2c3b18e67 test: Add RPC tests for same named parameter specified more than once (Ryan Ofsky)

Pull request description:

  Make the JSON-RPC server reject requests with the same named parameter specified multiple times, instead of silently overwriting earlier parameter values with later ones.

  Generally JSON keys are supposed to unique, and their order isn't supposed to be significant, so having the server silently discard duplicate keys is error-prone. Most likely if an RPC client is sending a request with duplicate keys it means something is wrong with the request and there should be an error.

  After this change, named parameters are still allowed to specified multiple times on the `bitcoin-cli` command line, since `bitcoin-cli` automatically replaces earlier values with later values before sending the JSON-RPC request. This makes sense, since it's not unusual for the order of command line options to be significant or for later command line options to override earlier ones.

ACKs for top commit:
  MarcoFalke:
    review ACK 8c3ff7d52a 🗂
  kristapsk:
    ACK 8c3ff7d52a
  stickies-v:
    ACK 8c3ff7d52

Tree-SHA512: 2d1357dcc2c171da287aeefc7b333ba4e67babfb64fc14d7fa0940256e18010a2a65054f3bf7fa1571b144d2de8b82d53076111b5f97ba29320cfe84b6ed986f
2022-12-13 17:57:23 +01:00
fanquake
968f03e65c
Merge bitcoin/bitcoin#26477: validation: fix broken maxtipage comparison
e4be0e9b06 test: add -maxtipage test for the maximum allowable value (James O'Beirne)
a451e832b4 fix: validation: cast now() to seconds for maxtipage comparison (James O'Beirne)

Pull request description:

  Since faf44876db, the maxtipage comparison in IsInitialBlockDownload() has been broken, since the NodeClock::now() time_point is in the system's native denomination (nanoseconds).

  Without this patch, specifying the maximum allowable -maxtipage (9223372036854775807) results in a SIGABRT crash:

  ```
  % gdb --args ./src/bitcoind -maxtipage=9223372036854775207 -minimumchainwork=0x00 -stopatheight=30000
  ...
  2022-11-09T15:55:17Z [dnsseed] dnsseed thread exit
  [Thread 0x7fff937fe640 (LWP 69883) exited]

  Thread 29 "b-msghand" received signal SIGABRT, Aborted.
  [Switching to Thread 0x7fff91ffb640 (LWP 69886)]
  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
  44      ./nptl/pthread_kill.c: No such file or directory.
  (gdb) bt
  #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
  #1  0x00007ffff768989f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
  #2  0x00007ffff763da52 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
  #3  0x00007ffff7628469 in __GI_abort () at ./stdlib/abort.c:79
  #4  0x00007ffff7cf79a4 in __mulvdi3 () from /lib/x86_64-linux-gnu/libgcc_s.so.1
  #5  0x00005555558d13ab in std::chrono::__duration_cast_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> >, std::ratio<1000000000l, 1l>, long, false, true>::__cast<long, std::ratio<1l, 1l> > (__d=...) at /usr/include/c++/12/bits/chrono.h:521
  #6  std::chrono::duration_cast<std::chrono::duration<long, std::ratio<1l, 1000000000l> >, long, std::ratio<1l, 1l> > (__d=...)
      at /usr/include/c++/12/bits/chrono.h:260
  #7  std::chrono::duration<long, std::ratio<1l, 1000000000l> >::duration<long, std::ratio<1l, 1l>, void> (__d=..., this=<optimized out>)
      at /usr/include/c++/12/bits/chrono.h:514
  #8  std::chrono::operator-<long, std::ratio<1l, 1000000000l>, long, std::ratio<1l, 1l> > (__rhs=..., __lhs=...)
      at /usr/include/c++/12/bits/chrono.h:650
  #9  std::chrono::operator-<NodeClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, long, std::ratio<1l, 1l> > (__rhs=...,
      __lhs=...) at /usr/include/c++/12/bits/chrono.h:1020
  #10 Chainstate::IsInitialBlockDownload (this=0x555556071940) at ./src/validation.cpp:1545
  #11 0x00005555556efd1e in operator() (__closure=<optimized out>) at ./src/net_processing.cpp:3369
  #12 (anonymous namespace)::PeerManagerImpl::ProcessMessage (this=0x555556219be0, pfrom=..., msg_type=..., vRecv=..., time_received=...,
      interruptMsgProc=...) at ./src/net_processing.cpp:3369
  #13 0x00005555556f75cc in (anonymous namespace)::PeerManagerImpl::ProcessMessages (this=0x555556219be0, pfrom=<optimized out>,
      interruptMsgProc=std::atomic<bool> = { false }) at ./src/net_processing.cpp:4985
  #14 0x00005555556a83c9 in CConnman::ThreadMessageHandler (this=0x5555560ebc70) at ./src/net.cpp:2014
  #15 0x0000555555c4d5d6 in std::function<void ()>::operator()() const (this=0x7fff91ffadb0) at /usr/include/c++/12/bits/std_function.h:591
  #16 util::TraceThread(std::basic_string_view<char, std::char_traits<char> >, std::function<void ()>) (
      thread_name="0\255\377\221\377\177\000\000\v\000\000\000\000\000\000\000TraceThread\000\000\000\000\000P\255\377\221\377\177\000\000\017\000\000\000\000\000\000\000util/thread.cpp\000\000\000\000\000\000\000\000\000\000ihB鵿6\000\000\000\000\000\000\000\000\260\255\377\221\377\177\000\000\277\211\321UUU\000\000p\324\304UUU\000\000\002\000\000\000\000\000\000\000\240xh\367\377\177\000\000\000\000\000\000\000\000\000\000]\340iUUU\000\000p\274\016VUU\000\000\000\000\000\000\000\000\000\000\300\303iUUU\000\000p\206jUUU", '\000' <repeats 11 times>, "ihB鵿6\200\251!VUU\000\000"..., thread_func=...) at util/thread.cpp:21
  #17 0x000055555569e05d in std::__invoke_impl<void, void (*)(std::basic_string_view<char>, std::function<void()>), char const*, CConnman::Start(CScheduler&, const Options&)::<lambda()> > (__f=<optimized out>) at /usr/include/c++/12/bits/invoke.h:61
  #18 std::__invoke<void (*)(std::basic_string_view<char>, std::function<void()>), char const*, CConnman::Start(CScheduler&, const Options&)::<lambda()> > (__fn=<optimized out>) at /usr/include/c++/12/bits/invoke.h:96
  #19 std:🧵:_Invoker<std::tuple<void (*)(std::basic_string_view<char, std::char_traits<char> >, std::function<void()>), char const*, CConnman::Start(CScheduler&, const Options&)::<lambda()> > >::_M_invoke<0, 1, 2> (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:252
  #20 std:🧵:_Invoker<std::tuple<void (*)(std::basic_string_view<char, std::char_traits<char> >, std::function<void()>), char const*, CConnman::Start(CScheduler&, const Options&)::<lambda()> > >::operator() (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:259
  #21 std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<void (*)(std::basic_string_view<char, std::char_traits<char> >, std::function<void()>), char const*, CConnman::Start(CScheduler&, const Options&)::<lambda()> > > >::_M_run(void) (this=<optimized out>)
      at /usr/include/c++/12/bits/std_thread.h:210
  #22 0x00007ffff7ad43d3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
  #23 0x00007ffff7687b27 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:435
  #24 0x00007ffff770a78c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
  (gdb)
  ```

ACKs for top commit:
  MarcoFalke:
    review ACK e4be0e9b06 🏽

Tree-SHA512: d892d6264a284d952a68a8631a6301277373b8df939dafd9e2652f2f22ab60712cde63b90c27c67ea2d05f02443452e3e4e1b9f25479bfaca00d4c4de13b9fbd
2022-12-13 10:07:37 +00:00
josibake
bcb7123406
test: add add_wallet_options to TestShell
without this, testShell runs with -disablewallet
2022-12-12 17:58:15 +01:00
fanquake
3b5fb6e77a
Merge bitcoin/bitcoin#26213: rpc: Strict type checking for RPC boolean parameters
fa0153e609 refactor: Replace isTrue with get_bool (MarcoFalke)
fa2cc5d1d6 bugfix: Strict type checking for RPC boolean parameters (MarcoFalke)

Pull request description:

ACKs for top commit:
  ryanofsky:
    Code review ACK fa0153e609
  furszy:
    Code ACK fa0153e6

Tree-SHA512: b221f823c69d90c94447fd491071ff3659cfd512872b495ebc3e711f50633351974102c9ef7e50fa4a393c4131d349adea8fd41cc9d66f1f31e1f5e7a5f78757
2022-12-10 09:58:33 +00:00
Andrew Chow
1647a11f39 tests: Reorder longer running tests in test_runner
The logest running tests should be at the front of the list in
test_runner.py. Since compiling with --enable-debug can have a
significant effect on test runtime, the order is based on the runtime
with that option configured.
2022-12-09 13:57:55 -05:00
Andrew Chow
ff6c9fe027 tests: Whitelist test p2p connection in rpc_packages
test_submit_child_with_parents creates a p2p connection which waits for
the node to announce transactions to it. By whitelisting this
connection, we can reduce the amount of time spent waiting for this
announcement which improves the test runtime and runtime variance.
2022-12-09 13:57:01 -05:00
Andrew Chow
8c20796aac tests: Use waitfornewblock for work queue test in interface_rpc
The work queue exceeded test in interface_rpc.py would repeatedly call
an RPC until the error was achieved. However hitting this error is
dependent on the processing speed of the computer and the optimization
level of the binary. Configurations that result in slower processing
would result in the RPC used being processed before the error could be
hit, resulting the test's runtime having a high variance.

Switching the RPC to waitfornewblock allows it to run in a much more
consistent time that is still fairly fast. waitfornewblock forces
the RPC server to allocate a thread and wait, occupying a spot in the
work queue. This is perfect for this test because the slower the RPC,
the more likely we will achieve the race condition necessary to pass the
test. Using a timeout of 500 ms appears to work reliably without causing
the test to take too long.
2022-12-09 13:57:01 -05:00
Andrew Chow
6c872d5e65 tests: Initialize sigops draining script with bytes in feature_taproot
The sigops draining script in feature_taproot's block_submit was
initialized with a list that would end up always being iterated by
CScript's constructor. Since this list is very large, a lot of time
would be wasted. By creating and passing a bytes object initialized from
that list, we can avoid this iteration and dramatically improve the
runtime of feature_taproot.
2022-12-09 13:57:01 -05:00
Andrew Chow
544cbf776c tests: Use batched RPC in feature_fee_estimation
feature_fee_estimation has a lot of loops that hit the RPC many times in
succession in order to setup scenarios. Using batched requests for these
can reduce the test's runtime without effecting the test's behavior.
2022-12-09 13:57:01 -05:00
Andrew Chow
4ad7272f8b tests: reduce number of generated blocks for wallet_import_rescan
Generating blocks is slow, especially when --enable-debug. There is no
need to generate a new block for each transaction, so avoid doing that
to improve this test's runtime.
2022-12-09 13:57:01 -05:00
MarcoFalke
fa7d71accc
test: Move rpc_fundrawtransaction.py to wallet_fundrawtransaction.py 2022-12-09 11:54:29 +01:00
MarcoFalke
fa933d6985
test: Move feature_backwards_compatibility.py to wallet_backwards_compatibility.py 2022-12-09 11:54:17 +01:00
MarcoFalke
16624e6ff3
Merge bitcoin/bitcoin#26660: test: Use last release in compatibility tests
fabb24cbef test: Use last release in compatibility tests (MarcoFalke)

Pull request description:

  In compatibility tests it makes sense to always use the last release without the new feature, as it is likely more in use than any even older previous release.

ACKs for top commit:
  Sjors:
    utACK fabb24c

Tree-SHA512: beb854f4d28ba313282e1e0303abb0e09377828b138bde5a3e209337210b6b4c24855ab90a68f8789387001e4ca33b15cc37dbc9b7809929f4e7d1b69833a527
2022-12-09 09:25:52 +01:00
MarcoFalke
fabb24cbef
test: Use last release in compatibility tests 2022-12-08 14:57:25 +01:00
MarcoFalke
fa34e5f3d3
test: Avoid intermittent timeout in feature_assumevalid.py 2022-12-08 14:24:46 +01:00
Aurèle Oulès
c29bff5b91
test: Fix backwards compatibility intermittent failure 2022-12-08 11:13:34 +01:00
Andrew Chow
a653f4bb1f
Merge bitcoin/bitcoin#25934: wallet, rpc: add label to listsinceblock
4e362c2b72 doc: add release note for 25934 (brunoerg)
fe488b4c4b test: add coverage for `label` in `listsinceblock` (brunoerg)
722e9a418d wallet, rpc: add `label` to `listsinceblock` (brunoerg)
852891ff98 refactor, wallet: use optional for `label` in `ListTransactions` (brunoerg)

Pull request description:

  This PR adds `label` parameter to `listsinceblock` to be able to fetch all incoming transactions having the specified label since a specific block.

  It's possible to use it in `listtransactions`, however, it's only possible to set the number of transactions to return, not a specific block to fetch from. `getreceivedbylabel` only returns the total amount received, not the txs info. `listreceivedbylabel` doesn't list all the informations about the transactions and it's not possible to fetch since a block.

ACKs for top commit:
  achow101:
    ACK 4e362c2b72
  w0xlt:
    ACK 4e362c2b72
  aureleoules:
    ACK 4e362c2b72

Tree-SHA512: fbde5db8cebf7a27804154fa61997b5155ad512e978cebb78c17acab9efcb624ea5f39d649899d12e5e675f80d4d0064cae8132b864de0d93a8d1e6fbcb9a737
2022-12-07 18:42:41 -05:00
MarcoFalke
fa2cc5d1d6
bugfix: Strict type checking for RPC boolean parameters 2022-12-07 17:55:58 +01:00
MarcoFalke
9052d869c9
Merge bitcoin/bitcoin#26517: test: Changed small_txpuzzle_randfee to return the virtual size instead of the transaction hex for feerate calculation.
6fb102c9f3 test: Changed small_txpuzzle_randfee to return the virtual size instead of the transaction hex for feerate calculation. (Randall Naar)

Pull request description:

  The fee rates used in feature_fee_estimation.py are calculated using the raw transaction size instead of the virtual transaction size (which is used in 'CBlockPolicyEstimator::processBlockTx' and 'CBlockPolicyEstimator::processBlock'). This leads to inconsistencies as the fee rates used in check_raw_estimates are incorrect and can cause assertions to fail.

  refs #25179

ACKs for top commit:
  MarcoFalke:
    ACK 6fb102c9f3

Tree-SHA512: b2bca85fffa605aeb17574f050736d4556506d782dd7fd969e165e48e108fd95ef4c4e2abbae83cce05ca777a00f4459cabfa0932694fa8bb93ddfba09d84357
2022-12-07 17:33:37 +01:00
brunoerg
fe488b4c4b test: add coverage for label in listsinceblock 2022-12-06 15:27:50 -03:00
Andrew Chow
ef744c03e5
Merge bitcoin/bitcoin#25729: wallet: Check max transaction weight in CoinSelection
c7c7ee9d0b test: Check max transaction weight in CoinSelection (Aurèle Oulès)
6b563cae92 wallet: Check max tx weight in coin selector (Aurèle Oulès)

Pull request description:

  This PR is an attempt to fix #5782.

  I have added 4 test scenarios, 3 of them provided here https://github.com/bitcoin/bitcoin/issues/5782#issuecomment-73819058 (slightly modified to use a segwit wallet).

  Here are my benchmarks :
  ## PR
  |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |        1,466,341.00 |              681.97 |    0.6% |   11,176,762.00 |    3,358,752.00 |  3.328 |   1,897,839.00 |    0.3% |      0.02 | `CoinSelection`

  ## Master

  |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |        1,526,029.00 |              655.30 |    0.5% |   11,142,188.00 |    3,499,200.00 |  3.184 |   1,994,156.00 |    0.2% |      0.02 | `CoinSelection`

ACKs for top commit:
  achow101:
    reACK c7c7ee9d0b
  w0xlt:
    ACK c7c7ee9d0b
  furszy:
    diff ACK c7c7ee9d

Tree-SHA512: ef0b28576ff845174651ba494aa9adee234c96e6f886d0e032eceb7050296e45b099dda0039d1dfb9944469f067627b2101f3ff855c70353cf39d1fc7ee81828
2022-12-06 12:08:58 -05:00
Andrew Chow
5d9b5305af
Merge bitcoin/bitcoin#19888: rpc, test: Improve getblockstats for unspendables
d885bb2f6e test: Test exclusion of OP_RETURN from getblockstats (Fabian Jahr)
ba9d288b24 test: Fix getblockstats test data generator (Fabian Jahr)
2ca5a496c2 rpc: Improve getblockstats (Fabian Jahr)
cb94db119f validation, index: Add unspendable coinbase helper functions (Fabian Jahr)

Pull request description:

  Fixes #19885

  The genesis block does not have undo data saved to disk so the RPC errored because of that.

ACKs for top commit:
  achow101:
    ACK d885bb2f6e
  aureleoules:
    ACK d885bb2f6e
  stickies-v:
    ACK d885bb2f6

Tree-SHA512: f37bda736ed605b7a41a81eeb4bfbb5d2b8518f847819e5d6a090548a61caf1455623e15165d72589ab3f4478252b00e7b624f9313ad6708cac06dd5edb62e9a
2022-12-05 17:46:54 -05:00
Andrew Chow
2ce3d26757
Merge bitcoin/bitcoin#26462: wallet: fix crash on loading descriptor wallet containing legacy key type entries
3198e4239e test: check that loading descriptor wallet with legacy entries throws error (Sebastian Falbesoner)
349ed2a0ee wallet: throw error if legacy entries are present on loading descriptor wallets (Sebastian Falbesoner)

Pull request description:

  Loading a descriptor wallet currently leads to a segfault if a legacy key type entry is present that can be deserialized successfully and needs SPKman-interaction. To reproduce with a "cscript" entry (see second commit for details):

  ```
  $ ./src/bitcoin-cli createwallet crashme
  $ ./src/bitcoin-cli unloadwallet crashme
  $ sqlite3 ~/.bitcoin/wallets/crashme/wallet.dat
  SQLite version 3.38.2 2022-03-26 13:51:10
  Enter ".help" for usage hints.
  sqlite> INSERT INTO main VALUES(x'07637363726970740000000000000000000000000000000000000000', x'00');
  $ ./src/bitcoin-cli loadwallet crashme

  --- bitcoind output: ---
  2022-11-06T13:51:01Z Using SQLite Version 3.38.2
  2022-11-06T13:51:01Z Using wallet /home/honey/.bitcoin/wallets/crashme
  2022-11-06T13:51:01Z init message: Loading wallet…
  2022-11-06T13:51:01Z [crashme] Wallet file version = 10500, last client version = 249900

  Segmentation fault (core dumped)
  ```

  Background: In the wallet key-value-loading routine, most legacy type entries require a `LegacyScriptPubKeyMan` instance after successful deserialization. On a descriptor wallet, creating that (via method `GetOrCreateLegacyScriptPubKeyMan`) fails and then leads to a null-pointer dereference crash. E.g. for CSCRIPT: 50422b770a/src/wallet/walletdb.cpp (L589-L594)

  ~~This PR fixes this by simply ignoring legacy entries if the wallet flags indicate that we have a descriptor wallet. The second commits adds a regression test to the descriptor wallet's functional test (fortunately Python includes sqlite3 support in the standard library).~~

  ~~Probably it would be even better to throw a warning to the user if unexpected legacy entries are found in descriptor wallets, but I think as a first mitigation everything is obvisouly better than crashing. As far as I'm aware, descriptor wallets created/migrated by Bitcoin Core should never end up in a state containing legacy type entries though.~~

  This PR fixes this by throwing an error if legacy entries are found in descriptor wallets on loading.

ACKs for top commit:
  achow101:
    ACK 3198e4239e
  aureleoules:
    ACK 3198e4239e

Tree-SHA512: ee43da3f61248e0fde55d9a705869202cb83df678ebf4816f0e77263f0beac0d7bae9490465d1753159efb093ee37182931d76b2e2b6e8c6f8761285700ace1c
2022-12-05 17:37:48 -05:00
Aurèle Oulès
6b563cae92
wallet: Check max tx weight in coin selector
Co-authored-by: Andrew Chow <github@achow101.com>
2022-12-05 19:32:11 +01:00
Andrew Chow
7734a0160d
Merge bitcoin/bitcoin#26640: test: Run mempool_compatibility.py with MiniWallet
fa43f60a0c test: Run mempool_compatibility.py with MiniWallet (MarcoFalke)

Pull request description:

  By using the already existing miniwallet, the test can be run even when no wallet is compiled.

ACKs for top commit:
  glozow:
    ACK fa43f60a0c
  achow101:
    ACK fa43f60a0c

Tree-SHA512: 6877b3f2f364663f04c28ab9f3d69780de6d1b77cc862379bba8c8242bbcfb0d26eb84c56cf721141407c393f1f3b49f667ae4fb32b3566108d71250e8b5d7bc
2022-12-05 12:39:15 -05:00
Andrew Chow
f0c4807a6a
Merge bitcoin/bitcoin#26560: wallet: bugfix, invalid CoinsResult cached total amount
7362f8e5e2 refactor: make CoinsResult total amounts members private (furszy)
3282fad599 wallet: add assert to SelectionResult::Merge for safety (S3RK)
c4e3b7d6a1 wallet: SelectCoins, return early if wallet's UTXOs cannot cover the target (furszy)
cac2725fd0 test: bugfix, coinselector_test, use 'CoinsResult::Erase/Add' instead of direct member access (furszy)
cf79384697 test: Coin Selection, duplicated preset inputs selection (furszy)
341ba7ffd8 test: wallet, coverage for CoinsResult::Erase function (furszy)
f930aefff9 wallet: bugfix, 'CoinsResult::Erase' is erasing only one output of the set (furszy)

Pull request description:

  This comes with #26559.

  Solving few bugs inside the wallet's transaction creation
  process and adding test coverage for them.
  Plus, making use of the `CoinsResult::total_amount` cached value
  inside the Coin Selection process to return early if we don't have
  enough funds to cover the target amount.

  ### Bugs

  1) The `CoinsResult::Erase` method removes only one
  output from the available coins vector (there is a [loop break](c1061be14a/src/wallet/spend.cpp (L112))
  that should have never been there) and not all the preset inputs.

     Which on master is not a problem, because since [#25685](https://github.com/bitcoin/bitcoin/pull/25685)
     we are no longer using the method. But, it's a bug on v24
     (check [#26559](https://github.com/bitcoin/bitcoin/pull/26559)).

     This method it's being fixed and not removed because I'm later using it to solve
     another bug inside this PR.

  2) As we update the total cached amount of the `CoinsResult` object inside
     `AvailableCoins` and we don't use such function inside the coin selection
     tests (we manually load up the `CoinsResult` object), there is a discrepancy
     between the outputs that we add/erase and the total amount cached value.

  ### Improvements

  * This makes use of the `CoinsResult` total amount field to early return
    with an "Insufficient funds" error inside Coin Selection if the tx target
    amount is greater than the sum of all the wallet available coins plus the
    preset inputs amounts (we don't need to perform the entire coin selection
    process if we already know that there aren't enough funds inside our wallet).

  ### Test Coverage

  1) Adds test coverage for the duplicated preset input selection bug that we have in v24.
    Where the wallet invalidly selects the preset inputs twice during the Coin Selection
    process. Which ends up with a "good" Coin Selection result that does not cover the
    total tx target amount. Which, alone, crashes the wallet due an insane fee.
    But.. to make it worst, adding the subtract fee from output functionality
    to this mix ends up with the wallet by-passing the "insane" fee assertion,
    decreasing the output amount to fulfill the insane fee, and.. sadly,
    broadcasting the tx to the network.

  2) Adds test coverage for the `CoinsResult::Erase` method.

  ------------------------------------

  TO DO:
  * [ ] Update [#26559 ](https://github.com/bitcoin/bitcoin/pull/26559) description.

ACKs for top commit:
  achow101:
    ACK 7362f8e5e2
  glozow:
    ACK 7362f8e5e2, I assume there will be a followup PR to add coin selection sanity checks and we can discuss the best way to do that there.
  josibake:
    ACK [7362f8e](7362f8e5e2)

Tree-SHA512: 37a6828ea10d8d36c8d5873ceede7c8bef72ae4c34bef21721fa9dad83ad6dba93711c3170a26ab6e05bdbc267bb17433da08ccb83b82956d05fb16090328cba
2022-12-05 12:00:45 -05:00
MarcoFalke
38cbf43dee
Merge bitcoin/bitcoin#26414: test: Move tx creation to create_self_transfer_multi
0b78110f73 test: Move tx creation to create_self_transfer_multi (kouloumos)

Pull request description:

  Two birds with one stone: replacement of https://github.com/bitcoin/bitcoin/pull/26278 with simplification of the MiniWallet's transaction creation logic.

  Currently the MiniWallet creates simple txns (1 input, 1 output) with `create_self_transfer`.  https://github.com/bitcoin/bitcoin/pull/24637 introduced `create_self_transfer_multi` **which uses** `create_self_transfer` to create a "transaction template" which then adjusts (copy and mutate inputs and outputs) in order to create more complex multi-input multi-output transactions.
  This can more easily lead to issues such as https://github.com/bitcoin/bitcoin/pull/26278 and is more of a maintenance burden.

  This PR simplifies the logic by going the other way around. Now `create_self_transfer` **uses** `create_self_transfer_multi`.
  The transaction creation logic has been moved to `create_self_transfer_multi` which is being called by `create_self_transfer` to construct the simple case of 1 input 1 output transaction.

ACKs for top commit:
  MarcoFalke:
    ACK 0b78110f73 👒

Tree-SHA512: 147e577ed5444bee57865bd375b37c9b49d6539e9875c30c2667e70fcba27fe80bcb4552a4e6efb42760d34b40d5dad826883b778eaeefe29425ec081787b4bd
2022-12-05 16:22:42 +01:00
MarcoFalke
fa43f60a0c
test: Run mempool_compatibility.py with MiniWallet 2022-12-05 13:13:00 +01:00
MarcoFalke
fadf7b8fef
test: Fix intermittent issue in rpc_net.py 2022-12-03 17:42:05 +01:00
MarcoFalke
cac29f5cd6
Merge bitcoin/bitcoin#26622: test: Add test for sendall min-fee setting
cb44c5923a test: Add sendall test for min-fee setting (Aurèle Oulès)

Pull request description:

  While experimenting with mutation testing it appeared that the minimum fee-rate check was not tested for the `sendall` RPC.

  https://bcm-ui.aureleoules.com/mutations/3581479318544ea6b97f788cec6e6ef1

ACKs for top commit:
  0xB10C:
    ACK cb44c5923a
  ishaanam:
    ACK cb44c5923a
  stickies-v:
    re-ACK [cb44c59](cb44c5923a)

Tree-SHA512: 31978436e1f01cc6abf44addc62b6887e65611e9a7ae7dc72e6a73cdfdb3a6a4f0a6c53043b47ecd1b10fc902385a172921e68818a7f5061c96e5e1ef5280b48
2022-12-03 12:28:13 +01:00
Ryan Ofsky
d1ca563825 bitcoin-cli: Make it an error to specify the "args" parameter two different ways
MarcoFalke reported the case of positional arguments silently overwriting the
named "args" parameter in bitcoin-cli
https://github.com/bitcoin/bitcoin/pull/19762#discussion_r1035761471 and this
behavior is confusing and was not intended when support for "args" parameters
was added to bitcoin-cli in #19762.

Instead of letting one "args" value overwrite the other in the client, just
pass the values to the server verbatim, and let the error be handled server
side.
2022-12-02 17:53:58 -05:00
Ryan Ofsky
e2c3b18e67 test: Add RPC tests for same named parameter specified more than once
Current behavior isn't ideal and will be changed in upcoming commits, but it's
useful to have test coverage regardless.

MarcoFalke reported the case of bitcoin-cli positional arguments overwriting
the named "args" parameter in
https://github.com/bitcoin/bitcoin/pull/19762#discussion_r1035761471
2022-12-02 17:37:08 -05:00
furszy
cf79384697
test: Coin Selection, duplicated preset inputs selection
This exercises the bug inside CoinsResult::Erase that
ends up on (1) a wallet crash or (2) a created and
broadcasted tx that contains a reduced recipient's amount.

This is covered by making the wallet selects the preset
inputs twice during the coin selection process.

Making the wallet think that the selection process result covers
the entire tx target when it does not. It's actually creating
a tx that sends more coins than what inputs are covering for.

Which, combined with the SFFO option, makes the wallet
incorrectly reduce the recipient's amount by the difference
between the original target and the wrongly counted inputs.
Which means, a created and relayed tx sending less coins to
the destination than what the user inputted.
2022-12-02 12:39:15 -03:00
fanquake
78aee0fe2c
Merge bitcoin/bitcoin#26569: p2p: Ensure transaction announcements are only queued for fully connected peers
8f2dac5409 [test] Add p2p_tx_privacy.py (dergoegge)
ce63fca13e [net processing] Assume that TxRelay::m_tx_inventory_to_send is empty pre-verack (dergoegge)
845e3a34c4 [net processing] Ensure transaction announcements are only queued for fully connected peers (dergoegge)

Pull request description:

  `TxRelay::m_next_inv_send_time` is initialized to 0, which means that any txids in `TxRelay::m_tx_inventory_to_send` will be announced on the first call to `PeerManagerImpl::SendMessages` for a fully connected peer (i.e. it completed the version handshake).

  Prior to #21160, `TxRelay::m_tx_inventory_to_send` was guaranteed to be empty on the first `SendMessages` call, as transaction announcements were only queued for fully connected peers. #21160 replaced a `CConnman::ForEachNode` call with a loop over `PeerManagerImpl::m_peer_map`, in which the txid for a transaction to be relayed is added to `TxRelay::m_tx_inventory_to_send` for all peers. Even for those peers that have not completed the version handshake. Prior to the PR this was not the case as `ForEachNode` has a "fully connected check" before calling a function for each node.

ACKs for top commit:
  MarcoFalke:
    ACK 8f2dac5409 🔝
  jnewbery:
    utACK 8f2dac5409

Tree-SHA512: e9eaccf7e00633ee0806fff1068b0e413a69a5e389d96c9659f68079915a6381ad5040c61f716cfcde77931d1b563b1049da97a232a95c6cd8355bd3d13404b9
2022-12-02 15:13:31 +00:00
Aurèle Oulès
cb44c5923a
test: Add sendall test for min-fee setting 2022-12-02 13:30:40 +01:00
MarcoFalke
4037478114
Merge bitcoin/bitcoin#26610: test: Remove unused blocktools imports from wallet_bumpfee
fa15c671f7 test: Remove unused blocktools imports from wallet_bumpfee (MarcoFalke)

Pull request description:

  Seems bloaty and confusing to use "tools" when a single RPC can already achieve the same.

ACKs for top commit:
  theStack:
    ACK fa15c671f7

Tree-SHA512: 87f9c31bbb286fee5e479ae54a1f9131f4d4294d665a985df8b14a0cc837a2a2e145ccd3660612768d88cfa0827a3eef392f85519b6cb7df365ba9fadafb0a41
2022-12-02 09:43:26 +01:00
MarcoFalke
02515117dc
Merge bitcoin/bitcoin#26620: test: refactor: eliminate genesis block timestamp magic numbers
dbed28968a test: refactor: eliminate genesis block timestamp magic numbers (Sebastian Falbesoner)

Pull request description:

  This tiny PR replaces all occurences of the regtest/testnet genesis block timestamp (found via `git grep 1296688602`) with the constant `TIME_GENESIS_BLOCK` to increase the readability.

ACKs for top commit:
  aureleoules:
    ACK dbed28968a

Tree-SHA512: be39d5c2631ad20eb775c2a077b1b1f056a1a4930aa44e6fdec73b974fd4bdf8da0103a3a38e3514b68fcf6a6316e007a371c523da5076a315545c9bf3091aee
2022-12-01 18:37:59 +01:00
MarcoFalke
a04121bdf9
Merge bitcoin/bitcoin#26617: test: add extra_args to BitcoinTestFramework class
150340aeac test: remove unneeded extra_args code (josibake)
989a52e0a5 test: add extra_args to BTF class (josibake)

Pull request description:

  ## problem
  If you try to add `extra_args` when using `TestShell`, you will get the following error:

  ```python
  >>> import sys
  >>>
  >>> sys.path.insert(0, "/home/josibake/bitcoin/test/functional")
  >>>
  >>> from test_framework.test_shell import TestShell
  >>> test = TestShell().setup(num_nodes=2, extra_args=[[],['-fallbackfee=0.0002']])
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/home/josibake/bitcoin/test/functional/test_framework/test_shell.py", line 41, in setup
      raise KeyError(key + " not a valid parameter key!")
  KeyError: 'extra_args not a valid parameter key!'
  >>>
  ```

  ## solution

  add `self.extra_args = None` so that `extra_args` is recognized as a valid parameter to be passed to `BitcoinTestFramework`

  ```python
  >>> import sys
  >>>
  >>> sys.path.insert(0, "/home/josibake/bitcoin/test/functional")
  >>>
  >>> from test_framework.test_shell import TestShell
  >>> test = TestShell().setup(num_nodes=2, extra_args=[[],['-fallbackfee=0.0002']])
  2022-12-01T11:23:23.765000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_sbwthbb_
  ```

ACKs for top commit:
  willcl-ark:
    re-ACK 150340aeac

Tree-SHA512: e6fa2a780a8f2d3472c322e8cdb00ec35cb220c3b4d6ca02291eb8b41c0d8676a635fbc79c6be80e3bb71d700a2501a4b73f762478f533ae453d492d449307bb
2022-12-01 17:50:46 +01:00
josibake
150340aeac
test: remove unneeded extra_args code 2022-12-01 16:55:43 +01:00
Sebastian Falbesoner
dbed28968a test: refactor: eliminate genesis block timestamp magic numbers 2022-12-01 12:59:59 +01:00