Commit graph

712 commits

Author SHA1 Message Date
John Newbery
7e158a6910 [test] Move MY_VERSION to p2p.py
The messages.py module should contain code and helpers for
[de]serializing p2p messages. Specific usage of those messages should
be in p2p.py. Therefore move MY_VERSION to p2p.py.

Also rename to P2P_VERSION to distinguish it from
other versioning used in Bitcoin/Bitcoin Core.

Also always set the nVersion field in CBlockLocator to 0 and ignore the
field in deserialized messages. The field is not currently used for
anything in Bitcoin Core.
2021-02-17 09:00:53 +00:00
John Newbery
652311165c [test] Move MIN_VERSION_SUPPORTED to p2p.py
The messages.py module should contain code and helpers for
[de]serializing p2p messages. Specific usage of those messages should
be in p2p.py. Therefore move MIN_VERSION_SUPPORTED to p2p.py.

Also rename to MIN_P2P_VERSION_SUPPORTED to distinguish it from
other versioning used in Bitcoin/Bitcoin Core.
2021-02-17 09:00:24 +00:00
MarcoFalke
69f7f50aa5
Merge #20993: test: store subversion (user agent) as string in msg_version
de85af5cce test: store subversion (user agent) as string in msg_version (Sebastian Falbesoner)

Pull request description:

  It seems more natural to treat the "subversion" field (=user agent string, see [BIP 14](https://github.com/bitcoin/bips/blob/master/bip-0014.mediawiki#Proposal)) of a node as pure string rather than a bytestring within the test framework. This is also suggested with the naming prefix in `msg_version.strSubVer`: one probably wouldn't expect a field starting with "str" to be a bytestring that needs further decoding to be useful. This PR moves the encoding/decoding parts to the serialization/deserialization routines so that the user doesn't have to bother with that anymore.

  Note that currently, in the master branch the `msg_version.strSubVer` is never read (only in `msg_version.__repr__`); However, one issue that is solved by this PR came up while testing #19509 (not merged yet): A decoding script for binary message capture files takes use of the functional test framework convert it into JSON format. Bytestrings will be convered to hexstrings, while pure strings will (surprise surprise) end up without modification in the file.

  So without this patch, we get:

  ```
  $ jq . out.json | grep -m5 strSubVer
        "strSubVer": "2f5361746f7368693a32312e39392e302f"
        "strSubVer": "2f5361746f7368693a302e32302e312f"
        "strSubVer": "2f5361746f7368693a32312e39392e302f"
        "strSubVer": "2f5361746f7368693a302e32302e312f"
        "strSubVer": "2f5361746f7368693a32312e39392e302f"
  ```

  After this patch:

  ```
  $ jq . out2.json | grep -m5 strSubVer
        "strSubVer": "/Satoshi:21.99.0/"
        "strSubVer": "/Satoshi:0.20.1/"
        "strSubVer": "/Satoshi:21.99.0/"
        "strSubVer": "/Satoshi:0.20.1/"
        "strSubVer": "/Satoshi:21.99.0/"
  ```

ACKs for top commit:
  jnewbery:
    utACK de85af5cce

Tree-SHA512: ff23642705c858e8387a625537dfec82e6b8a15da6d99b8d12152560e52d243ba17431b602b26f60996d897e00e3f37dcf8dc8a303ffb1d544df29a5937080f9
2021-02-17 09:36:30 +01:00
MarcoFalke
8d6994f93d
Merge #21100: test: remove unused function xor_bytes
f64adc1eed test: remove unused function xor_bytes (Sebastian Falbesoner)

Pull request description:

  The function `xor_bytes` was introduced in commit 3c226639eb (#19953, BIP340-342 validation), even [code-reviewed](https://github.com/bitcoin/bitcoin/pull/19953/files#r509383731), but actually never used. The [default signing algorithm in BIP340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki#Default_Signing) needs a xor operation, but this step is currently done by a single xor operation on large integer operands:

  ```
  t = (sec ^ int.from_bytes(TaggedHash("BIP0340/aux", aux), 'big')).to_bytes(32, 'big')
  ```

  Alternatively, we could keep the function and as well use it:
  ```diff
  --- a/test/functional/test_framework/key.py
  +++ b/test/functional/test_framework/key.py
  @@ -492,7 +492,7 @@ def sign_schnorr(key, msg, aux=None, flip_p=False, flip_r=False):
       P = SECP256K1.affine(SECP256K1.mul([(SECP256K1_G, sec)]))
       if SECP256K1.has_even_y(P) == flip_p:
           sec = SECP256K1_ORDER - sec
  -    t = (sec ^ int.from_bytes(TaggedHash("BIP0340/aux", aux), 'big')).to_bytes(32, 'big')
  +    t = xor_bytes(sec.to_bytes(32, 'big'), TaggedHash("BIP0340/aux", aux))
       kp = int.from_bytes(TaggedHash("BIP0340/nonce", t + P[0].to_bytes(32, 'big') + msg), 'big') % SECP256K1_ORDER
       assert kp != 0
       R = SECP256K1.affine(SECP256K1.mul([(SECP256K1_G, kp)]))
  ```

ACKs for top commit:
  practicalswift:
    cr ACK f64adc1eed: untested unused code should be removed

Tree-SHA512: e9afae303488f19c6f6f44874d3537ed1c8164a197490e2b4e34853db886b858825b719648fa1a30b95177dcee9cf241f94ee9b835f0a2cae07024ce38a8c090
2021-02-15 12:10:41 +01:00
Sebastian Falbesoner
f64adc1eed test: remove unused function xor_bytes 2021-02-09 17:58:21 +01:00
Bruno Garcia
c9095b738f test: remove unnecessary assignment in bdb 2021-02-09 10:22:50 -03:00
fanquake
9913419cc9
test: remove type: comments in favour of actual annotations
Now that we require Python 3.6+, we should be using variable type
annotations directly rather than # type: comments.

Also takes care of the discarded value issue in p2p_message_capture.py.
See: https://github.com/bitcoin/bitcoin/pull/19509/files#r571674446.
2021-02-08 13:24:44 +08:00
Wladimir J. van der Laan
a6b1bf6439
Merge #20267: Disable and fix tests for when BDB is not compiled
49797c3ccf tests: Disable bdb dump test when no bdb (Andrew Chow)
1194cf9269 Fix wallet_send.py wallet setup to work with descriptors (Andrew Chow)
fbaea7bfe4 Require legacy wallet for wallet_upgradewallet.py (Andrew Chow)
b1b679e0ab Explicitly mark legacy wallet tests as such (Andrew Chow)
09514e1bef Setup wallets for interface_zmq.py (Andrew Chow)
4d03ef9a73 Use MiniWallet in rpc_net.py (Andrew Chow)
4de23824b0 Setup wallets for interface_bitcoin_cli.py (Andrew Chow)
7c71c627d2 Setup wallets with descriptors for feature_notifications (Andrew Chow)
1f1bef8dba Have feature_filelock.py test both bdb and sqlite, depending on compiled (Andrew Chow)
c77975abc0 Disable upgrades tests that require BDB if BDB is not compiled (Andrew Chow)
1f20cac9d4 Disable wallet_descriptor.py bdb format check if BDB is not compiled (Andrew Chow)
3641597d7e tests: Don't make any wallets unless wallet is required (Andrew Chow)
b9b88f57a9 Skip legacy wallet reliant tests if BDB is not compiled (Andrew Chow)
6f36242389 tests: Set descriptors default based on compilation (Andrew Chow)

Pull request description:

  This PR fixes tests for when BDB is not compiled. Tests which rely on or test legacy wallet behavior are disabled and skipped when BDB is not compiled. For the components of some tests that are for legacy wallet things, those parts of the tests are skipped.

  For the majority of tests, changes are made so that they can be run with either legacy wallets or descriptor wallets without materially effecting the test. Most tests only need the wallet for balance and transactions, so the type of wallet is not an important part of those tests. Additionally, some tests are wallet agnostic and modified to instead use the test framework's MiniWallet.

ACKs for top commit:
  laanwj:
    ACK 49797c3ccf
  ryanofsky:
    Code review ACK 49797c3ccf. Only change since last review is dropping last commit. Previous review w/ suggestions for future followup is https://github.com/bitcoin/bitcoin/pull/20267#pullrequestreview-581508843

Tree-SHA512: 69659f8a81fb437ecbca962f4082c12835282dbf1fba7d9952f727a49e01981d749af9b09feda1c8ca737516c7d7a08ef17e782795df3fa69892d5021b41c1ed
2021-02-05 14:26:10 +01:00
MarcoFalke
384e090f93
Merge #19509: Per-Peer Message Capture
bff7c66e67 Add documentation to contrib folder (Troy Giorshev)
381f77be85 Add Message Capture Test (Troy Giorshev)
e4f378a505 Add capture parser (Troy Giorshev)
4d1a582549 Call CaptureMessage at appropriate locations (Troy Giorshev)
f2a77ff97b Add CaptureMessage (Troy Giorshev)
dbf779d5de Clean PushMessage and ProcessMessages (Troy Giorshev)

Pull request description:

  This PR introduces per-peer message capture into Bitcoin Core.  📓

  ## Purpose

  The purpose and scope of this feature is intentionally limited.  It answers a question anyone new to Bitcoin's P2P protocol has had: "Can I see what messages my node is sending and receiving?".

  ## Functionality

  When a new debug-only command line argument `capturemessages` is set, any message that the node receives or sends is captured.  The capture occurs in the MessageHandler thread.  When receiving a message, it is captured as soon as the MessageHandler thread takes the message off of the vProcessMsg queue.  When sending, the message is captured just before the message is pushed onto the vSendMsg queue.

  The message capture is as minimal as possible to reduce the performance impact on the node.  Messages are captured to a new `message_capture` folder in the datadir.  Each node has their own subfolder named with their IP address and port.  Inside, received and sent messages are captured into two binary files, msgs_recv.dat and msgs_sent.dat, like so:

  ```
  message_capture/203.0.113.7:56072/msgs_recv.dat
  message_capture/203.0.113.7:56072/msgs_sent.dat
  ```

  Because the messages are raw binary dumps, included in this PR is a Python parsing tool to convert the binary files into human-readable JSON.  This script has been placed on its own and out of the way in the new `contrib/message-capture` folder.  Its usage is simple and easily discovered by the autogenerated `-h` option.

  ## Future Maintenance

  I sympathize greatly with anyone who says "the best code is no code".

  The future maintenance of this feature will be minimal.  The logic to deserialize the payload of the p2p messages exists in our testing framework.  As long as our testing framework works, so will this tool.

  Additionally, I hope that the simplicity of this tool will mean that it gets used frequently, so that problems will be discovered and solved when they are small.

  ## FAQ

  "Why not just use Wireshark"

  Yes, Wireshark has the ability to filter and decode Bitcoin messages.  However, the purpose of the message capture added in this PR is to assist with debugging, primarily for new developers looking to improve their knowledge of the Bitcoin Protocol.  This drives the design in a different direction than Wireshark, in two different ways.  First, this tool must be convenient and simple to use.  Using an external tool, like Wireshark, requires setup and interpretation of the results.  To a new user who doesn't necessarily know what to expect, this is unnecessary difficulty.  This tool, on the other hand, "just works".  Turn on the command line flag, run your node, run the script, read the JSON.  Second, because this tool is being used for debugging, we want it to be as close to the true behavior of the node as possible.  A lot can happen in the SocketHandler thread that would be missed by Wireshark.

  Additionally, if we are to use Wireshark, we are at the mercy of whoever it maintaining the protocol in Wireshark, both as to it being accurate and recent.  As can be seen by the **many** previous attempts to include Bitcoin in Wireshark (google "bitcoin dissector") this is easier said than done.

  Lastly, I truly believe that this tool will be used significantly more by being included in the codebase.  It's just that much more discoverable.

ACKs for top commit:
  MarcoFalke:
    re-ACK bff7c66e67 only some minor changes: 👚
  jnewbery:
    utACK bff7c66e67
  theStack:
    re-ACK bff7c66e67

Tree-SHA512: e59e3160422269221f70f98720b47842775781c247c064071d546c24fa7a35a0e5534e8baa4b4591a750d7eb16de6b4ecf54cbee6d193b261f4f104e28c15f47
2021-02-02 13:11:28 +01:00
MarcoFalke
4c55f92c76
Merge #20954: test: Declare nodes type in test_framework.py.
5353b0c64d Change type definitions for "chain" and "setup_clean_chain" from type comments to Python 3.6+ types. Additionally, set type for "nodes". (Kiminuo)

Pull request description:

  ### Motivation

  When I wanted to understand better https://github.com/bitcoin/bitcoin/pull/19145/files#diff-4bebbd3b112dc222ea7e75ef051838ceffcee63b9e9234a98a4cc7251d34451b test, I noticed that navigation in PyCharm/VS Code did not work for `nodes` variable. I think this is frustrating, especially for newcomers.

  ### Summary

  * This PR modifies Python 3.5 [type comments](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#variables) to Python 3.6+ types and adds a proper type for `nodes` [instance attribute](https://mypy.readthedocs.io/en/stable/class_basics.html#instance-and-class-attributes).
  * This PR does not change behavior.
  * This PR is intentionally very small, if the concept is accepted, a follow-up PRs can be more ambitious.

  ### End result

  1. Open `test/functional/feature_abortnode.py`
  2. Move your caret to: `self.nodes[0].generate[caret here](3)`
  3. Use "Go to definition" [F12] should work now.

  I have tested this on PyCharm (Windows, Ubuntu) and VS Code (Windows, Ubuntu).

  Note: Some `TestNode` methods (e.g. `self.nodes[0].getblock(...)` ) use `__call__` mechanism and navigation does not work for them even with this PR.

ACKs for top commit:
  laanwj:
    ACK 5353b0c64d
  theStack:
    ACK 5353b0c64d

Tree-SHA512: 821773f052ab9b2889dc357d38c59407a4af09e3b86d7134fcca7d78e5edf3a5ede9bfb37595ea97caf9ebfcbda372bcf73763b7f89b0677670f21b3e396a12b
2021-01-31 09:28:21 +01:00
Andrew Chow
3641597d7e tests: Don't make any wallets unless wallet is required 2021-01-27 12:52:46 -05:00
Andrew Chow
b9b88f57a9 Skip legacy wallet reliant tests if BDB is not compiled 2021-01-27 12:52:46 -05:00
Andrew Chow
6f36242389 tests: Set descriptors default based on compilation
Determines whether descriptors should be used based on whether the
--descriptor or --legacy-wallet option is set,
and the compiled support. If no option is set and both BDB and SQLite
are available, it defaults to legacy.

This is used to switch descriptor agnostic tests between descriptors and
legacy wallet.
2021-01-27 12:52:46 -05:00
Troy Giorshev
e4f378a505 Add capture parser
This commit adds contrib/message-capture/message-capture-parser.py, a python
script to be used alongside -capturemessages to parse the captured
messages.

It is complete with arguments and will parse any file given, sorting the
messages in the files when creating the output.  If an output file is
specified with -o or --output, it will dump the messages in json format
to that file, otherwise it will print to stdout.

The small change to the unused msg_generic is to bring it in line with
the other message classes, purely to avoid a bug in the future.
2021-01-23 16:01:39 -05:00
Sebastian Falbesoner
de85af5cce test: store subversion (user agent) as string in msg_version 2021-01-23 15:04:35 +01:00
MarcoFalke
11cbd4bb54
Merge #17556: test: Change feature_config_args.py not to rely on strange regtest=0 behavior
ff44cae279 test: Change feature_config_args.py not to rely on strange regtest=0 behavior (Russell Yanofsky)

Pull request description:

  Update test to simply generate a normal mainnet configuration file instead of using a crazy setup where a regtest=1 config file using an includeconf in the [regtest] section includes another config file that specifies regtest=0, retroactively switching the network to mainnet.

  This setup was fragile and only worked because the triggered InitError happened early enough that none of the ignored [regtest] options mattered (only affecting log output).

  This change was originally made as part of #17493

Top commit has no ACKs.

Tree-SHA512: 3f77305454f04438493dfc2abd78a00434b30869454d1c3f54587b9c1f63239c49c90fb3b4d3a777ad130f2184e0f2dac87cee4cd23c50f1b3496a375943da01
2021-01-21 16:51:19 +01:00
Kiminuo
5353b0c64d Change type definitions for "chain" and "setup_clean_chain" from type comments to Python 3.6+ types. Additionally, set type for "nodes". 2021-01-18 09:01:07 +01:00
MarcoFalke
fa0aa87071
rpc: Return wtxid from testmempoolaccept 2021-01-12 18:43:43 +01:00
MarcoFalke
6af013792f
Merge #19315: [tests] Allow outbound & block-relay-only connections in functional tests.
b4dd2ef800 [test] Test the add_outbound_p2p_connection functionality (Amiti Uttarwar)
602e69e427 [test] P2PBlocksOnly - Test block-relay-only connections. (Amiti Uttarwar)
8bb6beacb1 [test/refactor] P2PBlocksOnly - Extract transaction violation test into helper. (Amiti Uttarwar)
99791e7560 [test/refactor] P2PBlocksOnly - simplify transaction creation using blocktool helper. (Amiti Uttarwar)
3997ab9154 [test] Add test framework support to create outbound connections. (Amiti Uttarwar)
5bc04e8837 [rpc/net] Introduce addconnection to test outbounds & blockrelay (Amiti Uttarwar)

Pull request description:

  The existing functional test framework uses the `addnode` RPC to spin up manual connections between bitcoind nodes. This limits our ability to add integration tests for our networking code, which often executes different code paths for different connection types.

  **This PR enables creating `outbound` & `block-relay-only` P2P connections in the functional tests.** This allows us to increase our p2p test coverage, since we can now verify expectations around these connection types.

  This builds out the [prototype](https://github.com/bitcoin/bitcoin/issues/14210#issuecomment-527421978) proposed by ajtowns in #14210. 🙌🏽

  An overview of this branch:
  - introduces a new test-only RPC function `addconnection` which initiates opening an `outbound` or `block-relay-only` connection. (conceptually similar to `addnode` but for different connection types & restricted to regtest)
  - adds `test_framework` support so a mininode can open an `outbound`/`block-relay-only` connection to a `P2PInterface`/`P2PConnection`.
  - updates `p2p_blocksonly` tests to create a `block-relay-only` connection & verify expectations around transaction relay.
  - introduces `p2p_add_connections` test that checks the behaviors of the newly introduced `add_outbound_p2p_connection` test framework function.

  With these changes, there are many more behaviors that we can add integration tests for. The blocksonly updates is just one example.

  Huge props to ajtowns for conceiving the approach & providing me feedback as I've built out this branch. Also thank you to jnewbery for lots of thoughtful input along the way.

ACKs for top commit:
  troygiorshev:
    reACK b4dd2ef800
  jnewbery:
    utACK b4dd2ef800
  MarcoFalke:
    Approach ACK b4dd2ef800 🍢

Tree-SHA512: d1cba768c19c9c80e6a38b1c340cc86a90701b14772c4a0791c458f9097f6a4574b4a4acc7d13d6790c7b1f1f197e2c3d87996270f177402145f084ef8519a6b
2021-01-11 21:06:57 +01:00
MarcoFalke
9c0b76c709
Merge #20876: test: Replace getmempoolentry with testmempoolaccept in MiniWallet
faabc26a61 test: Replace getmempoolentry with testmempoolaccept in MiniWallet (MarcoFalke)

Pull request description:

  This is a refactor to not use the return value of `sendrawtransaction` and `getmempoolentry` with the goal that submitting the tx to the mempool will become optional.

ACKs for top commit:
  mjdietzx:
    ACK faabc26a61

Tree-SHA512: 4260a59e65fed1c807530dad23f1996ba6e881bcce91995f5b498a0be6001f67b3e0251507c8801750fe105410147c68bb2f393ebe678c6a3a4d045e5d72fc19
2021-01-11 08:57:28 +01:00
MarcoFalke
faabc26a61
test: Replace getmempoolentry with testmempoolaccept in MiniWallet 2021-01-08 09:07:33 +01:00
Wladimir J. van der Laan
d7e2401c62
Merge #18077: net: Add NAT-PMP port forwarding support
a191e23b8e doc: Add release notes (Hennadii Stepanov)
ae749d12dd doc: Add libnatpmp stuff (Hennadii Stepanov)
e28f9be87a ci: Add libnatpmp-dev package to some builds (Hennadii Stepanov)
5a0185b6c9 gui: Add NAT-PMP network option (Hennadii Stepanov)
a39f7336a3 net: Add -natpmp command line option (Hennadii Stepanov)
28acffd9d5 net: Add NAT-PMP to port mapping loop (Hennadii Stepanov)
a8d9f275d0 net: Add libnatpmp support (Hennadii Stepanov)
58e8364dcd gui: Apply port mapping changes on dialog exit (Hennadii Stepanov)
cf151cc68c scripted-diff: Rename UPnP stuff (Hennadii Stepanov)
4e91b1e24d net: Add flags for port mapping protocols (Hennadii Stepanov)
8b50d1b5bb net: Keep trying to use UPnP when -upnp=1 (Hennadii Stepanov)
28e2961fd6 refactor: Replace magic number with named constant (Hennadii Stepanov)
02ccf69dd6 refactor: Move port mapping code to its own module (Hennadii Stepanov)

Pull request description:

  Close #11902
  This PR is an alternative to:
  - #12288
  - #15717

  To compile with NAT-PMP support on Ubuntu [`libnatpmp-dev`](https://packages.ubuntu.com/source/bionic/libnatpmp) should be available.

  Log excerpt:
  ```
  2020-02-05T20:12:28Z [mapport] NAT-PMP: public address = 95.164.65.194
  2020-02-05T20:12:28Z [mapport] AddLocal(95.164.65.194:18333,3)
  2020-02-05T20:12:28Z [mapport] NAT-PMP: port mapping successful.
  ```

  See: [`libnatpmp`](https://miniupnp.tuxfamily.org/libnatpmp.html)

  ---

  Some follow-ups are out of this PR's scope:
  - mention NAT-PMP library in the version message
  - ~integrate NAT-PMP into the GUI~ (already [added](https://github.com/bitcoin/bitcoin/pull/18077#issuecomment-589405068))

ACKs for top commit:
  laanwj:
    Tested and code review ACK a191e23b8e

Tree-SHA512: 10e19267c21bf30f20ff1abfc882d526049f0e790b95e12f109dc2bed7c0aef45de03eaf967f4e667e7509be04f1873a5c508087393d947205f3aab2ad6d7cf1
2021-01-07 19:41:55 +01:00
Amiti Uttarwar
3997ab9154 [test] Add test framework support to create outbound connections.
In the interest of increasing our P2P test coverage, add support to create
full-relay or block-relay-only connections. To support this, a P2P connection
spins up a listening thread & uses a callback to trigger the node initiating
the connection.

Co-authored-by: Anthony Towns <aj@erisian.com.au>
2021-01-07 10:15:56 -08:00
Wladimir J. van der Laan
b6a71b80d2
Merge #19055: Add MuHash3072 implementation
9815332d51 test: Change MuHash Python implementation to match cpp version again (Fabian Jahr)
01297fb3ca fuzz: Add MuHash consistency fuzz test (Fabian Jahr)
b111410914 test: Add MuHash3072 fuzz test (Fabian Jahr)
c122527385 bench: Add Muhash benchmarks (Fabian Jahr)
7b1242229d test: Add MuHash3072 unit tests (Fabian Jahr)
adc708c98d crypto: Add MuHash3072 implementation (Fabian Jahr)
0b4d290bf5 crypto: Add Num3072 implementation (Fabian Jahr)
589f958662 build: Check for 128 bit integer support (Fabian Jahr)

Pull request description:

  This is the first split of #18000 which implements the Muhash algorithm and uses it to calculate the UTXO set hash in `gettxoutsetinfo`.

ACKs for top commit:
  laanwj:
    Code review ACK 9815332d51

Tree-SHA512: 4bc090738f0e3d80b74bdd8122e24a8ce80121120fd37c7e4335a73e7ba4fcd7643f2a2d559e2eebf54b8e3a3bd5f12cfb27ba61ded135fda210a07a233eae45
2021-01-07 17:57:17 +01:00
Hennadii Stepanov
a39f7336a3
net: Add -natpmp command line option 2021-01-07 18:07:09 +02:00
Sawyer Billings
e8640849c7
doc: Use https URLs where possible 2021-01-04 12:23:16 +08:00
MarcoFalke
fa0074e2d8
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2020-12-31 09:45:41 +01:00
fanquake
0e1b57b4bb
Merge #20763: test: Fix comment typo in BitcoinTestFramework
40fdb2a212 test: Fix Comment Typo in BitcoinTestFramework (Joel Klabo)

Pull request description:

  Missing "override" in comment describing use of set_test_params

ACKs for top commit:
  michaelfolkson:
    ACK 40fdb2a212

Tree-SHA512: bf893a0d5f8dc86a3ec2eaf48cd7c0f0f832f3b3d254b3d99953336db7e294571b1d2c8686030bf8a27cbe67b1a85a54e53ebefb2e57d6d8d6ac864a15dce4e7
2020-12-27 17:51:15 +08:00
Joel Klabo
40fdb2a212 test: Fix Comment Typo in BitcoinTestFramework
Missing "override" in comment describing use of set_test_params
2020-12-24 10:54:56 -08:00
MarcoFalke
cc592a85ea
Merge #20189: test: Switch to BIP341's suggested scheme for outputs without script
812baaa1f8 Switch to BIP341's suggested scheme for outputs without script (Pieter Wuille)

Pull request description:

  BIP341 suggests using Hash<sub>TapTweak</sub>(pubkey) to derive the tweak in case of key-only outputs. The functional test framework currently uses Hash<sub>TapTweak</sub>(pubkey || 0x00...00) instead. Change this.

  There is no technical reason to prefer one over the other, but in case someone looks at it for inspiration, it's better to be consistent with the BIP.

ACKs for top commit:
  laanwj:
    ACK 812baaa1f8
  instagibbs:
    ACK 812baaa1f8

Tree-SHA512: 02576c38776ec786255f49d7edecdb1ed8a9dcf0f547d58c23099588b4c3296edf279b103a6eb80e0f07d3c5ee9743f67d152f5244fd63adc6613b004f6969ed
2020-12-24 07:56:33 +01:00
Fabian Jahr
9815332d51
test: Change MuHash Python implementation to match cpp version again 2020-12-22 01:48:34 +01:00
MarcoFalke
fab46b34f4
test: Fix restart node race 2020-12-17 15:06:31 +01:00
Wladimir J. van der Laan
5b6f970e3f
Merge #20171: Add functional test test_txid_inv_delay
bc4a230087 Remove redundant p2p lock tacking for tx download functional tests (Antoine Riard)
d3b5eac9a9 Add mutation for functional test test_preferred_inv (Antoine Riard)
06efb3163c Add functional test test_txid_inv_delay (Antoine Riard)
a07910abcd test: Makes wtxidrelay support a generic P2PInterface option (Antoine Riard)

Pull request description:

  This is a simple functional test to increase coverage of #19988, checking that txid announcements from txid-relay peers are delayed by TXID_RELAY_DELAY, assuming we have at least another wtxid-relay peer.

  You can verify new test with the following diff :

  ```
  diff --git a/src/net_processing.cpp b/src/net_processing.cpp
  index f14db379f..2a2805df5 100644
  --- a/src/net_processing.cpp
  +++ b/src/net_processing.cpp
  @@ -773,7 +773,7 @@ void PeerManager::AddTxAnnouncement(const CNode& node, const GenTxid& gtxid, std
       auto delay = std::chrono::microseconds{0};
       const bool preferred = state->fPreferredDownload;
       if (!preferred) delay += NONPREF_PEER_TX_DELAY;
  -    if (!gtxid.IsWtxid() && g_wtxid_relay_peers > 0) delay += TXID_RELAY_DELAY;
  +    //if (!gtxid.IsWtxid() && g_wtxid_relay_peers > 0) delay += TXID_RELAY_DELAY;
       const bool overloaded = !node.HasPermission(PF_RELAY) &&
           m_txrequest.CountInFlight(nodeid) >= MAX_PEER_TX_REQUEST_IN_FLIGHT;
       if (overloaded) delay += OVERLOADED_PEER_TX_DELAY;
  ```

ACKs for top commit:
  laanwj:
    ACK bc4a230087

Tree-SHA512: 150e806bc5289feda94738756ab375c7fdd23c80c12bd417d3112043e26a91a717dc325a01079ebd02a88b90975ead5bd397ec86eb745c7870ebec379a8aa711
2020-12-16 18:45:11 +01:00
MarcoFalke
fa918dd537
test: Use Popen.wait instead of RPC in assert_start_raises_init_error 2020-12-10 12:36:41 +01:00
Wladimir J. van der Laan
42ed7f51fa
Merge #20606: Remove unused bits from service flags enum
fa40168ab3 Remove unused bits from service flags enum (MarcoFalke)

Pull request description:

  Remove service bits that haven't been observed on the active network for years and won't ever be observed on the network with this meaning. Keeping this dead assignment in our source code forever doesn't add any value.

  I somehow forgot to do this in commit fa0d0ff6e1.

ACKs for top commit:
  laanwj:
    Code review ACK fa40168ab3
  practicalswift:
    cr ACK fa40168ab3
  fanquake:
    ACK fa40168ab3

Tree-SHA512: 376e5ac05940493cf2209fea60515c843e978c4b476f2524f6bf7a37a646d237c3ddcf6c0fa23641f9ba550f625609703d9b51b4be631a7f2a90e1092b557232
2020-12-09 16:37:56 +01:00
Pieter Wuille
1583498fb6 Send and require SENDADDRV2 before VERACK
See the corresponding BIP change: https://github.com/bitcoin/bips/pull/1043
2020-12-08 09:40:10 -08:00
MarcoFalke
fa40168ab3
Remove unused bits from service flags enum 2020-12-08 18:36:51 +01:00
MarcoFalke
f17e8ba3a1
Merge #20207: Follow-up extra comments on taproot code and tests
2d8099c713 Mention units of MAX_STANDARD_ policy constants (Pieter Wuille)
84e29c7c01 Mention in validation that IsWitnessStandard tests for P2TR (Pieter Wuille)
f867cbcc26 Clean up assets test minimizer LDFLAGS (Pieter Wuille)
ea0e78677b Document additional IsWitnessStandard behavior (Pieter Wuille)
6040de9a46 Add comments on CPubKey::IsValid (Pieter Wuille)
8dbb7de67c Add comments to VerifyTaprootCommitment (Pieter Wuille)
cdf900cbf2 Document need_vin_vout_mismatch argument to make_spender (Pieter Wuille)
18246ed5f0 Fix and improve taproot_construct comments (Pieter Wuille)

Pull request description:

  Addressing some review comments raised here: https://github.com/bitcoin/bitcoin/pull/19953#pullrequestreview-512238027 and https://github.com/bitcoin/bitcoin/pull/19953#pullrequestreview-513499921

ACKs for top commit:
  jonatack:
    ACK 2d8099c per `git range-diff 5009159 4f10965 2d8099c`
  ariard:
    ACK 2d8099c, only changes are comment light improvements on IsValid/IsWitnessStandard.

Tree-SHA512: c4881546c379ea8efc7ef99a43cbf3b9cd3f9dde5fd97a07ee66f2b593c78aef0bd8784853c5c9c737b66c269241a1048bbbdd6c964a3d872efd8ba0ec410b68
2020-12-01 15:11:51 +01:00
Amiti Uttarwar
3ebde2143a [test] Fix wait condition in disconnect_p2ps
MY_SUBVERSION is defined in messages.py as a byte string, but here we were
comparing this value to the value returned by the RPC. Convert to ensure the
types match.
2020-11-27 14:54:55 -08:00
Pieter Wuille
18246ed5f0 Fix and improve taproot_construct comments 2020-11-26 14:56:25 -08:00
Sjors Provoost
b87caf10b5
test: add is_bdb_compiled helper 2020-11-23 11:30:55 +01:00
Wladimir J. van der Laan
04670ef81e
Merge #20385: test: run mempool_spend_coinbase.py even with wallet disabled
21f2433601 test: run mempool_spend_coinbase.py even with wallet disabled (Michael Dietz)

Pull request description:

  Run the mempool spend coinbase test even when the wallet was not compiled, as proposed in #20078.

ACKs for top commit:
  laanwj:
    ACK 21f2433601

Tree-SHA512: 301582c04376371cfa8f1ebb2418a4341b42ddcd9ad4f48b58bcf888d867a97bdc409972856b67a8339ac5e60124aefee82a049b4f7fc6bca7a18d7e92e090be
2020-11-19 16:40:02 +01:00
Michael Dietz
21f2433601 test: run mempool_spend_coinbase.py even with wallet disabled 2020-11-16 09:05:34 -06:00
Wladimir J. van der Laan
c48e788246
Merge #18836: wallet: upgradewallet fixes and additional tests
5f9c0b6360 wallet: Remove -upgradewallet from dummywallet (MarcoFalke)
a314271f08 test: Remove unused wallet.dat (MarcoFalke)
bf7635963c tests: Test specific upgradewallet scenarios and that upgrades work (Andrew Chow)
4b418a9dec test: Add test_framework/bdb.py module for inspecting bdb files (Andrew Chow)
092fc43485 tests: Add a sha256sum_file function to util (Andrew Chow)
0bd995aa19 wallet: upgrade the CHDChain version number when upgrading to split hd (Andrew Chow)
8e32e1c41c wallet: remove nWalletMaxVersion (Andrew Chow)
bd7398cc62 wallet: have ScriptPubKeyMan::Upgrade check against the new version (Andrew Chow)
5f720544f3 wallet: Add GetClosestWalletFeature function (Andrew Chow)
842ae3842d wallet: Add utility method for CanSupportFeature (Andrew Chow)

Pull request description:

  This PR cleans up the wallet upgrade mechanism a bit, fixes some probably bugs, and adds more test cases.

  The `nWalletMaxVersion` member variable has been removed as it made `CanSupportFeature` unintuitive and was causing a couple of bugs. The reason this was introduced originally was to allow a wallet upgrade to only occur when the new feature is first used. While this makes sense for the old `-upgradewallet` option, for an RPC, this does not quite make sense. It's more intuitive for an upgrade to occur if possible if the `upgradewallet` RPC is used as that's an explicit request to upgrade a particular wallet to a newer version. `nWalletMaxVersion` was only relevant for upgrades to `FEATURE_WALLETCRYPT` and `FEATURE_COMPRPUBKEY` both of which are incredibly old features. So for such wallets, the behavior of `upgradewallet` will be that the feature is enabled immediately without the wallet needing to be encrypted at that time (note that `FEATURE_WALLETCRYPT` indicates support for encryption, not that the wallet is encrypted) or for a new key to be generated.

  `CanSupportFeature` would previously indicate whether we could upgrade to `nWalletMaxVersion` not just whether the current wallet version supported a feature. While this property was being used to determine whether we should upgrade to HD and HD chain split, it was also causing a few bugs. Determining whether we should upgrade to HD or HD chain split is resolved by passing into `ScriptPubKeyMan::Upgrade` the version we are upgrading to and checking against that. By removing `nWalletMaxVersion` we also fix a bug where you could upgrade to HD chain split without the pre-split keypool.

  `nWalletMaxVersion` was also the version that was being reported by `getwalletinfo` which meant that the version reported was not always consistent across restarts as it depended on whether `upgradewallet` was used. Additionally to make the wallet versions consistent with actually supported versions, instead of just setting the wallet version to whatever is given to `upgradewallet`, we normalize the version number to the closest supported version number. For example, if given 150000, we would store and report 139900.

  Another bug where CHDChain was not being upgraded to the version supporting HD chain split is also fixed by this PR.

  Lastly several more tests have been added. Some refactoring to the test was made to make these tests easier. These tests check specific upgrading scenarios, such as from non-HD (version 60000) to HD to pre-split keypool. Although not specifically related to `upgradewallet`, `UpgradeKeyMetadata` is now being tested too.

  Part of the new tests is checking that the wallet files are identical before and after failed upgrades. To facilitate this, a utility function `sha256sum_file` has been added. Another part of the tests is to examine the wallet file itself to ensure that the records in the wallet.dat file have been correctly modified. So a new `bdb.py` module has been added to deserialize the BDB db of the wallet.dat file. This format isn't explicitly documented anywhere, but the code and comments in BDB's source code in file `dbinc/db_page.h` describe it. This module just dumps all of the fields into a dict.

ACKs for top commit:
  MarcoFalke:
    approach ACK 5f9c0b6360
  laanwj:
    Code review ACK 5f9c0b6360
  jonatack:
    ACK 5f9c0b6360, approach seems fine, code review, only skimmed the test changes but they look well done, rebased on current master, debug built and verified the `wallet_upgradewallet.py` test runs green both before and after running `test/get_previous_releases.py -b v0.19.1 v0.18.1 v0.17.2 v0.16.3 v0.15.2`

Tree-SHA512: 7c4ebf420850d596a586cb6dd7f2ef39c6477847d12d105fcd362abb07f2a8aa4f7afc5bfd36cbc8b8c72fcdd1de8d2d3f16ad8e8ba736b6f4f31f133fe5feba
2020-11-16 11:03:25 +01:00
Wladimir J. van der Laan
4fd37d0a10
Merge #20292: test: Fix intermittent feature_taproot issue
fab900802d ci: Bump timeout factor (MarcoFalke)
50eb0c2512 Small improvements to the Taproot functional tests (Pieter Wuille)
fac865b72d test: Fix intermittent feature_taproot issue (MarcoFalke)
fa1dea19fc test: Fix deser issue in create_block (MarcoFalke)
fa762a3fd4 test: Remove unused unnamed parameter from block.serialize call (MarcoFalke)

Pull request description:

  This fixes three bugs. Also, fix some unrelated code style issues.

  Please refer to the commit messages for more information.

ACKs for top commit:
  laanwj:
    Code review ACK fab900802d

Tree-SHA512: 4e22c240cf345710f3b21fc63243126b90014b3656d0865ff87156e958dd1442e6572c6c0a5701dbbe503eee931a0ceb66eeeb3553137f3d1f5afd27a9f9cada
2020-11-09 15:47:04 +01:00
Stepan Snigirev
568a1d7261 fix ecdsa verify in test framework 2020-11-05 23:16:55 +01:00
Andrew Chow
4b418a9dec test: Add test_framework/bdb.py module for inspecting bdb files
For upgrade tests and possibly other tests, it is useful to inspect the
bdb file for the wallet (i.e. the wallet.dat file).
test_framework/bdb.py is an implementation of bdb file deserialization
specific for Bitcoin Core's usage.
2020-11-04 12:15:29 -05:00
Andrew Chow
092fc43485 tests: Add a sha256sum_file function to util 2020-11-04 12:15:25 -05:00
Pieter Wuille
50eb0c2512 Small improvements to the Taproot functional tests
The "whitelist" and "connect_nodes" is not needed in feature_taproot.py,
so remove it.

The changes to key.py are required when running the unit tests from the
test folder. Failure on current master:

[test]$ python -m unittest functional/test_framework/key.py
.E
======================================================================
ERROR: test_schnorr_testvectors (functional.test_framework.key.TestFrameworkKey)
Implement the BIP340 test vectors (read from bip340_test_vectors.csv).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/functional/test_framework/key.py", line 526, in test_schnorr_testvectors
    with open(os.path.join(sys.path[0], 'test_framework', 'bip340_test_vectors.csv'), newline='', encoding='utf8') as csvfile:
FileNotFoundError: [Errno 2] No such file or directory: 'test/test_framework/bip340_test_vectors.csv'

----------------------------------------------------------------------
Ran 2 tests in 0.775s

FAILED (errors=1)
2020-11-03 12:26:17 +01:00
MarcoFalke
fa1dea19fc
test: Fix deser issue in create_block
Without the fix a hex-string can not be parsed:

File "./test/functional/test_framework/blocktools.py", line 82, in create_block
    txo.deserialize(io.BytesIO(tx))
TypeError: a bytes-like object is required, not 'str'

Also, remove io import and repace it with our FromHex() helper
2020-11-03 12:25:55 +01:00