Commit graph

844 commits

Author SHA1 Message Date
MarcoFalke
fac62e6ff5
test: Delete generate* calls from TestNode 2021-10-06 13:41:08 +02:00
W. J. van der Laan
9e530c6352
Merge bitcoin/bitcoin#20487: Add syscall sandboxing using seccomp-bpf (Linux secure computing mode)
4747da3a5b Add syscall sandboxing (seccomp-bpf) (practicalswift)

Pull request description:

  Add experimental syscall sandboxing using seccomp-bpf (Linux secure computing mode).

  Enable filtering of system calls using seccomp-bpf: allow only explicitly allowlisted (expected) syscalls to be called.

  The syscall sandboxing implemented in this PR is an experimental feature currently available only under Linux x86-64.

  To enable the experimental syscall sandbox the `-sandbox=<mode>` option must be passed to `bitcoind`:

  ```
    -sandbox=<mode>
         Use the experimental syscall sandbox in the specified mode
         (-sandbox=log-and-abort or -sandbox=abort). Allow only expected
         syscalls to be used by bitcoind. Note that this is an
         experimental new feature that may cause bitcoind to exit or crash
         unexpectedly: use with caution. In the "log-and-abort" mode the
         invocation of an unexpected syscall results in a debug handler
         being invoked which will log the incident and terminate the
         program (without executing the unexpected syscall). In the
         "abort" mode the invocation of an unexpected syscall results in
         the entire process being killed immediately by the kernel without
         executing the unexpected syscall.
  ```

  The allowed syscalls are defined on a per thread basis.

  I've used this feature since summer 2020 and I find it to be a helpful testing/debugging addition which makes it much easier to reason about the actual capabilities required of each type of thread in Bitcoin Core.

  ---

  Quick start guide:

  ```
  $ ./configure
  $ src/bitcoind -regtest -debug=util -sandbox=log-and-abort
  …
  2021-06-09T12:34:56Z Experimental syscall sandbox enabled (-sandbox=log-and-abort): bitcoind will terminate if an unexpected (not allowlisted) syscall is invoked.
  …
  2021-06-09T12:34:56Z Syscall filter installed for thread "addcon"
  2021-06-09T12:34:56Z Syscall filter installed for thread "dnsseed"
  2021-06-09T12:34:56Z Syscall filter installed for thread "net"
  2021-06-09T12:34:56Z Syscall filter installed for thread "msghand"
  2021-06-09T12:34:56Z Syscall filter installed for thread "opencon"
  2021-06-09T12:34:56Z Syscall filter installed for thread "init"
  …
  # A simulated execve call to show the sandbox in action:
  2021-06-09T12:34:56Z ERROR: The syscall "execve" (syscall number 59) is not allowed by the syscall sandbox in thread "msghand". Please report.
  …
  Aborted (core dumped)
  $
  ```

  ---

  [About seccomp and seccomp-bpf](https://en.wikipedia.org/wiki/Seccomp):

  > In computer security, seccomp (short for secure computing mode) is a facility in the Linux kernel. seccomp allows a process to make a one-way transition into a "secure" state where it cannot make any system calls except exit(), sigreturn(), and read() and write() to already-open file descriptors. Should it attempt any other system calls, the kernel will terminate the process with SIGKILL or SIGSYS. In this sense, it does not virtualize the system's resources but isolates the process from them entirely.
  >
  > […]
  >
  > seccomp-bpf is an extension to seccomp that allows filtering of system calls using a configurable policy implemented using Berkeley Packet Filter rules. It is used by OpenSSH and vsftpd as well as the Google Chrome/Chromium web browsers on Chrome OS and Linux. (In this regard seccomp-bpf achieves similar functionality, but with more flexibility and higher performance, to the older systrace—which seems to be no longer supported for Linux.)

ACKs for top commit:
  laanwj:
    Code review and lightly tested ACK 4747da3a5b

Tree-SHA512: e1c28e323eb4409a46157b7cc0fc29a057ba58d1ee2de268962e2ade28ebd4421b5c2536c64a3af6e9bd3f54016600fec88d016adb49864b63edea51ad838e17
2021-10-04 22:45:43 +02:00
practicalswift
4747da3a5b Add syscall sandboxing (seccomp-bpf) 2021-10-01 13:51:10 +00:00
MarcoFalke
35a31d5f7e
Merge bitcoin/bitcoin#23136: test: update fee rate assertion helper in the functional test framework
b658d7d5c5 test: update assert_fee_amount() in test_framework/util.py (Jon Atack)

Pull request description:

  Follow-up to 42e1b5d979 (#12486).
  - update call to `round()` with our utility function `satoshi_round()` to avoid intermittent test failures
  - rename `fee_per_kB` to `feerate_BTC_kvB` for precision
  - store division result in `feerate_BTC_vB`

  Possibly resolves #19418.

ACKs for top commit:
  meshcollider:
    utACK b658d7d5c5

Tree-SHA512: f124ded98c913f98782dc047a85a05d3fdf5f0585041fa81129be562138f6261ec1bd9ee2af89729028277e75b591b0a7ad50244016c2b2fa935c6e400523183
2021-10-01 10:54:23 +02:00
Jon Atack
b658d7d5c5
test: update assert_fee_amount() in test_framework/util.py
- update call to round() with satoshi_round() to avoid intermittent test failures
- rename fee_per_kB to feerate_BTC_kvB for precision
- store division result in feerate_BTC_vB
2021-09-30 16:38:55 +02:00
W. J. van der Laan
1cf7fb9fd6
Merge bitcoin/bitcoin#23104: log: Avoid breaking single log lines over multiple lines in the log file
2222c04e1b log: Adjust coin selection log string (MarcoFalke)
fa6c1e850f test: Fix typos in tests (MarcoFalke)
faeae2980f log: Avoid broken DEBUG_LOCKORDER log (MarcoFalke)
faffaa85cd log: Avoid broken SELECTCOINS log (MarcoFalke)

Pull request description:

  Follow up to commit d8b4b3077f

ACKs for top commit:
  laanwj:
    re-ACK 2222c04e1b
  practicalswift:
    cr ACK 2222c04e1b

Tree-SHA512: e0daf76815a1b7c4898ceffedeaf7ede093223abf709874f9a0d78c8e41551c14e8b56d055c8fdf06ec698df64e67dfc168bbd8716131b23648d1d1294fa6636
2021-09-30 14:42:11 +02:00
MarcoFalke
fa6c1e850f
test: Fix typos in tests 2021-09-29 18:47:45 +02:00
MarcoFalke
33e31f8df9
Merge bitcoin/bitcoin#23079: test: use MiniWallet for p2p_filter.py
cfdb6baa22 test: use MiniWallet for p2p_filter.py (Sebastian Falbesoner)
6fc2cd3f09 test: introduce helper to create random P2WPKH scriptPubKeys (Sebastian Falbesoner)
aa26797f69 test: MiniWallet: add `send_to` method to create arbitrary txouts (Sebastian Falbesoner)

Pull request description:

  This PR enables one more of the non-wallet functional tests (p2p_filter.py) to be run even with the Bitcoin Core wallet disabled by using the MiniWallet instead, as proposed in #20078.
  For this purpose, a MiniWallet method `send_to` is introduced first, which allows to create arbitrary outputs (scriptPubKey/amount). Note that the implementation for this is already present in feature_rbf.py (recently added in PR #22998), i.e. it is simply moved to the MiniWallet interface.

ACKs for top commit:
  laanwj:
    Code review ACK cfdb6baa22

Tree-SHA512: 13b063631f0d7af065b7757cfe8b47c9be6cb9850ac5db2968a2bba4f5a18cdc9f89173a9b03971545356225082042f5fdbe49d3036027d18e8b7eb042d04f5e
2021-09-29 09:13:07 +02:00
MarcoFalke
fa54efda9b
test: pep-8 touched test
Can be reviewed with "--word-diff-regex=.".
2021-09-28 15:48:55 +02:00
MarcoFalke
fa46768059
test: Remove unused and confusing main parameter from script_util
Bitcoin script opcodes are equal on all chains (main and test) anyway.

Can be reviewed with "--word-diff-regex=.".
2021-09-28 15:46:57 +02:00
Sebastian Falbesoner
6fc2cd3f09 test: introduce helper to create random P2WPKH scriptPubKeys 2021-09-27 13:55:25 +02:00
Sebastian Falbesoner
aa26797f69 test: MiniWallet: add send_to method to create arbitrary txouts
With this new method, outputs to an arbitrary scriptPubKey/amount can
be created. Note that the implementation was already present in the
test feature_rbf.py and is just moved to the MiniWallet interface, in
order to enable other tests to also use it.
2021-09-27 13:55:25 +02:00
MarcoFalke
fad02274ba
test: Remove Windows workaround in authproxy
This reverts commit fab9899204.
2021-09-25 09:33:16 +02:00
merge-script
442e32e117
Merge bitcoin/bitcoin#22817: test: Avoid race after connect_nodes
fa04f26aa7 test: Avoid race after connect_nodes (MarcoFalke)

Pull request description:

  Wait until the connection is fully established on both sides (verack). Fixes #22714

ACKs for top commit:
  kiminuo:
    utACK fa04f26aa7

Tree-SHA512: bc2c44b44b688086ff84046924cf5251dd625584e93ce8fa17de27023855b32f3bb55109b846abbcec775e2836c7f3c5a81d6b4aff7c4ac065b9aefa044c1883
2021-09-25 08:50:52 +02:00
W. J. van der Laan
01b5cfb951
Merge bitcoin/bitcoin#23047: test: Use MiniWallet in mempool_persist
faae0988d6 test: Check other fields are loaded correctly as well (MarcoFalke)
fa4db92617 test: Remove unused self.connect_nodes (MarcoFalke)
fafb7b7a89 test: pep8 (MarcoFalke)
fa32cb2467 test: Use MiniWallet in mempool_persist (MarcoFalke)
faca688a85 test: Add MiniWallet get_descriptor function (MarcoFalke)

Pull request description:

ACKs for top commit:
  laanwj:
    Code review ACK faae0988d6

Tree-SHA512: 6124f16ee1f3f416c50dc07aebe8846ff7e2b7c8e5dd84f9517cb5f1df021b9e57ed7c7e17bc099a37c663cd93f6d417c5e0622c0b359956403d53e705eb5549
2021-09-24 17:09:44 +02:00
merge-script
8e9801bfc4
Merge bitcoin/bitcoin#22818: test: Activate all regtest softforks at height 1, unless overridden
fa4db8671b test: Activate all regtest softforks at height 1, unless overridden (MarcoFalke)
faad1e5ffd Introduce -testactivationheight=name@height setting (MarcoFalke)
fadb2ef2fa test: Add extra_args argument to TestChain100Setup constructor (MarcoFalke)
faa46986aa test: Remove version argument from build_next_block in p2p_segwit test (MarcoFalke)
fa086ef539 test: Remove unused ~TestChain100Setup (MarcoFalke)

Pull request description:

  All softforks that are active at the tip of mainnet, should also be active from genesis in regtest. Otherwise their rules might not be enforced in user testing, thus making their testing less useful.

  To still allow tests to check pre-softfork rules, a runtime argument can change the activation height.

ACKs for top commit:
  laanwj:
    Code review ACK fa4db8671b
  theStack:
    re-ACK fa4db8671b

Tree-SHA512: 6397d46ff56ebc48c007a4cda633904d6ac085bc76b4ecf83097c546c7eec93ac0c44b88083b2611b9091c8d1fb8ee1e314065de078ef15e922c015de7ade8bf
2021-09-24 14:04:51 +02:00
fanquake
86c3b84388
Merge bitcoin/bitcoin#23036: test: use test_framework.p2p P2P_SERVICES constant in functional tests
b69a106bcd test: use test_framework.p2p P2P_SERVICES in functional tests (Jon Atack)

Pull request description:

  `P2P_SERVICES` is defined in `test/functional/test_framework/p2p.py`, so we can use it as a single definition for our functional tests. It may also be a tiny bit more efficient to use the constant rather than calculating `NODE_NETWORK | NODE_WITNESS` every time we need it in the tests.

ACKs for top commit:
  laanwj:
    Code review ACK b69a106bcd
  klementtan:
    crACK b69a106bcd
  fanquake:
    ACK b69a106bcd - didn't look at the formatting changes.

Tree-SHA512: f83e593663a69182986325d9ba2b4b787b87896d6648973f4f802f191a2573201b9e7d7e10e69662ef1965fa63268845726ed1aa5742a2e38dcccf4aebc6a961
2021-09-23 17:13:02 +08:00
merge-script
ae674a0198
Merge bitcoin/bitcoin#22998: test: use MiniWallet for make_utxo helper in feature_rbf.py
f680d27155 test: use MiniWallet for make_utxo helper in feature_rbf.py (Sebastian Falbesoner)
0f27524602 test: scale amounts in test_doublespend_tree down by factor 10 (Sebastian Falbesoner)
d1e2481274 test: scale amounts in test_doublespend_chain down by factor 10 (Sebastian Falbesoner)

Pull request description:

  This PR aims to further increase MiniWallet usage in the functional test feature_rbf.py by using it in the `make_utxo(...)` helper, which is the only part that needs a wallet for most sub-tests. In order to do that, the amounts for the utxos have to be scaled down in two sub-tests first (`test_doublespend_chain` and `test_doublespend_tree`, see first two commits), since we need amounts passed to `make_utxo` than can be funded by only one input. For creating UTXOs with a value of 50 BTC, we'd need to implement a method for consolidating multiple utxos into one first, which seems to be overkill.

  Note that after this PR's change, there is only one sub-test left (`test_rpc`) that needs the wallet compiled into bitcoind.

ACKs for top commit:
  MarcoFalke:
    review ACK f680d27155 🦐

Tree-SHA512: 46c8c245086a9e79855c4ede2f8f412333cf2658136805196b203b3567c89398d77fcb80715c0bb72fdc84331cc67544b2fdc259193a3adcb2fc36e147c26fce
2021-09-21 15:20:07 +02:00
MarcoFalke
faca688a85
test: Add MiniWallet get_descriptor function 2021-09-20 15:48:12 +02:00
MarcoFalke
fa7e3f1fc1
test: Replace MiniWallet scan_blocks with rescan_utxos 2021-09-20 08:31:04 +02:00
Jon Atack
b69a106bcd
test: use test_framework.p2p P2P_SERVICES in functional tests 2021-09-19 14:20:48 +02:00
fanquake
de2af19dc8
Merge bitcoin/bitcoin#22987: qa: Fix "RuntimeError: Event loop is closed" on Windows
357f0c7233 ci: Enable more functional tests on Windows MSVC task (Hennadii Stepanov)
f55932678f qa: Fix "RuntimeError: Event loop is closed" on Windows (Hennadii Stepanov)

Pull request description:

  On master (2161a05855), running functional tests that use the P2P interface ends with an error:
  ```
  RuntimeError: Event loop is closed
  ```

  This PR fixes this bug, and enables more functional tests on Windows MSVC CI task.

  More details about bugfix:
  - [What’s New In Python 3.7](https://docs.python.org/3/whatsnew/3.7.html#asyncio)
  - https://bugs.python.org/issue33792
  - actual [change](https://docs.python.org/3.8/library/asyncio-policy.html#asyncio.WindowsSelectorEventLoopPolicy) done in Python 3.8

  Excluded tests, that are listed in the `EXCLUDE_TESTS` environment variable, need more thorough investigation to be enabled.

ACKs for top commit:
  MarcoFalke:
    review ACK 357f0c7233 🌆

Tree-SHA512: d0ba85be81d55c934959ce7402a9c726598125e9751a1de179d16759d0e8b8a915de879c3a62c12d3564c5e0d9649ebd86963744449626efaa42d9eaa99ad3d0
2021-09-18 16:49:19 +08:00
MarcoFalke
fa4db8671b
test: Activate all regtest softforks at height 1, unless overridden 2021-09-16 18:53:04 +02:00
Sebastian Falbesoner
f680d27155 test: use MiniWallet for make_utxo helper in feature_rbf.py 2021-09-16 16:48:55 +02:00
W. J. van der Laan
58e02395ba
Merge bitcoin/bitcoin#22955: p2p: Rename fBlocksOnly, Add test
fa66a7d732 p2p: Rename fBlocksOnly, Add test (MarcoFalke)
fac66d0a39 test: Simplify p2p_blocksonly test with new miniwallet rescan_utxos method (MarcoFalke)

Pull request description:

  `fBlocksOnly` has several issues:
  * The name is confusing
  * It is untested

  Fix both.

ACKs for top commit:
  laanwj:
    Code review ACK fa66a7d732

Tree-SHA512: 4218f455eeb37297f74603d7d44895288605844ae828a40dfb7a70215f1a058ac5ad945a22732f5ebcad3ad375d54ba360bea69ea79639a30d4c88b042448f0f
2021-09-16 16:38:14 +02:00
Hennadii Stepanov
f55932678f
qa: Fix "RuntimeError: Event loop is closed" on Windows 2021-09-15 20:33:28 +03:00
MarcoFalke
fad4f44645
test: Set peertimeout in write_config
This avoids having to remember to set it whenever mocktime is used with
peer connections. Also, it might help avoiding disconnects when
attaching a debugger to a running test.
2021-09-13 09:41:58 +02:00
MarcoFalke
fac66d0a39
test: Simplify p2p_blocksonly test with new miniwallet rescan_utxos method 2021-09-12 12:13:58 +02:00
MarcoFalke
fa04f26aa7
test: Avoid race after connect_nodes 2021-08-27 10:18:28 +02:00
MarcoFalke
faf7e485e9
Set regtest.BIP65Height = 111 to speed up tests 2021-08-26 11:08:24 +02:00
fanquake
eb09c26724
Merge bitcoin/bitcoin#22741: test: Add generate* calls to test framework
fab2e23b57 Use generate* from TestFramework (MarcoFalke)
faf7e92804 test: Add generate* calls to test framework (MarcoFalke)

Pull request description:

  This is needed for #22567.

  By making the calls to `generate*` member function of the test framework, it paves the way to make it easier to implicitly call the `sync_all` member function.

ACKs for top commit:
  jnewbery:
    utACK fab2e23b57

Tree-SHA512: 7a7be6be71f0602119689df45d63a1adec309f323eac2330ee0f200676001afe825605859bd02c6a8a8dcf85d925dc1bc37370ef1ceb8ad1d85a66eec0dbfff9
2021-08-24 16:22:30 +08:00
MarcoFalke
fab2e23b57
Use generate* from TestFramework
The changes in feature_rbf can be reviewed with
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2021-08-19 19:02:11 +02:00
MarcoFalke
faf7e92804
test: Add generate* calls to test framework 2021-08-19 15:30:07 +02:00
MarcoFalke
f5a406f003
Merge bitcoin/bitcoin#22633: refactor: Replace remaining binascii method calls
021daedfa1 refactor: replace remaining binascii method calls (Zero-1729)

Pull request description:

  This PR removes the remaining `binascii` method calls outside `test/functional` and `test_framework`, as pointed out here  https://github.com/bitcoin/bitcoin/pull/22619#pullrequestreview-722153458.

  Follow-up to #22593 and #22619
  Closes #22605

ACKs for top commit:
  josibake:
    re-ACK 021daedfa1
  theStack:
    re-ACK 021daedfa1

Tree-SHA512: 2ae9fee8917112c91a5406f219ca70f24cd8902b903db5a61fc2de85ad640d669a772f5c05970be0fcee6ef1cdd32fae2ca5d1ec6dc9798b43352c8160ddde6f
2021-08-21 16:29:55 +02:00
Zero-1729
021daedfa1
refactor: replace remaining binascii method calls 2021-08-16 19:30:56 +01:00
W. J. van der Laan
0b5344b0d1
Merge bitcoin/bitcoin#22632: test: Set regtest.BIP66Height = 102 to speed up tests
fafe896a0b test: Set regtest.BIP66Height = 102 to speed up tests (MarcoFalke)

Pull request description:

  No need to waste time by forcing creation of more than 1000 blocks to get the benefits of being able to test BIP 66. Also, reducing the height makes it more likely that (third-party) tests are conforming to BIP 66, which is enforced on mainnet for all new blocks.

ACKs for top commit:
  GeneFerneau:
    Concept + code review ACK [fafe896](fafe896a0b)
  0xB10C:
    crACK fafe896a0b
  laanwj:
    ACK fafe896a0b
  Zero-1729:
    tACK fafe896
  kristapsk:
    ACK fafe896a0b. Full functional test suite showed few second speed incrase on my laptop (although I didn't do proper benchmarking with multiple runs, just single `time ./test/functional/test_runner.py` on current master vs this PR).
  theStack:
    Tested ACK fafe896a0b
  hg333:
    tACK fafe896a0b

Tree-SHA512: 4bbee3c8587d612e74a59fde49b6439c1296f2fc27d3a7cf59a35e920f729fdd581c930290bd04def618f81412236676ddb99b4ceb4d80dfb9fd610b128a04b1
2021-08-10 16:18:26 +02:00
fanquake
21438d55d5
Merge bitcoin/bitcoin#21800: mempool/validation: mempool ancestor/descendant limits for packages
accf3d5868 [test] mempool package ancestor/descendant limits (glozow)
2b6b26e57c [test] parameterizable fee for make_chain and create_child_with_parents (glozow)
313c09f7b7 [test] helper function to increase transaction weight (glozow)
f8253d69d6 extract/rename helper functions from rpc_packages.py (glozow)
3cd663a5d3 [policy] ancestor/descendant limits for packages (glozow)
c6e016aa13 [mempool] check ancestor/descendant limits for packages (glozow)
f551841d3e [refactor] pass size/count instead of entry to CalculateAncestorsAndCheckLimits (glozow)
97dd1c729d MOVEONLY: add helper function for calculating ancestors and checking limits (glozow)
f95bbf58aa misc package validation doc improvements (glozow)

Pull request description:

  This PR implements a function to calculate mempool ancestors for a package and enforces ancestor/descendant limits on them as a whole. It reuses a portion of `CalculateMemPoolAncestors()`; there's also a small refactor to move the reused code into a generic helper function. Instead of calculating ancestors and descendants on every single transaction in the package and their ancestors, we use a "worst case" heuristic, treating every transaction in the package as each other's ancestor and descendant. This may overestimate everyone's counts, but is still pretty accurate in the our main package use cases, in which at least one of the transactions in the package is directly related to all the others (e.g. 1 parent + 1 child, multiple parents with 1 child, or chains).

  Note on Terminology: While "package" is often used to describe groups of related transactions _within_ the mempool, here, I only use package to mean the group of not-in-mempool transactions we are currently validating.

  #### Motivation

  It would be a potential DoS vector to allow submission of packages to mempool without a proper guard for mempool ancestors/descendants. In general, the purpose of mempool ancestor/descendant limits is to limit the computational complexity of dealing with families during removals and additions. We want to be able to validate multiple transactions on top of the mempool, but also avoid these scenarios:

  - We underestimate the ancestors/descendants during package validation and end up with extremely complex families in our mempool (potentially a DoS vector).
  - We expend an unreasonable amount of resources calculating everyone's ancestors and descendants during package validation.

ACKs for top commit:
  JeremyRubin:
    utACK accf3d5
  ariard:
    ACK accf3d5.

Tree-SHA512: 0d18ce4b77398fe872e0b7c2cc66d3aac2135e561b64029584339e1f4de2a6a16ebab3dd5784f376e119cbafc4d50168b28d3bd95d0b3d01158714ade2e3624d
2021-08-09 12:23:39 +08:00
glozow
2b6b26e57c [test] parameterizable fee for make_chain and create_child_with_parents 2021-08-06 10:04:59 +01:00
glozow
313c09f7b7 [test] helper function to increase transaction weight 2021-08-06 10:04:59 +01:00
glozow
f8253d69d6 extract/rename helper functions from rpc_packages.py
MOVEONLY; no change in behavior. Rename because there is another helper
funciton in chain_transaction in test_framework.util.py
2021-08-06 10:04:59 +01:00
MarcoFalke
f4328ebef5
Merge bitcoin/bitcoin#22619: test: refactor: use consistent bytes <-> hex-string conversion in functional test framework
5a1bef60a0 test: refactor: remove binascii from test_framework (Zero-1729)

Pull request description:

  This PR continues the work started in PR #22593, regarding using the `bytes` built-in module. In this PR specifically, instances of `binascii`'s methods `hexlify`, `unhexlify`,  and `a2b_hex` have been replaced with the build-in `bytes` module's `hex` and `fromhex` methods where appropriate to make bytes <-> hex-string conversions consistent across the functional test files and test_framework.

  Additionally, certain changes made are based on the following assumption:

  ```
  bytes.hex(data) == binascii.hexlify(data).decode()
  bytes.hex(data).encode() == binascii.hexlify(data)
  ```

  Ran the functional tests to ensure behaviour is still consistent and changes didn't break existing tests.

  closes #22605

ACKs for top commit:
  theStack:
    Code-review ACK 5a1bef60a0 🔢

Tree-SHA512: 8f28076cf0580a0d02a156f3e1e94c9badd3d41c3fbdfb2b87cd8a761dde2c94faa5f4c448d6747b1ccc9111c3ef1a1d7b42a11c806b241fa0410b7529e2445f
2021-08-05 12:15:01 +02:00
MarcoFalke
fafe896a0b
test: Set regtest.BIP66Height = 102 to speed up tests 2021-08-05 12:08:33 +02:00
Zero-1729
5a1bef60a0
test: refactor: remove binascii from test_framework 2021-08-04 19:59:13 +01:00
MarcoFalke
ad0fc453cc
Merge bitcoin/bitcoin#16333: test: Set BIP34Height = 2 for regtest
222290f543 test: Set BIP34Height = 2 for regtest (MarcoFalke)
fac90c55be test: Create all blocks with version 4 or higher (MarcoFalke)

Pull request description:

  BIP34 is active on the current tip of mainnet, so all miners must obey it. It would be nice if it also was active in fresh regtest instances from the earliest time possible.

  I changed the BIP34 height to `2`, so that the block at height=1 may be used to mine a duplicate coinbase. (Needed to test mainnet behaviour)

  This pull is done in two commits:

  *  test: Create all blocks with version 4 or higher:
     Now that BIP34 is activated earlier, we need to create blocks with a higher version number. Just bump it to 4 instead of 2 to avoid having to bump it again later.

  *  test: Set BIP34Height = 2 for regtest:
     This fixes the BIP34 implementation in the tests (to match the one of the Core codebase) and updates the tests where needed

ACKs for top commit:
  ajtowns:
    ACK 222290f543
  jonatack:
    ACK 222290f543 tested and reviewed rebased to current master 5e213822f8
  theStack:
    Tested ACK 222290f543

Tree-SHA512: d69c637a62a64b8e87de8c7f0b305823d8f4d115c1852514b923625dbbcf9a4854b5bb3771ff41702ebf47c4c182a4442c6d7c0b9f282c95a34b83e56a73939b
2021-08-03 10:10:43 +02:00
fanquake
06788c6705
Merge bitcoin/bitcoin#21528: [p2p] Reduce addr blackholes
3f7250b328 [test] Use the new endpoint to improve tests (Amiti Uttarwar)
3893da06db [RPC] Add field to getpeerinfo to indicate if addr relay is enabled (Amiti Uttarwar)
0980ca78cd [test] Test that we intentionally select addr relay peers. (Amiti Uttarwar)
c061599e40 [net_processing] Remove RelayAddrsWithPeer function (Amiti Uttarwar)
201e496481 [net_processing] Introduce new field to indicate if addr relay is enabled (Amiti Uttarwar)
1d1ef2db7e [net_processing] Defer initializing m_addr_known (Amiti Uttarwar)
6653fa3328 [test] Update p2p_addr_relay test to prepare (Amiti Uttarwar)
2fcaec7bbb [net_processing] Introduce SetupAddressRelay (Amiti Uttarwar)

Pull request description:

  This PR builds on the test refactors extracted into #22306 (first 5 commits).

  This PR aims to reduce addr blackholes. When we receive an `addr` message that contains 10 or less addresses, we forward them to 1-2 peers. This is the main technique we use for self advertisements, so sending to peers that wouldn't relay would effectively "blackhole" the trickle. Although we cannot prevent this in a malicious case, we can improve it for the normal, honest cases, and reduce the overall likelihood of occurrence. Two known cases where peers would not participate in addr relay are if they have connected to you as a block-relay-only connection, or if they are a light client.

  This implementation defers initialization of `m_addr_known` until it is needed, then uses its presence to decide if the peer is participating in addr relay. For outbound (not block-relay-only) peers, we initialize the filter before sending the initial self announcement when processing their `version` message. For inbound peers, we initialize the filter if/when we get an addr related message (`ADDR`, `ADDRV2`, `GETADDR`). We do NOT initialize the filter based on a `SENDADDRV2` message.

  To communicate about these changes beyond bitcoin core & to (try to) ensure that no other software would be disrupted, I have:
  - Posted to the [mailing list](https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-April/018784.html)
  - Researched other open source clients to confirm compatibility, opened issues in all the projects & documented in https://github.com/bitcoin/bitcoin/pull/21528#issuecomment-809906430. Many have confirmed that this change would not be problematic.
  - Raised as topic during [bitcoin-core-dev meeting](https://www.erisian.com.au/bitcoin-core-dev/log-2021-03-25.html#l-954)
  - Raised as topic during [bitcoin p2p meeting](https://www.erisian.com.au/bitcoin-core-dev/log-2021-04-20.html#l-439)

ACKs for top commit:
  jnewbery:
    reACK 3f7250b328
  glozow:
    ACK 3f7250b328
  ajtowns:
    utACK 3f7250b328

Tree-SHA512: 29069282af684c1cd37d107c395fdd432dcccb11626f3c2dabfe92fdc4c85e74c7c4056fbdfa88017fec240506639b72ac6c311f8ce7c583112eb15f47e421af
2021-08-03 09:47:51 +08:00
MarcoFalke
b620b2d58a
Merge bitcoin/bitcoin#22378: test: remove confusing MAX_BLOCK_BASE_SIZE
607076d01b test: remove confusing `MAX_BLOCK_BASE_SIZE` (Sebastian Falbesoner)
4af97c74ed test: introduce `get_weight()` helper for CBlock (Sebastian Falbesoner)
a084ebe133 test: introduce `get_weight()` helper for CTransaction (Sebastian Falbesoner)

Pull request description:

  This is a very late follow-up PR to #10618, which removed the constant `MAX_BLOCK_BASE_SIZE` from the core implementation about four years ago (see also #10608 in why it was considered confusing and superfluous).
  Since there is also no point in still keeping it in the functional test framework, the PR switches to weight-based accounting on the relevant test code parts and use `MAX_BLOCK_WEIGHT` instead for the block limit checks. To prepare that, the first two commits introduce `get_weight()` helpers for the classes CTransaction and CBlock, respectively.

ACKs for top commit:
  MarcoFalke:
    review ACK 607076d01b 🚴

Tree-SHA512: d59aa0b6b3dfd0a849b8063e66de275d252f705f99e25cd3bf6daec028b47d946777ee5b42a060f5283cb18e917ac073119c2c0e11bbc21211f69ef0a6ed335a
2021-08-02 15:51:48 +02:00
Sebastian Falbesoner
ca6c154ef1 test: refactor: remove hex_str_to_bytes helper
Use the built-in class method bytes.fromhex() instead,
which is available since Python 3.0.
2021-08-01 19:26:51 +02:00
MarcoFalke
f2e41d1109
Merge bitcoin/bitcoin#22429: test: refactor: fix segwit terminology (s/witness_program/witness_script/)
8a2b58db9e test: fix segwit terminology (s/witness_program/witness_script/) (Sebastian Falbesoner)

Pull request description:

  This PR fixes wrong uses of the term "witness program", which according to [BIP141](https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#Witness_program)  is defined as follows:
  > A scriptPubKey (or redeemScript as defined in BIP16/P2SH) that consists of a 1-byte push opcode (for 0 to 16) followed by a data push between 2 and 40 bytes gets a new special meaning. The value of the first push is called the "version byte". **The following byte vector pushed is called the "witness program".**

  In most cases where "witness program" is used in tests (concerns comments, variable names and in one instance even a function name) what we really want to denote is the "witness script". Thanks to [MarcoFalke for pointing this out in a review comment](https://github.com/bitcoin/bitcoin/pull/22363#discussion_r666794261)!

  Some historical background: At the time when the P2P segwit tests were first introduced (commit 330b0f31ee, PR #8149), the term "witness program" was not used consistently in BIP141: https://bitcoin.stackexchange.com/questions/46451/what-is-the-precise-definition-of-witness-program
  This was fixed in PR https://github.com/bitcoin/bips/pull/416 later.

  So in some way, this PR can be seen as a very late follow-up to the BIP141 fix that also reflects these changes in the tests.

ACKs for top commit:
  josibake:
    tACK 8a2b58db9e

Tree-SHA512: f36bb9e53d1b54b86bfa87ec12f33e3ebca64b5f59d97e9662fe35ba12c25e1c9a4f93a5425d0eaa3879dce9e50368d345555b927bfab76945511f873396892b
2021-08-01 16:59:19 +02:00
MarcoFalke
78f040a6b3
Merge bitcoin/bitcoin#22490: test: Disable automatic connections per default in the functional tests
8ca51af1ec test: Disable automatic connections by default (Martin Zumsande)

Pull request description:

  A node normally doesn't make automatic connections to peers in the functional tests because neither DNS seeds nor hardcoded peers are available on regtest. However, when random entries are inserted into addrman as part of a functional test (e.g. while testing addr relay), `ThreadOpenConnections` will periodically try to connect to them, resulting in log entries such as:
  `[opencon] [net.cpp:400] [ConnectNode] trying connection 18.166.1.1:8333 lastseen=0.0hrs`

  I don't think it's desirable that functional tests try to connect to random computers on the internet, aside from the possibility that at some point in time someone out there might actually answer in a way to ruin a test.

  This PR fixes this problem by disabling  `ThreadOpenConnections` by adding `-connect=0` to the default args, and adding exceptions only when needed for the test to pass.

ACKs for top commit:
  tryphe:
    Concept ACK, light code review ACK 8ca51af1ec

Tree-SHA512: bcfb2de610e6c35a97a2bd7ad6267e968b1ac7529638d99276993cd5bc93ce9919d54e22d6dc84e1b02ecd626ab6554e201693552ea065c29794eece38c43f7d
2021-07-30 13:26:50 +02:00
Amiti Uttarwar
1d1ef2db7e [net_processing] Defer initializing m_addr_known
Use SetupAddressRelay to only initialize `m_addr_known` as needed. For outbound
peers, we initialize the filter before sending our self announcement (not
applicable for block-relay-only connections). For inbound peers, we initialize
the filter when we get an addr related message (ADDR, ADDRV2, GETADDR).

These changes intend to mitigate address blackholes. Since an inbound peer has
to send us an addr related message to become eligible as a candidate for addr
relay, this should reduce our likelihood of sending them self-announcements.
2021-07-29 17:40:21 -07:00