b4dd2ef800 [test] Test the add_outbound_p2p_connection functionality (Amiti Uttarwar)
602e69e427 [test] P2PBlocksOnly - Test block-relay-only connections. (Amiti Uttarwar)
8bb6beacb1 [test/refactor] P2PBlocksOnly - Extract transaction violation test into helper. (Amiti Uttarwar)
99791e7560 [test/refactor] P2PBlocksOnly - simplify transaction creation using blocktool helper. (Amiti Uttarwar)
3997ab9154 [test] Add test framework support to create outbound connections. (Amiti Uttarwar)
5bc04e8837 [rpc/net] Introduce addconnection to test outbounds & blockrelay (Amiti Uttarwar)
Pull request description:
The existing functional test framework uses the `addnode` RPC to spin up manual connections between bitcoind nodes. This limits our ability to add integration tests for our networking code, which often executes different code paths for different connection types.
**This PR enables creating `outbound` & `block-relay-only` P2P connections in the functional tests.** This allows us to increase our p2p test coverage, since we can now verify expectations around these connection types.
This builds out the [prototype](https://github.com/bitcoin/bitcoin/issues/14210#issuecomment-527421978) proposed by ajtowns in #14210. 🙌🏽
An overview of this branch:
- introduces a new test-only RPC function `addconnection` which initiates opening an `outbound` or `block-relay-only` connection. (conceptually similar to `addnode` but for different connection types & restricted to regtest)
- adds `test_framework` support so a mininode can open an `outbound`/`block-relay-only` connection to a `P2PInterface`/`P2PConnection`.
- updates `p2p_blocksonly` tests to create a `block-relay-only` connection & verify expectations around transaction relay.
- introduces `p2p_add_connections` test that checks the behaviors of the newly introduced `add_outbound_p2p_connection` test framework function.
With these changes, there are many more behaviors that we can add integration tests for. The blocksonly updates is just one example.
Huge props to ajtowns for conceiving the approach & providing me feedback as I've built out this branch. Also thank you to jnewbery for lots of thoughtful input along the way.
ACKs for top commit:
troygiorshev:
reACK b4dd2ef800
jnewbery:
utACK b4dd2ef800
MarcoFalke:
Approach ACK b4dd2ef800🍢
Tree-SHA512: d1cba768c19c9c80e6a38b1c340cc86a90701b14772c4a0791c458f9097f6a4574b4a4acc7d13d6790c7b1f1f197e2c3d87996270f177402145f084ef8519a6b
aaaa987840 refactor: Use C++17 std::array deduction for ALL_FEE_ESTIMATE_HORIZONS (MarcoFalke)
fa39cdd072 refactor: Use C++17 std::array deduction for OUTPUT_TYPES (MarcoFalke)
Pull request description:
With the new C++17 array deduction rules, an array encompassing all values in an enum can be specified in the same header file that specifies the enum. This is useful to avoid having to repeatedly enumerate all enum values in the code. E.g. the RPC code, but also the fuzz code.
ACKs for top commit:
theStack:
cr ACK aaaa987840⚙️
fanquake:
ACK aaaa987840
Tree-SHA512: b71bd98f3ca07ddfec385735538ce89a4952e418b52dc990fb160187ccef1fc7ebc139d42988b6f7b48df24823af61f803b83d47fb7a3b82475f0c0b109bffb7
3642b2ed34 refactor, net: Increase CNode data member encapsulation (Hennadii Stepanov)
acebb79d3f refactor, move-only: Relocate CNode private members (Hennadii Stepanov)
Pull request description:
All protected `CNode` data members could be private.
ACKs for top commit:
jnewbery:
utACK 3642b2ed34
MarcoFalke:
review ACK 3642b2ed34 🏛
Tree-SHA512: 8435e3c43c3b7a3107d58cb809b8b5e1a1c0068677e249bdf0fc6ed24140ac4fc4efe2a280a1ee86df180d738c0c9e10772308690607954db6713000cf6e728d
39b43298d9 test: add test for banning of non-IP addresses (Vasil Dimov)
94d335da7f net: allow CSubNet of non-IP networks (Vasil Dimov)
Pull request description:
Allow creation of valid `CSubNet` objects of non-IP networks and only
match the single address they were created from (like /32 for IPv4 or
/128 for IPv6).
This fixes a deficiency in `CConnman::DisconnectNode(const CNetAddr& addr)`
and in `BanMan` which assume that creating a subnet from any address
using the `CSubNet(CNetAddr)` constructor would later match that address
only. Before this change a non-IP subnet would be invalid and would not
match any address.
ACKs for top commit:
jonatack:
Code review re-ACK 39b43298d9 per `git diff 5e95ce6 39b4329`; only change since last review is improvements to the functional test; verified the test fails on master @ 616eace0 where expected (`assert(self.is_banned(node, tor_addr))` fails and unban unfails)
laanwj:
code review ACK 39b43298d9
Tree-SHA512: 3239b26d0f2fa2d1388b4fdbc1d05ce4ac1980be699c6ec46049409baefcb2006b1e72b889871e2210e897f6725c48e873f68457eea7e6e4958ab4f959d20297
b3e9bcaac8 qt, refactor: Drop no longer used PeerTableModel::getNodeStats function (Hennadii Stepanov)
49c604077c qt: Use PeerTableModel::StatsRole (Hennadii Stepanov)
35007edf9c qt: Add PeerTableModel::StatsRole (Hennadii Stepanov)
Pull request description:
This PR allows to access to the `CNodeCombinedStats` instance directly from any view object.
The `PeerTableModel::getNodeStats` member function removed as a kind of layer violation.
No behavior changes.
Also other pulls (bugfixes) are based on this one: #18 and #164.
ACKs for top commit:
jonatack:
Tested re-ACK b3e9bcaac8 per `git range-diff ae8f797 4c05fe0 b3e9bca`
promag:
Code review ACK b3e9bcaac8.
jonasschnelli:
utACK b3e9bcaac8
Tree-SHA512: 6ba50d5dd2c0373655d491ce8b130c47d598da2db5ff4b00633f447404c7e70f8562ead53ddf166e851384d9632ff9146a770c99845c2cdd3ff7250677e4c130
faabc26a61 test: Replace getmempoolentry with testmempoolaccept in MiniWallet (MarcoFalke)
Pull request description:
This is a refactor to not use the return value of `sendrawtransaction` and `getmempoolentry` with the goal that submitting the tx to the mempool will become optional.
ACKs for top commit:
mjdietzx:
ACK faabc26a61
Tree-SHA512: 4260a59e65fed1c807530dad23f1996ba6e881bcce91995f5b498a0be6001f67b3e0251507c8801750fe105410147c68bb2f393ebe678c6a3a4d045e5d72fc19
faa8f68943 Replace boost::variant with std::variant (MarcoFalke)
Pull request description:
Now that we can use std::variant from the vanilla standard library, drop the third-party boost variant dependency
ACKs for top commit:
fjahr:
Code review ACK faa8f68943
fanquake:
ACK faa8f68943
Tree-SHA512: 6e3aecd33b00c2e31a763f999247944d5b2ce5e3018f1965c516c1000cd08ff6703a8d50fb0be64883153da2925ae72986b8a6b96586db74057bd05d6f4986e6
fad1f0fd33 net: Remove unused cs_feeFilter (MarcoFalke)
Pull request description:
A `RecursiveMutex` is overkill for setting or reading a plain integer. Even a `Mutex` is overkill, when a plain `std::atomic` can be used.
This removes 11 lines of code. Also, it is cutting down on the number of locks put on the stack at the same time, which complicates review looking out for potential lock contention.
ACKs for top commit:
jnewbery:
utACK fad1f0fd33
practicalswift:
cr ACK fad1f0fd33: patch looks correct
Tree-SHA512: 647f9b954fbf52e138d3e710937eb9131b390fef0deae03fd6a162d5a18b9f194010800bbddc8f89208d91be2802dff11c3884d04b3dd233865abd12aa3cde06
Allow creation of valid `CSubNet` objects of non-IP networks and only
match the single address they were created from (like /32 for IPv4 or
/128 for IPv6).
This fixes a deficiency in `CConnman::DisconnectNode(const CNetAddr& addr)`
and in `BanMan` which assume that creating a subnet from any address
using the `CSubNet(CNetAddr)` constructor would later match that address
only. Before this change a non-IP subnet would be invalid and would not
match any address.
06ba9b3008 rpc: move getpeerinfo connection_type help to correct place (Jon Atack)
c95fe6e38f gui: improve connection type tooltip (Hennadii Stepanov)
2c19ba2e1d gui: replace Direction with Connection Type in peer details (Jon Atack)
7e2beab2d2 gui: create GUIUtil::ConnectionTypeToQString utility function (Jon Atack)
Pull request description:
Closes#159.
ACKs for top commit:
hebasto:
re-ACK 06ba9b3008, the tooltip content is organized as unordered list.
jarolrod:
re-ACK 06ba9b3008
Tree-SHA512: 24f46494ceb42ed308e4a4f2a543dbc4f4e6409a6f738c145a9f16e175bf69d411cbc944a4fd969f1829d57644dfbc194182fa8d4e9e6bce82acbeca8c673748
fad327ca65 fuzz: net permission flags in net processing (MarcoFalke)
Pull request description:
to increase coverage
ACKs for top commit:
Crypt-iQ:
cr ACK fad327c
practicalswift:
ACK fad327ca65
Tree-SHA512: f8643d1774ff13524ab97ab228ad070489e080435e5742af26e6e325fd002e4c1fd78b9887e11622e79d6fe0c4daaddce5e033e6cd4b32e50fd68b434aab7333
3e61b8c800 doc: Add explicit macdeployqtplus dependencies install step (Hennadii Stepanov)
Pull request description:
This PR adds to macOS docs an explicit step to install `macdeployqtplus` script dependencies that are not part of the [Python Standard Library](https://docs.python.org/3/library/index.html):
- https://pypi.org/project/ds-store/
- https://pypi.org/project/mac-alias/
This change is required on macOS 11 Big Sur:
- #20371
- #20878Close#20878.
ACKs for top commit:
fanquake:
ACK 3e61b8c800
Tree-SHA512: d177139ee142d47cb27ad878d721cafcd03403ef861965ff532d712da461416380ec5878f70accf223a552a1f1e65eedb1e0ad72cb7a96791f8a55536ce28645
faecb74562 Expose integral m_conn_type in CNodeStats, remove m_conn_type_string (Jon Atack)
Pull request description:
Currently, strings are stored for what are actually integral (strong) enum types. This is fine, because the strings are only used as-is for the debug log and RPC. However, it complicates using them in the GUI. User facing strings in the GUI should be translated and only string literals can be picked up for translation, not runtime `std::string`s.
Fix that by removing the `std::string` members and replace them by strong enum integral types.
ACKs for top commit:
jonatack:
Code review ACK faecb74562
theStack:
Code review ACK faecb74562🌲
Tree-SHA512: 24df2bd0645432060e393eb44b8abaf20fe296457d07a867b0e735c3e2e75af7b03fc6bfeca734ec33ab816a7c8e1f8591a5ec342f3afe3098a4e41f5c2cfebb
fa44417fcb fuzz: Add missing muhash registration (MarcoFalke)
Pull request description:
otherwise it is not possible to run the target
ACKs for top commit:
laanwj:
Code review ACK fa44417fcb
Tree-SHA512: b6495b84890a778d3569deddfc5703d1bed95d7d5c1eb0766fd5bd9afe86a3a0ccf4726aa16cba4a5fd182d23bfb7e815f3af4eb635915397c90805585b5699e
a7599c80eb test: run mempool_compatibility.py even with wallet disabled (Michael Dietz)
Pull request description:
Another functional test rewritten as proposed in https://github.com/bitcoin/bitcoin/issues/20078
ACKs for top commit:
MarcoFalke:
review ACK a7599c80eb didn't test
Tree-SHA512: cc7a617e5489ed27bbdbdee85a82fa08525375061f7f4524577a6b8ecb340396adac88419b51f513be22ca53edd0a3bd5d572d9f43ffc2c18550b0ef9069d238
196b727649 depends: Add comment about cache invalidation (Carl Dong)
949c480e52 depends: Fully determine path for darwin cctools (Carl Dong)
880660acfa depends: Fully determine path for darwin_{CC,CXX} (Carl Dong)
8033110741 depends: Quote to prevent word splitting in config.site (Carl Dong)
77b1ef89a0 depends: Remove -fuse-ld line (Carl Dong)
3007339218 depends: Pin clang search paths for darwin host (Carl Dong)
107f33d434 depends: Delay expansion of per-package vars (Carl Dong)
Pull request description:
> Hello clang/lib/frontend,
> I search your headers once again.
> Because it's time for some housekeeping,
> Within the code I was tweaking,
> And the targets I was making with my build,
> Are unfulfilled,
> It's just language compliance.
>
> In reference works I scroll alone
> Pages cribbed from holy tomes
> In the details of a template
> My code's behaviour has now found its fate
> When my hopes were dashed as a note left it as described:
> As undefined
> It's not in compliance
>
> And from the standard text I saw
> Ten thousand errors, maybe more
> Threading used without locking
> Pointers referenced after freeing
> Linters writing warnings that coders will never fix
> But still they tick
> The box that claims compliance
>
> "Fools," said I, "you do not know"
> Errors, like a cancer, grow
> Hear my words that I might reach you
> Use -Wall and it might teach you
> But my words and compiler errors fade.
> Schedules forbade compliance.
>
> And the people bowed and prayed
> With static checking torn and frayed
> The markets flashed out their warning
> In the words that they were forming
> As recruiters said "The search for more profits leads to writing stuff in CSS,
> And node.js.
> Without a need for compliance"
Many thanks to ajtowns for the above contribution!
-----
This PR is ready for review!
When cross-compiling for macOS, the SDK gives us the entire context/sysroot on which we should base the build. This means that we can be extremely specific w/re our search path ordering in order to avoid build problems that arise out of a user's specific environment/system setup and improve the robustness of our macOS toolchain. This PR does 2 things to this end:
1. Unset environment variables which are known to alter search paths.
1. Makes us (in the case of macOS builds) explicitly specify the list of system include search paths and its ordering, rather than rely on `clang`'s unreliable autodetection routine. Here is the [rabbit-hole gist](https://gist.github.com/dongcarl/5cdc6990b7599e8a5bf6d2a9c70e82f9).
See the added comments in `depends/hosts/darwin.mk` for more details:
8b8296dc70/depends/hosts/darwin.mk (L37-L60)
We can be this specific _only_ because macOS builds are neatly contained in an SDK, **and** we are cross-compiling. Native toolchains should rely on the environment/distro/user to know how best to build for the running system.
Note: Although the `-u` flag of `env` is not a POSIX standard flag, it seems like it is useful enough to be implemented in [coreutils](https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html), [busybox](https://busybox.net/downloads/BusyBox.html#env), [FreeBSD](https://www.freebsd.org/cgi/man.cgi?env).
ACKs for top commit:
laanwj:
code review ACK 196b727649
Tree-SHA512: 406442df16d9aa0aef62f9fa94f72d7e48374301f3d826bf32f183e1610942aa44a4adfac7bead1f14aded0044fac400e1328fcd933b2337e55a024f034b5013
ef712298c3 util: Check for file being NULL in DirectoryCommit (Luke Dashjr)
4574904038 Fix possible data race when committing block files (Evan Klitzke)
220bb16cbe util: Introduce DirectoryCommit commit function to sync a directory (Evan Klitzke)
ce5cbaea63 util.h: Document FileCommit function (Evan Klitzke)
844d650eea util: Prefer Mac-specific F_FULLSYNC over fdatasync in FileCommit (Evan Klitzke)
f6cec0bcaf util: Refactor FileCommit from an #if sequence nested in #else, to a sequence of #elif (Evan Klitzke)
Pull request description:
Reviving #12696
ACKs for top commit:
laanwj:
Code review ACK ef712298c3
Tree-SHA512: 07d650990ef4c18d645dee3f9a199a940683ad17557d79d93979a76c4e710d8d70e6eae01d1a5991494a24a7654eb7db868be0c34a31e70b2509945d95bc9cce
Instead of doing the awkward /bin path prepending at config.site
creation time, set darwin_{CC,CXX} in a way that fully determines the
program's path (clang/clang++) similar to how AC_PATH_{TOOL,PROG} would
do.
Also see the added comment block in depends/Makefile for more context on
determining $PATH for our config.site.
clang warns when a command line option is unused, and some of our tests
use Werror, so unfortunately we cannot use this flag to pin our linker
for now. Leaving this commit in for future reference, as it would be
great if there's more granularity to Werror and we can be explicit about
what linker we want to use.
Prior to this commit, when int_vars was called for packages, it would
immediately expand the "single-dollar variables", which may be defined
in terms of variables which are not yet determined (e.g. variables
defined in package/*.mk, which are included after int_vars is called).
This is required for the next commit as after that commit, for darwin
cross-builds:
0. int_vars is defined in terms of $(1)_cc
1. $(1)_cc is defined in terms of darwin_CC
2. ... which is defined in terms of clang_resource_dir
3. ... which is defined in terms of native_cctools_clang_version
4. which is undetermined at the time when int_vars is being expanded and evaluated
a191e23b8e doc: Add release notes (Hennadii Stepanov)
ae749d12dd doc: Add libnatpmp stuff (Hennadii Stepanov)
e28f9be87a ci: Add libnatpmp-dev package to some builds (Hennadii Stepanov)
5a0185b6c9 gui: Add NAT-PMP network option (Hennadii Stepanov)
a39f7336a3 net: Add -natpmp command line option (Hennadii Stepanov)
28acffd9d5 net: Add NAT-PMP to port mapping loop (Hennadii Stepanov)
a8d9f275d0 net: Add libnatpmp support (Hennadii Stepanov)
58e8364dcd gui: Apply port mapping changes on dialog exit (Hennadii Stepanov)
cf151cc68c scripted-diff: Rename UPnP stuff (Hennadii Stepanov)
4e91b1e24d net: Add flags for port mapping protocols (Hennadii Stepanov)
8b50d1b5bb net: Keep trying to use UPnP when -upnp=1 (Hennadii Stepanov)
28e2961fd6 refactor: Replace magic number with named constant (Hennadii Stepanov)
02ccf69dd6 refactor: Move port mapping code to its own module (Hennadii Stepanov)
Pull request description:
Close#11902
This PR is an alternative to:
- #12288
- #15717
To compile with NAT-PMP support on Ubuntu [`libnatpmp-dev`](https://packages.ubuntu.com/source/bionic/libnatpmp) should be available.
Log excerpt:
```
2020-02-05T20:12:28Z [mapport] NAT-PMP: public address = 95.164.65.194
2020-02-05T20:12:28Z [mapport] AddLocal(95.164.65.194:18333,3)
2020-02-05T20:12:28Z [mapport] NAT-PMP: port mapping successful.
```
See: [`libnatpmp`](https://miniupnp.tuxfamily.org/libnatpmp.html)
---
Some follow-ups are out of this PR's scope:
- mention NAT-PMP library in the version message
- ~integrate NAT-PMP into the GUI~ (already [added](https://github.com/bitcoin/bitcoin/pull/18077#issuecomment-589405068))
ACKs for top commit:
laanwj:
Tested and code review ACK a191e23b8e
Tree-SHA512: 10e19267c21bf30f20ff1abfc882d526049f0e790b95e12f109dc2bed7c0aef45de03eaf967f4e667e7509be04f1873a5c508087393d947205f3aab2ad6d7cf1
Open max number of full-relay and block-relay-only connections from a
functional test with different sorts of behaviors to ensure it behaves as
expected.
In the interest of increasing our P2P test coverage, add support to create
full-relay or block-relay-only connections. To support this, a P2P connection
spins up a listening thread & uses a callback to trigger the node initiating
the connection.
Co-authored-by: Anthony Towns <aj@erisian.com.au>
Add a new RPC endpoint to enable opening outbound connections from
the tests. The functional test framework currently uses the addnode RPC, which
has different behavior than general outbound peers. These changes enable
creating both full-relay and block-relay-only connections. The new RPC
endpoint calls through to a newly introduced AddConnection method on
CConnman that ensures we stay within the allocated max.
9815332d51 test: Change MuHash Python implementation to match cpp version again (Fabian Jahr)
01297fb3ca fuzz: Add MuHash consistency fuzz test (Fabian Jahr)
b111410914 test: Add MuHash3072 fuzz test (Fabian Jahr)
c122527385 bench: Add Muhash benchmarks (Fabian Jahr)
7b1242229d test: Add MuHash3072 unit tests (Fabian Jahr)
adc708c98d crypto: Add MuHash3072 implementation (Fabian Jahr)
0b4d290bf5 crypto: Add Num3072 implementation (Fabian Jahr)
589f958662 build: Check for 128 bit integer support (Fabian Jahr)
Pull request description:
This is the first split of #18000 which implements the Muhash algorithm and uses it to calculate the UTXO set hash in `gettxoutsetinfo`.
ACKs for top commit:
laanwj:
Code review ACK 9815332d51
Tree-SHA512: 4bc090738f0e3d80b74bdd8122e24a8ce80121120fd37c7e4335a73e7ba4fcd7643f2a2d559e2eebf54b8e3a3bd5f12cfb27ba61ded135fda210a07a233eae45
