fa0a864b38 fuzz: Use mocktime in process_message* fuzz targets (MarcoFalke)
Pull request description:
Use mocktime to allow time to advance deterministically during execution of a fuzz input. This also allows to drop the call to `JumpOutOfIbd`.
ACKs for top commit:
practicalswift:
cr ACK fa0a864b38
Tree-SHA512: e92fc70ec6bd49760173cb202549f364304e22b3f7127b9a4da8447cf9341008e477ad42c2599c2fde167bbcbc0e2d139709b4ef6371788bc2c1c3b7f589e11d
fa75d40ef8 fuzz: Introduce CallOneOf helper to replace switch-case (MarcoFalke)
Pull request description:
The current `switch (fuzzed_data_provider.ConsumeIntegralInRange<int>(0, nn)) { case 0: ... case 1: ... case nn: ...` has several problems:
* It makes it hard to review newly added targets, because it requires manual counting of cases
* It makes it hard to update a target, because updating all case labels is trivial, but tedious to review and causes merge conflicts
* ~~Updating the target raises the question whether the case labels should be preserved to not invalidate the existing fuzz inputs format. Fuzz input format might already change implicitly on every commit, so this isn't something worthwhile to pursue.~~ Edit: This pull doesn't fix this problem.
Fix all issues by adding a new `CallOneOf` helper
ACKs for top commit:
ajtowns:
ACK fa75d40ef8 - code review only
jnewbery:
utACK fa75d40ef8
Tree-SHA512: 2daa602b240b86c8e85a024e008f03a57ba60349377eed771f4d21a97a9dba9b66e93fff16ff1992018d4330be7a1a276944c3dfdf698748ce135626c380e563
aaaa987840 refactor: Use C++17 std::array deduction for ALL_FEE_ESTIMATE_HORIZONS (MarcoFalke)
fa39cdd072 refactor: Use C++17 std::array deduction for OUTPUT_TYPES (MarcoFalke)
Pull request description:
With the new C++17 array deduction rules, an array encompassing all values in an enum can be specified in the same header file that specifies the enum. This is useful to avoid having to repeatedly enumerate all enum values in the code. E.g. the RPC code, but also the fuzz code.
ACKs for top commit:
theStack:
cr ACK aaaa987840⚙️
fanquake:
ACK aaaa987840
Tree-SHA512: b71bd98f3ca07ddfec385735538ce89a4952e418b52dc990fb160187ccef1fc7ebc139d42988b6f7b48df24823af61f803b83d47fb7a3b82475f0c0b109bffb7
fad327ca65 fuzz: net permission flags in net processing (MarcoFalke)
Pull request description:
to increase coverage
ACKs for top commit:
Crypt-iQ:
cr ACK fad327c
practicalswift:
ACK fad327ca65
Tree-SHA512: f8643d1774ff13524ab97ab228ad070489e080435e5742af26e6e325fd002e4c1fd78b9887e11622e79d6fe0c4daaddce5e033e6cd4b32e50fd68b434aab7333
9815332d51 test: Change MuHash Python implementation to match cpp version again (Fabian Jahr)
01297fb3ca fuzz: Add MuHash consistency fuzz test (Fabian Jahr)
b111410914 test: Add MuHash3072 fuzz test (Fabian Jahr)
c122527385 bench: Add Muhash benchmarks (Fabian Jahr)
7b1242229d test: Add MuHash3072 unit tests (Fabian Jahr)
adc708c98d crypto: Add MuHash3072 implementation (Fabian Jahr)
0b4d290bf5 crypto: Add Num3072 implementation (Fabian Jahr)
589f958662 build: Check for 128 bit integer support (Fabian Jahr)
Pull request description:
This is the first split of #18000 which implements the Muhash algorithm and uses it to calculate the UTXO set hash in `gettxoutsetinfo`.
ACKs for top commit:
laanwj:
Code review ACK 9815332d51
Tree-SHA512: 4bc090738f0e3d80b74bdd8122e24a8ce80121120fd37c7e4335a73e7ba4fcd7643f2a2d559e2eebf54b8e3a3bd5f12cfb27ba61ded135fda210a07a233eae45
eeee43bc48 fuzz: Use ConsumeWeakEnum for ServiceFlags (MarcoFalke)
fa9949b914 fuzz: Add ConsumeWeakEnum helper, Extract ALL_NET_PERMISSION_FLAGS (MarcoFalke)
faaef9434c fuzz: [refactor] Extract ALL_CONNECTION_TYPES constant (MarcoFalke)
fa42da2d54 fuzz: Use ConsumeNode in process_message target (MarcoFalke)
fa121f058f fuzz: Use ConsumeNode in process_messages target (MarcoFalke)
Pull request description:
The fuzz tests have several problems:
* The array passed to the fuzz engine to pick `net_permission_flags` is outdated
* The process_message* targets has the service flags as well as connection type hardcoded, limiting potential coverage
* The service flags deserialization from the fuzz engine doesn't allow for easy "exact matches". The fuzz engine has to explore a 64-bit space to hit an "exact match" (only one bit set)
Fix all issues in the commits in this pull
ACKs for top commit:
mzumsande:
ACK eeee43bc48 after rebase.
Tree-SHA512: 1ad9520c7e708b7f4994ae8f77886ffca33d7c542756e2a3e07dbbbe59e360f9fcaccf2e2fb57d9bc731d4aeb4938fb1c5c546e9d2744b007af5626f5cb377fe
efaf80e9bb fuzz: check that certain script TxoutType are nonstandard (Michael Dietz)
Pull request description:
- Every transaction of type NONSTANDARD must not be a standard script
- The only know types of nonstandard scripts are NONSTANDARD and certain NULL_DATA and MULTISIG scripts
When reviewing https://github.com/bitcoin/bitcoin/pull/20761 I figured this is very similar and might also be good to have
ACKs for top commit:
MarcoFalke:
ACK efaf80e9bb
Tree-SHA512: 6f563ee3104ea9d2633aad95f1d003474bea759d0f22636c37aa91b5536a6ff0800c42447285ca8ed12f1b3699bf781dae1e5e0a3362da578749cd3164a06ea4
86c495223f net: add CNode::IsInboundOnion() public getter and unit tests (Jon Atack)
6609eb8cb5 net: assert CNode::m_inbound_onion is inbound in ctor (Jon Atack)
993d1ecd19 test, fuzz: fix constructing CNode with invalid inbound_onion (Jon Atack)
Pull request description:
The goal of this PR is to be able to depend on `m_inbound_onion` in AttemptToEvictConnection in #20197:
- asserts `CNode::m_inbound_onion` is inbound in the CNode ctor to have a validity check at the class boundary
- fixes a unit test and a fuzz utility that were passing invalid inbound onion values to the CNode ctor
- drops an unneeded check in `CNode::ConnectedThroughNetwork()` for its inbound status
- adds a public getter `IsInboundOnion()` that also allows unit testing it
- adds unit test coverage
ACKs for top commit:
sipa:
utACK 86c495223f
LarryRuane:
ACK 86c495223f
vasild:
ACK 86c495223f
MarcoFalke:
review ACK 86c495223f🐍
Tree-SHA512: 21109105bc4e5e03076fadd489204be00eac710c9de0127708ca2d0a10a048ff81f640f589a7429967ac3eb51d35fe24bb2b12e53e7aa3efbc47aaff6396d204
fa0074e2d8 scripted-diff: Bump copyright headers (MarcoFalke)
Pull request description:
Needs to be done because no one has removed the years yet
ACKs for top commit:
practicalswift:
ACK fa0074e2d8
Tree-SHA512: 210e92acd7d400b556cf8259c3ec9967797420cfd19f0c2a4fa54cb2b3d32ad9ae27e771269201e7d554c0f4cd73a8b1c1a42c9f65d8685ca4d52e5134b071a3
4ddbcd0d9a fuzz: Add coverage for CDataStream consumer (practicalswift)
546a0764f3 fuzz: Fill various small fuzzing gaps (practicalswift)
Pull request description:
Fill various small fuzzing gaps.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
MarcoFalke:
review ACK 4ddbcd0d9a
Tree-SHA512: d20f2cc0172f39948673846d088121782f39b4556df8b38fa14859cfa062c1519d18ee9601d4503ef1ba9613976cc5349c1fc0f0b9601a3d68127ffce1b1854e
5a9ee0869b tests: Add fuzzing harness for node eviction logic (practicalswift)
Pull request description:
Add fuzzing harness for node eviction logic.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
MarcoFalke:
cr ACK 5a9ee0869b
Tree-SHA512: c2401d22134867e23dab1ba94ae7ef36fdf52aa0588fdc4705d9cb765ddf979fd775fdf153ce2359f1bc1787cf60bf0ebcd47c7aa29c672e6a253fa58cac292d
Upstream revision: 6d0488f75b/compiler-rt/include/fuzzer/FuzzedDataProvider.h
Changes:
* [compiler-rt] FuzzedDataProvider: add ConsumeData and method.
* [compiler-rt] Fix a typo in a comment in FuzzedDataProvider.h.
* [compiler-rt] Add ConsumeRandomLengthString() version without arguments.
* [compiler-rt] Refactor FuzzedDataProvider for better readability.
* [compiler-rt] FuzzedDataProvider: make linter happy.
* [compiler-rt] Mark FDP non-template methods inline to avoid ODR violations.
fa13e1b0c5 build: Add option --enable-danger-fuzz-link-all (MarcoFalke)
44444ba759 fuzz: Link all targets once (MarcoFalke)
Pull request description:
Currently the linker is invoked more than 150 times when compiling with `--enable-fuzz`. This is problematic for several reasons:
* It wastes disk space north of 20 GB, as all libraries and sanitizers are linked more than 150 times
* It wastes CPU time, as the link step can practically not be cached (similar to ccache for object files)
* It makes it a blocker to compile the fuzz tests by default for non-fuzz builds #19388, for the aforementioned reasons
* The build file is several thousand lines of code, without doing anything meaningful except listing each fuzz target in a highly verbose manner
* It makes writing new fuzz tests unnecessarily hard, as build system knowledge is required; Compare that to boost unit tests, which can be added by simply editing an existing cpp file
* It encourages fuzz tests that re-use the `buffer` or assume the `buffer` to be concatenations of seeds, which increases complexity of seeds and complexity for the fuzz engine to explore; Thus reducing the effectiveness of the affected fuzz targets
Fixes#20088
ACKs for top commit:
practicalswift:
Tested ACK fa13e1b0c5
sipa:
ACK fa13e1b0c5. Reviewed the code changes, and tested the 3 different test_runner.py modes (run once, merge, generate). I also tested building with the new --enable-danger-fuzz-link-all
Tree-SHA512: 962ab33269ebd51810924c51266ecc62edd6ddf2fcd9a8c359ed906766f58c3f73c223f8d3cc49f2c60f0053f65e8bdd86ce9c19e673f8c2b3cd676e913f2642
8c09c0c1d1 fuzz: Avoid time-based "non-determinism" in fuzzing harnesses by using mocked GetTime() (practicalswift)
Pull request description:
Avoid time-based "non-determinism" in fuzzing harnesses by using mocked `GetTime()`.
Prior to this commit the fuzzing harnesses `banman`, `connman`, `net` and `rbf` had time-based "non-determinism". `addrman` is fixed in #20425. `process_message` and `process_messages` are left to fix: simply using mock time is not enough for them due to interaction with `IsInitialBlockDownload()`.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
MarcoFalke:
review ACK 8c09c0c1d1
practicalswift:
> review ACK [8c09c0c](8c09c0c1d1)
Tree-SHA512: 32dfbead3dfd18cf4ff56dc2ea341aa977441b4e19a54879cf54fa5820c7e2b14b92c7e238d32fd785654f3b28cc82826ae66c03e94c292633c63c41196ba9a8
fac39c1983 wallet: document that tx in CreateTransaction is purely an out-param (MarcoFalke)
faac31521b Remove unused and confusing CTransaction constructor (MarcoFalke)
Pull request description:
The constructor is confusing and dangerous (as explained in the TODO), fix that by removing it.
ACKs for top commit:
laanwj:
Code review ACK fac39c1983
promag:
Code review ACK fac39c1983.
theStack:
Code review ACK fac39c1983
Tree-SHA512: e0c8cffce8d8ee0166b8e1cbfe85ed0657611e26e2af0d69fde70eceaa5d75cbde3eb489af0428fe4fc431360b4c791fb1cc21b8dee7d4c7a4f17df00836229d
