d0ea9bab28 walletdb: Don't remove database transaction logs and instead error (Andrew Chow)
Pull request description:
Instead of removing the database transaction logs and retrying the
wallet loading, just return an error message to the user. Additionally,
speciically for DB_RUNRECOVERY, notify the user that this could be due
to different BDB versions.
Kind of implements the suggestion from https://github.com/bitcoin/bitcoin/pull/18870#discussion_r421647964
ACKs for top commit:
Sjors:
re-utACK d0ea9bab28
ryanofsky:
Code review ACK d0ea9bab28. Only changes since last review are rebase and expanding error and commit messages.
Tree-SHA512: f6e67dc70f58188742a5c8af7cdc63a2b58779aa0d26ae7f1e75805a239f1a342433860e5a238d6577fae5ab04b9d15e7f11c55b867065dfd13781a6a62e4958
and also
- add missing parentheses in fuzz/scriptnum_ops.cpp
- remove useless unsigned int conditional in fuzz/script.cpp
These changes fix 5 compile warnings in gcc 10.
ec4c6a17e8 scripted-diff: replace MAX_BLOCKS_ONLY_CONNECTIONS with MAX_BLOCK_RELAY_ONLY_CONNECTIONS (glowang)
Pull request description:
We have two different concepts that have similar names: `-blocksonly` and `block-relay-only`, and the similarity of names could lead to confusion. `-blocksonly` disables all local receiving & relaying of transactions (with a few exceptions), while `block-relay-only`means that bitcoind will make 2 additional outbound connections that are only used for block relay.
In net.h and init.cpp, `MAX_BLOCKS_ONLY_CONNECTIONS` is used to represent the maximum number of `block-relay-only` outbound peers, which is 2. But this name sounds ambiguous, and I proposed a better name, `MAX_BLOCK_RELAY_ONLY_CONNECTION`.
ACKs for top commit:
jnewbery:
ACK ec4c6a17e8
Tree-SHA512: cfa592a7ff936f14d10cfc1e926a51b82bc0feaf104885a41ca8111b906cb3d1ec5536bab143a3cfca70aa49e9575c6995941eb6d3d7f4018d4535712342f155
12410b1feb test: fix intermittent p2p_ibd_txrelay race, add test_framework.py#wait_until (Jon Atack)
Pull request description:
To fix these intermittent failures in Travis CI.
```
162/163 - p2p_ibd_txrelay.py failed, Duration: 2 s
stdout:
2020-07-19T05:44:17.213000Z TestFramework (INFO):
Check that nodes set minfilter to MAX_MONEY while still in IBD
2020-07-19T05:44:17.216000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
File "/Users/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-apple-darwin16/test/functional/test_framework/test_framework.py", line 117, in main
self.run_test()
File "/Users/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-apple-darwin16/test/functional/p2p_ibd_txrelay.py", line 30, in run_test
assert_equal(conn_info['minfeefilter'], MAX_FEE_FILTER)
File "/Users/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-apple-darwin16/test/functional/test_framework/util.py", line 49, in assert_equal
raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
AssertionError: not(0E-8 == 0.09170997)
2020-07-19T05:44:17.293000Z TestFramework (INFO): Stopping nodes
```
At Marco's suggestion, cherry-picked part of #19134 to nicely simplify using `wait_until`.
ACKs for top commit:
vasild:
ACK 12410b1fe
Tree-SHA512: 615f509883682fd693e578b259cba35a9fa0bc519f1394e88c857e8b0650bfec5397bfa856cfa9e6d5ef81d0ee6ad02e4ad2b0eb0bd530b4c281cbe3e663790b
facdf530c7 contrib: Clean up previous_releases.py (MarcoFalke)
Pull request description:
ACKs for top commit:
fjahr:
tACK facdf530c7
Sjors:
tACK facdf53
hebasto:
ACK facdf530c7, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: c3543320572267035aa342dd170128bbdeb83ca4e2e36a8e46596dd76c8ff1b26ed6759a8073884228b133c45b06ec48889cd0ec83a13bef276b48073d8248e4
5fa067a27d Remove unnecessary blockfile SetPos (Tom Harding)
Pull request description:
Nothing could have changed the position since we retrieved it a few statements earlier. This dates from commit 16d5194165.
ACKs for top commit:
LarryRuane:
ACK 5fa067a27d
Tree-SHA512: 459cc7226e186c231ffb67f0613f550e8eb940f1b8933c3bc4a4e8dd519c8d5d45884e8cfd9347039dab90a093644bbbb31be063baed1c6fc7984b6cb4f17c9f
4455949d6f Make test DoS_mapOrphans deterministic (David Reikher)
Pull request description:
This pull request proposes a solution to make the test `DoS_mapOrphans` in denialofservice_tests.cpp have deterministic coverage.
The `RandomOrphan` function in denialofservice_tests.cpp and the implicitly called function `ecdsa_signature_parse_der_lax` in pubkey.cpp were causing the non-deterministic test coverage.
In the former, if a random orphan was selected the index of which is bigger than the max. orphan index in `mapOrphanTransactions`, the last orphan was returned from `RandomOrphan`. If the random number generated was never large enough, this condition would not be fulfilled and the corresponding branch wouldn't run. The proposed solution is to force one of the 50 dependant orphans to depend on the last orphan in `mapOrphanTransactions` using the newly introduced function `OrphanByIndex` (and passing it a large uint256), forcing this branch to run at least once.
In the latter, if values for ECDSA `R` or `S` (or both) had no leading zeros, some code would not be executed. The solution was to find a constant signature that would be comprised of `R` and `S` values with leading zeros and calling `CPubKey::Verify` at the end of the test with this signature forcing this code to always run at least once at the end even if it hadn't throughout the test.
To test that the coverage is (at least highly likely) deterministic, I ran
`contrib/devtools/test_deterministic_coverage.sh denialofservice_tests/DoS_mapOrphans 1000`
and the result was deterministic coverage across 1000 runs.
Also - removed denialofservice_tests test entry from the list of non-deterministic tests in the coverage script.
ACKs for top commit:
MarcoFalke:
ACK 4455949d6f
Tree-SHA512: 987eb1f94b80d5bec4d4944e91ef43b9b8603055750362d4b4665b7f011be27045808aa9f4c6ccf8ae009b61405f9a1b8671d65a843c3328e5b8acce1f1c00a6
The RandomOrphan function and the function ecdsa_signature_parse_der_lax
in pubkey.cpp were causing non-deterministic test coverage.
Force seed in the beginning of the test to make it deterministic.
The seed is selected carefully so that all branches of the function
ecdsa_signature_parse_der_lax are executed. Prior to this fix, the test
was exhibiting non-deterministic coverage since none of the ECDSA
signatures that were generated during the test had leading zeroes in
either R, S, or both, resulting in some branches of said function not
being executed. The seed ensures that both conditions are hit.
Removed denialofservice_tests test entry from the list of non-deterministic
tests in the coverage script.
6be3eebd74 doc: Update macOS cross compilation dependencies for Focal (Hennadii Stepanov)
Pull request description:
The [`libtinfo5`](https://packages.ubuntu.com/focal/libtinfo5) package is required on Ubuntu Focal for macOS cross compilation.
Fixes#19546.
ACKs for top commit:
fanquake:
ACK 6be3eebd74 I have been [installing the libtinfo5 package](https://github.com/fanquake/core-review/blob/master/docker/debian9.dockerfile#L18) into my cross-compiling images, so mostly forgot about this.
Tree-SHA512: f40ab1efae549276e520f9cc3200438e70297c3081a3f9acc3075ec11d494fd6176ed3f1d4889b39eb2a5706eac835844a75b79335bb7f3c21c985e0779a7562
55057ffc51 rpc: deduplicate WriteHDKeypath() used in decodepsbt (Sebastian Falbesoner)
Pull request description:
The functionality is already provided in the BIP32 utility library `util/bip32.h` with the exact same name and function signature.
ACKs for top commit:
achow101:
ACK 55057ffc51
instagibbs:
utACK 55057ffc51
jonatack:
ACK 55057ffc51
Tree-SHA512: 074c1a71ffb32908926bf07f0c5428a46309f6e0d21e7c20b1008197c820b97776a441736d0b6fd8ab0c0852522a0b5a5ddb26a1e4a1100ca02aabc65a07a018
Starting with the 10.1 release, GCC defaults to -fno-common. This causes
linking issues when building genisoimage:
```bash
[ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/checksum.o
[100%] Linking C executable genisoimage
/usr/bin/ld: CMakeFiles/genisoimage.dir/apple.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/desktop.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_file.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
```
Rather than patching genisoimage further, pass -fcommon to preserve the
legacy GCC behaviour.
Previously, TX_WITNESS_MUTATED could be returned during transaction validation
for either transactions that had a witness that was non-standard, or for
transactions that had no witness but were invalid due to segwit validation
rules.
However, for txid/wtxid-relay considerations, net_processing distinguishes the
witness stripped case separately, because it affects whether a wtxid should be
able to be added to the reject filter. It is safe to add the wtxid of a
witness-mutated transaction to the filter (as that wtxid shouldn't collide with
the txid, and hence it wouldn't interfere with transaction relay from
txid-relay peers), but it is not safe to add the wtxid (== txid) of a
witness-stripped transaction to the filter, because that would interfere with
relay of another transaction with the same txid (but different wtxid) when
relaying from txid-relay peers.
Also updates the comment explaining this logic, and explaining that we can get
rid of this complexity once there's a sufficient deployment of wtxid-relaying
peers on the network.
Using both txid and wtxid-based relay with peers means that we could sometimes
download the same transaction twice, if announced via two different hashes from
different peers.
Use a heuristic of delaying txid-peer-getdata requests by 2 seconds, if we have
at least one wtxid-based peer.
This adds a field to CNodeState that tracks whether to relay transactions with
that peer via wtxid, instead of txid. As of this commit the field will always
be false, but in a later commit we will add a way to negotiate turning this on
via p2p messages exchanged with the peer.
Previously, we only added txids to recentRejects if we were sure that the
transaction couldn't have had the wrong witness (either because the witness was
malleated or stripped).
In preparation for wtxid-based relay, we can observe that txid == wtxid for
transactions that have no witness, and add the wtxid of rejected transactions,
provided the transaction wasn't a witness-stripped one. This means that we now
add more data to the filter (as prior to this commit, any transaction with a
witness that failed to be accepted was being skipped for inclusion in the
filter) but witness malleation should still not interfere with relay of a valid
segwit transaction, because the txid of a segwit transaction would not be added
to the filter after failing validation.
In the future, having wtxids in the recent rejects filter will allow us to
skip downloading the same wtxid multiple times, once our peers use wtxids for
transaction relay.
This is in preparation for wtxid-based invs (we need to be able to tell whether
we AlreadyHave() a transaction based on either txid or wtxid).
This also double the size of the bloom filter, which is overkill, but still
uses a manageable amount of memory.
ad6c34881d tests: Add fuzzing harness for CBlockPolicyEstimator::{Read,Write} (policy/fees.h) (practicalswift)
614e0807a8 tests: Add fuzzing harness for CBufferedFile::{SetPos,GetPos,GetType,GetVersion} (stream.h) (practicalswift)
7bcc71e5f8 tests: Add fuzzing harness for LoadExternalBlockFile(...) (validation.h) (practicalswift)
9823376030 tests: Add fuzzing harness for CBufferedFile (streams.h) (practicalswift)
f3aa659be6 tests: Add fuzzing harness for CAutoFile (streams.h) (practicalswift)
e507c0799d tests: Add serialization/deserialization fuzzing helpers WriteToStream(…)/ReadFromStream(…) (practicalswift)
e48094a506 tests: Add FuzzedAutoFileProvider which provides a CAutoFile interface to FuzzedDataProvider (practicalswift)
9dbcd6854c tests: Add FuzzedFileProvider which provides a FILE* interface to FuzzedDataProvider using fopencookie (practicalswift)
Pull request description:
Add fuzzing harnesses for `CAutoFile`, `CBufferedFile`, `LoadExternalBlockFile` and other `FILE*` consumers:
* Add `FuzzedFileProvider` which provides a `FILE*` interface to `FuzzedDataProvider` using `fopencookie`
* Add `FuzzedAutoFileProvider` which provides a `CAutoFile` interface to `FuzzedDataProvider`
* Add serialization/deserialization fuzzing helpers `WriteToStream(…)`/`ReadFromStream(…)`
* Add fuzzing harness for `CAutoFile` (`streams.h`)
* Add fuzzing harness for `CBufferedFile` (`streams.h`)
* Add fuzzing harness for `LoadExternalBlockFile(...)` (`validation.h`)
* Add fuzzing harness for `CBlockPolicyEstimator::Read` and `CBlockPolicyEstimator::Write` (`policy/fees.h`)
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
Crypt-iQ:
Tested ACK ad6c348
Tree-SHA512: a38e142608218496796a527d7e59b74e30279a2815450408b7c27a76ed600cebc6b88491e831665a0639671e2d212453fcdca558500bbadbeb32b267751f8f72
0cdf2a77dd ci: add tsan debug symbols option (Russell Yanofsky)
9a2f12680b ci: Add tsan suppression for race in DatabaseBatch (Hennadii Stepanov)
Pull request description:
Since #19325 was merged, the corresponding change in TSan suppression file gets required.
This PR is:
- an analogous to #19226 and #19450, and
- a temporary workaround for CI fail like https://cirrus-ci.com/task/5741795508224000?command=ci#L4993
ACKs for top commit:
MarcoFalke:
ACK 0cdf2a77dd
Tree-SHA512: 7832f143887c8a0df99dea03e00694621710378fbe923e3592185fcd3658546a590693b513abffc5ab96e9ef76c9c4bff3330eeee69a0c5dbe7574f34c417220
6457361e90 qt: Fix QFileDialog for static builds (Hennadii Stepanov)
Pull request description:
This change partially reverts 248e22bbc0 (#16386) and makes `QFileDialog`s work again for static builds.
Fixes https://github.com/bitcoin-core/gui/issues/32.
ACKs for top commit:
fanquake:
ACK 6457361e90. Although it would be good to know exactly _why_ this fixes the issue. At this stage I also don't think this should be a blocker for 0.20.1.
theuni:
ACK 6457361e90
Tree-SHA512: 8ad27e0bcae6debd02f73b7c374743e37d4edd806922b103a2fe494cf2d9930fe9ef3107b5a6c61f3c466cf7462de2641171880398954e7f2c4f417f5bb820d7
cb31ee01b4 [test] feefilter during and after IBD (gzhao408)
Pull request description:
This is a followup to #19204 which uses `minfeefilter=MAX_MONEY` to effectively shut off txrelay, thereby reducing inv traffic, when nodes are in IBD. It was [missing](https://github.com/bitcoin/bitcoin/pull/19204#issuecomment-644040070) a functional test.
ACKs for top commit:
jnewbery:
utACK cb31ee01b4
Tree-SHA512: a9effc8193fa95fb42a2f9c66b258cc7b0941fc04c1ce3a6092f4426c9bfc7e72f702aca559b3e30e90652497f411f22fae3cf5cdb6cfd6ef6d37fed712cda67
e80259f197 Additionally treat Tx.nVersion as unsigned in joinpsbts (Matt Corallo)
970de70bdd Dump transaction version as an unsigned integer in RPC/TxToUniv (Matt Corallo)
Pull request description:
Consensus-wise we already treat it as an unsigned integer (the
only rules around it are in CSV/locktime handling), but changing
the underlying data type means touching consensus code for a
simple cleanup change, which isn't really worth it.
See-also, https://github.com/rust-bitcoin/rust-bitcoin/pull/299
ACKs for top commit:
sipa:
ACK e80259f197
practicalswift:
ACK e80259f197
ajtowns:
ACK e80259f197 code review -- checked all other uses of tx.nVersion treat it as unsigned (except for policy.cpp:IsStandard anyway), so looks good.
naumenkogs:
ACK e80259f
Tree-SHA512: 6760a2c77e24e9e1f79a336ca925f9bbca3a827ce02003c71d7f214b82ed3dea13fa7d9f87df9b9445cd58dff8b44a15571d821c876f22f8e5a372a014c9976b
Note that this does not _enable_ lto by default in any way, only hooks up the
machinery for -flto to work correctly.
enable-lto-support is explicitly used for pinned-clang because we know it
works. It is neither enabled nor disabled in the external clang case so that
it can be auto-detected.
de4fedb6c3 depends: Add documentation for FORCE_USE_SYSTEM_CLANG make flag (Carl Dong)
fe98999dcf depends: Reformat make options as definition list (Carl Dong)
60c55b1b9b depends: Add justifications for macOS clang flags (Carl Dong)
6b8e497eea depends: specify libc++ header location for darwin (Cory Fields)
156b604203 depends: force a new host id string if FORCE_USE_SYSTEM_CLANG is in use (Cory Fields)
c9c572a367 depends: Allow building with system clang (Carl Dong)
e6e5c8d6ca depends: Decouple toolchain + binutils (Carl Dong)
Pull request description:
This replaces: #17099
-----
This patchset allows us to force depends to use system clang.
Previously, #17099 removes our dependency on a specific clang we download from llvm.org, but theuni pointed out that since OSX builds are only ever built with a version of clang that is chosen and "blessed" by Apple, it is more likely that the user will encounter problems if they use their system clang. This patchset forces the user to set `FORCE_USE_SYSTEM_CLANG=1` in order to use their system clang (when they know what they're doing)
ACKs for top commit:
theuni:
ACK de4fedb6c3.
Tree-SHA512: 8774121e035f90c27030bcce06e1b79f7729b5e17802c718e49652ab06e19780632db974df47423c1d1b04f1ab1b7a763554fb922fec05d1cd6445b26578be1d