519b0bc5dc Make last disconnected block BLOCK_FAILED_VALID, even when aborted (Pieter Wuille)
8d220417cd Optimization: don't add txn back to mempool after 10 invalidates (Pieter Wuille)
9ce9c37004 Prevent callback overruns in InvalidateBlock and RewindBlockIndex (Pieter Wuille)
9bb32eb571 Release cs_main during InvalidateBlock iterations (Pieter Wuille)
9b1ff5c742 Call InvalidateBlock without cs_main held (Pieter Wuille)
241b2c74ac Make RewindBlockIndex interruptible (Pieter Wuille)
880ce7d46b Call RewindBlockIndex without cs_main held (Pieter Wuille)
436f7d735f Release cs_main during RewindBlockIndex operation (Pieter Wuille)
1d342875c2 Merge the disconnection and erasing loops in RewindBlockIndex (Pieter Wuille)
32b2696ab4 Move erasure of non-active blocks to a separate loop in RewindBlockIndex (Pieter Wuille)
9d6dcc52c6 Abstract EraseBlockData out of RewindBlockIndex (Pieter Wuille)
Pull request description:
This PR makes a number of improvements to the InvalidateBlock (`invalidateblock` RPC) and RewindBlockIndex functions, primarily around breaking up their long-term cs_main holding. In addition:
* They're made safely interruptible (`bitcoind` can be shutdown, and no progress in either will be lost, though if incomplete, `invalidateblock` won't continue after restart and will need to be called again)
* The validation queue is prevented from overflowing (meaning `invalidateblock` on a very old block will not drive bitcoind OOM) (see #14289).
* `invalidateblock` won't bother to move transactions back into the mempool after 10 blocks (optimization).
This is not an optimal solution, as we're relying on the scheduler call sites to make sure the scheduler doesn't overflow. Ideally, the scheduler would guarantee this directly, but that needs a few further changes (moving the signal emissions out of cs_main) to prevent deadlocks.
I have manually tested the `invalidateblock` changes (including interrupting, and running with -checkblockindex and -checkmempool), but haven't tried the rewinding (which is probably becoming increasingly unnecessary, as very few pre-0.13.1 nodes remain that would care to upgrade).
Tree-SHA512: 692e42758bd3d3efc2eb701984a8cb5db25fbeee32e7575df0183a00d0c2c30fdf72ce64c7625c32ad8c8bdc56313da72a7471658faeb0d39eefe39c4b8b8474
04cca33094 Style cleanup. (Jim Posen)
4c01e4e159 flatfile: Unit tests for FlatFileSeq methods. (Jim Posen)
65a489e93d scripted-diff: Rename CBlockDiskPos to FlatFilePos. (Jim Posen)
d6d8a78f26 Move CDiskBlockPos from chain to flatfile. (Jim Posen)
e0380933e3 validation: Refactor file flush logic into FlatFileSeq. (Jim Posen)
992404b31e validation: Refactor block file pre-allocation into FlatFileSeq. (Jim Posen)
e2d2abb99f validation: Refactor OpenDiskFile into method on FlatFileSeq. (Jim Posen)
9183d6ef65 validation: Extract basic block file logic into FlatFileSeq class. (Jim Posen)
62e7addb63 util: Move CheckDiskSpace to util. (Jim Posen)
Pull request description:
This cleans up and refactors block file helpers so that they may be used by the block filter indexer. Per [design discussion](https://github.com/bitcoin/bitcoin/pull/14121#issuecomment-451252591) about storing BIP 157 block filters, it has been suggested that they are stored in the same way as block and undo data. This refactor is sufficient to simplify file operations for this use case, though in the future perhaps more pruning-related logic ought to be moved into the new classes.
The basic abstraction is a `FlatFileSeq` which manages access to a sequence of numbered files into which raw data is written.
Tree-SHA512: b2108756777f2dad8964a1a2ef2764486e708a4a4a8cfac47b5de8bcb0625388964438eb096b10cfd9ea39212c299b5cb32fa943e768db2333cf49ea7def157e
Due to miners inserting garbage into the version numbers, the current
version signalling has become completely useless. This removes the
"unknown block versions" warning which has the tendency to scare
users unnecessarily (and might get them to "update" to something
bad).
It preserves the warning in the logs. Whether this is desirable can
be a point of discussion.
Note that the former 'else' branch in RewindBlockIndex is now
dealt with more naturally inside the EraseBlockData call (by
checking whether the parent needs to be re-added as candidate
after deleting a child).
04da9f4834 [RPC] Update getrawtransaction interface (Amiti Uttarwar)
Pull request description:
- stop checking unspent UTXOs for a transaction when txindex is not enabled, as per conversation here: https://github.com/bitcoin/bitcoin/issues/3220#issuecomment-377458383
- code contributed by sipa
Tree-SHA512: aa07353bccc14b81b7803992a25d076d6bc06d15ec7c1b85828dc10aea7e0498d9b49f71783e352ab8a14b0bb2010cfb7835de3dfd1bc6f2323f460449348e66
fa5e373365 validation: Add cs_main locking annotations (MarcoFalke)
fa5c346c5a doc: Add comment to cs_main and mempool::cs (MarcoFalke)
fafe941bdd test: Add missing validation locks (MarcoFalke)
fac4558462 sync: Add RecursiveMutex type alias (MarcoFalke)
Pull request description:
Both the chain state and the transaction pool are validation specific, but access to them is protected by two locks. The two locks have the following semantics:
* Writing to the chain state or adding transactions to the transaction pool -> Take both `cs_main` and `mempool::cs`
* Reading either or removing transactions from the the transaction pool -> Take only the appropriate lock
Tree-SHA512: 6f6e612ffc391904c6434a79a4f3f8de1b928bf0a3e3434b73561037b395e2b40a70a5a4bd8472dd230e9eacc8e5d5374c904a3c509910cf3971dd7ff59a626c
e58985c916 Log progress while verifying blocks at level 4. (Daniel Kraft)
Pull request description:
When verifying blocks at startup, the progress is printed in 10% increments to logs. When `-checklevel=4`, however, the second half of the verification (connecting the blocks again) does not log the progress anymore. (It is still computed and shown in the UI, but not printed to logs.)
This change makes the behaviour consistent, by adding the missing progress logging also for level-4 checks.
Tree-SHA512: 6a4c5914726fc1a1337de0c5130b20d4edf4e2feeb0aa0449d2ce422b2d8c41e56ede94163a02044d9a28ac4dc6624b1ad611da93ce5792ff32ad9fb1f0ea1e0
cb53b825c2 scripted-diff: Replace boost::bind with std::bind (Chun Kuan Lee)
2196c51821 refactor: Use boost::scoped_connection in signal/slot, also prefer range-based loop instead of std::transform (Chun Kuan Lee)
Pull request description:
Replace boost::bind with std::bind
- In `src/rpc/server.cpp`, replace `std::transform` with simple loop.
- In `src/validation.cpp`, store the `boost::signals2::connection` object and use it to disconnect.
- In `src/validationinterface.cpp`, use 2 map to store the `boost::signals2::scoped_connection` object.
Tree-SHA512: 6653cbe00036fecfc495340618efcba6d7be0227c752b37b81a27184433330f817e8de9257774e9b35828026cb55f11ee7f17d6c388aebe22c4a3df13b5092f0
66e15e8f97 Explain GetAncestor check for m_failed_blocks in AcceptBlockHeader (Sjors Provoost)
Pull request description:
Salvaged (but slightly modified) from #12138, the comment there was really helpful to wrap my head around that part of the code.
In addition, a naive reader like yours truly will first think `IsValid(BLOCK_VALID_SCRIPTS)` means the previous block was invalid. But IIUC that's not what it means. Instead, it means the block hasn't been checked for validity at the `BLOCK_VALID_SCRIPTS` level yet. So in that case the existing text "previous block index isn't valid" is wrong.
Tree-SHA512: 442a319a83290d94697fdf51376463b70454e0f3909d4a45594ddc2e7c26cd19dc703808385a25e26d6d2dddab0aa35ca41722f2e65ee6fe57bbaf62652d3ec8
fbaaf782ce validation: assert that pindexPrev is non-null when required (Karl-Johan Alm)
Pull request description:
In `ContextualCheckBlock`, we are checking if `pindexPrev == nullptr` conditionally at the start, but then assume it is non-`null` later. This removes the latter assumption.
Tree-SHA512: 95f1e9dc839b2cc0e099d155e6180634ece8c6760d00b53e7d27128762e64c92e82d98a5f4a5786b48a4851b17cdbb4b667d3b6a99adb651256e2032de67d05c
b7df96f456 refactor: Drop boost::this_thread::interruption_point and boost::thread_interrupted in main thread (Chun Kuan Lee)
Pull request description:
This PR drops useless `boost::this_thread::interruption_point` and `boost::thread_interrupted` catch. They are only executed in main thread.
Tree-SHA512: a980d098c1a8238e4f0da9493731d7e69b9ca8e010103f442722d0d4cce471cc40a1fafd5f05535ad0e18899b6cf7563ee20e4025f7c7bc15182a0058c028922
fa511e8dad Pass tx pool reference into CheckSequenceLocks (MarcoFalke)
Pull request description:
`CheckSequenceLocks` is called from ATMP and the member function `CTxMemPool::removeForReorg` without passing in the tx pool object that is used in those function's scope and instead using the global `::mempool` instance.
This fix should be refactoring only, since currently there is only one (global) tx pool in normal operation. Though, it fixes hard to track down issues in future settings where more than one mempool exists at a time. (E.g. for tests, rpc or p2p tx relay purposes)
Tree-SHA512: f0804588c7d29bb6ff05ec14f22a16422b89ab31ae714f38cd07f811d7dc7907bfd14e799c4c1c3121144ff22711019bbe9212b39e2fd4531936a4119950fa49
3ccfa34b32 convert C-style (void) parameter lists to C++ style () (Arvid Norberg)
Pull request description:
In C, an empty parameter list, `()`, means the function takes any arguments, and `(void)` means the function does not take any parameters.
In C++, an empty parameter list means the function does not take any parameters.
So, C++ still supports `(void)` parameter lists with the same semantics, why change to `()`?
1. removing the redundant `void` improves signal-to-noise ratio of the code
2. using `(void)` exposes a rare inconsistency in that a template taking a template `(T)` parameter list, cannot be instantiated with `T=void`
Tree-SHA512: be2897b6c5e474873aa878ed6bac098382cd21866aec33752fe40b089a6331aa6263cae749aba1b4a41e8467f1a47086d32eb74abaf09927fd5a2f44a4b2109a
893628be01 Drop minor GetSerializeSize template (Ben Woosley)
da74db0940 Drop unused GetType() from CSizeComputer (Ben Woosley)
Pull request description:
Based on conversation in #13462, it seems the serialization `GetType` has very narrow use/effect. In every case except for `CAddress`, which specifically relates to a network peer's address, not a wallet address etc., the serialized representation of an object is irrespective of its destination / type.
This removes the unused `GetType` method from `CSizeComputer` as a step to further narrowing that use.
Tree-SHA512: e72b8e9e5160396691e05aeaee3aba5a57935a75bd5005cfcc7fb51c936f3d1728a397f999da5c36696506dd815fafa5c738f3894df8864f25f91f639eba9c3d
f34c8c466a Make objects in range declarations immutable by default. Avoid unnecessary copying of objects in range declarations. (practicalswift)
Pull request description:
Make objects in range declarations immutable by default.
Rationale:
* Immutable objects are easier to reason about.
* Prevents accidental or hard-to-notice change of value.
Tree-SHA512: cad69d35f0cf8a938b848e65dd537c621d96fe3369be306b65ef0cd1baf6cc0a9f28bc230e1e383d810c555a6743d08cb6b2b0bd51856d4611f537a12e5abb8b
Call sync.h primitives "locks" and "mutexes" instead of "blocks" and "waitable
critical sections" to match current coding conventions and c++11 standard
names.
This PR does not rename the "CCriticalSection" class (though this could be done
as a followup) because it is used everywhere and would swamp the other changes
in this PR. Plain mutexes should mostly be preferred instead of recursive
mutexes in new code anyway.
-BEGIN VERIFY SCRIPT-
set -x
set -e
ren() { git grep -l $1 | xargs sed -i s/$1/$2/; }
ren CCriticalBlock UniqueLock
ren CWaitableCriticalSection Mutex
ren CConditionVariable std::condition_variable
ren cs_GenesisWait g_genesis_wait_mutex
ren condvar_GenesisWait g_genesis_wait_cv
perl -0777 -pi -e 's/.*typedef.*condition_variable.*\n\n?//g' src/sync.h
-END VERIFY SCRIPT-
9c4dc597dd Use LOCK macros for non-recursive locks (Russell Yanofsky)
1382913e61 Make LOCK, LOCK2, TRY_LOCK work with CWaitableCriticalSection (Russell Yanofsky)
ba1f095aad MOVEONLY Move AnnotatedMixin declaration (Russell Yanofsky)
41b88e9337 Add unit test for DEBUG_LOCKORDER code (Russell Yanofsky)
Pull request description:
Make LOCK macros work with non-recursive mutexes, and use wherever possible for better deadlock detection.
Also add unit test for DEBUG_LOCKORDER code.
Tree-SHA512: 64ef209307f28ecd0813a283f15c6406138c6ffe7f6cbbd084161044db60e2c099a7d0d2edcd1c5e7770a115e9b931b486e86c9a777bdc96d2e8a9f4dc192942
fa309dc305 validation: Log FormatStateMessage on ConnectBlock error in ConnectTip (MarcoFalke)
Pull request description:
This change additionally logs the validation state on error, which is not logged at all on current master.
Before:
```
ERROR: ConnectTip(): ConnectBlock ffffff.... failed
```
After:
```
ERROR: ConnectTip: ConnectBlock ffffff.... failed, bad-cb-amount (code 16)
```
Tree-SHA512: e69ee0266772b3f77c0193c4a959c2444bf1a51259bd29d790cf665582b037997e520c8567f70b36362c071dcfe1a8ebd7c0f2286cf1b842df5731960e7e1ba0
fa587773e5 scripted-diff: Remove unused first argument to addUnchecked (MarcoFalke)
fe5c49766c tx pool: Use the entry's hash instead of the one passed to addUnchecked (MarcoFalke)
ddd395f968 Mark CTxMemPoolEntry members that should not be modified const (MarcoFalke)
Pull request description:
Several years ago the transaction hash was not cached. For optimization the hash was instead passed into `addUnchecked` to avoid re-calculating it. See f77654a0e9
Passing in the hash is now redundant and the argument can safely be removed.
Tree-SHA512: 0206b65c7a014295f67574120e8c5397bf1b1bd70c918ae1360ab093676f7f89a6f084fd2c7000a141baebfe63fe6f515559e38c4ac71810ba64f949f9c0467f
When verifying blocks at startup, the progress is printed in 10%
increments to logs. When -checklevel=4, however, the second half
of the verification (connecting the blocks again) does not log the
progress anymore. (It is still computed and shown in the UI, but
not printed to logs.)
This change makes the behaviour consistent, by adding the missing
progress logging also for level-4 checks.
faa24441ec policy: Remove promiscuousmempoolflags (MarcoFalke)
Pull request description:
It seems odd to clutter validation code with features that can only ever be used for testing (testnet or regtest). Removing that test-only code makes the mempool logic less painful to understand and easier to reason about when changed or refactored in the future.
Tree-SHA512: 3b897aa9604ac8d82ebe9573c6efd468c93ddaa08d378ebc902e247b7aa6c68fcde71e5b449c08f17a067146cdc66dc50a67ce06d07607c27e5189a49c3fba3f
db6eb90094 [doc] Remove outdated comment about mining code ignoring CPFP (James O'Beirne)
Pull request description:
BlockAssembler chooses transactions on the basis of packages (which incorporate
unconfirmed ancestors into feerate), so the specified RBF comment about mining
code ignoring CPFP is out of date.
Tree-SHA512: a4c1e60fee0a8f450526d565951187f869d000febce0eea8a8d2e18bb140c3c1b8602953d9dcab2d1e8d0c4fc8d392c67eb0773d67e52080d48e6b9bf13f9ee2
BlockAssembler chooses transactions on the basis of packages (which incorporate
unconfirmed ancestors into feerate), so the specified RBF comment about mining
code ignoring CPFP is out of date.
ac51a26bdc During IBD, when doing pruning, prune 10% extra to avoid pruning again soon after (Luke Dashjr)
Pull request description:
Pruning forces a chainstate flush, which can defeat the dbcache and harm performance significantly.
Alternative to #11359
Tree-SHA512: 631e4e8f94f5699e98a2eff07204aa2b3b2325b2d92e8236b8c8d6a6730737a346e0ad86024e705f5a665b25e873ab0970ce7396740328a437c060f99e9ba4d9
fa324a8b15 doc: Rewrite some validation doc as lock annotations (MarcoFalke)
Pull request description:
#13402 added some lock annotations in comments. This pull removes them and adds clang-readable locking annotations instead.
Tree-SHA512: 2d392efa8ac4978830a9df08b2009e69d6f1ac031f62be2275ae8d7c7e483331c7f8d458d865443af907a7af27a592421c6cca6b2df3f2877e0f369b9198f383
1fabd59e7 Break circular dependency: init -> * -> init by extracting shutdown.h (Ben Woosley)
e62fdfeea Drop unused init.h includes (Ben Woosley)
Pull request description:
Most includers just wanted to react to pending shutdown.
This isolates access to `fRequestShutdown` and limits access to the shutdown api functions, including the new `CancelShutdown` for setting it to `false`.
Tree-SHA512: df42f75dfbba163576710e9a67cf1228531fd99d70a2f187bfba0bcc476d6749cf88180a97e66a81bb5b6c3c7f0917de7402d26039ba7b644cb7509b02f7e267
f618ebc4e4 validation: count blocks correctly for check level < 3 (Karl-Johan Alm)
Pull request description:
As noted in https://github.com/bitcoin/bitcoin/pull/13428#issuecomment-396129295 there is a bug where if check level < 3, the resulting count for blocks is wrong, because `pindexState` is never updated.
Post-commit `./bitcoin-cli verifychain 1 3`:
```
2018-06-11T07:12:28Z Verifying last 3 blocks at level 1
2018-06-11T07:12:28Z [0%]...[33%]...[66%]...[99%]...[DONE].
2018-06-11T07:12:28Z No coin database inconsistencies in last 3 blocks (0 transactions)
```
Pre-commit `./bitcoin-cli verifychain 1 3`:
```
2018-06-11T07:13:34Z Verifying last 3 blocks at level 1
2018-06-11T07:13:34Z [0%]...[33%]...[66%]...[99%]...[DONE].
2018-06-11T07:13:34Z No coin database inconsistencies in last 0 blocks (0 transactions)
```
Tree-SHA512: 3d82ed26665162c9615fb0e6e91a46ed4d229a5e6797c6c420e6b0bf1be6e5e02401c6e9a93b7a5aec503a2650d8c20d1b45fe300a922379e4cef8ee26e18d96
Most includers just wanted to react to pending shutdown.
This isolates access to `fRequestShutdown` and limits access to the shutdown
api functions, including the new `AbortShutdown` for setting it to `false`.
Note I originally called `AbortShutdown` `CancelShutdown` but that name was
already taken by winuser.h
https://travis-ci.org/bitcoin/bitcoin/jobs/386913329
This change also triggered a build error in bench. Fixing it required moving LIBBITCOIN_SERVER after LIBBITCOIN_WALLET in bench_bench_bitcoin_LDADD To make
server definitions in src/net.cpp available to wallet methods in
src/wallet/wallet.cpp. Specifically, solving:
libbitcoin_wallet.a(libbitcoin_wallet_a-wallet.o): In function `CWalletTx::RelayWalletTransaction(CConnman*)':
wallet.cpp:(.text+0x3f0e): undefined reference to `CConnman::NodeFullyConnected(CNode const*)'
collect2: error: ld returned 1 exit status
https://travis-ci.org/bitcoin/bitcoin/jobs/392133581
Need for remaining init.h includes confirmed via a thorough search with a more
specific regex:
\bInterrupt\(\)|\bShutdown\(\)|\bInitLogging\(\)|\bInitParameterInteraction\(\)|\bAppInitBasicSetup\(\)|\bAppInitParameterInteraction\(\)|\bAppInitSanityChecks\(\)|\bAppInitLockDataDirectory\(\)|\bAppInitMain\(\)|\bSetupServerArgs\(\)|\bLicenseInfo\(\)|g_wallet_init_interface|init.h
d92204c900 build: add warning to detect hidden copies in range-for loops (Cory Fields)
466e16e0e8 cleanup: avoid hidden copies in range-for loops (Cory Fields)
Pull request description:
Following-up on #13241, which was itself a follow-up of #12169.
See title. Fixing these would otherwise be a continuous process, adding the warning should keep them from cropping up.
Note that the warning seems to be Clang-only for now.
Tree-SHA512: ccfb769c3128b3f92c95715abcf21ee2496fe2aa384f80efead1529a28eeb56b98995b531b49a089f8142601389e63f7bb935963d724eacde4f5e1b4a024934b
f74894480 Only set fNewBlock to true in AcceptBlock when we write to disk (Matt Corallo)
fa6e49731 rpc: Avoid "duplicate" return value for invalid submitblock (MarcoFalke)
Pull request description:
This is #13395 with one more commit tacked on. MarcoFalke got tired of dealing with the stupidity of fixing a return code with too many rounds of review (not that I blame him). Honestly we should probably have no return whatsoever, but for now, this fixes it (as well as nLastBlockTime for eviction purposes).
Original description:
When `submitblock` of an invalid block, the return value should not be `"duplicate"`.
This is only seen when the header was previously found (denoted by the incorrectly named boolean `fBlockPresent`). Fix this bug by removing `fBlockPresent`.
Tree-SHA512: 0ce3092655d5d904b4c8c5ff7479f73ce387144a738f20472b8af132564005c6db5594ae366e589508f6258506ee7a28b1c7995a83a8328b334f99316006bf2d
fafa270328 Make ReceivedBlockTransactions return void (MarcoFalke)
Pull request description:
Instead of always returning `bool{true}` and forcing the caller to handle the return code, make it void and remove "a bunch" of dead code at the call sites.
Tree-SHA512: 10e41461c0516c0441d8b8eedcf6385874355c224b9e9d65e89addb142b4cf3e3be2d4ca0a7f2bd95c76aecdaa8537b6bd2d25631bf804bc42863ad5e84fa271
9b72c988a0 scripted-diff: Avoid temporary copies when looping over std::map (Ben Woosley)
Pull request description:
The ::value_type of the std::map/std::multimap/std::unordered_map containers is
std::pair<const Key, T>. Dropping the const results in an unnecessary copy,
for example in C++11 range-based loops.
For this I started with a more general scripted diff, then narrowed it down
based on the inspection showing that all actual map/multimap/unordered_map
variables used in loops start with m or have map in the name.
Tree-SHA512: b656d66b69ffa1eb954124aa8ae2bc5436ca50262abefa93bdda55cfcdaffc5ff90cd40539051a2bd06355ba69ddf245265cc8764eebff66d761b3aec06155a9
25bc9615b7 Document validationinterace callback blocking deadlock potential. (Matt Corallo)
Pull request description:
From the branches-I've-had-lying-around-and-forgot-to-PR department...
This is a comment-only PR, but the comments point out an API quirk that isn't exactly trivial. None of our use-cases right now hit this, but if we were to call SyncWithValidationInterfaceQueue (eg to limit queue depth) in ATMP, I'm pretty sure we'd hit a deadlock there.
Tree-SHA512: 889dd8fc9eb15d1f2aa5ca467e783bc8f07bc543b166b032741795b0db7a0df11a2846d3cb7c69bafa8d1acf970021001b742f52be06725a932813230c5b4a7b
f6f8026e40 validation: check the specified number of blocks (off-by-one) (Karl-Johan Alm)
Pull request description:
```
echeveria | 2018-06-11 02:03:03.384975 Verifying last 3 blocks at level 3
echeveria | 2018-06-11 02:03:23.676793 No coin database inconsistencies in last 4 blocks (6564 transactions)
echeveria | off by one?
sipa | echeveria: possibly!
kallewoof | Looks like it checks one more block than suggested. `if (pindex->nHeight < chainActive.Height()-nCheckDepth) break;` should probably be `<=`.
sipa | kallewoof: agree
```
Post-commit:
```
2018-06-11T05:24:02Z Verifying last 6 blocks at level 3
2018-06-11T05:24:02Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2018-06-11T05:25:07Z No coin database inconsistencies in last 6 blocks (7258 transactions)
```
Pre-commit:
```
2018-06-11T05:27:11Z Verifying last 6 blocks at level 3
2018-06-11T05:27:11Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2018-06-11T05:27:12Z No coin database inconsistencies in last 7 blocks (9832 transactions)
```
Tree-SHA512: 6e68dc4ba74232518c2ba8ea624d65893534f3619d43ccdf0b9c65992f25b68cb52cf54fa35e6e3d092d1eee5c9a8887057828895f1acdafc0ebb48f683fffdc
The only affect this should have is fixing the return code in
submitblock in cases where a block fails ContextualCheckBlock and
not setting nLastBlockTime on peers that provide blocks which fail
ContextualCheckBlock (which is only used in eviction and cosmetic).
The ::value_type of the std::map/std::multimap/std::unordered_map containers is
std::pair<const Key, T>. Dropping the const results in an unnecessary copy,
for example in C++11 range-based loops.
For this I started with a more general scripted diff, then narrowed it down
based on the inspection showing that all actual map/multimap/unordered_map
variables used in loops start with m or have map in the name.
-BEGIN VERIFY SCRIPT-
sed -i -E 's/for \(([^<]*)std::pair<([^c])(.+) : m/for (\1std::pair<const \2\3 : m/' src/*.cpp src/**/*.cpp
sed -i -E 's/for \(([^<]*)std::pair<([^c])(.+) : (.*)map/for (\1std::pair<const \2\3 : \4map/' src/*.cpp src/**/*.cpp
-END VERIFY SCRIPT-
0bf431870e net: Serve blocks directly from disk when possible (Wladimir J. van der Laan)
Pull request description:
In `ProcessGetBlockData`, send the block data directly from disk if type MSG_WITNESS_BLOCK is requested. This is a valid shortcut as the on-disk format matches the network format.
This is expected to increase performance because a deserialization and subsequent serialization roundtrip is avoided.
Tree-SHA512: 9a9500b4c1354eaae1a6f1c6ef2416c1c1985029852589266f3a70e808f6c7482c135e9ab251a527566935378ab7c32dba4ed43ba5451e802d8e72b77d1ba472
dd435ad Add unit tests for signals generated by ProcessNewBlock() (Jesse Cohen)
a3ae8e6 Fix concurrency-related bugs in ActivateBestChain (Jesse Cohen)
ecc3c4a Do not unlock cs_main in ABC unless we've actually made progress. (Matt Corallo)
Pull request description:
Originally this PR was just to add tests around concurrency in block validation - those tests seem to have uncovered another bug in ActivateBestChain - this now fixes that bug and adds tests.
ActivateBestChain (invoked after a new block is validated) proceeds in steps - acquiring and releasing cs_main while incrementally disconnecting and connecting blocks to sync to the most work chain known (FindMostWorkChain()). Every time cs_main is released the result of FindMostWorkChain() can change - but currently that value is cached across acquisitions of cs_main and only refreshed when an invalid chain is explored. It needs to be refreshed every time cs_main is reacquired. The test added in 6094ce7304 will occasionally fail without the commit fixing this issue 26bfdbaddb
Original description below
--
After a bug discovered where UpdatedBlockTip() notifications could be triggered out of order (#12978), these unit tests check certain invariants about these signals.
The scheduler test asserts that a SingleThreadedSchedulerClient processes callbacks fully and sequentially.
The block validation test generates a random chain and calls ProcessNewBlock from multiple threads at random and in parallel. ValidationInterface callbacks verify that the ordering of BlockConnected BlockDisconnected and UpdatedBlockTip events occur as expected.
Tree-SHA512: 4102423a03d2ea28580c7a70add8a6bdb22ef9e33b107c3aadef80d5af02644cdfaae516c44933924717599c81701e0b96fbf9cf38696e9e41372401a5ee1f3c
In `ProcessGetBlockData`, send the block data directly from disk if
type MSG_WITNESS_BLOCK is requested. This is a valid shortcut as the
on-disk format matches the network format.
This is expected to increase performance because a deserialization and
subsequent serialization roundtrip is avoided.
11fa6bb66e Bugfix: ensure consistency of m_failed_blocks after reconsiderblock (Suhas Daftuar)
Pull request description:
This was introduced in 015a5258ad and could cause a node to crash (due to assertion failure) when using the `reconsiderblock` rpc.
Tree-SHA512: 820dcd761bf983e36f5d0f16777ed75c833daaf62a6b3a4dbd17f6caaf9287223e3a202d06540ac62f8ba72926b73b0873bb76c6273ddcb19d9408f4c1cd325e
If multiple threads are invoking ActivateBestChain, it was possible to have
them working towards different tips, and we could arrive at a less work tip
than we should. Fix this by introducing a ChainState lock which must
be held for the entire duration of ActivateBestChain to enforce
exclusion in ABC.
Technically, some internal datastructures may be in an inconsistent
state if we do this, though there are no known bugs there. Still,
for future safety, its much better to only unlock cs_main if we've
made progress (not just tried a reorg which may make progress).
364bae5 qa: Pad scriptPubKeys to get minimum sized txs (MarcoFalke)
7485488 Policy to reject extremely small transactions (Johnson Lau)
0f8719b Add transaction tests for constant scriptCode (Johnson Lau)
9dabfe4 Add constant scriptCode policy in non-segwit scripts (Johnson Lau)
Pull request description:
This disables `OP_CODESEPARATOR` in non-segwit scripts (even in an unexecuted branch), and makes a positive `FindAndDelete` result invalid. This ensures that the `scriptCode` serialized in `SignatureHash` is always the same as the script passing to the `EvalScript`.
Tree-SHA512: a0552cb920294d130251c48053fa2ff1fbdd26332e62b52147d918837852750f0ce35ce2cd1cbdb86588943312f8154ccb4925e850dbb7c2254bc353070cd5f8
