bitcoin/src/test
Wladimir J. van der Laan ab765c2ec7
Merge #16577: util: CBufferedFile fixes and unit test
efd2474d17 util: CBufferedFile fixes (Larry Ruane)

Pull request description:

  The `CBufferedFile` object guarantees its user is able to "rewind" the data stream (that's being read from a file) up to a certain number of bytes, as specified by the user in the constructor. This guarantee is not honored due to a bug in the `SetPos` method.

  Such rewinding is done in `LoadExternalBlockFile()` (currently the only user of this object), which deserializes a series of `CBlock` objects. If that function encounters something unexpected in the data stream, which is coming from a `blocks/blk00???.dat` file, it "rewinds" to an earlier position in the stream to try to get in sync again. The `CBufferedFile` object does not actually rewind its file offset; it simply repositions its internal offset, `nReadPos`, to an earlier position within the object's private buffer; this is why there's a limit to how far the user may rewind.

  If `LoadExternalBlockFile()` needs to rewind (call `blkdat.SetPos()`), the stream may not be positioned as it should be, causing errors in deserialization. This need to rewind is probably rare, which is likely why this bug hasn't been noticed already. But if this object is used elsewhere in the future, this could be a serious problem, especially as, due to the nature of the bug, the `SetPos()` _sometimes_ works.

  This PR adds a unit test for `CBufferedFile` that fails due to this bug. (Until now it has had no unit tests.) The unit test provides good documentation and examples for developers trying to understand `LoadExternalBlockFile()` and for future users of this object.

  This PR also adds code to throw an exception from the constructor if the rewind argument is not less than the buffer size (since that doesn't make any sense).

  Finally, I discovered that the object is too restrictive in one respect: When the deserialization methods call this object's `read` method, a check ensures that the number of bytes being requested is less than the size of the buffer (adjusting for the rewind size), else it throws an exception. This restriction is unnecessary; the object being deserialized can be larger than the buffer because multiple reads from disk can satisfy the request.

ACKs for top commit:
  laanwj:
    ACK ~after squash.~ efd2474d17
  mzumsande:
    I had intended to follow up earlier on my last comment, ACK efd2474d17. I reviewed the code, ran tests and did a successful reindex on testnet with this branch.

Tree-SHA512: 695529e0af38bae2af4e0cc2895dda56a71b9059c3de04d32e09c0165a50f6aacee499f2042156ab5eaa6f0349bab6bcca4ef9f6f9ded4e60d4483beab7e4554
2019-09-26 13:38:39 +02:00
..
data Added comments referencing multiple CVEs in tests and production code. 2019-06-02 10:25:03 -07:00
fuzz tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
gen Integration of property based testing into Bitcoin Core 2018-08-27 08:51:51 -05:00
addrman_tests.cpp test: remove redundant setup in addrman_tests 2019-07-18 17:31:46 -07:00
allocator_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
amount_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
arith_uint256_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
base32_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
base58_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
base64_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
bech32_tests.cpp scripted-diff: Rename test_bitcoin to test/setup_common 2019-04-11 10:12:36 -04:00
bip32_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
blockchain_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
blockencodings_tests.cpp txmempool: Remove unused default value MemPoolRemovalReason::UNKNOWN 2019-07-22 07:40:24 -04:00
blockfilter_index_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
blockfilter_tests.cpp Merge #14121: Index for BIP 157 block filters 2019-04-18 09:48:25 -04:00
bloom_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
bswap_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
checkqueue_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
coins_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
compilerbug_tests.cpp Add test for GCC bug 90348 2019-06-02 10:19:30 +02:00
compress_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
crypto_tests.cpp Add ChaCha20Poly1305@Bitcoin tests 2019-07-03 11:48:48 +02:00
cuckoocache_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
dbwrapper_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
denialofservice_tests.cpp Add 2 outbound block-relay-only connections 2019-09-04 14:58:36 -04:00
descriptor_tests.cpp Make descriptor test deterministic 2019-08-21 08:22:55 +03:00
flatfile_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
fs_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
getarg_tests.cpp test: Make tests arg type specific 2019-07-31 21:58:14 +03:00
hash_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
key_io_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
key_properties.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
key_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
limitedmap_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
main.cpp test: Remove useless test_bitcoin_main.cpp 2019-02-28 15:44:02 -05:00
Makefile build: add stub makefiles for easier subdir builds 2014-06-05 16:05:43 -04:00
mempool_tests.cpp txmempool: Remove unused default value MemPoolRemovalReason::UNKNOWN 2019-07-22 07:40:24 -04:00
merkle_tests.cpp test: add some unit tests for merkle.cpp 2019-09-16 23:36:43 +04:30
merkleblock_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
miner_tests.cpp refactor: pcoinsTip -> CChainState::CoinsTip() 2019-08-06 13:13:06 -04:00
multisig_tests.cpp Move various SigningProviders to signingprovider.{cpp,h} 2019-07-09 16:20:18 -04:00
net_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
netbase_tests.cpp Make whitebind/whitelist permissions more flexible 2019-08-11 11:33:27 +09:00
pmt_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
policyestimator_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
pow_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
prevector_tests.cpp Merge #12324: speed up Unserialize_impl for prevector 2019-06-18 17:12:02 +02:00
raii_event_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
random_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
README.md doc: Delete stale URL in test README 2019-08-29 13:58:00 +03:00
reverselock_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
rpc_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
sanity_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
scheduler_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
script_p2sh_tests.cpp Move various SigningProviders to signingprovider.{cpp,h} 2019-07-09 16:20:18 -04:00
script_standard_tests.cpp Move various SigningProviders to signingprovider.{cpp,h} 2019-07-09 16:20:18 -04:00
script_tests.cpp Move various SigningProviders to signingprovider.{cpp,h} 2019-07-09 16:20:18 -04:00
scriptnum10.h tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
scriptnum_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
serialize_tests.cpp Support serialization of std::vector<bool> 2019-08-26 11:36:33 -07:00
setup_common.cpp Merge #16060: Bury bip9 deployments 2019-08-15 16:02:10 -04:00
setup_common.h tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
sighash_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
sigopcount_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
skiplist_tests.cpp Merge #15670: refactor: combine Chain::findFirstBlockWithTime/findFirstBlockWithTimeAndHeight 2019-04-19 12:03:12 -04:00
streams_tests.cpp util: CBufferedFile fixes 2019-09-10 07:53:09 -06:00
sync_tests.cpp scripted-diff: Bump copyright headers in test, bench 2019-04-11 13:34:43 -04:00
timedata_tests.cpp Add test for AddTimeData 2019-08-13 17:04:10 +02:00
torcontrol_tests.cpp Make reasoning about dependencies easier by not including unused dependencies 2019-06-02 17:15:23 +02:00
transaction_tests.cpp Move various SigningProviders to signingprovider.{cpp,h} 2019-07-09 16:20:18 -04:00
txindex_tests.cpp Make reasoning about dependencies easier by not including unused dependencies 2019-06-02 17:15:23 +02:00
txvalidation_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
txvalidationcache_tests.cpp Remove unused fScriptChecks parameter from CheckInputs 2019-08-26 16:27:58 -07:00
uint256_tests.cpp tests: Reduce compilation time and unneccessary recompiles by removing unused includes in tests 2019-06-26 20:37:48 +02:00
util.cpp Merge #16237: Have the wallet give out destinations instead of keys 2019-07-10 11:45:55 +02:00
util.h Add unit test NextString, ForEachNoDup functions 2019-05-07 11:51:30 -04:00
util_tests.cpp Testchains: Generic selection with -chain=<str> in addition of -testnet and -regtest 2019-09-06 22:05:33 +02:00
util_threadnames_tests.cpp tests: add threadutil tests 2019-04-29 13:43:01 -04:00
validation_block_tests.cpp Merge #14193: validation: Add missing mempool locks 2019-07-02 16:29:08 +02:00
validation_tests.cpp scripted-diff: Rename test_bitcoin to test/setup_common 2019-04-11 10:12:36 -04:00
versionbits_tests.cpp Merge #13555: Tests: parameterize adjustment period in versionbits_computeblockversion 2019-05-21 08:04:11 -04:00

Compiling/running unit tests

Unit tests will be automatically compiled if dependencies were met in ./configure and tests weren't explicitly disabled.

After configuring, they can be run with make check.

To run the bitcoind tests manually, launch src/test/test_bitcoin. To recompile after a test file was modified, run make and then run the test again. If you modify a non-test file, use make -C src/test to recompile only what's needed to run the bitcoind tests.

To add more bitcoind tests, add BOOST_AUTO_TEST_CASE functions to the existing .cpp files in the test/ directory or add new .cpp files that implement new BOOST_AUTO_TEST_SUITE sections.

To run the bitcoin-qt tests manually, launch src/qt/test/test_bitcoin-qt

To add more bitcoin-qt tests, add them to the src/qt/test/ directory and the src/qt/test/test_main.cpp file.

Running individual tests

test_bitcoin has some built-in command-line arguments; for example, to run just the getarg_tests verbosely:

test_bitcoin --log_level=all --run_test=getarg_tests

... or to run just the doubledash test:

test_bitcoin --run_test=getarg_tests/doubledash

Run test_bitcoin --help for the full list.

Note on adding test cases

The sources in this directory are unit test cases. Boost includes a unit testing framework, and since bitcoin already uses boost, it makes sense to simply use this framework rather than require developers to configure some other framework (we want as few impediments to creating unit tests as possible).

The build system is setup to compile an executable called test_bitcoin that runs all of the unit tests. The main source file is called setup_common.cpp. To add a new unit test file to our test suite you need to add the file to src/Makefile.test.include. The pattern is to create one test file for each class or source file for which you want to create unit tests. The file naming convention is <source_filename>_tests.cpp and such files should wrap their tests in a test suite called <source_filename>_tests. For an example of this pattern, examine uint256_tests.cpp.