Commit graph

93 commits

Author SHA1 Message Date
MarcoFalke
fa633aa690
streams: Teach AutoFile how to XOR 2023-07-19 18:12:42 +02:00
MarcoFalke
000019e158
Add AutoFile::detail_fread member function
New code can call the method without having first to retrieve the raw
FILE* pointer via Get().

Also, move implementation to the cpp file. Can be reviewed with:
--color-moved=dimmed-zebra  --color-moved-ws=ignore-all-space
2023-07-13 11:51:07 +02:00
MarcoFalke
fa7724bc9d
refactor: Modernize AutoFile
* Add m_ prefix to the std::FILE member variable
* Add std namespace where possible, to avoid confusion with member
  functions of the same name.
* Add AutoFile::feof() member function, to be used in place of
  std::feof(AutoFile::Get())
* Simplify fclose() in terms of release()
* Fix typo in the error message in the ignore member function.
2023-07-13 11:50:55 +02:00
MarcoFalke
fa8d227d58
doc: Remove comments that just repeat what the code does
No need to artificially bloat the code and waste space.
2023-07-12 10:01:51 +02:00
MarcoFalke
fafe2ca0ce
refactor: Remove redundant file check from AutoFile shift operators
The shift operators will call the write or read member function, which
already does the check. Also, call sites are free to directly call
::(Un)Serialize(s, obj) to skip this check, so removing it increases
consistency.
2023-07-12 10:00:56 +02:00
MarcoFalke
9999a49b32
Extract util::Xor, Add key_offset option, Add bench 2023-07-12 09:59:55 +02:00
Ryan Ofsky
5cd0717a54 streams: Drop confusing DataStream::Serialize method and << operator
DataStream Serialize method has surprising behavior because it just serializes
raw bytes without a length prefix. When you serialize a string or vector, a
length prefix is serialized before the raw object contents so the object can be
unambiguously deserialized later. But DataStreams don't support deserializing
at all and just dump the raw bytes.

Having this inconsistency is not necessary and could be confusing (see
https://github.com/bitcoin/bitcoin/pull/27790#discussion_r1212315030) so this
PR just drops the DataStream::Serialize method.
2023-06-01 10:27:33 -04:00
Larry Ruane
72efc26439 util: improve streams.h:FindByte() performance
Avoid use of the expensive mod operator (%) when calculating the
buffer offset. No functional difference.

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
2023-05-05 06:03:17 -06:00
fanquake
21138fe377
Merge bitcoin/bitcoin#26992: refactor: Remove unused CDataStream SerializeMany constructor
fa47b28dfc refactor: Remove unused CDataStream SerializeMany constructor (MarcoFalke)

Pull request description:

  Seems odd to have an unused method. Moreover, the function is fragile and dangerous, because one could have a `std::vector vec_a` and type `CDataStream{vec_a, 0, 0}.size()` and `CDataStream{0, 0, vec_a}.size()`, assuming they are the same thing, when in fact they are not. (The first takes over the memory as is, the second serializes the vector).

  So my suggestion would be to remove the unused method and introduce a new method when this functionality is needed. For example: `static DataStream FromMany(Args&&... args)`.

ACKs for top commit:
  stickies-v:
    ACK fa47b28dfc

Tree-SHA512: 9593a034b997e33a0794f779f76f02425b1097b218cf8cb1facb7f874fa69da328ce567a79138015baeebe004ae7d103dda4f64f83e8ad375b6dae6b66d3d950
2023-02-02 10:47:37 +00:00
Hennadii Stepanov
96ee992ac3
clang-tidy: Fix modernize-use-default-member-init in headers
See https://clang.llvm.org/extra/clang-tidy/checks/modernize/use-default-member-init.html
2023-01-31 11:50:10 +00:00
MarcoFalke
fa47b28dfc
refactor: Remove unused CDataStream SerializeMany constructor 2023-01-30 13:04:50 +01:00
MarcoFalke
fa035fe2d6
Remove unused CDataStream::SetType
The last use was removed in the previous commit.
2023-01-26 10:45:02 +01:00
MarcoFalke
fa9becfe1c
streams: Add DataStream without ser-type and ser-version
The moved parts can be reviewed with "--color-moved=dimmed-zebra".
The one-char changes can be reviewed with "--word-diff-regex=.".
2023-01-24 13:18:09 +01:00
Hennadii Stepanov
306ccd4927
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
- 2021: f47dda2c58
- 2020: fa0074e2d8
- 2019: aaaaad6ac9
2022-12-24 23:49:50 +00:00
Larry Ruane
c72de9990a util: add CBufferedFile::SkipTo() to move ahead in the stream
SkipTo() reads data from the file into the CBufferedFile object
(memory), but, unlike this object's read() method, SkipTo() doesn't
transfer data into a caller's memory buffer. This is useful because
after skipping forward in the stream in this way, the user can, if
needed, rewind the stream (SetPos()) and access the object's memory
buffer including ranges that were skipped over (without needing to
read from the disk file).
2022-10-24 13:02:37 -06:00
Andrew Chow
6912a28f08
Merge bitcoin/bitcoin#25667: assumeutxo: snapshot initialization
bf95976061 doc: add note about snapshot chainstate init (James O'Beirne)
e4d7995286 test: add testcases for snapshot initialization (James O'Beirne)
cced4e7336 test: move-only-ish: factor out LoadVerifyActivateChainstate() (James O'Beirne)
51fc9241c0 test: allow on-disk coins and block tree dbs in tests (James O'Beirne)
3c361391b8 test: add reset_chainstate parameter for snapshot unittests (James O'Beirne)
00b357c215 validation: add ResetChainstates() (James O'Beirne)
3a29dfbfb2 move-only: test: make snapshot chainstate setup reusable (James O'Beirne)
8153bd9247 blockmanager: avoid undefined behavior during FlushBlockFile (James O'Beirne)
ad67ff377c validation: remove snapshot datadirs upon validation failure (James O'Beirne)
34d1590331 add utilities for deleting on-disk leveldb data (James O'Beirne)
252abd1e8b init: add utxo snapshot detection (James O'Beirne)
f9f1735f13 validation: rename snapshot chainstate dir (James O'Beirne)
d14bebf100 db: add StoragePath to CDBWrapper/CCoinsViewDB (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11) (parent PR: https://github.com/bitcoin/bitcoin/pull/15606)

  ---

  Half of the replacement for #24232. The original PR grew larger than expected throughout the review process.

  This change adds the ability to initialize a snapshot-based chainstate during init if one is detected on disk. This is of course unused as of now (aside from in unittests) given that we haven't yet enabled actually loading snapshots.

  Don't be scared! There are some big move-only commits in here.

  Accompanying changes include:

  - moving the snapshot coinsdb directory from being called `chainstate_[base blockhash]` to `chainstate_snapshot`, since we only support one snapshot in use at a time. This simplifies some logic, but it necessitates writing that base blockhash out to a file within the coinsdb dir. See [discussion here](https://github.com/bitcoin/bitcoin/pull/24232#discussion_r832762880).
  - adding a simple fix in `FlushBlockFile()` that avoids a crash when attemping to flush to disk before `LoadBlockIndexDB()` is called, which happens when calling `MaybeRebalanceCaches()` during multiple chainstate init.
  - improving the unittest to allow testing with on-disk chainstates - necessary to test a simulated restart and re-initialization.

ACKs for top commit:
  naumenkogs:
    utACK bf95976061
  ariard:
    Code Review ACK bf9597606
  ryanofsky:
    Code review ACK bf95976061. Changes since last review: rebasing, switching from CAutoFile to AutoFile, adding comments, switching from BOOST_CHECK to Assert in test util, using chainman.GetMutex() in tests, destroying one ChainstateManager before creating a new one in tests
  fjahr:
    utACK bf95976061
  aureleoules:
    ACK bf95976061

Tree-SHA512: 15ae75caf19f8d12a12d2647c52897904d27b265a7af6b4ae7b858592eeadb8f9da6c2394b6baebec90adc28742c053e3eb506119577dae7c1e722ebb3b7bcc0
2022-10-13 10:19:27 -04:00
MacroFake
fabbbe32ee
Remove unused CDataStream::rdbuf method
It is unused and seems unlikely to be ever used.
2022-10-05 15:29:36 +02:00
fanquake
b95633121b
refactor: use <cstdio> over stdio.h
We currently use both. Consolidate on the former.
2022-09-21 16:53:11 +01:00
James O'Beirne
f9f1735f13 validation: rename snapshot chainstate dir
This changes the snapshot's leveldb chainstate dir name from
`chainstate_[blockhash]` to `chainstate_snapshot`. This simplifies
later logic that loads snapshot data, and enforces the limitation
of a single snapshot at any given time.

Since we still need to persis the blockhash of the base block, we
write that out to a file (`chainstate_snapshot/base_blockhash`) for
later use during initialization, so that we can reinitialize the
snapshot chainstate.

Co-authored-by: Russell Yanofsky <russ@yanofsky.org>
2022-09-13 13:30:12 -04:00
MacroFake
6666803c89
streams: Add AutoFile without ser-type and ser-version
The moved parts can be reviewed with "--color-moved=dimmed-zebra".
The one-char changes can be reviewed with "--word-diff-regex=.".
2022-06-29 10:31:53 +02:00
MarcoFalke
fa1b89a6bd
scripted-diff: Rename nReadPos to m_read_pos in streams.h
-BEGIN VERIFY SCRIPT-
 sed -i 's/nReadPos/m_read_pos/g' ./src/streams.h
-END VERIFY SCRIPT-
2022-02-09 17:21:04 +01:00
MarcoFalke
fa56c79df9
Make CDataStream work properly on 64-bit systems 2022-02-09 17:21:01 +01:00
MarcoFalke
fab02f7991
streams: Fix read-past-the-end and integer overflows 2022-02-09 17:20:22 +01:00
MarcoFalke
fa1b227a72
Remove broken and unused CDataStream methods 2022-02-03 20:16:50 +01:00
MarcoFalke
fa24493d63
Use spans of std::byte in serialize
This switches .read() and .write() to take spans of bytes.
2022-01-02 11:40:31 +01:00
Hennadii Stepanov
f47dda2c58
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
* 2020: fa0074e2d8
* 2019: aaaaad6ac9
2021-12-30 19:36:57 +02:00
Pieter Wuille
31ba1af74a Remove unused (and broken) functionality in SpanReader
This removes the ability to set an offset in the SpanReader constructor,
as the current code is broken. All call sites use pos=0, so it is actually
unused. If future call sites need it, SpanReader{a, b, c, d} is equivalent
to SpanReader{a, b, c.subspan(d)}.

It also removes the ability to deserialize from SpanReader directly from
the constructor. This too is unused, and can be more idiomatically
simulated using (SpanReader{a, b, c} >> x >> y >> z) instead of
SpanReader{a, b, c, x, y, z}.
2021-12-06 16:18:14 -05:00
Pieter Wuille
2c35a93b3c Generalize/simplify VectorReader into SpanReader 2021-12-02 14:47:17 -05:00
MarcoFalke
fabe18d0b3
Use value_type in CDataStream where possible
Also, simplify unit tests with the CDataStream::str method.
2021-11-09 17:41:21 +01:00
MarcoFalke
fa2204f6ad
streams: Accept URef obj for VectorReader unserialize 2021-05-05 20:19:03 +02:00
Wladimir J. van der Laan
2c0fc856a6
Merge #20464: refactor: Treat CDataStream bytes as uint8_t
fa29272459 Remove redundant MakeUCharSpan wrappers (MarcoFalke)
faf4aa2f47 Remove CDataStream::Init in favor of C++11 member initialization (MarcoFalke)
fada14b948 Treat CDataStream bytes as uint8_t (MarcoFalke)
fa8bdb048e refactor: Drop CDataStream constructors in favor of one taking a Span of bytes (MarcoFalke)
faa96f841f Remove unused CDataStream methods (MarcoFalke)

Pull request description:

  Using `uint8_t` for raw bytes has a style benefit:
  * The signedness is clear from reading the code, as it does not depend on the architecture

  Other clean-ups in this pull include:
  * Remove unused methods
  * Constructor is simplified with `Span`
  * Remove `Init()` member in favor of C++11 member initialization

ACKs for top commit:
  laanwj:
    code review ACK fa29272459
  theStack:
    ACK fa29272459 🍾

Tree-SHA512: 931ee28bd99843d7e894b48e90e1187ffb0278677c267044b3c0c255069d9bbd9298ab2e539b1002a30b543d240450eaec718ef4ee95a7fd4be0a295e926343f
2021-02-01 15:17:28 +01:00
MarcoFalke
fa0074e2d8
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2020-12-31 09:45:41 +01:00
MarcoFalke
faf4aa2f47
Remove CDataStream::Init in favor of C++11 member initialization 2020-12-31 09:05:51 +01:00
MarcoFalke
fada14b948
Treat CDataStream bytes as uint8_t
Also, rename CSerializeData to SerializeData
2020-11-23 21:19:50 +01:00
MarcoFalke
fa8bdb048e
refactor: Drop CDataStream constructors in favor of one taking a Span of bytes 2020-11-23 21:19:22 +01:00
MarcoFalke
faa96f841f
Remove unused CDataStream methods 2020-11-23 19:24:54 +01:00
Vasil Dimov
201a4596d9
net: CAddress & CAddrMan: (un)serialize as ADDRv2
Change the serialization of `CAddrMan` to serialize its addresses
in ADDRv2/BIP155 format by default. Introduce a new `CAddrMan` format
version (3).

Add support for ADDRv2 format in `CAddress` (un)serialization.

Co-authored-by: Carl Dong <contact@carldong.me>
2020-10-09 16:42:49 +02:00
Hennadii Stepanov
7b3851e947
refactor: Drop unused CBufferedFile::Seek() 2020-07-26 22:46:28 +03:00
MarcoFalke
aaaaad6ac9
scripted-diff: Bump copyright of files changed in 2019
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2019-12-30 10:42:20 +13:00
practicalswift
084e17cebd Remove unused includes 2019-10-15 22:56:43 +00:00
Larry Ruane
efd2474d17 util: CBufferedFile fixes 2019-09-10 07:53:09 -06:00
Ben Woosley
70e7cee960
Trivial: Doxygenize existing CBufferedFile and VectorReader comments 2019-01-25 12:32:37 -08:00
MarcoFalke
88bbcdc4e9
Merge #14357: streams: Fix broken streams_vector_reader test. Remove unused seek(size_t).
4f4993fe2a Remove UBSan suppression (practicalswift)
958e1a307e streams: Remove unused seek(size_t) (practicalswift)

Pull request description:

  Fix broken `streams_vector_reader` test. Remove unused `seek(size_t)`.

  Before this change the test `streams_vector_reader` triggered an unintended unsigned integer wraparound. It tried so seek using a negative value in `reader.seek(-6)`.

  Changes in this PR:
  * Fix broken `VectorReader::seek(size_t)` test case
  * Remove unused `seek(size_t)`

Tree-SHA512: 6c6affd680626363eef9e496748f2f86a522325abab9d6b13161f41125cdc29ceb36c2c1509c90b8ff108d606df7629e55e094cc2b6253b05a892b81ce176b71
2019-01-05 15:06:43 +01:00
MarcoFalke
5b6b371c77
Merge #14855: test: Correct ineffectual WithOrVersion from transactions_tests
75778a0724 test: Correct ineffectual WithOrVersion from transactions_tests (Ben Woosley)

Pull request description:

  `WithOrVersion` uses `|` to combine the versions, and `|` with 0 is a no-op.

  NicolasDorier / sipa do you recall why the version is being overridden here?

  Introduced in ab48c5e721
  Last updated 81e3228fcb

Tree-SHA512: 2aea925497bab2da973f17752410a6759d67181a57c3b12a685d184fbfcca2984c45b702ab0bd641d75e086696a0424f1bf77c5578ca765d6882dc03b42d5f9a
2019-01-04 12:20:35 +01:00
Ben Woosley
75778a0724
test: Correct ineffectual WithOrVersion from transactions_tests
WithOrVersion uses | to combine the versions, and | with 0 is a no-op.

Instead I run it with PROTOCOL_VERSION and 0 separately, as the original
code only tested PROTOCOL_VERSION but apparently only intended to test
version 0.

Introduced in ab48c5e721
Last updated 81e3228fcb
2019-01-03 09:10:27 -08:00
practicalswift
cf4b0327ed
Use std::numeric_limits<UNSIGNED>::max()) instead of (UNSIGNED)-1 2018-12-04 19:55:04 +02:00
practicalswift
958e1a307e streams: Remove unused seek(size_t) 2018-11-23 16:58:44 +01:00
practicalswift
ada356208e Fix typos reported by codespell 2018-09-04 13:11:26 +02:00
Jim Posen
fe943f99bf streams: Implement BitStreamReader/Writer classes.
Golomb-Rice coding, as specified in BIP 158, involves operations on
individual bits. These classes will be used to implement the
encoding/decoding operations.
2018-08-25 10:02:37 -07:00
Jim Posen
947133dec9 streams: Create VectorReader stream interface for vectors.
This is a read analogue for the existing CVectorWriter.
2018-08-25 10:02:37 -07:00