Merge #16802: scripts: In linearize, search for next position of magic bytes rather than fail

3284e6c09a scripts: search for next position of magic bytes rather than fail (Tim Akinbo)

Pull request description:

  When using the `linearize-data.py` contrib script to export block data, there are edge cases where the script fails with an `Invalid magic: 00000000` error. This error occurs due to the presence of padding bytes that occasionally appears between consecutive blocks in the block data file.

  There's an ongoing conversation about this in #14986. sipa also admitted that it is a bug in #5028. Fortunately, this is not an issue in bitcoin core as it handles this type of situation gracefully and so no fix in bitcoin core is required.

  This PR is an improvement on how the script handles these "invalid magic bytes". Rather than failing, this patch allows the script to search for the next occurrence of the magic bytes and then starts reading the block from there.

ACKs for top commit:
  laanwj:
    ACK 3284e6c09a

Tree-SHA512: 18067ae0b4b62e822dfc558a86439ad6acaf939b98479e38e8e4248536574643b26eb48e96ec7139375c88b42cbe7705a64deb13a3c239e16025a6aad3d69bfa
This commit is contained in:
Wladimir J. van der Laan 2019-10-08 10:41:58 +02:00
commit df50fd194f
No known key found for this signature in database
GPG key ID: 1E4AED62986CD25D

View file

@ -213,8 +213,11 @@ class BlockDataCopier:
inMagic = inhdr[:4]
if (inMagic != self.settings['netmagic']):
print("Invalid magic: " + inMagic.hex())
return
# Seek backwards 7 bytes (skipping the first byte in the previous search)
# and continue searching from the new position if the magic bytes are not
# found.
self.inF.seek(-7, os.SEEK_CUR)
continue
inLenLE = inhdr[4:]
su = struct.unpack("<I", inLenLE)
inLen = su[0] - 80 # length without header