Commit graph

25672 commits

Author SHA1 Message Date
MarcoFalke
2a784723f0
Merge #19597: test: test decodepsbt fee calculation (count input value only once per UTXO)
82dee87933 test: test decodepsbt fee calculation (count input value only once per UTXO) (Sebastian Falbesoner)

Pull request description:

  Fixes #19523, adding a simple test to `rpc_psbt.py` that checks that the decodepsbt fee matches the one given by the wallet (`walletcreatefundedpsbt`). This is in particular important for PSBTs with segwit inputs that have both a witness- and a non-witness-UTXO type set.

  Example test run after reverting commit 75122780e2 ("Increment input value sum only once per UTXO in decodepsbt"):

  ```
  $ test/functional/rpc_psbt.py
  2020-07-26T11:31:44.862000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test__sutcd4y
  20.00007580
  2020-07-26T11:31:47.073000Z TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
    File "/home/honeybadger/buidl/bitcoin_thestack/test/functional/test_framework/test_framework.py", line 118, in main
      self.run_test()
    File "test/functional/rpc_psbt.py", line 166, in run_test
      assert_equal(decoded['fee'], created_psbt['fee'])
    File "/home/honeybadger/buidl/bitcoin_thestack/test/functional/test_framework/util.py", line 49, in assert_equal
      raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
  AssertionError: not(20.00007580 == 0.00007580)
  2020-07-26T11:31:47.125000Z TestFramework (INFO): Stopping nodes
  ......
  ```

ACKs for top commit:
  achow101:
    ACK 82dee87933

Tree-SHA512: 296b8a701f851d482ef6200c6cbf0cf0257a79a828ac6dbc39b05d8c2d839c6fdb9d3f5a084015295cfa3eac7c11faa2f2d52e619c11627b04c75150eead8330
2020-07-30 09:46:16 +02:00
MarcoFalke
149eca433d
Merge #19599: test: clean message_count and last_message
2c6a02e024 Clean message_count and last_message (Troy Giorshev)

Pull request description:

  From #19580

  This PR changes comments to clarify the intended usage of `message_count` and `last_message`.  Additionally it changes the only usage of `message_count` to use `last_message` instead, bringing the code into alignment with the intended usage.

  Note: Now `message_count` is completely unused.  However, it is ready to be used (i.e. the supporting code works) and likely will be used in some test in the future.

ACKs for top commit:
  jnewbery:
    utACK 2c6a02e024

Tree-SHA512: 07c7684c9586de4f845e10d7aac36c1aab9fb56b409949c1c70d5ca705bc3971ca7d5943245a0472def4efd7b4e1c5dad2f713db5ead8fca08404daf4891e98b
2020-07-30 09:15:49 +02:00
Andrew Chow
0fcff547d5 walletdb: Ensure that having no database handle is a failure
Previously having no database handle could still be considered a success
when BerkeleyDatabase and BerkeleyBatch were used for dummy database
things. With dedicated DummyDatabase and DummyBatch classes now, these
should fail.
2020-07-29 12:30:24 -04:00
Andrew Chow
da039d2a91 Remove BDB dummy databases 2020-07-29 12:30:23 -04:00
Andrew Chow
0103d6434e Introduce DummyDatabase and use it in the tests 2020-07-29 12:28:30 -04:00
Wladimir J. van der Laan
8db23349fe
Merge #19335: wallet: Cleanup and separate BerkeleyDatabase and BerkeleyBatch
74507ce71e walletdb: Remove BerkeleyBatch friend class from BerkeleyDatabase (Andrew Chow)
00f0041351 No need to check for duplicate fileids in all dbenvs (Andrew Chow)
d86efab370 walletdb: Move Db->open to BerkeleyDatabase::Open (Andrew Chow)
4fe4b3bf1b walletdb: track database file use as m_refcount within BerkeleyDatabase (Andrew Chow)
65fb8807ac Combine BerkeleyEnvironment::Verify into BerkeleyDatabase::Verify (Andrew Chow)

Pull request description:

  `BerkeleyBatch` and `BerkeleyDatabase` are kind of messy. The goal of this is to clean up them up so that they are logically separated.

  `BerkeleyBatch` currently handles the creation of the `BerkeleyDatabase`'s `Db` handle. This is instead moved into `BerkeleyDatabase` and is called by `BerkeleyBatch`.

  Instead of having `BerkeleyEnvironment` track each database's usage, have `BerkeleyDatabase` track this usage itself with the `m_refcount` variable that is present in `WalletDatabase`.

  Lastly, instead of having each `BerkeleyEnvironment` store the fileids of the databases open in it, have a global `g_fileids` to track those fileids. We were already checking fileid uniqueness globally (by checking the fileids in every environment when opening a database) so it's cleaner to do this with a global variable.

  All of these changes allow us to make `BerkeleyBatch` and `BerkeleyDatabase` no longer be friend classes.

  The diff of this PR is currently the same as in ##18971

  Requires #19334

ACKs for top commit:
  laanwj:
    Code review ACK 74507ce71e
  ryanofsky:
    Code review ACK 74507ce71e. No changes since last review other than rebase

Tree-SHA512: 845d84ee1a470e2bf5d2e2e3d7738183d8ce43ddd06a0bbd57edecf5779b2f55d70728b1b57f5daab0f078650a8d60c3e19dc30b75b36e7aa952ce268399d5f6
2020-07-29 18:24:16 +02:00
Wladimir J. van der Laan
400f45ec9b
Merge #19525: build: add -Wl,-z,separate-code to hardening flags
65d0f1a533 devtools: Add security check for separate_code (Wladimir J. van der Laan)
2e9e6377f1 build: add -Wl,-z,separate-code to hardening flags (fanquake)

Pull request description:

  TLDR: We are generally explicit about the hardening related flags we use,
  rather than letting the distro / toolchain decide via their defaults. This PR
  adds `-z,separate-code` which has been enabled by default for Linux targets
   since binutils 2.31. Ubuntu Bionic (currently used for gitian) ships with
  binutils 2.30, so this will enable the option for those builds.

  This flag was added to binutils/ld in the 2.30 release,
  see commit c11c786f0b45617bb8807ab6a57220d5ff50e414:

  > The new "-z separate-code" option will generate separate code LOAD
  segment which must be in wholly disjoint pages from any other data.

  It was made the default for Linux/x86 targets in the 2.31 release, see commit
  f6aec96dce1ddbd8961a3aa8a2925db2021719bb:

  > This patch adds --enable-separate-code to ld configure to turn on
  -z separate-code by default and enables it by default for Linux/x86.
  This avoids mixing code pages with data to improve cache performance
  as well as security.

  > To reduce x86-64 executable and shared object sizes, the maximum page
  size is reduced from 2MB to 4KB when -z separate-code is turned on by
  default.  Note: -z max-page-size= can be used to set the maximum page
  size.

  > We compared SPEC CPU 2017 performance before and after this change on
  Skylake server.  There are no any significant performance changes.
  Everything is mostly below +/-1%.

  Support was also added to LLVMs lld: https://reviews.llvm.org/D64903, however
  there it remains off by default.

  There were concerns about an increase in binary size, however in our case, the
  difference would seem negligible, given we are shipping a
  multi-megabyte binary, which then downloads 100's of GBs of data.

  Also note that most recent versions of distros are shipping a new enough version
  of binutils that this is available and/or already on by default (assuming the distro
  has not turned it off, I haven't checked everywhere):

  CentOS 8: 2.30
  Debian Buster 2.31.1
  Fedora 29: 2.31.1
  FreeBSD: 2.33
  GNU Guix: 2.33 / 2.34
  Ubuntu 18.04: 2.30

  Related threads / discussion:
  https://bugzilla.redhat.com/show_bug.cgi?id=1623218

  The ELF header when building on Debian Buster (where it's already enabled by default in binutils):
  ```bash
  Program Header:
      PHDR off    0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 2**3
           filesz 0x00000000000002a0 memsz 0x00000000000002a0 flags r--
    INTERP off    0x00000000000002e0 vaddr 0x00000000000002e0 paddr 0x00000000000002e0 align 2**0
           filesz 0x000000000000001c memsz 0x000000000000001c flags r--
      LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12
           filesz 0x0000000000038f10 memsz 0x0000000000038f10 flags r--
      LOAD off    0x0000000000039000 vaddr 0x0000000000039000 paddr 0x0000000000039000 align 2**12
           filesz 0x00000000006b9389 memsz 0x00000000006b9389 flags r-x
      LOAD off    0x00000000006f3000 vaddr 0x00000000006f3000 paddr 0x00000000006f3000 align 2**12
           filesz 0x0000000000204847 memsz 0x0000000000204847 flags r--
      LOAD off    0x00000000008f7920 vaddr 0x00000000008f8920 paddr 0x00000000008f8920 align 2**12
           filesz 0x00000000000183e0 memsz 0x0000000000022fd0 flags rw-
   DYNAMIC off    0x000000000090adb0 vaddr 0x000000000090bdb0 paddr 0x000000000090bdb0 align 2**3
           filesz 0x0000000000000240 memsz 0x0000000000000240 flags rw-
  ```
   vs when opting out using `-Wl,-z,noseparate-code`:
  ```bash
  Program Header:
      PHDR off    0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 2**3
           filesz 0x0000000000000230 memsz 0x0000000000000230 flags r--
    INTERP off    0x0000000000000270 vaddr 0x0000000000000270 paddr 0x0000000000000270 align 2**0
           filesz 0x000000000000001c memsz 0x000000000000001c flags r--
      LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12
           filesz 0x00000000008f6a87 memsz 0x00000000008f6a87 flags r-x
      LOAD off    0x00000000008f7920 vaddr 0x00000000008f8920 paddr 0x00000000008f8920 align 2**12
           filesz 0x00000000000183e0 memsz 0x0000000000022fd0 flags rw-
   DYNAMIC off    0x000000000090adb0 vaddr 0x000000000090bdb0 paddr 0x000000000090bdb0 align 2**3
           filesz 0x0000000000000240 memsz 0x0000000000000240 flags rw-
  ```

ACKs for top commit:
  laanwj:
    ACK 65d0f1a533

Tree-SHA512: 6e40e434efea8a8e39f6cb244dfd16aaa5a9db5a2ea762a05d1727357b20e33b7e47c1a652ee88490c9d7952a4caa2f992396fb30346239300d37ae123e36d49
2020-07-29 16:52:51 +02:00
Wladimir J. van der Laan
a76ccb01b9
Merge #19534: net: save the network type explicitly in CNetAddr
bcfebb6d55 net: save the network type explicitly in CNetAddr (Vasil Dimov)
100c64a95b net: document `enum Network` (Vasil Dimov)

Pull request description:

  (chopped off from https://github.com/bitcoin/bitcoin/pull/19031 to ease review)

  Before this change, we would analyze the contents of `CNetAddr::ip[16]`
  in order to tell which type is an address. Change this by introducing a
  new member `CNetAddr::m_net` that explicitly tells the type of the
  address.

  This is necessary because in BIP155 we will not be able to tell the
  address type by just looking at its raw representation (e.g. both TORv3
  and I2P are "seemingly random" 32 bytes).

  As a side effect of this change we no longer need to store IPv4
  addresses encoded as IPv6 addresses - we can store them in proper 4
  bytes (will be done in a separate commit). Also the code gets
  somewhat simplified - instead of
  `memcmp(ip, pchIPv4, sizeof(pchIPv4)) == 0` we can use
  `m_net == NET_IPV4`.

ACKs for top commit:
  troygiorshev:
    reACK bcfebb6d55 via `git range-diff master 64897c5 bcfebb6`
  jonatack:
    re-ACK bcfebb6 per `git diff 662bb25 bcfebb6`, code review, debug build/tests clean, ran bitcoind.
  laanwj:
    Code review ACK bcfebb6d55

Tree-SHA512: 9347e2a50feac617a994bfb46a8f77e31c236bde882e4fd4f03eea4766cd5110216f5f3d24dee91d25218bab7f8bb6e1d2d6212a44db9e34594299fd6ff7606b
2020-07-29 13:31:16 +02:00
MarcoFalke
fae8c28dae
Pass mempool pointer to GetCoinsCacheSizeState 2020-07-29 12:30:11 +02:00
MarcoFalke
fac674db20
Pass mempool pointer to UnloadBlockIndex 2020-07-29 12:29:51 +02:00
Ivan Metlushko
8ed9002cd1 refactor: use local argsmanager in CRegTestParams 2020-07-29 16:39:03 +07:00
Ivan Metlushko
9b20f66828 scripted-diff: Replace gArgs with local argsman
-BEGIN VERIFY SCRIPT-
sed -i -e 's/gArgs.Add/argsman.Add/g' `git grep -l "gArgs.Add"`
-END VERIFY SCRIPT-
2020-07-29 16:39:00 +07:00
Ivan Metlushko
a316e9ce26 refactor: add unused ArgsManager to replace gArgs 2020-07-29 16:36:44 +07:00
MarcoFalke
faec851b6e
test: Simplify cs_main locks 2020-07-29 08:00:54 +02:00
MarcoFalke
2f71a1ea35
Merge #18637: coins: allow cache resize after init
f19fdd47a6 test: add test for CChainState::ResizeCoinsCaches() (James O'Beirne)
8ac3ef4699 add ChainstateManager::MaybeRebalanceCaches() (James O'Beirne)
f36aaa6392 Add CChainState::ResizeCoinsCaches (James O'Beirne)
b223111da2 txdb: add CCoinsViewDB::ChangeCacheSize (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: #15606
  Issue: #15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal

  ---

  In the assumeutxo implementation draft (#15056), once a UTXO snapshot is loaded, a new chainstate object is created after initialization. This means that we have to reclaim some of the cache that we've allocated to the original chainstate (per `dbcache=`) to repurpose for the snapshot chainstate.

  Furthermore, it makes sense to have different cache allocations depending on which chainstate is more active. While the snapshot chainstate is working to get to the network tip (and the background validation chainstate is idle), it makes sense that the snapshot chainstate should have the majority of cache allocation. And contrariwise once the snapshot has reached network tip, most of the cache should be given to the background validation chainstate.

  This set of changes (detailed in the commit messages) allows us to dynamically resize the various coins caches. None of the functionality introduced here is used at the moment, but will be in the next AU PR (which introduces `ActivateSnapshot`).

  `ChainstateManager::MaybeRebalanceCaches()` defines the (somewhat normative) cache allocations between the snapshot and background validation chainstates. I'd be interested in feedback if anyone has thoughts on the proportions I've set there.

ACKs for top commit:
  ajtowns:
    weak utACK f19fdd47a6 -- didn't find any major problems, but not super confident that I didn't miss anything
  fjahr:
    Code review ACK f19fdd4
  ryanofsky:
    Code review ACK f19fdd47a6. Only change since last review is constructor cleanup (no change in behavior). I think the suggestions here from ajtowns and others are good, but shouldn't delay merging the PR (and hold up assumeutxo)

Tree-SHA512: fffb7847fb6993dd4a1a41cf11179b211b0b20b7eb5f7cf6266442136bfe9d43b830bbefcafd475bfd4af273f5573500594aa41fff03e0ed5c2a1e8562ff9269
2020-07-29 07:53:19 +02:00
fanquake
e60ef21b8a
doc: Clang 8 or later is required with FORCE_USE_SYSTEM_CLANG
The usage of pragmas within the macOS SDK requires LLVM Clang 8. This is
the version as our prebuilt Clang, however the minimum is worth noting here
as they may diverge and/or expert users might expect they could use an
earlier version.

If you compile using Clang 7 you'll see output like:
```bash
In file included from kernel/qcore_mac_objc.mm:44:
In file included from /bitcoin/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/System/Library/Frameworks/AppKit.framework/Headers/NSText.h:9:
In file included from /bitcoin/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/System/Library/Frameworks/AppKit.framework/Headers/NSView.h:19:
In file included from /bitcoin/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/System/Library/Frameworks/AppKit.framework/Headers/NSResponder.h:10:
/bitcoin/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/System/Library/Frameworks/AppKit.framework/Headers/NSEvent.h:19:1: error:
      expected 'push' or 'pop' after '#pragma clang attribute'
/bitcoin/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/usr/include/os/availability.h:104:273: note: expanded from macro
      'API_UNAVAILABLE_BEGIN'
  ...__API_UNAVAILABLE_BEGIN5, __API_UNAVAILABLE_BEGIN4, __API_UNAVAILABLE_BEGIN3, __API_UNAVAILABLE_BEGIN2, __API_UNAVAILABLE_BEGIN1, 0)(__VA_A...
                                                                                                             ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
```
2020-07-29 10:49:54 +08:00
MarcoFalke
a41ae68053
Merge #19613: travis: Re-enable s390x
fae656144e travis: Re-enable s390x (MarcoFalke)

Pull request description:

  According to travis, the issue has been solved. Quote

  > I would like to confirm that we have resolved this issue and most of our users are reported that this issue has been resolved on their end as well. Could you please re-check and see if that still exists for you?

ACKs for top commit:
  theStack:
    ACK fae656144e

Tree-SHA512: cf42f96d25474a9dcf0817a049e30e29714731d708f73c40a3042b0c70a71ff08f07dd96a89f0dcd5a50a63a355cf30b3511172a32b8af7d5a2e13ad222a4b49
2020-07-28 17:51:58 +02:00
fanquake
1d8338d6b7
util: use HAVE_FDATASYNC to determine fdatasync() use
Rather than just using it on Linux and NetBSD, use `fdatasync()` based
on whether it's available. i.e it is available in newer versions
of FreeBSD (11.1 and later).

This also aligns our code more closely with what is being done in leveldb.

Was pointed out by Luke in #19430.
2020-07-28 22:39:56 +08:00
MarcoFalke
fae656144e
travis: Re-enable s390x 2020-07-28 16:01:53 +02:00
fanquake
01cd24c226
doc: set CC_FOR_BUILD when building on OpenBSD
Closes: #19559

While #19559 has been fixed upstream, it makes sense to not only
recommend using `CC_FOR_BUILD`here  until the fix is pulled in as
part of our next libsecp update, but after discussing with Cory,
he suggested we should be setting this on OpenBSD (which still has
the an ancient GCC) regardless.
2020-07-28 16:29:47 +08:00
MarcoFalke
24ead1a923
Merge #19519: ci: Increase CCACHE_SIZE in some builds on Travis
fdf697fe75 ci: Increase CCACHE_SIZE in some builds on Travis (Hennadii Stepanov)

Pull request description:

  This PR will decrease build time for the longest Travis builds.
  Up to ~20 min improvement with warmed up cache is expected.

Top commit has no ACKs.

Tree-SHA512: ee3a22162d03537be11e60b53de043247f6a65e35e630e0807b758a846b7e05ef2059d18846644aafa3cd5dc7d3e3f56a8ccb94cc71b95227debc9201d3142a2
2020-07-28 10:25:11 +02:00
fanquake
e0ed4cc50d
Merge #19530: depends: build LTO support into Apple's ld64
5962522fbc depends: bump native_cctools for fixed lto with external clang (Cory Fields)
00d1ba7aaa depends: enable lto support for Apple's ld64 (Cory Fields)

Pull request description:

  This didn't work for a few reasons (various toolchain compatibility issues) the last time I tested it, but after the last round of bumps it works with no apparent issues.

  Note that this does not _enable_ LTO by default in any way, only hooks up the machinery for ```-flto``` to work correctly when specified.

  Lines were split for an easier rebase after #17919 is merged.

ACKs for top commit:
  fanquake:
    ACK 5962522fbc. The relevant option upstream is [here](https://github.com/tpoechtrager/cctools-port/blob/master/cctools/m4/llvm.m4#L4).

Tree-SHA512: df2775e74e7bc847e6cef94cb8457d503d6c9e2fdea861e51386fa6ed5a7ba688241db3685561ae1a32f66724c1b3801727252025f00c04b90a3bdc8a4f6f93b
2020-07-28 16:01:26 +08:00
fanquake
cea39fc268
Merge #19595: Fix .gitignore for src/test/fuzz directory
623f66554d Fix .gitignore for src/test/fuzz directory (Hennadii Stepanov)

Pull request description:

  On master (31d2b4098a):
  ```
  $ git ls-files --ignored --exclude-standard
  contrib/init/org.bitcoin.bitcoind.plist
  contrib/macdeploy/fancy.plist
  src/qt/Makefile
  src/qt/test/Makefile
  src/test/Makefile
  src/test/fuzz/FuzzedDataProvider.h
  src/test/fuzz/addition_overflow.cpp
  src/test/fuzz/addrdb.cpp
  src/test/fuzz/asmap.cpp
  src/test/fuzz/asmap_direct.cpp
  src/test/fuzz/autofile.cpp
  src/test/fuzz/banman.cpp
  src/test/fuzz/base_encode_decode.cpp
  src/test/fuzz/bech32.cpp
  src/test/fuzz/block.cpp
  src/test/fuzz/block_header.cpp
  src/test/fuzz/blockfilter.cpp
  src/test/fuzz/bloom_filter.cpp
  src/test/fuzz/buffered_file.cpp
  src/test/fuzz/chain.cpp
  src/test/fuzz/checkqueue.cpp
  src/test/fuzz/coins_view.cpp
  src/test/fuzz/crypto.cpp
  src/test/fuzz/crypto_aes256.cpp
  src/test/fuzz/crypto_aes256cbc.cpp
  src/test/fuzz/crypto_chacha20.cpp
  src/test/fuzz/crypto_chacha20_poly1305_aead.cpp
  src/test/fuzz/crypto_common.cpp
  src/test/fuzz/crypto_hkdf_hmac_sha256_l32.cpp
  src/test/fuzz/crypto_poly1305.cpp
  src/test/fuzz/cuckoocache.cpp
  src/test/fuzz/decode_tx.cpp
  src/test/fuzz/descriptor_parse.cpp
  src/test/fuzz/deserialize.cpp
  src/test/fuzz/eval_script.cpp
  src/test/fuzz/fee_rate.cpp
  src/test/fuzz/fees.cpp
  src/test/fuzz/flatfile.cpp
  src/test/fuzz/float.cpp
  src/test/fuzz/fuzz.cpp
  src/test/fuzz/fuzz.h
  src/test/fuzz/golomb_rice.cpp
  src/test/fuzz/hex.cpp
  src/test/fuzz/http_request.cpp
  src/test/fuzz/integer.cpp
  src/test/fuzz/key.cpp
  src/test/fuzz/key_io.cpp
  src/test/fuzz/kitchen_sink.cpp
  src/test/fuzz/load_external_block_file.cpp
  src/test/fuzz/locale.cpp
  src/test/fuzz/merkleblock.cpp
  src/test/fuzz/message.cpp
  src/test/fuzz/multiplication_overflow.cpp
  src/test/fuzz/net_permissions.cpp
  src/test/fuzz/netaddress.cpp
  src/test/fuzz/p2p_transport_deserializer.cpp
  src/test/fuzz/parse_hd_keypath.cpp
  src/test/fuzz/parse_iso8601.cpp
  src/test/fuzz/parse_numbers.cpp
  src/test/fuzz/parse_script.cpp
  src/test/fuzz/parse_univalue.cpp
  src/test/fuzz/policy_estimator.cpp
  src/test/fuzz/policy_estimator_io.cpp
  src/test/fuzz/pow.cpp
  src/test/fuzz/prevector.cpp
  src/test/fuzz/primitives_transaction.cpp
  src/test/fuzz/process_message.cpp
  src/test/fuzz/process_messages.cpp
  src/test/fuzz/protocol.cpp
  src/test/fuzz/psbt.cpp
  src/test/fuzz/random.cpp
  src/test/fuzz/rbf.cpp
  src/test/fuzz/rolling_bloom_filter.cpp
  src/test/fuzz/script.cpp
  src/test/fuzz/script_bitcoin_consensus.cpp
  src/test/fuzz/script_descriptor_cache.cpp
  src/test/fuzz/script_flags.cpp
  src/test/fuzz/script_interpreter.cpp
  src/test/fuzz/script_ops.cpp
  src/test/fuzz/script_sigcache.cpp
  src/test/fuzz/script_sign.cpp
  src/test/fuzz/scriptnum_ops.cpp
  src/test/fuzz/signature_checker.cpp
  src/test/fuzz/span.cpp
  src/test/fuzz/spanparsing.cpp
  src/test/fuzz/string.cpp
  src/test/fuzz/strprintf.cpp
  src/test/fuzz/system.cpp
  src/test/fuzz/timedata.cpp
  src/test/fuzz/transaction.cpp
  src/test/fuzz/tx_in.cpp
  src/test/fuzz/tx_out.cpp
  src/test/fuzz/util.h
  src/univalue/gen/gen.cpp
  test/functional/data/wallets/high_minversion/db.log
  test/functional/data/wallets/high_minversion/wallet.dat
  ```

  With this PR:
  ```
  $ git ls-files --ignored --exclude-standard
  contrib/init/org.bitcoin.bitcoind.plist
  contrib/macdeploy/fancy.plist
  src/qt/Makefile
  src/qt/test/Makefile
  src/test/Makefile
  src/univalue/gen/gen.cpp
  test/functional/data/wallets/high_minversion/db.log
  test/functional/data/wallets/high_minversion/wallet.dat
  ```

ACKs for top commit:
  MarcoFalke:
    review ACK 623f66554d seems like an improvement when writing new fuzz tests
  practicalswift:
    ACK 623f66554d -- thanks for fixing! ❤️
  theStack:
    tested ACK 623f66554d

Tree-SHA512: 16b3854bf4fd8c3096d915a4efc5cbc63d28b18854b051bafee374508dfbb5871ae7dc6f303dbf57469473082d2c3a7df0a8170da22d60d13878544679363b5c
2020-07-28 15:19:00 +08:00
fanquake
2979a7aff0
Merge #19583: p2p: clean up Misbehaving()
a8865f8b72 [net processing] Tidy up Misbehaving() (John Newbery)
d15b3afb4c [net processing] Always supply debug message to Misbehaving() (John Newbery)
634144a1c2 [net processing] Fixup MaybeDiscourageAndDisconnect() style (John Newbery)

Pull request description:

  This PR makes a few minor clean-ups to `Misbehaving()` in preparation to move it out of the cs_main lock.

  There are very minor logging changes but otherwise no functional changes.

ACKs for top commit:
  troygiorshev:
    tACK a8865f8b72
  jonatack:
    ACK a8865f8
  fjahr:
    Code review ACK a8865f8b72
  promag:
    Code review ACK a8865f8b72.

Tree-SHA512: 98fb4f5f76399715545a1ea19290dcebfc8cb4eff72a1d3555dd3de6e184040bb8668c9651dab21db0dfd8e674e53a5977105ef76547146c9f6fa6b4b9d2ba59
2020-07-28 15:15:35 +08:00
fanquake
a1da180b1b
Merge #19589: rpc: Avoid useless mempool query in gettxoutproof
fa5979d12f rpc: Avoid useless mempool query in gettxoutproof (MarcoFalke)
fa1f7f28cb rpc: Style fixups in gettxoutproof (MarcoFalke)

Pull request description:

  `GetTransaction` implicitly and unconditionally asks the mempool global for a transaction. This is problematic for several reasons:

  * `gettxoutproof` is for on-chain txs only and asking the mempool for on-chain txs is confusing and minimally wasteful
  * Globals are confusing and make code harder to test with unit tests

  Fix both issues by passing in an optional mempool. This also helps with #19556

ACKs for top commit:
  hebasto:
    re-ACK fa5979d12f
  jnewbery:
    utACK fa5979d12f
  promag:
    Code review ACK fa5979d12f.

Tree-SHA512: 048361b82abfcc40481181bd44f70cfc9e97d5d6356549df34bbe30b9de7a0a72d2207a3ad0279b21f06293509b284d8967f58ca7e716263a22b20aa4e7f9c54
2020-07-28 14:54:00 +08:00
Wladimir J. van der Laan
65d0f1a533
devtools: Add security check for separate_code
Check that sections are appropriately separated in virtual memory,
based on their (expected) permissions. This checks for missing
-Wl,-z,separate-code and potentially other problems.

Co-authored-by: fanquake <fanquake@gmail.com>
2020-07-28 12:57:35 +08:00
fanquake
2e9e6377f1
build: add -Wl,-z,separate-code to hardening flags
This flag was added to binutils/ld in the 2.30 release, 
see commit c11c786f0b45617bb8807ab6a57220d5ff50e414:

> The new "-z separate-code" option will generate separate code LOAD
segment which must be in wholly disjoint pages from any other data.


It was made the default for Linux/x86 targets in the 2.31 release, see commit
f6aec96dce1ddbd8961a3aa8a2925db2021719bb:

> This patch adds --enable-separate-code to ld configure to turn on
-z separate-code by default and enables it by default for Linux/x86.
This avoids mixing code pages with data to improve cache performance
as well as security.

> To reduce x86-64 executable and shared object sizes, the maximum page
size is reduced from 2MB to 4KB when -z separate-code is turned on by
default.  Note: -z max-page-size= can be used to set the maximum page
size.

> We compared SPEC CPU 2017 performance before and after this change on
Skylake server.  There are no any significant performance changes.
Everything is mostly below +/-1%.

Support was also added to LLVMs lld: https://reviews.llvm.org/D64903, however
there is remains off by default.

There were concerns about an increase in binary size, however in our case, the
increase (1 page worth of bytes) would seem negligible, given we are shipping a
multi-megabyte binary, which then downloads 100's of GBs of data.

Also note that most recent versions of distros are shipping a new enough version
of binutils that this is available and/or on by default (assuming the distro has
not turned it off, I haven't checked everywhere):

CentOS 8: 2.30
Debian Buster 2.31.1
Fedora 29: 2.31.1
FreeBSD: 2.33
GNU Guix: 2.33 / 2.34
Ubuntu 18.04: 2.30

Related threads / discussion:
https://bugzilla.redhat.com/show_bug.cgi?id=1623218
2020-07-28 12:57:35 +08:00
Vasil Dimov
bcfebb6d55
net: save the network type explicitly in CNetAddr
Before this change, we would analyze the contents of `CNetAddr::ip[16]`
in order to tell which type is an address. Change this by introducing a
new member `CNetAddr::m_net` that explicitly tells the type of the
address.

This is necessary because in BIP155 we will not be able to tell the
address type by just looking at its raw representation (e.g. both TORv3
and I2P are "seemingly random" 32 bytes).

As a side effect of this change we no longer need to store IPv4
addresses encoded as IPv6 addresses - we can store them in proper 4
bytes (will be done in a separate commit). Also the code gets
somewhat simplified - instead of
`memcmp(ip, pchIPv4, sizeof(pchIPv4)) == 0` we can use
`m_net == NET_IPV4`.

Co-authored-by: Carl Dong <contact@carldong.me>
2020-07-27 15:13:24 +02:00
Wladimir J. van der Laan
b62fbf9e1c
Merge #19593: refactor: Drop unused CBufferedFile::Seek()
7b3851e947 refactor: Drop unused CBufferedFile::Seek() (Hennadii Stepanov)

Pull request description:

ACKs for top commit:
  practicalswift:
    ACK 7b3851e947 -- deleted code is better than unused untested code:)
  MarcoFalke:
    ACK 7b3851e947, assuming that removing this should either be correct or result in a compile failure
  jonasschnelli:
    utACK 7b3851e947
  promag:
    Code review ACK 7b3851e947.

Tree-SHA512: 7bfd172aa4bbe349855c1303fd9cd58093d66833fefe46bd29081bfcca4ab434b84c6b84e76e94d06b8749a5abe1dc1e184f5189136cd1403d0e5bc25ad6d456
2020-07-27 13:57:34 +02:00
Troy Giorshev
2c6a02e024 Clean message_count and last_message
This commit clarifies the intended usage of message_count and
last_message.  Additionally it changes the only usage of message_count
to using last_message instead, bringing the code further along the
intended usage.
2020-07-27 07:55:49 -04:00
Jon Atack
c251d710a4
p2p, refactoring: use CInv helpers in net_processing.cpp
to simplify the code and reach less from it into the CInv class internals
2020-07-27 11:06:48 +02:00
Jon Atack
4254cd9f8f
p2p: add CInv transaction message helper methods 2020-07-27 11:06:39 +02:00
Andrew Chow
0e279fe489 walletdb: Remove unused static functions from walletdb.h
VerifyEnvironment and VerifyDatabaseFile were removed, but their
declarations weren't. Remove those.
2020-07-26 20:22:49 -04:00
Andrew Chow
9f536d4fe9 wallettool: Have RecoverDatabaseFile return errors and warnings
Instead of logging or printing these errors and warnings, return them to
the caller.
2020-07-26 20:22:45 -04:00
Hennadii Stepanov
623f66554d
Fix .gitignore for src/test/fuzz directory 2020-07-27 00:56:37 +03:00
Hennadii Stepanov
7b3851e947
refactor: Drop unused CBufferedFile::Seek() 2020-07-26 22:46:28 +03:00
MarcoFalke
31d2b4098a
Merge #19508: Work around memory-aliasing in descriptor ParsePubkey
fa2ae0ac8d span: Add Span::empty() and use it in script/descriptor (MarcoFalke)
fa8a992589 Work around memory-aliasing in descriptor ParsePubkey (MarcoFalke)

Pull request description:

  While this is not undefined behaviour, the memory aliasing trick is confusing when reading the code. Having `a.size()==0` and then access `a[0]` works in this particular case, but should probably be avoided to harden the code for the future.

ACKs for top commit:
  theStack:
    re-ACK fa2ae0ac8d
  elichai:
    ACK fa2ae0ac8d
  jonatack:
    ACK fa2ae0ac8d

Tree-SHA512: 0ec7b09eef45504973a195923cdf1aa8522117c8e2f69b453e5ce9aa8a7e327c71138518022c32d05133dc99cb861101ed0f60fa891814ee3e9dab3a6fa61a84
2020-07-26 19:01:51 +02:00
MarcoFalke
fa5979d12f
rpc: Avoid useless mempool query in gettxoutproof 2020-07-26 16:44:07 +02:00
MarcoFalke
fa1f7f28cb
rpc: Style fixups in gettxoutproof 2020-07-26 15:43:35 +02:00
Sebastian Falbesoner
82dee87933 test: test decodepsbt fee calculation (count input value only once per UTXO)
Checks that the RPC decodepsbt calculates the fee correctly, in particular for
PSBTs with segwit inputs that have both a witness- and a non-witness-UTXO type
set. Before commit 75122780e2 ("Increment input
value sum only once per UTXO in decodepsbt") the values for those inputs were
double counted.
2020-07-26 13:25:16 +02:00
Hennadii Stepanov
b6dcc6d741
gui: Clarify block height label 2020-07-25 20:41:35 +03:00
John Newbery
a8865f8b72 [net processing] Tidy up Misbehaving()
- Make const things const.
- Replace conditional return with assert.
- Don't log the peer's IP address.
- Log the name Misbehaving directly instead of relying on __func__.
2020-07-25 15:52:23 +01:00
John Newbery
d15b3afb4c [net processing] Always supply debug message to Misbehaving()
Misbehaving() could optionally take a debug string for printing to the
log file. Make this mandatory and always provide the string.

A couple of additional minor changes:

- remove the unnecessary forward declaration of Misbehaving()
- don't include the nodeid or newline in the passed debug message.
Misbehaving() adds these itself.
2020-07-25 15:50:34 +01:00
John Newbery
634144a1c2 [net processing] Fixup MaybeDiscourageAndDisconnect() style
Based on review comments from Marco Falke and Jon Atack.
2020-07-25 15:49:24 +01:00
Fabian Jahr
e7448d6680
wallet: Don't override signing errors 2020-07-25 00:00:36 +02:00
Chris L
ae4958be95 rpc: RPCResult Type of MempoolEntryDescription should be OBJ. If multiple entries are possible, wrapping Type should be OBJ_DYN. fixes #19579 2020-07-24 18:15:51 +02:00
Wladimir J. van der Laan
40a04814d1
Merge #19472: [net processing] Reduce cs_main scope in MaybeDiscourageAndDisconnect()
655b195747 [net processing] Continue SendMessages processing if not disconnecting peer (John Newbery)
a49781e56d [net processing] Only call MaybeDiscourageAndDisconnect from SendMessages (John Newbery)
a1d5a428a2 [net processing] Fix bad indentation in SendMessages() (John Newbery)
1a1c23f8d4 [net processing] Change cs_main TRY_LOCK to LOCK in SendMessages() (John Newbery)

Pull request description:

  The motivation for this PR is to reduce the scope of cs_main locking in misbehavior logic. It is the first set of commits from a larger branch to move the misbehavior data out of CNodeState and into a new struct that doesn't take cs_main.

  There are some very minor behavior changes in this branch, such as:

  - Not checking for discouragement/disconnect in `ProcessMessages()` (and instead relying on the following check in `SendMessages()`)
  - Checking for discouragement/disconnect as the first action in `SendMessages()` (and not doing ping message sending first)
  - Continuing through `SendMessages()` if `MaybeDiscourageAndDisconnect()` doesn't disconnect the peer (rather than dropping out of `SendMessages()`

ACKs for top commit:
  jonatack:
    re-ACK 655b195 per `git range-diff 505b4ed f54af5e 655b195`, code/commit messages review, a bit of code history, and debug build.
  MarcoFalke:
    ACK 655b195747 only some style-nits 🚁
  promag:
    Code review ACK 655b195747.
  ariard:
    Code Review ACK 655b195

Tree-SHA512: fd6d7bc6bb789f5fb7771fb6a45f61a8faba32af93b766554f562144f9631d15c9cc849a383e71743ef73e610b4ee14853666f6fbf08a3ae35176d48c76c65d3
2020-07-24 17:20:58 +02:00
Gleb Naumenko
7cc0e8101f Remove useless 2500 limit on AddrMan queries 2020-07-24 18:02:20 +03:00
Gleb Naumenko
ded742bc5b Move filtering banned addrs inside GetAddresses() 2020-07-24 18:02:20 +03:00
fanquake
007e15dcd7
Merge #19565: build: call AC_PATH_TOOL for dsymutil in macOS cross-compile
ef3d4ce4c3 build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing #19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce4c3
  theuni:
    ACK ef3d4ce4c3.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
2020-07-24 08:57:51 +08:00