228d6a2969 build: Fix regression in "ARMv8 CRC32 intrinsics" test (Hennadii Stepanov)
Pull request description:
In the master branch, the `aarch64` binaries lack support for CRC32 intrinsics.
The `vmull_p64` is a part of the Crypto extensions from the ACLE. They are optional extensions, so they get enabled with a `+crypto` for architecture flags.
The regression was introduced in https://github.com/bitcoin/bitcoin/pull/26183 (v25.0).
The `./configure` script log excerpts:
- the master branch @ d752349029:
```
checking whether C++ compiler accepts -march=armv8-a+crc... yes
checking whether C++ compiler accepts -march=armv8-a+crypto... yes
checking for ARMv8 CRC32 intrinsics... no
checking for ARMv8 SHA-NI intrinsics... yes
```
- this PR:
```
checking whether C++ compiler accepts -march=armv8-a+crc+crypto... yes
checking whether C++ compiler accepts -march=armv8-a+crypto... yes
checking for ARMv8 CRC32 intrinsics... yes
checking for ARMv8 SHA-NI intrinsics... yes
```
Guix build:
```
x86_64
2afd81f540c6d3b36ff305e88bafe935e4272cd3efef3130aa69d49a0522541b guix-build-228d6a2969e4/output/aarch64-linux-gnu/SHA256SUMS.part
6c704d6d30d495adb3fb86befdb500eb389a02c1167163f14ab5c3c3e630e6b3 guix-build-228d6a2969e4/output/aarch64-linux-gnu/bitcoin-228d6a2969e4-aarch64-linux-gnu-debug.tar.gz
e4419963c9c0d99adc4e38538900b648f2c14f793b60c8ee2e6f5acc9d3fadd3 guix-build-228d6a2969e4/output/aarch64-linux-gnu/bitcoin-228d6a2969e4-aarch64-linux-gnu.tar.gz
7d11052b6bd28cdf26d5f2a4987f02d32c93a061907bcd048fb6d161a0466ca9 guix-build-228d6a2969e4/output/dist-archive/bitcoin-228d6a2969e4.tar.gz
```
ACKs for top commit:
TheCharlatan:
ACK 228d6a2969
Tree-SHA512: 4c27ca8acb953bf56e972d907a282ee19e3f30f7a4bf8a9822395fe0e28977cd6233e8b65b4a25cc1d3d5ff6a796d7af07653e18531c44ee3efaff1563d96d32
f95af98128 guix: default ssp for Windows GCC (fanquake)
95d55b96c2 guix: remove ssp workaround from Windows GCC (fanquake)
8f43302a0a build: remove explicit libssp linking from Windows build (fanquake)
Pull request description:
I was expecting this to fail to compile somewhere, maybe in the CI, but that doesn't seem to be the case?
Seems workable given the SSP related changes in the newer mingw-w64 headers (which are in Guix):
> Implement some of the stack protector functions/variables so -lssp is now optional when _FORTIFY_SOURCE or -fstack-protector-strong is used.
However I think this would still be broken in some older environments, so we might have to wait for a compiler bump, or similar. The optional -lssp also seems to work when using older headers, which doesn't make sense.
Would fix#28104.
ACKs for top commit:
hebasto:
ACK f95af98128, I've verified binaries from `bitcoin-f95af98128f1-win64.zip` on Windows 11 Pro 23H2.
TheCharlatan:
ACK f95af98128
Tree-SHA512: 71169ec513cfe692dfa7741d2bf37b45da05627c0af1cbd50cf8c3c04cc21c4bf88f3284532bddc1e3e648391ec78dbaca5170987a13c21ac204a7bcaf27f349
fa01f884d3 ci: Add missing COPY for ./test/lint/test_runner (MarcoFalke)
faff3e3b46 lint: Report all lint errors instead of early exit (MarcoFalke)
Pull request description:
`all-lint.py` currently collects all failures. However, the `06_script.sh` does not, since July this year (https://github.com/bitcoin/bitcoin/pull/28103#discussion_r1268115806).
Fix this by printing all failures before exiting.
Can be tested by modifying (for example) two subtrees in the same commit and then running the linters.
ACKs for top commit:
kevkevinpal:
ACK [fa01f88](fa01f884d3)
TheCharlatan:
lgtm ACK fa01f884d3
Tree-SHA512: c0f3110f2907d87e29c755e3b77a67dfae1f8a25833fe6ef8f2f2c58cfecf1aa46f1a20881576b62252b04930140a9e416c78b4edba0780d3c4fa7aaebabba81
21bfee0720 depends: bump libmultiprocess to fix capnproto deprecation warnings (Ryan Ofsky)
Pull request description:
This incorporates PR chaincodelabs/libmultiprocess#88 and reverts the NO_WERROR CI workaround added in #28735
Upstream diff: 61d5a0e661...414542f81e
---
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/issues/28722).
ACKs for top commit:
maflcko:
lgtm ACK 21bfee0720
hebasto:
ACK 21bfee0720, I have reviewed the code and it looks OK. I've also skimmed through the related changes in the https://github.com/chaincodelabs/libmultiprocess repository.
Tree-SHA512: b5addb0deed694eeec62a0ae08b4715a811110201f39f3e6cadee8fc4e6231b0e66c844a98512072a1445bac122ab561dc1711e27fb4d7ac5c08ac46780a4acf
5e7cc4144b test: add unit test for CConnman::AddedNodesContain() (Jon Atack)
cc62716920 p2p: do not make automatic outbound connections to addnode peers (Jon Atack)
Pull request description:
to allocate our limited outbound slots correctly, and to ensure addnode
connections benefit from their intended protections.
Our addnode logic usually connects the addnode peers before the automatic
outbound logic does, but not always, as a connection race can occur. If an
addnode peer disconnects us and if it was the only one from its network, there
can be a race between reconnecting to it with the addnode thread, and it being
picked as automatic network-specific outbound peer. Or our internet connection
or router or the addnode peer could be temporarily offline, and then return
online during the automatic outbound thread. Or we could add a new manual peer
using the addnode RPC at that time.
The race can be more apparent when our node doesn't know many peers, or with
networks like cjdns that currently have few bitcoin peers.
When an addnode peer is connected as an automatic outbound peer and is the only
connection we have to a network, it can be protected by our new outbound
eviction logic and persist in the "wrong role".
Finally, there does not seem to be a reason to make block-relay or short-lived
feeler connections to addnode peers, as the addnode logic will ensure we connect
to them if they are up, within the addnode connection limit.
Fix these issues by checking if the address is an addnode peer in our automatic
outbound connection logic.
ACKs for top commit:
mzumsande:
Tested ACK 5e7cc4144b
brunoerg:
utACK 5e7cc4144b
vasild:
ACK 5e7cc4144b
guggero:
utACK 5e7cc4144b
Tree-SHA512: 2438c3ec92e98aebca2a0da960534e4655a9c6e1192a24a085fc01326d95cdb1b67d8c44e4ee706bc1d8af8564126d446a21b5579dcbec61bdea5fce2f0115ee
007d6f0e85 test: fix `AddNode` unit test failure on OpenBSD (Sebastian Falbesoner)
Pull request description:
On OpenBSD 7.4, the following check of the unit test `test_addnode_getaddednodeinfo_and_connection_detection` currently fails:
```
BOOST_CHECK(!connman->AddNode({/*m_added_node=*/"127.1", /*m_use_v2transport=*/true}));
```
The reason for that is that this OS seemingly doesn't support the IPv4 shorthand notation with omitted zero-bytes:
```
$ ping 127.1
ping: no address associated with name
```
As a simple fix, this PR skips the check for this with a pre-processor #if. On NetBSD and FreeBSD, `127.1` is resolved correctly to localhost and hence the test passes (thanks to vasild for verifying on the latter!).
ACKs for top commit:
vasild:
ACK 007d6f0e85
Tree-SHA512: 8ab8393c490e1ecc140e8ff74f6fa4d26d0dd77e6a77a241cd198314b8c5afee7422f95351ca05f4c1742433dab77016a8ccb8d28062f8edd4b703a918a2bbda
d5b4c0b69e pool: change memusage_test to use int64_t, add allocation check (Martin Leitner-Ankerl)
ce881bf9fc pool: make sure PoolAllocator uses the correct alignment (Martin Leitner-Ankerl)
Pull request description:
The class `CTxOut` has a member `CAmount` which is an int64_t, and on ARM 32bit int64_t are 8 byte aligned, which is larger than the pointer alignment of 4 bytes.
So for `CCoinsMap` to be able to use the pool, we need to use the alignment of the member instead of just `alignof(void*)`.
This fixes#28906 (first noted in https://github.com/bitcoin/bitcoin/issues/28718#issuecomment-1807197107) and #28440.
ACKs for top commit:
pinheadmz:
ACK d5b4c0b69e
hebasto:
re-ACK d5b4c0b69e, the only change since my recent [review](https://github.com/bitcoin/bitcoin/pull/28913#pullrequestreview-1739334189) is an updated test.
theStack:
Tested ACK d5b4c0b69e
Tree-SHA512: 4446793fad6d56f0fe22e09ac9ade051e86de11ac039cd61c0f6b7f79874242878a6a46a2c76ac3b8f1d53464872620d39139f54b1471daccad26d6bb1ae8ca1
`CBlockPolicyEstimator` will implement `CValidationInterface` and
subscribe to its notification to process transactions added and removed
from the mempool.
Re-delegate calculation of `validForFeeEstimation` from validation to fee estimator.
Also clean up the validForFeeEstimation arg thats no longer needed in `CTxMempool`.
Co-authored-by: Matt Corallo <git@bluematt.me>
This commit adds a new callback `MempoolTransactionsRemovedForBlock` which notify
its listeners of the transactions that are removed from the mempool because a new
block is connected, along with the block height the transactions were removed.
The transactions are in `RemovedMempoolTransactionInfo` format.
`CTransactionRef`, base fee, virtual size, and height which the transaction was added
to the mempool are all members of the struct called `RemovedMempoolTransactionInfo`.
A struct `NewMempoolTransactionInfo`, which has fields similar to `RemovedMempoolTransactionInfo`,
will be added in a later commit, create a struct `TransactionInfo` with all similar fields.
They can both have a member with type `TransactionInfo`.
If the removal reason of a transaction is BLOCK, then the `removeTx`
boolean argument should be true.
Before this PR, `CBlockPolicyEstimator` have to complete updating the fee stats
before the mempool clears that's why having removeTx call outside reason!= `BLOCK`
in `addUnchecked` was not a bug.
But in a case where the `CBlockPolicyEstimator` update is asynchronous, the mempool might
clear before we update the `CBlockPolicyEstimator` fee stats.
Transactions that are removed for `BLOCK` reasons will also be incorrectly removed from
`CBlockPolicyEstimator` stats as failures.
710da28c72 ci: Switch from `apt` to `apt-get` (Hennadii Stepanov)
a6cc059ea5 ci: Update apt cache (Hennadii Stepanov)
Pull request description:
This PR aims to fix the recent errors in the "test each commit" CI job.
ACKs for top commit:
maflcko:
lgtm ACK 710da28c72
ismaelsadeeq:
utACK 710da28c72
Tree-SHA512: b42340aea00e80f791000e19791629f27df2da98adefb839cb4389d81b5eee094089ea5092a2d7b56b3990683a72e4d2fa986fc86c823c7245649af37873b790
If alignment of the PoolAllocator would be insufficient, then the test would fail. This also catches the issue with ARM 32bit,
where int64_t is aligned to 8 bytes but void* is aligned to 4 bytes. The test adds a check to ensure the pool has allocated
a minimum number of chunks
The `vmull_p64` is a part of the Crypto extensions from the ACLE. They
are optional extensions, so they get enabled with a `+crypto` for
architecture flags.
The nVersion field is unused, so remove it.
This is also required for future commits.
Also, add PushMessage aliases in PeerManagerImpl to make calling code
less verbose.
Co-Authored-By: Anthony Towns <aj@erisian.com.au>
This changes the PoolAllocator to default the alignment to the given type. This makes the code simpler, and most importantly
fixes a bug on ARM 32bit that caused OOM: The class CTxOut has a member CAmount which is an int64_t and on ARM 32bit int64_t
are 8 byte aligned which is larger than the pointer alignment of 4 bytes. So for CCoinsMap to be able to use the pool, we
need to use the alignment of the member instead of just alignof(void*).
This has the goal of prohibiting users from accidentally creating
runtime failures, e.g. by interacting with iterator_to with a copied
entry.
CTxMemPoolEntry is already implicitly not move-constructable. So be
explicit about this and use a std::list to collect the values in the
policy_estimator fuzz test instead of a std::vector.
Co-authored-by: Anthony Towns <aj@erisian.com.au>
fa552e8a4e doc: Simplify guix install doc, after 1.4 release (MarcoFalke)
Pull request description:
Now that 1.4 is out (for a while), remove the recommendation to build a random commit.
ACKs for top commit:
fanquake:
ACK fa552e8a4e
hebasto:
ACK fa552e8a4e.
Tree-SHA512: f5642df201ff0e2af8a7ae9660a66920ddbb5f522b3e921f6f4aa7c411ced23afa91bdfe43b943ac012228eebbaad3396df505d00aa8f721a4358f03fda9d8e3
a478c817b2 test: replace `Callable`/`Iterable` with their `collections.abc` alternative (PEP 585) (stickies-v)
4b9afb18e6 scripted-diff: use PEP 585 built-in collection types for verify-binary script (Sebastian Falbesoner)
d516cf83ed test: use built-in collection types for type hints (Python 3.9 / PEP 585) (Sebastian Falbesoner)
Pull request description:
With Python 3.9 / [PEP 585](https://peps.python.org/pep-0585/), [type hinting has become a little less awkward](https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generics-in-standard-collections), as for collection types one doesn't need to import the corresponding capitalized types (`Dict`, `List`, `Set`, `Tuple`, ...) anymore, but can use the built-in types directly (see https://peps.python.org/pep-0585/#implementation for the full list).
This PR applies the replacement for all Python scripts (i.e. in the contrib and test folders) for the basic types, i.e.:
- typing.Dict -> dict
- typing.List -> list
- typing.Set -> set
- typing.Tuple -> tuple
For an additional check, I ran mypy 1.6.1 on both master and the PR branch via
```
$ mypy --ignore-missing-imports --explicit-package-bases $(git ls-files "*.py")
```
and verified that the output is identical -- (from the 22 identified problems, most look like false-positives, it's probably worth it to go deeper here and address them in a follow-up though).
ACKs for top commit:
stickies-v:
ACK a478c817b2
fanquake:
ACK a478c817b2
Tree-SHA512: 6948c905f6abd644d84f09fcb3661d7edb2742e8f2b28560008697d251d77a61a1146ab4b070e65b0d27acede7a5256703da7bf6eb1c7c3a897755478c76c6e8
83986f464c Include version.h in fewer places (Anthony Towns)
c7b61fd61b Convert some CDataStream to DataStream (Anthony Towns)
1410d300df serialize: Drop useless version param from GetSerializeSize() (Anthony Towns)
bf574a7501 serialize: drop GetSerializeSizeMany (Anthony Towns)
efa9eb6d7c serialize: Drop nVersion from [C]SizeComputer (Anthony Towns)
Pull request description:
Drops the version field from `GetSerializeSize()`, simplifying the code in various places. Also drop `GetSerializeSizeMany()` (as just removing the version parameter could result in silent bugs) and remove unnecessary instances of `#include <version.h>`.
ACKs for top commit:
maflcko:
ACK 83986f464c📒
theuni:
ACK 83986f464c.
Tree-SHA512: 36617b6dfbb1b4b0afbf673e905525fc6d623d3f568d3f86e3b9d4f69820db97d099e83a88007bfff881f731ddca6755ebf1549e8d8a7762437dfadbf434c62e
faa25718b3 fuzz: AutoFile with XOR (MarcoFalke)
fab5cb9066 fuzz: Reduce LIMITED_WHILE limit for file fuzzing (MarcoFalke)
fa5388fad3 fuzz: Remove FuzzedAutoFileProvider (MarcoFalke)
Pull request description:
This should help to get fuzz coverage for https://maflcko.github.io/b-c-cov/fuzz.coverage/src/streams.cpp.gcov.html
Also, remove unused code and fix a timeout bug.
ACKs for top commit:
dergoegge:
ACK faa25718b3
Tree-SHA512: 56f1e6fd5cb2b66ffd9a7d9c09c9b8e396be3e7485feb03b35b6bd3c48e624fdaed50b472e4ffec21f09efb5e949d7ee32a13851849c9140b6b4cf25917dd7ac
to allocate our limited outbound slots correctly, and to ensure addnode
connections benefit from their intended protections.
Our addnode logic usually connects the addnode peers before the automatic
outbound logic does, but not always, as a connection race can occur. If an
addnode peer disconnects us and if it was the only one from its network, there
can be a race between reconnecting to it with the addnode thread, and it being
picked as automatic network-specific outbound peer. Or our internet connection
or router, or the addnode peer, could be temporarily offline, and then return
online during the automatic outbound thread. Or we could add a new manual peer
using the addnode RPC at that time.
The race can be more apparent when our node doesn't know many peers, or with
networks like cjdns that currently have few bitcoin peers.
When an addnode peer is connected as an automatic outbound peer and is the only
connection we have to a network, it can be protected by our new outbound
eviction logic and persist in the "wrong role".
Examples on mainnet using logging added in the same pull request:
2023-08-12T14:51:05.681743Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic network-specific outbound-full-relay connection
to i2p peer selected for manual (addnode) connection: [geh...odq.b32.i2p]:0
2023-08-13T03:59:28.050853Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic block-relay-only connection to onion peer
selected for manual (addnode) connection: kpg...aid.onion:8333
2023-08-13T16:21:26.979052Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic network-specific outbound-full-relay connection
to cjdns peer selected for manual (addnode) connection: [fcc...8ce]:8333
2023-08-14T20:43:53.401271Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic network-specific outbound-full-relay connection
to cjdns peer selected for manual (addnode) connection: [fc7...59e]:8333
2023-08-15T00:10:01.894147Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic feeler connection to i2p peer selected for
manual (addnode) connection: geh...odq.b32.i2p:8333
Finally, there does not seem to be a reason to make block-relay or short-lived
feeler connections to addnode peers, as the addnode logic will ensure we connect
to them if they are up, within the addnode connection limit.
Fix these issues by checking if the address is an addnode peer in our automatic
outbound connection logic.
43de4d3630 doc: fix typos (Sjors Provoost)
Pull request description:
This PR fixes typos found by lint-spelling.py using codespell 2.2.6.
Our CI linter job uses codespell 2.2.5 and found fewer typos that I did locally. In any case it's happy now.
ACKs for top commit:
pablomartin4btc:
re ACK 43de4d3630
Tree-SHA512: c032fe86cb49c924a468385653b31f309a9db68c478d70335bba3e65a1ff3826abe80284fe00a090ab5a509e1edbf17e476f6922fb15d055e50f1103dad2ccb0
