Commit graph

358 commits

Author SHA1 Message Date
fanquake
e4e9dd3a28
contrib: fix implicit function decleration in win symbol check
```bash
test3.c: In function 'main':
test3.c:6:21: warning: implicit declaration of function 'CoFreeUnusedLibrariesEx' [-Wimplicit-function-declaration]
    6 |                     CoFreeUnusedLibrariesEx(0,0);
```
2022-03-07 11:51:11 +00:00
fanquake
5a89bed410
contrib: address gen-manpages feedback from #24263
Co-authored-by: Carl Dong <contact@carldong.me>
2022-02-22 15:36:19 +00:00
laanwj
42c202893b doc: Fix gen-manpages, rewrite in Python
Rewrite the manual page generation script in Python.

This:
- Solves '-' stripping issue (fixes #22681)
- Makes that copyright footer is generated again
2022-02-04 17:03:31 +01:00
fanquake
6fe55160dd
contrib: support arm64 darwin in security checks 2022-01-26 17:32:10 +08:00
fanquake
446e73cc0b
build: use macOS 11 SDK (Xcode 12.2)
This should be sufficient to support building for Apple ARM when
cross-compiling.
2022-01-26 17:28:16 +08:00
laanwj
792d0d8d51
Merge bitcoin/bitcoin#23839: Linux: build with and test for control flow instrumentation on x86_64
5a8f907c93 scripts: add CONTROL_FLOW to ELF security checks (fanquake)
e13f8f775d build: build x86_64 Linux Boost with -fcf-protection=full (fanquake)
6ca5efa8ed script rename control flow check to MACHO specific (fanquake)

Pull request description:

  Closes #21888.

  TODO:
  * Duplication in security-check-tests

  Guix build:
  ```bash
  bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
  8025e2e6859247eaf865a4a5009d0a39570ec5e8ab51739fa4da0d8ee4ab2117  guix-build-5a8f907c93f1/output/aarch64-linux-gnu/SHA256SUMS.part
  75576482994493262dbf7d3567b0450c0804bdc75c186d6b4c6a856bd0d8f0a3  guix-build-5a8f907c93f1/output/aarch64-linux-gnu/bitcoin-5a8f907c93f1-aarch64-linux-gnu-debug.tar.gz
  7cb89c9327cc540859334b597c041a2250156e3e83ce2aa7d16177376056302f  guix-build-5a8f907c93f1/output/aarch64-linux-gnu/bitcoin-5a8f907c93f1-aarch64-linux-gnu.tar.gz
  e264053e4c7a5d65340dd7bbd2c664bcd596292ab80f00a0bf0026bfe0c480bc  guix-build-5a8f907c93f1/output/arm-linux-gnueabihf/SHA256SUMS.part
  6c3ee9b3c0c4583620301b183226678f1618605dd07dbed0bbdf7d06d3591314  guix-build-5a8f907c93f1/output/arm-linux-gnueabihf/bitcoin-5a8f907c93f1-arm-linux-gnueabihf-debug.tar.gz
  f217cb8d8e233a8dbdbfde7dabf12c5d867d7de53e8c652b8ed1d4a55da82fd9  guix-build-5a8f907c93f1/output/arm-linux-gnueabihf/bitcoin-5a8f907c93f1-arm-linux-gnueabihf.tar.gz
  48b4b2a1b52b3098f4e92c11cb60f1e5e9696a2a960560cd6adea72277eaa4a4  guix-build-5a8f907c93f1/output/dist-archive/bitcoin-5a8f907c93f1.tar.gz
  40832db2446e129879caa9fbc9d682c53069876dbb2e0d4d76592e5dcb40bb12  guix-build-5a8f907c93f1/output/powerpc64-linux-gnu/SHA256SUMS.part
  26fdfa9b7b77d5db415ef34054cf6f6d020a5dab73239db6dd05539f654bc5d5  guix-build-5a8f907c93f1/output/powerpc64-linux-gnu/bitcoin-5a8f907c93f1-powerpc64-linux-gnu-debug.tar.gz
  3adbdb9d3eb1cb5f9adc38b29450054f286bd6d74cef8619adaee89494853605  guix-build-5a8f907c93f1/output/powerpc64-linux-gnu/bitcoin-5a8f907c93f1-powerpc64-linux-gnu.tar.gz
  8ec2baf82483a698350bfdabf530cd9b5241690c916f597c746210e95ac451de  guix-build-5a8f907c93f1/output/powerpc64le-linux-gnu/SHA256SUMS.part
  1797e75e1f66ec9068fa4e57e0108960475e863f8f054fbe854358b1f995c4df  guix-build-5a8f907c93f1/output/powerpc64le-linux-gnu/bitcoin-5a8f907c93f1-powerpc64le-linux-gnu-debug.tar.gz
  448bd289ef26c777a1fc4498e7ba7fb17d0f6a932dcac91b2f89cbba63704bb8  guix-build-5a8f907c93f1/output/powerpc64le-linux-gnu/bitcoin-5a8f907c93f1-powerpc64le-linux-gnu.tar.gz
  a23ee91eeae515c2a6a31eb25d659fab833839aaafa4676ccee364bdad2a468c  guix-build-5a8f907c93f1/output/riscv64-linux-gnu/SHA256SUMS.part
  82df1b6d5020d0af8268ecc8e823f752f20dec308277763b2dd675804dfa4bbd  guix-build-5a8f907c93f1/output/riscv64-linux-gnu/bitcoin-5a8f907c93f1-riscv64-linux-gnu-debug.tar.gz
  09a6098ce83896a6ee6d5c8aff12eaca51595bd724c8e0b2a6f90b6410dc168c  guix-build-5a8f907c93f1/output/riscv64-linux-gnu/bitcoin-5a8f907c93f1-riscv64-linux-gnu.tar.gz
  4fac2951f80eaa2bd1747a263be6be1b76282cac5062f7d86db631a2fb80f8db  guix-build-5a8f907c93f1/output/x86_64-apple-darwin/SHA256SUMS.part
  3392f417b09efca5916c384f0b2d0c177a72ec4921399c62e84484f0054cc8c4  guix-build-5a8f907c93f1/output/x86_64-apple-darwin/bitcoin-5a8f907c93f1-osx-unsigned.dmg
  bfb4f8ade6107996ec4bc9efdb53959151b8fb19b6790c34472fe218dd02383d  guix-build-5a8f907c93f1/output/x86_64-apple-darwin/bitcoin-5a8f907c93f1-osx-unsigned.tar.gz
  4de92e149bad46fc863efb3b650753d194aad96be991d020e0b859452cf27457  guix-build-5a8f907c93f1/output/x86_64-apple-darwin/bitcoin-5a8f907c93f1-osx64.tar.gz
  9329549a2f275a59e329afc8744ff8cbc297f3042e0ad03b47626805c7aae2e8  guix-build-5a8f907c93f1/output/x86_64-linux-gnu/SHA256SUMS.part
  d87deb1eca8a1d3780f02edde78350d27f700e13c7ff444be0bfb34e7369904a  guix-build-5a8f907c93f1/output/x86_64-linux-gnu/bitcoin-5a8f907c93f1-x86_64-linux-gnu-debug.tar.gz
  76f3bc2fad010d9373e854d941e8205b68fa6c9a8ecaff34c4978ae3ae76c806  guix-build-5a8f907c93f1/output/x86_64-linux-gnu/bitcoin-5a8f907c93f1-x86_64-linux-gnu.tar.gz
  2948631081c4bd475529da2b7bc2b32b5aa2e44de46dfdaa6cf9432b3c9fe869  guix-build-5a8f907c93f1/output/x86_64-w64-mingw32/SHA256SUMS.part
  c5d56fac163b73c00e9745aa7b0f9a0bd9fcac7517e39b677869b6e76faf7218  guix-build-5a8f907c93f1/output/x86_64-w64-mingw32/bitcoin-5a8f907c93f1-win-unsigned.tar.gz
  f12c7cd72511544c67f4934981c090cee0d9c17e931c059edbbbef6e843dd651  guix-build-5a8f907c93f1/output/x86_64-w64-mingw32/bitcoin-5a8f907c93f1-win64-debug.zip
  eb952cc4554f92ba6787353f4305d7cbcb1c6dafc4a3867b3088106252a573b8  guix-build-5a8f907c93f1/output/x86_64-w64-mingw32/bitcoin-5a8f907c93f1-win64-setup-unsigned.exe
  7bd2b071f1cdf9410535e6a10dd1da519f942bd2c1e47ed52c5b8c4f977e8f27  guix-build-5a8f907c93f1/output/x86_64-w64-mingw32/bitcoin-5a8f907c93f1-win64.zip
  ```

ACKs for top commit:
  laanwj:
    Code review ACK 5a8f907c93
  hebasto:
    ACK 5a8f907c93, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: 75702ac01175ccf08e73b5c3bce266cee9dd15ccf8fb38d46108cfada156de9a8c18e27d996f9343ae33f7a75a6904d335cbe25256d90af1f4ccbe72ce1788bb
2022-01-25 16:01:21 +01:00
fanquake
3ccfba1c7f
guix: use GCC 10 (over GCC 8) to build releases
This currently points to the version-1.4.0 branch.
2022-01-05 10:32:19 +08:00
fanquake
5a8f907c93
scripts: add CONTROL_FLOW to ELF security checks 2022-01-04 22:45:45 +08:00
fanquake
6ca5efa8ed
script rename control flow check to MACHO specific 2022-01-04 22:45:44 +08:00
fanquake
b9898aeeaa
scripts: make security checks architecture independent
This paves the way for using and checking for architecture dependent
flags like -fcf-protection on x86_64 Linux and -mbranch-protection on 64
bit ARM.
2022-01-03 21:36:55 +08:00
Hennadii Stepanov
f47dda2c58
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
* 2020: fa0074e2d8
* 2019: aaaaad6ac9
2021-12-30 19:36:57 +02:00
Hennadii Stepanov
c29105efdc
script: Fix copyright_header.py
This change prevents updating copyright years from "2021" to
"2021-2017".
2021-12-30 19:33:20 +02:00
fanquake
d1711a40b3
scripts: match on exe type over str in symbol-check.py 2021-12-18 11:41:49 +08:00
fanquake
feee308401
scripts: match on exe type over str in security-check.py 2021-12-18 11:41:36 +08:00
fanquake
c870027cc2
build: qt 5.15.2 2021-12-03 14:07:12 +08:00
fanquake
f31d4bd214
scripts: test for PE control flow instrumentation 2021-11-17 16:12:47 +08:00
fanquake
dbfca4a815
build: require glibc 2.18+ for release builds
From what I can see the only platform this drops support for is CentOS
7. CentOS 7 reached the end of it's "full update" support at the end of
2020. It does receive maintenance updates until 2024, however I don't
think supporting glibc 2.17 until 2024 is realistic. Note that anyone
wanting to self-compile and target a glibc 2.17 runtime could build with
--disable-threadlocal.

glibc 2.18 was released in August 2013.
https://sourceware.org/legacy-ml/libc-alpha/2013-08/msg00160.html
2021-11-16 07:51:24 +08:00
Hennadii Stepanov
9a1ad7bc0d
test: Enable SC2086 shellcheck rule 2021-11-13 16:54:56 +02:00
Gleb Naumenko
0659f12b13
Add minisketch dependency 2021-10-21 09:38:55 +08:00
josibake
a46f71bb70
lint: enable mypy checking for missing imports
Achieve this by adding some ignore, and making data/ importable.

Co-authored-by: João Barbosa <joao.paulo.barbosa@gmail.com>
2021-10-16 09:14:37 +08:00
Carl Dong
1527b7e8a1 symbol-check: Check requested ELF interpreter
It is important that binaries request a standard interpreter location
where most distros would place the linker-loader. Otherwise, the user
would be met with a very confusing message:

    bash: <path>/<to>/bitcoind: No such file or directory

When really it's the interpreter that's not found.
2021-10-13 08:39:48 -04:00
fanquake
ce69e18947
scripts: remove pixie.py 2021-10-12 08:36:21 +08:00
fanquake
00b85d0b13
scripts: only parse the binary once in security-check.py 2021-10-12 08:36:21 +08:00
fanquake
cad40a5b16
scripts: use LIEF for ELF checks in security-check.py 2021-10-12 08:36:21 +08:00
fanquake
8242ae230e
scripts: only parse the binary once in symbol-check.py 2021-10-12 08:36:21 +08:00
fanquake
309eac9019
scripts: use LIEF for ELF checks in symbol-check.py
Co-authored-by: Carl Dong <contact@carldong.me>
2021-10-12 08:36:15 +08:00
Carl Dong
610a8a8e39
test-*-check: Pass in *FLAGS and compile with them
These test-*-check scripts should compile "test" binaries in a way that
is as close to what autotools would do, since the goal is to make sure
that if we run the *-check script, they can correctly detect flaws in
binaries which are compiled by our autotools-based system.

Therefore, we should emulate what happens when the binary is linked in
autotools, meaning that for C binaries, we need to supply the CFLAGS,
CPPFLAGS, and LDFLAGS flags in that order.

Note to future developers: perhaps it'd be nice to have these
test-*-check scripts be part of configure.ac to avoid having to manually
replicate autoconf-like behaviour every time we find a discrepancy. Of
course, that would also mean you'd have to write more m4...
2021-10-12 08:28:25 +08:00
fanquake
a43b8e9555
build: set OSX_MIN_VERSION to 10.15
This is required to use std::filesystem on macOS as support for it only
landed in the libc++ dylib shipped with 10.15.

See also: https://developer.apple.com/documentation/xcode-release-notes/xcode-11-release-notes

Clang now supports the C++17 <filesystem> library for iOS 13, macOS 10.15, watchOS 6, and tvOS 13.
2021-09-16 17:50:19 +08:00
fanquake
ab9c34237a
release: remove gitian 2021-08-31 09:37:23 +08:00
fanquake
5449d44e37
scripts: prevent GCC optimising test symbols in test-symbol-check 2021-08-09 13:59:48 +08:00
fanquake
6cf3345297
scripts: adjust test-symbol-check for guix release environment
Now that our release binaries are build in a glibc 2.24 and 2.27
environment, we can't use a symbol from glibc 2.28 to test our checks.
Replace renameat2() with nextup(), which was introduced in 2.24.

Note that this also means re-disabling the test for RISC-V, however
RISC-V is built in a glibc 2.27 environment, and our minimum required
glibc for that binary is 2.27.
2021-07-07 20:29:01 +08:00
fanquake
1946b5f77c
scripts: more robustly test macOS symbol checks 2021-07-07 19:31:37 +08:00
Carl Dong
a8127b34bc
build: Use and test PE binutils with --reloc-section
Also fix test-security-check.py to account for new PE PIE failure
indication.
2021-07-07 19:31:37 +08:00
Carl Dong
9fdc8afe11
devtools: Improve *-check.py tool detection
This is important to make sure that we're not testing tools different
from the one we're building with.

Introduce determine_wellknown_cmd, which encapsulates how we
should handle well-known tools specification (IFS splitting, env
override, etc.).
2021-07-07 19:31:37 +08:00
fanquake
797b3ed909
script: remove gitian reference from symbol-check.py 2021-07-06 20:50:47 +08:00
W. J. van der Laan
a305a687e7
Merge bitcoin/bitcoin#22244: devtools: Correctly extract symbol versions in symbol-check
e8cd3700ee devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in symbol-check.py (W. J. van der Laan)
a33381acf5 devtools: Add xkb version to symbol-check (W. J. van der Laan)
19e598bab0 devtools: Fix verneed section parsing in pixie (W. J. van der Laan)

Pull request description:

  I misunderstood the ELF specification for version symbols (verneed): The `vn_aux` pointer is relative to the main verneed record, not the start of the section.

  This caused many symbols to not be versioned properly in the return value of `elf.dyn_symbols`. This was discovered in #21454.

  Fix it by correcting the offset computation.

  - xkb versions symbols (using the prefix `V`), as this library is used by bitcoin-qt, add it to the valid versions in `symbol-check.py`

  This unfortunately brings to light some symbols that have been introduced since and weren't caught (from a gitian compile of master):

  ```
  bitcoin-cli: symbol getrandom from unsupported version GLIBC_2.25
  bitcoin-cli: failed IMPORTED_SYMBOLS
  bitcoind: symbol getrandom from unsupported version GLIBC_2.25
  bitcoind: symbol log from unsupported version GLIBC_2.29
  bitcoind: symbol fcntl64 from unsupported version GLIBC_2.28
  bitcoind: symbol pow from unsupported version GLIBC_2.29
  bitcoind: symbol exp from unsupported version GLIBC_2.29
  bitcoind: failed IMPORTED_SYMBOLS
  bitcoin-qt: symbol exp from unsupported version GLIBC_2.29
  bitcoin-qt: symbol fcntl64 from unsupported version GLIBC_2.28
  bitcoin-qt: symbol log from unsupported version GLIBC_2.29
  bitcoin-qt: symbol pow from unsupported version GLIBC_2.29
  bitcoin-qt: symbol statx from unsupported version GLIBC_2.28
  bitcoin-qt: symbol getrandom from unsupported version GLIBC_2.25
  bitcoin-qt: symbol renameat2 from unsupported version GLIBC_2.28
  bitcoin-qt: symbol getentropy from unsupported version GLIBC_2.25
  bitcoin-qt: failed IMPORTED_SYMBOLS
  bitcoin-wallet: symbol exp from unsupported version GLIBC_2.29
  bitcoin-wallet: symbol log from unsupported version GLIBC_2.29
  bitcoin-wallet: symbol fcntl64 from unsupported version GLIBC_2.28
  bitcoin-wallet: failed IMPORTED_SYMBOLS
  test_bitcoin: symbol getrandom from unsupported version GLIBC_2.25
  test_bitcoin: symbol log from unsupported version GLIBC_2.29
  test_bitcoin: symbol fcntl64 from unsupported version GLIBC_2.28
  test_bitcoin: symbol pow from unsupported version GLIBC_2.29
  test_bitcoin: symbol exp from unsupported version GLIBC_2.29
  test_bitcoin: failed IMPORTED_SYMBOLS
  ```

ACKs for top commit:
  hebasto:
    ACK e8cd3700ee

Tree-SHA512: 8c15e3478eb642f01a1ddaadef03f80583f088f9fa8e3bf171ce16b0ec05ffb4675ec147d7ffc6a4360637ed47fca517c6ca2bac7bb30d794c03783cfb964b79
2021-06-21 07:58:12 +02:00
W. J. van der Laan
e8cd3700ee devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in symbol-check.py
The (ancient) versions specified here were deceptive. Entries older than
MAX_VERSIONS['GLIBC'], which is 2.17, are ignored here. So reorganize
the code to avoid confusion for other people reading this code.
2021-06-18 13:25:57 +02:00
W. J. van der Laan
a33381acf5 devtools: Add xkb version to symbol-check
xkb versions symbols (using the prefix `V`), as this library is used by
bitcoin-qt, add it to the valid versions in `symbol-check.py`.
2021-06-14 20:32:09 +02:00
W. J. van der Laan
19e598bab0 devtools: Fix verneed section parsing in pixie
I misunderstood the ELF specification for version symbols (verneed):
The `vn_aux` pointer is relative to the main verneed record, not the
start of the section.

This caused many symbols to not be versioned properly in the return
value of `elf.dyn_symbols`. This was discovered in #21454.

Fix it by correcting the offset computation.
2021-06-14 20:31:58 +02:00
fanquake
aa80b5759d
scripts: check macOS SDK version is set
Clangs Darwin driver should infer the SDK version used during compilation, and
forward that through to the linker. Add a check that this has been done, and the
expected SDK version is set.

Should help prevent issues like #21771 in future.
2021-06-10 15:43:50 +08:00
fanquake
c972345bac
scripts: check minimum required Windows version is set
We use linker flags (-Wl,--major/minor-subsystem-version) to set the
minimum required version of Windows needed to run our binaries. This
adds a sanity check that the version is being set as expected.
2021-06-10 10:40:53 +08:00
fanquake
29615aef52
scripts: check minimum required macOS vesion is set
We use a compile flag (-mmacosx-version-min) to set the minimum required
version of macOS needed to run our binaries. This adds a sanity check
that the version is being set as expected.
2021-06-10 10:40:53 +08:00
fanquake
42b589d18f
scripts: test for MACHO control flow instrumentation 2021-05-09 14:26:09 +08:00
fanquake
955140b326
contrib: consolidate PIE and NX security checks 2021-05-04 20:48:00 +08:00
fanquake
2aa1631822
contrib: use LIEF in PE symbol checks 2021-05-04 20:48:00 +08:00
fanquake
e93ac26b85
contrib: use LIEF in macOS symbol checks 2021-05-04 20:48:00 +08:00
fanquake
a632cbcee5
contrib: use f strings in symbol-check.py 2021-05-04 20:48:00 +08:00
fanquake
0f5d77c8e4
contrib: add PE PIE check to security checks 2021-05-04 20:48:00 +08:00
fanquake
8e1f40dd9a
contrib: use LIEF for PE security checks 2021-05-04 20:47:59 +08:00
fanquake
a25b2e965c
contrib: use LIEF for macOS security checks 2021-05-04 20:47:59 +08:00