Commit graph

374 commits

Author SHA1 Message Date
fanquake
797b3ed909
script: remove gitian reference from symbol-check.py 2021-07-06 20:50:47 +08:00
W. J. van der Laan
a305a687e7
Merge bitcoin/bitcoin#22244: devtools: Correctly extract symbol versions in symbol-check
e8cd3700ee devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in symbol-check.py (W. J. van der Laan)
a33381acf5 devtools: Add xkb version to symbol-check (W. J. van der Laan)
19e598bab0 devtools: Fix verneed section parsing in pixie (W. J. van der Laan)

Pull request description:

  I misunderstood the ELF specification for version symbols (verneed): The `vn_aux` pointer is relative to the main verneed record, not the start of the section.

  This caused many symbols to not be versioned properly in the return value of `elf.dyn_symbols`. This was discovered in #21454.

  Fix it by correcting the offset computation.

  - xkb versions symbols (using the prefix `V`), as this library is used by bitcoin-qt, add it to the valid versions in `symbol-check.py`

  This unfortunately brings to light some symbols that have been introduced since and weren't caught (from a gitian compile of master):

  ```
  bitcoin-cli: symbol getrandom from unsupported version GLIBC_2.25
  bitcoin-cli: failed IMPORTED_SYMBOLS
  bitcoind: symbol getrandom from unsupported version GLIBC_2.25
  bitcoind: symbol log from unsupported version GLIBC_2.29
  bitcoind: symbol fcntl64 from unsupported version GLIBC_2.28
  bitcoind: symbol pow from unsupported version GLIBC_2.29
  bitcoind: symbol exp from unsupported version GLIBC_2.29
  bitcoind: failed IMPORTED_SYMBOLS
  bitcoin-qt: symbol exp from unsupported version GLIBC_2.29
  bitcoin-qt: symbol fcntl64 from unsupported version GLIBC_2.28
  bitcoin-qt: symbol log from unsupported version GLIBC_2.29
  bitcoin-qt: symbol pow from unsupported version GLIBC_2.29
  bitcoin-qt: symbol statx from unsupported version GLIBC_2.28
  bitcoin-qt: symbol getrandom from unsupported version GLIBC_2.25
  bitcoin-qt: symbol renameat2 from unsupported version GLIBC_2.28
  bitcoin-qt: symbol getentropy from unsupported version GLIBC_2.25
  bitcoin-qt: failed IMPORTED_SYMBOLS
  bitcoin-wallet: symbol exp from unsupported version GLIBC_2.29
  bitcoin-wallet: symbol log from unsupported version GLIBC_2.29
  bitcoin-wallet: symbol fcntl64 from unsupported version GLIBC_2.28
  bitcoin-wallet: failed IMPORTED_SYMBOLS
  test_bitcoin: symbol getrandom from unsupported version GLIBC_2.25
  test_bitcoin: symbol log from unsupported version GLIBC_2.29
  test_bitcoin: symbol fcntl64 from unsupported version GLIBC_2.28
  test_bitcoin: symbol pow from unsupported version GLIBC_2.29
  test_bitcoin: symbol exp from unsupported version GLIBC_2.29
  test_bitcoin: failed IMPORTED_SYMBOLS
  ```

ACKs for top commit:
  hebasto:
    ACK e8cd3700ee

Tree-SHA512: 8c15e3478eb642f01a1ddaadef03f80583f088f9fa8e3bf171ce16b0ec05ffb4675ec147d7ffc6a4360637ed47fca517c6ca2bac7bb30d794c03783cfb964b79
2021-06-21 07:58:12 +02:00
W. J. van der Laan
e8cd3700ee devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in symbol-check.py
The (ancient) versions specified here were deceptive. Entries older than
MAX_VERSIONS['GLIBC'], which is 2.17, are ignored here. So reorganize
the code to avoid confusion for other people reading this code.
2021-06-18 13:25:57 +02:00
W. J. van der Laan
a33381acf5 devtools: Add xkb version to symbol-check
xkb versions symbols (using the prefix `V`), as this library is used by
bitcoin-qt, add it to the valid versions in `symbol-check.py`.
2021-06-14 20:32:09 +02:00
W. J. van der Laan
19e598bab0 devtools: Fix verneed section parsing in pixie
I misunderstood the ELF specification for version symbols (verneed):
The `vn_aux` pointer is relative to the main verneed record, not the
start of the section.

This caused many symbols to not be versioned properly in the return
value of `elf.dyn_symbols`. This was discovered in #21454.

Fix it by correcting the offset computation.
2021-06-14 20:31:58 +02:00
fanquake
aa80b5759d
scripts: check macOS SDK version is set
Clangs Darwin driver should infer the SDK version used during compilation, and
forward that through to the linker. Add a check that this has been done, and the
expected SDK version is set.

Should help prevent issues like #21771 in future.
2021-06-10 15:43:50 +08:00
fanquake
c972345bac
scripts: check minimum required Windows version is set
We use linker flags (-Wl,--major/minor-subsystem-version) to set the
minimum required version of Windows needed to run our binaries. This
adds a sanity check that the version is being set as expected.
2021-06-10 10:40:53 +08:00
fanquake
29615aef52
scripts: check minimum required macOS vesion is set
We use a compile flag (-mmacosx-version-min) to set the minimum required
version of macOS needed to run our binaries. This adds a sanity check
that the version is being set as expected.
2021-06-10 10:40:53 +08:00
fanquake
42b589d18f
scripts: test for MACHO control flow instrumentation 2021-05-09 14:26:09 +08:00
fanquake
955140b326
contrib: consolidate PIE and NX security checks 2021-05-04 20:48:00 +08:00
fanquake
2aa1631822
contrib: use LIEF in PE symbol checks 2021-05-04 20:48:00 +08:00
fanquake
e93ac26b85
contrib: use LIEF in macOS symbol checks 2021-05-04 20:48:00 +08:00
fanquake
a632cbcee5
contrib: use f strings in symbol-check.py 2021-05-04 20:48:00 +08:00
fanquake
0f5d77c8e4
contrib: add PE PIE check to security checks 2021-05-04 20:48:00 +08:00
fanquake
8e1f40dd9a
contrib: use LIEF for PE security checks 2021-05-04 20:47:59 +08:00
fanquake
a25b2e965c
contrib: use LIEF for macOS security checks 2021-05-04 20:47:59 +08:00
fanquake
7e7eae7aa8
contrib: use f strings in security-check.py 2021-05-04 20:47:59 +08:00
wodry
ea76f4ac7d Doc: Tell howto install clang-format on Debian/Ubuntu
Because only macOS wasy mentioned, I was unsure if this would be a macOS specific tool. I guess Linux is more used than Mac, so Linux guide should be there, too.
2021-03-19 19:24:40 +01:00
Hennadii Stepanov
0fc0c00f7a
test: Drop unused get_machine function
The removed get_machine function is no longer used since #21255.
2021-03-13 18:29:08 +02:00
Hennadii Stepanov
61a0f8f9cc
test: Cleanup test files in test-{security,symbol}-check.py 2021-03-13 18:27:11 +02:00
fanquake
06cd0da21f
build: qt 5.12.10
remove fix_configure_mac.patch
Fixed upstream: https://bugreports.qt.io/browse/QTBUG-67286

remove fix_riscv64_arch.patch
Was fixed upstream in 6a39e49a6cdeb28a04a3657bb6a22f848d5dfa9d

remove fix_rcc_determinism.patch
Fixed upstream in https://bugreports.qt.io/browse/QTBUG-62511

remove freetype_back_compat.patch
By the time we ship a release with Qt 5.12, we'll certainly no-longer be
supporting Ubuntu 14.04 and Ubuntu 16.04 ships with FreeType 2.6.1,
which is new enough that using the symbol is no-longer an issue.
The renaming of FT_Get_X11_Font_Format() happened in FreeType 2.6

remove xkb-default.patch
This was removed upstream in d5abf545971da717014d316127045fc19edbcd65

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
2021-03-10 12:57:22 +08:00
Hennadii Stepanov
3272e34f9c
build: Add xkbcommon 0.8.4
Co-authored-by: fanquake <fanquake@gmail.com>
2021-03-10 08:23:47 +08:00
fanquake
ccb7b6a976
contrib: run test-symbol check for RISCV
Now that we are using Focal for Gitian building (glibc 2.31), we can
user a newer introduced  symbol, and include RISCV in this test.
2021-02-22 08:35:06 +08:00
Wladimir J. van der Laan
b5e93f873a doc: Add manual page generation for bitcoin-util
- Add `-version` option to `bitcoin-util`
- Add `bitcoin-util` call to `gen-manpages.sh`
- Add stub manual page `bitcoin-util.1`
- Add install of `bitcoin-util.1` to build system
2021-01-12 14:09:21 +01:00
MarcoFalke
fa0074e2d8
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2020-12-31 09:45:41 +01:00
fanquake
1ef2138c0d
lint: run mypy over contrib/devtools 2020-12-28 14:25:06 +08:00
Wladimir J. van der Laan
f1dbf92ff0
Merge #20434: contrib: Parse ELF directly for symbol and security checks
a0a771843f contrib: Changes to checks for PowerPC64 (Luke Dashjr)
634f6ec4eb contrib: Parse ELF directly for symbol and security checks (Wladimir J. van der Laan)

Pull request description:

  Instead of the ever-messier text parsing of the output of the readelf tool (which is clearly meant for human consumption not to be machine parseable), parse the ELF binaries directly.

  Add a small dependency-less ELF parser specific to the checks.

  This is slightly more secure, too, because it removes potential ambiguity due to misparsing and changes in the output format of `elfread`. It also allows for stricter and more specific ELF format checks in the future.

  This removes the build-time dependency for `readelf`.

  It passes the test-security-check for me locally, ~~though I haven't checked on all platforms~~. I've checked that this works on the cross-compile output for all ELF platforms supported by Bitcoin Core at the moment, as well as PPC64 LE and BE.

Top commit has no ACKs.

Tree-SHA512: 7f9241fec83ee512642fecf5afd90546964561efd8c8c0f99826dcf6660604a4db2b7255e1afb1e9bb0211fd06f5dbad18a6175dfc03e39761a40025118e7bfc
2020-12-18 12:16:40 +01:00
fanquake
ae9b48995b
contrib: add symbol check test for PE 2020-12-09 23:13:24 +08:00
Wladimir J. van der Laan
f3e17686b3
Merge #20468: build: warn when generating man pages for binaries built from a dirty branch
6690adba08 Warn when binaries are built from a dirty branch. (Tyler Chambers)

Pull request description:

  - Adjusted `--version` flag behavior in bitcoind and bitcoin-wallet to have the same behavior.
  - Added `--version` flag to bitcoin-tx to match.
  - Added functionality in gen-manpages.sh to error when attempting to generate man pages for binaries built from a dirty branch.

  mitigates problem with  issue #20412

ACKs for top commit:
  laanwj:
    Tested ACK 6690adba08

Tree-SHA512: b5ca509f1a57f66808c2bebc4b710ca00c6fec7b5ebd7eef58018e28e716f5f2358e36551b8a4df571bf3204baed565a297aeefb93990e7a99add502b97ee1b8
2020-12-07 10:51:45 +01:00
fanquake
ed1bbcefea contrib: add MACHO tests to symbol-check tests 2020-12-03 12:16:19 +01:00
Wladimir J. van der Laan
5bab08df17 contrib: Add test for ELF symbol-check
Check both failure cases:
- Use a glibc symbol from a version that is too new
- Use a symbol from a library that is not in the allowlist

And also check a conforming binary.

Adding a similar check for Windows PE can be done in a separate PR.
2020-12-03 12:15:12 +01:00
Tyler Chambers
6690adba08
Warn when binaries are built from a dirty branch.
Adjusted version flag behavior in bitcoin-tx, bitcoin-wallet, and
bitcoind to match. Added functionality in gen-manpages.sh to warning when
attempting to generate man pages for binaries built from a dirty
branch.
2020-11-28 18:43:55 -05:00
Luke Dashjr
a0a771843f contrib: Changes to checks for PowerPC64
Changes from #14066.
2020-11-22 11:11:32 +01:00
Wladimir J. van der Laan
634f6ec4eb contrib: Parse ELF directly for symbol and security checks
Instead of the ever-messier text parsing of the output of the readelf
tool (which is clearly meant for human consumption not to be machine
parseable), parse the ELF binaries directly.

Add a small dependency-less ELF parser specific to the checks.

This is slightly more secure, too, because it removes potential
ambiguity due to misparsing and changes in the output format of `elfread`. It
also allows for stricter and more specific ELF format checks in the future.

This removes the build-time dependency for `readelf`.

It passes the test-security-check for me locally, though I haven't
checked on all platforms.
2020-11-22 11:11:32 +01:00
Tyler Chambers
b6121edf70 swapped "is" for "==" in literal comparison
update lint-python.sh to include check F632
2020-11-09 10:21:51 -05:00
Hennadii Stepanov
bd897ce79f
scripted-diff: Move previous_release.py to test/get_previous_releases.py
-BEGIN VERIFY SCRIPT-
OLD=contrib/devtools/previous_release.py
NEW=test/get_previous_releases.py
sed -i "s|$OLD|$NEW|g" $(git grep -l $OLD)
git mv $OLD $NEW
-END VERIFY SCRIPT-
2020-08-29 11:26:25 +03:00
Wladimir J. van der Laan
4ebe2f6e75
Merge #18011: Replace current benchmarking framework with nanobench
78c312c983 Replace current benchmarking framework with nanobench (Martin Ankerl)

Pull request description:

  Replace current benchmarking framework with nanobench

  This replaces the current benchmarking framework with nanobench [1], an
  MIT licensed single-header benchmarking library, of which I am the
  autor. This has in my opinion several advantages, especially on Linux:

  * fast: Running all benchmarks takes ~6 seconds instead of 4m13s on
    an Intel i7-8700 CPU @ 3.20GHz.

  * accurate: I ran e.g. the benchmark for SipHash_32b 10 times and
    calculate standard deviation / mean = coefficient of variation:

    * 0.57% CV for old benchmarking framework
    * 0.20% CV for nanobench

    So the benchmark results with nanobench seem to vary less than with
    the old framework.

  * It automatically determines runtime based on clock precision, no need
    to specify number of evaluations.

  * measure instructions, cycles, branches, instructions per cycle,
    branch misses (only Linux, when performance counters are available)

  * output in markdown table format.

  * Warn about unstable environment (frequency scaling, turbo, ...)

  * For better profiling, it is possible to set the environment variable
    NANOBENCH_ENDLESS to force endless running of a particular benchmark
    without the need to recompile. This makes it to e.g. run "perf top"
    and look at hotspots.

  Here is an example copy & pasted from the terminal output:

  |             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |                2.52 |      396,529,415.94 |    0.6% |           25.42 |            8.02 |  3.169 |           0.06 |    0.0% |      0.03 | `bench/crypto_hash.cpp RIPEMD160`
  |                1.87 |      535,161,444.83 |    0.3% |           21.36 |            5.95 |  3.589 |           0.06 |    0.0% |      0.02 | `bench/crypto_hash.cpp SHA1`
  |                3.22 |      310,344,174.79 |    1.1% |           36.80 |           10.22 |  3.601 |           0.09 |    0.0% |      0.04 | `bench/crypto_hash.cpp SHA256`
  |                2.01 |      496,375,796.23 |    0.0% |           18.72 |            6.43 |  2.911 |           0.01 |    1.0% |      0.00 | `bench/crypto_hash.cpp SHA256D64_1024`
  |                7.23 |      138,263,519.35 |    0.1% |           82.66 |           23.11 |  3.577 |           1.63 |    0.1% |      0.00 | `bench/crypto_hash.cpp SHA256_32b`
  |                3.04 |      328,780,166.40 |    0.3% |           35.82 |            9.69 |  3.696 |           0.03 |    0.0% |      0.03 | `bench/crypto_hash.cpp SHA512`

  [1] https://github.com/martinus/nanobench

ACKs for top commit:
  laanwj:
    ACK 78c312c983

Tree-SHA512: 9e18770b18b6f95a7d0105a4a5497d31cf4eb5efe6574f4482f6f1b4c88d7e0946b9a4a1e9e8e6ecbf41a3f2d7571240677dcb45af29a6f0584e89b25f32e49e
2020-07-30 15:34:17 +02:00
Wladimir J. van der Laan
65d0f1a533
devtools: Add security check for separate_code
Check that sections are appropriately separated in virtual memory,
based on their (expected) permissions. This checks for missing
-Wl,-z,separate-code and potentially other problems.

Co-authored-by: fanquake <fanquake@gmail.com>
2020-07-28 12:57:35 +08:00
MarcoFalke
facdf530c7
contrib: Clean up previous_releases.py
* Replace curl single char options with their verbose counterpart
* Stricter check for tarballHash
2020-07-21 11:03:35 +02:00
MarcoFalke
ea595d39f7
Merge #19205: script: previous_release.sh rewritten in python
9c34aff393 Remove previous_release.sh (Brian Liotti)
e1e5960e10 script: Add previous_release.py (Brian Liotti)

Pull request description:

  Closes #18132

  Added functionality:
  1) checks file hash before untarring when using the binary download option

ACKs for top commit:
  fjahr:
    re-ACK 9c34aff393
  Sjors:
    tACK 9c34aff393

Tree-SHA512: 323f11828736a372a47f048592de8b027ddcd75b38f312dfc73f7b495d1e078bfeb384d9cdf434b3e70f2c6c0ce2da2df48e9a6460ac0e1967c6829a411c52d5
2020-07-21 10:11:39 +02:00
David Reikher
4455949d6f Make test DoS_mapOrphans deterministic
The RandomOrphan function and the function ecdsa_signature_parse_der_lax
in pubkey.cpp were causing non-deterministic test coverage.

Force seed in the beginning of the test to make it deterministic.
The seed is selected carefully so that all branches of the function
ecdsa_signature_parse_der_lax are executed. Prior to this fix, the test
was exhibiting non-deterministic coverage since none of the ECDSA
signatures that were generated during the test had leading zeroes in
either R, S, or both, resulting in some branches of said function not
being executed. The seed ensures that both conditions are hit.

Removed denialofservice_tests test entry from the list of non-deterministic
tests in the coverage script.
2020-07-21 09:18:57 +03:00
Brian Liotti
9c34aff393 Remove previous_release.sh 2020-07-05 04:05:02 -04:00
Brian Liotti
e1e5960e10 script: Add previous_release.py
closes #18132
added GPG verify for binaries

co-authored-by: bboot <bboot@cisco.com>
2020-07-05 04:04:52 -04:00
fanquake
9fe71a57a6
test: use subprocess.run() in test-security-check.py 2020-06-16 19:52:30 +08:00
fanquake
968aaae940
tests: run test-security-check.py in CI 2020-06-16 19:52:30 +08:00
Martin Ankerl
78c312c983 Replace current benchmarking framework with nanobench
This replaces the current benchmarking framework with nanobench [1], an
MIT licensed single-header benchmarking library, of which I am the
autor. This has in my opinion several advantages, especially on Linux:

* fast: Running all benchmarks takes ~6 seconds instead of 4m13s on
  an Intel i7-8700 CPU @ 3.20GHz.

* accurate: I ran e.g. the benchmark for SipHash_32b 10 times and
  calculate standard deviation / mean = coefficient of variation:

  * 0.57% CV for old benchmarking framework
  * 0.20% CV for nanobench

  So the benchmark results with nanobench seem to vary less than with
  the old framework.

* It automatically determines runtime based on clock precision, no need
  to specify number of evaluations.

* measure instructions, cycles, branches, instructions per cycle,
  branch misses (only Linux, when performance counters are available)

* output in markdown table format.

* Warn about unstable environment (frequency scaling, turbo, ...)

* For better profiling, it is possible to set the environment variable
  NANOBENCH_ENDLESS to force endless running of a particular benchmark
  without the need to recompile. This makes it to e.g. run "perf top"
  and look at hotspots.

Here is an example copy & pasted from the terminal output:

|             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|                2.52 |      396,529,415.94 |    0.6% |           25.42 |            8.02 |  3.169 |           0.06 |    0.0% |      0.03 | `bench/crypto_hash.cpp RIPEMD160`
|                1.87 |      535,161,444.83 |    0.3% |           21.36 |            5.95 |  3.589 |           0.06 |    0.0% |      0.02 | `bench/crypto_hash.cpp SHA1`
|                3.22 |      310,344,174.79 |    1.1% |           36.80 |           10.22 |  3.601 |           0.09 |    0.0% |      0.04 | `bench/crypto_hash.cpp SHA256`
|                2.01 |      496,375,796.23 |    0.0% |           18.72 |            6.43 |  2.911 |           0.01 |    1.0% |      0.00 | `bench/crypto_hash.cpp SHA256D64_1024`
|                7.23 |      138,263,519.35 |    0.1% |           82.66 |           23.11 |  3.577 |           1.63 |    0.1% |      0.00 | `bench/crypto_hash.cpp SHA256_32b`
|                3.04 |      328,780,166.40 |    0.3% |           35.82 |            9.69 |  3.696 |           0.03 |    0.0% |      0.03 | `bench/crypto_hash.cpp SHA512`

[1] https://github.com/martinus/nanobench

* Adds support for asymptotes

  This adds support to calculate asymptotic complexity of a benchmark.
  This is similar to #17375, but currently only one asymptote is
  supported, and I have added support in the benchmark `ComplexMemPool`
  as an example.

  Usage is e.g. like this:

  ```
  ./bench_bitcoin -filter=ComplexMemPool -asymptote=25,50,100,200,400,600,800
  ```

  This runs the benchmark `ComplexMemPool` several times but with
  different complexityN settings. The benchmark can extract that number
  and use it accordingly. Here, it's used for `childTxs`. The output is
  this:

  | complexityN |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |     total | benchmark
  |------------:|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|----------:|:----------
  |          25 |        1,064,241.00 |              939.64 |    1.4% |    3,960,279.00 |    2,829,708.00 |  1.400 |      0.01 | `ComplexMemPool`
  |          50 |        1,579,530.00 |              633.10 |    1.0% |    6,231,810.00 |    4,412,674.00 |  1.412 |      0.02 | `ComplexMemPool`
  |         100 |        4,022,774.00 |              248.58 |    0.6% |   16,544,406.00 |   11,889,535.00 |  1.392 |      0.04 | `ComplexMemPool`
  |         200 |       15,390,986.00 |               64.97 |    0.2% |   63,904,254.00 |   47,731,705.00 |  1.339 |      0.17 | `ComplexMemPool`
  |         400 |       69,394,711.00 |               14.41 |    0.1% |  272,602,461.00 |  219,014,691.00 |  1.245 |      0.76 | `ComplexMemPool`
  |         600 |      168,977,165.00 |                5.92 |    0.1% |  639,108,082.00 |  535,316,887.00 |  1.194 |      1.86 | `ComplexMemPool`
  |         800 |      310,109,077.00 |                3.22 |    0.1% |1,149,134,246.00 |  984,620,812.00 |  1.167 |      3.41 | `ComplexMemPool`

  |   coefficient |   err% | complexity
  |--------------:|-------:|------------
  |   4.78486e-07 |   4.5% | O(n^2)
  |   6.38557e-10 |  21.7% | O(n^3)
  |   3.42338e-05 |  38.0% | O(n log n)
  |   0.000313914 |  46.9% | O(n)
  |     0.0129823 | 114.4% | O(log n)
  |     0.0815055 | 133.8% | O(1)

  The best fitting curve is O(n^2), so the algorithm seems to scale
  quadratic with `childTxs` in the range 25 to 800.
2020-06-13 12:24:18 +02:00
MarcoFalke
fa4cd1fdae
ci: Switch to bitcoincore.org download 2020-06-07 10:30:38 -04:00
fanquake
eacedfb023
scripts: add additional type annotations to security-check.py 2020-05-14 15:30:52 +08:00
fanquake
83d063e954
scripts: add run_command to security-check.py
Deduplicate all the subprocess code as mentioned in 18713.
2020-05-14 15:29:58 +08:00
fanquake
13f606b4f9
scripts: remove NONFATAL from security-check.py 2020-05-14 14:36:27 +08:00
fanquake
061acf62a1
scripts: no-longer check for 32 bit windows in security-check.py 2020-05-14 14:36:27 +08:00
MarcoFalke
fbd522721c
Merge #18885: contrib: Move optimize-pngs.py script to the maintainer repo
fa13090d20 contrib: Remove optimize-pngs.py script, which lives in the maintainer repo (MarcoFalke)

Pull request description:

  Moved to https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/optimize-pngs.py

  Bitcoin Core should focus on the full node implementation, not on scripts to compress png images.

  This script is only used when new PNG files are added to the repo. This happens about once every two years. So fetching the script from the other repo should not be a burden, but removing it from this repo is going to cut down on the meta files we need to maintain in the main repo.

ACKs for top commit:
  practicalswift:
    ACK fa13090d20 -- `+0 lines, -82 lines` :)
  promag:
    ACK fa13090d20.
  hebasto:
    ACK fa13090d20, verified that script is already [moved](https://github.com/bitcoin-core/bitcoin-maintainer-tools/pull/56).

Tree-SHA512: 37d111adae769bcddc6ae88041032d5a2b8b228fec67f555c8333c38de3992f5138b30bea868d7d6d6b7f966a47133e5853134373b149ab23cba3b8b560ecb31
2020-05-05 10:40:39 -04:00
MarcoFalke
fa13090d20
contrib: Remove optimize-pngs.py script, which lives in the maintainer repo
https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/optimize-pngs.py
2020-05-05 07:25:58 -04:00
fanquake
3e38023af7
scripts: add PE .reloc section check to security-check.py 2020-04-23 08:40:24 +08:00
fanquake
8334ee31f8
scripts: add MACHO LAZY_BINDINGS test to test-security-check.py
I didn't add the relevant test in #18295.
2020-04-21 11:32:06 +08:00
fanquake
7b99c7454c
scripts: add MACHO Canary check to security-check.py 2020-04-21 11:32:01 +08:00
MarcoFalke
fa488f131f
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2020-04-16 13:33:09 -04:00
fanquake
5ca90f8b59
scripts: add MACHO lazy bindings check to security-check.py 2020-04-04 09:54:25 +08:00
fanquake
6ec42df32b
Merge #18426: scripts: previous_release: improve behaviour on failed download
332f373a9d [scripts] previous_release: improve failed download error message (Sebastian Falbesoner)

Pull request description:

  Currently, if the earlier release build/fetch script `previous_release.sh` is invoked with the option `-b` (intending to fetch a binary package from `https://bitcoin.org`) and the download fails, the user sees the following confusing output:
  ```
  $ contrib/devtools/previous_release.sh -r -b v0.9.5
  [...]
  gzip: stdin: not in gzip format
  tar: Child returned status 1
  tar: Error is not recoverable: exiting now
  ```
  This implies that the download worked, but the archive is corrupted, when in reality the HTML document containing the delivery fail reason (most likely 404 Not Found) is saved and tried to get unpacked. In contrast to wget, curl is a bit stubborn and needs explicit instructions to react to server errors via the flag `-f` (outputs error message and returns error code, ideal for scripts): https://curl.haxx.se/docs/manpage.html#-f

  On the PR branch, the output on failed download looks now the following:
  ```
  $ contrib/devtools/previous_release.sh -r -b v0.9.5
  [...]
  curl: (22) The requested URL returned error: 404 Not Found
  Download failed.
  ```

ACKs for top commit:
  fanquake:
    ACK 332f373a9d

Tree-SHA512: 046c931ad9e78aeb2d13faa4866d46122ed325aa142483547c2b04032d03223ed2411783b00106fcab0cd91b2f78691531ac526ed7bb3ed7547b6e2adbfb2e93
2020-04-03 18:10:28 +08:00
fanquake
7142d50ac3
scripts: rename test_64bit_PE to test_PE 2020-03-26 11:39:34 +08:00
fanquake
edaca2dd12
scripts: add MACHO NX check to security-check.py 2020-03-26 11:39:34 +08:00
fanquake
1a4e9f32ef
scripts: add MACHO tests to test-security-check.py 2020-03-26 11:39:34 +08:00
Sebastian Falbesoner
332f373a9d [scripts] previous_release: improve failed download error message
before:
------------------------------------------------------------
$ contrib/devtools/previous_release.sh -r -b v0.9.5
[...]
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
------------------------------------------------------------

now:
------------------------------------------------------------
$ contrib/devtools/previous_release.sh -r -b v0.9.5
[...]
curl: (22) The requested URL returned error: 404 Not Found
Download failed.
------------------------------------------------------------
2020-03-25 09:06:48 +01:00
fanquake
1a0993ae35
scripts: add PE dylib checking to symbol-check.py 2020-03-22 10:47:38 +08:00
Sjors Provoost
c7ca630896
[scripts] support release candidates of earlier releases 2020-02-11 21:46:22 +01:00
Sjors Provoost
ae379cf7d1
[scripts] build earlier releases 2020-02-11 19:37:37 +01:00
Wladimir J. van der Laan
3a037d0067 test: Add crc32c exception to various linters and generation scripts 2020-01-28 17:01:48 +01:00
Sebastian Falbesoner
b3c4d9bac6 test: rename test suite name "tx_validationcache_tests" to match filename
Quoting src/test/README.md, 'Adding test cases':
    "The file naming convention is `<source_filename>_tests.cpp`
     and such files should wrap their tests in a test suite
     called `<source_filename>_tests`."

Currently the unit test source file txvalidationcache_tests.cpp contains a unit
test suite with the name tx_validationcache_tests, which is fixed by this commit.
The following shell script shows that this is the only mismatch and for all other
unit test source files the test suite names are correct:

 #!/bin/bash
 shopt -s globstar
 for test_full_filename in **/*_tests.cpp; do
     test_name_file=`basename $test_full_filename .cpp`
     test_name_suite=`sed -n "s/^.*TEST_SUITE(\(.*_tests\).*$/\1/p" $test_full_filename`
     if [ $test_name_file != $test_name_suite ]; then
         echo "TestFilename: $test_name_file != TestSuitname: $test_name_suite"
     fi
 done
2020-01-27 22:44:02 +01:00
Wladimir J. van der Laan
a3d198c93c
Merge #17863: scripts: Add MACHO dylib checks to symbol-check.py
c491368d8c scripts: add MACHO dylib checking to symbol-check.py (fanquake)
76bf97213f scripts: fix check-symbols & check-security argument passing (fanquake)

Pull request description:

  Based on #17857.

  This adds dynamic library checks for MACHO executables to symbol-check.py. The script has been modified to function more like `security-check.py`. The error output is now also slightly different. i.e:
  ```bash
  # Linux x86
  bitcoin-cli: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4
  bitcoin-cli: export of symbol vtable for std::basic_ios<char, std::char_traits<char> > not allowed
  bitcoin-cli: NEEDED library libstdc++.so.6 is not allowed
  bitcoin-cli: failed IMPORTED_SYMBOLS EXPORTED_SYMBOLS LIBRARY_DEPENDENCIES

  # RISCV (skips exported symbols checks)
  bitcoin-tx: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4
  bitcoin-tx: NEEDED library libstdc++.so.6 is not allowed
  bitcoin-tx: failed IMPORTED_SYMBOLS LIBRARY_DEPENDENCIES

  # macOS
  Checking macOS dynamic libraries...
  libboost_filesystem.dylib is not in ALLOWED_LIBRARIES!
  bitcoind: failed DYNAMIC_LIBRARIES
  ```

  Compared to `v0.19.0.1` the macOS allowed dylibs has been slimmed down somewhat:
  ```diff
   src/qt/bitcoin-qt:
   /usr/lib/libSystem.B.dylib
  -/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
   /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
   /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
   /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
   /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
   /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
   /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
  -/System/Library/Frameworks/Security.framework/Versions/A/Security
  -/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
   /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
  -/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
  -/System/Library/Frameworks/AGL.framework/Versions/A/AGL
   /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
   /usr/lib/libc++.1.dylib
  -/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
   /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
   /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
   /usr/lib/libobjc.A.dylib
  ```

ACKs for top commit:
  laanwj:
    ACK c491368d8c

Tree-SHA512: f8624e4964e80b3e0d34e8d3cc33f3107938f3ef7a01c07828f09b902b5ea31a53c50f9be03576e1896ed832cf2c399e03a7943a4f537a1e1c705f3804aed979
2020-01-22 20:33:44 +01:00
MarcoFalke
95ca6aeec7
Merge #17691: doc: Add missed copyright headers
fac86ac7b3 scripted-diff: Add missed copyright headers (Hennadii Stepanov)
6fde9d5e47 script: Update EXLUDE list in copyright_header.py (Hennadii Stepanov)
1998152f15 script: Add empty line after C++ copyright (Hennadii Stepanov)
071f2fc204 script: Add ability to insert copyright to *.sh (Hennadii Stepanov)

Pull request description:

  This PR improves `contrib/devtools/copyright_header.py` script and adds copyright headers to the files in `src` and `test` directories with two exceptions:
  - [`src/reverse_iterator.h`](https://github.com/bitcoin/bitcoin/blob/master/src/reverse_iterator.h) (added to exceptions)
  - [`src/test/fuzz/FuzzedDataProvider.h`](https://github.com/bitcoin/bitcoin/blob/master/src/test/fuzz/FuzzedDataProvider.h) (added to exceptions)

  On master 5622d8f315:
  ```
  $ ./contrib/devtools/copyright_header.py report . | grep zero
    25 with zero copyrights
  ```

  With this PR:
  ```
  $ ./contrib/devtools/copyright_header.py report . | grep zero
     2 with zero copyrights
  ```

  ~I am uncertain about our copyright policy with `build_msvc` and `contrib` directories content, so they are out of scope of this PR.~

ACKs for top commit:
  MarcoFalke:
    ACK fac86ac7b3

Tree-SHA512: d7832c4a7a1a3b7806119775b40ec35d7982f49ff0e6199b8cee4c0e0a36e68d51728b6ee9924b1c161df4bc6105bd93391b79d42914357fa522f499cb113fa8
2020-01-16 15:58:35 -05:00
MarcoFalke
e09c701e01 scripted-diff: Bump copyright of files changed in 2020
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2020-01-15 02:18:00 +07:00
Hennadii Stepanov
fac86ac7b3
scripted-diff: Add missed copyright headers
-BEGIN VERIFY SCRIPT-
s() { contrib/devtools/copyright_header.py insert "$1"; }

s build_msvc/bitcoin_config.h
s build_msvc/msvc-autogen.py
s build_msvc/testconsensus/testconsensus.cpp
s contrib/devtools/circular-dependencies.py
s contrib/devtools/gen-manpages.sh
s contrib/filter-lcov.py
s contrib/gitian-build.py
s contrib/install_db4.sh
s src/crypto/sha256_avx2.cpp
s src/crypto/sha256_sse41.cpp
s src/fs.cpp
s src/qt/test/addressbooktests.cpp
s src/qt/test/addressbooktests.h
s src/qt/test/util.cpp
s src/qt/test/util.h
s src/qt/test/wallettests.cpp
s src/qt/test/wallettests.h
s src/test/blockchain_tests.cpp
s test/functional/combine_logs.py
s test/lint/lint-locale-dependence.sh
sed -i '1G' test/lint/lint-shebang.sh
s test/lint/lint-shebang.sh
-END VERIFY SCRIPT-
2020-01-04 20:18:28 +02:00
fanquake
c491368d8c
scripts: add MACHO dylib checking to symbol-check.py 2020-01-04 11:24:42 +08:00
Hennadii Stepanov
6fde9d5e47
script: Update EXLUDE list in copyright_header.py 2020-01-03 17:48:37 +02:00
Hennadii Stepanov
1998152f15
script: Add empty line after C++ copyright 2020-01-03 17:48:37 +02:00
Hennadii Stepanov
071f2fc204
script: Add ability to insert copyright to *.sh 2020-01-03 17:47:54 +02:00
fanquake
7c9e821c4e
scripts: add MACHO NOUNDEFS check to security-check.py 2020-01-02 14:42:23 +08:00
fanquake
4ca92dc6d3
scripts: add MACHO PIE check to security-check.py 2020-01-02 14:42:21 +08:00
MarcoFalke
aaaaad6ac9
scripted-diff: Bump copyright of files changed in 2019
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2019-12-30 10:42:20 +13:00
Wladimir J. van der Laan
f6e42256fe build: Allow export of environ symbols
This export was introduced in #17270 which added
```
//! Necessary on some platforms
extern char** environ;
```
2019-11-23 10:30:41 +01:00
Wladimir J. van der Laan
16a7be1663
build: Bump minimum versions in symbol checker
Debian 8 (Jessie) has:
- g++ version 4.9.2
- libc version 2.19

Ubuntu 16.04.4 (Xenial) has:
- g++ version 5.3.1
- libc version 2.23.0

CentOS 7 has:
- g++ version 4.8.5
- libc version 2.17

Taking the minimum of these as our target.
According to the GNU ABI document this corresponds to:

- GCC 4.8.5: GCC_4.8.0
- (glibc)    GLIBC_2_17

Co-Authored-By: fanquake <fanquake@gmail.com>
2019-11-22 15:40:05 -05:00
Wladimir J. van der Laan
b77d5ad59f
build: Disallow dynamic linking against c++ library
Ever since statically linking Qt, we've been linking the C++ library
statically too (-static-libstdc++). Take this into
account in the symbol checker.
2019-11-22 15:38:52 -05:00
fanquake
ec89d2882a
build: remove libanl.so.1 from ALLOWED_LIBRARIES 2019-11-20 15:05:35 -05:00
James O'Beirne
c1ccbc3dde devtools: add utxo_snapshot.sh
to allow easy (if not time-consuming) generation and verification of
snapshots.
2019-11-05 13:36:04 -05:00
gchuf
dffae5a5ad scripts: update copyright_header script to include additional files
Includes .sh and .bash-completion files in the script as well
2019-10-04 17:47:55 +02:00
Ben Woosley
3d0a82cff8
devtools: Accomodate block-style copyright blocks
Without this, `copyright_header.py report . verbose` reports:
-------------------------------------------------------------------------------
1 with unexpected copyright holder names
	./build_msvc/libsecp256k1_config.h
-------------------------------------------------------------------------------
2019-09-03 14:41:43 -04:00
Ben Woosley
b21680baf5
test/contrib: Fix invalid escapes in regex strings
Flagged by flake8 v3.6.0, as W605, plus a few others identified
incidentally, e.g. 59ffecf66cf4d08c4b431e457b083878d66a3fd6.

Note that r"\n" matches to "\n" under re.match/search.
2019-09-03 14:38:38 -04:00
fanquake
3d50fe2c1f
scripts: move update-translations.py to maintainer-tools repo 2019-08-20 11:14:00 +08:00
fanquake
6c27fa0f09
scripts: remove github-merge.py
This script has been moved to the bitcoin-core/bitcoin-maintainer-tools repository,
after discussion in a core dev meeting. The rationale being that it is also useful to
other projects, and thus should be moved to a more general repository.

Meeting log: http://www.erisian.com.au/bitcoin-core-dev/log-2019-08-15.html
See also: https://github.com/bitcoin-core/bitcoin-maintainer-tools/pull/28
2019-08-19 19:16:57 +08:00
Carl Dong
65f8da08df
symbol-check: Disallow libX11-*.so.* shared libraries
They should no longer be needed as we build QT without libX11/XLib
libraries now.
2019-07-17 17:09:48 -04:00
fanquake
1088b90cba
Merge #16327: scripts and tools: Update ShellCheck linter
1ac454a384 Enable ShellCheck rules (Hennadii Stepanov)

Pull request description:

  Enable some simple ShellCheck rules.

  Note for reviewers: `bash` and `shellcheck` on macOS are different from ones on Ubuntu.
  For local tests the latest `shellcheck` version 0.6.0 should be used (see #15166).

ACKs for top commit:
  practicalswift:
    utACK 1ac454a384
  dongcarl:
    utACK 1ac454a
  fanquake:
    ACK 1ac454a384

Tree-SHA512: 8d0a3a5c09fe1a0c22120178f5e6b80f81f746f8c3356b7701ff301c117acb2edea8fe08f08fb54ed73f94b1617515fb239fa28e7ab4121f74872e6494b6f20e
2019-07-05 09:19:23 +08:00
Hennadii Stepanov
1ac454a384
Enable ShellCheck rules
Enabled ShellCheck rules:
  SC1087
  SC2001
  SC2004
  SC2005
  SC2006
  SC2016
  SC2028
  SC2048
  SC2066 (note that IFS already contains only a line feed)
  SC2116
  SC2166
  SC2181
  SC2206
  SC2207
  SC2230
  SC2236
2019-07-04 19:35:25 +03:00
Hennadii Stepanov
ca11606858
Fix: "Bitcoin Core" -> "The Bitcoin Core" 2019-06-29 18:58:52 +03:00
Hennadii Stepanov
621463d7df
Drop no-longer-relevant copyright holder name 2019-06-29 18:50:39 +03:00
Hennadii Stepanov
01fafe55a1
Include Objective-C source files
The copyright_header.py script will process Objective-C source files 
(*.mm) as other ones.
2019-06-29 18:35:45 +03:00
MarcoFalke
e115a21f79
Merge #16223: devtools: Fetch and display ACKs at sign-off time in github-merge
0e01e4522e devtools: Fetch and display ACKs at sign-off time in github-merge (Wladimir J. van der Laan)

Pull request description:

  - Fetch the ACKs only at sign-off time. This makes sure that any last-minute ACKs are included (fixes #16200)
  - Show a list of ACKs that will be included and their author before signing off, and warn if there are none

  ![1](https://user-images.githubusercontent.com/126646/59605250-ad070980-910e-11e9-9f9a-d789c7f06ebb.png)
  ![2](https://user-images.githubusercontent.com/126646/59605255-b1332700-910e-11e9-80a5-d1e244f48264.png)

  There's a slight change to the merge commit format—before it was
  ```
      ACKs for commit 88884c:
  (list of ACKs, could be empty)
  ```
  now it is
  ```
  ACKs for top commit:
        jnewbery:
          ACK 5ebc6b0eb
      ... (list of ACKs cannot be empty)
  ```
  or
  ```
  Top commit has no ACKs.
  ```
  I don't think there's a reason to have the abbreviated commit ID there, after all the full commit id is already in the beginning of the merge commit message, and at least the abbreviated one is in every single ACK message.

ACKs for commit 0e01e4:
  fanquake:
    ACK 0e01e4522e

Tree-SHA512: 8576de016137d71cfc101747e9bb6779c13e0953cf2babee7afc9972bf2bd46f6912be4982b54fa5abf4d91e98e8fdae6b4ca3eef7d6892b7a5f04a7017b6882
2019-06-24 15:09:24 -04:00
Wladimir J. van der Laan
0e01e4522e devtools: Fetch and display ACKs at sign-off time in github-merge
- Fetch the ACKs only at sign-off time. This makes sure that any
  last-minute ACKs are included (fixes #16200)
- Show a list of ACKs and their author before signing off, and warn if
  there are none
2019-06-17 14:54:35 +02:00
fanquake
47d981e827
Merge #16186: doc/lint: Fix spelling errors identified by codespell 1.15.0
b748bf6f50 Fix spelling errors identified by codespell 1.15.0 (Ben Woosley)

Pull request description:

  Note all changes are to comments / documentation.

  After this commit, the only remaining output is:

  ```
    $ test/lint/lint-spelling.sh
    src/test/base32_tests.cpp:14: fo  ==> of, for
    src/test/base64_tests.cpp:14: fo  ==> of, for
    ^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in test/lint/lint-spelling.ignore-words.txt
  ```

  Note:
  * I ignore several valid alternative spellings ~, but changed homogenous
  to homogeneous as the latter is a more specific term according to the
  Google dictionary definitions I found~
  * homogenous is present in tinyformat, hence should be addressed upstream
  * process' is correct only if there are plural processes

ACKs for commit b748bf:
  practicalswift:
    ACK b748bf6f50
  fanquake:
    ACK b748bf6f50

Tree-SHA512: 9add7044643ce015e0a44d8b27a3f300d72c485ffff550fb6491a17f14528085289ec5caddfe02f291ea9b2cded38a0dd3079652a054e2d7fe2ff4f7b53db5d7
2019-06-16 09:57:09 +08:00
fanquake
7524376a81
Merge #16201: devtools: Always use unabbreviated commit IDs in github-merge.py
067fba563 devtools: Always use unabbreviated commit IDs in github-merge.py (Wladimir J. van der Laan)

Pull request description:

  Always put the unabbreviated commit IDs in the generated commit messages and other places. This prevents the developer's `core.abbrev` git setting from leaking through and is better against ambiguity too.

ACKs for commit 067fba:
  MarcoFalke:
    ACK 067fba5631 (replaces `h` with `H`, didn't test)
  promag:
    ACK 067fba5631, from the documentation https://git-scm.com/docs/pretty-formats:
  fanquake:
    ACK 067fba5631. Tested by merging this PR into master, then merging a second PR ontop and checking that full commit hashes were being used. Also checked documentation linked to above. Did not check that this works when a different `core.abbrev` is set locally.

Tree-SHA512: a851d10490cd8bcd8bca29094b08a6b9f883cfe1b0767ccda7ca789e4c8eff6260a4d82c33cb3d9bab01dd30ac8c9100cb7adbcb1911bb399d9385c1e1f15ecd
2019-06-14 13:38:52 +08:00
Wladimir J. van der Laan
067fba5631 devtools: Always use unabbreviated commit IDs in github-merge.py
Always put the unabbreviated commit IDs in the generated commit messages
and other places. This prevents the developer's `core.abbrev` git setting
from leaking through and is better against ambiguity too.
2019-06-13 10:21:36 +02:00