bitcoin/depends
fanquake ab5b26f072
Merge bitcoin/bitcoin#21995: build: Make dependency package archive timestamps deterministic
6ebe57622c build: Make dependency package archive timestamps deterministic (Hennadii Stepanov)

Pull request description:

  This PR makes testing changes like bitcoin/bitcoin#20641, bitcoin/bitcoin#21593, bitcoin/bitcoin#22142, bitcoin/bitcoin#24279, bitcoin/bitcoin#24285 as easy as comparing hashes.

  With this PR:
  ```
  $ make -C depends clean
  $ make -C depends HOST=x86_64-w64-mingw32
  $ find depends/built/x86_64-w64-mingw32 -name '*.hash' | sort | xargs cat
  1f685a61cbf205f81977ecf88cba91fa1ccdfbe77ab4ec3405dcd33ceb778af4  bdb-4.8.30-ca950bd6d13.tar.gz
  08a9acde276e6e5e5c8913e3ad07eeecda184a996882ae226b3ed056c7ec1b01  boost-1.80.0-b537c466dcb.tar.gz
  144c6d92e4108fcc90740bee27007db58a88336a97be6367f9c8ba4cc208af27  libevent-2.1.12-stable-e13b2bdd8b8.tar.gz
  e3c9c9609bf32bfd460432c6ab99a64e9f8750ed775a193925ff4f5aed363e4c  libnatpmp-07004b97cf691774efebe70404cf22201e4d330d-82255b84667.tar.gz
  62c6a089a4b24a413eccd2f389bf4c8b0716423b0ace5e87e984069635da9f83  miniupnpc-2.2.2-c43fc4cf2f6.tar.gz
  78762700066273e597698a78479a506b33532ea565d18ef561614b9fc3820cf5  qrencode-3.4.4-663de0dc628.tar.gz
  5e2183faf91838510a48e6dbb4b65ae74a7d48ba1abc070b82767c4076582360  qt-5.15.5-986926343e2.tar.gz
  9f8459f8d27fc3af9146712be6ba6577f15741429936504a950cc51c17da1ba8  sqlite-3380500-bec6a4d3299.tar.gz
  0eca5d01d427de50be4bd57c8bb100ab69b017792c32b8733e2b20443f4c9c28  zeromq-4.3.4-8ae81bab6f4.tar.gz
  ```

  As an example, here is an evidence that bitcoin/bitcoin#24279 is a strict refactoring change:
  ```
  $ git fetch origin pull/24279/head
  $ git cherry-pick 706026838d917a3d853e03e83db040f1fd4aeb74
  $ git cherry-pick 3f90ddea8a6a2061cfb347a1d77df2c0a6fa238c
  $ make -C depends clean
  $ make -C depends HOST=x86_64-w64-mingw32
  $ find depends/built/x86_64-w64-mingw32 -name '*.hash' | sort | xargs cat
  1f685a61cbf205f81977ecf88cba91fa1ccdfbe77ab4ec3405dcd33ceb778af4  bdb-4.8.30-c7faf31d5ca.tar.gz
  08a9acde276e6e5e5c8913e3ad07eeecda184a996882ae226b3ed056c7ec1b01  boost-1.80.0-1af3dd1d99e.tar.gz
  144c6d92e4108fcc90740bee27007db58a88336a97be6367f9c8ba4cc208af27  libevent-2.1.12-stable-6228a9f8534.tar.gz
  e3c9c9609bf32bfd460432c6ab99a64e9f8750ed775a193925ff4f5aed363e4c  libnatpmp-07004b97cf691774efebe70404cf22201e4d330d-41aa6194ecc.tar.gz
  62c6a089a4b24a413eccd2f389bf4c8b0716423b0ace5e87e984069635da9f83  miniupnpc-2.2.2-6a93027769c.tar.gz
  78762700066273e597698a78479a506b33532ea565d18ef561614b9fc3820cf5  qrencode-3.4.4-d40cb2d45c9.tar.gz
  5e2183faf91838510a48e6dbb4b65ae74a7d48ba1abc070b82767c4076582360  qt-5.15.5-120c3cb745d.tar.gz
  9f8459f8d27fc3af9146712be6ba6577f15741429936504a950cc51c17da1ba8  sqlite-3380500-bbd4d813c69.tar.gz
  0eca5d01d427de50be4bd57c8bb100ab69b017792c32b8733e2b20443f4c9c28  zeromq-4.3.4-df0858a19d2.tar.gz
  ```

ACKs for top commit:
  TheCharlatan:
    Code review ACK 6ebe57622c

Tree-SHA512: 20e0222781f5dcb50126c11677d0671bcdd7be144b2e528c75a02983acc494206552fb35039697ccd094de27a21b3fb439e9965c34feb8a6d74627fa20a9a5e7
2023-02-07 11:59:11 +00:00
..
builders build: Let the depends build system define a path to dsymutil tool 2022-04-21 10:08:46 +02:00
hosts Merge bitcoin/bitcoin#23619: build: Propagate user-defined flags to host packages 2023-01-30 14:32:32 +00:00
packages depends: ensure we are appending to sqlite cflags 2023-01-30 17:15:01 +00:00
patches depends: systemtap: remove variadic params that trigger compiler warnings 2023-01-23 18:12:26 +00:00
.gitignore depends: Support for S390X targets 2019-12-05 12:12:46 -05:00
config.guess depends: latest config.guess (2021-05-24) 2021-05-26 13:28:47 +08:00
config.site.in build: Do not export PKG_CONFIG_{PATH|LIBDIR} variables 2022-07-27 10:07:34 +01:00
config.sub depends: latest config.sub (2021-04-30) 2021-05-26 13:30:18 +08:00
description.md doc: Fix whitespace errs in .md files, bitcoin.conf, Info.plist.in, and find_bdb48.m4 2019-09-17 03:21:22 -04:00
funcs.mk Merge bitcoin/bitcoin#21995: build: Make dependency package archive timestamps deterministic 2023-02-07 11:59:11 +00:00
gen_id build: add *_STANDARD vars to depends gen_id 2022-06-17 14:44:49 +01:00
Makefile build: allow NO_LIBEVENT=1 in depends 2023-01-14 12:31:22 +00:00
packages.md doc: Fix typos in packages.md 2021-11-28 18:13:52 +02:00
README.md doc: add new NO_* options from #26833 2023-01-18 16:59:01 +00:00

Usage

To build dependencies for the current arch+OS:

make

To build for another arch/OS:

make HOST=host-platform-triplet

For example:

make HOST=x86_64-w64-mingw32 -j4

Bitcoin Core's configure script by default will ignore the depends output. In order for it to pick up libraries, tools, and settings from the depends build, you must set the CONFIG_SITE environment variable to point to a config.site settings file. In the above example, a file named depends/x86_64-w64-mingw32/share/config.site will be created. To use it during compilation:

CONFIG_SITE=$PWD/depends/x86_64-w64-mingw32/share/config.site ./configure

The default install prefix when using config.site is --prefix=depends/<host-platform-triplet>, so depends build outputs will be installed in that location.

Common host-platform-triplets for cross compilation are:

  • i686-pc-linux-gnu for Linux 32 bit
  • x86_64-pc-linux-gnu for x86 Linux
  • x86_64-w64-mingw32 for Win64
  • x86_64-apple-darwin for macOS
  • arm64-apple-darwin for ARM macOS
  • arm-linux-gnueabihf for Linux ARM 32 bit
  • aarch64-linux-gnu for Linux ARM 64 bit
  • powerpc64-linux-gnu for Linux POWER 64-bit (big endian)
  • powerpc64le-linux-gnu for Linux POWER 64-bit (little endian)
  • riscv32-linux-gnu for Linux RISC-V 32 bit
  • riscv64-linux-gnu for Linux RISC-V 64 bit
  • s390x-linux-gnu for Linux S390X
  • armv7a-linux-android for Android ARM 32 bit
  • aarch64-linux-android for Android ARM 64 bit
  • x86_64-linux-android for Android x86 64 bit

The paths are automatically configured and no other options are needed unless targeting Android.

Install the required dependencies: Ubuntu & Debian

For macOS cross compilation

sudo apt-get install curl bsdmainutils cmake libz-dev python3-setuptools libtinfo5 xorriso

Note: You must obtain the macOS SDK before proceeding with a cross-compile. Under the depends directory, create a subdirectory named SDKs. Then, place the extracted SDK under this new directory. For more information, see SDK Extraction.

For Win64 cross compilation

For linux (including i386, ARM) cross compilation

Common linux dependencies:

sudo apt-get install make automake cmake curl g++-multilib libtool binutils bsdmainutils pkg-config python3 patch bison

For linux ARM cross compilation:

sudo apt-get install g++-arm-linux-gnueabihf binutils-arm-linux-gnueabihf

For linux AARCH64 cross compilation:

sudo apt-get install g++-aarch64-linux-gnu binutils-aarch64-linux-gnu

For linux POWER 64-bit cross compilation (there are no packages for 32-bit):

sudo apt-get install g++-powerpc64-linux-gnu binutils-powerpc64-linux-gnu g++-powerpc64le-linux-gnu binutils-powerpc64le-linux-gnu

For linux RISC-V 64-bit cross compilation (there are no packages for 32-bit):

sudo apt-get install g++-riscv64-linux-gnu binutils-riscv64-linux-gnu

For linux S390X cross compilation:

sudo apt-get install g++-s390x-linux-gnu binutils-s390x-linux-gnu

Install the required dependencies: OpenBSD

pkg_add bash gtar

Dependency Options

The following can be set when running make: make FOO=bar

  • SOURCES_PATH: Downloaded sources will be placed here
  • BASE_CACHE: Built packages will be placed here
  • SDK_PATH: Path where SDKs can be found (used by macOS)
  • FALLBACK_DOWNLOAD_PATH: If a source file can't be fetched, try here before giving up
  • C_STANDARD: Set the C standard version used. Defaults to c11.
  • CXX_STANDARD: Set the C++ standard version used. Defaults to c++17.
  • NO_BOOST: Don't download/build/cache Boost
  • NO_LIBEVENT: Don't download/build/cache Libevent
  • NO_QT: Don't download/build/cache Qt and its dependencies
  • NO_QR: Don't download/build/cache packages needed for enabling qrencode
  • NO_ZMQ: Don't download/build/cache packages needed for enabling ZeroMQ
  • NO_WALLET: Don't download/build/cache libs needed to enable the wallet
  • NO_BDB: Don't download/build/cache BerkeleyDB
  • NO_SQLITE: Don't download/build/cache SQLite
  • NO_UPNP: Don't download/build/cache packages needed for enabling UPnP
  • NO_NATPMP: Don't download/build/cache packages needed for enabling NAT-PMP
  • ALLOW_HOST_PACKAGES: Packages that are missed in dependencies (due to NO_* option or build script logic) are searched for among the host system packages using pkg-config. It allows building with packages of other (newer) versions
  • MULTIPROCESS: Build libmultiprocess (experimental, requires CMake)
  • DEBUG: Disable some optimizations and enable more runtime checking
  • HOST_ID_SALT: Optional salt to use when generating host package ids
  • BUILD_ID_SALT: Optional salt to use when generating build package ids
  • FORCE_USE_SYSTEM_CLANG: (EXPERTS ONLY) When cross-compiling for macOS, use Clang found in the system's $PATH rather than the default prebuilt release of Clang from llvm.org. Clang 8 or later is required
  • LOG: Use file-based logging for individual packages. During a package build its log file resides in the depends directory, and the log file is printed out automatically in case of build error. After successful build log files are moved along with package archives
  • LTO: Use LTO when building packages.

If some packages are not built, for example make NO_WALLET=1, the appropriate options will be passed to bitcoin's configure. In this case, --disable-wallet.

Additional targets

download: run 'make download' to fetch all sources without building them
download-osx: run 'make download-osx' to fetch all sources needed for macOS builds
download-win: run 'make download-win' to fetch all sources needed for win builds
download-linux: run 'make download-linux' to fetch all sources needed for linux builds

Other documentation