Merge bitcoin/bitcoin#30991: test: enable running independent functional test sub-tests
Some checks are pending
CI / test each commit (push) Waiting to run
CI / macOS 14 native, arm64, no depends, sqlite only, gui (push) Waiting to run
CI / macOS 14 native, arm64, fuzz (push) Waiting to run
CI / Win64 native, VS 2022 (push) Waiting to run
CI / Win64 native fuzz, VS 2022 (push) Waiting to run
CI / ASan + LSan + UBSan + integer, no depends, USDT (push) Waiting to run

409d0d6293 test: enable running individual independent functional test methods (ismaelsadeeq)

Pull request description:

  - Some test methods in the functional test framework are independent and do not require any prior context or setup in `run_test`.
  - This commit adds a new option for running these specific methods within a test file, allowing them to be executed individually without running the entire test suite.
  - Using this option reduces the time you need to wait before the test you are interested in starts executing.
  - The functionality added by this PR can be achieved manually by commenting out code, but having a pragmatic option to do this is more convenient.

  Note: Running test methods that require arguments or context will fail.

  **Example Usage**:
  ```zsh
  build/test/functional/feature_reindex.py --test_methods continue_reindex_after_shutdown
  ```

  ```zsh
  build/test/functional/feature_config_args.py --test_methods test_log_buffer test_args_log test_connect_with_seednode
  ```

ACKs for top commit:
  maflcko:
    review ACK 409d0d6293
  rkrux:
    reACK 409d0d6293
  ryanofsky:
    Code review ACK 409d0d6293. This seems like a good step towards making it easy to run independent tests quickly. I think ideally there would be some naming convention or @ annotation added to test methods that can run independently, so the test framework could provide more functionality like being able to list test methods, being able to show command lines to quickly reproduce problems when tests fails, and calling test methods automatically instead of requiring individual tests to call them. But these ideas are all compatible with the new `--test_methods` option

Tree-SHA512: b0daac7c3b322e6fd9b946962335d8279e8cb004ff76f502c8d597b9c4b0073840945be198a79d44c5aaa64bda421429829d5c84ceeb8c6139eb6ed079a35878
This commit is contained in:
Ryan Ofsky 2024-12-02 11:01:08 -05:00
commit ebe4cac38b
No known key found for this signature in database
GPG key ID: 46800E30FC748A66

View file

@ -129,7 +129,11 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
try:
self.setup()
self.run_test()
if self.options.test_methods:
self.run_test_methods()
else:
self.run_test()
except JSONRPCException:
self.log.exception("JSONRPC error")
self.success = TestStatus.FAILED
@ -155,6 +159,13 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
exit_code = self.shutdown()
sys.exit(exit_code)
def run_test_methods(self):
for method_name in self.options.test_methods:
self.log.info(f"Attempting to execute method: {method_name}")
method = getattr(self, method_name)
method()
self.log.info(f"Method '{method_name}' executed successfully.")
def parse_args(self, test_file):
previous_releases_path = os.getenv("PREVIOUS_RELEASES_DIR") or os.getcwd() + "/releases"
parser = argparse.ArgumentParser(usage="%(prog)s [options]")
@ -194,6 +205,8 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
help="use BIP324 v2 connections between all nodes by default")
parser.add_argument("--v1transport", dest="v1transport", default=False, action="store_true",
help="Explicitly use v1 transport (can be used to overwrite global --v2transport option)")
parser.add_argument("--test_methods", dest="test_methods", nargs='*',
help="Run specified test methods sequentially instead of the full test. Use only for methods that do not depend on any context set up in run_test or other methods.")
self.add_options(parser)
# Running TestShell in a Jupyter notebook causes an additional -f argument