This PR adds initial support for type hints checking in python scripts.

Support for type hints was introduced in Python 3.5. Type hints make it easier to read and review code in my opinion. Also an IDE may discover a potential bug sooner. Yet, as PEP 484 says: "It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention."

Mypy is used in lint-python.sh to do the type checking. The package is standard so there is little chance that it will be abandoned. Mypy checks that type hints in source code are correct when they are not, it fails with an error.

Useful resources:

* https://docs.python.org/3.5/library/typing.html
* https://www.python.org/dev/peps/pep-0484/
This commit is contained in:
Kiminuo 2020-02-25 20:05:48 +01:00
parent 9bc7751cad
commit bd7e530f01
9 changed files with 45 additions and 33 deletions

1
.gitignore vendored
View file

@ -127,6 +127,7 @@ linux-build
win32-build win32-build
test/config.ini test/config.ini
test/cache/* test/cache/*
test/.mypy_cache/
!src/leveldb*/Makefile !src/leveldb*/Makefile

View file

@ -9,6 +9,7 @@ export LC_ALL=C
travis_retry pip3 install codespell==1.15.0 travis_retry pip3 install codespell==1.15.0
travis_retry pip3 install flake8==3.7.8 travis_retry pip3 install flake8==3.7.8
travis_retry pip3 install yq travis_retry pip3 install yq
travis_retry pip3 install mypy==0.700
SHELLCHECK_VERSION=v0.6.0 SHELLCHECK_VERSION=v0.6.0
curl -s "https://storage.googleapis.com/shellcheck/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | tar --xz -xf - --directory /tmp/ curl -s "https://storage.googleapis.com/shellcheck/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | tar --xz -xf - --directory /tmp/

View file

@ -26,10 +26,12 @@ don't have test cases for.
The Travis linter also checks this, but [possibly not in all cases](https://github.com/bitcoin/bitcoin/pull/14884#discussion_r239585126). The Travis linter also checks this, but [possibly not in all cases](https://github.com/bitcoin/bitcoin/pull/14884#discussion_r239585126).
- See [the python lint script](/test/lint/lint-python.sh) that checks for violations that - See [the python lint script](/test/lint/lint-python.sh) that checks for violations that
could lead to bugs and issues in the test code. could lead to bugs and issues in the test code.
- Use [type hints](https://docs.python.org/3/library/typing.html) in your code to improve code readability
and to detect possible bugs earlier.
- Avoid wildcard imports - Avoid wildcard imports
- Use a module-level docstring to describe what the test is testing, and how it - Use a module-level docstring to describe what the test is testing, and how it
is testing it. is testing it.
- When subclassing the BitcoinTestFramwork, place overrides for the - When subclassing the BitcoinTestFramework, place overrides for the
`set_test_params()`, `add_options()` and `setup_xxxx()` methods at the top of `set_test_params()`, `add_options()` and `setup_xxxx()` methods at the top of
the subclass, then locally-defined helper methods, then the `run_test()` method. the subclass, then locally-defined helper methods, then the `run_test()` method.
- Use `'{}'.format(x)` for string formatting, not `'%s' % x`. - Use `'{}'.format(x)` for string formatting, not `'%s' % x`.
@ -45,7 +47,7 @@ don't have test cases for.
- `rpc` for tests for individual RPC methods or features, eg `rpc_listtransactions.py` - `rpc` for tests for individual RPC methods or features, eg `rpc_listtransactions.py`
- `tool` for tests for tools, eg `tool_wallet.py` - `tool` for tests for tools, eg `tool_wallet.py`
- `wallet` for tests for wallet features, eg `wallet_keypool.py` - `wallet` for tests for wallet features, eg `wallet_keypool.py`
- use an underscore to separate words - Use an underscore to separate words
- exception: for tests for specific RPCs or command line options which don't include underscores, name the test after the exact RPC or argument name, eg `rpc_decodescript.py`, not `rpc_decode_script.py` - exception: for tests for specific RPCs or command line options which don't include underscores, name the test after the exact RPC or argument name, eg `rpc_decodescript.py`, not `rpc_decode_script.py`
- Don't use the redundant word `test` in the name, eg `interface_zmq.py`, not `interface_zmq_test.py` - Don't use the redundant word `test` in the name, eg `interface_zmq.py`, not `interface_zmq_test.py`

View file

@ -21,6 +21,7 @@ Invalid tx cases not covered here can be found by running:
""" """
import abc import abc
from typing import Optional
from test_framework.messages import ( from test_framework.messages import (
COutPoint, COutPoint,
CTransaction, CTransaction,
@ -56,7 +57,7 @@ class BadTxTemplate:
__metaclass__ = abc.ABCMeta __metaclass__ = abc.ABCMeta
# The expected error code given by bitcoind upon submission of the tx. # The expected error code given by bitcoind upon submission of the tx.
reject_reason = "" reject_reason = "" # type: Optional[str]
# Only specified if it differs from mempool acceptance error. # Only specified if it differs from mempool acceptance error.
block_reject_reason = "" block_reject_reason = ""

View file

@ -295,7 +295,7 @@ class SegWitTest(BitcoinTestFramework):
return func_wrapper return func_wrapper
@subtest @subtest # type: ignore
def test_non_witness_transaction(self): def test_non_witness_transaction(self):
"""See if sending a regular transaction works, and create a utxo to use in later tests.""" """See if sending a regular transaction works, and create a utxo to use in later tests."""
# Mine a block with an anyone-can-spend coinbase, # Mine a block with an anyone-can-spend coinbase,
@ -324,7 +324,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.append(UTXO(tx.sha256, 0, 49 * 100000000)) self.utxo.append(UTXO(tx.sha256, 0, 49 * 100000000))
self.nodes[0].generate(1) self.nodes[0].generate(1)
@subtest @subtest # type: ignore
def test_unnecessary_witness_before_segwit_activation(self): def test_unnecessary_witness_before_segwit_activation(self):
"""Verify that blocks with witnesses are rejected before activation.""" """Verify that blocks with witnesses are rejected before activation."""
@ -355,7 +355,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(tx.sha256, 0, tx.vout[0].nValue)) self.utxo.append(UTXO(tx.sha256, 0, tx.vout[0].nValue))
@subtest @subtest # type: ignore
def test_block_relay(self): def test_block_relay(self):
"""Test that block requests to NODE_WITNESS peer are with MSG_WITNESS_FLAG. """Test that block requests to NODE_WITNESS peer are with MSG_WITNESS_FLAG.
@ -451,7 +451,7 @@ class SegWitTest(BitcoinTestFramework):
self.old_node.announce_tx_and_wait_for_getdata(block4.vtx[0]) self.old_node.announce_tx_and_wait_for_getdata(block4.vtx[0])
assert block4.sha256 not in self.old_node.getdataset assert block4.sha256 not in self.old_node.getdataset
@subtest @subtest # type: ignore
def test_v0_outputs_arent_spendable(self): def test_v0_outputs_arent_spendable(self):
"""Test that v0 outputs aren't spendable before segwit activation. """Test that v0 outputs aren't spendable before segwit activation.
@ -533,7 +533,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(txid, 2, value)) self.utxo.append(UTXO(txid, 2, value))
@subtest @subtest # type: ignore
def test_getblocktemplate_before_lockin(self): def test_getblocktemplate_before_lockin(self):
txid = int(self.nodes[0].sendtoaddress(self.nodes[0].getnewaddress(), 1), 16) txid = int(self.nodes[0].sendtoaddress(self.nodes[0].getnewaddress(), 1), 16)
@ -559,7 +559,7 @@ class SegWitTest(BitcoinTestFramework):
self.nodes[0].generate(1) self.nodes[0].generate(1)
self.sync_blocks() self.sync_blocks()
@subtest @subtest # type: ignore
def test_witness_tx_relay_before_segwit_activation(self): def test_witness_tx_relay_before_segwit_activation(self):
# Generate a transaction that doesn't require a witness, but send it # Generate a transaction that doesn't require a witness, but send it
@ -601,7 +601,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(tx_hash, 0, tx_value)) self.utxo.append(UTXO(tx_hash, 0, tx_value))
@subtest @subtest # type: ignore
def test_standardness_v0(self): def test_standardness_v0(self):
"""Test V0 txout standardness. """Test V0 txout standardness.
@ -679,7 +679,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue)) self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue))
assert_equal(len(self.nodes[1].getrawmempool()), 0) assert_equal(len(self.nodes[1].getrawmempool()), 0)
@subtest @subtest # type: ignore
def advance_to_segwit_active(self): def advance_to_segwit_active(self):
"""Mine enough blocks to activate segwit.""" """Mine enough blocks to activate segwit."""
assert not softfork_active(self.nodes[0], 'segwit') assert not softfork_active(self.nodes[0], 'segwit')
@ -690,7 +690,7 @@ class SegWitTest(BitcoinTestFramework):
assert softfork_active(self.nodes[0], 'segwit') assert softfork_active(self.nodes[0], 'segwit')
self.segwit_active = True self.segwit_active = True
@subtest @subtest # type: ignore
def test_p2sh_witness(self): def test_p2sh_witness(self):
"""Test P2SH wrapped witness programs.""" """Test P2SH wrapped witness programs."""
@ -759,7 +759,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(spend_tx.sha256, 0, spend_tx.vout[0].nValue)) self.utxo.append(UTXO(spend_tx.sha256, 0, spend_tx.vout[0].nValue))
@subtest @subtest # type: ignore
def test_witness_commitments(self): def test_witness_commitments(self):
"""Test witness commitments. """Test witness commitments.
@ -849,7 +849,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue)) self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue))
@subtest @subtest # type: ignore
def test_block_malleability(self): def test_block_malleability(self):
# Make sure that a block that has too big a virtual size # Make sure that a block that has too big a virtual size
@ -889,7 +889,7 @@ class SegWitTest(BitcoinTestFramework):
block.vtx[0].wit.vtxinwit[0].scriptWitness.stack = [ser_uint256(0)] block.vtx[0].wit.vtxinwit[0].scriptWitness.stack = [ser_uint256(0)]
test_witness_block(self.nodes[0], self.test_node, block, accepted=True) test_witness_block(self.nodes[0], self.test_node, block, accepted=True)
@subtest @subtest # type: ignore
def test_witness_block_size(self): def test_witness_block_size(self):
# TODO: Test that non-witness carrying blocks can't exceed 1MB # TODO: Test that non-witness carrying blocks can't exceed 1MB
# Skipping this test for now; this is covered in p2p-fullblocktest.py # Skipping this test for now; this is covered in p2p-fullblocktest.py
@ -967,7 +967,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(block.vtx[-1].sha256, 0, block.vtx[-1].vout[0].nValue)) self.utxo.append(UTXO(block.vtx[-1].sha256, 0, block.vtx[-1].vout[0].nValue))
@subtest @subtest # type: ignore
def test_submit_block(self): def test_submit_block(self):
"""Test that submitblock adds the nonce automatically when possible.""" """Test that submitblock adds the nonce automatically when possible."""
block = self.build_next_block() block = self.build_next_block()
@ -1003,7 +1003,7 @@ class SegWitTest(BitcoinTestFramework):
# Tip should not advance! # Tip should not advance!
assert self.nodes[0].getbestblockhash() != block_2.hash assert self.nodes[0].getbestblockhash() != block_2.hash
@subtest @subtest # type: ignore
def test_extra_witness_data(self): def test_extra_witness_data(self):
"""Test extra witness data in a transaction.""" """Test extra witness data in a transaction."""
@ -1076,7 +1076,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue)) self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue))
@subtest @subtest # type: ignore
def test_max_witness_push_length(self): def test_max_witness_push_length(self):
"""Test that witness stack can only allow up to 520 byte pushes.""" """Test that witness stack can only allow up to 520 byte pushes."""
@ -1113,7 +1113,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop() self.utxo.pop()
self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue)) self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue))
@subtest @subtest # type: ignore
def test_max_witness_program_length(self): def test_max_witness_program_length(self):
"""Test that witness outputs greater than 10kB can't be spent.""" """Test that witness outputs greater than 10kB can't be spent."""
@ -1161,7 +1161,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop() self.utxo.pop()
self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue)) self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue))
@subtest @subtest # type: ignore
def test_witness_input_length(self): def test_witness_input_length(self):
"""Test that vin length must match vtxinwit length.""" """Test that vin length must match vtxinwit length."""
@ -1243,7 +1243,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop() self.utxo.pop()
self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue)) self.utxo.append(UTXO(tx2.sha256, 0, tx2.vout[0].nValue))
@subtest @subtest # type: ignore
def test_tx_relay_after_segwit_activation(self): def test_tx_relay_after_segwit_activation(self):
"""Test transaction relay after segwit activation. """Test transaction relay after segwit activation.
@ -1336,7 +1336,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue)) self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue))
@subtest @subtest # type: ignore
def test_segwit_versions(self): def test_segwit_versions(self):
"""Test validity of future segwit version transactions. """Test validity of future segwit version transactions.
@ -1418,7 +1418,7 @@ class SegWitTest(BitcoinTestFramework):
# Add utxo to our list # Add utxo to our list
self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue)) self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue))
@subtest @subtest # type: ignore
def test_premature_coinbase_witness_spend(self): def test_premature_coinbase_witness_spend(self):
block = self.build_next_block() block = self.build_next_block()
@ -1453,7 +1453,7 @@ class SegWitTest(BitcoinTestFramework):
test_witness_block(self.nodes[0], self.test_node, block2, accepted=True) test_witness_block(self.nodes[0], self.test_node, block2, accepted=True)
self.sync_blocks() self.sync_blocks()
@subtest @subtest # type: ignore
def test_uncompressed_pubkey(self): def test_uncompressed_pubkey(self):
"""Test uncompressed pubkey validity in segwit transactions. """Test uncompressed pubkey validity in segwit transactions.
@ -1558,7 +1558,7 @@ class SegWitTest(BitcoinTestFramework):
test_witness_block(self.nodes[0], self.test_node, block, accepted=True) test_witness_block(self.nodes[0], self.test_node, block, accepted=True)
self.utxo.append(UTXO(tx5.sha256, 0, tx5.vout[0].nValue)) self.utxo.append(UTXO(tx5.sha256, 0, tx5.vout[0].nValue))
@subtest @subtest # type: ignore
def test_signature_version_1(self): def test_signature_version_1(self):
key = ECKey() key = ECKey()
@ -1740,7 +1740,7 @@ class SegWitTest(BitcoinTestFramework):
for i in range(len(tx.vout)): for i in range(len(tx.vout)):
self.utxo.append(UTXO(tx.sha256, i, tx.vout[i].nValue)) self.utxo.append(UTXO(tx.sha256, i, tx.vout[i].nValue))
@subtest @subtest # type: ignore
def test_non_standard_witness_blinding(self): def test_non_standard_witness_blinding(self):
"""Test behavior of unnecessary witnesses in transactions does not blind the node for the transaction""" """Test behavior of unnecessary witnesses in transactions does not blind the node for the transaction"""
@ -1794,7 +1794,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue)) self.utxo.append(UTXO(tx3.sha256, 0, tx3.vout[0].nValue))
@subtest @subtest # type: ignore
def test_non_standard_witness(self): def test_non_standard_witness(self):
"""Test detection of non-standard P2WSH witness""" """Test detection of non-standard P2WSH witness"""
pad = chr(1).encode('latin-1') pad = chr(1).encode('latin-1')
@ -1894,7 +1894,7 @@ class SegWitTest(BitcoinTestFramework):
self.utxo.pop(0) self.utxo.pop(0)
@subtest @subtest # type: ignore
def test_upgrade_after_activation(self): def test_upgrade_after_activation(self):
"""Test the behavior of starting up a segwit-aware node after the softfork has activated.""" """Test the behavior of starting up a segwit-aware node after the softfork has activated."""
@ -1916,7 +1916,7 @@ class SegWitTest(BitcoinTestFramework):
assert_equal(self.nodes[0].getblock(block_hash), self.nodes[2].getblock(block_hash)) assert_equal(self.nodes[0].getblock(block_hash), self.nodes[2].getblock(block_hash))
height -= 1 height -= 1
@subtest @subtest # type: ignore
def test_witness_sigops(self): def test_witness_sigops(self):
"""Test sigop counting is correct inside witnesses.""" """Test sigop counting is correct inside witnesses."""

View file

@ -9,6 +9,7 @@ This file is modified from python-bitcoinlib.
import hashlib import hashlib
import struct import struct
import unittest import unittest
from typing import List, Dict
from .messages import ( from .messages import (
CTransaction, CTransaction,
@ -21,7 +22,7 @@ from .messages import (
) )
MAX_SCRIPT_ELEMENT_SIZE = 520 MAX_SCRIPT_ELEMENT_SIZE = 520
OPCODE_NAMES = {} OPCODE_NAMES = {} # type: Dict[CScriptOp, str]
def hash160(s): def hash160(s):
return hashlib.new('ripemd160', sha256(s)).digest() return hashlib.new('ripemd160', sha256(s)).digest()
@ -37,7 +38,7 @@ def bn2vch(v):
# Serialize to bytes # Serialize to bytes
return encoded_v.to_bytes(n_bytes, 'little') return encoded_v.to_bytes(n_bytes, 'little')
_opcode_instances = [] _opcode_instances = [] # type: List[CScriptOp]
class CScriptOp(int): class CScriptOp(int):
"""A single script opcode""" """A single script opcode"""
__slots__ = () __slots__ = ()

View file

@ -91,6 +91,9 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
This class also contains various public and private helper methods.""" This class also contains various public and private helper methods."""
chain = None # type: str
setup_clean_chain = None # type: bool
def __init__(self): def __init__(self):
"""Sets test framework defaults. Do not override this method. Instead, override the set_test_params() method""" """Sets test framework defaults. Do not override this method. Instead, override the set_test_params() method"""
self.chain = 'regtest' self.chain = 'regtest'
@ -407,7 +410,7 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
# Public helper methods. These can be accessed by the subclass test scripts. # Public helper methods. These can be accessed by the subclass test scripts.
def add_nodes(self, num_nodes, extra_args=None, *, rpchost=None, binary=None, binary_cli=None, versions=None): def add_nodes(self, num_nodes: int, extra_args=None, *, rpchost=None, binary=None, binary_cli=None, versions=None):
"""Instantiate TestNode objects. """Instantiate TestNode objects.
Should only be called once after the nodes have been specified in Should only be called once after the nodes have been specified in

View file

@ -42,7 +42,7 @@ except UnicodeDecodeError:
if os.name != 'nt' or sys.getwindowsversion() >= (10, 0, 14393): if os.name != 'nt' or sys.getwindowsversion() >= (10, 0, 14393):
if os.name == 'nt': if os.name == 'nt':
import ctypes import ctypes
kernel32 = ctypes.windll.kernel32 kernel32 = ctypes.windll.kernel32 # type: ignore
ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4 ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4
STD_OUTPUT_HANDLE = -11 STD_OUTPUT_HANDLE = -11
STD_ERROR_HANDLE = -12 STD_ERROR_HANDLE = -12

View file

@ -7,6 +7,7 @@
# Check for specified flake8 warnings in python files. # Check for specified flake8 warnings in python files.
export LC_ALL=C export LC_ALL=C
export MYPY_CACHE_DIR="${BASE_ROOT_DIR}/test/.mypy_cache"
enabled=( enabled=(
E101 # indentation contains mixed spaces and tabs E101 # indentation contains mixed spaces and tabs
@ -96,3 +97,5 @@ PYTHONWARNINGS="ignore" flake8 --ignore=B,C,E,F,I,N,W --select=$(IFS=","; echo "
echo "$@" echo "$@"
fi fi
) )
mypy --ignore-missing-imports $(git ls-files "test/functional/*.py")