4fb7e0bc29
AGENTS.md has always said that a skill shipping scripts/ puts its tests in
tests/<name>/, but nothing checked it: 54 of the 100 such skills had no suite
at all, including docx, pptx, xlsx, pdf and scanpy. All 100 now do.
tests/_meta is the guard. It runs the shared structural contract across every
skill in a single process -- safe because it parses scripts with ast and never
imports them -- and fails when a skill ships scripts/ without a suite or
without a [skills.<name>] entry in skill-requirements.toml. It needs no
scientific packages and finishes in seconds, so skill-tests.yml blocks every
pull request on it, plus the packages=[] suites.
tests/_contract holds what the per-skill suites were each reimplementing:
frontmatter conformance, the 500-line limit, no tests or bytecode under
skills/, local links resolving, scripts parsing, no eval/exec/os.system, no
standard-library shadowing, no hardcoded local paths, valid shell scripts. Also
the --help contract, which skips when a skill's packages are absent and runs
for real under --isolated, and shared behaviour for the files docx/pptx/xlsx
and five schematic-shipping skills carry byte-identical copies of, with drift
detection so they cannot diverge silently.
67 new suites, 3325 test functions. The existing suites were retrofitted: 18
no longer pin an exact skill version, so a version bump no longer breaks a
test; 11 duplicated structural methods removed; 19 wired to the --help
contract; and 5 that failed collection without their packages now skip
cleanly. run_all.py in the bare project environment goes from 6 failures to 0.
Writing the tests surfaced 19 defects in the skills, fixed here with version
bumps. The ones that changed scientific output:
- openpiv reported vorticity 0 for a rotating flow, from a sign error in
openpiv's y-up coordinate relabelling; solid-body rotation now gives 2w
exactly, on grids of either orientation
- deepchem returned solubility predictions in z-scored space while labelling
them log(mol/L), because it transformed a y-less dataset instead of
untransforming the output
- neuropixels-analysis had the Allen and IBL ISI thresholds swapped,
contradicting its own references/QUALITY_METRICS.md and inverting the two
standards' relative strictness
- scanpy's summarize() raised TypeError on every AnnData under anndata 0.13,
which reports an unnamed None key on .layers; scanpy convert was broken
- experimental-design's Latin hypercube was never reproducible: pyDOE3 draws
from its own default_rng and ignores numpy's global seed
- primekg shipped a hardcoded path naming a person, which is why
no_personal_paths is now a contract rule
The remainder is upstream API drift, each verified against the installed
package: retired symbols in bioservices 1.16, gget helpers that returned lists
where a string was written, ArviZ 1.x kwargs in pymc, a positional-only
factory in pymoo, ReduceLROnPlateau(verbose=) in torch 2.13, a removed scvelo
parameter, and PyPDF2 in scientific-slides.
Two manifest environments could not build and are pinned: gget, where an
unpinned scanpy walked back to 1.9.8 and pulled llvmlite 0.36 which does not
compile on 3.13, and pymatgen, pinned to the snapshot its own _common.py
enforces rather than loosening that check. deepchem gains torch, without which
no model class exists.
python tests/run_all.py --isolated: 101 passed, 0 failed.
106 lines
3.9 KiB
Python
106 lines
3.9 KiB
Python
"""Session guard for the repo-wide test tree.
|
|
|
|
Every skill's `scripts/` directory is self-contained and owns plain top-level
|
|
module names -- 32 skills ship a `scripts/_common.py`, and names like
|
|
`cluster.py` or `validate_manifest.py` are shared too. Tests import those
|
|
scripts by putting the skill's `scripts/` directory on `sys.path`, so two
|
|
skills collected into one interpreter would resolve `_common` to whichever
|
|
skill was imported first and silently test the wrong files.
|
|
|
|
Each skill therefore gets its own process:
|
|
|
|
pytest tests/<skill> # one skill
|
|
python tests/run_all.py # every skill, one process each
|
|
|
|
This module also installs `tests/_contract/` as the importable module
|
|
`skill_contract` -- the assertions every skill shares, factored out of the
|
|
per-skill suites. See `_install_contract` for why it is loaded by file
|
|
location rather than by putting `tests/` on `sys.path`.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib.util
|
|
import os
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
# Several suites assert that a skill directory ships no .pyc files. Importing
|
|
# and subprocessing the skill's scripts is what creates them, so keep the
|
|
# interpreter -- and any child it spawns -- from writing bytecode at all.
|
|
sys.dont_write_bytecode = True
|
|
os.environ["PYTHONDONTWRITEBYTECODE"] = "1"
|
|
|
|
TESTS_DIR = Path(__file__).resolve().parent
|
|
|
|
|
|
def _install_contract() -> None:
|
|
"""Make the shared contract importable as `skill_contract`.
|
|
|
|
`tests/` must never go on `sys.path`: in prepend/append import mode that
|
|
turns every `tests/<skill>/` directory into an importable namespace
|
|
package, so `tests/simpy/`, `tests/qutip/` and `tests/neurokit2/` would
|
|
shadow the real libraries (see `addopts` in pyproject.toml). Loading the
|
|
package by file location and registering it under a name no skill uses
|
|
gives suites `import skill_contract` without that hazard.
|
|
"""
|
|
if "skill_contract" in sys.modules:
|
|
return
|
|
package = TESTS_DIR / "_contract"
|
|
spec = importlib.util.spec_from_file_location(
|
|
"skill_contract",
|
|
package / "__init__.py",
|
|
submodule_search_locations=[str(package)],
|
|
)
|
|
module = importlib.util.module_from_spec(spec)
|
|
# Registered before exec_module so the package's own `from . import cli`
|
|
# resolves against the name suites will use.
|
|
sys.modules["skill_contract"] = module
|
|
spec.loader.exec_module(module)
|
|
|
|
|
|
_install_contract()
|
|
|
|
|
|
def _skill_dirs(config: pytest.Config) -> set[str]:
|
|
"""Return the names of the skill directories this session would collect.
|
|
|
|
Underscore-prefixed directories are infrastructure, not skills:
|
|
`_contract/` is a library and `_meta/` runs the repo-wide contract without
|
|
importing any skill code, so neither participates in the one-skill-per-
|
|
process rule.
|
|
"""
|
|
everything = {
|
|
path.name
|
|
for path in TESTS_DIR.iterdir()
|
|
if path.is_dir() and not path.name.startswith((".", "_"))
|
|
}
|
|
selected: set[str] = set()
|
|
for argument in config.args:
|
|
path = Path(str(argument).split("::")[0])
|
|
if not path.is_absolute():
|
|
path = Path(config.invocation_params.dir) / path
|
|
try:
|
|
relative = path.resolve().relative_to(TESTS_DIR)
|
|
except ValueError:
|
|
continue
|
|
if relative.parts:
|
|
selected.add(relative.parts[0])
|
|
else:
|
|
selected |= everything
|
|
return selected & everything
|
|
|
|
|
|
def pytest_sessionstart(session: pytest.Session) -> None:
|
|
skills = _skill_dirs(session.config)
|
|
if len(skills) > 1:
|
|
raise pytest.UsageError(
|
|
f"cannot collect {len(skills)} skills in one process: their scripts/ "
|
|
"directories share module names (_common.py and friends), so imports "
|
|
"would resolve to the wrong skill. Run one skill at a time with "
|
|
"`pytest tests/<skill>`, or the whole tree with "
|
|
"`python tests/run_all.py`."
|
|
)
|