github/scientific-agent-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Test every script-bearing skill, and enforce that it stays that way

Browse Source 
AGENTS.md has always said that a skill shipping scripts/ puts its tests in
tests/<name>/, but nothing checked it: 54 of the 100 such skills had no suite
at all, including docx, pptx, xlsx, pdf and scanpy. All 100 now do.

tests/_meta is the guard. It runs the shared structural contract across every
skill in a single process -- safe because it parses scripts with ast and never
imports them -- and fails when a skill ships scripts/ without a suite or
without a [skills.<name>] entry in skill-requirements.toml. It needs no
scientific packages and finishes in seconds, so skill-tests.yml blocks every
pull request on it, plus the packages=[] suites.

tests/_contract holds what the per-skill suites were each reimplementing:
frontmatter conformance, the 500-line limit, no tests or bytecode under
skills/, local links resolving, scripts parsing, no eval/exec/os.system, no
standard-library shadowing, no hardcoded local paths, valid shell scripts. Also
the --help contract, which skips when a skill's packages are absent and runs
for real under --isolated, and shared behaviour for the files docx/pptx/xlsx
and five schematic-shipping skills carry byte-identical copies of, with drift
detection so they cannot diverge silently.

67 new suites, 3325 test functions. The existing suites were retrofitted: 18
no longer pin an exact skill version, so a version bump no longer breaks a
test; 11 duplicated structural methods removed; 19 wired to the --help
contract; and 5 that failed collection without their packages now skip
cleanly. run_all.py in the bare project environment goes from 6 failures to 0.

Writing the tests surfaced 19 defects in the skills, fixed here with version
bumps. The ones that changed scientific output:

  - openpiv reported vorticity 0 for a rotating flow, from a sign error in
    openpiv's y-up coordinate relabelling; solid-body rotation now gives 2w
    exactly, on grids of either orientation
  - deepchem returned solubility predictions in z-scored space while labelling
    them log(mol/L), because it transformed a y-less dataset instead of
    untransforming the output
  - neuropixels-analysis had the Allen and IBL ISI thresholds swapped,
    contradicting its own references/QUALITY_METRICS.md and inverting the two
    standards' relative strictness
  - scanpy's summarize() raised TypeError on every AnnData under anndata 0.13,
    which reports an unnamed None key on .layers; scanpy convert was broken
  - experimental-design's Latin hypercube was never reproducible: pyDOE3 draws
    from its own default_rng and ignores numpy's global seed
  - primekg shipped a hardcoded path naming a person, which is why
    no_personal_paths is now a contract rule

The remainder is upstream API drift, each verified against the installed
package: retired symbols in bioservices 1.16, gget helpers that returned lists
where a string was written, ArviZ 1.x kwargs in pymc, a positional-only
factory in pymoo, ReduceLROnPlateau(verbose=) in torch 2.13, a removed scvelo
parameter, and PyPDF2 in scientific-slides.

Two manifest environments could not build and are pinned: gget, where an
unpinned scanpy walked back to 1.9.8 and pulled llvmlite 0.36 which does not
compile on 3.13, and pymatgen, pinned to the snapshot its own _common.py
enforces rather than loosening that check. deepchem gains torch, without which
no model class exists.

python tests/run_all.py --isolated: 101 passed, 0 failed.
This commit is contained in:
Timothy Kassis
2026-07-28 10:20:12 -07:00
parent d33b2671cf
commit 4fb7e0bc29
155 changed files with 24403 additions and 431 deletions
Download Patch File Download Diff File Expand all files Collapse all files

103
.github/workflows/skill-tests.yml vendored Normal file
View File

@@ -0,0 +1,103 @@
name: Skill Tests
on:
pull_request:
paths:
- "skills/**"
- "tests/**"
- "pyproject.toml"
- "uv.lock"
- ".github/workflows/skill-tests.yml"
push:
branches:
- main
paths:
- "skills/**"
- "tests/**"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: skill-tests-${{ github.ref }}
cancel-in-progress: true
jobs:
contract:
name: Repo-wide contract and coverage guard
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up uv
uses: astral-sh/setup-uv@v8.0.0
with:
enable-cache: true
cache-dependency-glob: uv.lock
python-version: "3.13"
- name: Install dependencies
run: uv sync --python 3.13
# tests/_meta checks every skill against the shared structural contract
# (frontmatter, SKILL.md length, local links, scripts parse, no shipped
# bytecode, no hardcoded local paths, ...) and enforces the repo rule that
# a skill shipping scripts/ has a suite under tests/ and an entry in
# tests/skill-requirements.toml. It imports no skill code and needs no
# scientific packages, so it runs in seconds on every pull request.
- name: Structural contract and coverage
run: uv run --python 3.13 python -m pytest tests/_meta -q
suites:
name: Standard-library-only skill suites
runs-on: ubuntu-latest
timeout-minutes: 30
needs: contract
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up uv
uses: astral-sh/setup-uv@v8.0.0
with:
enable-cache: true
cache-dependency-glob: uv.lock
python-version: "3.13"
# The skills whose bundled tooling is standard-library only -- read from
# `packages = []` in tests/skill-requirements.toml, so the list needs no
# separate maintenance. Each still gets a clean throwaway environment.
#
# The full `--isolated` sweep across every skill is deliberately NOT run
# here: it builds ~100 environments including torch, qiskit, and scanpy,
# and several skills need CUDA, a JDK, or a MATLAB install that CI does
# not have. Run it locally or on a schedule:
# python tests/run_all.py --isolated
- name: Select standard-library-only skills
id: select
run: |
set -euo pipefail
SKILLS=$(python3 - <<'PY'
import pathlib, tomllib
manifest = tomllib.loads(
pathlib.Path("tests/skill-requirements.toml").read_text()
)
names = sorted(
name
for name, entry in manifest["skills"].items()
if not entry.get("packages") and "python" not in entry
and (pathlib.Path("tests") / name).is_dir()
)
print(" ".join(names))
PY
)
echo "Selected: $SKILLS"
echo "skills=$SKILLS" >> "$GITHUB_OUTPUT"
- name: Run suites, one environment each
run: uv run --python 3.13 python tests/run_all.py --isolated ${{ steps.select.outputs.skills }}

118
AGENTS.md
View File

@@ -45,6 +45,10 @@ tests/<skill-name>/ # same name as the skill directory
└── fixtures/ # optional test data
```
**Diagrams never live under `skills/` either.** Every skill has one generated workflow diagram at
`docs/images/<skill-name>.png`, produced by `scripts/generate_skill_image.py` and kept in step with
the skill's documentation — see [Skill diagrams](#skill-diagrams).
Tests reach their skill through an explicit anchor, never a relative walk:
```python
@@ -62,6 +66,11 @@ SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "<skill-name>"
5. If the skill ships `scripts/`, put their tests in **`tests/<name>/`** — never in the skill
directory. Fixtures go in `tests/<name>/fixtures/`.
6. Validate and scan (below).
7. Generate the skill's diagram — a new skill without `docs/images/<name>.png` is incomplete:
```bash
uv run python scripts/generate_skill_image.py --skill <name>
```
```markdown
---
@@ -97,7 +106,17 @@ Use this skill when...
4. **Bump `metadata.version` in the same change**: minor for normal improvements (`"1.2"` →
`"1.3"`), major only for a breaking change or substantial redesign (`"1.9"` → `"2.0"`).
5. Re-run any example, command, or script you touched, plus `tests/<name>/` if that suite exists.
Some suites assert the skill's exact version string, so a version bump can require a test edit.
Suites check that `metadata.version` is present and quoted, not what it equals, so a version bump
never needs a matching test edit.
6. **Regenerate the diagram in the same change** whenever the edit changes what the skill does or
how its workflow runs — the picture is generated from `SKILL.md` and `references/`, so it goes
stale silently. The command overwrites `docs/images/<name>.png` in place:
```bash
uv run python scripts/generate_skill_image.py --skill <name>
```
A typo fix, a link repair, or a version bump alone does not need a new image.
## Frontmatter
@@ -222,7 +241,7 @@ If the skill has tests in `tests/<name>/`, run them:
```bash
uv run --with pytest python -m pytest tests/<name> -q
# every skill's suite, one process each
# every skill's suite, one process each, after the repo-wide guard
uv run --with pytest python tests/run_all.py
```
@@ -231,6 +250,47 @@ uv run --with pytest python tests/run_all.py
`_common` to whichever skill imported first and silently tests the wrong files. `tests/conftest.py`
refuses such a session; `tests/run_all.py` forks per skill.
### The repo-wide guard
```bash
uv run --with pytest python -m pytest tests/_meta -q
```
`tests/_meta` is the fastest useful signal in the repo: pure standard library, no scientific
packages, a couple of seconds. It runs the shared structural contract against **every** skill and
fails if a skill ships `scripts/` without a suite under `tests/<name>/` or an entry in
`tests/skill-requirements.toml`. `.github/workflows/skill-tests.yml` runs it on every pull request,
so a skill with untested scripts cannot land. A full run of `tests/run_all.py` starts with it.
It is not one of the per-skill processes because it deliberately spans all of them at once — safe
because it never imports skill code, only parses it.
### The shared contract
`tests/_contract/` holds the assertions every skill shares, so a per-skill suite contains only what
is actually specific to that skill. `tests/conftest.py` registers it as the importable module
`skill_contract`:
```python
import skill_contract
# every argparse script answers --help; skips when its packages are absent,
# runs for real under --isolated
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
# for library-style scripts with an `if __name__ == "__main__"` worked example
DemoBlockTests = skill_contract.cli.demo_test_case(SKILL_ROOT, ("doe_designs.py",))
```
- `structure` — frontmatter conformance, the 500-line limit, no tests or bytecode under `skills/`,
local links resolve, scripts parse, no `eval`/`exec`/`os.system`, no standard-library shadowing,
no hardcoded local paths, shell scripts valid. Run repo-wide by `tests/_meta`; do not duplicate
these in a per-skill suite.
- `cli` — the `--help` and demo-block cases above.
- `office` / `schematic` — behaviour for files several skills ship byte-identical copies of (the
OOXML tree under docx/pptx/xlsx; the AI schematic generator under five skills). `tests/_meta`
separately fails if those copies drift apart, so fix them together.
### One environment per skill
The project environment deliberately does not carry the skills' scientific packages. Their upstream
@@ -253,9 +313,50 @@ run on the default interpreter; uv downloads that interpreter on demand. Package
installed at all — a GitHub-only SDK, a conda-forge-only library, a CUDA build — are recorded under
`[unavailable]` with the reason, and the runner prints them so the gap shows up in test output.
Adding a skill with `scripts/` means adding its `[skills.<name>]` entry. Use `packages = []` for
skills whose bundled tooling is standard-library only; they still get a clean environment. uv caches
wheels globally, so repeat runs create each environment in milliseconds.
Adding a skill with `scripts/` means adding its `[skills.<name>]` entry — `tests/_meta` fails
without one. Use `packages = []` for skills whose bundled tooling is standard-library only; they
still get a clean environment, and CI runs exactly that set on every pull request. uv caches wheels
globally, so repeat runs create each environment in milliseconds.
The full `--isolated` sweep is not run in CI: it builds one environment per skill, several of which
need a CUDA toolchain, a JDK, or a local MATLAB install. Run it before a release, or whenever you
touch the shared contract.
## Skill diagrams
Every skill carries one generated workflow diagram at `docs/images/<skill-name>.png`. Creating a
skill means creating its image; changing what a skill does means regenerating it. The image is not
optional decoration — it is derived from the documentation, so an out-of-date one misrepresents the
skill.
`scripts/generate_skill_image.py` is local repository tooling, standard library only, and runs in
two stages on one `OPENROUTER_API_KEY` (environment variable, repository `.env`, or `--api-key`):
a text model reads `SKILL.md` plus everything under `references/` and a manifest of `scripts/` and
`assets/`, distils it into a description of one diagram, then an image model draws it. Because it
reads the whole skill, run it **after** the documentation is final, not before.
```bash
# one skill -> docs/images/<name>.png, replacing any existing image
uv run python scripts/generate_skill_image.py --skill <name>
# see which files feed the reader, and where the image lands — no API calls, nothing billed
uv run python scripts/generate_skill_image.py --skill <name> --dry-run
# read the skill and print the diagram prompt without drawing it
uv run python scripts/generate_skill_image.py --skill <name> --prompt-only
# several skills in one batch
uv run python scripts/generate_skill_image.py --skill <name-a> <name-b>
# backfill everything missing an image, six at a time
uv run python scripts/generate_skill_image.py --all --skip-existing -j 6
```
Look at the result before committing it. Image models misspell labels and occasionally point an
arrow at the wrong card; regenerate rather than ship a diagram whose text is wrong. `--quality low`
makes iteration cheap while checking composition, but commit a `high` render. Both the art direction
and the reader's instructions live at the top of the script — change them there rather than
hand-tuning one skill's prompt, so the set stays visually consistent.
## Before opening a PR
@@ -266,5 +367,12 @@ wheels globally, so repeat runs create each environment in milliseconds.
- `metadata.version` exists, is quoted, and is bumped if you changed an existing skill.
- `metadata` is a block mapping; `openclaw` / `hermes` blocks are nested mappings.
- `uv run skills-ref validate skills/<name>` passes.
- `uv run --with pytest python -m pytest tests/_meta -q` passes — this is what CI blocks on, and it
catches a missing suite, a missing `skill-requirements.toml` entry, a broken local link, and a
leaked local path.
- If the skill ships `scripts/`: a suite exists at `tests/<name>/`, a `[skills.<name>]` entry exists
in `tests/skill-requirements.toml`, and `python tests/run_all.py --isolated <name>` passes.
- `docs/images/<name>.png` exists, and was regenerated if the change altered what the skill does.
Its labels are spelled correctly and its arrows point where they should.
- Examples and scripts are tested, or clearly marked illustrative.
- No secrets or private data; scan results clean or explained in the PR.

40
CONTRIBUTING.md
View File

@@ -228,7 +228,7 @@ Good skills are specific, practical, and easy for an agent to apply.
3. Make the smallest useful change that fixes or improves the skill.
4. Increment `metadata.version`.
5. Test changed examples, commands, and scripts.
6. Run the skill's suite if it has one: `uv run --with pytest python -m pytest tests/skill-name -q`. Some suites assert the skill's exact version string, so a version bump can require a matching test edit.
6. Run the skill's suite if it has one: `uv run --with pytest python -m pytest tests/skill-name -q`. Suites check that `metadata.version` is present and quoted, not what it equals, so a version bump never needs a matching test edit.
7. Note any behavior changes in the pull request description.
## Validation
@@ -278,12 +278,42 @@ Run one skill's suite, or the whole tree:
```bash
uv run --with pytest python -m pytest tests/skill-name -q
# every skill, in a separate process each
# every skill, in a separate process each, after the repo-wide guard
uv run --with pytest python tests/run_all.py
```
Each skill's suite must run in its own process. Skills' `scripts/` directories own plain top-level module names — 32 skills ship a `scripts/_common.py`, and names like `cluster.py` and `validate_manifest.py` recur — so collecting two skills into one interpreter would resolve those imports to whichever skill was imported first and silently test the wrong files. `tests/conftest.py` rejects a multi-skill session, and `tests/run_all.py` forks per skill.
### The repo-wide guard, and what you no longer have to write
```bash
uv run --with pytest python -m pytest tests/_meta -q
```
`tests/_meta` is the check to run first and the one CI blocks on. It needs no scientific packages and finishes in seconds. It spans every skill at once — safe, because it parses scripts with `ast` and never imports them — and it enforces the rule this whole layout exists for: **a skill that ships `scripts/` must have a suite at `tests/<name>/` and a `[skills.<name>]` entry in `skill-requirements.toml`.** It also runs the shared structural contract over every skill: frontmatter conformance, the 500-line `SKILL.md` limit, no tests or compiled bytecode under `skills/`, every local link resolving, every script parsing, no `eval`/`exec`/`os.system`, no script shadowing a standard-library module, no hardcoded local path, and valid shell scripts.
Because `tests/_meta` already covers all of that repo-wide, a per-skill suite should not repeat it. Write only what is specific to the skill, and pull the shared pieces from `tests/_contract/`, which `tests/conftest.py` registers as the importable module `skill_contract`:
```python
import skill_contract
# every argparse script answers --help; skips when the skill's packages are
# absent, and runs for real under --isolated
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
# for scripts that are importable libraries with a worked example under
# `if __name__ == "__main__":` rather than argparse CLIs
DemoBlockTests = skill_contract.cli.demo_test_case(SKILL_ROOT, ("doe_designs.py",))
```
`skill_contract.office` and `skill_contract.schematic` cover files that several skills ship byte-identical copies of — the OOXML `office/` tree under `docx`/`pptx`/`xlsx`, and the AI schematic generator under five skills. Instantiate them against your skill root rather than writing the tests again; `tests/_meta` separately fails if the copies drift apart, so those files have to be changed together.
Guard heavy imports at module scope so a suite degrades to skips rather than a collection error when a package is missing:
```python
np = pytest.importorskip("numpy", reason="skill-name needs numpy")
```
### One environment per skill
Four suites fail on this repository's default environment because their scientific dependencies are not installed (`exa-search`, `qutip`, `scikit-survival`, `simpy`), and installing them all into one environment is not possible: the skills' upstream pins contradict each other. `opentrons` requires `numpy<2`; `esm` caps `transformers` below the release the `transformers` skill targets; `geniml` and `spikeinterface` pin `zarr<3` while the `zarr-python` skill targets 3.x; `bioservices` caps `lxml<6` while `matchms` requires 6.0.2+; and `pytdc`, `molfeat`, `deepchem`, `histolab`, `vaex`, and `ete3` each need an interpreter older than 3.13.
@@ -297,7 +327,9 @@ python tests/run_all.py --isolated qutip exa-search # just these
Nothing is installed into the project environment, so `uv sync` is unaffected. Each `[skills.<name>]` entry lists the packages that skill documents and, where needed, a `python` version for that skill alone — uv downloads the interpreter on demand. Packages that cannot be installed at all (a GitHub-only SDK, a conda-forge-only library, a CUDA build) are listed under `[unavailable]` with the reason, and the runner prints them so the gap appears in the test output.
A new skill that ships `scripts/` needs a `[skills.<name>]` entry. Use `packages = []` when its bundled tooling is standard-library only — the skill still gets a clean environment with just pytest. uv caches wheels globally, so repeat runs create each environment in milliseconds.
A new skill that ships `scripts/` needs a `[skills.<name>]` entry — `tests/_meta` fails without one. Use `packages = []` when its bundled tooling is standard-library only — the skill still gets a clean environment with just pytest. uv caches wheels globally, so repeat runs create each environment in milliseconds.
`.github/workflows/skill-tests.yml` runs `tests/_meta` plus every `packages = []` suite on each pull request, which is fast and needs no wheels beyond pytest. The full `--isolated` sweep is not run in CI: it builds an environment per skill, and several of them need a CUDA toolchain, a JDK, or a local MATLAB install that a runner does not have. Run it locally before a release, and whenever you change anything under `tests/_contract/`.
## Pull Request Checklist
@@ -314,6 +346,8 @@ Before submitting a pull request, confirm:
- `metadata.version` exists and is quoted.
- Existing skills have a version bump when changed.
- The `description` clearly says what the skill does and when to use it.
- `uv run --with pytest python -m pytest tests/_meta -q` passes. This is what CI blocks on, and it catches a missing suite, a missing `skill-requirements.toml` entry, a broken local link, a leaked local path, and a `SKILL.md` over 500 lines.
- If the skill ships `scripts/`: a suite exists at `tests/<skill-name>/`, a `[skills.<skill-name>]` entry exists in `tests/skill-requirements.toml`, and `python tests/run_all.py --isolated <skill-name>` passes.
- Examples and scripts have been tested or clearly marked as illustrative.
- No secrets, credentials, private data, or unsafe instructions are included.
- Relevant official documentation is linked where useful.

BIN
docs/images/paperclip.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

BIN
docs/images/pkpd-modeling.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

12
skills/bioservices/SKILL.md
View File

@@ -5,7 +5,7 @@ license: GPLv3 license
allowed-tools: Read Write Edit Bash
compatibility: Requires Python 3.9–3.12 and internet access to 40+ bioinformatics web APIs. NCBI BLAST requires a contact email (`NCBI_EMAIL` env var or explicit parameter).
metadata:
version: "1.2"
version: "1.3"
skill-author: K-Dense Inc.
openclaw:
envVars:
@@ -122,6 +122,12 @@ u = UniChem()
chembl_id = u.get_compound_id_from_kegg("C11222") # Returns CHEMBL278315
```
**Version caveat:** the per-source `get_compound_id_from_*` helpers are gone from
bioservices 1.16.0 — check `hasattr(u, "get_compound_id_from_kegg")` first, and
otherwise use the current UniChem API (`u.get_compounds(compound, source_type)`
and read `res["compounds"][0]["sources"]`). ChEMBL lookups follow the same rule:
`get_molecule`, not the pre-1.6 `get_compound_by_chemblId`.
**Common workflow:**
1. Search compound by name in KEGG
2. Extract KEGG compound ID
@@ -206,7 +212,9 @@ annotations = g.Annotation(protein="P43403", format="tsv")
### 7. Protein-Protein Interactions
Query interaction databases via PSICQUIC:
Query interaction databases via PSICQUIC. **PSICQUIC is not shipped by every
release — it is absent from 1.16.0** — so import it defensively and fall back to
`IntactComplex`, `OmniPath`, or `STRING` when it is missing:
```python
from bioservices import PSICQUIC

11
skills/bioservices/scripts/compound_cross_reference.py
View File

@@ -92,6 +92,13 @@ def get_kegg_info(kegg, kegg_id):
current_section = None
for line in entry.split("\n"):
# KEGG field names start in column 1 and their continuation lines are
# indented. A new field therefore ends any multi-line section --
# without this reset, the indented DBLINKS lines that follow PATHWAY
# would be collected as pathways.
if line and not line.startswith(" "):
current_section = None
if line.startswith("NAME"):
compound_info['name'] = line.replace("NAME", "").strip().rstrip(";")
@@ -234,7 +241,9 @@ def get_chembl_info(chembl_id):
print(f"Retrieving ChEMBL entry for {chembl_id}...")
compound = c.get_compound_by_chemblId(chembl_id)
# `get_compound_by_chemblId` was a pre-1.6 name; current releases expose
# the same lookup as `get_molecule`.
compound = c.get_molecule(chembl_id)
if compound:
print(f"\n✓ ChEMBL Information:")

14
skills/bioservices/scripts/protein_analysis_workflow.py
View File

@@ -27,7 +27,15 @@ import re
import sys
import time
import argparse
from bioservices import UniProt, KEGG, NCBIblast, PSICQUIC, QuickGO
from bioservices import UniProt, KEGG, NCBIblast, QuickGO
try:
# PSICQUIC is not shipped by every bioservices release (it is absent from
# 1.16.0). Importing it unconditionally would take the whole workflow down
# over one optional step, so degrade instead.
from bioservices import PSICQUIC
except ImportError: # pragma: no cover - depends on the installed release
PSICQUIC = None
_EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
@@ -260,6 +268,10 @@ def find_interactions(protein_query):
print("STEP 5: Protein-Protein Interactions")
print(f"{'='*70}")
if PSICQUIC is None:
print("⊘ Skipped (this bioservices release does not ship PSICQUIC)")
return []
try:
p = PSICQUIC()

2
skills/deepchem/SKILL.md
View File

@@ -5,7 +5,7 @@ license: MIT license
allowed-tools: Read Write Edit Bash
compatibility: Requires Python 3.7–3.11 (PyPI 2.8.0 caps at <3.12). Install PyTorch, TensorFlow, or JAX before the matching deepchem extra. RDKit is a core dependency.
metadata:
version: "1.2"
version: "1.4"
skill-author: K-Dense Inc.
---

10
skills/deepchem/references/core_capabilities.md
View File

@@ -268,9 +268,9 @@ new_smiles = ['CCO', 'c1ccccc1', 'CC(C)O']
new_features = featurizer.featurize(new_smiles)
new_dataset = dc.data.NumpyDataset(X=new_features)
# Apply same transformations as training
for transformer in transformers:
new_dataset = transformer.transform(new_dataset)
predictions = model.predict(new_dataset)
# Untransform the output, not the input. A NormalizationTransformer built with
# transform_y=True touches y, and a prediction dataset has no y -- transforming
# it does nothing, and the predictions come back in z-scored space. Passing the
# transformers to predict() untransforms them into the target's real units.
predictions = model.predict(new_dataset, transformers=transformers)
```

10
skills/deepchem/references/workflows.md
View File

@@ -115,11 +115,11 @@ new_featurizer = dc.feat.CircularFingerprint(radius=2, size=2048)
new_features = new_featurizer.featurize(new_smiles)
new_dataset = dc.data.NumpyDataset(X=new_features)
# Apply same transformations
for transformer in transformers:
new_dataset = transformer.transform(new_dataset)
predictions = model.predict(new_dataset)
# Untransform the output, not the input. A NormalizationTransformer built with
# transform_y=True touches y, and a prediction dataset has no y -- transforming
# it does nothing, and the predictions come back in z-scored space. Passing the
# transformers to predict() untransforms them into the target's real units.
predictions = model.predict(new_dataset, transformers=transformers)
```
---

14
skills/deepchem/scripts/graph_neural_network.py
View File

@@ -34,6 +34,18 @@ MOLNET_DATASETS = {
'lipo': ('regression', 1)
}
# MoleculeNet loader names that are not simply load_<dataset>. BACE ships as
# separate classification and regression loaders, so dc.molnet.load_bace does
# not exist; the entry above treats it as a classification benchmark.
MOLNET_LOADERS = {
'bace': 'load_bace_classification'
}
def molnet_loader(dataset_name):
"""Return the dc.molnet loader function for a MoleculeNet dataset name."""
return getattr(dc.molnet, MOLNET_LOADERS.get(dataset_name, f'load_{dataset_name}'))
def create_model(model_type, n_tasks, mode='classification'):
"""
@@ -108,7 +120,7 @@ def train_on_molnet(dataset_name, model_type, n_epochs=50):
# Load dataset with graph featurization
print(f"\nLoading {dataset_name} dataset with GraphConv featurizer...")
load_func = getattr(dc.molnet, f'load_{dataset_name}')
load_func = molnet_loader(dataset_name)
tasks, datasets, transformers = load_func(
featurizer='GraphConv',
splitter='scaffold'

13
skills/deepchem/scripts/predict_solubility.py
View File

@@ -139,13 +139,12 @@ def predict_new_molecules(model, smiles_list, transformers=None):
# Create dataset
new_dataset = dc.data.NumpyDataset(X=features)
# Apply transformers (if any)
if transformers:
for transformer in transformers:
new_dataset = transformer.transform(new_dataset)
# Predict
predictions = model.predict(new_dataset)
# Both training paths normalize with transform_y=True, so the model learned
# in z-scored target space. Hand the transformers to predict() so it
# untransforms the output back to log(mol/L) -- transforming the dataset
# instead would do nothing (these transformers touch y, not X, and a
# prediction dataset has no y) and leave the printed numbers z-scored.
predictions = model.predict(new_dataset, transformers=transformers or [])
# Display results
print("\nPredictions:")

2
skills/diffdock/SKILL.md
View File

@@ -5,7 +5,7 @@ allowed-tools: Read Write Edit Bash Glob Grep
compatibility: Requires the DiffDock repository, Python 3.9 environment from upstream environment.yml or the official Docker image, RDKit, PyTorch/PyG, and optional CUDA GPU acceleration. Current guidance targets DiffDock v1.1.3 / DiffDock-L.
license: MIT license
metadata:
version: "1.1"
version: "1.2"
skill-author: K-Dense Inc.
---

7
skills/diffdock/scripts/analyze_results.py
View File

@@ -121,8 +121,11 @@ def extract_confidence_score(sdf_file, complex_dir):
try:
with open(sdf_file) as f:
content = f.read()
# Look for confidence score in SDF properties
conf_match = re.search(r'confidence[:\s]+(-?\d+\.?\d*)', content, re.IGNORECASE)
# Look for confidence score in SDF properties. An SDF data item is
# written as `> <confidence>` followed by the value on the next
# line, so the separator class has to admit the closing angle
# bracket as well as a colon.
conf_match = re.search(r'confidence[>:\s]+(-?\d+\.?\d*)', content, re.IGNORECASE)
if conf_match:
return float(conf_match.group(1))
except Exception:

13
skills/diffdock/scripts/prepare_batch_csv.py
View File

@@ -95,8 +95,11 @@ def validate_csv(csv_path, base_dir=None):
if base_dir is None:
base_dir = Path(csv_path).parent
# Validate each row
for idx, row in df.iterrows():
# Validate each row. The per-row checks index the required columns
# directly, so they can only run once every one of them is present --
# otherwise a CSV missing a column raises KeyError instead of reporting it.
rows = [] if missing_cols else df.iterrows()
for idx, row in rows:
row_msgs = []
# Check complex name
@@ -192,8 +195,8 @@ Examples:
# Create template CSV
python prepare_batch_csv.py --create --output batch_template.csv
# Create template with 5 example rows
python prepare_batch_csv.py --create --output template.csv --num-examples 5
# Create template with 2 example rows
python prepare_batch_csv.py --create --output template.csv --num-examples 2
# Validate with custom base directory for relative paths
python prepare_batch_csv.py input.csv --validate --base-dir /path/to/data/
@@ -207,7 +210,7 @@ Examples:
help='Create a template CSV file')
parser.add_argument('--output', '-o', help='Output path for template CSV')
parser.add_argument('--num-examples', type=int, default=3,
help='Number of example rows in template (default: 3)')
help='Number of example rows in template, 1-3 (default: 3)')
parser.add_argument('--base-dir', help='Base directory for relative file paths')
args = parser.parse_args()

2
skills/experimental-design/SKILL.md
View File

@@ -5,7 +5,7 @@ allowed-tools: Read Write Edit Bash
compatibility: Requires Python >=3.10. Scripts use numpy, pandas, and pyDOE3 (DOE matrices). Install with uv as shown below.
license: MIT license
metadata:
version: "1.0"
version: "1.1"
skill-author: K-Dense Inc.
---

8
skills/experimental-design/scripts/doe_designs.py
View File

@@ -148,10 +148,12 @@ def latin_hypercube(factors, n_samples, criterion="maximin", seed=0,
'center'/'centermaximin'/'correlation' are alternatives.
"""
from pyDOE3 import lhs
rng_state = int(seed) # pyDOE3 lhs uses numpy global RNG; seed it for repeatability
np.random.seed(rng_state)
# pyDOE3 draws from its own default_rng, so seeding numpy's global RNG has
# no effect -- the seed has to be handed to lhs itself.
names = list(factors)
unit = lhs(len(names), samples=n_samples, criterion=criterion) # in [0,1]
unit = lhs( # in [0,1]
len(names), samples=n_samples, criterion=criterion, seed=int(seed)
)
out = {}
for j, n in enumerate(names):
low, high = factors[n][0], factors[n][1]

2
skills/gget/SKILL.md
View File

@@ -5,7 +5,7 @@ license: BSD-2-Clause license
allowed-tools: Read Write Edit Bash
compatibility: Requires Python >=3.8 and gget 0.30.5-compatible APIs. Optional setup modules may install scientific dependencies that lag the newest Python releases; use Python 3.9 or 3.10 if `gget setup cellxgene` or `gget setup alphafold` fails.
metadata:
version: "1.2"
version: "1.3"
skill-author: K-Dense Inc.
---

25
skills/gget/references/workflows.md
View File

@@ -39,11 +39,12 @@ print("\nStep 3: Retrieving sequences...")
nucleotide_seqs = gget.seq(gene_ids)
protein_seqs = gget.seq(gene_ids, translate=True)
# Save sequences
# Save sequences. gget.seq returns a list of FASTA lines, so join it first --
# f.write(list) raises TypeError.
with open("gaba_receptors_nt.fasta", "w") as f:
f.write(nucleotide_seqs)
f.write("\n".join(nucleotide_seqs) + "\n")
with open("gaba_receptors_aa.fasta", "w") as f:
f.write(protein_seqs)
f.write("\n".join(protein_seqs) + "\n")
# Step 4: Get expression data
print("\nStep 4: Getting tissue expression...")
@@ -108,22 +109,20 @@ print("\n2. Retrieving protein sequences...")
human_seq = gget.seq(human_gene, translate=True)
mouse_seq = gget.seq(mouse_gene, translate=True)
# Save to file for alignment
# Save to file for alignment (gget.seq returns a list of FASTA lines)
with open("pcsk9_sequences.fasta", "w") as f:
f.write(human_seq)
f.write("\n")
f.write(mouse_seq)
f.write("\n".join(human_seq) + "\n")
f.write("\n".join(mouse_seq) + "\n")
# Step 3: Align sequences
# Step 3: Align sequences. gget.muscle returns None -- it writes to `out`, or
# prints the ClustalW alignment when `out` is omitted.
print("\n3. Aligning sequences...")
alignment = gget.muscle("pcsk9_sequences.fasta")
print("Alignment completed. Visualizing in ClustalW format:")
print(alignment)
gget.muscle("pcsk9_sequences.fasta", out="pcsk9_aligned.afa")
# Step 4: Get existing structures from PDB
print("\n4. Searching PDB for existing structures...")
# Search by sequence using BLAST
pdb_results = gget.blast(human_seq, database="pdbaa", limit=5)
# Search by sequence using BLAST (the amino-acid line, not the FASTA header)
pdb_results = gget.blast(human_seq[1], database="pdbaa", limit=5)
print("Top PDB matches:")
print(pdb_results[["Description", "Max Score", "Query Coverage"]])

7
skills/gget/scripts/batch_sequence_analysis.py
View File

@@ -92,10 +92,11 @@ def analyze_sequences(
print("\n\nStep 2: Multiple sequence alignment...")
print("-" * 60)
try:
alignment = gget.muscle(fasta_file)
# gget.muscle returns None: it writes the alignment to `out` (or
# prints it when `out` is omitted), so the file must be requested
# rather than written from the return value.
alignment_file = output_path / "alignment.afa"
with open(alignment_file, "w") as f:
f.write(alignment)
gget.muscle(fasta_file, out=str(alignment_file))
print(f"Alignment saved to: {alignment_file}")
except Exception as e:
print(f"Error in alignment: {e}")

18
skills/gget/scripts/gene_analysis.py
View File

@@ -9,6 +9,20 @@ import sys
import gget
def fasta_text(sequences):
"""Render what gget.seq returned as FASTA text.
gget.seq returns a list of FASTA lines (header, sequence, ...); older
releases returned one already-joined string. Writing the list straight to a
file raises TypeError, so normalise both shapes here.
"""
if sequences is None:
return ""
if isinstance(sequences, str):
return sequences if sequences.endswith("\n") else sequences + "\n"
return "\n".join(sequences) + "\n"
def analyze_gene(gene_name, species="homo_sapiens", output_prefix=None):
"""
Perform comprehensive analysis of a gene.
@@ -53,11 +67,11 @@ def analyze_gene(gene_name, species="homo_sapiens", output_prefix=None):
protein_seq = gget.seq([gene_id], translate=True)
with open(f"{output_prefix}_nucleotide.fasta", "w") as f:
f.write(nucleotide_seq)
f.write(fasta_text(nucleotide_seq))
print(f" Nucleotide sequence saved to: {output_prefix}_nucleotide.fasta")
with open(f"{output_prefix}_protein.fasta", "w") as f:
f.write(protein_seq)
f.write(fasta_text(protein_seq))
print(f" Protein sequence saved to: {output_prefix}_protein.fasta")
# Step 4: Get tissue expression

2
skills/neuropixels-analysis/SKILL.md
View File

@@ -3,7 +3,7 @@ name: neuropixels-analysis
description: Analyze Neuropixels extracellular recordings end-to-end with SpikeInterface. Covers loading SpikeGLX/Open Ephys/NWB data, preprocessing, drift/motion correction, Kilosort4 (and CPU) spike sorting, quality metrics, and unit curation (threshold-based, model-based UnitRefine, and AI-assisted visual review). Use when working with Neuropixels 1.0/2.0 recordings, spike sorting, or extracellular electrophysiology analysis.
license: MIT license
metadata:
version: "2.1"
version: "2.2"
skill-author: K-Dense Inc.
openclaw:
primaryEnv: ANTHROPIC_API_KEY

14
skills/neuropixels-analysis/scripts/compute_metrics.py
View File

@@ -14,19 +14,23 @@ import pandas as pd
import spikeinterface.full as si
# Curation criteria presets
# Curation criteria presets. snr and presence_ratio are minima, the other two are
# maxima, and each preset is at least as strict as the one above it. The ISI,
# presence, and amplitude thresholds follow references/QUALITY_METRICS.md:
# Allen Visual Coding uses isi_violations_ratio < 0.5, IBL's reproducible-ephys
# criteria tighten that to < 0.1, and the strict single-unit set to < 0.01.
CURATION_CRITERIA = {
'allen': {
'snr': 3.0,
'isi_violations_ratio': 0.1,
'isi_violations_ratio': 0.5,
'presence_ratio': 0.9,
'amplitude_cutoff': 0.1,
},
'ibl': {
'snr': 4.0,
'isi_violations_ratio': 0.5,
'presence_ratio': 0.5,
'amplitude_cutoff': None,
'isi_violations_ratio': 0.1,
'presence_ratio': 0.9,
'amplitude_cutoff': 0.1,
},
'strict': {
'snr': 5.0,

8
skills/neuropixels-analysis/scripts/neuropixels_pipeline.py
View File

@@ -244,6 +244,14 @@ def curate_units(qm, method: str = 'allen') -> dict:
'ibl': IBL standards
'strict': Strict single-unit criteria
"""
methods = ('allen', 'ibl', 'strict')
if method not in methods:
# Without this, an unrecognised method leaves every non-noise unit out of
# `labels` entirely, so export_results() silently reports zero good units.
raise ValueError(
f"unknown curation method {method!r}; choose one of {', '.join(methods)}"
)
print(f"Curating units (method: {method})...")
labels = {}

8
skills/openpiv/SKILL.md
View File

@@ -5,7 +5,7 @@ license: BSD-3-Clause
compatibility: Requires Python 3.10+ with openpiv installed (uv pip install openpiv). numpy, scipy, scikit-image, and matplotlib arrive as dependencies. No network access needed after install.
allowed-tools: Read Write Edit Bash
metadata:
version: "1.0"
version: "1.1"
skill-author: OpenPIV Team
tested-against: "openpiv 0.25.4"
---
@@ -333,6 +333,12 @@ def compute_vorticity(u, v, dx=1.0, dy=None):
The grid spacing is `(window_size - overlap) / scaling_factor` in physical units, so leaving `dx=1.0`
yields vorticity per grid cell, not per unit length.
**Sign convention:** `runner.py` ends with `transform_coordinates`, which relabels the grid into a
right-handed y-up frame but leaves the rows in image order, so the saved `y` *decreases* as the row
index grows. The standalone forms above assume the opposite, so on a `params.npz` field they return
`-du/dy` and flip the sign of the vorticity and the shear strain — negate the `axis=0` derivatives, or
use `PIVAnalyzer`, which reads the orientation off the saved coordinates.
### Strain Rate
```python

55
skills/openpiv/scripts/analyze.py
View File

@@ -36,6 +36,34 @@ class PIVAnalyzer:
dy = float(np.abs(np.diff(self.y, axis=0)).mean()) if self.y.shape[0] > 1 else 1.0
return dx, dy
@property
def axis_signs(self) -> Tuple[float, float]:
"""(+-1, +-1): whether x and y increase or decrease along the array axes.
runner.py finishes with openpiv.tools.transform_coordinates(), which relabels
the grid into a physical right-handed (y-up) frame while leaving the rows in
image order -- so physical y *decreases* as the row index grows. np.gradient
only sees the array, so differentiating with a positive spacing would return
-du/dy there and silently flip the sign of the vorticity and of the shear
strain rate. These signs put the derivatives back on the physical axes.
"""
x_sign = -1.0 if self.x.shape[1] > 1 and self.x[0, 1] < self.x[0, 0] else 1.0
y_sign = -1.0 if self.y.shape[0] > 1 and self.y[1, 0] < self.y[0, 0] else 1.0
return x_sign, y_sign
def _steps(
self, dx: Optional[float], dy: Optional[float]
) -> Tuple[float, float, float, float]:
"""(dx, dy, x_sign, y_sign) for a gradient call, defaulting to the grid."""
gx, gy = self.grid_spacing
x_sign, y_sign = self.axis_signs
return (
gx if dx is None else abs(float(dx)),
gy if dy is None else abs(float(dy)),
x_sign,
y_sign,
)
def plot_vector_field(
self,
scale: int = 50,
@@ -72,23 +100,26 @@ class PIVAnalyzer:
def compute_vorticity(
self, dx: Optional[float] = None, dy: Optional[float] = None
) -> np.ndarray:
"""Out-of-plane vorticity, dv/dx - du/dy. Defaults to the inferred grid spacing."""
gx, gy = self.grid_spacing
dx = gx if dx is None else dx
dy = gy if dy is None else dy
return np.gradient(self.v, dx, axis=1) - np.gradient(self.u, dy, axis=0)
"""Out-of-plane vorticity, dv/dx - du/dy. Defaults to the inferred grid spacing.
dx and dy are spacing magnitudes; the axis orientation comes from the saved
coordinates (see axis_signs), so a counter-clockwise flow gives positive
vorticity whichever way the rows run.
"""
dx, dy, x_sign, y_sign = self._steps(dx, dy)
dv_dx = x_sign * np.gradient(self.v, dx, axis=1)
du_dy = y_sign * np.gradient(self.u, dy, axis=0)
return dv_dx - du_dy
def compute_strain(
self, dx: Optional[float] = None, dy: Optional[float] = None
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""Return (exx, eyy, exy) of the 2D strain-rate tensor."""
gx, gy = self.grid_spacing
dx = gx if dx is None else dx
dy = gy if dy is None else dy
du_dx = np.gradient(self.u, dx, axis=1)
du_dy = np.gradient(self.u, dy, axis=0)
dv_dx = np.gradient(self.v, dx, axis=1)
dv_dy = np.gradient(self.v, dy, axis=0)
dx, dy, x_sign, y_sign = self._steps(dx, dy)
du_dx = x_sign * np.gradient(self.u, dx, axis=1)
du_dy = y_sign * np.gradient(self.u, dy, axis=0)
dv_dx = x_sign * np.gradient(self.v, dx, axis=1)
dv_dy = y_sign * np.gradient(self.v, dy, axis=0)
return du_dx, dv_dy, 0.5 * (du_dy + dv_dx)
def compute_statistics(self) -> Dict[str, float]:

6
skills/phylogenetics/SKILL.md
View File

@@ -3,7 +3,7 @@ name: phylogenetics
description: Build and analyze phylogenetic trees using MAFFT (multiple alignment), IQ-TREE 2 (maximum likelihood), and FastTree (fast NJ/ML). Visualize with ETE3 or FigTree. For evolutionary analysis, microbial genomics, viral phylodynamics, protein family analysis, and molecular clock studies.
license: Unknown
metadata:
version: "1.0"
version: "1.1"
skill-author: Kuan-lin Huang
---
@@ -23,6 +23,10 @@ Phylogenetic analysis reconstructs the evolutionary history of biological sequen
# Conda (recommended for CLI tools)
conda install -c bioconda mafft iqtree fasttree
pip install ete3
# ete3's TreeStyle/NodeStyle rendering lives in its Qt backend, so image output
# needs PyQt5 as well; tree parsing and statistics work without it.
pip install PyQt5
```
## When to Use This Skill

8
skills/phylogenetics/scripts/phylogenetic_analysis.py
View File

@@ -138,9 +138,11 @@ def visualize_tree(tree_file: str, output_png: str, outgroup: str = None) -> Non
"""Visualize the phylogenetic tree with ETE3."""
try:
from ete3 import Tree, TreeStyle, NodeStyle
except ImportError:
print("ETE3 not installed. Skipping visualization.")
print(" Install: pip install ete3")
except ImportError as exc:
# TreeStyle and NodeStyle live in ete3's Qt-backed treeview module, so
# this also fires when ete3 itself imported fine but PyQt5 is missing.
print(f"ETE3 rendering unavailable ({exc}). Skipping visualization.")
print(" Install: pip install ete3 PyQt5")
return
t = Tree(tree_file)

5
skills/primekg/SKILL.md
View File

@@ -3,7 +3,7 @@ name: primekg
description: Query the Precision Medicine Knowledge Graph (PrimeKG) for multiscale biological data including genes, drugs, diseases, phenotypes, and more.
license: Unknown
metadata:
version: "1.0"
version: "1.1"
skill-author: K-Dense Inc. (PrimeKG original from Harvard MIMS)
---
@@ -92,7 +92,8 @@ The graph contains several key relationship types including:
- `scripts/query_primekg.py`: Core functions for searching and querying the knowledge graph.
### Data Path
- Data: `/mnt/c/Users/eamon/Documents/Data/PrimeKG/kg.csv`
- Data: `kg.csv`, downloaded from the [PrimeKG Harvard Dataverse](https://dataverse.harvard.edu/dataverse/primekg).
- Point the scripts at it with `export PRIMEKG_DATA=/path/to/kg.csv` (default: `data/PrimeKG/kg.csv`).
- Total nodes: ~129,000
- Total edges: ~4,000,000
- Database: CSV-based, optimized for pandas querying.

11
skills/primekg/scripts/query_primekg.py
View File

@@ -3,13 +3,18 @@ import os
import json
from typing import List, Dict, Optional, Union
# Default data path
DATA_PATH = "/mnt/c/Users/eamon/Documents/Data/PrimeKG/kg.csv"
# Where kg.csv lives. Override with the PRIMEKG_DATA environment variable, or
# by assigning to DATA_PATH before calling any query function.
DATA_PATH = os.environ.get("PRIMEKG_DATA", "data/PrimeKG/kg.csv")
def _load_kg():
"""Internal helper to load the KG efficiently."""
if not os.path.exists(DATA_PATH):
raise FileNotFoundError(f"PrimeKG data not found at {DATA_PATH}. Please ensure the file is downloaded.")
raise FileNotFoundError(
f"PrimeKG data not found at {DATA_PATH}. Download kg.csv from "
"https://dataverse.harvard.edu/dataverse/primekg and set "
"PRIMEKG_DATA to its path."
)
# For very large files, we might want to use a database or specialized graph library.
# For now, we'll use pandas for simplicity but with low_memory=True.
return pd.read_csv(DATA_PATH, low_memory=True)

2
skills/pydeseq2/SKILL.md
View File

@@ -5,7 +5,7 @@ allowed-tools: Read Write Edit Bash
compatibility: Requires Python >=3.11 and PyDESeq2 0.5.4-compatible dependencies. Examples target PyDESeq2 0.5.x, formulaic design strings, explicit contrasts, and uv-based installs.
license: MIT license
metadata:
version: "1.2"
version: "1.3"
skill-author: K-Dense Inc.
---

6
skills/pydeseq2/scripts/run_deseq2_analysis.py
View File

@@ -47,8 +47,10 @@ def load_and_validate_data(counts_path, metadata_path, transpose_counts=True):
print(f" Counts shape: {counts_df.shape} (samples × genes)")
print(f" Metadata shape: {metadata.shape} (samples × variables)")
# Validate
if not all(counts_df.index == metadata.index):
# Validate. `Index.equals` also covers a length mismatch, which element-wise
# comparison cannot: `index_a == index_b` raises when the two differ in
# length, so the intersection fallback below would never be reached.
if not counts_df.index.equals(metadata.index):
print("\nWarning: Sample indices don't match perfectly. Taking intersection...")
common_samples = counts_df.index.intersection(metadata.index)
counts_df = counts_df.loc[common_samples]

6
skills/pymc/SKILL.md
View File

@@ -5,7 +5,7 @@ allowed-tools: Read Write Edit Bash
compatibility: Requires Python 3.12+ and PyMC 6.0.1-compatible dependencies. Install reproducible environments with `uv pip install "pymc[nutpie]==6.0.1"`; optional NumPyro or BlackJAX samplers require separately pinned JAX-compatible dependencies.
license: Apache License, Version 2.0
metadata:
version: "1.2"
version: "1.3"
skill-author: K-Dense Inc.
---
@@ -160,7 +160,7 @@ Creates:
- Rank plots (mixing check)
- Autocorrelation plots
- Energy plots
- ESS evolution
- Local ESS plots
- Summary statistics CSV
### Quick Diagnostic Check
@@ -258,7 +258,7 @@ This skill includes:
- **`model_diagnostics.py`**: Automated diagnostic checking and report generation. Functions: `check_diagnostics()` for quick checks, `create_diagnostic_report()` for comprehensive analysis with plots.
- **`model_comparison.py`**: Model comparison utilities using LOO/WAIC. Functions: `compare_models()`, `check_loo_reliability()`, `model_averaging()`.
- **`model_comparison.py`**: Model comparison utilities built on PSIS-LOO ELPD, the only criterion ArviZ 1.x `compare()` ranks on. Functions: `compare_models()`, `check_loo_reliability()`, `model_averaging()`.
### Templates (`assets/`)

10
skills/pymc/references/model_patterns.md
View File

@@ -109,11 +109,11 @@ comparison = compare_models(models, ic='loo')
check_loo_reliability(models)
```
**Interpretation:**
- **Δloo < 2**: Models are similar, choose simpler model
- **2 < Δloo < 4**: Weak evidence for better model
- **4 < Δloo < 10**: Moderate evidence
- **Δloo > 10**: Strong evidence for better model
**Interpretation** — ArviZ 1.x reports `elpd_diff` on the ELPD scale (higher is
better, so the best model's `elpd_diff` is 0 and the others are negative):
- **|elpd_diff| < 4**: Models are similar, choose the simpler model
- **|elpd_diff| > 4 but within 2 `dse`**: Moderate evidence for the better model
- **|elpd_diff| > 4 and beyond 2 `dse`**: Strong evidence for the better model
**Check Pareto-k values:**
- k < 0.7: LOO reliable

118
skills/pymc/scripts/model_comparison.py
View File

@@ -24,12 +24,16 @@ import matplotlib.pyplot as plt
from typing import Any, Dict
#: ArviZ 1.x compares models on PSIS-LOO ELPD only; there is no `ic=` switch and
#: no deviance scale. WAIC is still available on its own via `az.waic()`.
SUPPORTED_IC = ('loo', 'elpd')
def compare_models(models_dict: Dict[str, Any],
ic='loo',
scale='deviance',
verbose=True):
"""
Compare multiple models using information criteria.
Compare multiple models by expected log pointwise predictive density.
Parameters
----------
@@ -37,29 +41,38 @@ def compare_models(models_dict: Dict[str, Any],
Dictionary mapping model names to PyMC posterior objects.
All models must have log_likelihood computed.
ic : str
Information criterion to use: 'loo' (default) or 'waic'
scale : str
Scale for IC: 'deviance' (default), 'log', or 'negative_log'
Information criterion. Only 'loo' (equivalently 'elpd') is supported:
ArviZ 1.x ranks models on PSIS-LOO ELPD.
verbose : bool
Print detailed comparison results (default: True)
Returns
-------
pd.DataFrame
Comparison DataFrame with model rankings and statistics
Comparison DataFrame with model rankings and statistics, on the ELPD
scale (higher is better, so `elpd_diff` is 0 for the best model and
negative for the others).
Notes
-----
Models must be fit with idata_kwargs={'log_likelihood': True} or
log-likelihood computed afterwards with pm.compute_log_likelihood().
Models must have a log_likelihood group, computed during sampling or
afterwards with pm.compute_log_likelihood(idata).
"""
if ic.lower() not in SUPPORTED_IC:
raise ValueError(
f"unknown information criterion {ic!r}: ArviZ 1.x ranks models on "
"PSIS-LOO ELPD, so pass ic='loo'. For WAIC, call az.waic() per "
"model directly."
)
if verbose:
print("="*70)
print(f" " * 25 + f"MODEL COMPARISON ({ic.upper()})")
print(" " * 25 + "MODEL COMPARISON (LOO)")
print("="*70)
# Perform comparison
comparison = az.compare(models_dict, ic=ic, scale=scale)
# round_to='none' keeps the columns numeric; the default formats them for
# display, which turns every comparison below into a string comparison.
comparison = az.compare(models_dict, round_to='none')
if verbose:
print("\nModel Rankings:")
@@ -69,15 +82,15 @@ def compare_models(models_dict: Dict[str, Any],
print("\n" + "="*70)
print("INTERPRETATION GUIDE")
print("="*70)
print(f"• rank: Model ranking (0 = best)")
print(f"• {ic}: {ic.upper()} estimate (lower is better)")
print(f"• p_{ic}: Effective number of parameters")
print(f"• d{ic}: Difference from best model")
print(f"• weight: Model probability (pseudo-BMA)")
print(f"• se: Standard error of {ic.upper()}")
print(f"• dse: Standard error of the difference")
print(f"• warning: True if model has reliability issues")
print(f"• scale: {scale}")
print("• rank: Model ranking (0 = best)")
print("• elpd: PSIS-LOO ELPD estimate (higher is better)")
print("• p: Effective number of parameters")
print("• elpd_diff: ELPD minus the best model's ELPD (0 for the best)")
print("• weight: Model probability (stacking weights)")
print("• se: Standard error of the ELPD estimate")
print("• dse: Standard error of the difference")
print("• p_worse: Probability the model is worse than the best one")
print("• diag_elpd: Reliability diagnostic for the ELPD estimate")
print("\n" + "="*70)
print("MODEL SELECTION GUIDELINES")
@@ -86,33 +99,38 @@ def compare_models(models_dict: Dict[str, Any],
best_model = comparison.index[0]
print(f"\n✓ Best model: {best_model}")
# Check for clear winner
# Check for a clear winner. Vehtari et al. recommend treating an ELPD
# difference below 4 as small, and otherwise judging it against the
# standard error of the difference.
if len(comparison) > 1:
delta = comparison.iloc[1][f'd{ic}']
delta = abs(comparison.iloc[1]['elpd_diff'])
delta_se = comparison.iloc[1]['dse']
if delta > 10:
print(f" → STRONG evidence for {best_model} (Δ{ic} > 10)")
elif delta > 4:
print(f" → MODERATE evidence for {best_model} (4 < Δ{ic} < 10)")
elif delta > 2:
print(f" → WEAK evidence for {best_model} (2 < Δ{ic} < 4)")
if delta < 4:
print(f" → Models are SIMILAR (ELPD difference {delta:.1f} < 4)")
print(" Consider model averaging or choose based on simplicity")
elif delta > 2 * delta_se:
print(
f" → STRONG evidence for {best_model} "
f"(ELPD difference {delta:.1f} > 2 SE)"
)
else:
print(f" → Models are SIMILAR (Δ{ic} < 2)")
print(f" Consider model averaging or choose based on simplicity")
print(
f" → MODERATE evidence for {best_model} "
f"(ELPD difference {delta:.1f}, within 2 SE)"
)
# Check if difference is significant relative to SE
if delta > 2 * delta_se:
print(f" → Difference is > 2 SE, likely reliable")
else:
print(f" → Difference is < 2 SE, uncertain distinction")
# Check for warnings
if comparison['warning'].any():
# Reliability. ArviZ 1.x reports this per row as a diagnostic string
# instead of the old boolean `warning` column.
flagged = [
name
for name, diagnostic in comparison['diag_elpd'].items()
if isinstance(diagnostic, str) and diagnostic.strip() not in ('', 'ok')
]
if flagged:
print("\n⚠️ WARNING: Some models have reliability issues")
warned_models = comparison[comparison['warning']].index.tolist()
print(f" Models with warnings: {', '.join(warned_models)}")
print(f" → Check Pareto-k diagnostics with check_loo_reliability()")
print(f" Models with warnings: {', '.join(flagged)}")
print(" → Check Pareto-k diagnostics with check_loo_reliability()")
return comparison
@@ -210,19 +228,21 @@ def plot_model_comparison(comparison, output_path=None, show=True):
matplotlib.figure.Figure
The comparison figure
"""
fig = plt.figure(figsize=(10, 6))
az.plot_compare(comparison)
plt.title('Model Comparison', fontsize=14, fontweight='bold')
plt.tight_layout()
# ArviZ 1.x returns a PlotCollection and does not draw into pyplot's
# current figure, so the figure has to come back out of the collection --
# plt.savefig() would write a blank image.
collection = az.plot_compare(comparison)
fig = collection.viz['figure'].item()
fig.suptitle('Model Comparison', fontsize=14, fontweight='bold')
if output_path:
plt.savefig(output_path, dpi=300, bbox_inches='tight')
fig.savefig(output_path, dpi=300, bbox_inches='tight')
print(f"Comparison plot saved to {output_path}")
if show:
plt.show()
else:
plt.close()
plt.close(fig)
return fig
@@ -239,7 +259,7 @@ def model_averaging(models_dict: Dict[str, Any],
models_dict : dict
Dictionary mapping model names to PyMC posterior objects
weights : array-like, optional
Model weights. If None, computed from IC (pseudo-BMA weights)
Model weights. If None, taken from `compare_models` (stacking weights)
var_name : str
Name of the predicted variable (default: 'y_obs')
ic : str
@@ -253,7 +273,7 @@ def model_averaging(models_dict: Dict[str, Any],
Model weights used
"""
if weights is None:
comparison = az.compare(models_dict, ic=ic)
comparison = compare_models(models_dict, ic=ic, verbose=False)
weights = comparison['weight'].values
model_names = comparison.index.tolist()
else:

139
skills/pymc/scripts/model_diagnostics.py
View File

@@ -15,7 +15,6 @@ Usage:
"""
import arviz as az
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path
@@ -44,8 +43,10 @@ def check_diagnostics(idata, var_names=None, ess_threshold=400, rhat_threshold=1
print(" " * 20 + "MCMC DIAGNOSTICS REPORT")
print("="*70)
# Get summary statistics
summary = az.summary(idata, var_names=var_names)
# Get summary statistics. round_to="none" is required: ArviZ 1.x formats the
# default summary for display, returning strings, which makes every numeric
# comparison below raise TypeError.
summary = az.summary(idata, var_names=var_names, round_to="none")
results = {
'summary': summary,
@@ -197,60 +198,51 @@ def create_diagnostic_report(idata, var_names=None, output_dir='diagnostics/', s
print(f"\nGenerating diagnostic plots in '{output_dir}'...")
# 1. Trace plots
az.plot_trace_dist(idata, var_names=var_names)
plt.tight_layout()
plt.savefig(output_path / 'trace_plots.png', dpi=300, bbox_inches='tight')
print(f" ✓ Saved trace plots")
# ArviZ 1.x plots return a PlotCollection and do not draw into pyplot's
# current figure, so the figure must be saved through the collection --
# plt.savefig() would write a blank image.
def _save(plot_collection, filename, label):
plot_collection.savefig(
output_path / filename, dpi=300, bbox_inches='tight'
)
print(f" ✓ Saved {label}")
if show:
plt.show()
else:
plt.close()
plt.close(plot_collection.viz['figure'].item())
# 1. Trace plots
_save(
az.plot_trace_dist(idata, var_names=var_names),
'trace_plots.png',
'trace plots',
)
# 2. Rank plots (check mixing)
fig = plt.figure(figsize=(12, 8))
az.plot_rank(idata, var_names=var_names)
plt.tight_layout()
plt.savefig(output_path / 'rank_plots.png', dpi=300, bbox_inches='tight')
print(f" ✓ Saved rank plots")
if show:
plt.show()
else:
plt.close()
_save(
az.plot_rank(idata, var_names=var_names),
'rank_plots.png',
'rank plots',
)
# 3. Autocorrelation plots
fig = plt.figure(figsize=(12, 8))
az.plot_autocorr(idata, var_names=var_names, combined=True)
plt.tight_layout()
plt.savefig(output_path / 'autocorr_plots.png', dpi=300, bbox_inches='tight')
print(f" ✓ Saved autocorrelation plots")
if show:
plt.show()
else:
plt.close()
_save(
az.plot_autocorr(idata, var_names=var_names),
'autocorr_plots.png',
'autocorrelation plots',
)
# 4. Energy plot (if available)
if hasattr(idata.sample_stats, 'energy'):
fig = plt.figure(figsize=(10, 6))
az.plot_energy(idata)
plt.tight_layout()
plt.savefig(output_path / 'energy_plot.png', dpi=300, bbox_inches='tight')
print(f" ✓ Saved energy plot")
if show:
plt.show()
else:
plt.close()
_save(az.plot_energy(idata), 'energy_plot.png', 'energy plot')
# 5. ESS plot
fig = plt.figure(figsize=(10, 6))
az.plot_ess(idata, var_names=var_names, kind='evolution')
plt.tight_layout()
plt.savefig(output_path / 'ess_evolution.png', dpi=300, bbox_inches='tight')
print(f" ✓ Saved ESS evolution plot")
if show:
plt.show()
else:
plt.close()
# 5. ESS plot. ArviZ 1.x offers 'local' and 'quantile'; the old
# 'evolution' kind no longer exists.
_save(
az.plot_ess(idata, var_names=var_names, kind='local'),
'ess_local.png',
'local ESS plot',
)
# Save summary to CSV
results['summary'].to_csv(output_path / 'summary_statistics.csv')
@@ -272,53 +264,44 @@ def compare_prior_posterior(idata, prior_idata, var_names=None, output_path=None
prior_idata : xarray.DataTree or arviz.InferenceData
Prior object with prior samples
var_names : list, optional
Variables to compare
Variables to compare. Defaults to the first three posterior variables.
output_path : str, optional
If provided, save plot to this path
Returns
-------
None
arviz_plots.PlotCollection
The overlaid prior/posterior figure.
"""
fig, axes = plt.subplots(
len(var_names) if var_names else 3,
1,
figsize=(10, 8)
if var_names is None:
var_names = list(idata.posterior.data_vars)[:3]
# ArviZ 1.x plot functions take a DataTree and a group name, not a flat
# array plus an axis. Passing the first call's PlotCollection back into the
# second is what overlays the two distributions on the same axes.
collection = az.plot_dist(
prior_idata,
group='prior',
var_names=var_names,
visuals={'dist': {'color': 'blue'}},
)
if not isinstance(axes, np.ndarray):
axes = [axes]
for idx, var in enumerate(var_names if var_names else list(idata.posterior.data_vars)[:3]):
# Plot prior
az.plot_dist(
prior_idata.prior[var].values.flatten(),
label='Prior',
ax=axes[idx],
color='blue',
alpha=0.3
idata,
group='posterior',
var_names=var_names,
plot_collection=collection,
visuals={'dist': {'color': 'green'}},
)
# Plot posterior
az.plot_dist(
idata.posterior[var].values.flatten(),
label='Posterior',
ax=axes[idx],
color='green',
alpha=0.3
)
axes[idx].set_title(f'{var}: Prior vs Posterior')
axes[idx].legend()
plt.tight_layout()
if output_path:
plt.savefig(output_path, dpi=300, bbox_inches='tight')
collection.savefig(output_path, dpi=300, bbox_inches='tight')
print(f"Prior-posterior comparison saved to {output_path}")
print("Prior is blue, posterior is green")
else:
plt.show()
return collection
# Example usage
if __name__ == '__main__':

2
skills/pymoo/SKILL.md
View File

@@ -5,7 +5,7 @@ license: Apache-2.0 license
allowed-tools: Read Write Edit Bash
compatibility: Requires Python 3.10+ and pymoo (uv pip install). Optional matplotlib for visualization plots; optional autograd for gradient-based features; optional joblib for JoblibParallelization.
metadata:
version: "1.2"
version: "1.3"
skill-author: K-Dense Inc.
---

2
skills/pymoo/references/algorithms.md
View File

@@ -152,7 +152,7 @@ algorithm = SPEA2(pop_size=100)
from pymoo.algorithms.moo.nsga3 import NSGA3
from pymoo.util.ref_dirs import get_reference_directions
ref_dirs = get_reference_directions("das-dennis", n_obj=4, n_partitions=12)
ref_dirs = get_reference_directions("das-dennis", 4, n_partitions=12) # n_dim is positional
algorithm = NSGA3(ref_dirs=ref_dirs)
```

2
skills/pymoo/references/quick_start_workflows.md
View File

@@ -114,7 +114,7 @@ from pymoo.visualization.pcp import PCP
problem = get_problem("dtlz2", n_obj=5)
# Generate reference directions (required for NSGA-III)
ref_dirs = get_reference_directions("das-dennis", n_obj=5, n_partitions=12)
ref_dirs = get_reference_directions("das-dennis", 5, n_partitions=12) # n_dim is positional
# Configure NSGA-III
algorithm = NSGA3(ref_dirs=ref_dirs)

6
skills/pymoo/scripts/many_objective_example.py
View File

@@ -21,8 +21,10 @@ def run_many_objective_optimization():
problem = get_problem("dtlz2", n_obj=n_obj)
# Generate reference directions for NSGA-III
# Das-Dennis method for uniform distribution
ref_dirs = get_reference_directions("das-dennis", n_obj=n_obj, n_partitions=12)
# Das-Dennis method for uniform distribution.
# The dimension is positional: the factory's first argument is n_dim, and
# passing it as n_obj= raises TypeError.
ref_dirs = get_reference_directions("das-dennis", n_obj, n_partitions=12)
print(f"Number of reference directions: {len(ref_dirs)}")

2
skills/pytorch-lightning/SKILL.md
View File

@@ -5,7 +5,7 @@ allowed-tools: Read Write Edit Bash
license: Apache-2.0 license
compatibility: Requires Python 3.10+ and lightning 2.6+ (or pytorch-lightning 2.6+). GPU training needs CUDA-capable PyTorch. Optional loggers (wandb, mlflow, comet-ml) and DeepSpeed require separate installs.
metadata:
version: "1.0"
version: "1.1"
skill-author: K-Dense Inc.
---

1
skills/pytorch-lightning/references/distributed_training.md
View File

@@ -636,6 +636,7 @@ Save checkpoints to resume from failures:
```python
checkpoint_callback = ModelCheckpoint(
monitor="val_loss", # save_top_k > 1 needs a quantity to rank on
save_top_k=3,
save_last=True, # Always save last for resuming
every_n_epochs=5

21
skills/pytorch-lightning/scripts/quick_trainer_setup.py
View File

@@ -256,6 +256,8 @@ def deepspeed_trainer(
checkpoint_callback = ModelCheckpoint(
dirpath=checkpoint_dir,
filename="{epoch:02d}-{step:06d}",
monitor="val_loss", # Required: save_top_k > 1 needs a quantity
mode="min",
save_top_k=3,
save_last=True,
every_n_train_steps=1000, # Save every N steps
@@ -351,6 +353,8 @@ def time_limited_trainer(
checkpoint_callback = ModelCheckpoint(
dirpath=checkpoint_dir,
monitor="val_loss", # Required: save_top_k > 1 needs a quantity
mode="min",
save_top_k=3,
save_last=True, # Important for resuming
every_n_epochs=5,
@@ -410,6 +414,12 @@ def reproducible_trainer(seed=42, max_epochs=100):
# =============================================================================
if __name__ == "__main__":
from lightning.pytorch.accelerators import CUDAAccelerator
# Trainer construction validates the requested hardware, so the GPU-only
# configurations below can only be built where that hardware exists.
cuda_devices = CUDAAccelerator.auto_device_count() if CUDAAccelerator.is_available() else 0
print("PyTorch Lightning Trainer Configurations\n")
# Example 1: Basic training
@@ -423,29 +433,38 @@ if __name__ == "__main__":
print("2. Debug Trainer:")
trainer = debug_trainer()
print(f" - Fast dev run: {trainer.fast_dev_run}")
print(f" - Detect anomaly: {trainer.detect_anomaly}")
print(f" - Accelerator: {trainer.accelerator}")
print()
# Example 3: Production single GPU
print("3. Production Single GPU Trainer:")
if cuda_devices >= 1:
trainer = production_single_gpu_trainer(max_epochs=100)
print(f" - Max epochs: {trainer.max_epochs}")
print(f" - Precision: {trainer.precision}")
print(f" - Callbacks: {len(trainer.callbacks)}")
else:
print(" - skipped: no CUDA GPU on this machine")
print()
# Example 4: Multi-GPU DDP
print("4. Multi-GPU DDP Trainer:")
if cuda_devices >= 4:
trainer = multi_gpu_ddp_trainer(num_gpus=4)
print(f" - Strategy: {trainer.strategy}")
print(f" - Devices: {trainer.num_devices}")
else:
print(f" - skipped: needs 4 CUDA GPUs, found {cuda_devices}")
print()
# Example 5: FSDP for large models
print("5. FSDP Trainer for Large Models:")
if cuda_devices >= 8:
trainer = large_model_fsdp_trainer(num_gpus=8)
print(f" - Strategy: {trainer.strategy}")
print(f" - Precision: {trainer.precision}")
else:
print(f" - skipped: needs 8 CUDA GPUs, found {cuda_devices}")
print()
print("\nTo use these configurations:")

7
skills/pytorch-lightning/scripts/template_lightning_module.py
View File

@@ -162,13 +162,13 @@ class TemplateLightningModule(L.LightningModule):
weight_decay=1e-5,
)
# Define scheduler
# Define scheduler. PyTorch removed the schedulers' `verbose` argument;
# use LearningRateMonitor or the logged learning_rate metric instead.
scheduler = ReduceLROnPlateau(
optimizer,
mode="min",
factor=0.5,
patience=5,
verbose=True,
)
# Return configuration
@@ -215,5 +215,6 @@ if __name__ == "__main__":
# Train (you need to provide train_dataloader and val_dataloader)
# trainer.fit(model, train_dataloader, val_dataloader)
print(f"Model created with {model.num_parameters:,} parameters")
n_parameters = sum(p.numel() for p in model.parameters())
print(f"Model created with {n_parameters:,} parameters")
print(f"Hyperparameters: {model.hparams}")

2
skills/scanpy/SKILL.md
View File

@@ -3,7 +3,7 @@ name: scanpy
description: Standard single-cell RNA-seq analysis pipeline. Use for QC, normalization, dimensionality reduction (PCA/UMAP/t-SNE), clustering, differential expression, visualization, and converting R-friendly single-cell formats such as Seurat or SingleCellExperiment RDS files into h5ad for Scanpy. Best for exploratory scRNA-seq analysis with established workflows. For deep learning models use scvi-tools; for data format questions use anndata.
license: BSD-3-Clause
metadata:
version: "1.4"
version: "1.5"
skill-author: K-Dense Inc.
---

19
skills/scanpy/scripts/_common.py
View File

@@ -104,13 +104,24 @@ def add_io_args(parser, default_output=None):
return parser
def _named_keys(mapping):
"""Named keys of an AnnData mapping, in order.
anndata >= 0.13 reports an unnamed `None` key on `.layers` standing for X
itself. Joining that into a string raises TypeError, so filter it out.
"""
return [key for key in mapping.keys() if isinstance(key, str)]
def summarize(adata):
"""Return a short human-readable summary string of an AnnData object."""
lines = [f"{adata.n_obs} cells x {adata.n_vars} genes"]
if len(adata.obs.columns):
lines.append("obs: " + ", ".join(adata.obs.columns[:20]))
if list(adata.obsm.keys()):
lines.append("obsm: " + ", ".join(adata.obsm.keys()))
if list(adata.layers.keys()):
lines.append("layers: " + ", ".join(adata.layers.keys()))
obsm = _named_keys(adata.obsm)
if obsm:
lines.append("obsm: " + ", ".join(obsm))
layers = _named_keys(adata.layers)
if layers:
lines.append("layers: " + ", ".join(layers))
return "\n".join(lines)

2
skills/scientific-slides/SKILL.md
View File

@@ -4,7 +4,7 @@ description: Build slide decks and presentations for research talks. Use this fo
allowed-tools: Read Write Edit Bash
license: MIT license
metadata:
version: "1.3"
version: "1.4"
skill-author: K-Dense Inc.
openclaw:
primaryEnv: OPENROUTER_API_KEY

19
skills/scientific-slides/scripts/validate_presentation.py
View File

@@ -16,12 +16,17 @@ import subprocess
from pathlib import Path
from typing import Dict, List, Tuple, Optional
# Try to import PyPDF2 for PDF analysis
# PDF page and geometry analysis. pypdf is the maintained continuation of
# PyPDF2 and exposes the same PdfReader API, so accept whichever is installed.
try:
import PyPDF2
HAS_PYPDF2 = True
from pypdf import PdfReader
HAS_PDF_READER = True
except ImportError:
HAS_PYPDF2 = False
try:
from PyPDF2 import PdfReader
HAS_PDF_READER = True
except ImportError:
HAS_PDF_READER = False
# Try to import python-pptx for PowerPoint analysis
try:
@@ -97,15 +102,15 @@ class PresentationValidator:
def _validate_pdf(self):
"""Validate PDF presentation."""
if not HAS_PYPDF2:
if not HAS_PDF_READER:
self.warnings.append(
"PyPDF2 not installed. Install with: pip install PyPDF2"
"pypdf not installed. Install with: pip install pypdf"
)
return
try:
with open(self.filepath, 'rb') as f:
reader = PyPDF2.PdfReader(f)
reader = PdfReader(f)
num_pages = len(reader.pages)
self.info.append(f"Number of slides: {num_pages}")

18
skills/scvelo/SKILL.md
View File

@@ -2,8 +2,9 @@
name: scvelo
description: RNA velocity analysis with scVelo. Estimate cell state transitions from unspliced/spliced mRNA dynamics, infer trajectory directions, compute latent time, and identify driver genes in single-cell RNA-seq data. Complements Scanpy/scVI-tools for trajectory inference.
license: BSD-3-Clause
compatibility: Requires Python 3.10+ with scvelo, scanpy, and anndata. Verified against scvelo 0.3.4, whose dynamical model and pl.scatter need pandas<3 and whose stochastic estimator needs numpy<2; the deterministic estimator works on current releases.
metadata:
version: "1.0"
version: "1.1"
skill-author: Kuan-lin Huang
---
@@ -74,12 +75,15 @@ print(adata)
### 2. Preprocessing
```python
# Filter and normalize (follows Scanpy conventions)
# Filter and normalize. As of scVelo 0.3, filter_and_normalize() only filters
# genes and normalizes per cell -- it no longer takes n_top_genes and no longer
# log-transforms, so the log step and HVG selection come from Scanpy.
scv.pp.filter_and_normalize(
adata,
min_shared_counts=20, # Minimum counts in spliced+unspliced
n_top_genes=2000 # Top highly variable genes
min_shared_counts=20 # Minimum counts in spliced+unspliced
)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000, subset=True)
# Compute first and second order moments (means and variances)
# knn_connectivities must be computed first
@@ -238,8 +242,10 @@ def run_rna_velocity(adata, n_top_genes=2000, mode='dynamical', n_jobs=4):
"""
scv.settings.verbosity = 2
# 1. Preprocessing
scv.pp.filter_and_normalize(adata, min_shared_counts=20, n_top_genes=n_top_genes)
# 1. Preprocessing (scVelo 0.3 dropped log/HVG from filter_and_normalize)
scv.pp.filter_and_normalize(adata, min_shared_counts=20)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=n_top_genes, subset=True)
if 'neighbors' not in adata.uns:
sc.pp.neighbors(adata, n_neighbors=30)

10
skills/scvelo/scripts/rna_velocity_workflow.py
View File

@@ -65,7 +65,15 @@ def run_velocity_analysis(
# ── Step 2: Preprocessing ─────────────────────────────────────────────────
print("Step 1/5: Preprocessing...")
scv.pp.filter_and_normalize(adata, min_shared_counts=20, n_top_genes=n_top_genes)
# scVelo 0.3 narrowed filter_and_normalize() to gene filtering plus
# per-cell normalization: it no longer accepts n_top_genes and no longer
# log-transforms, so the log step and HVG selection come from Scanpy.
scv.pp.filter_and_normalize(adata, min_shared_counts=20)
if "log1p" not in adata.uns:
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(
adata, n_top_genes=min(n_top_genes, adata.n_vars), subset=True
)
if "neighbors" not in adata.uns:
sc.pp.neighbors(adata, n_neighbors=n_neighbors, n_pcs=30)

38
tests/_contract/__init__.py Normal file
View File

@@ -0,0 +1,38 @@
"""Shared test contract for every skill in this repository.
Suites do not import this package by name from `sys.path` -- putting `tests/`
on `sys.path` would turn `tests/simpy/`, `tests/qutip/`, `tests/neurokit2/` and
friends into importable namespace packages that shadow the real libraries (see
the comment on `addopts` in `pyproject.toml`). Instead `tests/conftest.py`
loads this package by file location and registers it as `skill_contract`, so a
suite writes:
import skill_contract
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
Two halves, split by what they need from the environment:
`structure`
Never imports skill code -- scripts are parsed with `ast`, not executed --
so it is safe to run across every skill in one interpreter.
`tests/_meta` does exactly that.
`cli`
Runs each script's `--help` in a subprocess. Skips when the skill's
packages are absent, and runs for real under
`python tests/run_all.py --isolated`.
`office`
The OOXML tree that docx, pptx, and xlsx each ship a byte-identical copy of.
`schematic`
The AI schematic generator that scientific-schematics, latex-posters, and
literature-review each ship a byte-identical copy of.
"""
from __future__ import annotations
from . import cli, office, schematic, structure
__all__ = ["cli", "office", "schematic", "structure"]

211
tests/_contract/cli.py Normal file
View File

@@ -0,0 +1,211 @@
"""`--help` contract for a skill's bundled command-line scripts.
Unlike `structure`, this half actually runs the scripts, so what it proves
depends on the environment. In the bare project environment most scientific
packages are absent and the affected scripts skip; under
`python tests/run_all.py --isolated`, where each skill gets the packages it
documents, the same tests run for real. That is the point of the split: one
definition, honest in both places.
Each script runs in its own subprocess -- `--help` must not need this
interpreter's `sys.path`, and a script that hangs must not hang the suite.
"""
from __future__ import annotations
import ast
import os
import re
import subprocess
import sys
import unittest
from pathlib import Path
DEFAULT_TIMEOUT = 120
#: Never invoked directly: shared helpers and package markers, not CLIs.
NON_CLI_NAMES = frozenset({"__init__.py", "__main__.py", "_common.py", "conftest.py"})
def cli_scripts(skill_root: Path) -> list[Path]:
"""Top-level scripts under `scripts/` that build an argparse parser.
Only the top level: nested directories such as the `office/` trees under
docx/pptx/xlsx are importable libraries, not entry points.
"""
scripts = []
for path in sorted((skill_root / "scripts").glob("*.py")):
if path.name in NON_CLI_NAMES:
continue
try:
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
except SyntaxError:
continue # structure.compile_problems reports it
imports_argparse = any(
(isinstance(node, ast.Import) and any(a.name == "argparse" for a in node.names))
or (isinstance(node, ast.ImportFrom) and node.module == "argparse")
for node in ast.walk(tree)
)
if imports_argparse:
scripts.append(path)
return scripts
def run_help(script: Path, timeout: int = DEFAULT_TIMEOUT) -> subprocess.CompletedProcess:
environment = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}
return subprocess.run(
[sys.executable, str(script), "--help"],
capture_output=True,
text=True,
timeout=timeout,
env=environment,
cwd=script.parent,
)
#: A script that guards its own imports and prints a friendly install hint --
#: better behaviour than a traceback -- must still be recognised as "package
#: absent" rather than reported as a broken CLI. `exa-search` does exactly this.
_GRACEFUL_IMPORT_GUARD = re.compile(
r"(?:^|\n)\s*([\w.\-]+) (?:is )?not installed\b"
r"|No module named ['\"]?([\w.]+)"
r"|pip install ([\w.\-\[\]]+)",
re.IGNORECASE,
)
def _missing_dependency(output: str) -> str | None:
"""The package name, when a failed `--help` was caused by an absent import.
Deliberately narrow. `ImportError: cannot import name 'PSICQUIC' from
'bioservices'` means the package IS installed and the script is calling an
API that no longer exists -- upstream drift, the most valuable thing this
contract can catch. Treating that as "package absent" would skip the test
and hide the breakage, so only a genuinely absent module counts:
* `ModuleNotFoundError`, or an `ImportError` naming no such module;
* a script that catches the ImportError itself and prints an install hint.
"""
for line in reversed(output.strip().splitlines()):
if line.startswith("ModuleNotFoundError:"):
return line.split(":", 1)[1].strip()
if line.startswith("ImportError:"):
message = line.split(":", 1)[1].strip()
return message if "No module named" in message else None
match = _GRACEFUL_IMPORT_GUARD.search(output)
if match:
return next(group for group in match.groups() if group)
return None
def help_test_case(
skill_root: Path,
*,
skip: frozenset[str] | set[str] | tuple[str, ...] = (),
timeout: int = DEFAULT_TIMEOUT,
) -> type[unittest.TestCase]:
"""Build the `--help` TestCase for one skill.
`skip` names scripts (by filename) that legitimately have no working
`--help` -- pass it sparingly and say why at the call site.
"""
skipped = frozenset(skip)
class CliHelpContractTests(unittest.TestCase):
maxDiff = None
def test_every_cli_answers_help(self) -> None:
scripts = [
path for path in cli_scripts(skill_root) if path.name not in skipped
]
self.assertTrue(
scripts,
f"{skill_root.name} ships no argparse CLI -- drop this test case "
"or fix the discovery",
)
for script in scripts:
with self.subTest(script=script.name):
try:
result = run_help(script, timeout=timeout)
except subprocess.TimeoutExpired:
self.fail(
f"{script.name} --help did not return within {timeout}s; "
"it is doing work before parsing arguments"
)
if result.returncode != 0:
missing = _missing_dependency(result.stderr + result.stdout)
if missing:
self.skipTest(
f"{script.name} needs an uninstalled package ({missing}); "
"run under `tests/run_all.py --isolated` to exercise it"
)
self.fail(
f"{script.name} --help exited {result.returncode}:\n"
f"{result.stderr.strip()}"
)
self.assertIn(
"usage:",
result.stdout.lower(),
f"{script.name} --help printed no usage line",
)
self.assertNotIn("Traceback", result.stderr)
CliHelpContractTests.__qualname__ = f"CliHelpContractTests[{skill_root.name}]"
return CliHelpContractTests
def demo_test_case(
skill_root: Path,
scripts: tuple[str, ...],
*,
timeout: int = DEFAULT_TIMEOUT,
) -> type[unittest.TestCase]:
"""Build a TestCase that runs a library script's `__main__` demo block.
Some skills ship importable modules rather than CLIs, with a worked example
under `if __name__ == "__main__":`. That example is documentation, and
documentation rots -- so run it and require a clean exit.
Opt-in and explicit: pass the filenames, because a demo block that writes
files or takes minutes should not be run by accident.
"""
class DemoBlockTests(unittest.TestCase):
maxDiff = None
def test_every_demo_block_runs_clean(self) -> None:
for name in scripts:
script = skill_root / "scripts" / name
with self.subTest(script=name):
self.assertTrue(script.is_file(), f"{name} is not shipped")
environment = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}
# Headless: a demo that plots must not open a window.
environment.setdefault("MPLBACKEND", "Agg")
try:
result = subprocess.run(
[sys.executable, str(script)],
capture_output=True,
text=True,
timeout=timeout,
env=environment,
cwd=script.parent,
)
except subprocess.TimeoutExpired:
self.fail(f"{name} did not finish within {timeout}s")
if result.returncode != 0:
missing = _missing_dependency(result.stderr + result.stdout)
if missing:
self.skipTest(
f"{name} needs an uninstalled package ({missing}); "
"run under `tests/run_all.py --isolated`"
)
self.fail(
f"{name} exited {result.returncode}:\n{result.stderr.strip()}"
)
self.assertTrue(
result.stdout.strip(), f"{name} printed nothing"
)
DemoBlockTests.__qualname__ = f"DemoBlockTests[{skill_root.name}]"
return DemoBlockTests

296
tests/_contract/office.py Normal file
View File

@@ -0,0 +1,296 @@
"""Shared tests for the OOXML `office/` tree bundled by docx, pptx, and xlsx.
Those three skills each ship a byte-identical copy of `scripts/office/` --
`helpers/`, `validators/`, `validate.py`, `soffice.py`, and the vendored
ISO/ECMA schemas. Rather than write the same tests three times, each suite
instantiates `office_test_case()` against its own copy, so a change that lands
in one copy and not the others fails on the copy that was missed as well as on
`identical_tree_problems()` in `tests/_meta`.
The interesting surface is the safety logic: `safe_extract` guards zip-slip and
symlink entries, and `opc_target` refuses relationship targets that escape the
package. Both are reachable with nothing but the standard library.
"""
from __future__ import annotations
import hashlib
import importlib
import stat
import sys
import tempfile
import unittest
import zipfile
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
SKILLS_DIR = REPO_ROOT / "skills"
#: The skills that ship the shared tree. Kept here so adding a fourth is one edit.
OFFICE_SKILLS = ("docx", "pptx", "xlsx")
#: Other files several skills ship byte-identical copies of. A copy that drifts
#: is a skill quietly behaving differently from its siblings, so `tests/_meta`
#: pins them together. Each entry is (path relative to the skill, skills).
SHARED_FILES: tuple[tuple[str, tuple[str, ...]], ...] = (
(
"scripts/generate_schematic.py",
(
"scientific-schematics",
"latex-posters",
"literature-review",
"citation-management",
"scientific-slides",
),
),
(
"scripts/generate_schematic_ai.py",
(
"scientific-schematics",
"latex-posters",
"literature-review",
"citation-management",
"scientific-slides",
),
),
)
def shared_file_problems(skills_dir: Path = SKILLS_DIR) -> list[str]:
"""Every skill in a `SHARED_FILES` group ships the same bytes."""
problems = []
for relative, skills in SHARED_FILES:
digests = {}
for name in skills:
path = skills_dir / name / relative
if not path.is_file():
problems.append(f"{name}: {relative} is missing")
continue
digests[name] = hashlib.sha256(path.read_bytes()).hexdigest()
if len(set(digests.values())) > 1:
reference = skills[0]
problems.extend(
f"{name}: {relative} has drifted from {reference}'s copy"
for name, digest in digests.items()
if digest != digests.get(reference)
)
return problems
def _tree_digest(office_dir: Path) -> str:
"""A digest over every path and byte under an `office/` tree."""
digest = hashlib.sha256()
for path in sorted(office_dir.rglob("*")):
if not path.is_file():
continue
digest.update(str(path.relative_to(office_dir)).encode("utf-8"))
digest.update(path.read_bytes())
return digest.hexdigest()
def identical_tree_problems(skills_dir: Path = SKILLS_DIR) -> list[str]:
"""The shared `office/` trees have not drifted apart."""
digests = {}
for name in OFFICE_SKILLS:
office = skills_dir / name / "scripts" / "office"
if not office.is_dir():
return [f"{name}: scripts/office/ is missing"]
digests[name] = _tree_digest(office)
reference = digests[OFFICE_SKILLS[0]]
return [
f"{name}: scripts/office/ has drifted from {OFFICE_SKILLS[0]}'s copy "
f"({digests[name][:12]} vs {reference[:12]})"
for name in OFFICE_SKILLS[1:]
if digests[name] != reference
]
def office_test_case(skill_root: Path) -> type[unittest.TestCase]:
"""Build the shared `office/` TestCase for one of docx / pptx / xlsx."""
office_dir = skill_root / "scripts" / "office"
def load_helpers():
# `office/` puts `helpers` and `validators` at the top level -- see the
# `from helpers import ...` in validate.py -- so office/ itself goes on
# the path. Safe because this repo runs one skill per process.
if str(office_dir) not in sys.path:
sys.path.insert(0, str(office_dir))
return importlib.import_module("helpers")
class OfficeSharedTreeTests(unittest.TestCase):
maxDiff = None
def setUp(self) -> None:
self.helpers = load_helpers()
# -- safe_extract ------------------------------------------------
def test_safe_extract_unpacks_an_ordinary_package(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
archive = root / "package.docx"
with zipfile.ZipFile(archive, "w") as handle:
handle.writestr("[Content_Types].xml", "<Types/>")
handle.writestr("word/document.xml", "<document/>")
destination = root / "unpacked"
destination.mkdir()
with zipfile.ZipFile(archive) as handle:
self.helpers.safe_extract(handle, destination)
self.assertTrue((destination / "[Content_Types].xml").is_file())
self.assertEqual(
(destination / "word" / "document.xml").read_text(), "<document/>"
)
def test_safe_extract_refuses_a_traversal_entry(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
archive = root / "evil.docx"
with zipfile.ZipFile(archive, "w") as handle:
handle.writestr("../escaped.xml", "<pwned/>")
destination = root / "unpacked"
destination.mkdir()
with zipfile.ZipFile(archive) as handle:
with self.assertRaisesRegex(ValueError, "unsafe archive entry"):
self.helpers.safe_extract(handle, destination)
self.assertFalse((root / "escaped.xml").exists())
def test_safe_extract_refuses_a_symlink_entry(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
archive = root / "link.docx"
info = zipfile.ZipInfo("word/document.xml")
info.external_attr = (stat.S_IFLNK | 0o777) << 16
with zipfile.ZipFile(archive, "w") as handle:
handle.writestr(info, "/etc/passwd")
destination = root / "unpacked"
destination.mkdir()
with zipfile.ZipFile(archive) as handle:
with self.assertRaisesRegex(ValueError, "symlink archive entry"):
self.helpers.safe_extract(handle, destination)
# -- opc_target --------------------------------------------------
def test_opc_target_resolves_relative_and_absolute_part_names(self) -> None:
resolve = self.helpers.opc_target
self.assertEqual(
resolve("media/image1.png", "word/document.xml"),
"word/media/image1.png",
)
self.assertEqual(
resolve("/word/styles.xml", "word/document.xml"), "word/styles.xml"
)
self.assertEqual(
resolve("../customXml/item1.xml", "word/document.xml"),
"customXml/item1.xml",
)
# Percent-encoding is decoded before resolution.
self.assertEqual(
resolve("media/my%20image.png", "word/document.xml"),
"word/media/my image.png",
)
def test_opc_target_ignores_external_and_scheme_targets(self) -> None:
resolve = self.helpers.opc_target
self.assertIsNone(resolve("", "word/document.xml"))
self.assertIsNone(
resolve("https://example.invalid/x", "word/document.xml", "External")
)
self.assertIsNone(resolve("mailto:someone@example.invalid", "word/document.xml"))
def test_opc_target_refuses_targets_that_escape_the_package(self) -> None:
resolve = self.helpers.opc_target
with self.assertRaisesRegex(ValueError, "escapes the package"):
resolve("../../outside.xml", "word/document.xml")
with self.assertRaisesRegex(ValueError, "not a POSIX part name"):
resolve("word\\styles.xml", "word/document.xml")
# A bare "/" is absolute and strips to nothing at all.
with self.assertRaisesRegex(ValueError, "resolves to nothing"):
resolve("/", "word/document.xml")
def test_a_dot_target_resolves_to_its_own_directory(self) -> None:
# normpath collapses it rather than treating it as an error.
self.assertEqual(
self.helpers.opc_target(".", "word/document.xml"), "word"
)
# -- text rendering ----------------------------------------------
def test_rendered_text_honours_xml_space_preserve(self) -> None:
render = self.helpers.rendered_text
self.assertEqual(render(" spaced ", True), " spaced ")
self.assertEqual(render(" spaced ", False), "spaced")
self.assertEqual(render("\t\nmixed\r\n", False), "mixed")
def test_part_text_survives_undecodable_bytes(self) -> None:
# surrogateescape, not a crash: a corrupt part must still be
# reportable rather than taking the whole validation run down.
self.assertEqual(self.helpers.part_text(b"ok"), "ok")
self.assertEqual(len(self.helpers.part_text(b"\xff\xfe")), 2)
# -- rezip -------------------------------------------------------
def test_rezip_stores_content_types_first_and_uncompressed(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "unpacked"
(source / "word").mkdir(parents=True)
(source / "[Content_Types].xml").write_text("<Types/>")
(source / "word" / "document.xml").write_text("<document/>")
output = root / "rebuilt.docx"
self.helpers.rezip(source, output)
with zipfile.ZipFile(output) as handle:
entries = handle.namelist()
# OOXML readers require [Content_Types].xml first and stored.
self.assertEqual(entries[0], "[Content_Types].xml")
self.assertEqual(
handle.getinfo("[Content_Types].xml").compress_type,
zipfile.ZIP_STORED,
)
self.assertIn("word/document.xml", entries)
def test_rezip_leaves_no_temporary_file_behind(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "unpacked"
source.mkdir()
(source / "part.xml").write_text("<part/>")
output = root / "rebuilt.docx"
self.helpers.rezip(source, output)
self.assertEqual(
sorted(path.name for path in root.iterdir()),
["rebuilt.docx", "unpacked"],
)
# -- packaging ---------------------------------------------------
def test_ooxml_family_covers_this_skill(self) -> None:
families = set(self.helpers.OOXML_FAMILY.values())
self.assertEqual(families, {"docx", "pptx", "xlsx"})
self.assertIn(f".{skill_root.name}", self.helpers.OOXML_FAMILY)
def test_every_validator_schema_path_exists(self) -> None:
schemas = office_dir / "schemas"
self.assertTrue(schemas.is_dir(), "vendored schemas are missing")
referenced = set()
for module in sorted((office_dir / "validators").glob("*.py")):
for line in module.read_text(encoding="utf-8").splitlines():
for token in line.split('"'):
if token.endswith(".xsd"):
referenced.add(token.lstrip("/"))
missing = sorted(
name
for name in referenced
if not any(schemas.rglob(Path(name).name))
)
self.assertEqual(missing, [], "validators name schemas that are not shipped")
OfficeSharedTreeTests.__qualname__ = f"OfficeSharedTreeTests[{skill_root.name}]"
return OfficeSharedTreeTests

102
tests/_contract/schematic.py Normal file
View File

@@ -0,0 +1,102 @@
"""Shared tests for the AI schematic generator bundled by three skills.
`scientific-schematics`, `latex-posters`, and `literature-review` each ship a
byte-identical `scripts/generate_schematic.py` (a thin CLI) and
`scripts/generate_schematic_ai.py` (the generator). Rather than write the same
tests three times, each suite instantiates `schematic_test_case()` against its
own copy; `tests/_meta` separately pins the copies together.
The behaviour worth testing offline is the environment allowlist. The CLI
re-executes the generator as a subprocess, and it deliberately forwards a
named set of variables rather than the whole parent environment -- copying
everything would hand the child every unrelated secret exported in the calling
shell. Nothing here makes a network call or needs an API key.
"""
from __future__ import annotations
import importlib
import sys
import unittest
from pathlib import Path
from unittest import mock
def schematic_test_case(skill_root: Path) -> type[unittest.TestCase]:
"""Build the shared schematic-generator TestCase for one skill."""
scripts = skill_root / "scripts"
def load():
if str(scripts) not in sys.path:
sys.path.insert(0, str(scripts))
return importlib.import_module("generate_schematic")
class SchematicContractTests(unittest.TestCase):
maxDiff = None
def setUp(self) -> None:
self.module = load()
def test_only_allowlisted_variables_reach_the_subprocess(self) -> None:
environment = {
"PATH": "/usr/bin",
"HOME": "/home/someone",
"AWS_SECRET_ACCESS_KEY": "should-not-be-forwarded",
"GITHUB_TOKEN": "also-not",
}
with mock.patch.dict("os.environ", environment, clear=True):
built = self.module.build_subprocess_env(None)
self.assertEqual(built["PATH"], "/usr/bin")
self.assertEqual(built["HOME"], "/home/someone")
self.assertNotIn("AWS_SECRET_ACCESS_KEY", built)
self.assertNotIn("GITHUB_TOKEN", built)
def test_the_api_key_is_injected_under_the_expected_name(self) -> None:
with mock.patch.dict("os.environ", {"PATH": "/usr/bin"}, clear=True):
built = self.module.build_subprocess_env("sk-or-test")
self.assertEqual(built["OPENROUTER_API_KEY"], "sk-or-test")
def test_no_key_means_no_key_variable_rather_than_an_empty_one(self) -> None:
# An empty OPENROUTER_API_KEY would look configured to the child and
# fail deep inside the request instead of at the boundary.
with mock.patch.dict("os.environ", {"PATH": "/usr/bin"}, clear=True):
for value in (None, ""):
with self.subTest(value=value):
self.assertNotIn(
"OPENROUTER_API_KEY", self.module.build_subprocess_env(value)
)
def test_absent_variables_are_omitted_not_blanked(self) -> None:
with mock.patch.dict("os.environ", {"PATH": "/usr/bin"}, clear=True):
built = self.module.build_subprocess_env(None)
self.assertEqual(set(built), {"PATH"})
def test_the_allowlist_covers_proxies_tls_and_windows_startup(self) -> None:
forwarded = set(self.module.FORWARDED_ENV_VARS)
# Dropping any of these breaks the child in a way that looks like a
# model failure rather than a configuration one.
for required in (
"PATH", "HOME",
"HTTPS_PROXY", "https_proxy",
"SSL_CERT_FILE", "REQUESTS_CA_BUNDLE",
"SYSTEMROOT", "COMSPEC",
):
with self.subTest(variable=required):
self.assertIn(required, forwarded)
def test_the_allowlist_carries_no_credential_variables(self) -> None:
forwarded = " ".join(self.module.FORWARDED_ENV_VARS).upper()
for banned in ("SECRET", "TOKEN", "PASSWORD", "API_KEY", "CREDENTIAL"):
with self.subTest(term=banned):
self.assertNotIn(banned, forwarded)
def test_the_allowlist_has_no_duplicates(self) -> None:
forwarded = self.module.FORWARDED_ENV_VARS
self.assertEqual(len(set(forwarded)), len(forwarded))
def test_the_generator_script_is_shipped_alongside_the_cli(self) -> None:
self.assertTrue((scripts / "generate_schematic_ai.py").is_file())
SchematicContractTests.__qualname__ = f"SchematicContractTests[{skill_root.name}]"
return SchematicContractTests

432
tests/_contract/structure.py Normal file
View File

@@ -0,0 +1,432 @@
"""The structural contract every skill in this repository must satisfy.
Nothing here imports skill code. Scripts are read and parsed with `ast`, never
executed, so the whole contract is safe to run against every skill inside a
single interpreter -- which is what `tests/_meta` does. Per-skill suites that
*do* import their scripts still need one process each; see `tests/conftest.py`.
Every check takes a skill directory and returns a list of human-readable
problems, empty when the skill conforms. Returning problems rather than
asserting keeps the checks reusable: `tests/_meta` reports them per rule and
per skill, so one broken link names the file and the line rather than failing
an opaque assertion.
"""
from __future__ import annotations
import ast
import re
import subprocess
import sys
from collections.abc import Callable, Iterable
from functools import lru_cache
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
SKILLS_DIR = REPO_ROOT / "skills"
# The Agent Skills specification defines a closed set of top-level keys. Any
# other key is a validation error, and because strictyaml rejects the whole
# document, it takes `name` and `description` down with it.
ALLOWED_FRONTMATTER_FIELDS = frozenset(
{"name", "description", "license", "compatibility", "allowed-tools", "metadata"}
)
MAX_SKILL_MD_LINES = 500
# `__import__` is deliberately absent: several skills use it for a legitimate
# availability probe (`try: __import__("torch")`) or to reach pathlib before
# the sys.path insert that makes `_common` importable.
BANNED_BUILTIN_CALLS = frozenset({"eval", "exec"})
BANNED_OS_CALLS = frozenset({"system", "popen"})
# Inline-code and markdown-link references to files the skill ships.
_INLINE_PATH = re.compile(r"`((?:assets|references|scripts)/[A-Za-z0-9_./-]+)`")
_MARKDOWN_LINK = re.compile(r"\]\(((?:assets|references|scripts)/[A-Za-z0-9_./-]+)\)")
# An absolute path under someone's home or drive mount is a leaked local
# environment: it names a person, and it cannot work on any other machine.
_PERSONAL_PATH = re.compile(
r"/(?:mnt/[a-z]/Users|home|Users)/(?!<|\$|\{)([A-Za-z0-9._-]+)/"
)
# Service and platform accounts that legitimately appear in documentation --
# `/home/dnanexus/` is where a DNAnexus worker runs, not somebody's laptop --
# plus the placeholder spellings skills use for "your username".
_IMPERSONAL_ACCOUNTS = frozenset(
{
"dnanexus", "ubuntu", "root", "runner", "ec2-user", "jovyan", "vscode",
"airflow", "nextflow", "opt", "shared", "linuxbrew",
"user", "username", "you", "me", "youruser", "your-user", "name",
}
)
def script_bearing_skills(skills_dir: Path = SKILLS_DIR) -> list[Path]:
"""Skill directories that ship at least one file under `scripts/`.
An empty `scripts/` directory does not count. Git cannot track empty
directories, so one only ever exists as local cruft, and requiring a test
suite for a skill that ships no scripts would be nonsense.
"""
return [
skill
for skill in sorted(skills_dir.iterdir())
if skill.is_dir()
and (skill / "SKILL.md").is_file()
and any((skill / "scripts").rglob("*"))
]
def all_skill_names(skills_dir: Path = SKILLS_DIR) -> set[str]:
return {
skill.name
for skill in skills_dir.iterdir()
if skill.is_dir() and (skill / "SKILL.md").is_file()
}
def _frontmatter(skill: Path) -> str | None:
"""The raw YAML between the opening and closing `---`, or None."""
text = (skill / "SKILL.md").read_text(encoding="utf-8")
if not text.startswith("---\n"):
return None
end = text.find("\n---", 3)
if end == -1:
return None
return text[4:end]
def _top_level_entries(frontmatter: str) -> list[tuple[str, str]]:
"""(key, value) for every unindented `key: value` line."""
entries = []
for line in frontmatter.splitlines():
match = re.match(r"^([A-Za-z][A-Za-z0-9_-]*):(.*)$", line)
if match:
entries.append((match.group(1), match.group(2).strip()))
return entries
def _metadata_scalars(frontmatter: str) -> list[tuple[str, str]]:
"""(key, value) for scalars nested one level under `metadata:`.
Host manifest blocks (`openclaw`, `hermes`) nest further; their contents
are deliberately skipped -- the spec exempts them from the string-only
rule, and they carry booleans on purpose.
"""
lines = frontmatter.splitlines()
try:
start = next(i for i, line in enumerate(lines) if line.startswith("metadata:"))
except StopIteration:
return []
scalars = []
for line in lines[start + 1 :]:
if line.strip() and not line.startswith((" ", "\t")):
break
match = re.match(r"^ ([A-Za-z][A-Za-z0-9_-]*):(.*)$", line)
if match and match.group(2).strip():
scalars.append((match.group(1), match.group(2).strip()))
return scalars
def frontmatter_problems(skill: Path) -> list[str]:
"""Frontmatter parses, uses only spec fields, and is versioned."""
problems: list[str] = []
if not (skill / "SKILL.md").is_file():
return [f"{skill.name}: no SKILL.md"]
frontmatter = _frontmatter(skill)
if frontmatter is None:
return [f"{skill.name}: SKILL.md has no `---` delimited frontmatter"]
entries = _top_level_entries(frontmatter)
keys = [key for key, _ in entries]
for key in keys:
if key not in ALLOWED_FRONTMATTER_FIELDS:
problems.append(
f"{skill.name}: top-level `{key}` is not one of the six spec fields "
"-- move it under `metadata`"
)
for required in ("name", "description", "metadata"):
if required not in keys:
problems.append(f"{skill.name}: frontmatter is missing `{required}`")
values = dict(entries)
name = values.get("name", "").strip("\"'")
if name and name != skill.name:
problems.append(
f"{skill.name}: frontmatter name is `{name}`, must equal the directory name"
)
# strictyaml rejects JSON flow style outright, taking the whole document
# with it -- so `name` and `description` become unreadable too.
for key, value in entries:
if value.startswith(("{", "[")):
problems.append(
f"{skill.name}: `{key}` uses JSON flow style; strictyaml rejects it"
)
tools = values.get("allowed-tools")
if tools is not None and ("," in tools or tools.startswith("[")):
problems.append(
f"{skill.name}: `allowed-tools` must be a space-separated string"
)
scalars = dict(_metadata_scalars(frontmatter))
if "version" not in scalars:
problems.append(f"{skill.name}: `metadata.version` is required")
for key, value in scalars.items():
# Only values YAML would actually coerce. `1.0.0` has two dots, so it
# is already a string; `1.0` is a float and must be quoted.
ambiguous = re.fullmatch(
r"\d+|\d+\.\d+|true|false|yes|no|on|off|\d{4}-\d{2}-\d{2}", value, re.I
)
if ambiguous:
problems.append(
f"{skill.name}: `metadata.{key}: {value}` must be quoted to stay a string"
)
return problems
def length_problems(skill: Path) -> list[str]:
"""SKILL.md stays under 500 lines; longer material belongs in references/."""
lines = len((skill / "SKILL.md").read_text(encoding="utf-8").splitlines())
if lines > MAX_SKILL_MD_LINES:
return [
f"{skill.name}: SKILL.md is {lines} lines, over the {MAX_SKILL_MD_LINES}-line "
"limit -- move reference material into references/"
]
return []
def stray_test_problems(skill: Path) -> list[str]:
"""Tests never live under skills/ -- a skill ships only what an agent loads."""
strays = [
str(path.relative_to(skill))
for path in sorted(skill.rglob("*"))
if (path.is_dir() and path.name == "tests")
or (path.is_file() and path.name.startswith("test_") and path.suffix == ".py")
]
return [
f"{skill.name}: ships {stray}; tests belong in tests/{skill.name}/"
for stray in strays
]
@lru_cache(maxsize=1)
def _tracked_files(skills_dir: Path = SKILLS_DIR) -> frozenset[str] | None:
"""Repo-relative paths git tracks under `skills/`, or None outside a checkout."""
try:
listed = subprocess.run(
["git", "ls-files", "--", str(skills_dir)],
capture_output=True,
text=True,
timeout=60,
cwd=REPO_ROOT,
)
except (OSError, subprocess.SubprocessError):
return None
if listed.returncode != 0:
return None
return frozenset(listed.stdout.split())
def bytecode_problems(skill: Path) -> list[str]:
"""No compiled bytecode ships with a skill.
"Ships" means committed. Importing a skill's scripts is what creates
`__pycache__`, and running one by hand outside pytest -- which sets
`PYTHONDONTWRITEBYTECODE` -- leaves some behind, so scanning the working
tree would fail on local cruft that is already gitignored. Fall back to a
filesystem scan only when git cannot answer.
"""
tracked = _tracked_files()
candidates = [
path
for path in sorted(skill.rglob("*"))
if path.suffix in {".pyc", ".pyo"} or path.name == "__pycache__"
]
if tracked is not None:
candidates = [
path
for path in candidates
if str(path.relative_to(REPO_ROOT)) in tracked
or any(entry.startswith(f"{path.relative_to(REPO_ROOT)}/") for entry in tracked)
]
return [
f"{skill.name}: ships bytecode artifact {path.relative_to(skill)}"
for path in candidates
]
def link_problems(skill: Path, known_skills: Iterable[str] | None = None) -> list[str]:
"""Every `assets/`, `references/`, or `scripts/` path in the docs resolves.
Skills routinely point at each other -- bulk-rnaseq documents
`pathway-enrichment`'s `scripts/run_enrichment.py`, citation-management
points at `pyzotero` -> `references/exports.md`. A path that does not
resolve inside this skill is therefore accepted when another skill is
named on the same line and owns it.
"""
names = set(known_skills) if known_skills is not None else all_skill_names()
documents = [skill / "SKILL.md", *sorted((skill / "references").glob("*.md"))]
problems = []
for document in documents:
if not document.is_file():
continue
for number, line in enumerate(document.read_text(encoding="utf-8").splitlines(), 1):
for relative in set(_INLINE_PATH.findall(line)) | set(
_MARKDOWN_LINK.findall(line)
):
if (skill / relative).exists():
continue
owners = [
other
for other in names
if other != skill.name
and other in line
and (SKILLS_DIR / other / relative).exists()
]
if owners:
continue
problems.append(
f"{skill.name}: {document.name}:{number} references "
f"`{relative}`, which does not exist"
)
return problems
def _script_paths(skill: Path, suffix: str = ".py") -> list[Path]:
return sorted((skill / "scripts").rglob(f"*{suffix}"))
def compile_problems(skill: Path) -> list[str]:
"""Every bundled Python script parses -- a syntax error can never ship."""
problems = []
for path in _script_paths(skill):
try:
ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
except SyntaxError as error:
problems.append(
f"{skill.name}: {path.relative_to(skill)} does not parse: "
f"line {error.lineno}: {error.msg}"
)
return problems
def dynamic_execution_problems(skill: Path) -> list[str]:
"""No `eval`, `exec`, `os.system`, or `os.popen` in bundled scripts.
A narrower rule than the Cisco scanner's, and a complementary one: this is
deterministic and runs on every skill every time, where the scanner is
LLM-backed and runs only on changed skills.
"""
problems = []
for path in _script_paths(skill):
try:
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
except SyntaxError:
continue # compile_problems already reports it
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
function = node.func
if isinstance(function, ast.Name) and function.id in BANNED_BUILTIN_CALLS:
problems.append(
f"{skill.name}: {path.relative_to(skill)}:{node.lineno} "
f"calls {function.id}()"
)
if (
isinstance(function, ast.Attribute)
and function.attr in BANNED_OS_CALLS
and isinstance(function.value, ast.Name)
and function.value.id == "os"
):
problems.append(
f"{skill.name}: {path.relative_to(skill)}:{node.lineno} "
f"calls os.{function.attr}()"
)
return problems
def shadow_module_problems(skill: Path) -> list[str]:
"""No script shadows a standard-library module.
Tests and the scripts themselves put `scripts/` on `sys.path`, so a file
named `json.py` or `csv.py` there wins over the real module for the rest
of the process.
"""
return [
f"{skill.name}: scripts/{path.name} shadows the standard-library module "
f"`{path.stem}`"
for path in sorted((skill / "scripts").glob("*.py"))
if path.stem in sys.stdlib_module_names
]
def personal_path_problems(skill: Path) -> list[str]:
"""No script or document hardcodes somebody's home directory.
A path like `/mnt/c/Users/<name>/Data/kg.csv` names a real person and works
on exactly one machine. Placeholders are fine -- `/home/$USER/`,
`/Users/<you>/` -- so the pattern skips anything that looks templated.
"""
documents = [
skill / "SKILL.md",
*sorted((skill / "references").glob("*.md")),
*_script_paths(skill),
*_script_paths(skill, ".sh"),
]
problems = []
for document in documents:
if not document.is_file():
continue
for number, line in enumerate(
document.read_text(encoding="utf-8", errors="replace").splitlines(), 1
):
for match in _PERSONAL_PATH.finditer(line):
if match.group(1).lower() in _IMPERSONAL_ACCOUNTS:
continue
problems.append(
f"{skill.name}: {document.relative_to(skill)}:{number} hardcodes "
f"the local path `{match.group(0)}`"
)
return problems
def shell_script_problems(skill: Path) -> list[str]:
"""Bundled shell scripts have a shebang, the executable bit, and parse."""
problems = []
for path in _script_paths(skill, ".sh"):
relative = path.relative_to(skill)
first_line = path.read_text(encoding="utf-8").splitlines()[:1]
if not first_line or not first_line[0].startswith("#!"):
problems.append(f"{skill.name}: {relative} has no shebang")
if not path.stat().st_mode & 0o111:
problems.append(f"{skill.name}: {relative} is not executable")
syntax = subprocess.run(
["bash", "-n", str(path)], capture_output=True, text=True, timeout=30
)
if syntax.returncode != 0:
problems.append(
f"{skill.name}: {relative} fails `bash -n`: {syntax.stderr.strip()}"
)
return problems
#: Rule name -> check. `tests/_meta` iterates this so a new rule is picked up
#: repo-wide by adding one entry.
CHECKS: dict[str, Callable[[Path], list[str]]] = {
"frontmatter": frontmatter_problems,
"skill_md_length": length_problems,
"no_tests_under_skills": stray_test_problems,
"no_bytecode": bytecode_problems,
"local_links_resolve": link_problems,
"scripts_compile": compile_problems,
"no_dynamic_execution": dynamic_execution_problems,
"no_stdlib_shadowing": shadow_module_problems,
"no_personal_paths": personal_path_problems,
"shell_scripts": shell_script_problems,
}

126
tests/_meta/test_repo_contract.py Normal file
View File

@@ -0,0 +1,126 @@
"""Repo-wide guards: every skill conforms, and every skill with scripts is tested.
This suite is deliberately not per-skill. It imports no skill code -- the
structural contract parses scripts with `ast` and never executes them -- so
running it across all skills in one interpreter is safe, and it is the only
place that can see the whole repository at once. That is what lets it enforce
the rule `AGENTS.md` states but nothing previously checked:
If the skill ships `scripts/`, put their tests in `tests/<name>/`.
It runs in the project environment and needs no scientific packages, so CI can
run it on every pull request in seconds.
"""
from __future__ import annotations
import tomllib
import unittest
from pathlib import Path
import skill_contract
REPO_ROOT = Path(__file__).resolve().parents[2]
SKILLS_DIR = REPO_ROOT / "skills"
TESTS_DIR = REPO_ROOT / "tests"
REQUIREMENTS = TESTS_DIR / "skill-requirements.toml"
structure = skill_contract.structure
office = skill_contract.office
SCRIPT_BEARING = structure.script_bearing_skills(SKILLS_DIR)
KNOWN_SKILLS = structure.all_skill_names(SKILLS_DIR)
def _suite_names() -> set[str]:
"""Test directories that stand for a skill, excluding infrastructure."""
return {
path.name
for path in TESTS_DIR.iterdir()
if path.is_dir() and not path.name.startswith((".", "_"))
}
class CoverageTests(unittest.TestCase):
"""The rule this whole suite exists to enforce."""
maxDiff = None
def test_every_skill_with_scripts_has_a_test_suite(self) -> None:
suites = _suite_names()
untested = sorted(
skill.name for skill in SCRIPT_BEARING if skill.name not in suites
)
self.assertEqual(
untested,
[],
"these skills ship scripts/ but have no tests/<name>/ suite; add one "
"(see AGENTS.md, 'Creating a skill' step 5)",
)
def test_every_suite_has_a_test_file(self) -> None:
empty = sorted(
name
for name in _suite_names()
if not any((TESTS_DIR / name).glob("test_*.py"))
)
self.assertEqual(empty, [], "test directories with no test_*.py")
def test_no_test_suite_is_orphaned(self) -> None:
orphans = sorted(name for name in _suite_names() if name not in KNOWN_SKILLS)
self.assertEqual(
orphans, [], "test directories that do not name a skill under skills/"
)
def test_every_skill_with_scripts_has_a_requirements_entry(self) -> None:
"""`--isolated` needs a `[skills.<name>]` entry or it cannot build the env."""
manifest = tomllib.loads(REQUIREMENTS.read_text(encoding="utf-8"))
entries = manifest.get("skills", {})
missing = sorted(
skill.name for skill in SCRIPT_BEARING if skill.name not in entries
)
self.assertEqual(
missing,
[],
f"add a [skills.<name>] block to {REQUIREMENTS.name} "
"(packages = [] for standard-library-only skills)",
)
def test_requirements_entries_name_real_skills(self) -> None:
manifest = tomllib.loads(REQUIREMENTS.read_text(encoding="utf-8"))
unknown = sorted(set(manifest.get("skills", {})) - KNOWN_SKILLS)
self.assertEqual(unknown, [], f"stale entries in {REQUIREMENTS.name}")
class StructuralContractTests(unittest.TestCase):
"""Every rule in `skill_contract.structure`, against every script-bearing skill."""
maxDiff = None
def test_all_skills_satisfy_every_structural_rule(self) -> None:
self.assertTrue(SCRIPT_BEARING, "no skills found -- the anchor is wrong")
for rule, check in structure.CHECKS.items():
for skill in SCRIPT_BEARING:
with self.subTest(rule=rule, skill=skill.name):
problems = (
check(skill, KNOWN_SKILLS)
if rule == "local_links_resolve"
else check(skill)
)
self.assertEqual(problems, [])
class SharedCopyTests(unittest.TestCase):
"""Files several skills ship identical copies of must not drift apart."""
maxDiff = None
def test_docx_pptx_xlsx_ship_the_same_office_tree(self) -> None:
self.assertEqual(office.identical_tree_problems(SKILLS_DIR), [])
def test_shared_scripts_are_identical_across_their_skills(self) -> None:
self.assertEqual(office.shared_file_problems(SKILLS_DIR), [])
if __name__ == "__main__":
unittest.main()

9
tests/analytical-method-validation/test_scripts.py
View File

@@ -17,6 +17,8 @@ import sys
import unittest
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "analytical-method-validation"
SCRIPTS_DIR = SKILL_ROOT / "scripts"
FIXTURES = Path(__file__).resolve().parent / "fixtures"
@@ -58,7 +60,7 @@ class TestSkillStructure(unittest.TestCase):
end = text.index("\n---\n", 4)
front = text[4:end]
self.assertIn("name: analytical-method-validation", front)
self.assertIn('version: "1.0"', front)
self.assertRegex(front, r'\n version: "\d+\.\d+"\n')
# allowed-tools must be a space-separated string, not a YAML list
for line in front.splitlines():
if line.startswith("allowed-tools:"):
@@ -945,5 +947,10 @@ class TestInputHandling(unittest.TestCase):
self.assertIn("note:", res.stderr)
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
if __name__ == "__main__":
unittest.main()

461
tests/arbor/test_scripts.py Normal file
View File

@@ -0,0 +1,461 @@
"""Tests for the Arbor hypothesis-tree state manager.
`tree.py` is the durable state of an Autonomous Optimization run, so the tests
drive it the way a coordinator does -- through the parser, one subcommand at a
time, against a real `.arbor/` directory in a temporary run dir -- and assert
on the JSON that lands on disk. Testing the persisted state rather than return
values is deliberate: the script's contract *is* the file it writes, and a
coordinator resuming a run reads nothing else.
The behaviours worth pinning are the ones a corrupted run would hinge on: the
merge gate honouring `metric_direction`, pruning stopping at merged nodes, and
`validate` catching an inconsistent tree.
"""
from __future__ import annotations
import io
import json
import sys
import tempfile
import unittest
from contextlib import redirect_stdout, redirect_stderr
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "arbor"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import tree as arbor_tree # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
class ArborRunTestCase(unittest.TestCase):
"""A temporary run directory plus helpers for driving the CLI."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.run_dir = Path(self._temporary.name)
self.parser = arbor_tree.build_parser()
def run_command(self, *argv: str) -> tuple[str, str]:
"""Invoke one subcommand; return its (stdout, stderr)."""
args = self.parser.parse_args(["--run-dir", str(self.run_dir), *argv])
out, err = io.StringIO(), io.StringIO()
with redirect_stdout(out), redirect_stderr(err):
args.func(args)
return out.getvalue(), err.getvalue()
def init(self, **overrides: str) -> None:
argv = [
"init",
"--objective", overrides.get("objective", "Improve F1 on the dev split"),
"--dev-eval", "python eval.py --split dev",
"--test-eval", "python eval.py --split test",
]
for flag in ("metric-direction", "max-depth", "budget", "branching"):
if flag.replace("-", "_") in overrides:
argv += [f"--{flag}", str(overrides[flag.replace('-', '_')])]
self.run_command(*argv)
@property
def tree(self) -> dict:
return json.loads((self.run_dir / ".arbor" / "tree.json").read_text())
@property
def run_config(self) -> dict:
return json.loads((self.run_dir / ".arbor" / "run.json").read_text())
class InitTests(ArborRunTestCase):
def test_init_creates_both_state_files_with_a_root_node(self) -> None:
self.init()
self.assertTrue((self.run_dir / ".arbor" / "tree.json").is_file())
self.assertTrue((self.run_dir / ".arbor" / "run.json").is_file())
tree = self.tree
self.assertEqual(tree["root"], "n0")
root = tree["nodes"]["n0"]
self.assertEqual(root["status"], "root")
self.assertIsNone(root["parent"])
self.assertEqual(root["depth"], 0)
self.assertEqual(root["hypothesis"], "Improve F1 on the dev split")
def test_run_config_records_the_budget_and_starts_unspent(self) -> None:
self.init(budget=7, max_depth=3, branching=5)
run = self.run_config
self.assertEqual(run["budget_cycles"], 7)
self.assertEqual(run["max_depth"], 3)
self.assertEqual(run["branching"], 5)
self.assertEqual(run["cycles_used"], 0)
self.assertIsNone(run["best_node"])
self.assertIsNone(run["best_test_score"])
def test_init_refuses_to_erase_an_existing_run(self) -> None:
self.init()
with self.assertRaises(SystemExit) as raised:
self.init()
self.assertIn("--force", str(raised.exception))
def test_force_overwrites_and_resets_the_counter(self) -> None:
self.init()
self.run_command("add-node", "--parent", "n0", "--hypothesis", "h")
self.assertEqual(len(self.tree["nodes"]), 2)
args = self.parser.parse_args(
[
"--run-dir", str(self.run_dir),
"init",
"--objective", "A fresh objective",
"--dev-eval", "d",
"--test-eval", "t",
"--force",
]
)
with redirect_stdout(io.StringIO()):
args.func(args)
self.assertEqual(list(self.tree["nodes"]), ["n0"])
self.assertEqual(self.tree["_counter"], 0)
def test_commands_before_init_explain_how_to_start(self) -> None:
with self.assertRaises(SystemExit) as raised:
self.run_command("observe")
self.assertIn("tree.py init", str(raised.exception))
class NodeTests(ArborRunTestCase):
def setUp(self) -> None:
super().setUp()
self.init(max_depth=2)
def test_ids_are_sequential_and_depth_is_derived_from_the_parent(self) -> None:
self.run_command("add-node", "--parent", "n0", "--hypothesis", "direction A")
self.run_command("add-node", "--parent", "n0", "--hypothesis", "direction B")
self.run_command("add-node", "--parent", "n1", "--hypothesis", "intervention A1")
nodes = self.tree["nodes"]
self.assertEqual(sorted(nodes), ["n0", "n1", "n2", "n3"])
self.assertEqual(nodes["n1"]["depth"], 1)
self.assertEqual(nodes["n2"]["depth"], 1)
self.assertEqual(nodes["n3"]["depth"], 2)
self.assertEqual(nodes["n3"]["parent"], "n1")
def test_new_nodes_start_pending_with_empty_evidence(self) -> None:
self.run_command("add-node", "--parent", "n0", "--hypothesis", "h")
node = self.tree["nodes"]["n1"]
self.assertEqual(node["status"], "pending")
self.assertEqual(node["insight"], "")
self.assertIsNone(node["metadata"]["dev_score"])
self.assertIsNone(node["metadata"]["branch_ref"])
def test_depth_one_is_a_direction_and_deeper_is_an_intervention(self) -> None:
out, _ = self.run_command("add-node", "--parent", "n0", "--hypothesis", "h")
self.assertIn("direction node n1", out)
out, _ = self.run_command("add-node", "--parent", "n1", "--hypothesis", "h")
self.assertIn("intervention node n2", out)
def test_exceeding_max_depth_warns_but_still_records(self) -> None:
self.run_command("add-node", "--parent", "n0", "--hypothesis", "h")
self.run_command("add-node", "--parent", "n1", "--hypothesis", "h")
_, err = self.run_command("add-node", "--parent", "n2", "--hypothesis", "too deep")
self.assertIn("exceeds max_depth", err)
self.assertIn("n3", self.tree["nodes"])
def test_an_unknown_parent_is_refused_with_a_pointer_to_status(self) -> None:
with self.assertRaises(SystemExit) as raised:
self.run_command("add-node", "--parent", "n99", "--hypothesis", "h")
self.assertIn("tree.py status", str(raised.exception))
def test_status_must_come_from_the_documented_set(self) -> None:
self.run_command("add-node", "--parent", "n0", "--hypothesis", "h")
self.run_command("set-status", "--node", "n1", "--status", "running")
self.assertEqual(self.tree["nodes"]["n1"]["status"], "running")
with self.assertRaises(SystemExit):
self.run_command("set-status", "--node", "n1", "--status", "abandoned")
self.assertEqual(self.tree["nodes"]["n1"]["status"], "running")
class EvidenceTests(ArborRunTestCase):
def setUp(self) -> None:
super().setUp()
self.init()
self.run_command("add-node", "--parent", "n0", "--hypothesis", "direction")
self.run_command("add-node", "--parent", "n1", "--hypothesis", "intervention")
def test_evidence_defaults_the_status_to_executed(self) -> None:
self.run_command(
"set-evidence", "--node", "n2", "--dev-score", "0.81", "--result", "ran clean"
)
node = self.tree["nodes"]["n2"]
self.assertEqual(node["status"], "executed")
self.assertEqual(node["metadata"]["dev_score"], 0.81)
self.assertEqual(node["metadata"]["result"], "ran clean")
def test_an_explicit_status_overrides_the_default(self) -> None:
self.run_command(
"set-evidence", "--node", "n2", "--dev-score", "0.1", "--status", "pruned"
)
self.assertEqual(self.tree["nodes"]["n2"]["status"], "pruned")
def test_a_partial_update_leaves_untouched_fields_alone(self) -> None:
self.run_command(
"set-evidence",
"--node", "n2",
"--dev-score", "0.5",
"--result", "first pass",
"--branch-ref", "wt/n2",
)
self.run_command("set-evidence", "--node", "n2", "--dev-score", "0.6")
metadata = self.tree["nodes"]["n2"]["metadata"]
self.assertEqual(metadata["dev_score"], 0.6)
self.assertEqual(metadata["result"], "first pass")
self.assertEqual(metadata["branch_ref"], "wt/n2")
def test_a_leaf_with_ancestors_is_reminded_to_propagate(self) -> None:
out, _ = self.run_command("set-evidence", "--node", "n2", "--insight", "lr matters")
self.assertIn("tree.py propagate", out)
self.assertIn("['n1', 'n0']", out)
def test_a_zero_dev_score_is_recorded_rather_than_treated_as_absent(self) -> None:
self.run_command("set-evidence", "--node", "n2", "--dev-score", "0")
self.assertEqual(self.tree["nodes"]["n2"]["metadata"]["dev_score"], 0.0)
class PropagateTests(ArborRunTestCase):
def setUp(self) -> None:
super().setUp()
self.init()
self.run_command("add-node", "--parent", "n0", "--hypothesis", "direction")
self.run_command("add-node", "--parent", "n1", "--hypothesis", "intervention")
def test_default_propagation_reaches_only_the_immediate_parent(self) -> None:
self.run_command("propagate", "--node", "n2", "--insight", "small batches help")
nodes = self.tree["nodes"]
self.assertIn("[from n2] small batches help", nodes["n1"]["insight"])
self.assertEqual(nodes["n0"]["insight"], "")
def test_to_root_reaches_every_ancestor(self) -> None:
self.run_command(
"propagate", "--node", "n2", "--insight", "generalises", "--to-root"
)
nodes = self.tree["nodes"]
self.assertIn("generalises", nodes["n1"]["insight"])
self.assertIn("generalises", nodes["n0"]["insight"])
def test_insights_accumulate_rather_than_overwrite(self) -> None:
self.run_command("propagate", "--node", "n2", "--insight", "first lesson")
self.run_command("propagate", "--node", "n2", "--insight", "second lesson")
insight = self.tree["nodes"]["n1"]["insight"]
self.assertIn("first lesson", insight)
self.assertIn("second lesson", insight)
self.assertEqual(len(insight.splitlines()), 2)
def test_every_line_records_which_node_it_came_from(self) -> None:
self.run_command("propagate", "--node", "n2", "--insight", "lesson")
self.assertTrue(self.tree["nodes"]["n1"]["insight"].startswith("[from n2]"))
def test_the_root_has_no_ancestors_to_propagate_to(self) -> None:
with self.assertRaises(SystemExit) as raised:
self.run_command("propagate", "--node", "n0", "--insight", "x")
self.assertIn("no ancestors", str(raised.exception))
class PruneTests(ArborRunTestCase):
def setUp(self) -> None:
super().setUp()
self.init(max_depth=4)
self.run_command("add-node", "--parent", "n0", "--hypothesis", "direction")
self.run_command("add-node", "--parent", "n1", "--hypothesis", "child")
self.run_command("add-node", "--parent", "n2", "--hypothesis", "grandchild")
def test_pruning_takes_the_whole_subtree(self) -> None:
self.run_command("prune", "--node", "n1", "--reason", "dead end")
statuses = {nid: n["status"] for nid, n in self.tree["nodes"].items()}
self.assertEqual(statuses["n1"], "pruned")
self.assertEqual(statuses["n2"], "pruned")
self.assertEqual(statuses["n3"], "pruned")
self.assertEqual(statuses["n0"], "root")
def test_the_reason_is_recorded_on_the_target_only(self) -> None:
self.run_command("prune", "--node", "n1", "--reason", "falsified by n3")
nodes = self.tree["nodes"]
self.assertEqual(nodes["n1"]["metadata"]["prune_reason"], "falsified by n3")
self.assertNotIn("prune_reason", nodes["n2"]["metadata"])
def test_a_merged_node_survives_a_prune_of_its_ancestor(self) -> None:
# A merged node is already in M_best; pruning its parent must not
# retroactively mark the promoted work as a dead end.
self.run_command("set-status", "--node", "n2", "--status", "merged")
self.run_command("prune", "--node", "n1")
statuses = {nid: n["status"] for nid, n in self.tree["nodes"].items()}
self.assertEqual(statuses["n1"], "pruned")
self.assertEqual(statuses["n2"], "merged")
# The traversal stops at the merged node, so its child is untouched too.
self.assertEqual(statuses["n3"], "pending")
def test_pruning_the_root_is_a_no_op(self) -> None:
self.run_command("prune", "--node", "n0")
self.assertEqual(self.tree["nodes"]["n0"]["status"], "root")
class MergeGateTests(ArborRunTestCase):
def _prepare(self, direction: str = "max") -> None:
self.init(metric_direction=direction)
self.run_command("add-node", "--parent", "n0", "--hypothesis", "direction")
self.run_command("add-node", "--parent", "n0", "--hypothesis", "rival")
def test_the_first_candidate_always_passes(self) -> None:
self._prepare()
out, _ = self.run_command(
"merge", "--node", "n1", "--test-score", "0.7", "--branch-ref", "wt/n1"
)
self.assertIn("MERGE GATE PASSED", out)
self.assertEqual(self.run_config["best_node"], "n1")
self.assertEqual(self.run_config["best_test_score"], 0.7)
self.assertEqual(self.run_config["best_branch_ref"], "wt/n1")
self.assertEqual(self.tree["nodes"]["n1"]["status"], "merged")
def test_a_worse_candidate_is_rejected_and_leaves_m_best_alone(self) -> None:
self._prepare()
self.run_command("merge", "--node", "n1", "--test-score", "0.7")
out, _ = self.run_command("merge", "--node", "n2", "--test-score", "0.6")
self.assertIn("MERGE GATE REJECTED", out)
self.assertEqual(self.run_config["best_node"], "n1")
self.assertEqual(self.run_config["best_test_score"], 0.7)
self.assertEqual(self.tree["nodes"]["n2"]["status"], "pending")
# The test score is still recorded -- a rejection is evidence.
self.assertEqual(self.tree["nodes"]["n2"]["metadata"]["test_score"], 0.6)
def test_minimisation_runs_invert_the_comparison(self) -> None:
self._prepare(direction="min")
self.run_command("merge", "--node", "n1", "--test-score", "0.7")
out, _ = self.run_command("merge", "--node", "n2", "--test-score", "0.6")
self.assertIn("MERGE GATE PASSED", out)
self.assertEqual(self.run_config["best_node"], "n2")
self.assertEqual(self.run_config["best_test_score"], 0.6)
def test_an_exact_tie_does_not_displace_the_incumbent(self) -> None:
self._prepare()
self.run_command("merge", "--node", "n1", "--test-score", "0.7")
out, _ = self.run_command("merge", "--node", "n2", "--test-score", "0.7")
self.assertIn("MERGE GATE REJECTED", out)
self.assertEqual(self.run_config["best_node"], "n1")
def test_the_node_branch_ref_is_used_when_none_is_supplied(self) -> None:
self._prepare()
self.run_command("set-evidence", "--node", "n1", "--branch-ref", "wt/from-evidence")
self.run_command("merge", "--node", "n1", "--test-score", "0.7")
self.assertEqual(self.run_config["best_branch_ref"], "wt/from-evidence")
def test_a_negative_score_can_still_be_the_first_best(self) -> None:
# `better()` special-cases only `old is None`, not falsiness.
self._prepare()
self.run_command("merge", "--node", "n1", "--test-score", "-3.5")
self.assertEqual(self.run_config["best_test_score"], -3.5)
class CycleTests(ArborRunTestCase):
def test_cycles_count_up_and_report_the_remainder(self) -> None:
self.init(budget=2)
out, _ = self.run_command("cycle")
self.assertIn("Cycle 1/2 (1 remaining)", out)
out, _ = self.run_command("cycle")
self.assertIn("Cycle 2/2 (0 remaining)", out)
self.assertIn("Budget exhausted", out)
self.assertEqual(self.run_config["cycles_used"], 2)
def test_the_counter_keeps_going_past_the_budget(self) -> None:
self.init(budget=1)
self.run_command("cycle")
out, _ = self.run_command("cycle")
self.assertIn("Budget exhausted", out)
self.assertEqual(self.run_config["cycles_used"], 2)
class ValidateTests(ArborRunTestCase):
def setUp(self) -> None:
super().setUp()
self.init()
self.run_command("add-node", "--parent", "n0", "--hypothesis", "direction")
def _write_tree(self, tree: dict) -> None:
(self.run_dir / ".arbor" / "tree.json").write_text(json.dumps(tree))
def test_a_healthy_tree_validates(self) -> None:
out, _ = self.run_command("validate")
self.assertIn("OK", out)
self.assertIn("2 nodes", out)
def test_a_dangling_parent_is_caught(self) -> None:
tree = self.tree
tree["nodes"]["n1"]["parent"] = "n42"
self._write_tree(tree)
with self.assertRaises(SystemExit):
self.run_command("validate")
def test_an_invalid_status_is_caught(self) -> None:
tree = self.tree
tree["nodes"]["n1"]["status"] = "hallucinated"
self._write_tree(tree)
with self.assertRaises(SystemExit):
self.run_command("validate")
def test_a_best_node_that_was_never_merged_is_caught(self) -> None:
run = self.run_config
run["best_node"] = "n1"
(self.run_dir / ".arbor" / "run.json").write_text(json.dumps(run))
with self.assertRaises(SystemExit):
self.run_command("validate")
def test_a_merged_best_node_validates(self) -> None:
self.run_command("merge", "--node", "n1", "--test-score", "0.9")
out, _ = self.run_command("validate")
self.assertIn("OK", out)
self.assertIn("best=n1", out)
class ProjectionTests(ArborRunTestCase):
def test_observe_reports_the_objective_and_every_node(self) -> None:
self.init(objective="Reduce inference latency")
self.run_command("add-node", "--parent", "n0", "--hypothesis", "quantise weights")
self.run_command("set-evidence", "--node", "n1", "--dev-score", "0.42")
out, _ = self.run_command("observe")
self.assertIn("Reduce inference latency", out)
self.assertIn("quantise weights", out)
self.assertIn("0.42", out)
def test_status_renders_the_tree_without_mutating_it(self) -> None:
self.init()
self.run_command("add-node", "--parent", "n0", "--hypothesis", "a direction")
before = self.tree
out, _ = self.run_command("status")
self.assertIn("a direction", out)
self.assertEqual(self.tree, before)
class PersistenceTests(ArborRunTestCase):
def test_writes_leave_no_temporary_file_behind(self) -> None:
# _save writes to a .tmp sibling and replaces, so a crash cannot leave
# a half-written tree.json -- but it must also clean up on success.
self.init()
self.run_command("add-node", "--parent", "n0", "--hypothesis", "h")
leftovers = sorted(p.name for p in (self.run_dir / ".arbor").iterdir())
self.assertEqual(leftovers, ["run.json", "tree.json"])
def test_state_survives_a_fresh_read(self) -> None:
self.init()
self.run_command("add-node", "--parent", "n0", "--hypothesis", "persisted")
reloaded = json.loads((self.run_dir / ".arbor" / "tree.json").read_text())
self.assertEqual(reloaded["nodes"]["n1"]["hypothesis"], "persisted")
if __name__ == "__main__":
unittest.main()

127
tests/arboreto/test_scripts.py Normal file
View File

@@ -0,0 +1,127 @@
"""Tests for the Arboreto GRN inference wrapper.
GRNBoost2 on real data is a distributed, minutes-long job, so the tests stub
`grnboost2` and assert on the contract between the wrapper and the algorithm:
the expression matrix is read genes-as-columns, `tf_names` defaults to the
sentinel `'all'` rather than an empty list, the seed and limit are forwarded,
and the network is written headerless -- the format every downstream Arboreto
consumer expects.
Getting `tf_names` wrong is the failure worth guarding: passing `[]` instead of
`'all'` produces an empty network with no error at all.
"""
from __future__ import annotations
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "arboreto"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pd = pytest.importorskip("pandas", reason="arboreto needs pandas")
pytest.importorskip("arboreto", reason="basic_grn_inference imports arboreto at module scope")
import basic_grn_inference # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
NETWORK = pd.DataFrame(
{
"TF": ["TF1", "TF1", "TF2"],
"target": ["G1", "G2", "G1"],
"importance": [9.5, 4.2, 1.1],
}
)
class InferenceWrapperTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.expression = self.root / "expression.tsv"
self.expression.write_text(
"TF1\tTF2\tG1\tG2\n"
"1.0\t2.0\t3.0\t4.0\n"
"1.5\t2.5\t3.5\t4.5\n"
"2.0\t3.0\t4.0\t5.0\n",
encoding="utf-8",
)
self.output = self.root / "network.tsv"
def run_inference(self, **kwargs):
with mock.patch.object(
basic_grn_inference, "grnboost2", return_value=NETWORK
) as algorithm:
basic_grn_inference.run_grn_inference(
str(self.expression), str(self.output), **kwargs
)
return algorithm
def test_the_expression_matrix_is_read_with_genes_as_columns(self) -> None:
algorithm = self.run_inference()
passed = algorithm.call_args.kwargs["expression_data"]
self.assertEqual(list(passed.columns), ["TF1", "TF2", "G1", "G2"])
self.assertEqual(len(passed), 3) # three observations
def test_without_a_tf_file_every_gene_is_a_candidate_regulator(self) -> None:
# The sentinel is the string 'all'; an empty list would silently return
# an empty network.
algorithm = self.run_inference()
self.assertEqual(algorithm.call_args.kwargs["tf_names"], "all")
def test_a_tf_file_restricts_the_candidate_regulators(self) -> None:
tf_file = self.root / "tfs.txt"
tf_file.write_text("TF1\nTF2\n", encoding="utf-8")
algorithm = self.run_inference(tf_file=str(tf_file))
self.assertEqual(list(algorithm.call_args.kwargs["tf_names"]), ["TF1", "TF2"])
def test_the_seed_is_forwarded_so_runs_are_reproducible(self) -> None:
self.assertEqual(self.run_inference().call_args.kwargs["seed"], 777)
self.assertEqual(self.run_inference(seed=42).call_args.kwargs["seed"], 42)
def test_the_link_limit_is_forwarded_and_defaults_to_unlimited(self) -> None:
self.assertIsNone(self.run_inference().call_args.kwargs["limit"])
self.assertEqual(self.run_inference(limit=100).call_args.kwargs["limit"], 100)
def test_the_network_is_written_headerless_and_tab_separated(self) -> None:
# Arboreto's own consumers expect three unlabelled columns.
self.run_inference()
lines = self.output.read_text(encoding="utf-8").strip().splitlines()
self.assertEqual(len(lines), 3)
self.assertNotIn("importance", lines[0])
self.assertEqual(lines[0].split("\t"), ["TF1", "G1", "9.5"])
def test_no_index_column_is_written(self) -> None:
self.run_inference()
for line in self.output.read_text(encoding="utf-8").strip().splitlines():
with self.subTest(line=line):
self.assertEqual(len(line.split("\t")), 3)
class ParserTests(unittest.TestCase):
def test_the_documented_flags_are_all_accepted(self) -> None:
source = (SCRIPTS / "basic_grn_inference.py").read_text(encoding="utf-8")
for flag in ("--tf-file", "--seed", "--limit"):
with self.subTest(flag=flag):
self.assertIn(flag, source)
def test_the_positional_arguments_are_required(self) -> None:
result = skill_contract.cli.run_help(SCRIPTS / "basic_grn_inference.py")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("expression_file", result.stdout)
self.assertIn("output_file", result.stdout)
if __name__ == "__main__":
unittest.main()

9
tests/autoskill/test_cli.py
View File

@@ -1,7 +1,11 @@
from pathlib import Path
import pytest
import autoskill
import skill_contract
def test_dispatches_to_run_subcommand(monkeypatch):
calls = {}
@@ -59,3 +63,8 @@ def test_missing_subcommand_errors_out():
def test_unknown_subcommand_errors_out():
with pytest.raises(SystemExit):
autoskill.main(["nope"])
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(Path(__file__).resolve().parents[2] / "skills" / "autoskill")

147
tests/bids/test_scripts.py Normal file
View File

@@ -0,0 +1,147 @@
"""Tests for the BIDS schema updater.
`update_schema.py` is a maintenance script that overwrites files in the skill's
own `references/` directory from the network. Neither of those is acceptable in
a test, so every test here redirects `REFERENCES_DIR` at a temporary directory
and replaces `fetch` with a stub. Nothing reaches the network and nothing
touches the shipped references.
The other half of the suite checks the artefacts that *are* shipped: a schema
the script wrote once must still be parseable and carry its version fields,
because the skill's documentation quotes them.
"""
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "bids"
SCRIPTS = SKILL_ROOT / "scripts"
REFERENCES = SKILL_ROOT / "references"
sys.path.insert(0, str(SCRIPTS))
import update_schema # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
class TemporaryReferencesTestCase(unittest.TestCase):
"""Point the script's output directory at a scratch dir for the test."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.references = Path(self._temporary.name)
patcher = mock.patch.object(update_schema, "REFERENCES_DIR", self.references)
patcher.start()
self.addCleanup(patcher.stop)
class SchemaWriteTests(TemporaryReferencesTestCase):
SCHEMA = {
"schema_version": "1.2.1",
"bids_version": "1.11.1",
"objects": {"entities": {"subject": {"name": "sub"}}},
}
def test_downloaded_schema_is_reserialised_with_stable_formatting(self) -> None:
# Upstream ships minified JSON; the script re-indents so the checked-in
# copy diffs sanely on the next update.
minified = json.dumps(self.SCHEMA, separators=(",", ":")).encode("utf-8")
with mock.patch.object(update_schema, "fetch", return_value=minified):
update_schema.update_schema("https://example.invalid/schema.json")
written = self.references / "bids_schema.json"
text = written.read_text(encoding="utf-8")
self.assertEqual(json.loads(text), self.SCHEMA)
self.assertIn("\n ", text, "output should be indented")
self.assertTrue(text.endswith("\n"), "output should end with a newline")
def test_the_reported_versions_come_from_the_payload(self) -> None:
payload = json.dumps(self.SCHEMA).encode("utf-8")
with mock.patch.object(update_schema, "fetch", return_value=payload):
with mock.patch("builtins.print") as printed:
update_schema.update_schema("https://example.invalid/schema.json")
output = " ".join(str(call.args[0]) for call in printed.call_args_list)
self.assertIn("schema 1.2.1", output)
self.assertIn("BIDS 1.11.1", output)
def test_a_schema_without_version_fields_reports_unknown(self) -> None:
with mock.patch.object(update_schema, "fetch", return_value=b"{}"):
with mock.patch("builtins.print") as printed:
update_schema.update_schema("https://example.invalid/schema.json")
output = " ".join(str(call.args[0]) for call in printed.call_args_list)
self.assertIn("schema ? / BIDS ?", output)
def test_a_non_json_response_fails_before_anything_is_written(self) -> None:
with mock.patch.object(update_schema, "fetch", return_value=b"<html>404</html>"):
with self.assertRaises(json.JSONDecodeError):
update_schema.update_schema("https://example.invalid/schema.json")
self.assertFalse((self.references / "bids_schema.json").exists())
class BepsWriteTests(TemporaryReferencesTestCase):
def test_beps_are_written_verbatim_and_counted(self) -> None:
payload = (
b"# template\n"
b"- number: '004'\n title: Diffusion\n"
b"- number: '011'\n title: Structural\n"
)
with mock.patch.object(update_schema, "fetch", return_value=payload):
with mock.patch("builtins.print") as printed:
update_schema.update_beps()
written = self.references / "beps.yml"
self.assertEqual(written.read_bytes(), payload)
output = " ".join(str(call.args[0]) for call in printed.call_args_list)
self.assertIn("2 BEPs", output)
def test_the_counter_matches_the_format_of_the_shipped_file(self) -> None:
# The count is a byte-substring search, so it only stays correct while
# upstream keeps this exact indentation. Pin it against the real file.
shipped = (REFERENCES / "beps.yml").read_bytes()
self.assertEqual(shipped.count(b"\n- number:"), 25)
class FetchTests(unittest.TestCase):
def test_requests_carry_an_identifying_user_agent(self) -> None:
response = mock.MagicMock()
response.__enter__.return_value.read.return_value = b"{}"
with mock.patch("urllib.request.urlopen", return_value=response) as opened:
update_schema.fetch("https://example.invalid/x.json")
request = opened.call_args.args[0]
self.assertEqual(request.full_url, "https://example.invalid/x.json")
self.assertIn("bids-skill-updater", request.get_header("User-agent"))
def test_the_default_sources_are_https_and_upstream(self) -> None:
self.assertTrue(update_schema.SCHEMA_URL.startswith("https://"))
self.assertTrue(update_schema.BEPS_URL.startswith("https://"))
self.assertIn("bids-specification", update_schema.SCHEMA_URL)
self.assertIn("bids-standard", update_schema.BEPS_URL)
class ShippedReferenceTests(unittest.TestCase):
"""The artefacts the script produced last time it ran must still be usable."""
def test_the_shipped_schema_parses_and_is_versioned(self) -> None:
schema = json.loads((REFERENCES / "bids_schema.json").read_text(encoding="utf-8"))
self.assertRegex(schema["schema_version"], r"^\d+\.\d+\.\d+$")
self.assertRegex(schema["bids_version"], r"^\d+\.\d+\.\d+$")
self.assertIn("objects", schema)
def test_the_references_directory_is_where_the_script_writes(self) -> None:
self.assertEqual(update_schema.REFERENCES_DIR, REFERENCES)
self.assertTrue((REFERENCES / "beps.yml").is_file())
self.assertTrue((REFERENCES / "bids_schema.json").is_file())
if __name__ == "__main__":
unittest.main()

938
tests/bioservices/test_scripts.py Normal file
View File

@@ -0,0 +1,938 @@
"""Tests for the BioServices cross-database workflow scripts.
BioServices *is* the network, so every service class is replaced here and the
tests assert on the request that would have been issued and on the parsing of a
canned response. Nothing in this file opens a connection.
The parsers are where these scripts can be quietly wrong, so the fixtures are
real records with published values: the KEGG flat file for ATP (C00002) carries
formula C10H16N5O13P3, exact mass 506.9957 and ChEBI 15422, and the parser must
pull exactly those out -- and must not mistake the indented DBLINKS lines that
follow the PATHWAY block for pathways.
The rest of the coverage is the batch machinery: chunking a large identifier
list into requests of the requested size, retrying a failed chunk one identifier
at a time, marking everything that never came back as failed, and writing a CSV
whose "Failed" rows really are the unmapped ones. Alias normalisation is tested
in both directions -- an alias must resolve and an already-official code must
survive untouched.
"""
from __future__ import annotations
import csv
import io
import sys
import tempfile
import unittest
from contextlib import redirect_stdout
from pathlib import Path
from unittest.mock import patch
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "bioservices"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pytest.importorskip("bioservices", reason="bioservices skill needs bioservices")
import batch_id_converter as converter # noqa: E402
import compound_cross_reference as compound # noqa: E402
import pathway_analysis as pathways # noqa: E402
import protein_analysis_workflow as workflow # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
# KEGG's flat file for ATP, trimmed to the fields the parser reads. Field names
# start in column 1; continuation lines are indented 12 spaces.
ATP_ENTRY = """\
ENTRY C00002 Compound
NAME ATP;
Adenosine 5'-triphosphate
FORMULA C10H16N5O13P3
EXACT_MASS 506.9957
MOL_WEIGHT 507.181
REACTION R00002 R00076
PATHWAY map00190 Oxidative phosphorylation
map00230 Purine metabolism
map00730 Thiamine metabolism
ENZYME 1.1.98.6 1.2.1.101
DBLINKS CAS: 56-65-5
PubChem: 3304
ChEBI: 15422
KNApSAcK: C00001491
PDB-CCD: ATP
ATOM 31
///
"""
def quietly(function, *args, **kwargs):
"""Run one of these very chatty functions without its progress output."""
with redirect_stdout(io.StringIO()):
return function(*args, **kwargs)
class DatabaseCodeTests(unittest.TestCase):
"""Aliases must resolve, and official codes must pass through untouched."""
def test_lowercase_aliases_resolve_to_the_official_code(self) -> None:
self.assertEqual(converter.normalize_database_code("uniprot"), "UniProtKB_AC-ID")
self.assertEqual(converter.normalize_database_code("entrez"), "GeneID")
self.assertEqual(converter.normalize_database_code("refseq"), "RefSeq_Protein")
def test_aliases_are_matched_case_insensitively(self) -> None:
self.assertEqual(converter.normalize_database_code("UniProt"), "UniProtKB_AC-ID")
self.assertEqual(converter.normalize_database_code("ENSEMBL"), "Ensembl")
def test_an_official_code_survives_normalisation(self) -> None:
# Rewriting a valid code would break the mapping request.
for code in ("UniProtKB_AC-ID", "Ensembl_Protein", "RefSeq_Protein", "GO"):
with self.subTest(code=code):
self.assertEqual(converter.normalize_database_code(code), code)
def test_every_alias_maps_to_a_code_that_is_itself_stable(self) -> None:
# Normalisation must be idempotent, or a second pass would corrupt it.
for alias, code in converter.DATABASE_CODES.items():
with self.subTest(alias=alias):
self.assertEqual(converter.normalize_database_code(code), code)
def test_an_unknown_code_is_passed_through_rather_than_rejected(self) -> None:
# UniProt supports far more codes than the alias table lists.
self.assertEqual(converter.normalize_database_code("Ensembl_Genomes"),
"Ensembl_Genomes")
class IdentifierFileTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def read(self, text: str):
path = self.root / "ids.txt"
path.write_text(text, encoding="utf-8")
return quietly(converter.read_ids_from_file, path)
def test_identifiers_are_read_one_per_line_and_trimmed(self) -> None:
self.assertEqual(self.read("P43403\n P04637 \n"), ["P43403", "P04637"])
def test_comments_and_blank_lines_are_ignored(self) -> None:
self.assertEqual(
self.read("# kinases\nP43403\n\n#P00000\nP04637\n"),
["P43403", "P04637"],
)
def test_an_empty_file_yields_no_identifiers(self) -> None:
self.assertEqual(self.read("\n\n"), [])
def test_a_missing_file_raises(self) -> None:
with self.assertRaises(OSError):
converter.read_ids_from_file(self.root / "absent.txt")
class RecordingUniProt:
"""Stands in for `bioservices.UniProt`, recording every mapping request."""
def __init__(self, verbose: bool = True) -> None:
self.requests: list[dict] = []
RecordingUniProt.instances.append(self)
instances: list["RecordingUniProt"] = []
def mapping(self, fr: str, to: str, query: str):
self.requests.append({"fr": fr, "to": to, "query": query})
return RecordingUniProt.responder(query)
class BatchConversionTests(unittest.TestCase):
def setUp(self) -> None:
RecordingUniProt.instances = []
RecordingUniProt.responder = lambda query: {
identifier: [f"hsa:{identifier}"] for identifier in query.split(",")
}
patcher = patch.object(converter, "UniProt", RecordingUniProt)
patcher.start()
self.addCleanup(patcher.stop)
# Rate limiting is real behaviour, but the suite must not sleep for it.
sleeper = patch.object(converter.time, "sleep")
self.sleep = sleeper.start()
self.addCleanup(sleeper.stop)
@property
def requests(self) -> list[dict]:
self.assertEqual(len(RecordingUniProt.instances), 1)
return RecordingUniProt.instances[0].requests
def convert(self, ids, **kwargs):
return quietly(
converter.batch_convert, ids, "UniProtKB_AC-ID", "KEGG", **kwargs
)
def test_a_short_list_is_one_comma_separated_request(self) -> None:
mapping, failed = self.convert(["P43403", "P04637"])
self.assertEqual(len(self.requests), 1)
self.assertEqual(self.requests[0]["query"], "P43403,P04637")
self.assertEqual(self.requests[0]["fr"], "UniProtKB_AC-ID")
self.assertEqual(self.requests[0]["to"], "KEGG")
self.assertEqual(mapping["P43403"], ["hsa:P43403"])
self.assertEqual(failed, [])
def test_a_long_list_is_split_into_chunks_of_the_requested_size(self) -> None:
# 250 identifiers at 100 per request is 3 requests: 100, 100, 50.
identifiers = [f"P{index:05d}" for index in range(250)]
mapping, failed = self.convert(identifiers, chunk_size=100, delay=0)
self.assertEqual(len(self.requests), 3)
self.assertEqual(
[len(request["query"].split(",")) for request in self.requests],
[100, 100, 50],
)
# Every identifier appears exactly once across the chunks.
sent = [
identifier
for request in self.requests
for identifier in request["query"].split(",")
]
self.assertEqual(sent, identifiers)
self.assertEqual(len(mapping), 250)
self.assertEqual(failed, [])
def test_a_chunk_size_larger_than_the_list_still_sends_one_request(self) -> None:
self.convert(["P43403"], chunk_size=500)
self.assertEqual(len(self.requests), 1)
def test_identifiers_that_never_map_are_reported_as_failed(self) -> None:
RecordingUniProt.responder = lambda query: {"P43403": ["hsa:7535"]}
mapping, failed = self.convert(["P43403", "P99999"], delay=0)
self.assertEqual(mapping["P43403"], ["hsa:7535"])
# Absent from the response, so present in the result with no target --
# a silently dropped identifier would inflate the mapping rate.
self.assertIsNone(mapping["P99999"])
self.assertEqual(len(mapping), 2)
def test_an_empty_response_marks_the_whole_chunk_failed(self) -> None:
RecordingUniProt.responder = lambda query: {}
mapping, failed = self.convert(["P43403", "P04637"], delay=0)
self.assertEqual(sorted(failed), ["P04637", "P43403"])
self.assertTrue(all(value is None for value in mapping.values()))
def test_a_failed_chunk_is_retried_one_identifier_at_a_time(self) -> None:
# UniProt rejects a whole batch when one identifier in it is malformed,
# so the per-identifier retry is what rescues the rest.
def responder(query: str):
if "," in query:
raise RuntimeError("400 Bad Request")
if query == "BROKEN":
return {}
return {query: [f"hsa:{query}"]}
RecordingUniProt.responder = staticmethod(responder)
mapping, failed = self.convert(["P43403", "BROKEN", "P04637"], delay=0)
queries = [request["query"] for request in self.requests]
self.assertEqual(queries[0], "P43403,BROKEN,P04637") # the failed batch
self.assertEqual(queries[1:], ["P43403", "BROKEN", "P04637"])
self.assertEqual(mapping["P43403"], ["hsa:P43403"])
self.assertEqual(failed, ["BROKEN"])
def test_a_retry_that_also_fails_records_the_identifier_once(self) -> None:
def responder(query: str):
raise RuntimeError("service unavailable")
RecordingUniProt.responder = staticmethod(responder)
mapping, failed = self.convert(["P43403"], delay=0)
self.assertEqual(failed, ["P43403"])
self.assertIsNone(mapping["P43403"])
def test_the_delay_is_only_taken_between_chunks(self) -> None:
self.convert([f"P{index}" for index in range(3)], chunk_size=1, delay=0.5)
# Three chunks, two gaps.
self.assertEqual(
[call.args[0] for call in self.sleep.call_args_list], [0.5, 0.5]
)
def test_no_delay_is_taken_when_it_is_switched_off(self) -> None:
self.convert([f"P{index}" for index in range(3)], chunk_size=1, delay=0)
self.sleep.assert_not_called()
class MappingCsvTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.path = self.root / "mapping.csv"
quietly(
converter.save_mapping_csv,
{"P43403": ["hsa:7535", "mmu:22637"], "P99999": None, "P04637": ["hsa:7157"]},
self.path,
"UniProtKB_AC-ID",
"KEGG",
)
with self.path.open(newline="", encoding="utf-8") as handle:
self.rows = list(csv.reader(handle))
def test_the_header_is_the_documented_five_columns(self) -> None:
self.assertEqual(
self.rows[0],
["Source_ID", "Source_DB", "Target_IDs", "Target_DB", "Mapping_Status"],
)
def test_rows_are_sorted_by_source_identifier(self) -> None:
self.assertEqual(
[row[0] for row in self.rows[1:]], ["P04637", "P43403", "P99999"]
)
def test_multiple_targets_are_joined_with_semicolons(self) -> None:
# A comma would collide with the CSV delimiter.
row = next(row for row in self.rows if row[0] == "P43403")
self.assertEqual(row[2], "hsa:7535;mmu:22637")
self.assertEqual(row[4], "Success")
def test_an_unmapped_identifier_is_written_as_an_empty_failed_row(self) -> None:
row = next(row for row in self.rows if row[0] == "P99999")
self.assertEqual(row[2], "")
self.assertEqual(row[4], "Failed")
def test_the_source_and_target_databases_are_recorded_on_every_row(self) -> None:
for row in self.rows[1:]:
self.assertEqual((row[1], row[3]), ("UniProtKB_AC-ID", "KEGG"))
def test_failed_identifiers_are_saved_only_when_there_are_some(self) -> None:
target = self.root / "failed.txt"
quietly(converter.save_failed_ids, [], target)
self.assertFalse(target.exists())
quietly(converter.save_failed_ids, ["P99999", "P88888"], target)
self.assertEqual(
target.read_text(encoding="utf-8").split(), ["P99999", "P88888"]
)
class FakeKegg:
"""Records KEGG requests and replays canned flat-file responses."""
def __init__(self, find_result: str = "", entries: dict | None = None) -> None:
self.find_result = find_result
self.entries = entries or {}
self.requests: list[tuple[str, tuple]] = []
def find(self, database: str, query: str):
self.requests.append(("find", (database, query)))
return self.find_result
def get(self, identifier: str):
self.requests.append(("get", (identifier,)))
return self.entries.get(identifier, "")
class KeggCompoundSearchTests(unittest.TestCase):
def search(self, find_result: str):
fake = FakeKegg(find_result=find_result)
with patch.object(compound, "KEGG", lambda *a, **k: fake):
_, identifier = quietly(compound.search_kegg_compound, "ATP")
return fake, identifier
def test_the_compound_database_is_searched_by_name(self) -> None:
fake, _ = self.search("cpd:C00002\tATP; Adenosine 5'-triphosphate\n")
self.assertEqual(fake.requests[0], ("find", ("compound", "ATP")))
def test_the_cpd_prefix_is_stripped_from_the_first_hit(self) -> None:
# Downstream calls rebuild "cpd:<id>", so keeping the prefix here would
# produce "cpd:cpd:C00002".
_, identifier = self.search("cpd:C00002\tATP\ncpd:C00008\tADP\n")
self.assertEqual(identifier, "C00002")
def test_no_results_returns_no_identifier(self) -> None:
for empty in ("", "\n", " "):
with self.subTest(response=repr(empty)):
_, identifier = self.search(empty)
self.assertIsNone(identifier)
def test_a_service_error_returns_no_identifier_rather_than_raising(self) -> None:
class Broken:
def find(self, *args):
raise RuntimeError("KEGG is down")
with patch.object(compound, "KEGG", lambda *a, **k: Broken()):
_, identifier = quietly(compound.search_kegg_compound, "ATP")
self.assertIsNone(identifier)
class KeggEntryParsingTests(unittest.TestCase):
"""Parsed against the published values in the ATP entry above."""
def setUp(self) -> None:
self.kegg = FakeKegg(entries={"cpd:C00002": ATP_ENTRY})
self.info = quietly(compound.get_kegg_info, self.kegg, "C00002")
def test_the_entry_is_requested_with_the_cpd_prefix(self) -> None:
self.assertEqual(self.kegg.requests, [("get", ("cpd:C00002",))])
def test_the_published_formula_and_masses_are_extracted(self) -> None:
self.assertEqual(self.info["formula"], "C10H16N5O13P3")
self.assertEqual(self.info["exact_mass"], "506.9957")
self.assertEqual(self.info["mol_weight"], "507.181")
def test_the_name_loses_its_trailing_semicolon(self) -> None:
# KEGG separates synonyms with ";"; keeping it would corrupt the label.
self.assertEqual(self.info["name"], "ATP")
def test_the_chebi_identifier_is_pulled_out_of_dblinks(self) -> None:
# ATP is CHEBI:15422.
self.assertEqual(self.info["chebi_id"], "15422")
def test_only_the_pathway_block_becomes_pathways(self) -> None:
# Three PATHWAY lines. The indented DBLINKS lines that follow must not
# be collected, or the pathway count is inflated by database links.
self.assertEqual(len(self.info["pathways"]), 3)
self.assertEqual(self.info["pathways"][0], "map00190 Oxidative phosphorylation")
self.assertTrue(
all("PubChem" not in pathway for pathway in self.info["pathways"])
)
self.assertTrue(
all("KNApSAcK" not in pathway for pathway in self.info["pathways"])
)
def test_an_entry_without_a_pathway_block_has_no_pathways(self) -> None:
entry = "ENTRY C99999\nNAME Nothing\nFORMULA CH4\n///\n"
info = quietly(
compound.get_kegg_info, FakeKegg(entries={"cpd:C99999": entry}), "C99999"
)
self.assertEqual(info["pathways"], [])
self.assertIsNone(info["chebi_id"])
self.assertEqual(info["formula"], "CH4")
def test_an_empty_response_is_reported_as_no_information(self) -> None:
self.assertIsNone(quietly(compound.get_kegg_info, FakeKegg(), "C00002"))
class ChebiAndChemblTests(unittest.TestCase):
def test_a_bare_chebi_number_is_prefixed_before_the_request(self) -> None:
# The ChEBI service requires the "CHEBI:" namespace.
requested: list[str] = []
class FakeChebi:
def getCompleteEntity(self, identifier):
requested.append(identifier)
return type(
"Entity",
(),
{
"chebiId": "CHEBI:15422",
"chebiAsciiName": "ATP",
"Formulae": "C10H16N5O13P3",
"mass": "507.18100",
},
)()
with patch.object(compound, "ChEBI", lambda *a, **k: FakeChebi()):
info = quietly(compound.get_chebi_info, "15422")
self.assertEqual(requested, ["CHEBI:15422"])
self.assertEqual(info["chebi_id"], "CHEBI:15422")
self.assertEqual(info["name"], "ATP")
def test_an_already_prefixed_identifier_is_not_prefixed_twice(self) -> None:
requested: list[str] = []
class FakeChebi:
def getCompleteEntity(self, identifier):
requested.append(identifier)
return None
with patch.object(compound, "ChEBI", lambda *a, **k: FakeChebi()):
quietly(compound.get_chebi_info, "CHEBI:15422")
self.assertEqual(requested, ["CHEBI:15422"])
def test_no_chebi_identifier_means_no_request_at_all(self) -> None:
with patch.object(compound, "ChEBI") as chebi:
self.assertIsNone(quietly(compound.get_chebi_info, None))
chebi.assert_not_called()
def test_the_chembl_lookup_uses_the_current_method_name(self) -> None:
# `get_compound_by_chemblId` was removed after bioservices 1.6; the
# current release exposes `get_molecule`, and the installed class must
# actually have whatever the script calls.
from bioservices import ChEMBL
self.assertTrue(hasattr(ChEMBL, "get_molecule"))
requested: list[str] = []
class FakeChembl:
def get_molecule(self, identifier):
requested.append(identifier)
return {
"pref_name": "ASPIRIN",
"molecule_properties": {"full_mwt": "180.16", "alogp": "1.31"},
"molecule_structures": {"canonical_smiles": "CC(=O)Oc1ccccc1C(=O)O"},
}
with patch.object(compound, "ChEMBL", lambda *a, **k: FakeChembl()):
result = quietly(compound.get_chembl_info, "CHEMBL25")
self.assertEqual(requested, ["CHEMBL25"])
self.assertEqual(result["pref_name"], "ASPIRIN")
def test_no_chembl_identifier_means_no_request(self) -> None:
with patch.object(compound, "ChEMBL") as chembl:
self.assertIsNone(quietly(compound.get_chembl_info, None))
chembl.assert_not_called()
def test_a_chembl_error_is_swallowed_into_a_none_result(self) -> None:
class Broken:
def get_molecule(self, identifier):
raise RuntimeError("ChEMBL is down")
with patch.object(compound, "ChEMBL", lambda *a, **k: Broken()):
self.assertIsNone(quietly(compound.get_chembl_info, "CHEMBL25"))
class CompoundReportTests(unittest.TestCase):
def test_the_report_records_every_identifier_that_was_resolved(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "report.txt"
quietly(
compound.save_results,
"ATP",
{
"kegg_id": "C00002",
"name": "ATP",
"formula": "C10H16N5O13P3",
"exact_mass": "506.9957",
"mol_weight": "507.181",
"chebi_id": "15422",
"pathways": ["map00190", "map00230"],
},
"CHEMBL14249",
path,
)
text = path.read_text(encoding="utf-8")
self.assertIn("ATP", text)
self.assertIn("C10H16N5O13P3", text)
self.assertIn("KEGG: C00002", text)
self.assertIn("ChEBI: 15422", text)
self.assertIn("CHEMBL14249", text)
self.assertIn("Pathways: 2 found", text)
def test_a_report_without_kegg_information_still_writes(self) -> None:
# The compound may resolve in ChEMBL but not in KEGG.
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "report.txt"
quietly(compound.save_results, "Mystery", None, "CHEMBL1", path)
text = path.read_text(encoding="utf-8")
self.assertIn("Mystery", text)
self.assertIn("CHEMBL1", text)
self.assertNotIn("KEGG Compound\n", text)
class FakeKgmlKegg:
"""A KEGG stand-in for the KGML pathway walk."""
def __init__(self, parsed: dict | None = None, entry: str = "") -> None:
self.parsed = parsed
self.entry = entry
self.parsed_ids: list[str] = []
def parse_kgml_pathway(self, pathway_id: str):
self.parsed_ids.append(pathway_id)
if self.parsed is None:
raise RuntimeError("no KGML for this pathway")
return self.parsed
def get(self, pathway_id: str):
return self.entry
KGML = {
"entries": [{"id": "1", "gene_names": "TP53"}, {"id": "2", "gene_names": "MDM2"}],
"relations": [
{"entry1": "1", "entry2": "2", "name": "activation", "link": "PPrel"},
{"entry1": "2", "entry2": "1", "name": "inhibition", "link": "PPrel"},
{"entry1": "1", "entry2": "2", "name": "binding/association", "link": "PPrel"},
{"entry1": "1", "entry2": "2", "name": "activation", "link": "PPrel"},
{"entry1": "2", "entry2": "2", "name": "methylation", "link": "PPrel"},
],
}
class PathwayAnalysisTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def analyse(self, kegg, pathway_id: str = "hsa04115"):
return quietly(pathways.analyze_pathway, kegg, pathway_id)
def test_entries_and_relations_are_counted_and_typed(self) -> None:
kegg = FakeKgmlKegg(parsed=KGML, entry="NAME p53 signaling pathway\n")
result = self.analyse(kegg)
self.assertEqual(result["num_entries"], 2)
self.assertEqual(result["num_relations"], 5)
# Hand-counted from KGML above.
self.assertEqual(
result["relation_types"],
{"activation": 2, "inhibition": 1, "binding/association": 1,
"methylation": 1},
)
self.assertEqual(result["pathway_name"], "p53 signaling pathway")
def test_a_pathway_without_kgml_is_skipped_not_fatal(self) -> None:
# Many KEGG pathways have no KGML; one of them must not end the run.
self.assertIsNone(self.analyse(FakeKgmlKegg(parsed=None)))
def test_a_missing_name_falls_back_to_unknown(self) -> None:
result = self.analyse(FakeKgmlKegg(parsed=KGML, entry="ENTRY hsa04115\n"))
self.assertEqual(result["pathway_name"], "Unknown")
def test_an_empty_pathway_reports_zero_rather_than_failing(self) -> None:
result = self.analyse(FakeKgmlKegg(parsed={}, entry=""))
self.assertEqual((result["num_entries"], result["num_relations"]), (0, 0))
self.assertEqual(result["relation_types"], {})
def test_failed_pathways_are_dropped_from_the_batch(self) -> None:
class Flaky(FakeKgmlKegg):
def parse_kgml_pathway(self, pathway_id):
if pathway_id == "bad":
raise RuntimeError("no KGML")
return KGML
results = quietly(
pathways.analyze_all_pathways, Flaky(entry=""), ["good", "bad", "good2"]
)
self.assertEqual([r["pathway_id"] for r in results], ["good", "good2"])
def test_the_limit_truncates_the_pathway_list(self) -> None:
kegg = FakeKgmlKegg(parsed=KGML, entry="")
results = quietly(
pathways.analyze_all_pathways, kegg, ["a", "b", "c", "d"], 2
)
self.assertEqual(len(results), 2)
self.assertEqual(kegg.parsed_ids, ["a", "b"])
def test_the_summary_buckets_the_four_named_interaction_types(self) -> None:
result = self.analyse(FakeKgmlKegg(parsed=KGML, entry="NAME p53\n"))
path = self.root / "summary.csv"
quietly(pathways.save_pathway_summary, [result], path)
with path.open(newline="", encoding="utf-8") as handle:
header, row = list(csv.reader(handle))
self.assertEqual(
header,
["Pathway_ID", "Pathway_Name", "Num_Genes", "Num_Interactions",
"Activation", "Inhibition", "Phosphorylation", "Binding", "Other"],
)
columns = dict(zip(header, row))
self.assertEqual(columns["Activation"], "2")
self.assertEqual(columns["Inhibition"], "1")
self.assertEqual(columns["Phosphorylation"], "0")
self.assertEqual(columns["Binding"], "1")
# methylation is not one of the four named buckets.
self.assertEqual(columns["Other"], "1")
# Every relation is accounted for exactly once.
self.assertEqual(
sum(int(columns[name]) for name in
("Activation", "Inhibition", "Phosphorylation", "Binding", "Other")),
int(columns["Num_Interactions"]),
)
def test_interactions_are_written_as_three_column_sif(self) -> None:
result = self.analyse(FakeKgmlKegg(parsed=KGML, entry=""))
path = self.root / "network.sif"
quietly(pathways.save_interactions_sif, [result], path)
lines = path.read_text(encoding="utf-8").splitlines()
self.assertEqual(len(lines), 5)
# SIF is source<TAB>interaction<TAB>target; Cytoscape reads nothing else.
self.assertEqual(lines[0].split("\t"), ["1", "activation", "2"])
for line in lines:
self.assertEqual(len(line.split("\t")), 3)
def test_per_pathway_files_are_named_without_a_colon(self) -> None:
# "path:hsa04115" would be an illegal filename on some systems.
result = self.analyse(FakeKgmlKegg(parsed=KGML, entry=""), "path:hsa04115")
quietly(pathways.save_detailed_pathway_info, [result], str(self.root))
written = [p.name for p in (self.root / "pathways").iterdir()]
self.assertEqual(written, ["path_hsa04115_interactions.csv"])
def test_the_organism_is_set_on_the_client_before_listing_pathways(self) -> None:
class FakeKeggOrganism:
organism = None
pathwayIds = ["path:hsa00010", "path:hsa04115"]
kegg = FakeKeggOrganism()
ids = quietly(pathways.get_all_pathways, kegg, "hsa")
self.assertEqual(kegg.organism, "hsa")
self.assertEqual(ids, ["path:hsa00010", "path:hsa04115"])
class NcbiEmailTests(unittest.TestCase):
"""BLAST submissions need a contact address; a bad one gets rejected."""
def test_a_valid_address_on_the_command_line_wins(self) -> None:
with patch.dict("os.environ", {"NCBI_EMAIL": "env@lab.org"}):
self.assertEqual(
workflow.resolve_ncbi_email("cli@lab.org"), "cli@lab.org"
)
def test_the_environment_is_the_fallback(self) -> None:
with patch.dict("os.environ", {"NCBI_EMAIL": "env@lab.org"}):
self.assertEqual(workflow.resolve_ncbi_email(None), "env@lab.org")
def test_surrounding_whitespace_is_stripped(self) -> None:
self.assertEqual(workflow.resolve_ncbi_email(" a@b.org "), "a@b.org")
def test_no_address_anywhere_yields_none(self) -> None:
with patch.dict("os.environ", {}, clear=True):
self.assertIsNone(workflow.resolve_ncbi_email(None))
self.assertIsNone(workflow.resolve_ncbi_email(""))
def test_malformed_addresses_are_refused(self) -> None:
for candidate in ("not-an-email", "a@b", "a b@c.org", "@b.org", "a@.org"):
with self.subTest(candidate=candidate):
with patch.dict("os.environ", {}, clear=True):
self.assertIsNone(workflow.resolve_ncbi_email(candidate))
class ProteinSearchTests(unittest.TestCase):
TABLE = (
"Entry\tGene names\tOrganism\tLength\tProtein names\n"
"P43403\tZAP70 SRK\tHomo sapiens\t619\tTyrosine-protein kinase ZAP-70\n"
"P04637\tTP53\tHomo sapiens\t393\tCellular tumor antigen p53\n"
)
class FakeUniProt:
def __init__(self, verbose: bool = True, retrieve=None, search="") -> None:
self.retrieve_result = retrieve
self.search_result = search
self.calls: list[tuple[str, tuple, dict]] = []
def retrieve(self, *args, **kwargs):
self.calls.append(("retrieve", args, kwargs))
if self.retrieve_result is None:
raise RuntimeError("not an accession")
return self.retrieve_result
def search(self, *args, **kwargs):
self.calls.append(("search", args, kwargs))
return self.search_result
def run_search(self, query: str, **kwargs):
fake = self.FakeUniProt(**kwargs)
with patch.object(workflow, "UniProt", lambda **k: fake):
_, identifier = quietly(workflow.search_protein, query)
return fake, identifier
def test_an_accession_shaped_query_is_retrieved_directly(self) -> None:
# P43403 is six characters starting with P, so it is fetched rather
# than searched -- one request instead of a full-text query.
fake, identifier = self.run_search("P43403", retrieve="Entry\tP43403\n")
self.assertEqual(identifier, "P43403")
self.assertEqual([call[0] for call in fake.calls], ["retrieve"])
def test_a_name_query_falls_through_to_search(self) -> None:
fake, identifier = self.run_search("ZAP70_HUMAN", search=self.TABLE)
self.assertEqual([call[0] for call in fake.calls], ["search"])
self.assertEqual(identifier, "P43403") # the first data row
def test_a_failed_direct_retrieval_falls_back_to_search(self) -> None:
fake, identifier = self.run_search("P43403", retrieve=None, search=self.TABLE)
self.assertEqual([call[0] for call in fake.calls], ["retrieve", "search"])
self.assertEqual(identifier, "P43403")
def test_a_header_only_response_finds_nothing(self) -> None:
# One line means the table had no data rows; indexing lines[1] would
# raise instead of reporting "not found".
_, identifier = self.run_search("NOSUCHPROTEIN", search="Entry\tGene names\n")
self.assertIsNone(identifier)
def test_an_empty_response_finds_nothing(self) -> None:
_, identifier = self.run_search("NOSUCHPROTEIN", search="")
self.assertIsNone(identifier)
class SequenceRetrievalTests(unittest.TestCase):
class FakeUniProt:
def __init__(self, fasta) -> None:
self.fasta = fasta
self.formats: list[str] = []
def retrieve(self, identifier, frmt=None):
self.formats.append(frmt)
return self.fasta
def test_the_fasta_header_is_dropped_and_the_lines_joined(self) -> None:
fake = self.FakeUniProt(">sp|P43403|ZAP70_HUMAN\nMPDPAAHL\nPFFYGSIS\n")
sequence = quietly(workflow.retrieve_sequence, fake, "P43403")
# 16 residues across two wrapped lines; a header left in would corrupt
# the BLAST submission that follows.
self.assertEqual(sequence, "MPDPAAHLPFFYGSIS")
self.assertEqual(fake.formats, ["fasta"])
def test_an_empty_response_yields_no_sequence(self) -> None:
self.assertIsNone(
quietly(workflow.retrieve_sequence, self.FakeUniProt(""), "P43403")
)
def test_a_service_error_yields_no_sequence(self) -> None:
class Broken:
def retrieve(self, *args, **kwargs):
raise RuntimeError("UniProt is down")
self.assertIsNone(quietly(workflow.retrieve_sequence, Broken(), "P43403"))
class BlastGuardTests(unittest.TestCase):
def test_blast_is_not_submitted_without_a_contact_address(self) -> None:
# EBI rejects anonymous submissions; sending one wastes a job slot.
with patch.object(workflow, "NCBIblast") as blast:
self.assertIsNone(quietly(workflow.run_blast, "MPDPAAHL", None))
blast.assert_not_called()
def test_blast_is_not_submitted_when_explicitly_skipped(self) -> None:
with patch.object(workflow, "NCBIblast") as blast:
self.assertIsNone(
quietly(workflow.run_blast, "MPDPAAHL", "a@b.org", skip=True)
)
blast.assert_not_called()
def test_a_finished_job_returns_its_result(self) -> None:
class FakeBlast:
def __init__(self, verbose: bool = True) -> None:
self.submitted: dict = {}
def run(self, **kwargs):
self.submitted = kwargs
return "ncbiblast-1"
def getStatus(self, jobid):
return "FINISHED"
def getResult(self, jobid, kind):
return "BLAST report\nhit 1\n"
fake = FakeBlast()
with patch.object(workflow, "NCBIblast", lambda **k: fake):
result = quietly(workflow.run_blast, "MPDPAAHL", "a@b.org")
self.assertIn("BLAST report", result)
self.assertEqual(fake.submitted["program"], "blastp")
self.assertEqual(fake.submitted["stype"], "protein")
self.assertEqual(fake.submitted["database"], "uniprotkb")
self.assertEqual(fake.submitted["email"], "a@b.org")
def test_a_failed_job_returns_nothing_rather_than_polling_forever(self) -> None:
class FailingBlast:
def __init__(self, verbose: bool = True) -> None:
pass
def run(self, **kwargs):
return "ncbiblast-2"
def getStatus(self, jobid):
return "ERROR"
with patch.object(workflow, "NCBIblast", lambda **k: FailingBlast()):
with patch.object(workflow.time, "sleep"):
self.assertIsNone(quietly(workflow.run_blast, "MPDPAAHL", "a@b.org"))
class InteractionTests(unittest.TestCase):
#: A PSI-MI TAB record: the first 12 columns are what the parser reads.
LINE = "\t".join(
[
"uniprotkb:P43403", "uniprotkb:P07948",
"intact:EBI-1", "intact:EBI-2",
"uniprotkb:ZAP70", "uniprotkb:LYN",
"psi-mi:\"MI:0018\"(two hybrid)", "Smith et al.",
"pubmed:12345", "taxid:9606", "taxid:9606",
"psi-mi:\"MI:0407\"(direct interaction)",
]
)
def test_the_query_is_scoped_to_human_and_parsed_by_column(self) -> None:
recorded: list[tuple] = []
payload = self.LINE
class FakePsicquic:
def query(self, database, query):
recorded.append((database, query))
return payload
with patch.object(workflow, "PSICQUIC", lambda *a, **k: FakePsicquic()):
interactions = quietly(workflow.find_interactions, "ZAP70")
self.assertEqual(recorded, [("mint", "ZAP70 AND species:9606")])
# Columns 5 and 6 hold the aliases; column 12 the interaction type.
self.assertEqual(interactions[0][0], "ZAP70")
self.assertEqual(interactions[0][1], "LYN")
self.assertIn("direct interaction", interactions[0][2])
def test_a_truncated_record_is_ignored_rather_than_indexed(self) -> None:
class FakePsicquic:
def query(self, database, query):
return "uniprotkb:P43403\tuniprotkb:P07948\n"
with patch.object(workflow, "PSICQUIC", lambda *a, **k: FakePsicquic()):
self.assertEqual(quietly(workflow.find_interactions, "ZAP70"), [])
def test_no_interactions_returns_an_empty_list(self) -> None:
class FakePsicquic:
def query(self, database, query):
return ""
with patch.object(workflow, "PSICQUIC", lambda *a, **k: FakePsicquic()):
self.assertEqual(quietly(workflow.find_interactions, "ZAP70"), [])
def test_a_release_without_psicquic_skips_the_step(self) -> None:
# bioservices 1.16.0 does not ship PSICQUIC; the workflow must degrade
# rather than fail, and must not try to construct it.
with patch.object(workflow, "PSICQUIC", None):
self.assertEqual(quietly(workflow.find_interactions, "ZAP70"), [])
class GoAnnotationTests(unittest.TestCase):
ANNOTATIONS = (
"DB\tID\tSymbol\tQualifier\tRef\tEvidence\tGO_ID\tGO_NAME\tASPECT\n"
"UniProtKB\tP43403\tZAP70\t\tPMID:1\tIDA\tGO:0004713\tprotein tyrosine kinase activity\tF\n"
"UniProtKB\tP43403\tZAP70\t\tPMID:2\tIDA\tGO:0002250\tadaptive immune response\tP\n"
"UniProtKB\tP43403\tZAP70\t\tPMID:3\tIDA\tGO:0005886\tplasma membrane\tC\n"
"UniProtKB\tP43403\tZAP70\t\tPMID:4\tIDA\tGO:0046777\tprotein autophosphorylation\tP\n"
)
def annotations(self, payload: str):
recorded: list[dict] = []
class FakeQuickGo:
def Annotation(self, **kwargs):
recorded.append(kwargs)
return payload
with patch.object(workflow, "QuickGO", lambda *a, **k: FakeQuickGo()):
result = quietly(workflow.get_go_annotations, "P43403")
return recorded, result
def test_terms_are_grouped_by_the_three_go_aspects(self) -> None:
recorded, aspects = self.annotations(self.ANNOTATIONS)
self.assertEqual(recorded, [{"protein": "P43403", "format": "tsv"}])
self.assertEqual(len(aspects["P"]), 2) # two biological processes
self.assertEqual(len(aspects["F"]), 1)
self.assertEqual(len(aspects["C"]), 1)
self.assertEqual(aspects["F"][0], ("GO:0004713", "protein tyrosine kinase activity"))
def test_an_unknown_aspect_letter_is_discarded(self) -> None:
payload = self.ANNOTATIONS + (
"UniProtKB\tP43403\tZAP70\t\tPMID:5\tIDA\tGO:0000001\tmystery\tX\n"
)
_, aspects = self.annotations(payload)
self.assertEqual(set(aspects), {"P", "F", "C"})
self.assertEqual(sum(len(terms) for terms in aspects.values()), 4)
def test_no_annotations_yields_an_empty_list(self) -> None:
_, result = self.annotations("")
self.assertEqual(result, [])
def test_a_header_only_response_yields_empty_aspects(self) -> None:
_, aspects = self.annotations("DB\tID\tSymbol\n")
self.assertEqual(aspects, {"P": [], "F": [], "C": []})
if __name__ == "__main__":
unittest.main()

384
tests/bulk-rnaseq/test_scripts.py Normal file
View File

@@ -0,0 +1,384 @@
"""Tests for the bulk RNA-seq samplesheet validator and counts-matrix builder.
`validate_samplesheet` is the gate in front of an expensive pipeline run, and
the distinction it draws between an error and a warning is the whole product:
a duplicated FASTQ is fatal, the same sample across lanes is not. Every test
here asserts on that split, not merely that "something was reported".
The design checks matter just as much. A group with one replicate cannot
estimate variance, and a batch nested inside condition cannot be separated
from the biology -- both are silent disasters downstream, so both are pinned.
"""
from __future__ import annotations
import sys
import tempfile
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "bulk-rnaseq"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pd = pytest.importorskip("pandas", reason="bulk-rnaseq needs pandas")
import build_counts_matrix # noqa: E402
import validate_samplesheet # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
GOOD_SHEET = """\
sample,fastq_1,fastq_2,strandedness
ctrl_1,ctrl_1_R1.fastq.gz,ctrl_1_R2.fastq.gz,auto
ctrl_2,ctrl_2_R1.fastq.gz,ctrl_2_R2.fastq.gz,auto
treat_1,treat_1_R1.fastq.gz,treat_1_R2.fastq.gz,auto
treat_2,treat_2_R1.fastq.gz,treat_2_R2.fastq.gz,auto
"""
GOOD_METADATA = """\
sample,condition,batch
ctrl_1,control,b1
ctrl_2,control,b2
treat_1,treated,b1
treat_2,treated,b2
"""
class SamplesheetTestCase(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.report = validate_samplesheet.Report()
def write(self, name: str, text: str) -> Path:
path = self.root / name
path.write_text(text, encoding="utf-8")
return path
def validate(self, text: str, check_files: bool = False):
return validate_samplesheet.validate_samplesheet(
self.write("samplesheet.csv", text), check_files, self.report
)
class SamplesheetShapeTests(SamplesheetTestCase):
def test_a_well_formed_sheet_passes_cleanly(self) -> None:
sheet = self.validate(GOOD_SHEET)
self.assertIsNotNone(sheet)
self.assertEqual(self.report.errors, [])
self.assertEqual(self.report.warnings, [])
self.assertEqual(len(sheet), 4)
def test_a_missing_file_is_an_error_not_a_crash(self) -> None:
result = validate_samplesheet.validate_samplesheet(
self.root / "absent.csv", False, self.report
)
self.assertIsNone(result)
self.assertTrue(any("not found" in e for e in self.report.errors))
def test_the_required_columns_are_sample_and_fastq_1(self) -> None:
self.validate("sample,reads\nctrl_1,x.fastq.gz\n")
self.assertTrue(
any("must have at least" in e for e in self.report.errors)
)
def test_whitespace_around_headers_and_values_is_stripped(self) -> None:
sheet = self.validate(
" sample , fastq_1 \n ctrl_1 , ctrl_1_R1.fastq.gz \n"
)
self.assertEqual(list(sheet.columns), ["sample", "fastq_1"])
self.assertEqual(sheet.loc[0, "sample"], "ctrl_1")
self.assertEqual(self.report.errors, [])
def test_a_missing_strandedness_column_is_only_advisory(self) -> None:
self.validate("sample,fastq_1\nctrl_1,a.fastq.gz\n")
self.assertEqual(self.report.errors, [])
self.assertTrue(any("strandedness" in w for w in self.report.warnings))
class SamplesheetContentTests(SamplesheetTestCase):
def test_empty_sample_or_read_cells_are_errors_with_a_row_number(self) -> None:
self.validate("sample,fastq_1\n,a.fastq.gz\nctrl_2,\n")
self.assertTrue(any("row 2" in e and "'sample'" in e for e in self.report.errors))
self.assertTrue(any("row 3" in e and "'fastq_1'" in e for e in self.report.errors))
def test_the_same_file_in_both_read_columns_is_an_error(self) -> None:
# A copy-paste that would silently halve the library.
self.validate("sample,fastq_1,fastq_2\nctrl_1,same.fastq.gz,same.fastq.gz\n")
self.assertTrue(any("the same file" in e for e in self.report.errors))
def test_a_reused_read_one_file_is_an_error(self) -> None:
self.validate(
"sample,fastq_1\nctrl_1,shared.fastq.gz\nctrl_2,shared.fastq.gz\n"
)
self.assertTrue(any("appears 2 times" in e for e in self.report.errors))
def test_unknown_strandedness_values_are_rejected(self) -> None:
self.validate("sample,fastq_1,strandedness\nctrl_1,a.fastq.gz,sideways\n")
self.assertTrue(any("strandedness 'sideways'" in e for e in self.report.errors))
def test_every_documented_strandedness_value_is_accepted(self) -> None:
rows = "\n".join(
f"s{i},r{i}.fastq.gz,{value}"
for i, value in enumerate(sorted(validate_samplesheet.VALID_STRANDEDNESS))
)
self.validate(f"sample,fastq_1,strandedness\n{rows}\n")
self.assertEqual(self.report.errors, [])
def test_strandedness_is_matched_case_insensitively(self) -> None:
self.validate("sample,fastq_1,strandedness\nctrl_1,a.fastq.gz,REVERSE\n")
self.assertEqual(self.report.errors, [])
def test_one_sample_across_lanes_warns_but_does_not_fail(self) -> None:
# nf-core merges lanes, so this is normal -- it must stay a warning.
self.validate(
"sample,fastq_1,fastq_2\n"
"ctrl_1,L1_R1.fastq.gz,L1_R2.fastq.gz\n"
"ctrl_1,L2_R1.fastq.gz,L2_R2.fastq.gz\n"
)
self.assertEqual(self.report.errors, [])
self.assertTrue(any("lane-merged" in w for w in self.report.warnings))
def test_mixing_paired_and_single_rows_for_one_sample_is_an_error(self) -> None:
self.validate(
"sample,fastq_1,fastq_2\n"
"ctrl_1,L1_R1.fastq.gz,L1_R2.fastq.gz\n"
"ctrl_1,L2_R1.fastq.gz,\n"
)
self.assertTrue(
any("mixes paired-end and single-end" in e for e in self.report.errors)
)
class FileExistenceTests(SamplesheetTestCase):
def test_absent_local_reads_are_errors_when_checking_is_on(self) -> None:
self.validate("sample,fastq_1\nctrl_1,missing.fastq.gz\n", check_files=True)
self.assertTrue(any("fastq_1 not found" in e for e in self.report.errors))
def test_present_local_reads_pass(self) -> None:
(self.root / "real.fastq.gz").write_bytes(b"")
self.validate(
f"sample,fastq_1\nctrl_1,{self.root / 'real.fastq.gz'}\n", check_files=True
)
self.assertEqual(self.report.errors, [])
def test_an_unusual_extension_on_an_existing_file_is_only_a_warning(self) -> None:
(self.root / "reads.txt").write_bytes(b"")
self.validate(
f"sample,fastq_1\nctrl_1,{self.root / 'reads.txt'}\n", check_files=True
)
self.assertEqual(self.report.errors, [])
self.assertTrue(any("unusual extension" in w for w in self.report.warnings))
def test_remote_urls_are_skipped_rather_than_reported_missing(self) -> None:
for prefix in validate_samplesheet.REMOTE_PREFIXES:
with self.subTest(prefix=prefix):
report = validate_samplesheet.Report()
path = self.write(
"remote.csv", f"sample,fastq_1\nctrl_1,{prefix}bucket/r1.fastq.gz\n"
)
validate_samplesheet.validate_samplesheet(path, True, report)
self.assertEqual(report.errors, [])
self.assertTrue(any("remote" in w for w in report.warnings))
def test_nothing_is_checked_when_file_checking_is_off(self) -> None:
self.validate("sample,fastq_1\nctrl_1,missing.fastq.gz\n", check_files=False)
self.assertEqual(self.report.errors, [])
class MetadataTests(SamplesheetTestCase):
def _validate_metadata(self, metadata: str, sheet_text: str = GOOD_SHEET,
condition_col: str = "condition", min_rep: int = 3):
sheet = validate_samplesheet.validate_samplesheet(
self.write("sheet.csv", sheet_text), False, validate_samplesheet.Report()
)
validate_samplesheet.validate_metadata(
self.write("meta.csv", metadata), sheet, condition_col, min_rep, self.report
)
return sheet
def test_a_matching_design_passes_apart_from_the_replicate_advisory(self) -> None:
self._validate_metadata(GOOD_METADATA)
self.assertEqual(self.report.errors, [])
def test_a_missing_condition_column_names_the_columns_it_found(self) -> None:
self._validate_metadata("sample,group\nctrl_1,control\n")
self.assertTrue(any("no 'condition' column" in e for e in self.report.errors))
self.assertTrue(any("group" in e for e in self.report.errors))
def test_samples_missing_from_the_metadata_are_an_error(self) -> None:
self._validate_metadata("sample,condition\nctrl_1,control\nctrl_2,control\n")
self.assertTrue(
any("missing from metadata" in e for e in self.report.errors)
)
def test_extra_metadata_samples_are_only_a_warning(self) -> None:
self._validate_metadata(GOOD_METADATA + "spare,control,b3\n")
self.assertEqual(self.report.errors, [])
self.assertTrue(
any("not in samplesheet" in w for w in self.report.warnings)
)
def test_a_singleton_group_cannot_estimate_variance_and_is_fatal(self) -> None:
sheet_text = (
"sample,fastq_1\n"
"ctrl_1,c1.fastq.gz\nctrl_2,c2.fastq.gz\ntreat_1,t1.fastq.gz\n"
)
self._validate_metadata(
"sample,condition\nctrl_1,control\nctrl_2,control\ntreat_1,treated\n",
sheet_text=sheet_text,
)
self.assertTrue(
any("need >=2 to estimate variance" in e for e in self.report.errors)
)
def test_two_replicates_pass_but_warn_below_the_recommended_minimum(self) -> None:
self._validate_metadata(GOOD_METADATA, min_rep=3)
self.assertEqual(self.report.errors, [])
self.assertTrue(any("recommended" in w for w in self.report.warnings))
def test_an_entirely_empty_condition_column_is_fatal(self) -> None:
self._validate_metadata(
"sample,condition\nctrl_1,\nctrl_2,\ntreat_1,\ntreat_2,\n"
)
self.assertTrue(any("is empty for all samples" in e for e in self.report.errors))
def test_batch_fully_nested_in_condition_is_flagged_as_confounded(self) -> None:
# Each batch holding a single condition means the batch effect and the
# biology cannot be told apart -- the most expensive silent mistake here.
self._validate_metadata(
"sample,condition,batch\n"
"ctrl_1,control,b1\nctrl_2,control,b1\n"
"treat_1,treated,b2\ntreat_2,treated,b2\n"
)
self.assertTrue(any("confounded" in w for w in self.report.warnings))
def test_a_crossed_batch_design_is_not_flagged(self) -> None:
self._validate_metadata(GOOD_METADATA)
self.assertFalse(any("confounded" in w for w in self.report.warnings))
class ReportTests(unittest.TestCase):
def test_errors_fail_the_run_and_warnings_do_not(self) -> None:
clean = validate_samplesheet.Report()
self.assertEqual(clean.summarize(), 0)
warned = validate_samplesheet.Report()
warned.warn("advisory")
self.assertEqual(warned.summarize(), 0)
failed = validate_samplesheet.Report()
failed.error("fatal")
self.assertEqual(failed.summarize(), 1)
class SampleNameTests(unittest.TestCase):
def test_common_quantifier_suffixes_are_stripped(self) -> None:
clean = build_counts_matrix._clean_sample_name
for raw in ("ctrl_1", "ctrl_1.bam", "ctrl_1_S1_L001"):
with self.subTest(raw=raw):
self.assertTrue(clean(raw).startswith("ctrl_1"))
def test_cleaning_is_idempotent(self) -> None:
clean = build_counts_matrix._clean_sample_name
once = clean("ctrl_1.bam")
self.assertEqual(clean(once), once)
def test_the_star_strand_columns_match_the_documented_layout(self) -> None:
# STAR's ReadsPerGene.out.tab puts unstranded/forward/reverse in
# columns 1-3; picking the wrong one silently halves the counts.
self.assertEqual(
build_counts_matrix.STAR_STRAND_COL,
{"unstranded": 1, "forward": 2, "reverse": 3},
)
class CountsMatrixTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def _star_table(self, sample: str, rows: str) -> None:
# STAR tables sit flat in the directory, named <sample>.ReadsPerGene.out.tab.
(self.root / f"{sample}.ReadsPerGene.out.tab").write_text(
"N_unmapped\t1\t1\t1\n"
"N_multimapping\t2\t2\t2\n"
"N_noFeature\t3\t3\t3\n"
"N_ambiguous\t4\t4\t4\n" + rows,
encoding="utf-8",
)
def test_star_counts_are_assembled_from_per_sample_tables(self) -> None:
self._star_table("ctrl_1", "GENE1\t10\t0\t0\nGENE2\t20\t0\t0\n")
self._star_table("treat_1", "GENE1\t30\t0\t0\nGENE2\t40\t0\t0\n")
counts = build_counts_matrix.build_from_star(self.root, "unstranded")
self.assertEqual(sorted(counts.index), ["GENE1", "GENE2"])
self.assertEqual(set(counts.columns), {"ctrl_1", "treat_1"})
self.assertEqual(counts.loc["GENE1", "ctrl_1"], 10)
self.assertEqual(counts.loc["GENE2", "treat_1"], 40)
def test_star_summary_rows_are_excluded_from_the_gene_matrix(self) -> None:
# The first four rows are alignment statistics, not genes; counting
# them would inflate the library size for every sample.
self._star_table("s1", "GENE1\t5\t0\t0\n")
counts = build_counts_matrix.build_from_star(self.root, "unstranded")
self.assertEqual(list(counts.index), ["GENE1"])
def test_genes_absent_from_one_sample_become_zero_not_nan(self) -> None:
self._star_table("s1", "GENE1\t5\t0\t0\nGENE2\t7\t0\t0\n")
self._star_table("s2", "GENE1\t9\t0\t0\n")
counts = build_counts_matrix.build_from_star(self.root, "unstranded")
self.assertEqual(counts.loc["GENE2", "s2"], 0)
self.assertEqual(str(counts.dtypes.unique()[0]), "int64")
def test_the_strand_choice_selects_a_different_column(self) -> None:
self._star_table("s1", "GENE1\t100\t60\t40\n")
for strand, expected in (("unstranded", 100), ("forward", 60), ("reverse", 40)):
with self.subTest(strand=strand):
counts = build_counts_matrix.build_from_star(self.root, strand)
self.assertEqual(counts.loc["GENE1", "s1"], expected)
def test_an_empty_directory_exits_with_a_message(self) -> None:
with self.assertRaises(SystemExit) as raised:
build_counts_matrix.build_from_star(self.root, "unstranded")
self.assertIn("ReadsPerGene.out.tab", str(raised.exception))
def test_featurecounts_output_is_reduced_to_genes_by_samples(self) -> None:
path = self.root / "counts.txt"
path.write_text(
"# Program:featureCounts\n"
"Geneid\tChr\tStart\tEnd\tStrand\tLength\tctrl_1.bam\ttreat_1.bam\n"
"GENE1\tchr1\t1\t100\t+\t100\t10\t30\n"
"GENE2\tchr1\t200\t300\t+\t100\t20\t40\n",
encoding="utf-8",
)
counts = build_counts_matrix.build_from_featurecounts(path)
self.assertEqual(list(counts.index), ["GENE1", "GENE2"])
self.assertEqual(len(counts.columns), 2)
self.assertEqual(counts.iloc[0, 0], 10)
# The annotation columns must not survive as samples.
self.assertFalse({"Chr", "Start", "End", "Strand", "Length"} & set(counts.columns))
def test_outputs_are_written_where_downstream_steps_expect_them(self) -> None:
counts = pd.DataFrame(
{"ctrl_1": [10, 20], "treat_1": [30, 40]}, index=["GENE1", "GENE2"]
)
output = self.root / "out"
build_counts_matrix.write_outputs(counts, output)
written = sorted(path.name for path in output.iterdir())
self.assertTrue(written, "write_outputs produced nothing")
for name in written:
with self.subTest(file=name):
self.assertGreater((output / name).stat().st_size, 0)
if __name__ == "__main__":
unittest.main()

348
tests/citation-management/test_scripts.py Normal file
View File

@@ -0,0 +1,348 @@
"""Tests for the citation-management BibTeX tooling.
Two of this skill's scripts reach the network (Crossref, PubMed, Scholar) and
two are pure text processing. The pure half is where citation errors are
actually introduced -- a page range silently rewritten, a DOI left with its URL
prefix so lookups fail, two distinct papers merged because they share a key --
so that is what the suite drives, end to end through real `.bib` files.
The schematic scripts this skill also ships come from the shared contract;
five skills carry byte-identical copies.
"""
from __future__ import annotations
import sys
import tempfile
import unittest
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "citation-management"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import format_bibtex # noqa: E402
import validate_citations # noqa: E402
SchematicTests = skill_contract.schematic.schematic_test_case(SKILL_ROOT)
BIBLIOGRAPHY = """\
@article{jumper2021,
author = {Jumper, John and Evans, Richard},
title = {Highly accurate protein structure prediction},
journal = {Nature},
year = {2021},
volume = {596},
pages = {583-589},
doi = {https://doi.org/10.1038/s41586-021-03819-2}
}
@inproceedings{vaswani2017,
author = {Vaswani, Ashish; Shazeer, Noam},
title = {Attention Is All You Need},
booktitle = {NeurIPS},
year = {2017},
pages = {pp. 5998--6008}
}
"""
class BibTeXTestCase(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.formatter = format_bibtex.BibTeXFormatter()
def bib(self, text: str = BIBLIOGRAPHY) -> str:
path = self.root / "refs.bib"
path.write_text(text, encoding="utf-8")
return str(path)
class ParsingTests(BibTeXTestCase):
def test_entries_are_parsed_with_type_key_and_fields(self) -> None:
entries = self.formatter.parse_bibtex_file(self.bib())
self.assertEqual(len(entries), 2)
self.assertEqual(entries[0]["key"], "jumper2021")
self.assertEqual(entries[0]["type"], "article")
self.assertEqual(entries[0]["fields"]["journal"], "Nature")
self.assertEqual(entries[1]["type"], "inproceedings")
def test_an_empty_file_parses_to_no_entries(self) -> None:
self.assertEqual(self.formatter.parse_bibtex_file(self.bib("")), [])
def test_comments_outside_entries_are_ignored(self) -> None:
entries = self.formatter.parse_bibtex_file(
self.bib(
"% a leading comment\n"
"@article{a,\n title = {T},\n year = {2020}\n}\n"
"% a trailing comment\n"
)
)
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0]["key"], "a")
def test_an_entry_closed_on_the_same_line_is_not_recognised(self) -> None:
# The entry pattern anchors on `\n}`, so the closing brace must start a
# line. Single-line entries are legal BibTeX but are skipped silently --
# pinned here so the limitation is visible rather than surprising.
self.assertEqual(
self.formatter.parse_bibtex_file(
self.bib("@article{a, title = {T}, year = {2020}}\n")
),
[],
)
def test_quoted_field_values_are_read_as_well_as_braced_ones(self) -> None:
entries = self.formatter.parse_bibtex_file(
self.bib('@article{a,\n title = "Quoted Title",\n year = {2020}\n}\n')
)
self.assertEqual(entries[0]["fields"]["title"], "Quoted Title")
def test_an_unreadable_file_returns_no_entries_rather_than_raising(self) -> None:
self.assertEqual(
self.formatter.parse_bibtex_file(str(self.root / "absent.bib")), []
)
class FixTests(BibTeXTestCase):
def _fixed_fields(self, **fields) -> dict:
entry = {"key": "k", "type": "article", "fields": fields}
return self.formatter.fix_common_issues(entry)["fields"]
def test_a_single_hyphen_page_range_becomes_an_en_dash_range(self) -> None:
# BibTeX renders `583-589` as a hyphen, not an en dash.
self.assertEqual(self._fixed_fields(pages="583-589")["pages"], "583--589")
def test_an_already_correct_range_is_left_alone(self) -> None:
self.assertEqual(self._fixed_fields(pages="583--589")["pages"], "583--589")
def test_a_pp_prefix_is_stripped(self) -> None:
for raw in ("pp. 5998--6008", "PP.5998--6008"):
with self.subTest(raw=raw):
self.assertEqual(self._fixed_fields(pages=raw)["pages"], "5998--6008")
def test_a_doi_loses_its_url_prefix(self) -> None:
# A DOI stored as a URL fails every downstream Crossref lookup.
for raw in (
"https://doi.org/10.1038/x",
"http://doi.org/10.1038/x",
"doi:10.1038/x",
):
with self.subTest(raw=raw):
self.assertEqual(self._fixed_fields(doi=raw)["doi"], "10.1038/x")
def test_a_bare_doi_is_untouched(self) -> None:
self.assertEqual(self._fixed_fields(doi="10.1038/x")["doi"], "10.1038/x")
def test_author_separators_are_normalised_to_and(self) -> None:
self.assertEqual(
self._fixed_fields(author="Vaswani, A.; Shazeer, N.")["author"],
"Vaswani, A. and Shazeer, N.",
)
self.assertEqual(
self._fixed_fields(author="Smith, J. & Jones, K.")["author"],
"Smith, J. and Jones, K.",
)
def test_a_doubled_and_is_collapsed(self) -> None:
self.assertEqual(
self._fixed_fields(author="A and and B")["author"], "A and B"
)
def test_fixing_does_not_mutate_the_original_entry(self) -> None:
entry = {"key": "k", "type": "article", "fields": {"pages": "1-2"}}
self.formatter.fix_common_issues(entry)
self.assertEqual(entry["fields"]["pages"], "1-2")
def test_absent_fields_are_not_invented(self) -> None:
self.assertEqual(self._fixed_fields(title="T"), {"title": "T"})
class DeduplicationTests(BibTeXTestCase):
def _entries(self, *specs) -> list:
return [
{"key": key, "type": "article", "fields": fields} for key, fields in specs
]
def test_entries_sharing_a_doi_collapse_to_one(self) -> None:
entries = self._entries(
("a2021", {"doi": "10.1/x"}), ("b2021", {"doi": "10.1/x"})
)
self.assertEqual(len(self.formatter.deduplicate_entries(entries)), 1)
def test_entries_sharing_a_citation_key_collapse_to_one(self) -> None:
entries = self._entries(("same", {"doi": "10.1/a"}), ("same", {"doi": "10.1/b"}))
self.assertEqual(len(self.formatter.deduplicate_entries(entries)), 1)
def test_distinct_entries_all_survive(self) -> None:
entries = self._entries(
("a", {"doi": "10.1/a"}), ("b", {"doi": "10.1/b"}), ("c", {})
)
self.assertEqual(len(self.formatter.deduplicate_entries(entries)), 3)
def test_the_first_occurrence_is_kept(self) -> None:
entries = self._entries(
("first", {"doi": "10.1/x", "title": "T1"}),
("second", {"doi": "10.1/x", "title": "T2"}),
)
kept = self.formatter.deduplicate_entries(entries)
self.assertEqual(kept[0]["key"], "first")
def test_entries_without_a_doi_are_deduplicated_by_key_alone(self) -> None:
entries = self._entries(("a", {}), ("a", {}), ("b", {}))
self.assertEqual(len(self.formatter.deduplicate_entries(entries)), 2)
class SortTests(BibTeXTestCase):
def setUp(self) -> None:
super().setUp()
self.entries = [
{"key": "zeta", "type": "article", "fields": {"year": "2019", "author": "Young, A.", "title": "Beta"}},
{"key": "alpha", "type": "article", "fields": {"year": "2021", "author": "Adams, B.", "title": "Alpha"}},
]
def test_the_default_sort_is_by_citation_key(self) -> None:
self.assertEqual(
[e["key"] for e in self.formatter.sort_entries(self.entries)],
["alpha", "zeta"],
)
def test_sorting_by_year_author_and_title(self) -> None:
expected = {
"year": ["zeta", "alpha"],
"author": ["alpha", "zeta"],
"title": ["alpha", "zeta"],
}
for field, order in expected.items():
with self.subTest(sort_by=field):
self.assertEqual(
[e["key"] for e in self.formatter.sort_entries(self.entries, field)],
order,
)
def test_descending_reverses_the_order(self) -> None:
self.assertEqual(
[e["key"] for e in self.formatter.sort_entries(self.entries, "key", True)],
["zeta", "alpha"],
)
def test_entries_missing_the_sort_field_go_last(self) -> None:
entries = self.entries + [{"key": "omega", "type": "article", "fields": {}}]
ordered = self.formatter.sort_entries(entries, "year")
self.assertEqual(ordered[-1]["key"], "omega")
def test_an_unknown_sort_field_falls_back_to_the_key(self) -> None:
self.assertEqual(
[e["key"] for e in self.formatter.sort_entries(self.entries, "nonsense")],
["alpha", "zeta"],
)
class RenderTests(BibTeXTestCase):
def test_a_formatted_entry_reparses_to_the_same_fields(self) -> None:
original = self.formatter.parse_bibtex_file(self.bib())[0]
rendered = self.formatter.format_entry(original)
path = self.root / "round-trip.bib"
path.write_text(rendered, encoding="utf-8")
reparsed = self.formatter.parse_bibtex_file(str(path))[0]
self.assertEqual(reparsed["key"], original["key"])
self.assertEqual(reparsed["type"], original["type"])
self.assertEqual(reparsed["fields"], original["fields"])
def test_fields_are_emitted_in_the_documented_order(self) -> None:
entry = {
"key": "k",
"type": "article",
"fields": {"year": "2021", "title": "T", "author": "A"},
}
rendered = self.formatter.format_entry(entry)
self.assertLess(rendered.index("author"), rendered.index("title"))
self.assertLess(rendered.index("title"), rendered.index("year"))
def test_braces_are_balanced(self) -> None:
rendered = self.formatter.format_entry(
self.formatter.parse_bibtex_file(self.bib())[0]
)
self.assertEqual(rendered.count("{"), rendered.count("}"))
class EndToEndTests(BibTeXTestCase):
def test_formatting_a_file_applies_every_fix(self) -> None:
output = self.root / "clean.bib"
self.formatter.format_file(self.bib(), output=str(output))
text = output.read_text(encoding="utf-8")
self.assertIn("583--589", text)
self.assertIn("10.1038/s41586-021-03819-2", text)
self.assertNotIn("https://doi.org/", text)
self.assertNotIn("pp. ", text)
self.assertIn("Vaswani, Ashish and Shazeer, Noam", text)
def test_the_result_still_parses(self) -> None:
output = self.root / "clean.bib"
self.formatter.format_file(self.bib(), output=str(output))
self.assertEqual(len(self.formatter.parse_bibtex_file(str(output))), 2)
class ValidationTests(BibTeXTestCase):
def setUp(self) -> None:
super().setUp()
self.validator = validate_citations.CitationValidator()
def test_a_complete_article_raises_no_errors(self) -> None:
entry = {
"key": "jumper2021",
"type": "article",
"fields": {
"author": "Jumper, John",
"title": "A title",
"journal": "Nature",
"year": "2021",
},
}
errors, _ = self.validator.validate_entry(entry)
self.assertEqual(errors, [])
def test_a_missing_required_field_is_reported(self) -> None:
entry = {
"key": "incomplete",
"type": "article",
"fields": {"title": "A title", "year": "2021"},
}
errors, _ = self.validator.validate_entry(entry)
self.assertTrue(errors)
self.assertIn("author", " ".join(str(error) for error in errors).lower())
def test_duplicate_detection_finds_repeated_entries(self) -> None:
entries = self.formatter.parse_bibtex_file(self.bib(BIBLIOGRAPHY + BIBLIOGRAPHY))
self.assertTrue(self.validator.detect_duplicates(entries))
def test_distinct_entries_are_not_reported_as_duplicates(self) -> None:
entries = self.formatter.parse_bibtex_file(self.bib())
self.assertEqual(self.validator.detect_duplicates(entries), [])
def test_manuscript_citation_keys_are_extracted(self) -> None:
manuscript = self.root / "paper.tex"
manuscript.write_text(
"As shown \\cite{jumper2021} and \\citep{vaswani2017,smith2020}.\n",
encoding="utf-8",
)
keys = self.validator.parse_manuscript_citations(str(manuscript))
self.assertIn("jumper2021", keys)
self.assertIn("vaswani2017", keys)
self.assertIn("smith2020", keys)
def test_a_manuscript_with_no_citations_yields_none(self) -> None:
manuscript = self.root / "paper.tex"
manuscript.write_text("No citations here.\n", encoding="utf-8")
self.assertEqual(self.validator.parse_manuscript_citations(str(manuscript)), [])
if __name__ == "__main__":
unittest.main()

28
tests/clinical-decision-support/test_scripts.py
View File

@@ -5,7 +5,6 @@ from __future__ import annotations
import ast
import copy
import json
import re
import sys
import unittest
from pathlib import Path
@@ -26,6 +25,8 @@ import model_biomarker_evaluation # noqa: E402
import survival_plan_validator # noqa: E402
import validate_cds_artifact # noqa: E402
import skill_contract
def load_asset(filename: str) -> dict:
return json.loads((ASSETS / filename).read_text(encoding="utf-8"))
@@ -94,28 +95,10 @@ class StaticSafetyTests(unittest.TestCase):
with self.assertRaises(_common.InputError):
_common.local_input_path("https://example.invalid/data.json")
def test_documented_local_paths_exist(self) -> None:
pattern = re.compile(r"\b(?:assets|references|scripts)/[A-Za-z0-9_.-]+")
documents = [ROOT / "SKILL.md", *sorted((ROOT / "references").glob("*.md"))]
missing: list[str] = []
for document in documents:
for relative in pattern.findall(document.read_text(encoding="utf-8")):
if not (ROOT / relative).is_file():
missing.append(f"{document.name}: {relative}")
self.assertEqual(missing, [])
def test_no_bytecode_artifacts(self) -> None:
artifacts = [
str(path.relative_to(ROOT))
for path in ROOT.rglob("*")
if path.suffix in {".pyc", ".pyo"}
]
self.assertEqual(artifacts, [])
def test_skill_is_progressively_disclosed_and_versioned(self) -> None:
text = (ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertLess(len(text.splitlines()), 500)
self.assertIn('version: "2.1"', text)
self.assertRegex(text, r'\n version: "\d+\.\d+"\n')
self.assertIn("license: MIT", text)
self.assertIn("metadata:\n version:", text)
@@ -264,5 +247,10 @@ class PrivacyChecklistTests(unittest.TestCase):
self.assertFalse(summary["hipaa_compliance_determined"])
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(ROOT)
if __name__ == "__main__":
unittest.main()

7
tests/clinical-reports/test_scripts.py
View File

@@ -31,6 +31,8 @@ from terminology_validator import validate_terminology_manifest # noqa: E402
from validate_case_report import CARE_ITEMS, validate_case_manifest # noqa: E402
from validate_trial_report import E3_SECTIONS, validate_trial_manifest # noqa: E402
import skill_contract
def valid_case_manifest() -> dict:
return {
@@ -486,5 +488,10 @@ class AssetAndGeneratorTests(unittest.TestCase):
self.assertFalse(data["authorization_verified"])
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
if __name__ == "__main__":
unittest.main()

44
tests/conftest.py
View File

@@ -11,10 +11,16 @@ Each skill therefore gets its own process:
pytest tests/<skill> # one skill
python tests/run_all.py # every skill, one process each
This module also installs `tests/_contract/` as the importable module
`skill_contract` -- the assertions every skill shares, factored out of the
per-skill suites. See `_install_contract` for why it is loaded by file
location rather than by putting `tests/` on `sys.path`.
"""
from __future__ import annotations
import importlib.util
import os
import sys
from pathlib import Path
@@ -30,12 +36,46 @@ os.environ["PYTHONDONTWRITEBYTECODE"] = "1"
TESTS_DIR = Path(__file__).resolve().parent
def _install_contract() -> None:
"""Make the shared contract importable as `skill_contract`.
`tests/` must never go on `sys.path`: in prepend/append import mode that
turns every `tests/<skill>/` directory into an importable namespace
package, so `tests/simpy/`, `tests/qutip/` and `tests/neurokit2/` would
shadow the real libraries (see `addopts` in pyproject.toml). Loading the
package by file location and registering it under a name no skill uses
gives suites `import skill_contract` without that hazard.
"""
if "skill_contract" in sys.modules:
return
package = TESTS_DIR / "_contract"
spec = importlib.util.spec_from_file_location(
"skill_contract",
package / "__init__.py",
submodule_search_locations=[str(package)],
)
module = importlib.util.module_from_spec(spec)
# Registered before exec_module so the package's own `from . import cli`
# resolves against the name suites will use.
sys.modules["skill_contract"] = module
spec.loader.exec_module(module)
_install_contract()
def _skill_dirs(config: pytest.Config) -> set[str]:
"""Return the names of the skill directories this session would collect."""
"""Return the names of the skill directories this session would collect.
Underscore-prefixed directories are infrastructure, not skills:
`_contract/` is a library and `_meta/` runs the repo-wide contract without
importing any skill code, so neither participates in the one-skill-per-
process rule.
"""
everything = {
path.name
for path in TESTS_DIR.iterdir()
if path.is_dir() and not path.name.startswith((".", "__"))
if path.is_dir() and not path.name.startswith((".", "_"))
}
selected: set[str] = set()
for argument in config.args:

448
tests/deepchem/test_scripts.py Normal file
View File

@@ -0,0 +1,448 @@
"""Tests for the DeepChem training scripts.
All three scripts end in a `model.fit(...)` that needs a GPU-scale budget and a
deep-learning backend, so none of them can be run to completion here. What can
be checked is everything that decides *what* gets trained, and that is where
these scripts can go wrong in ways a user only discovers minutes into a run:
* the MoleculeNet table -- `train_on_molnet` resolves a dataset name to
`dc.molnet.load_<name>` by attribute lookup, so a name with no matching
loader raises `AttributeError` only after the CLI has accepted it. BACE is
exactly that case: DeepChem ships `load_bace_classification` and
`load_bace_regression` but no `load_bace`.
* the model factory -- every `--model` choice must reach a real branch of
`create_model`, not the fall-through `ValueError`.
* featurizer selection in the transfer-learning script -- ChemBERTa and
MolFormer consume raw SMILES while GROVER needs graph features, and handing a
model the wrong representation fails deep inside the fit.
* the fingerprint width, which is declared twice in `predict_solubility.py`:
once as the regressor's `n_features` and once as the featurizer's `size`. A
mismatch is a shape error at prediction time, after training has finished.
MoleculeNet loaders are stubbed rather than called, so no test downloads a
benchmark. The custom-CSV path is driven for real on twenty molecules, where the
80/10/10 scaffold split has a known answer.
"""
from __future__ import annotations
import ast
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "deepchem"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
deepchem = pytest.importorskip("deepchem", reason="deepchem skill needs deepchem")
pytest.importorskip("numpy", reason="deepchem skill needs numpy")
import graph_neural_network # noqa: E402
import predict_solubility # noqa: E402
import transfer_learning # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
#: Published MoleculeNet task types. Delaney (ESOL), FreeSolv and Lipophilicity
#: are regression benchmarks; Tox21, BBBP, BACE and HIV are classification.
PUBLISHED_TASK_TYPES = {
"tox21": "classification",
"bbbp": "classification",
"bace": "classification",
"hiv": "classification",
"delaney": "regression",
"freesolv": "regression",
"lipo": "regression",
}
#: Tox21 comprises 12 toxicity assays; the other benchmarks here are single-task.
PUBLISHED_TASK_COUNTS = {
"tox21": 12,
"bbbp": 1,
"bace": 1,
"hiv": 1,
"delaney": 1,
"freesolv": 1,
"lipo": 1,
}
#: Twenty distinct molecules -- enough scaffolds for an 80/10/10 split to give
#: whole numbers, and small enough to featurize in milliseconds.
SAMPLE_SMILES = [
"CCO", "CCC", "CCCC", "c1ccccc1", "CC(=O)O",
"CN1C=NC2=C1C(=O)N(C(=O)N2C)C", "CCN", "CCCN", "c1ccncc1", "CC(C)O",
"CCOC", "CCCl", "CCBr", "c1ccc(O)cc1", "CC(N)=O",
"CCS", "CC#N", "CCC=O", "c1ccc2ccccc2c1", "CC(C)(C)O",
]
def parser_choices(module, destination: str) -> list[str]:
"""The choices argparse offers for one option of a script's parser.
The parsers are built inside `main`, so they cannot be obtained without
running it; the choice lists are read off the module source instead.
"""
tree = ast.parse((SCRIPTS / f"{module.__name__}.py").read_text(encoding="utf-8"))
flag = "--" + destination.replace("_", "-")
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
if not (node.args and isinstance(node.args[0], ast.Constant)):
continue
if node.args[0].value != flag:
continue
for keyword in node.keywords:
if keyword.arg != "choices":
continue
source = ast.unparse(keyword.value)
if source.startswith("list("):
# `choices=list(SOME_DICT)` or `choices=list(SOME_DICT.keys())`
name = source[len("list(") : -1].removesuffix(".keys()")
return list(getattr(module, name))
return list(ast.literal_eval(keyword.value))
raise AssertionError(f"{module.__name__} has no {flag} with choices")
def dotted_name(node: ast.AST) -> str:
return ast.unparse(node)
def keyword_constants(source: str, callee: str, keyword: str) -> list[object]:
"""Every constant passed as `keyword` to calls of `callee` in `source`."""
found = []
for node in ast.walk(ast.parse(source)):
if isinstance(node, ast.Call) and dotted_name(node.func) == callee:
for argument in node.keywords:
if argument.arg == keyword and isinstance(argument.value, ast.Constant):
found.append(argument.value.value)
return found
def run_script(name: str, *arguments: str, cwd: Path) -> subprocess.CompletedProcess:
environment = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1", "MPLBACKEND": "Agg"}
return subprocess.run(
[sys.executable, str(SCRIPTS / name), *arguments],
capture_output=True,
text=True,
timeout=300,
env=environment,
cwd=str(cwd),
)
class MolnetCatalogueTests(unittest.TestCase):
"""The dataset table in graph_neural_network.py."""
def test_every_dataset_name_resolves_to_a_real_molnet_loader(self) -> None:
# The regression this guards: `getattr(dc.molnet, f"load_{name}")` for a
# dataset DeepChem exposes only under a suffixed loader name.
for name in graph_neural_network.MOLNET_DATASETS:
with self.subTest(dataset=name):
loader = graph_neural_network.molnet_loader(name)
self.assertTrue(callable(loader))
def test_bace_resolves_to_the_classification_loader(self) -> None:
# DeepChem has no `load_bace`; the table declares BACE a classification
# benchmark, so the classification loader is the matching one.
self.assertIs(
graph_neural_network.molnet_loader("bace"),
deepchem.molnet.load_bace_classification,
)
self.assertFalse(hasattr(deepchem.molnet, "load_bace"))
def test_the_declared_task_types_match_the_published_benchmarks(self) -> None:
# The task type selects the metric set (ROC-AUC vs R²) and the model
# mode, so calling a regression benchmark "classification" produces
# numbers that look plausible and mean nothing.
self.assertEqual(
{
name: task_type
for name, (task_type, _) in graph_neural_network.MOLNET_DATASETS.items()
},
PUBLISHED_TASK_TYPES,
)
def test_the_declared_task_counts_match_the_published_benchmarks(self) -> None:
self.assertEqual(
{
name: count
for name, (_, count) in graph_neural_network.MOLNET_DATASETS.items()
},
PUBLISHED_TASK_COUNTS,
)
def test_the_dataset_flag_offers_exactly_the_table(self) -> None:
self.assertEqual(
parser_choices(graph_neural_network, "dataset"),
list(graph_neural_network.MOLNET_DATASETS),
)
def test_the_model_flag_offers_exactly_the_model_table(self) -> None:
self.assertEqual(
parser_choices(graph_neural_network, "model"),
list(graph_neural_network.AVAILABLE_MODELS),
)
class ModelFactoryTests(unittest.TestCase):
"""`create_model` must cover every `--model` choice."""
def test_an_unknown_model_is_refused_by_name(self) -> None:
with self.assertRaisesRegex(ValueError, "Unknown model type: transformer"):
graph_neural_network.create_model("transformer", 1)
def test_every_advertised_model_reaches_a_branch(self) -> None:
# Constructing these needs a torch backend, which the documented install
# does not always provide, so the assertion is narrower than "it works":
# whatever happens, it must not be the unknown-model fall-through.
for name in graph_neural_network.AVAILABLE_MODELS:
with self.subTest(model=name):
try:
graph_neural_network.create_model(name, 1)
except ValueError as error: # pragma: no cover - backend dependent
self.fail(f"{name} is advertised but unreachable: {error}")
except Exception:
# A missing deep-learning backend is an environment problem,
# not a wiring problem.
pass
def test_every_advertised_model_has_a_human_readable_description(self) -> None:
# The descriptions are printed as the run banner, so an empty one leaves
# the log ambiguous about what was trained.
for name, description in graph_neural_network.AVAILABLE_MODELS.items():
with self.subTest(model=name):
self.assertTrue(description.strip())
self.assertNotEqual(description, name)
class PretrainedCatalogueTests(unittest.TestCase):
"""The pretrained-model table in transfer_learning.py."""
def test_the_model_flag_offers_exactly_the_pretrained_table(self) -> None:
self.assertEqual(
parser_choices(transfer_learning, "model"),
list(transfer_learning.PRETRAINED_MODELS),
)
def test_each_entry_carries_a_name_and_a_description(self) -> None:
for key, entry in transfer_learning.PRETRAINED_MODELS.items():
with self.subTest(model=key):
self.assertEqual(set(entry), {"name", "description", "model_id"})
self.assertTrue(entry["name"].strip())
self.assertTrue(entry["description"].strip())
def test_the_hub_backed_models_name_a_hugging_face_repository(self) -> None:
# These strings are passed straight to HuggingFaceModel; a bare model
# name without the owner prefix cannot be resolved.
for key in ("chemberta", "molformer"):
with self.subTest(model=key):
model_id = transfer_learning.PRETRAINED_MODELS[key]["model_id"]
self.assertRegex(model_id, r"^[\w.-]+/[\w.-]+$")
def test_grover_declares_no_hub_id_because_it_loads_itself(self) -> None:
# GroverModel takes a model_dir, not a hub id; a placeholder string here
# would be passed to a loader that ignores it.
self.assertIsNone(transfer_learning.PRETRAINED_MODELS["grover"]["model_id"])
def test_every_pretrained_model_has_a_fine_tuning_entry_point(self) -> None:
# main() dispatches on the key; an unhandled one falls through to a
# "not yet implemented" message after the dataset has been loaded.
for key in transfer_learning.PRETRAINED_MODELS:
with self.subTest(model=key):
self.assertTrue(callable(getattr(transfer_learning, f"train_{key}")))
class MolnetFeaturizerSelectionTests(unittest.TestCase):
"""`load_molnet_dataset` picks the representation each model can consume."""
def load(self, dataset: str, model: str, loader: str | None = None) -> dict:
"""Call `load_molnet_dataset` with the real MoleculeNet loader stubbed."""
captured: dict = {}
def stub(**keywords):
captured.update(keywords)
return (["task"], ("train", "valid", "test"), [])
with mock.patch.object(deepchem.molnet, loader or f"load_{dataset}", stub):
transfer_learning.load_molnet_dataset(dataset, model)
return captured
def test_smiles_models_get_raw_strings_and_grover_gets_graphs(self) -> None:
# ChemBERTa and MolFormer tokenise SMILES themselves; GROVER is a graph
# transformer and cannot read a string.
self.assertEqual(self.load("bbbp", "chemberta")["featurizer"], "Raw")
self.assertEqual(self.load("bbbp", "molformer")["featurizer"], "Raw")
self.assertEqual(self.load("bbbp", "grover")["featurizer"], "GraphConv")
def test_an_unrecognised_model_falls_back_to_fingerprints(self) -> None:
self.assertEqual(self.load("bbbp", "random-forest")["featurizer"], "ECFP")
def test_every_dataset_is_loaded_with_a_scaffold_split(self) -> None:
# A random split shares scaffolds between train and test and inflates
# every reported score, so the split must not depend on the model.
for model in ("chemberta", "grover", "molformer"):
with self.subTest(model=model):
self.assertEqual(self.load("delaney", model)["splitter"], "scaffold")
def test_every_offered_dataset_is_in_the_loader_table(self) -> None:
# `--dataset` and the dict inside load_molnet_dataset are written out
# separately, so a name in one and not the other is a live failure --
# the CLI accepts the run and it dies on "Unknown dataset".
loaders = (
"load_tox21",
"load_bbbp",
"load_bace_classification",
"load_hiv",
"load_delaney",
"load_freesolv",
"load_lipo",
)
def stub(**keywords):
return (["task"], ("train", "valid", "test"), [])
with mock.patch.multiple(
deepchem.molnet, **{name: stub for name in loaders}
):
for name in parser_choices(transfer_learning, "dataset"):
with self.subTest(dataset=name):
tasks, datasets, _ = transfer_learning.load_molnet_dataset(
name, "chemberta"
)
self.assertEqual(len(datasets), 3)
self.assertEqual(tasks, ["task"])
def test_an_unknown_dataset_is_refused_before_anything_downloads(self) -> None:
with self.assertRaisesRegex(ValueError, "Unknown dataset: not-a-benchmark"):
transfer_learning.load_molnet_dataset("not-a-benchmark", "chemberta")
class CustomDatasetTests(unittest.TestCase):
"""`load_custom_dataset` on a real CSV, where the split sizes are known."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.csv = self.root / "molecules.csv"
rows = "\n".join(
f"{smiles},{index % 2}" for index, smiles in enumerate(SAMPLE_SMILES)
)
self.csv.write_text(f"smiles,target\n{rows}\n", encoding="utf-8")
def test_the_split_is_eighty_ten_ten(self) -> None:
train, valid, test = transfer_learning.load_custom_dataset(
str(self.csv), ["target"], "smiles", "chemberta"
)
self.assertEqual((len(train), len(valid), len(test)), (16, 2, 2))
# Nothing is lost or duplicated by the split.
self.assertEqual(len(train) + len(valid) + len(test), len(SAMPLE_SMILES))
def test_a_smiles_model_keeps_the_strings_unfeaturized(self) -> None:
# DummyFeaturizer is deliberate: the tokenizer inside the model does the
# featurizing, so turning the SMILES into a fingerprint here would
# destroy the input it needs.
train, _, _ = transfer_learning.load_custom_dataset(
str(self.csv), ["target"], "smiles", "chemberta"
)
self.assertEqual(train.X[0], "CCO")
def test_a_fingerprint_model_gets_a_2048_bit_vector(self) -> None:
train, _, _ = transfer_learning.load_custom_dataset(
str(self.csv), ["target"], "smiles", "random-forest"
)
# CircularFingerprint's documented default width.
self.assertEqual(train.X.shape, (16, 2048))
def test_the_named_smiles_column_is_the_one_featurized(self) -> None:
path = self.root / "renamed.csv"
rows = "\n".join(
f"{index % 2},{smiles}" for index, smiles in enumerate(SAMPLE_SMILES)
)
path.write_text(f"target,structure\n{rows}\n", encoding="utf-8")
train, _, _ = transfer_learning.load_custom_dataset(
str(path), ["target"], "structure", "chemberta"
)
self.assertEqual(train.X[0], "CCO")
class SolubilityScriptTests(unittest.TestCase):
"""Static consistency in predict_solubility.py."""
def test_the_fingerprint_width_matches_the_regressor_input_size(self) -> None:
# `n_features` is fixed when the model is built and `size` when new
# molecules are featurized. A mismatch is a shape error at predict
# time -- after the training run has already been paid for.
source = (SCRIPTS / "predict_solubility.py").read_text(encoding="utf-8")
declared = keyword_constants(source, "dc.models.MultitaskRegressor", "n_features")
featurized = keyword_constants(source, "dc.feat.CircularFingerprint", "size")
self.assertEqual(len(set(declared)), 1, declared)
self.assertEqual(set(declared), set(featurized))
def test_the_fingerprint_radius_is_the_same_everywhere(self) -> None:
# Training on ECFP4 and predicting with ECFP6 silently produces garbage
# rather than an error, because the vector width is unchanged.
source = (SCRIPTS / "predict_solubility.py").read_text(encoding="utf-8")
radii = keyword_constants(source, "dc.feat.CircularFingerprint", "radius")
self.assertEqual(len(set(radii)), 1, radii)
def test_the_default_target_column_is_the_delaney_column_name(self) -> None:
# The function's default has to match the column in the published
# Delaney (ESOL) CSV, or the benchmark path finds no target.
import inspect
default = inspect.signature(
predict_solubility.train_solubility_model
).parameters["target_col"].default
self.assertEqual(default, "measured log solubility in mols per litre")
class ArgumentValidationTests(unittest.TestCase):
"""Both scripts refuse an ambiguous invocation before loading anything."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def test_a_gnn_run_needs_a_dataset_or_a_csv(self) -> None:
result = run_script("graph_neural_network.py", cwd=self.root)
self.assertEqual(result.returncode, 1)
self.assertIn("Must specify either", result.stderr)
def test_a_gnn_run_refuses_both_a_dataset_and_a_csv(self) -> None:
# Silently preferring one would train on data the user did not ask for.
result = run_script(
"graph_neural_network.py", "--dataset", "bbbp", "--data", "custom.csv",
cwd=self.root,
)
self.assertEqual(result.returncode, 1)
self.assertIn("Cannot specify both", result.stderr)
def test_transfer_learning_requires_a_pretrained_model(self) -> None:
result = run_script("transfer_learning.py", "--dataset", "bbbp", cwd=self.root)
# argparse's own exit code for a missing required option.
self.assertEqual(result.returncode, 2)
self.assertIn("--model", result.stderr)
def test_transfer_learning_refuses_both_input_sources(self) -> None:
result = run_script(
"transfer_learning.py", "--model", "chemberta",
"--dataset", "bbbp", "--data", "custom.csv",
cwd=self.root,
)
self.assertEqual(result.returncode, 1)
self.assertIn("Cannot specify both", result.stderr)
if __name__ == "__main__":
unittest.main()

319
tests/deeptools/test_scripts.py Normal file
View File

@@ -0,0 +1,319 @@
"""Tests for the deepTools helper scripts.
`workflow_generator` writes bash that a user is expected to run, so its input
sanitising is the security boundary of this skill: every path that reaches a
generated script passes `sanitize_path`, and every interpolation goes through
`shlex.quote`. Those tests come first, and the generated scripts are checked
with `bash -n` so a template that stops parsing cannot ship.
`validate_files` inspects real files, so the tests build genuine BED fixtures
in a temporary directory rather than mocking `open`.
"""
from __future__ import annotations
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "deeptools"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import validate_files # noqa: E402
import workflow_generator # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
# Every generator's required parameters, so the templates can be driven
# uniformly and a new workflow shows up here as a KeyError rather than silently
# going untested.
WORKFLOW_PARAMS = {
"chipseq_qc": {
"input_bam": "data/input.bam",
"chip_bams": ["data/chip1.bam", "data/chip2.bam"],
"output_dir": "results/qc",
},
"chipseq_analysis": {
"input_bam": "data/input.bam",
"chip_bam": "data/chip.bam",
"genes_bed": "annotation/genes.bed",
"peaks_bed": "peaks/macs2.bed",
"output_dir": "results/analysis",
},
"rnaseq_coverage": {
"rnaseq_bam": "data/rnaseq.bam",
"output_dir": "results/coverage",
},
"atacseq": {
"atac_bam": "data/atac.bam",
"peaks_bed": "peaks/atac.bed",
"output_dir": "results/atac",
},
}
GENERATORS = {
"chipseq_qc": workflow_generator.generate_chipseq_qc_workflow,
"chipseq_analysis": workflow_generator.generate_chipseq_analysis_workflow,
"rnaseq_coverage": workflow_generator.generate_rnaseq_coverage_workflow,
"atacseq": workflow_generator.generate_atacseq_workflow,
}
class PathSanitisingTests(unittest.TestCase):
def test_ordinary_paths_pass_through_unchanged(self) -> None:
for path in ("sample.bam", "data/sub-01_run-1.bam", "./x.bed", "a-b_c.2.bw"):
with self.subTest(path=path):
self.assertEqual(
workflow_generator.sanitize_path(path, "default", "--bam"), path
)
def test_empty_and_none_fall_back_to_the_default(self) -> None:
for value in (None, ""):
with self.subTest(value=value):
self.assertEqual(
workflow_generator.sanitize_path(value, "fallback.bam", "--bam"),
"fallback.bam",
)
def test_shell_metacharacters_are_refused(self) -> None:
injections = (
"sample.bam; rm -rf /",
"$(whoami).bam",
"`id`.bam",
"a|b.bam",
"file name.bam",
"x.bam\nrm -rf /",
"~/secret.bam",
"a&b.bam",
"x>out.bam",
)
for value in injections:
with self.subTest(value=value):
with self.assertRaisesRegex(ValueError, "unsupported characters"):
workflow_generator.sanitize_path(value, "d", "--bam")
def test_parent_directory_segments_are_refused(self) -> None:
for value in ("../escape.bam", "data/../../etc/passwd", "..", "a/../b"):
with self.subTest(value=value):
with self.assertRaisesRegex(ValueError, r"'\.\.' path segments"):
workflow_generator.sanitize_path(value, "d", "--bam")
def test_a_dotdot_substring_inside_a_name_is_allowed(self) -> None:
# Only whole segments are parent references; `a..b.bam` is a filename.
self.assertEqual(
workflow_generator.sanitize_path("a..b.bam", "d", "--bam"), "a..b.bam"
)
class PathListTests(unittest.TestCase):
def test_space_separated_values_are_split_and_validated(self) -> None:
self.assertEqual(
workflow_generator.sanitize_path_list("a.bam b.bam", [], "--chip"),
["a.bam", "b.bam"],
)
def test_an_empty_list_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "at least one file path"):
workflow_generator.sanitize_path_list(None, [], "--chip")
def test_one_bad_entry_rejects_the_whole_list(self) -> None:
with self.assertRaisesRegex(ValueError, "unsupported characters"):
workflow_generator.sanitize_path_list("good.bam bad;rm.bam", [], "--chip")
def test_quoting_cannot_smuggle_a_space_into_a_single_path(self) -> None:
# shlex.split honours the quotes, but sanitize_path then rejects the
# space -- the two layers have to agree, not just the first.
with self.assertRaisesRegex(ValueError, "unsupported characters"):
workflow_generator.sanitize_path_list("'my file.bam'", [], "--chip")
class PositiveIntTests(unittest.TestCase):
def test_integers_and_integer_strings_are_accepted(self) -> None:
self.assertEqual(workflow_generator.sanitize_positive_int(8, "--threads"), 8)
self.assertEqual(workflow_generator.sanitize_positive_int("16", "--threads"), 16)
def test_zero_negative_and_non_numeric_are_refused(self) -> None:
with self.assertRaisesRegex(ValueError, ">= 1"):
workflow_generator.sanitize_positive_int(0, "--threads")
with self.assertRaisesRegex(ValueError, ">= 1"):
workflow_generator.sanitize_positive_int(-4, "--threads")
for value in ("eight", None, "8.5"):
with self.subTest(value=value):
with self.assertRaisesRegex(ValueError, "must be an integer"):
workflow_generator.sanitize_positive_int(value, "--threads")
class ShellQuotingTests(unittest.TestCase):
def test_values_are_quoted_for_bash(self) -> None:
self.assertEqual(workflow_generator.shell_literal("plain.bam"), "plain.bam")
self.assertIn("'", workflow_generator.shell_literal("has space.bam"))
def test_arrays_quote_each_element(self) -> None:
rendered = workflow_generator.shell_array(["a.bam", "b c.bam"])
self.assertEqual(rendered.split()[0], "a.bam")
self.assertIn("'b c.bam'", rendered)
def test_relative_run_hints_are_prefixed_so_bash_finds_them(self) -> None:
self.assertEqual(
workflow_generator.runnable_script_path("run.sh"), "./run.sh"
)
self.assertEqual(
workflow_generator.runnable_script_path("/tmp/run.sh"), "/tmp/run.sh"
)
class GeneratedScriptTests(unittest.TestCase):
"""Every generated workflow is valid bash and carries safe defaults."""
def _generate(self, workflow: str) -> str:
with tempfile.TemporaryDirectory() as directory:
output = Path(directory) / "workflow.sh"
GENERATORS[workflow](str(output), WORKFLOW_PARAMS[workflow])
return output.read_text(encoding="utf-8")
def test_every_advertised_workflow_has_a_generator(self) -> None:
self.assertEqual(set(workflow_generator.WORKFLOWS), set(GENERATORS))
def test_generated_scripts_parse_as_bash(self) -> None:
for workflow in GENERATORS:
with self.subTest(workflow=workflow):
script = self._generate(workflow)
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "generated.sh"
path.write_text(script, encoding="utf-8")
result = subprocess.run(
["bash", "-n", str(path)],
capture_output=True,
text=True,
timeout=30,
)
self.assertEqual(result.returncode, 0, result.stderr)
def test_generated_scripts_fail_fast(self) -> None:
for workflow in GENERATORS:
with self.subTest(workflow=workflow):
script = self._generate(workflow)
self.assertTrue(script.startswith("#!/bin/bash"))
self.assertIn("set -euo pipefail", script)
def test_parameters_reach_the_script_quoted(self) -> None:
script = self._generate("chipseq_qc")
self.assertIn("data/input.bam", script)
self.assertIn("data/chip1.bam", script)
self.assertIn("data/chip2.bam", script)
self.assertIn("results/qc", script)
class FileValidationTests(unittest.TestCase):
def test_missing_file_is_reported_not_raised(self) -> None:
ok, message = validate_files.check_file_exists("/no/such/file.bam")
self.assertFalse(ok)
self.assertIn("File not found", message)
def test_bam_index_is_found_under_either_naming_convention(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
bam = root / "sample.bam"
bam.write_bytes(b"BAM\1")
ok, message = validate_files.check_bam_index(str(bam))
self.assertFalse(ok)
self.assertIn("samtools index", message)
# sample.bam.bai
(root / "sample.bam.bai").write_bytes(b"BAI\1")
ok, _ = validate_files.check_bam_index(str(bam))
self.assertTrue(ok)
(root / "sample.bam.bai").unlink()
# sample.bai
(root / "sample.bai").write_bytes(b"BAI\1")
ok, _ = validate_files.check_bam_index(str(bam))
self.assertTrue(ok)
def test_tiny_bigwig_is_flagged_as_suspicious(self) -> None:
with tempfile.TemporaryDirectory() as directory:
small = Path(directory) / "small.bw"
small.write_bytes(b"x" * 10)
ok, message = validate_files.check_bigwig_file(str(small))
self.assertFalse(ok)
self.assertIn("suspiciously small", message)
big = Path(directory) / "big.bw"
big.write_bytes(b"x" * 1024)
ok, _ = validate_files.check_bigwig_file(str(big))
self.assertTrue(ok)
def test_well_formed_bed_passes_and_counts_regions(self) -> None:
with tempfile.TemporaryDirectory() as directory:
bed = Path(directory) / "peaks.bed"
bed.write_text(
"# a comment\n"
"chr1\t100\t200\tpeak1\n"
"chr1\t300\t400\tpeak2\n"
"\n",
encoding="utf-8",
)
ok, message = validate_files.check_bed_file(str(bed))
self.assertTrue(ok, message)
self.assertIn("2 regions", message)
def test_bed_rejects_too_few_columns_bad_types_and_inverted_ranges(self) -> None:
cases = {
"chr1\t100\n": "at least 3 columns",
"chr1\tstart\tend\n": "must be integers",
"chr1\t500\t100\n": "start >= end",
"chr1\t100\t100\n": "start >= end",
}
with tempfile.TemporaryDirectory() as directory:
for index, (content, expected) in enumerate(cases.items()):
with self.subTest(content=content.strip()):
bed = Path(directory) / f"bad{index}.bed"
bed.write_text(content, encoding="utf-8")
ok, message = validate_files.check_bed_file(str(bed))
self.assertFalse(ok)
self.assertIn(expected, message)
def test_comment_only_bed_is_treated_as_empty(self) -> None:
with tempfile.TemporaryDirectory() as directory:
bed = Path(directory) / "comments.bed"
bed.write_text("# header\n\n# more\n", encoding="utf-8")
ok, message = validate_files.check_bed_file(str(bed))
self.assertFalse(ok)
self.assertIn("empty", message)
def test_validate_files_aggregates_across_types(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
bam = root / "s.bam"
bam.write_bytes(b"BAM\1")
(root / "s.bam.bai").write_bytes(b"BAI\1")
bed = root / "p.bed"
bed.write_text("chr1\t1\t2\n", encoding="utf-8")
ok, messages = validate_files.validate_files(
bam_files=[str(bam)], bed_files=[str(bed)]
)
self.assertTrue(ok, messages)
self.assertTrue(any("BAM Files" in line for line in messages))
self.assertTrue(any("BED Files" in line for line in messages))
# One bad file fails the whole run.
ok, _ = validate_files.validate_files(
bam_files=[str(bam), "/no/such.bam"], bed_files=[str(bed)]
)
self.assertFalse(ok)
def test_no_files_means_nothing_to_report(self) -> None:
ok, messages = validate_files.validate_files()
self.assertTrue(ok)
self.assertEqual(messages, [])
if __name__ == "__main__":
unittest.main()

657
tests/diffdock/test_scripts.py Normal file
View File

@@ -0,0 +1,657 @@
"""Tests for the DiffDock helper scripts.
Docking itself needs a GPU, the upstream repository, and downloaded
checkpoints, so nothing here runs inference. What these scripts do around a run
is what the tests cover, and each group guards a specific way the surrounding
work goes wrong:
* `analyze_results` turns a directory of pose files into a ranked table. Get the
confidence bands wrong and a low-confidence pose is reported as trustworthy;
parse `rank10_...sdf` before `rank2_...sdf` and the "top" pose is not the top
one. The published bands (> 0 high, -1.5 to 0 moderate, < -1.5 low, from
DiffDock's README and `references/confidence_and_limitations.md`) are asserted
literally, including their boundaries.
* `prepare_batch_csv` is a validator, so both directions matter: a usable CSV
must pass silently, and each malformed row must be named. A CSV missing a
required column used to raise KeyError instead of reporting it.
* `setup_check` only reports; the tests stub the interpreter, the import
machinery, and the working directory so its verdicts are checked against a
known environment rather than this machine's.
"""
from __future__ import annotations
import contextlib
import csv
import io
import os
import sys
import tempfile
import types
import unittest
from pathlib import Path
from unittest import mock
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "diffdock"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
# Only prepare_batch_csv needs a third-party package; the other two scripts are
# standard library, but a module-scope skip keeps the guard in one place.
pytest.importorskip("pandas", reason="diffdock's prepare_batch_csv needs pandas")
import analyze_results # noqa: E402
import prepare_batch_csv # noqa: E402
import setup_check # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
#: Aspirin, the SMILES the bundled template ships as its first example.
ASPIRIN = "CC(=O)Oc1ccccc1C(=O)O"
def quietly(function, *args, **kwargs):
"""Call `function`, returning (result, everything it printed)."""
stream = io.StringIO()
with contextlib.redirect_stdout(stream):
result = function(*args, **kwargs)
return result, stream.getvalue()
class TemporaryDirectoryTestCase(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
class ConfidenceClassificationTests(unittest.TestCase):
"""The bands come from DiffDock upstream, so they are asserted literally."""
def test_the_three_published_bands(self) -> None:
self.assertEqual(analyze_results.classify_confidence(1.2), "High")
self.assertEqual(analyze_results.classify_confidence(-0.7), "Moderate")
self.assertEqual(analyze_results.classify_confidence(-3.0), "Low")
def test_the_band_boundaries_fall_on_the_documented_side(self) -> None:
# Exactly 0 is not "High" -- the high band is strictly above zero -- and
# exactly -1.5 is "Low", so the moderate band is (-1.5, 0].
self.assertEqual(analyze_results.classify_confidence(0.0), "Moderate")
self.assertEqual(analyze_results.classify_confidence(-1.5), "Low")
self.assertEqual(analyze_results.classify_confidence(-1.4999), "Moderate")
def test_a_missing_score_is_unknown_rather_than_low(self) -> None:
# A pose whose score could not be parsed must not be reported as bad.
self.assertEqual(analyze_results.classify_confidence(None), "Unknown")
class ConfidenceExtractionTests(TemporaryDirectoryTestCase):
def test_the_score_is_read_from_the_current_filename_format(self) -> None:
path = self.root / "rank1_confidence0.87.sdf"
path.write_text("")
self.assertEqual(
analyze_results.extract_confidence_score(path, self.root), 0.87
)
def test_a_negative_score_keeps_its_sign(self) -> None:
# Most real poses score below zero; dropping the minus sign would turn
# every low-confidence pose into a high-confidence one.
path = self.root / "rank3_confidence-2.35.sdf"
path.write_text("")
self.assertEqual(
analyze_results.extract_confidence_score(path, self.root), -2.35
)
def test_a_legacy_run_falls_back_to_confidence_scores_txt(self) -> None:
# Older DiffDock wrote rank_1.sdf plus one score per line, in rank order.
(self.root / "confidence_scores.txt").write_text("0.5\n-0.25\n-2.0\n")
path = self.root / "rank_2.sdf"
path.write_text("")
self.assertEqual(
analyze_results.extract_confidence_score(path, self.root), -0.25
)
def test_a_rank_past_the_end_of_the_legacy_file_is_not_invented(self) -> None:
(self.root / "confidence_scores.txt").write_text("0.5\n")
path = self.root / "rank_4.sdf"
path.write_text("")
self.assertIsNone(analyze_results.extract_confidence_score(path, self.root))
def test_an_sdf_data_item_is_the_last_resort(self) -> None:
# The SDF data-item spelling: a `> <tag>` line, then the value.
path = self.root / "rank_1.sdf"
path.write_text(" 1 0 0\nM END\n> <confidence>\n-1.75\n\n$$$$\n")
self.assertEqual(
analyze_results.extract_confidence_score(path, self.root), -1.75
)
def test_an_inline_confidence_property_is_also_read(self) -> None:
path = self.root / "rank_1.sdf"
path.write_text("M END\nconfidence: 0.31\n$$$$\n")
self.assertEqual(
analyze_results.extract_confidence_score(path, self.root), 0.31
)
def test_a_file_with_no_score_anywhere_yields_none(self) -> None:
path = self.root / "rank_1.sdf"
path.write_text("no properties here\n")
self.assertIsNone(analyze_results.extract_confidence_score(path, self.root))
class ComplexParsingTests(TemporaryDirectoryTestCase):
@staticmethod
def write(directory: Path, *names: str) -> None:
directory.mkdir(parents=True, exist_ok=True)
for name in names:
(directory / name).write_text("")
def test_ranks_are_ordered_numerically_not_lexicographically(self) -> None:
# "rank10" sorts before "rank2" as text; the top pose would be wrong.
self.write(
self.root,
"rank1_confidence0.90.sdf",
"rank2_confidence0.10.sdf",
"rank10_confidence-2.00.sdf",
)
parsed = analyze_results.parse_single_complex(self.root)
self.assertEqual([p["rank"] for p in parsed["predictions"]], [1, 2, 10])
def test_the_scored_filename_wins_over_the_bare_one_for_the_same_rank(self) -> None:
# A run can leave both spellings behind; keeping the unscored one would
# silently drop the confidence for that rank.
self.write(self.root, "rank_1.sdf", "rank1_confidence0.42.sdf")
parsed = analyze_results.parse_single_complex(self.root)
self.assertEqual(len(parsed["predictions"]), 1)
self.assertEqual(parsed["predictions"][0]["confidence"], 0.42)
def test_a_directory_without_pose_files_parses_to_nothing(self) -> None:
self.write(self.root, "log.txt")
self.assertIsNone(analyze_results.parse_single_complex(self.root))
def test_a_single_complex_directory_is_reported_under_one_key(self) -> None:
self.write(self.root, "rank1_confidence0.5.sdf")
results = analyze_results.parse_confidence_scores(self.root)
self.assertEqual(list(results), ["single_complex"])
def test_a_batch_directory_is_keyed_by_subdirectory_name(self) -> None:
self.write(self.root / "complex_a", "rank1_confidence0.5.sdf")
self.write(self.root / "complex_b", "rank1_confidence-1.0.sdf")
self.write(self.root / "failed_run", "log.txt")
results = analyze_results.parse_confidence_scores(self.root)
# The subdirectory with no poses is omitted rather than listed empty.
self.assertEqual(sorted(results), ["complex_a", "complex_b"])
class TopPredictionTests(unittest.TestCase):
@staticmethod
def results(*entries):
"""Build the {complex: {'predictions': [...]}} shape the parsers return."""
built = {}
for name, predictions in entries:
built[name] = {
"predictions": [
{"rank": rank, "file": f"rank{rank}.sdf", "path": f"{name}/rank{rank}.sdf",
"confidence": confidence}
for rank, confidence in predictions
]
}
return built
def test_the_best_poses_are_ranked_across_every_complex(self) -> None:
results = self.results(
("a", [(1, -0.5), (2, -2.0)]),
("b", [(1, 0.9), (2, 0.2)]),
)
top = analyze_results.get_top_predictions(results, n=3)
self.assertEqual(
[(entry["complex"], entry["confidence"]) for entry in top],
[("b", 0.9), ("b", 0.2), ("a", -0.5)],
)
def test_unscored_poses_are_excluded_rather_than_sorted_as_zero(self) -> None:
results = self.results(("a", [(1, None), (2, -1.0)]))
top = analyze_results.get_top_predictions(results, n=10)
self.assertEqual([entry["rank"] for entry in top], [2])
def test_asking_for_more_than_exists_returns_what_exists(self) -> None:
results = self.results(("a", [(1, 0.5)]))
self.assertEqual(len(analyze_results.get_top_predictions(results, n=50)), 1)
class CsvExportTests(TemporaryDirectoryTestCase):
def test_every_pose_becomes_one_row_with_its_band(self) -> None:
results = {
"kinase_1": {
"predictions": [
{"rank": 1, "file": "r1.sdf", "path": "/x/r1.sdf", "confidence": 0.8},
{"rank": 2, "file": "r2.sdf", "path": "/x/r2.sdf", "confidence": -2.5},
{"rank": 3, "file": "r3.sdf", "path": "/x/r3.sdf", "confidence": None},
]
}
}
destination = self.root / "summary.csv"
quietly(analyze_results.export_to_csv, results, destination)
with destination.open(newline="") as handle:
rows = list(csv.DictReader(handle))
self.assertEqual(
[row["confidence_class"] for row in rows], ["High", "Low", "Unknown"]
)
# An unscored pose must leave the numeric column empty, not write "None",
# which would parse as a string and break any downstream aggregation.
self.assertEqual(rows[2]["confidence"], "")
self.assertEqual(rows[0]["complex_name"], "kinase_1")
self.assertEqual(rows[0]["file_path"], "/x/r1.sdf")
class SummaryFilterTests(unittest.TestCase):
RESULTS = {
"a": {
"predictions": [
{"rank": 1, "file": "r1.sdf", "path": "a/r1.sdf", "confidence": 0.5},
{"rank": 2, "file": "r2.sdf", "path": "a/r2.sdf", "confidence": -1.0},
{"rank": 3, "file": "r3.sdf", "path": "a/r3.sdf", "confidence": None},
]
}
}
def test_without_a_threshold_every_pose_is_listed(self) -> None:
_, output = quietly(analyze_results.print_summary, self.RESULTS)
for rank in ("Rank 1", "Rank 2", "Rank 3"):
self.assertIn(rank, output)
def test_a_threshold_keeps_the_poses_at_or_above_it(self) -> None:
# Inclusive at the boundary: -1.0 survives a threshold of -1.0.
_, output = quietly(
analyze_results.print_summary, self.RESULTS, None, -1.0
)
self.assertIn("Rank 1", output)
self.assertIn("Rank 2", output)
def test_a_threshold_drops_lower_and_unscored_poses(self) -> None:
_, output = quietly(analyze_results.print_summary, self.RESULTS, None, 0.0)
self.assertIn("Rank 1", output)
self.assertNotIn("Rank 2", output)
# An unscored pose cannot be shown to clear a threshold.
self.assertNotIn("Rank 3", output)
def test_top_n_truncates_the_listing(self) -> None:
_, output = quietly(analyze_results.print_summary, self.RESULTS, 1)
self.assertIn("Rank 1", output)
self.assertNotIn("Rank 2", output)
def test_an_empty_complex_is_reported_rather_than_skipped(self) -> None:
_, output = quietly(analyze_results.print_summary, {"a": {"predictions": []}})
self.assertIn("No predictions found", output)
class SmilesValidationTests(unittest.TestCase):
def setUp(self) -> None:
if not prepare_batch_csv.RDKIT_AVAILABLE:
self.skipTest("RDKit is not installed; SMILES validation is a no-op")
def test_a_real_drug_smiles_is_accepted(self) -> None:
valid, message = prepare_batch_csv.validate_smiles(ASPIRIN)
self.assertTrue(valid, message)
def test_a_malformed_smiles_is_rejected(self) -> None:
# An unclosed ring bond: RDKit returns None rather than raising.
valid, message = prepare_batch_csv.validate_smiles("C1CCCC")
self.assertFalse(valid)
self.assertIn("Invalid SMILES", message)
def test_an_impossible_valence_is_rejected(self) -> None:
valid, _ = prepare_batch_csv.validate_smiles("C(C)(C)(C)(C)C")
self.assertFalse(valid)
class FilePathValidationTests(TemporaryDirectoryTestCase):
def test_an_existing_file_passes(self) -> None:
(self.root / "protein.pdb").write_text("")
valid, _ = prepare_batch_csv.validate_file_path("protein.pdb", self.root)
self.assertTrue(valid)
def test_a_missing_file_is_named_in_the_message(self) -> None:
valid, message = prepare_batch_csv.validate_file_path("absent.pdb", self.root)
self.assertFalse(valid)
self.assertIn("absent.pdb", message)
def test_a_relative_path_resolves_against_the_base_directory(self) -> None:
# Without the base directory the same path is looked up in the process
# working directory, where it does not exist.
(self.root / "sub").mkdir()
(self.root / "sub" / "protein.pdb").write_text("")
self.assertTrue(
prepare_batch_csv.validate_file_path("sub/protein.pdb", self.root)[0]
)
self.assertFalse(prepare_batch_csv.validate_file_path("sub/protein.pdb")[0])
def test_an_empty_path_is_allowed_because_a_sequence_may_replace_it(self) -> None:
for value in ("", float("nan")):
with self.subTest(value=value):
valid, message = prepare_batch_csv.validate_file_path(value, self.root)
self.assertTrue(valid)
self.assertIn("protein_sequence", message)
class BatchCsvValidationTests(TemporaryDirectoryTestCase):
HEADER = "complex_name,protein_path,ligand_description,protein_sequence"
def write_csv(self, *rows: str) -> Path:
path = self.root / "batch.csv"
path.write_text("\n".join([self.HEADER, *rows]) + "\n")
return path
def test_a_usable_csv_passes_without_complaint(self) -> None:
(self.root / "protein.pdb").write_text("")
path = self.write_csv(f"target_1,protein.pdb,{ASPIRIN},")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertTrue(valid, "\n".join(messages))
joined = "\n".join(messages)
self.assertIn("All required columns present", joined)
self.assertIn("PASSED", joined)
# A clean CSV must not accumulate per-row complaints.
self.assertNotIn("Row 1", joined)
def test_a_sequence_only_row_needs_no_protein_file(self) -> None:
path = self.write_csv(f"target_1,,{ASPIRIN},MSKGEELFTG")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertTrue(valid, "\n".join(messages))
def test_a_row_with_neither_protein_input_is_rejected(self) -> None:
path = self.write_csv(f"target_1,,{ASPIRIN},")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertFalse(valid)
self.assertIn("Must provide either protein_path or protein_sequence",
"\n".join(messages))
def test_a_missing_complex_name_is_reported(self) -> None:
path = self.write_csv(f",protein.pdb,{ASPIRIN},")
(self.root / "protein.pdb").write_text("")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertFalse(valid)
self.assertIn("Missing complex_name", "\n".join(messages))
def test_a_missing_protein_file_is_reported_with_its_row(self) -> None:
path = self.write_csv(f"target_1,gone.pdb,{ASPIRIN},")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertFalse(valid)
joined = "\n".join(messages)
self.assertIn("Row 1", joined)
self.assertIn("Protein file issue", joined)
def test_a_missing_ligand_description_is_reported(self) -> None:
(self.root / "protein.pdb").write_text("")
path = self.write_csv("target_1,protein.pdb,,")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertFalse(valid)
self.assertIn("Missing ligand_description", "\n".join(messages))
def test_a_ligand_path_is_checked_as_a_file_not_as_smiles(self) -> None:
# Anything containing a separator is treated as a path, so a missing
# SDF must be reported as a file problem rather than as bad chemistry.
(self.root / "protein.pdb").write_text("")
path = self.write_csv("target_1,protein.pdb,ligands/gone.sdf,")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertFalse(valid)
self.assertIn("Ligand file issue", "\n".join(messages))
def test_both_protein_inputs_together_warn_but_still_validate(self) -> None:
(self.root / "protein.pdb").write_text("")
path = self.write_csv(f"target_1,protein.pdb,{ASPIRIN},MSKGEELFTG")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertTrue(valid, "\n".join(messages))
self.assertIn("will use protein_path", "\n".join(messages))
def test_a_missing_column_is_reported_instead_of_raising(self) -> None:
# Regression: the per-row checks index protein_path directly, so a CSV
# without that column used to abort with KeyError.
path = self.root / "partial.csv"
path.write_text(f"complex_name,ligand_description\ntarget_1,{ASPIRIN}\n")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertFalse(valid)
joined = "\n".join(messages)
self.assertIn("Missing required columns", joined)
self.assertIn("protein_path", joined)
self.assertIn("protein_sequence", joined)
def test_an_unreadable_csv_is_reported_not_raised(self) -> None:
valid, messages = prepare_batch_csv.validate_csv(self.root / "absent.csv")
self.assertFalse(valid)
self.assertIn("Error reading CSV", messages[0])
def test_a_header_only_csv_is_vacuously_valid(self) -> None:
path = self.write_csv()
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertTrue(valid, "\n".join(messages))
self.assertIn("0 rows", messages[0])
def test_an_invalid_smiles_row_fails_validation(self) -> None:
if not prepare_batch_csv.RDKIT_AVAILABLE:
self.skipTest("RDKit is not installed; SMILES validation is a no-op")
(self.root / "protein.pdb").write_text("")
path = self.write_csv("target_1,protein.pdb,C1CCCC,")
valid, messages = prepare_batch_csv.validate_csv(path)
self.assertFalse(valid)
self.assertIn("SMILES issue", "\n".join(messages))
class TemplateCsvTests(TemporaryDirectoryTestCase):
def test_the_template_carries_the_four_columns_diffdock_requires(self) -> None:
destination = self.root / "template.csv"
frame = prepare_batch_csv.create_template_csv(destination)
self.assertEqual(
list(frame.columns),
["complex_name", "protein_path", "ligand_description", "protein_sequence"],
)
self.assertTrue(destination.is_file())
def test_the_requested_number_of_example_rows_is_written(self) -> None:
frame = prepare_batch_csv.create_template_csv(self.root / "t.csv", 2)
self.assertEqual(len(frame), 2)
def test_only_three_examples_exist_so_larger_requests_are_capped(self) -> None:
# The CLI advertises 1-3; asking for more must not produce empty rows.
frame = prepare_batch_csv.create_template_csv(self.root / "t.csv", 9)
self.assertEqual(len(frame), 3)
self.assertFalse(frame["complex_name"].isna().any())
def test_the_first_example_ligand_is_a_parseable_molecule(self) -> None:
frame = prepare_batch_csv.create_template_csv(self.root / "t.csv", 1)
self.assertEqual(frame["ligand_description"][0], ASPIRIN)
if prepare_batch_csv.RDKIT_AVAILABLE:
self.assertTrue(prepare_batch_csv.validate_smiles(ASPIRIN)[0])
class ShippedTemplateAssetTests(unittest.TestCase):
"""`assets/batch_template.csv` is what a user copies, so it must be valid."""
def setUp(self) -> None:
with (SKILL_ROOT / "assets" / "batch_template.csv").open(newline="") as handle:
self.rows = list(csv.DictReader(handle))
def test_it_declares_the_columns_the_validator_requires(self) -> None:
self.assertEqual(
set(self.rows[0]),
{"complex_name", "protein_path", "ligand_description", "protein_sequence"},
)
def test_every_row_supplies_a_protein_by_path_or_by_sequence(self) -> None:
for row in self.rows:
with self.subTest(complex_name=row["complex_name"]):
self.assertTrue(
row["protein_path"] or row["protein_sequence"],
"row has neither protein_path nor protein_sequence",
)
def test_every_smiles_in_the_asset_parses(self) -> None:
if not prepare_batch_csv.RDKIT_AVAILABLE:
self.skipTest("RDKit is not installed; SMILES validation is a no-op")
for row in self.rows:
ligand = row["ligand_description"]
if "/" in ligand or ligand.endswith((".sdf", ".mol2")):
continue # a file reference, not a molecule
with self.subTest(ligand=ligand):
self.assertTrue(prepare_batch_csv.validate_smiles(ligand)[0])
class PythonVersionCheckTests(unittest.TestCase):
@staticmethod
def version(major: int, minor: int, micro: int = 0):
return mock.patch.object(
sys, "version_info", types.SimpleNamespace(major=major, minor=minor, micro=micro)
)
def test_the_upstream_minimum_of_three_nine_passes(self) -> None:
# DiffDock's environment.yml pins Python 3.9.18, so 3.9 is the floor.
with self.version(3, 9, 18):
passed, output = quietly(setup_check.check_python_version)
self.assertTrue(passed)
self.assertIn("3.9.18", output)
def test_anything_older_fails_and_says_what_is_required(self) -> None:
with self.version(3, 8, 10):
passed, output = quietly(setup_check.check_python_version)
self.assertFalse(passed)
self.assertIn("3.9", output)
def test_a_newer_interpreter_is_accepted(self) -> None:
with self.version(3, 13, 1):
passed, _ = quietly(setup_check.check_python_version)
self.assertTrue(passed)
class PackageProbeTests(unittest.TestCase):
def test_a_present_package_reports_its_version(self) -> None:
module = types.ModuleType("diffdock_probe_present")
module.__version__ = "1.2.3"
with mock.patch.dict(sys.modules, {"diffdock_probe_present": module}):
found, output = quietly(
setup_check.check_package, "probe", "diffdock_probe_present"
)
self.assertTrue(found)
self.assertIn("1.2.3", output)
def test_a_nested_version_attribute_is_followed(self) -> None:
# RDKit's version lives at rdkit.rdBase.__version__, not on the package.
module = types.ModuleType("diffdock_probe_nested")
module.rdBase = types.SimpleNamespace(__version__="2024.03.1")
with mock.patch.dict(sys.modules, {"diffdock_probe_nested": module}):
found, output = quietly(
setup_check.check_package,
"rdkit",
"diffdock_probe_nested",
"rdBase.__version__",
)
self.assertTrue(found)
self.assertIn("2024.03.1", output)
def test_a_package_without_a_version_still_counts_as_installed(self) -> None:
module = types.ModuleType("diffdock_probe_bare")
with mock.patch.dict(sys.modules, {"diffdock_probe_bare": module}):
found, output = quietly(
setup_check.check_package, "probe", "diffdock_probe_bare"
)
self.assertTrue(found)
self.assertIn("unknown", output)
def test_an_absent_package_is_reported_as_not_installed(self) -> None:
found, output = quietly(
setup_check.check_package, "probe", "diffdock_probe_absent_xyz"
)
self.assertFalse(found)
self.assertIn("not installed", output)
def test_torch_reports_cuda_separately_from_the_import(self) -> None:
torch = types.ModuleType("torch")
torch.__version__ = "2.4.0"
torch.cuda = types.SimpleNamespace(is_available=lambda: False)
with mock.patch.dict(sys.modules, {"torch": torch}):
(installed, has_cuda), output = quietly(setup_check.check_pytorch)
# Present but CPU-only: the check must pass while flagging no GPU.
self.assertEqual((installed, has_cuda), (True, False))
self.assertIn("CUDA not available", output)
def test_a_visible_gpu_is_named_in_the_report(self) -> None:
torch = types.ModuleType("torch")
torch.__version__ = "2.4.0"
torch.cuda = types.SimpleNamespace(
is_available=lambda: True,
get_device_name=lambda index: "NVIDIA A100",
device_count=lambda: 2,
)
torch.version = types.SimpleNamespace(cuda="12.1")
with mock.patch.dict(sys.modules, {"torch": torch}):
(installed, has_cuda), output = quietly(setup_check.check_pytorch)
self.assertEqual((installed, has_cuda), (True, True))
self.assertIn("NVIDIA A100", output)
self.assertIn("12.1", output)
def test_missing_torch_reports_no_gpu_rather_than_raising(self) -> None:
# None in sys.modules makes `import torch` raise ImportError.
with mock.patch.dict(sys.modules, {"torch": None}):
(installed, has_cuda), output = quietly(setup_check.check_pytorch)
self.assertEqual((installed, has_cuda), (False, False))
self.assertIn("not installed", output)
def test_missing_esm_is_reported_as_optional(self) -> None:
with mock.patch.dict(sys.modules, {"esm": None}):
found, output = quietly(setup_check.check_esm)
self.assertFalse(found)
# ESM is only needed to fold a sequence, so the message must say so
# rather than reading as a hard failure.
self.assertIn("protein sequence folding", output)
self.assertIn("fair-esm", output)
class InstallationProbeTests(TemporaryDirectoryTestCase):
def setUp(self) -> None:
super().setUp()
origin = os.getcwd()
self.addCleanup(os.chdir, origin)
os.chdir(self.root)
def test_an_unrelated_directory_is_not_a_diffdock_checkout(self) -> None:
found, output = quietly(setup_check.check_diffdock_installation)
self.assertFalse(found)
self.assertIn("repository root", output)
def test_the_upstream_entry_points_are_what_it_looks_for(self) -> None:
for name in ("inference.py", "default_inference_args.yaml", "environment.yml"):
(self.root / name).write_text("")
found, output = quietly(setup_check.check_diffdock_installation)
self.assertTrue(found)
self.assertIn("inference.py", output)
def test_absent_checkpoints_are_a_note_not_a_failure(self) -> None:
(self.root / "inference.py").write_text("")
found, output = quietly(setup_check.check_diffdock_installation)
# Weights download on first run, so their absence must not fail setup.
self.assertTrue(found)
self.assertIn("downloaded on first run", output)
def test_present_checkpoints_are_recognised_at_the_documented_path(self) -> None:
(self.root / "inference.py").write_text("")
for name in ("score_model", "confidence_model"):
(self.root / "workdir" / "v1.1" / name).mkdir(parents=True)
_, output = quietly(setup_check.check_diffdock_installation)
self.assertIn("Model checkpoints found", output)
class PerformanceNoteTests(unittest.TestCase):
def test_the_gpu_and_cpu_guidance_do_not_swap(self) -> None:
_, with_gpu = quietly(setup_check.print_performance_notes, True)
_, without = quietly(setup_check.print_performance_notes, False)
self.assertIn("GPU detected", with_gpu)
self.assertNotIn("No GPU detected", with_gpu)
# CPU docking is hours per complex; the warning is the point of the note.
self.assertIn("No GPU detected", without)
self.assertIn("SIGNIFICANTLY slower", without)
if __name__ == "__main__":
unittest.main()

582
tests/dnanexus-integration/test_scripts.py Normal file
View File

@@ -0,0 +1,582 @@
"""Tests for the DNAnexus integration scripts.
`validate_dxapp` is an offline linter for `dxapp.json`, so the tests start from
one manifest that must be clean and mutate a single field per test. That shape
matters: a validator is only useful if a valid manifest stays silent, and a
suite built only from broken inputs never proves that.
`inspect_dxpy` introspects the installed `dxpy` SDK. Its report-shaping and
version-comparison logic is pure and tested here; anything needing the SDK
itself skips unless `dxpy` is installed, which it is under
`tests/run_all.py --isolated`.
"""
from __future__ import annotations
import copy
import importlib
import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "dnanexus-integration"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import inspect_dxpy # noqa: E402
import validate_dxapp # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
def valid_app_manifest() -> dict:
"""A minimal dxapp.json that must produce no errors and no warnings."""
return {
"name": "my-analysis-app",
"title": "My Analysis App",
"version": "1.2.3",
"inputSpec": [
{"name": "reads", "class": "file"},
{"name": "threads", "class": "int", "optional": True},
],
"outputSpec": [{"name": "report", "class": "file"}],
"runSpec": {
"file": "src/code.sh",
"interpreter": "bash",
"distribution": "Ubuntu",
"release": "24.04",
"version": "0",
},
}
def issue_codes(manifest: dict, kind: str = "auto") -> set[str]:
_, issues = validate_dxapp.Validator(manifest, kind).validate()
return {issue.code for issue in issues}
class ValidBaselineTests(unittest.TestCase):
def test_a_correct_app_manifest_produces_nothing(self) -> None:
kind, issues = validate_dxapp.Validator(valid_app_manifest(), "auto").validate()
self.assertEqual(kind, "app")
self.assertEqual([f"{i.code} at {i.path}" for i in issues], [])
def test_kind_is_inferred_from_the_presence_of_version(self) -> None:
manifest = valid_app_manifest()
del manifest["version"]
kind, _ = validate_dxapp.Validator(manifest, "auto").validate()
self.assertEqual(kind, "applet")
kind, _ = validate_dxapp.Validator(manifest, "app").validate()
self.assertEqual(kind, "app")
def test_an_applet_only_gets_a_warning_for_missing_specs(self) -> None:
applet = {
"name": "my-applet",
"runSpec": {
"file": "src/code.sh",
"interpreter": "bash",
"distribution": "Ubuntu",
"release": "24.04",
"version": "0",
},
}
_, issues = validate_dxapp.Validator(applet, "applet").validate()
severities = {issue.code: issue.severity for issue in issues}
self.assertEqual(severities.get("missing-spec"), "warning")
# The same omission is an error for an app.
_, issues = validate_dxapp.Validator(applet, "app").validate()
severities = {issue.code: issue.severity for issue in issues}
self.assertEqual(severities.get("missing-spec"), "error")
def test_a_non_object_root_is_rejected_without_further_checks(self) -> None:
kind, issues = validate_dxapp.Validator(["not", "an", "object"], "auto").validate()
self.assertEqual(kind, "unknown")
self.assertEqual([issue.code for issue in issues], ["root-type"])
class MetadataTests(unittest.TestCase):
def test_name_is_required_and_character_restricted(self) -> None:
manifest = valid_app_manifest()
del manifest["name"]
self.assertIn("missing-name", issue_codes(manifest))
for bad in ("has space", "slash/name", "quote'name"):
with self.subTest(name=bad):
manifest = valid_app_manifest()
manifest["name"] = bad
self.assertIn("invalid-name", issue_codes(manifest))
def test_app_versions_must_be_semantic(self) -> None:
for bad in ("1.2", "v1.2.3", "1.02.3", "1.2.3.4", ""):
with self.subTest(version=bad):
manifest = valid_app_manifest()
manifest["version"] = bad
self.assertIn("invalid-version", issue_codes(manifest))
for good in ("0.0.1", "1.2.3-beta.1", "1.2.3+build5", "10.20.30"):
with self.subTest(version=good):
manifest = valid_app_manifest()
manifest["version"] = good
self.assertNotIn("invalid-version", issue_codes(manifest))
def test_top_level_resources_is_flagged_as_deprecated(self) -> None:
manifest = valid_app_manifest()
manifest["resources"] = ["project-xxxx:/assets"]
self.assertIn("deprecated-resources", issue_codes(manifest))
class ParameterSpecTests(unittest.TestCase):
def test_parameter_names_follow_the_identifier_rule(self) -> None:
for bad in ("2reads", "has-dash", "has space", ""):
with self.subTest(name=bad):
manifest = valid_app_manifest()
manifest["inputSpec"][0]["name"] = bad
self.assertIn("parameter-name", issue_codes(manifest))
def test_duplicate_parameter_names_are_rejected(self) -> None:
manifest = valid_app_manifest()
manifest["inputSpec"][1]["name"] = "reads"
self.assertIn("duplicate-parameter", issue_codes(manifest))
def test_unknown_and_array_classes(self) -> None:
manifest = valid_app_manifest()
manifest["inputSpec"][0]["class"] = "array:file"
self.assertNotIn("parameter-class", issue_codes(manifest))
manifest["inputSpec"][0]["class"] = "array:blob"
self.assertIn("parameter-class", issue_codes(manifest))
def test_optional_must_be_a_boolean(self) -> None:
manifest = valid_app_manifest()
manifest["inputSpec"][1]["optional"] = "yes"
self.assertIn("optional-type", issue_codes(manifest))
def test_input_only_fields_are_rejected_in_the_output_spec(self) -> None:
for field in ("default", "suggestions", "choices"):
with self.subTest(field=field):
manifest = valid_app_manifest()
manifest["outputSpec"][0][field] = "x"
self.assertIn("output-only-field", issue_codes(manifest))
# ...and accepted in the input spec.
manifest = valid_app_manifest()
manifest["inputSpec"][1]["default"] = 4
self.assertNotIn("output-only-field", issue_codes(manifest))
def test_a_spec_that_is_not_a_list_is_reported_once(self) -> None:
manifest = valid_app_manifest()
manifest["inputSpec"] = {"reads": "file"}
self.assertIn("spec-type", issue_codes(manifest))
class RunSpecTests(unittest.TestCase):
def test_runspec_is_required(self) -> None:
manifest = valid_app_manifest()
del manifest["runSpec"]
self.assertIn("missing-runspec", issue_codes(manifest))
def test_an_entry_source_is_required_and_should_not_be_doubled(self) -> None:
manifest = valid_app_manifest()
del manifest["runSpec"]["file"]
self.assertIn("missing-entry-source", issue_codes(manifest))
manifest = valid_app_manifest()
manifest["runSpec"]["code"] = "echo hi"
self.assertIn("multiple-entry-sources", issue_codes(manifest))
def test_interpreter_distribution_release_and_aee_version_are_pinned(self) -> None:
checks = {
"interpreter": ("perl", "interpreter"),
"distribution": ("Debian", "distribution"),
"release": ("22.04", "release"),
"version": ("1", "aee-version"),
}
for field, (value, code) in checks.items():
with self.subTest(field=field):
manifest = valid_app_manifest()
manifest["runSpec"][field] = value
self.assertIn(code, issue_codes(manifest))
def test_ubuntu_2004_is_supported_but_flagged_as_legacy(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["release"] = "20.04"
codes = issue_codes(manifest)
self.assertNotIn("release", codes)
self.assertIn("legacy-release", codes)
def test_deprecated_system_requirements_location_is_flagged(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["systemRequirements"] = {"*": {"instanceType": "mem1_ssd1_v2_x4"}}
self.assertIn("deprecated-system-requirements", issue_codes(manifest))
def test_restartable_entry_points_are_constrained_and_warned_about(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["restartableEntryPoints"] = "some"
self.assertIn("restartable-entry-points", issue_codes(manifest))
manifest = valid_app_manifest()
manifest["runSpec"]["restartableEntryPoints"] = "all"
codes = issue_codes(manifest)
self.assertNotIn("restartable-entry-points", codes)
self.assertIn("restart-idempotency", codes)
class PolicyTests(unittest.TestCase):
def test_unpinned_dependencies_are_warned_about(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["execDepends"] = [{"name": "samtools"}]
self.assertIn("floating-dependency", issue_codes(manifest))
manifest["runSpec"]["execDepends"] = [{"name": "samtools", "version": "1.19"}]
self.assertNotIn("floating-dependency", issue_codes(manifest))
def test_dependency_entries_must_be_named_objects(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["execDepends"] = ["samtools"]
self.assertIn("dependency-type", issue_codes(manifest))
manifest["runSpec"]["execDepends"] = [{"version": "1.19"}]
self.assertIn("dependency-name", issue_codes(manifest))
def test_max_restarts_is_bounded_and_rejects_booleans(self) -> None:
for bad in (-1, 10, 99, True, "3"):
with self.subTest(value=bad):
manifest = valid_app_manifest()
manifest["runSpec"]["executionPolicy"] = {"maxRestarts": bad}
self.assertIn("max-restarts", issue_codes(manifest))
manifest = valid_app_manifest()
manifest["runSpec"]["executionPolicy"] = {"maxRestarts": 0}
self.assertNotIn("max-restarts", issue_codes(manifest))
def test_restart_reasons_are_checked_against_the_documented_set(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["executionPolicy"] = {"restartOn": {"MadeUpError": 2}}
self.assertIn("unknown-restart-reason", issue_codes(manifest))
manifest["runSpec"]["executionPolicy"] = {"restartOn": {"ExecutionError": 2}}
self.assertNotIn("unknown-restart-reason", issue_codes(manifest))
manifest["runSpec"]["executionPolicy"] = {"restartOn": {"ExecutionError": -1}}
self.assertIn("restart-count", issue_codes(manifest))
def test_timeout_units_and_values_are_constrained(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["timeoutPolicy"] = {"*": {"hours": 12}}
self.assertEqual(issue_codes(manifest), set())
manifest["runSpec"]["timeoutPolicy"] = {"*": {"weeks": 1}}
self.assertIn("timeout-unit", issue_codes(manifest))
manifest["runSpec"]["timeoutPolicy"] = {"*": {"hours": -1}}
self.assertIn("timeout-value", issue_codes(manifest))
manifest["runSpec"]["timeoutPolicy"] = {"*": {}}
self.assertIn("timeout-duration", issue_codes(manifest))
class RegionalOptionTests(unittest.TestCase):
def test_region_identifiers_should_be_provider_qualified(self) -> None:
manifest = valid_app_manifest()
manifest["regionalOptions"] = {"us-east-1": {}}
self.assertIn("region-name", issue_codes(manifest))
manifest["regionalOptions"] = {"aws:us-east-1": {}}
self.assertNotIn("region-name", issue_codes(manifest))
def test_system_requirements_must_cover_every_region_or_none(self) -> None:
manifest = valid_app_manifest()
manifest["regionalOptions"] = {
"aws:us-east-1": {"systemRequirements": {"*": {"instanceType": "mem1_ssd1_v2_x4"}}},
"azure:westus": {},
}
self.assertIn("inconsistent-regional-requirements", issue_codes(manifest))
manifest["regionalOptions"]["azure:westus"] = {
"systemRequirements": {"*": {"instanceType": "azure:mem1_ssd1_x4"}}
}
self.assertNotIn("inconsistent-regional-requirements", issue_codes(manifest))
def test_resource_selectors_are_mutually_exclusive(self) -> None:
manifest = valid_app_manifest()
manifest["regionalOptions"] = {
"aws:us-east-1": {
"systemRequirements": {
"*": {
"instanceType": "mem1_ssd1_v2_x4",
"clusterSpec": {"type": "spark"},
}
}
}
}
self.assertIn("resource-selector-conflict", issue_codes(manifest))
def test_instance_type_selector_needs_a_non_empty_string_list(self) -> None:
def with_selector(selector):
manifest = valid_app_manifest()
manifest["regionalOptions"] = {
"aws:us-east-1": {
"systemRequirements": {"*": {"instanceTypeSelector": selector}}
}
}
return issue_codes(manifest)
self.assertIn("allowed-instance-types", with_selector({"allowedInstanceTypes": []}))
self.assertIn("allowed-instance-types", with_selector({"allowedInstanceTypes": [1]}))
self.assertIn(
"duplicate-instance-type",
with_selector({"allowedInstanceTypes": ["a", "a"]}),
)
self.assertNotIn(
"allowed-instance-types",
with_selector({"allowedInstanceTypes": ["mem1_ssd1_v2_x4"]}),
)
class AccessAndSecretTests(unittest.TestCase):
def test_broad_and_privileged_access_is_warned_about(self) -> None:
manifest = valid_app_manifest()
manifest["access"] = {"network": ["*"]}
self.assertIn("broad-network", issue_codes(manifest))
manifest["access"] = {"network": ["api.example.invalid"]}
self.assertNotIn("broad-network", issue_codes(manifest))
manifest["access"] = {"project": "ADMINISTER"}
self.assertIn("admin-project-access", issue_codes(manifest))
manifest["access"] = {"allProjects": "VIEW"}
self.assertIn("all-projects-access", issue_codes(manifest))
manifest["access"] = {"developer": True}
self.assertIn("developer-access", issue_codes(manifest))
def test_access_levels_are_checked_against_the_documented_set(self) -> None:
manifest = valid_app_manifest()
manifest["access"] = {"project": "READWRITE"}
self.assertIn("access-level", issue_codes(manifest))
def test_embedded_credentials_are_found_at_any_depth(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["assetDepends"] = [{"api_token": "sk-live-abc123"}]
self.assertIn("embedded-secret", issue_codes(manifest))
manifest = valid_app_manifest()
manifest["details"] = {"nested": {"private-key": "-----BEGIN..."}}
self.assertIn("embedded-secret", issue_codes(manifest))
def test_placeholder_credentials_are_not_flagged(self) -> None:
for placeholder in ("", "changeme", "PLACEHOLDER", "<token>", " redacted "):
with self.subTest(value=placeholder):
manifest = valid_app_manifest()
manifest["details"] = {"password": placeholder}
self.assertNotIn("embedded-secret", issue_codes(manifest))
def test_a_secret_key_holding_false_is_a_setting_not_a_credential(self) -> None:
manifest = valid_app_manifest()
manifest["details"] = {"use_token": False}
self.assertNotIn("embedded-secret", issue_codes(manifest))
def test_secret_detection_matches_whole_words_only(self) -> None:
# SECRET_KEY_RE anchors on word boundaries so `tokenizer` is not a token.
manifest = valid_app_manifest()
manifest["details"] = {"tokenizer": "bpe"}
self.assertNotIn("embedded-secret", issue_codes(manifest))
def test_https_ports_are_restricted(self) -> None:
manifest = valid_app_manifest()
manifest["httpsApp"] = {"ports": [443]}
self.assertNotIn("https-ports", issue_codes(manifest))
for bad in ([], [22], [443, 9000], "443"):
with self.subTest(ports=bad):
manifest["httpsApp"] = {"ports": bad}
self.assertIn("https-ports", issue_codes(manifest))
class CommandLineTests(unittest.TestCase):
def _run(self, manifest, *flags) -> subprocess.CompletedProcess:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "dxapp.json"
path.write_text(json.dumps(manifest), encoding="utf-8")
return subprocess.run(
[sys.executable, str(SCRIPTS / "validate_dxapp.py"), str(path), *flags],
capture_output=True,
text=True,
timeout=60,
)
def test_a_clean_manifest_exits_zero(self) -> None:
result = self._run(valid_app_manifest())
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("Errors: 0", result.stdout)
def test_errors_exit_one(self) -> None:
manifest = valid_app_manifest()
del manifest["name"]
self.assertEqual(self._run(manifest).returncode, 1)
def test_strict_promotes_warnings_to_a_failure(self) -> None:
manifest = valid_app_manifest()
manifest["runSpec"]["release"] = "20.04" # warning only
self.assertEqual(self._run(manifest).returncode, 0)
self.assertEqual(self._run(manifest, "--strict").returncode, 1)
def test_json_output_is_machine_readable(self) -> None:
manifest = valid_app_manifest()
manifest["access"] = {"network": ["*"]}
result = self._run(manifest, "--json")
report = json.loads(result.stdout)
self.assertTrue(report["valid"])
self.assertEqual(report["kind"], "app")
self.assertEqual(report["warnings"], 1)
self.assertEqual(report["issues"][0]["code"], "broad-network")
def test_unreadable_manifest_exits_two_with_a_parse_issue(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "dxapp.json"
path.write_text("{not json", encoding="utf-8")
result = subprocess.run(
[
sys.executable,
str(SCRIPTS / "validate_dxapp.py"),
str(path),
"--json",
],
capture_output=True,
text=True,
timeout=60,
)
self.assertEqual(result.returncode, 2)
report = json.loads(result.stdout)
self.assertFalse(report["valid"])
self.assertEqual(report["issues"][0]["code"], "parse")
class SummaryTests(unittest.TestCase):
def test_summarize_counts_both_severities(self) -> None:
issues = [
validate_dxapp.Issue("error", "a", "$", "m"),
validate_dxapp.Issue("warning", "b", "$", "m"),
validate_dxapp.Issue("warning", "c", "$", "m"),
]
self.assertEqual(validate_dxapp.summarize(issues), {"error": 1, "warning": 2})
def test_summarize_of_nothing_is_zeroed_not_empty(self) -> None:
self.assertEqual(validate_dxapp.summarize([]), {"error": 0, "warning": 0})
class SdkInspectionTests(unittest.TestCase):
"""`inspect_dxpy`'s pure helpers; the SDK-dependent report needs dxpy."""
def test_version_comparison_is_numeric_not_lexicographic(self) -> None:
parse = inspect_dxpy.numeric_version
self.assertEqual(parse("0.410.0"), (0, 410, 0))
# The bug this guards: "0.99" > "0.410" as strings, but not as versions.
self.assertLess(parse("0.99.0"), parse("0.410.0"))
self.assertGreater(parse("1.0.0"), parse("0.410.0"))
def test_version_comparison_tolerates_non_numeric_segments(self) -> None:
# Never raise on an unexpected version string -- the script's job is to
# report, not to crash on a pre-release tag.
self.assertIsInstance(inspect_dxpy.numeric_version("0.410.0rc1"), tuple)
self.assertIsInstance(inspect_dxpy.numeric_version(""), tuple)
def test_documented_baseline_is_a_parseable_version(self) -> None:
self.assertRegex(inspect_dxpy.DOCUMENTED_BASELINE, r"^\d+\.\d+\.\d+$")
self.assertTrue(inspect_dxpy.numeric_version(inspect_dxpy.DOCUMENTED_BASELINE))
def test_required_symbols_are_dxpy_qualified_names(self) -> None:
self.assertTrue(inspect_dxpy.REQUIRED_SYMBOLS)
for name in inspect_dxpy.REQUIRED_SYMBOLS:
with self.subTest(symbol=name):
self.assertTrue(name.startswith("dxpy."))
def test_missing_symbol_detection_reads_the_report_not_the_sdk(self) -> None:
symbol = "dxpy.upload_local_file"
self.assertIn(symbol, inspect_dxpy.REQUIRED_SYMBOLS)
# An empty report means nothing was observed, so everything is missing.
self.assertEqual(
inspect_dxpy.missing_required_symbols({}),
sorted(inspect_dxpy.REQUIRED_SYMBOLS),
)
report = {
"symbols": {
"dxpy": [{"qualified_name": symbol, "available": False}],
}
}
self.assertIn(symbol, inspect_dxpy.missing_required_symbols(report))
report["symbols"]["dxpy"][0]["available"] = True
self.assertNotIn(symbol, inspect_dxpy.missing_required_symbols(report))
def test_missing_method_detection_reads_the_report(self) -> None:
missing = inspect_dxpy.missing_required_methods({})
self.assertIn("DXFile.describe", missing)
self.assertEqual(missing, sorted(missing))
report = {"methods": {"DXFile": {"describe": {"available": True}}}}
self.assertNotIn("DXFile.describe", inspect_dxpy.missing_required_methods(report))
def test_report_generation_needs_the_sdk(self) -> None:
try:
importlib.import_module("dxpy")
except ImportError:
self.skipTest("dxpy is not installed; run under --isolated")
report = inspect_dxpy.build_report()
self.assertEqual(
set(report),
{
"schema_version",
"python",
"platform",
"documented_baseline",
"dxpy_version",
"version_meets_baseline",
"symbols",
"methods",
"known_legacy_checks",
},
)
self.assertEqual(
report["documented_baseline"], inspect_dxpy.DOCUMENTED_BASELINE
)
# The report must be usable by the two consumers below.
self.assertEqual(inspect_dxpy.missing_required_symbols(report), [])
self.assertEqual(inspect_dxpy.missing_required_methods(report), [])
def test_the_installed_sdk_meets_the_documented_baseline(self) -> None:
try:
importlib.import_module("dxpy")
except ImportError:
self.skipTest("dxpy is not installed; run under --isolated")
report = inspect_dxpy.build_report()
# Recomputed independently of the script's own comparison.
self.assertEqual(
report["version_meets_baseline"],
inspect_dxpy.numeric_version(report["dxpy_version"])
>= inspect_dxpy.numeric_version(inspect_dxpy.DOCUMENTED_BASELINE),
)
def test_the_report_is_json_serialisable(self) -> None:
try:
importlib.import_module("dxpy")
except ImportError:
self.skipTest("dxpy is not installed; run under --isolated")
# --json is the documented machine-readable path.
json.dumps(inspect_dxpy.build_report())
if __name__ == "__main__":
unittest.main()

226
tests/docx/test_scripts.py Normal file
View File

@@ -0,0 +1,226 @@
"""Tests for the docx skill's OOXML editors.
The shared `office/` tree -- zip safety, relationship resolution, repacking --
is covered by the contract, since pptx and xlsx ship byte-identical copies.
What is specific to docx is the run merger and the comment writer, and both
edit `word/document.xml` in place, so the tests build a real unpacked package
in a temporary directory and assert on the XML that comes back out.
`merge_runs` is the one with a genuine correctness risk: Word splits a single
sentence across many `<w:r>` runs, and merging them must preserve the visible
text exactly, including whitespace held by `xml:space="preserve"`.
"""
from __future__ import annotations
import sys
import tempfile
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "docx"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pytest.importorskip("defusedxml", reason="docx scripts need defusedxml")
import merge_runs # noqa: E402
OfficeTests = skill_contract.office.office_test_case(SKILL_ROOT)
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
WORDML = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
def document(body: str) -> str:
return (
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>\n'
f'<w:document xmlns:w="{WORDML}"><w:body>{body}</w:body></w:document>'
)
def run(text: str, *, preserve: bool = False, properties: str = "") -> str:
space = ' xml:space="preserve"' if preserve else ""
return f"<w:r>{properties}<w:t{space}>{text}</w:t></w:r>"
class UnpackedPackage:
"""A minimal unpacked .docx directory."""
def __init__(self, root: Path, body: str) -> None:
self.root = root
(root / "word").mkdir(parents=True, exist_ok=True)
(root / "word" / "document.xml").write_text(document(body), encoding="utf-8")
(root / "[Content_Types].xml").write_text("<Types/>", encoding="utf-8")
@property
def xml(self) -> str:
return (self.root / "word" / "document.xml").read_text(encoding="utf-8")
def visible_text(self) -> str:
import re
return "".join(re.findall(r"<w:t[^>]*>(.*?)</w:t>", self.xml, re.DOTALL))
class MergeRunsTestCase(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def package(self, body: str) -> UnpackedPackage:
return UnpackedPackage(self.root, body)
class TextPreservationTests(MergeRunsTestCase):
def test_adjacent_identical_runs_merge_without_changing_the_text(self) -> None:
# The trailing space needs xml:space="preserve" to be significant --
# without it OOXML says the space is not part of the text, which is
# what `office/helpers.rendered_text` implements.
package = self.package(
"<w:p>" + run("Hello ", preserve=True) + run("world") + run("!") + "</w:p>"
)
before = package.visible_text()
merge_runs.merge_runs(str(self.root))
self.assertEqual(package.visible_text(), before)
self.assertEqual(package.visible_text(), "Hello world!")
def test_unpreserved_trailing_whitespace_is_insignificant(self) -> None:
# The counterpart: without xml:space="preserve" the space is dropped,
# matching how Word itself reads the part.
package = self.package("<w:p>" + run("Hello ") + run("world") + "</w:p>")
merge_runs.merge_runs(str(self.root))
self.assertEqual(package.visible_text(), "Helloworld")
def test_merging_reduces_the_run_count(self) -> None:
package = self.package(
"<w:p>" + run("a") + run("b") + run("c") + "</w:p>"
)
merged, _ = merge_runs.merge_runs(str(self.root))
self.assertGreater(merged, 0)
self.assertEqual(package.xml.count("<w:r>"), 1)
def test_runs_with_different_formatting_are_not_merged(self) -> None:
bold = "<w:rPr><w:b/></w:rPr>"
package = self.package(
"<w:p>" + run("plain") + run("bold", properties=bold) + "</w:p>"
)
merge_runs.merge_runs(str(self.root))
self.assertEqual(package.xml.count("<w:r>"), 2)
self.assertEqual(package.visible_text(), "plainbold")
def test_preserved_whitespace_survives_a_merge(self) -> None:
# Losing xml:space="preserve" silently deletes the space between words.
package = self.package(
"<w:p>" + run("one ", preserve=True) + run("two", preserve=True) + "</w:p>"
)
merge_runs.merge_runs(str(self.root))
self.assertEqual(package.visible_text(), "one two")
def test_runs_in_different_paragraphs_are_not_merged_across(self) -> None:
package = self.package(
"<w:p>" + run("first") + "</w:p><w:p>" + run("second") + "</w:p>"
)
merge_runs.merge_runs(str(self.root))
self.assertEqual(package.xml.count("<w:p>"), 2)
self.assertEqual(package.visible_text(), "firstsecond")
def test_a_document_with_nothing_to_merge_keeps_its_content(self) -> None:
# The XML declaration is rewritten by minidom's serializer, so compare
# the document element rather than the whole byte stream.
package = self.package("<w:p>" + run("only one run") + "</w:p>")
merged, _ = merge_runs.merge_runs(str(self.root))
self.assertEqual(merged, 0)
self.assertEqual(package.visible_text(), "only one run")
self.assertEqual(package.xml.count("<w:r>"), 1)
def test_an_empty_document_does_not_raise(self) -> None:
self.package("")
merged, _ = merge_runs.merge_runs(str(self.root))
self.assertEqual(merged, 0)
def test_the_result_is_still_well_formed_xml(self) -> None:
import defusedxml.ElementTree as ElementTree
package = self.package(
"<w:p>" + "".join(run(str(index)) for index in range(20)) + "</w:p>"
)
merge_runs.merge_runs(str(self.root))
ElementTree.fromstring(package.xml)
self.assertEqual(package.visible_text(), "".join(str(i) for i in range(20)))
def test_merging_is_idempotent(self) -> None:
package = self.package("<w:p>" + run("a") + run("b") + "</w:p>")
merge_runs.merge_runs(str(self.root))
once = package.xml
merged, _ = merge_runs.merge_runs(str(self.root))
self.assertEqual(merged, 0)
self.assertEqual(package.xml, once)
class HelperTests(unittest.TestCase):
"""`merge_runs` walks a minidom tree, so the helpers take minidom nodes."""
def _runs(self, body: str) -> list:
import defusedxml.minidom
dom = defusedxml.minidom.parseString(document(body))
root = dom.documentElement
return merge_runs._find_runs(root, merge_runs._run_tag_names(root))
def test_adjacency_requires_the_elements_to_be_siblings_in_order(self) -> None:
runs = self._runs("<w:p>" + run("a") + run("b") + "</w:p>")
self.assertEqual(len(runs), 2)
self.assertTrue(merge_runs._is_adjacent(runs[0], runs[1]))
self.assertFalse(merge_runs._is_adjacent(runs[1], runs[0]))
def test_an_intervening_element_breaks_adjacency(self) -> None:
runs = self._runs(
"<w:p>" + run("a") + "<w:bookmarkStart/>" + run("b") + "</w:p>"
)
self.assertFalse(merge_runs._is_adjacent(runs[0], runs[1]))
def test_element_matching_is_namespace_aware(self) -> None:
runs = self._runs("<w:p>" + run("a") + "</w:p>")
self.assertTrue(merge_runs._is_element(runs[0], "r"))
self.assertFalse(merge_runs._is_element(runs[0], "p"))
def test_run_tag_names_are_discovered_from_the_document(self) -> None:
names = merge_runs._run_tag_names(
__import__("defusedxml.minidom", fromlist=["parseString"])
.parseString(document("<w:p>" + run("a") + "</w:p>"))
.documentElement
)
self.assertTrue(names)
class TemplateTests(unittest.TestCase):
def test_the_comment_templates_are_shipped_and_parse(self) -> None:
import defusedxml.ElementTree as ElementTree
templates = sorted((SCRIPTS / "templates").glob("*.xml"))
self.assertTrue(templates, "no comment templates shipped")
for template in templates:
with self.subTest(template=template.name):
ElementTree.fromstring(template.read_text(encoding="utf-8"))
def test_the_documented_comment_parts_are_all_present(self) -> None:
# Word needs every one of these to open a commented document.
shipped = {path.name for path in (SCRIPTS / "templates").glob("*.xml")}
for required in (
"comments.xml",
"commentsExtended.xml",
"commentsIds.xml",
"people.xml",
):
with self.subTest(part=required):
self.assertIn(required, shipped)
if __name__ == "__main__":
unittest.main()

912
tests/etetoolkit/test_scripts.py Normal file
View File

@@ -0,0 +1,912 @@
"""Tests for the ETE 4 tree helpers.
Three things in these scripts can be wrong in ways a user would not notice.
The diagnostics in `tree_stats` are numbers people quote in papers, so they are
checked against a Newick string whose leaf count, branch lengths, farthest-leaf
distance, and support values are all known by construction -- including the
detail that the Newick *parser number* decides whether an internal label is a
name or a support value. The validators in both scripts are proved in both
directions: acceptable input passes silently, and each rejection names the
argument at fault. And `validate_bind_address` is a security boundary --
SmartView serves an unauthenticated interactive viewer, so binding it past
loopback must require the explicit `--allow-remote-bind` opt-in -- so every
loopback spelling and every remote spelling is exercised.
Everything here needs ete4 itself; there is no pure-Python half worth testing
separately, because the interesting logic is about what ETE returns.
"""
from __future__ import annotations
import argparse
import contextlib
import io
import json
import sys
import tempfile
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "etetoolkit"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pytest.importorskip("ete4", reason="etetoolkit skill needs ete4")
import quick_visualize # noqa: E402
import tree_operations # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
#: Four taxa, two clades, *named* internal nodes -- so it needs parser 1, which
#: reads internal labels as names. Total branch length 8 (1+1+1+1+2+2); the
#: farthest leaf sits 3 from the root.
BALANCED = "((A:1,B:1)AB:1,(C:2,D:2)CD:1)root;"
#: The same shape with unlabelled internal nodes, so the default parser 0 reads
#: it. Total branch length 6, and every leaf is 2 from the root.
PLAIN = "((A:1,B:1):1,(C:1,D:1):1);"
class TreeFixtureCase(unittest.TestCase):
"""Base class giving each test a scratch directory and a `newick` helper."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def newick(self, text: str, name: str = "tree.nwk") -> Path:
path = self.root / name
path.write_text(text + "\n", encoding="utf-8")
return path
class ParserSpecTests(unittest.TestCase):
"""`--parser` accepts both ETE's numeric ids and its named aliases."""
def test_a_numeric_parser_id_stays_an_integer(self) -> None:
# ETE looks parsers up by int; passing "0" as a string selects nothing.
for text, expected in (("0", 0), ("1", 1), (" 5 ", 5), ("-1", -1)):
with self.subTest(text=text):
self.assertEqual(tree_operations.parser_spec(text), expected)
def test_a_named_parser_stays_a_string(self) -> None:
self.assertEqual(tree_operations.parser_spec("newick"), "newick")
def test_an_empty_parser_is_rejected(self) -> None:
for text in ("", " "):
with self.subTest(text=text):
with self.assertRaises(argparse.ArgumentTypeError):
tree_operations.parser_spec(text)
def test_both_scripts_agree_on_the_parser_type(self) -> None:
# The two CLIs define the converter separately; a divergence would make
# `--parser 1` mean different things in `stats` and in `quick_visualize`.
self.assertEqual(quick_visualize.parser_spec("1"), tree_operations.parser_spec("1"))
self.assertEqual(
quick_visualize.parser_spec("newick"), tree_operations.parser_spec("newick")
)
class CommaSeparatedTests(unittest.TestCase):
def test_whitespace_and_empty_fields_are_dropped(self) -> None:
self.assertEqual(
tree_operations.comma_separated("name, dist ,,support,"),
["name", "dist", "support"],
)
def test_an_empty_string_selects_no_properties(self) -> None:
# `--props ""` must mean "none", not a single empty property name, or
# tree.write() is asked for a property no node has.
self.assertEqual(tree_operations.comma_separated(""), [])
class ModeSpecTests(unittest.TestCase):
def test_short_and_long_layout_names_normalise_to_the_long_form(self) -> None:
for text in ("r", "R", "rectangular", "RECTANGULAR"):
with self.subTest(text=text):
self.assertEqual(quick_visualize.mode_spec(text), "rectangular")
for text in ("c", "C", "circular"):
with self.subTest(text=text):
self.assertEqual(quick_visualize.mode_spec(text), "circular")
def test_an_unknown_layout_is_rejected(self) -> None:
for text in ("radial", "", "rect"):
with self.subTest(text=text):
with self.assertRaisesRegex(
argparse.ArgumentTypeError, "rectangular/r or circular/c"
):
quick_visualize.mode_spec(text)
class TreeLoadingTests(TreeFixtureCase):
def test_a_valid_newick_file_loads_with_its_leaves(self) -> None:
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
self.assertEqual(sorted(tree.leaf_names()), ["A", "B", "C", "D"])
def test_a_missing_file_is_reported_as_user_error(self) -> None:
with self.assertRaisesRegex(tree_operations.UserInputError, "does not exist"):
tree_operations.load_tree(self.root / "absent.nwk", 0)
def test_a_directory_is_not_a_tree(self) -> None:
with self.assertRaises(tree_operations.UserInputError):
tree_operations.load_tree(self.root, 0)
def test_malformed_newick_becomes_a_user_error_not_a_traceback(self) -> None:
# A truncated tree is the commonest real failure; the CLI must name the
# file and the parser rather than surfacing NewickError.
path = self.newick("((A:1,B:1)")
with self.assertRaisesRegex(tree_operations.UserInputError, "could not parse"):
tree_operations.load_tree(path, 0)
def test_both_scripts_load_trees_identically(self) -> None:
path = self.newick(BALANCED)
self.assertEqual(
sorted(quick_visualize.load_tree(path, 1).leaf_names()),
sorted(tree_operations.load_tree(path, 1).leaf_names()),
)
class NumericSummaryTests(unittest.TestCase):
def test_an_empty_list_summarises_to_none_rather_than_raising(self) -> None:
# A tree with no branch lengths at all must report "none", not crash in
# statistics.fmean on an empty sequence.
self.assertIsNone(tree_operations.numeric_summary([]))
def test_the_four_statistics_are_the_textbook_ones(self) -> None:
summary = tree_operations.numeric_summary([1.0, 2.0, 3.0, 6.0])
self.assertEqual(summary["minimum"], 1.0)
self.assertEqual(summary["maximum"], 6.0)
self.assertEqual(summary["mean"], 3.0)
self.assertEqual(summary["median"], 2.5)
def test_a_single_value_is_its_own_summary(self) -> None:
self.assertEqual(
tree_operations.numeric_summary([4.0]),
{"minimum": 4.0, "maximum": 4.0, "mean": 4.0, "median": 4.0},
)
class TreeStatsTests(TreeFixtureCase):
"""Every number below is read off the Newick string, not off the code."""
def stats(self, text: str, parser: int = 1) -> dict:
path = self.newick(text)
return tree_operations.tree_stats(tree_operations.load_tree(path, parser), path)
def test_the_node_counts_match_the_newick_string(self) -> None:
stats = self.stats(BALANCED)
self.assertEqual(stats["leaf_count"], 4)
# root, AB, CD.
self.assertEqual(stats["internal_node_count"], 3)
self.assertEqual(stats["total_node_count"], 7)
self.assertEqual(stats["root_child_count"], 2)
self.assertEqual(stats["polytomy_count"], 0)
self.assertEqual(stats["unary_node_count"], 0)
def test_branch_length_statistics_exclude_the_root(self) -> None:
# Six non-root branches: AB=1, A=1, B=1, CD=1, C=2, D=2.
summary = self.stats(BALANCED)["branch_lengths"]
self.assertEqual(summary["minimum"], 1.0)
self.assertEqual(summary["maximum"], 2.0)
self.assertEqual(summary["median"], 1.0)
self.assertAlmostEqual(summary["mean"], 8 / 6)
def test_the_farthest_leaf_distance_is_the_root_to_tip_path(self) -> None:
stats = self.stats(BALANCED)
# root -> CD (1) -> C or D (2). C and D tie, so accept either name.
self.assertEqual(stats["farthest_leaf_distance"], 3.0)
self.assertIn(stats["farthest_leaf"], {"C", "D"})
def test_a_polytomy_is_counted_and_a_bifurcating_tree_reports_none(self) -> None:
self.assertEqual(self.stats("((A,B,C),D);")["polytomy_count"], 1)
self.assertEqual(self.stats("((A,B),(C,D));")["polytomy_count"], 0)
def test_a_unary_node_is_counted(self) -> None:
# Single-child nodes survive some pruning routines and break downstream
# tools, so they get their own counter.
stats = self.stats("((A),(B,C));")
self.assertEqual(stats["unary_node_count"], 1)
self.assertEqual(stats["leaf_count"], 3)
def test_duplicate_leaf_names_are_listed_once_each(self) -> None:
self.assertEqual(self.stats("((A,A),(B,C));")["duplicate_leaf_names"], ["A"])
self.assertEqual(self.stats("((A,B),(C,D));")["duplicate_leaf_names"], [])
def test_unnamed_leaves_are_reported_by_position(self) -> None:
# An unnamed leaf has no name to report, so the node id (the path of
# child indices from the root) identifies it instead.
self.assertEqual(self.stats("((A,),(B,C));")["unnamed_leaf_ids"], [[0, 1]])
def test_the_parser_number_decides_whether_a_label_is_a_name_or_support(self) -> None:
# Parser 1 reads "AB"/"CD" as internal *names*, so no support exists;
# parser 0 reads the same position as a support value. Getting this
# backwards silently reports support statistics for a tree that has
# none, or none for a tree that has them.
self.assertIsNone(self.stats(BALANCED, parser=1)["internal_support"])
supported = self.stats("((A:1,B:1)0.95:1,(C:1,D:1)0.80:1);", parser=0)
summary = supported["internal_support"]
self.assertEqual(summary["minimum"], 0.80)
self.assertEqual(summary["maximum"], 0.95)
self.assertAlmostEqual(summary["mean"], 0.875)
def test_the_reported_source_and_version_identify_the_run(self) -> None:
path = self.newick(BALANCED)
stats = tree_operations.tree_stats(tree_operations.load_tree(path, 1), path)
self.assertEqual(stats["source"], str(path))
# The statistics depend on ETE's traversal, so the version is recorded.
self.assertTrue(stats["ete_version"])
class PrintStatsTests(TreeFixtureCase):
def test_json_output_is_machine_readable_and_complete(self) -> None:
path = self.newick(BALANCED)
stats = tree_operations.tree_stats(tree_operations.load_tree(path, 1), path)
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
tree_operations.print_stats(stats, as_json=True)
self.assertEqual(json.loads(buffer.getvalue()), stats)
def test_text_output_says_none_where_a_summary_is_missing(self) -> None:
# Parser 1 leaves support unset; the text report must say so rather
# than crash formatting None.
path = self.newick(BALANCED)
stats = tree_operations.tree_stats(tree_operations.load_tree(path, 1), path)
buffer = io.StringIO()
with contextlib.redirect_stdout(buffer):
tree_operations.print_stats(stats, as_json=False)
text = buffer.getvalue()
self.assertIn("Internal support: none", text)
self.assertIn("Leaves: 4", text)
self.assertIn("Duplicate leaf names: none", text)
class NodeResolutionTests(TreeFixtureCase):
def test_a_unique_name_resolves_to_that_node(self) -> None:
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
node = tree_operations.resolve_unique_node(tree, "AB")
self.assertEqual(sorted(node.leaf_names()), ["A", "B"])
def test_an_absent_name_is_refused(self) -> None:
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
with self.assertRaisesRegex(tree_operations.UserInputError, "node not found"):
tree_operations.resolve_unique_node(tree, "Z")
def test_an_ambiguous_name_is_refused_rather_than_silently_taking_the_first(self) -> None:
# Rooting on "whichever A ETE traversed first" would give a different
# tree run to run, so a duplicate name has to be an error.
tree = tree_operations.load_tree(self.newick("((A,A),(B,C));"), 1)
with self.assertRaisesRegex(tree_operations.UserInputError, "ambiguous"):
tree_operations.resolve_unique_node(tree, "A")
class KeepNameTests(TreeFixtureCase):
def test_names_passed_on_the_command_line_are_returned_in_order(self) -> None:
self.assertEqual(tree_operations.read_keep_names(["A", "B"], None), ["A", "B"])
def test_a_taxon_file_skips_blank_lines_and_comments(self) -> None:
path = self.root / "keep.txt"
path.write_text("# keep these\nA\n\n B \n#C\n", encoding="utf-8")
self.assertEqual(tree_operations.read_keep_names(None, path), ["A", "B"])
def test_an_empty_request_is_refused(self) -> None:
for values, file_path in ((None, None), ([], None)):
with self.subTest(values=values):
with self.assertRaisesRegex(
tree_operations.UserInputError, "at least one taxon"
):
tree_operations.read_keep_names(values, file_path)
def test_a_taxon_file_of_only_comments_is_refused(self) -> None:
path = self.root / "keep.txt"
path.write_text("# nothing here\n\n", encoding="utf-8")
with self.assertRaisesRegex(tree_operations.UserInputError, "at least one taxon"):
tree_operations.read_keep_names(None, path)
def test_duplicate_requests_are_refused(self) -> None:
# `prune --keep A A` would silently retain one leaf while reporting two.
with self.assertRaisesRegex(tree_operations.UserInputError, "duplicate"):
tree_operations.read_keep_names(["A", "B", "A"], None)
def test_a_missing_taxon_file_is_refused(self) -> None:
with self.assertRaisesRegex(tree_operations.UserInputError, "taxon file"):
tree_operations.read_keep_names(None, self.root / "absent.txt")
class RequestedNameValidationTests(TreeFixtureCase):
def test_names_that_all_resolve_pass_silently(self) -> None:
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
self.assertIsNone(tree_operations.validate_requested_names(tree, ["A", "D"]))
def test_an_internal_node_name_is_not_a_leaf(self) -> None:
# `prune` works on leaf names; "AB" exists in the tree but is not one.
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
with self.assertRaisesRegex(tree_operations.UserInputError, "absent"):
tree_operations.validate_requested_names(tree, ["AB"])
def test_a_leaf_name_duplicated_in_the_tree_is_refused(self) -> None:
tree = tree_operations.load_tree(self.newick("((A,A),(B,C));"), 1)
with self.assertRaisesRegex(tree_operations.UserInputError, "duplicated"):
tree_operations.validate_requested_names(tree, ["A"])
class SaveTreeTests(TreeFixtureCase):
def test_a_saved_tree_reloads_with_the_same_topology(self) -> None:
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
output = self.root / "out.nwk"
tree_operations.save_tree(tree, output, 1, [])
reloaded = tree_operations.load_tree(output, 1)
self.assertEqual(sorted(reloaded.leaf_names()), ["A", "B", "C", "D"])
# Branch lengths survive the round trip, so the total is unchanged.
self.assertEqual(
sum(float(node.dist) for node in reloaded.traverse() if not node.is_root),
8.0,
)
def test_the_file_ends_in_exactly_one_newline(self) -> None:
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
output = self.root / "out.nwk"
tree_operations.save_tree(tree, output, 1, [])
text = output.read_text(encoding="utf-8")
self.assertTrue(text.endswith(";\n"))
self.assertFalse(text.endswith("\n\n"))
def test_a_missing_output_directory_is_refused_before_writing(self) -> None:
tree = tree_operations.load_tree(self.newick(BALANCED), 1)
with self.assertRaisesRegex(tree_operations.UserInputError, "output directory"):
tree_operations.save_tree(tree, self.root / "nope" / "out.nwk", 1, [])
class CommandLineTests(TreeFixtureCase):
"""`main` end to end: the exit codes and the numbers it prints."""
def run_main(self, argv: list[str]) -> tuple[int, str, str]:
out, err = io.StringIO(), io.StringIO()
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
code = tree_operations.main(argv)
return code, out.getvalue(), err.getvalue()
def test_a_missing_input_exits_two_with_a_message_on_stderr(self) -> None:
code, _, err = self.run_main(["stats", str(self.root / "absent.nwk")])
self.assertEqual(code, 2)
self.assertIn("error:", err)
def test_stats_json_round_trips_through_the_cli(self) -> None:
code, out, _ = self.run_main(
["stats", str(self.newick(BALANCED)), "--parser", "1", "--json"]
)
self.assertEqual(code, 0)
self.assertEqual(json.loads(out)["leaf_count"], 4)
def test_the_wrong_parser_number_is_reported_rather_than_guessed(self) -> None:
# BALANCED labels its internal nodes, so parser 0 -- which reads that
# position as a support value -- cannot convert "AB" to a float. Falling
# back to another parser would silently reinterpret the tree.
code, _, err = self.run_main(["stats", str(self.newick(BALANCED))])
self.assertEqual(code, 2)
self.assertIn("could not parse", err)
def test_leaves_prints_one_name_per_line(self) -> None:
code, out, _ = self.run_main(["leaves", str(self.newick(PLAIN))])
self.assertEqual(code, 0)
self.assertEqual(sorted(out.split()), ["A", "B", "C", "D"])
def test_an_unnamed_leaf_still_occupies_a_line(self) -> None:
# Otherwise the leaf count and the line count disagree downstream.
code, out, _ = self.run_main(["leaves", str(self.newick("((A,),(B,C));"))])
self.assertEqual(code, 0)
self.assertEqual(len(out.splitlines()), 4)
def test_the_ascii_drawing_contains_every_leaf(self) -> None:
code, out, _ = self.run_main(["ascii", str(self.newick(PLAIN))])
self.assertEqual(code, 0)
for name in ("A", "B", "C", "D"):
self.assertIn(name, out)
def test_identical_trees_have_a_robinson_foulds_distance_of_zero(self) -> None:
path = self.newick("((A:1,B:1):1,(C:1,D:1):1);")
code, out, _ = self.run_main(["compare", str(path), str(path), "--unrooted"])
self.assertEqual(code, 0)
result = json.loads(out)
self.assertEqual(result["rf"], 0)
self.assertEqual(result["normalized_rf"], 0.0)
self.assertEqual(result["common_leaf_count"], 4)
def test_the_two_distinct_four_taxon_topologies_are_maximally_apart(self) -> None:
# Unrooted, ((A,B),(C,D)) and ((A,C),(B,D)) each have exactly one
# non-trivial split and the splits differ, so RF = 1 + 1 = 2 and the
# maximum possible RF for four taxa is also 2.
first = self.newick("((A:1,B:1):1,(C:1,D:1):1);", "a.nwk")
second = self.newick("((A:1,C:1):1,(B:1,D:1):1);", "b.nwk")
code, out, _ = self.run_main(["compare", str(first), str(second), "--unrooted"])
self.assertEqual(code, 0)
result = json.loads(out)
self.assertEqual(result["rf"], 2)
self.assertEqual(result["max_rf"], 2)
self.assertEqual(result["normalized_rf"], 1.0)
self.assertEqual(result["common_leaves"], ["A", "B", "C", "D"])
def test_comparing_a_tree_with_duplicate_names_is_refused(self) -> None:
# Robinson-Foulds is defined over sets of leaf labels; duplicates make
# the split sets meaningless, so the comparison must not run.
good = self.newick("((A,B),(C,D));", "good.nwk")
bad = self.newick("((A,A),(C,D));", "bad.nwk")
code, _, err = self.run_main(["compare", str(good), str(bad)])
self.assertEqual(code, 2)
self.assertIn("duplicate leaf names", err)
def test_comparing_a_tree_with_unnamed_leaves_is_refused(self) -> None:
good = self.newick("((A,B),(C,D));", "good.nwk")
bad = self.newick("((A,),(C,D));", "bad.nwk")
code, _, err = self.run_main(["compare", str(good), str(bad)])
self.assertEqual(code, 2)
self.assertIn("unnamed leaves", err)
def test_pruning_keeps_only_the_requested_leaves(self) -> None:
output = self.root / "pruned.nwk"
code, out, _ = self.run_main(
[
"prune",
str(self.newick("((A:1,B:1):1,(C:1,D:1):1);")),
str(output),
"--keep",
"A",
"B",
"--output-parser",
"1",
]
)
self.assertEqual(code, 0)
self.assertIn("Retained 2 leaves", out)
self.assertEqual(
sorted(tree_operations.load_tree(output, 1).leaf_names()), ["A", "B"]
)
def test_pruning_preserves_the_path_length_between_retained_leaves(self) -> None:
# A--B is 1+1 = 2 in the input; with --preserve-branch-length the
# collapsed internal node's length must be folded in, not discarded.
output = self.root / "pruned.nwk"
code, _, _ = self.run_main(
[
"prune",
str(self.newick("((A:1,B:1):5,(C:1,D:1):1);")),
str(output),
"--keep",
"A",
"B",
"--output-parser",
"1",
]
)
self.assertEqual(code, 0)
pruned = tree_operations.load_tree(output, 1)
leaves = {leaf.name: leaf for leaf in pruned.leaves()}
self.assertEqual(pruned.get_distance(leaves["A"], leaves["B"]), 2.0)
def test_pruning_to_an_absent_leaf_is_refused(self) -> None:
code, _, err = self.run_main(
[
"prune",
str(self.newick(PLAIN)),
str(self.root / "pruned.nwk"),
"--keep",
"Z",
]
)
self.assertEqual(code, 2)
self.assertIn("absent", err)
def test_rerooting_on_an_outgroup_preserves_the_leaves_and_total_length(self) -> None:
# Rerooting relocates the root along an existing branch; it must not
# create, destroy, or rescale any evolutionary distance.
output = self.root / "rerooted.nwk"
code, out, _ = self.run_main(
[
"reroot",
str(self.newick("((A:1,B:1):1,(C:1,D:1):1);")),
str(output),
"--outgroup",
"A",
"--output-parser",
"1",
]
)
self.assertEqual(code, 0)
self.assertIn("outgroup 'A'", out)
rerooted = tree_operations.load_tree(output, 1)
self.assertEqual(sorted(rerooted.leaf_names()), ["A", "B", "C", "D"])
self.assertAlmostEqual(
sum(float(node.dist) for node in rerooted.traverse() if not node.is_root),
6.0,
)
def test_midpoint_rooting_reports_which_midpoint_it_used(self) -> None:
source = self.newick("((A:1,B:1):1,(C:1,D:1):1);")
code, out, _ = self.run_main(
["reroot", str(source), str(self.root / "m.nwk"), "--midpoint"]
)
self.assertEqual(code, 0)
self.assertIn("branch-length midpoint", out)
code, out, _ = self.run_main(
[
"reroot",
str(source),
str(self.root / "t.nwk"),
"--midpoint",
"--topological",
]
)
self.assertEqual(code, 0)
self.assertIn("topological midpoint", out)
def test_topological_without_midpoint_is_refused(self) -> None:
# --topological only changes how the midpoint is found, so pairing it
# with --outgroup would silently do nothing.
code, _, err = self.run_main(
[
"reroot",
str(self.newick(PLAIN)),
str(self.root / "out.nwk"),
"--outgroup",
"A",
"--topological",
]
)
self.assertEqual(code, 2)
self.assertIn("--topological applies only to --midpoint", err)
def test_converting_between_parsers_keeps_the_leaf_set(self) -> None:
output = self.root / "converted.nwk"
code, _, _ = self.run_main(
[
"convert",
str(self.newick(BALANCED)),
str(output),
"--input-parser",
"1",
"--output-parser",
"1",
]
)
self.assertEqual(code, 0)
self.assertEqual(
sorted(tree_operations.load_tree(output, 1).leaf_names()),
["A", "B", "C", "D"],
)
class BindAddressTests(unittest.TestCase):
"""SmartView serves an unauthenticated viewer, so the bind host is a gate."""
def test_every_loopback_spelling_is_allowed_without_the_opt_in(self) -> None:
# A validator that rejected these would make the default `--host
# 127.0.0.1` unusable, so the permissive direction matters as much as
# the restrictive one.
for host in ("127.0.0.1", "127.0.0.2", "::1", "localhost", "LocalHost"):
with self.subTest(host=host):
self.assertIsNone(quick_visualize.validate_bind_address(host, False))
def test_the_wildcard_address_is_refused_without_the_opt_in(self) -> None:
# 0.0.0.0 is not loopback: it publishes the viewer on every interface.
with self.assertRaisesRegex(quick_visualize.UserInputError, "non-loopback"):
quick_visualize.validate_bind_address("0.0.0.0", False)
def test_routable_addresses_are_refused_without_the_opt_in(self) -> None:
for host in ("192.168.1.10", "10.0.0.5", "8.8.8.8", "::", "2001:db8::1"):
with self.subTest(host=host):
with self.assertRaises(quick_visualize.UserInputError):
quick_visualize.validate_bind_address(host, False)
def test_a_host_name_that_merely_starts_with_localhost_is_refused(self) -> None:
# "localhost.example.com" resolves wherever its DNS says; only the
# exact name is treated as loopback.
for host in ("localhost.example.com", "notlocalhost", "example.com"):
with self.subTest(host=host):
with self.assertRaisesRegex(
quick_visualize.UserInputError, "--allow-remote-bind"
):
quick_visualize.validate_bind_address(host, False)
def test_the_opt_in_permits_exactly_what_it_says(self) -> None:
for host in ("0.0.0.0", "192.168.1.10", "example.com", "::"):
with self.subTest(host=host):
self.assertIsNone(quick_visualize.validate_bind_address(host, True))
class SupportFractionTests(unittest.TestCase):
def test_percentages_are_normalised_and_fractions_left_alone(self) -> None:
# Bootstrap support is written both as 0-1 and as 0-100; colouring by
# support has to mean the same thing either way.
self.assertEqual(quick_visualize.support_fraction(95), 0.95)
self.assertEqual(quick_visualize.support_fraction(0.95), 0.95)
self.assertEqual(quick_visualize.support_fraction(100), 1.0)
def test_the_boundary_value_one_is_treated_as_a_fraction(self) -> None:
# `numeric > 1` -- a support of exactly 1 is full support, not 1%.
self.assertEqual(quick_visualize.support_fraction(1), 1.0)
def test_zero_support_stays_zero_and_missing_support_stays_none(self) -> None:
self.assertEqual(quick_visualize.support_fraction(0), 0.0)
self.assertIsNone(quick_visualize.support_fraction(None))
class SupportColorTests(TreeFixtureCase):
def args(self, extra: list[str] | None = None) -> argparse.Namespace:
return quick_visualize.build_parser().parse_args(
["tree.nwk", *(extra or [])]
)
def test_each_support_band_gets_its_own_colour(self) -> None:
path = self.newick("((A:1,B:1)95:1,((C:1,D:1)70:1,(E:1,F:1)50:1)80:1)100;")
tree = quick_visualize.load_tree(path, 0)
args = self.args()
by_support = {
float(node.support): quick_visualize.support_color(node, args)
for node in tree.traverse()
if not node.is_leaf
}
self.assertEqual(by_support[95.0], args.high_support_color)
# 70 normalises to exactly the moderate threshold, which is inclusive.
self.assertEqual(by_support[70.0], args.moderate_support_color)
self.assertEqual(by_support[80.0], args.moderate_support_color)
self.assertEqual(by_support[50.0], args.low_support_color)
def test_a_node_with_no_support_gets_the_missing_colour(self) -> None:
tree = quick_visualize.load_tree(self.newick("((A:1,B:1):1,(C:1,D:1):1);"), 0)
args = self.args()
internal = next(node for node in tree.traverse() if not node.is_leaf)
self.assertIsNone(internal.support)
self.assertEqual(
quick_visualize.support_color(internal, args), args.missing_support_color
)
def test_custom_thresholds_move_the_bands(self) -> None:
tree = quick_visualize.load_tree(self.newick("((A:1,B:1)80:1,(C:1,D:1)80:1);"), 0)
# The root itself carries no support here, so pick a labelled clade.
node = next(
node
for node in tree.traverse()
if not node.is_leaf and node.support is not None
)
strict = self.args(["--high-support", "0.99", "--moderate-support", "0.9"])
lenient = self.args(["--high-support", "0.8", "--moderate-support", "0.5"])
self.assertEqual(
quick_visualize.support_color(node, strict), strict.low_support_color
)
self.assertEqual(
quick_visualize.support_color(node, lenient), lenient.high_support_color
)
class ValidateArgsTests(unittest.TestCase):
def args(self, extra: list[str] | None = None) -> argparse.Namespace:
return quick_visualize.build_parser().parse_args(["tree.nwk", *(extra or [])])
def test_the_shipped_defaults_validate(self) -> None:
# If the defaults failed their own validator the CLI would be unusable
# with no arguments at all.
self.assertIsNone(quick_visualize.validate_args(self.args()))
def test_support_thresholds_must_be_ordered_and_within_zero_to_one(self) -> None:
for extra in (
["--moderate-support", "0.95", "--high-support", "0.9"],
["--high-support", "1.5"],
["--moderate-support", "-0.1"],
):
with self.subTest(extra=extra):
with self.assertRaisesRegex(
quick_visualize.UserInputError, "support thresholds"
):
quick_visualize.validate_args(self.args(extra))
def test_equal_thresholds_are_allowed(self) -> None:
# The comparison chain is <=, so a single cut-off between "high" and
# "low" is a legitimate configuration.
quick_visualize.validate_args(
self.args(["--moderate-support", "0.9", "--high-support", "0.9"])
)
def test_negative_sizes_are_refused_and_zero_is_allowed(self) -> None:
for flag in ("--label-size", "--leaf-size", "--internal-size"):
with self.subTest(flag=flag):
with self.assertRaisesRegex(
quick_visualize.UserInputError, "cannot be negative"
):
quick_visualize.validate_args(self.args([flag, "-1"]))
# Zero is how a user hides a marker, so it must be accepted.
quick_visualize.validate_args(self.args([flag, "0"]))
def test_dimensions_must_be_positive_when_given(self) -> None:
for flag in ("--width", "--height", "--dpi"):
with self.subTest(flag=flag):
with self.assertRaisesRegex(
quick_visualize.UserInputError, "must be positive"
):
quick_visualize.validate_args(self.args([flag, "0"]))
quick_visualize.validate_args(self.args([flag, "1"]))
def test_the_port_range_is_the_tcp_range(self) -> None:
for port in ("0", "65536", "-1"):
with self.subTest(port=port):
with self.assertRaisesRegex(quick_visualize.UserInputError, "port"):
quick_visualize.validate_args(self.args(["--port", port]))
for port in ("1", "8080", "65535"):
with self.subTest(port=port):
quick_visualize.validate_args(self.args(["--port", port]))
def test_the_arc_bounds_are_degrees(self) -> None:
for extra in (["--arc-start", "361"], ["--arc-start", "-361"]):
with self.subTest(extra=extra):
with self.assertRaisesRegex(quick_visualize.UserInputError, "arc-start"):
quick_visualize.validate_args(self.args(extra))
for extra in (["--arc-span", "0"], ["--arc-span", "361"], ["--arc-span", "-90"]):
with self.subTest(extra=extra):
with self.assertRaisesRegex(quick_visualize.UserInputError, "arc-span"):
quick_visualize.validate_args(self.args(extra))
# A full circle and a negative start are both legitimate.
quick_visualize.validate_args(
self.args(["--arc-start", "-360", "--arc-span", "360"])
)
class EngineChoiceTests(unittest.TestCase):
def args(self, extra: list[str]) -> argparse.Namespace:
return quick_visualize.build_parser().parse_args(["tree.nwk", *extra])
def test_no_output_means_the_interactive_engine(self) -> None:
self.assertEqual(quick_visualize.choose_engine(self.args([])), "smartview")
def test_the_suffix_picks_the_engine_that_can_write_it(self) -> None:
# SmartView only screenshots PNG; vector output needs Qt treeview.
cases = {"out.png": "smartview", "out.pdf": "treeview", "out.svg": "treeview"}
for name, expected in cases.items():
with self.subTest(name=name):
self.assertEqual(
quick_visualize.choose_engine(self.args([name])), expected
)
def test_suffix_matching_is_case_insensitive(self) -> None:
self.assertEqual(quick_visualize.choose_engine(self.args(["OUT.PDF"])), "treeview")
def test_an_unwritable_suffix_is_refused_rather_than_guessed(self) -> None:
for name in ("out.tiff", "out.eps", "out"):
with self.subTest(name=name):
with self.assertRaisesRegex(
quick_visualize.UserInputError, "cannot infer renderer"
):
quick_visualize.choose_engine(self.args([name]))
def test_an_explicit_engine_overrides_the_suffix(self) -> None:
self.assertEqual(
quick_visualize.choose_engine(self.args(["out.tiff", "--engine", "treeview"])),
"treeview",
)
self.assertEqual(
quick_visualize.choose_engine(self.args(["out.pdf", "--engine", "smartview"])),
"smartview",
)
class RenderDestinationTests(TreeFixtureCase):
"""Destination checks run before any renderer is loaded or invoked."""
def args(self, extra: list[str] | None = None) -> argparse.Namespace:
return quick_visualize.build_parser().parse_args(["tree.nwk", *(extra or [])])
def setUp(self) -> None:
super().setUp()
self.tree = quick_visualize.load_tree(self.newick(BALANCED), 1)
def test_smartview_refuses_a_non_png_destination(self) -> None:
with self.assertRaisesRegex(quick_visualize.UserInputError, "PNG"):
quick_visualize.render_smartview(
self.tree, None, self.root / "out.pdf", self.args()
)
def test_smartview_refuses_a_missing_output_directory(self) -> None:
with self.assertRaisesRegex(quick_visualize.UserInputError, "output directory"):
quick_visualize.render_smartview(
self.tree, None, self.root / "nope" / "out.png", self.args()
)
def test_treeview_refuses_a_suffix_it_cannot_write(self) -> None:
with self.assertRaisesRegex(quick_visualize.UserInputError, r"\.png, \.pdf, or \.svg"):
quick_visualize.render_treeview(self.tree, self.root / "out.txt", self.args())
def test_treeview_refuses_a_missing_output_directory(self) -> None:
with self.assertRaisesRegex(quick_visualize.UserInputError, "output directory"):
quick_visualize.render_treeview(
self.tree, self.root / "nope" / "out.pdf", self.args()
)
def test_the_missing_qt_extra_is_reported_as_an_install_hint(self) -> None:
try:
import ete4.treeview # noqa: F401
except ImportError:
pass
else:
self.skipTest("ete4[treeview] is installed, so the hint cannot be triggered")
with self.assertRaisesRegex(quick_visualize.UserInputError, r"ete4\[treeview\]"):
quick_visualize.create_treeview_style(self.tree, self.args())
class SmartViewLayoutTests(TreeFixtureCase):
"""The CLI's display options have to reach the SmartView style dictionary."""
def args(self, extra: list[str] | None = None) -> argparse.Namespace:
return quick_visualize.build_parser().parse_args(["tree.nwk", *(extra or [])])
def elements(self, args: argparse.Namespace) -> list:
return list(quick_visualize.create_smartview_layout(args).draw_tree(None))
def style(self, args: argparse.Namespace) -> dict:
# ete4's Layout prepends its own DEFAULT_TREE_STYLE, so the style the
# script yields is the second element.
return self.elements(args)[1]
def test_the_layout_mode_becomes_the_smartview_shape(self) -> None:
self.assertEqual(self.style(self.args())["shape"], "rectangular")
self.assertEqual(self.style(self.args(["--mode", "c"]))["shape"], "circular")
def test_arc_options_are_only_emitted_for_the_circular_layout(self) -> None:
# A rectangular tree has no arc; passing angle keys would be ignored at
# best and misread at worst.
rectangular = self.style(self.args())
self.assertNotIn("angle-start", rectangular)
circular = self.style(self.args(["--mode", "c", "--arc-span", "180"]))
self.assertEqual(circular["angle-span"], 180)
self.assertEqual(circular["angle-start"], 0)
def test_collapse_thresholds_are_passed_through_unchanged(self) -> None:
style = self.style(self.args(["--collapse-pixels", "20", "--content-pixels", "6"]))
self.assertEqual(style["node-height-min"], 20)
self.assertEqual(style["content-height-min"], 6)
def dots(self, args: argparse.Namespace, newick: str, parser: int = 0) -> dict:
"""{leaf-or-support key: dot spec} for every node of `newick`."""
layout = quick_visualize.create_smartview_layout(args)
tree = quick_visualize.load_tree(self.newick(newick), parser)
found = {}
for node in tree.traverse():
elements = [
element
for element in layout.draw_node(node, ())
if isinstance(element, dict) and "dot" in element
]
self.assertEqual(len(elements), 1, "every node gets exactly one dot")
found[node.name if node.is_leaf else node.support] = elements[0]["dot"]
return found
def test_leaves_and_internal_nodes_get_their_own_colour_and_size(self) -> None:
args = self.args()
dots = self.dots(args, "((A:1,B:1)80:1,(C:1,D:1)80:1);")
self.assertEqual(dots["A"]["fill"], args.leaf_color)
self.assertEqual(dots["A"]["radius"], args.leaf_size)
self.assertEqual(dots[80.0]["fill"], args.internal_color)
self.assertEqual(dots[80.0]["radius"], args.internal_size)
def test_colour_by_support_repaints_internal_nodes_only(self) -> None:
# Leaves have no bootstrap support, so they must keep the leaf colour
# even when --color-by-support is on.
args = self.args(["--color-by-support"])
dots = self.dots(args, "((A:1,B:1)95:1,(C:1,D:1)50:1);")
self.assertEqual(dots["A"]["fill"], args.leaf_color)
self.assertEqual(dots[95.0]["fill"], args.high_support_color)
self.assertEqual(dots[50.0]["fill"], args.low_support_color)
def test_a_title_adds_a_header_face_and_no_title_adds_nothing(self) -> None:
# An empty --title must not yield a blank face that reserves header space.
self.assertEqual(len(self.elements(self.args())), 2)
self.assertEqual(len(self.elements(self.args(["--title", ""]))), 2)
self.assertEqual(len(self.elements(self.args(["--title", "Fig 1"]))), 3)
if __name__ == "__main__":
unittest.main()

13
tests/exa-search/test_exa_search.py
View File

@@ -17,10 +17,18 @@ import os
import sys
import tempfile
import unittest
import pytest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
import skill_contract
# Guarded so a bare project-environment run skips cleanly instead of failing;
# the real run is `tests/run_all.py --isolated exa-search`.
pytest.importorskip("exa_py", reason="exa-search needs exa-py")
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "exa-search"
SCRIPTS_DIR = SKILL_ROOT / "scripts"
@@ -213,5 +221,10 @@ class IntegrationHeaderAndFlowTests(unittest.TestCase):
self.assertEqual(payload["results"][0]["title"], "Attention Is All You Need")
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
if __name__ == "__main__":
unittest.main()

322
tests/experimental-design/test_scripts.py Normal file
View File

@@ -0,0 +1,322 @@
"""Tests for the experimental-design generators.
Randomisation and DOE code is easy to get subtly wrong in ways that no error
message reveals -- a block that does not actually balance, an allocation ratio
applied to the wrong arm, a design matrix decoded to the wrong real units. So
the assertions here are about the statistical properties the docstrings
promise, not just about shapes: every permuted block is exactly balanced, a
2:1 ratio produces twice as many treatment units, and a two-level design
decodes to the factor's own low/high values.
Determinism matters as much as correctness: every generator takes a seed, and
an experimental design that cannot be reproduced from its seed is not a design.
"""
from __future__ import annotations
import sys
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "experimental-design"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
np = pytest.importorskip("numpy", reason="experimental-design needs numpy")
pd = pytest.importorskip("pandas", reason="experimental-design needs pandas")
import randomization # noqa: E402
# Both scripts are importable libraries with a worked example under
# `if __name__ == "__main__":` rather than argparse CLIs, so the contract's
# demo-block case applies instead of its `--help` case.
DemoBlockTests = skill_contract.cli.demo_test_case(
SKILL_ROOT, ("doe_designs.py", "randomization.py")
)
def doe():
"""Import the DOE module, skipping when pyDOE3 is absent."""
pytest.importorskip("pyDOE3", reason="doe_designs needs pyDOE3")
import doe_designs
return doe_designs
class RatioTests(unittest.TestCase):
def test_no_ratio_means_one_of_each_arm(self) -> None:
self.assertEqual(
randomization._normalize_ratio(["a", "b", "c"], None), ["a", "b", "c"]
)
def test_a_ratio_repeats_each_arm(self) -> None:
self.assertEqual(
randomization._normalize_ratio(["treatment", "control"], (2, 1)),
["treatment", "treatment", "control"],
)
def test_a_mismatched_ratio_length_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "one entry per arm"):
randomization._normalize_ratio(["a", "b"], (1, 1, 1))
def test_non_positive_ratio_entries_are_refused(self) -> None:
# A zero-weight arm would silently vanish from the allocation.
for ratio in ((1, 0), (-1, 1), (0, 0)):
with self.subTest(ratio=ratio):
with self.assertRaisesRegex(ValueError, "positive integers"):
randomization._normalize_ratio(["a", "b"], ratio)
class SimpleRandomizationTests(unittest.TestCase):
def test_every_unit_is_assigned_exactly_once(self) -> None:
frame = randomization.simple_randomization(25, seed=1)
self.assertEqual(len(frame), 25)
self.assertEqual(list(frame["unit_id"]), list(range(1, 26)))
self.assertTrue(set(frame["arm"]) <= {"treatment", "control"})
def test_the_same_seed_reproduces_the_same_allocation(self) -> None:
first = randomization.simple_randomization(40, seed=7)
second = randomization.simple_randomization(40, seed=7)
pd.testing.assert_frame_equal(first, second)
def test_different_seeds_generally_differ(self) -> None:
first = randomization.simple_randomization(60, seed=1)
second = randomization.simple_randomization(60, seed=2)
self.assertFalse(first["arm"].equals(second["arm"]))
def test_an_allocation_ratio_shifts_the_expected_split(self) -> None:
# Simple randomisation only balances in expectation, so assert the
# direction over a large sample rather than an exact count.
frame = randomization.simple_randomization(
4000, arms=("treatment", "control"), ratio=(3, 1), seed=3
)
share = (frame["arm"] == "treatment").mean()
self.assertAlmostEqual(share, 0.75, delta=0.03)
def test_three_arms_are_supported(self) -> None:
frame = randomization.simple_randomization(
300, arms=("a", "b", "c"), seed=5
)
self.assertEqual(set(frame["arm"]), {"a", "b", "c"})
class BlockRandomizationTests(unittest.TestCase):
def test_each_complete_block_is_exactly_balanced(self) -> None:
# This is the entire point of permuted blocks: balance holds throughout
# enrollment, not only at the end.
frame = randomization.block_randomization(24, block_size=4, seed=1)
for block, rows in frame.groupby("block"):
with self.subTest(block=block):
counts = rows["arm"].value_counts().to_dict()
self.assertEqual(counts, {"treatment": 2, "control": 2})
def test_the_allocation_ratio_holds_within_every_block(self) -> None:
frame = randomization.block_randomization(
18, arms=["treatment", "control"], ratio=(2, 1), block_size=6, seed=1
)
for block, rows in frame.groupby("block"):
with self.subTest(block=block):
counts = rows["arm"].value_counts().to_dict()
self.assertEqual(counts, {"treatment": 4, "control": 2})
def test_a_block_size_incompatible_with_the_ratio_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "must be a multiple of"):
randomization.block_randomization(12, ratio=(2, 1), block_size=4)
def test_the_default_block_size_is_valid_for_the_ratio(self) -> None:
frame = randomization.block_randomization(12, ratio=(2, 1), seed=1)
sizes = frame.groupby("block").size().unique()
self.assertEqual(list(sizes), [6])
def test_a_partial_final_block_is_truncated_not_padded(self) -> None:
frame = randomization.block_randomization(10, block_size=4, seed=1)
self.assertEqual(len(frame), 10)
self.assertEqual(list(frame["unit_id"]), list(range(1, 11)))
def test_the_same_seed_reproduces_the_same_blocks(self) -> None:
pd.testing.assert_frame_equal(
randomization.block_randomization(20, seed=11),
randomization.block_randomization(20, seed=11),
)
class StratifiedTests(unittest.TestCase):
def test_every_stratum_is_balanced_independently(self) -> None:
frame = randomization.stratified_block_randomization(
{"siteA": 8, "siteB": 12}, block_size=4, seed=1
)
self.assertEqual(len(frame), 20)
balance = randomization.arm_balance(frame, by="stratum")
self.assertEqual(balance.loc["siteA", "treatment"], 4)
self.assertEqual(balance.loc["siteA", "control"], 4)
self.assertEqual(balance.loc["siteB", "treatment"], 6)
self.assertEqual(balance.loc["siteB", "control"], 6)
def test_unit_ids_are_renumbered_across_the_whole_cohort(self) -> None:
frame = randomization.stratified_block_randomization(
{"a": 4, "b": 4}, block_size=4, seed=1
)
self.assertEqual(list(frame["unit_id"]), list(range(1, 9)))
def test_a_label_sequence_is_accepted_as_well_as_a_count_map(self) -> None:
labels = ["north"] * 4 + ["south"] * 8
frame = randomization.stratified_block_randomization(
labels, block_size=4, seed=1
)
self.assertEqual(len(frame), 12)
self.assertEqual(
frame["stratum"].value_counts().to_dict(), {"south": 8, "north": 4}
)
def test_strata_are_seeded_apart_so_they_do_not_share_a_pattern(self) -> None:
# Every stratum getting the same permutation would defeat stratification.
frame = randomization.stratified_block_randomization(
{"a": 8, "b": 8}, block_size=4, seed=1
)
first = list(frame[frame["stratum"] == "a"]["arm"])
second = list(frame[frame["stratum"] == "b"]["arm"])
self.assertNotEqual(first, second)
class ClusterTests(unittest.TestCase):
def test_one_row_per_cluster_keyed_by_cluster_id(self) -> None:
frame = randomization.cluster_randomization(
["clinic-1", "clinic-2", "clinic-3", "clinic-4"], seed=1
)
self.assertEqual(len(frame), 4)
self.assertEqual(
list(frame["cluster_id"]),
["clinic-1", "clinic-2", "clinic-3", "clinic-4"],
)
self.assertNotIn("unit_id", frame.columns)
def test_an_integer_count_generates_cluster_labels(self) -> None:
frame = randomization.cluster_randomization(3, seed=1)
self.assertEqual(
list(frame["cluster_id"]), ["cluster_1", "cluster_2", "cluster_3"]
)
def test_clusters_are_blocked_so_arms_stay_balanced(self) -> None:
frame = randomization.cluster_randomization(8, block_size=4, seed=1)
counts = frame["arm"].value_counts().to_dict()
self.assertEqual(counts, {"treatment": 4, "control": 4})
class RunOrderTests(unittest.TestCase):
def test_run_order_is_a_permutation_of_every_row(self) -> None:
design = pd.DataFrame({"temp": [20, 20, 60, 60], "ph": [6, 8, 6, 8]})
randomized = randomization.assign_factorial_runs(design, seed=1)
self.assertEqual(sorted(randomized["run_order"]), [1, 2, 3, 4])
self.assertEqual(len(randomized), len(design))
def test_rows_are_returned_sorted_by_run_order(self) -> None:
design = pd.DataFrame({"x": range(10)})
randomized = randomization.assign_factorial_runs(design, seed=2)
self.assertEqual(list(randomized["run_order"]), list(range(1, 11)))
def test_the_original_design_is_not_mutated(self) -> None:
design = pd.DataFrame({"x": range(5)})
before = design.copy()
randomization.assign_factorial_runs(design, seed=1)
pd.testing.assert_frame_equal(design, before)
def test_randomising_actually_reorders_the_runs(self) -> None:
# A systematic order confounds factors with drift, which is the whole
# reason this function exists.
design = pd.DataFrame({"x": range(12)})
randomized = randomization.assign_factorial_runs(design, seed=1)
self.assertNotEqual(list(randomized["x"]), list(range(12)))
class BalanceReportTests(unittest.TestCase):
def test_counts_are_reported_per_arm(self) -> None:
frame = pd.DataFrame({"arm": ["a", "a", "b"]})
self.assertEqual(randomization.arm_balance(frame).to_dict(), {"a": 2, "b": 1})
def test_grouping_produces_a_stratum_by_arm_table(self) -> None:
frame = pd.DataFrame(
{"arm": ["a", "b", "a", "a"], "stratum": ["x", "x", "y", "y"]}
)
table = randomization.arm_balance(frame, by="stratum")
self.assertEqual(table.loc["x", "a"], 1)
self.assertEqual(table.loc["y", "a"], 2)
# unstack(fill_value=0) -- an absent combination is 0, not NaN.
self.assertEqual(table.loc["y", "b"], 0)
class DesignMatrixTests(unittest.TestCase):
def test_full_factorial_covers_every_combination(self) -> None:
design = doe().full_factorial(
{"temp": [20, 40, 60], "catalyst": ["A", "B"]}, randomize=False
)
self.assertEqual(len(design), 6)
combinations = set(zip(design["temp"], design["catalyst"]))
self.assertEqual(len(combinations), 6)
def test_two_level_factorial_decodes_to_the_stated_low_and_high(self) -> None:
design = doe().two_level_factorial(
{"temp": (20, 60), "ph": (6.0, 8.0)}, randomize=False
)
self.assertEqual(len(design), 4)
self.assertEqual(set(design["temp"]), {20.0, 60.0})
self.assertEqual(set(design["ph"]), {6.0, 8.0})
def test_a_generator_that_names_the_wrong_factor_count_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "generator defines"):
doe().fractional_factorial({"a": (0, 1), "b": (0, 1)}, "a b c")
def test_a_valid_fraction_halves_the_full_factorial(self) -> None:
factors = {f: (0, 1) for f in ("a", "b", "c", "d")}
design = doe().fractional_factorial(factors, "a b c abc", randomize=False)
self.assertEqual(len(design), 8) # 2^(4-1), not 2^4
self.assertEqual(list(design.columns), list(factors))
def test_plackett_burman_drops_the_dummy_columns(self) -> None:
factors = {f: (0, 1) for f in ("a", "b", "c", "d", "e")}
design = doe().plackett_burman(factors, randomize=False)
self.assertEqual(list(design.columns), list(factors))
def test_box_behnken_needs_three_factors(self) -> None:
with self.assertRaisesRegex(ValueError, "at least 3 factors"):
doe().box_behnken({"a": (0, 1), "b": (0, 1)})
def test_box_behnken_avoids_the_extreme_corners(self) -> None:
# Never using the all-low / all-high corner is the design's reason for
# existing -- those runs are the unsafe or infeasible ones.
factors = {"a": (0, 10), "b": (0, 10), "c": (0, 10)}
design = doe().box_behnken(factors, randomize=False)
corners = {(0.0, 0.0, 0.0), (10.0, 10.0, 10.0)}
rows = set(zip(design["a"], design["b"], design["c"]))
self.assertFalse(rows & corners)
def test_latin_hypercube_stays_inside_every_factor_range(self) -> None:
design = doe().latin_hypercube({"x": (2.0, 5.0), "y": (-1.0, 1.0)}, 20, seed=3)
self.assertEqual(len(design), 20)
self.assertTrue((design["x"] >= 2.0).all() and (design["x"] <= 5.0).all())
self.assertTrue((design["y"] >= -1.0).all() and (design["y"] <= 1.0).all())
def test_latin_hypercube_is_reproducible_from_its_seed(self) -> None:
factors = {"x": (0.0, 1.0), "y": (0.0, 1.0)}
pd.testing.assert_frame_equal(
doe().latin_hypercube(factors, 15, seed=9),
doe().latin_hypercube(factors, 15, seed=9),
)
def test_randomize_adds_a_run_order_column_and_sorts_by_it(self) -> None:
factors = {"a": (0, 1), "b": (0, 1)}
design = doe().two_level_factorial(factors, randomize=True, seed=4)
self.assertEqual(design.columns[0], "run_order")
self.assertEqual(list(design["run_order"]), [1, 2, 3, 4])
def test_not_randomizing_leaves_the_design_in_canonical_order(self) -> None:
factors = {"a": (0, 1), "b": (0, 1)}
design = doe().two_level_factorial(factors, randomize=False)
self.assertNotIn("run_order", design.columns)
if __name__ == "__main__":
unittest.main()

2
tests/exploratory-data-analysis/test_static.py
View File

@@ -86,7 +86,7 @@ class StaticSafetyTests(unittest.TestCase):
skill = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertIn("license: MIT", skill)
self.assertIn("compatibility:", skill)
self.assertIn('metadata:\n version: "1.1"', skill)
self.assertRegex(skill, r'\nmetadata:\n version: "\d+\.\d+"\n')
self.assertNotIn("200+", skill)
self.assertLess(len(skill.splitlines()), 500)

458
tests/flowio/test_scripts.py Normal file
View File

@@ -0,0 +1,458 @@
"""Tests for the FlowIO inspection helper.
`inspect_fcs` exists to read an untrusted FCS file without letting it decide how
much memory to allocate, so the guards are the product: the byte-size ceiling,
the estimated-array ceiling that is enforced *before* DATA is loaded, and the
multi-dataset offset walk that refuses a negative, non-increasing, or
out-of-file `$NEXTDATA` chain. Those tests build the pathological offsets by
stubbing `FlowData`, because a well-formed FCS file cannot express them.
The rest of the suite pins the reported numbers against a file this test wrote:
channel classification (a scatter channel silently reported as fluorescence
misleads every downstream gate), and the per-channel statistics, which must
ignore NaN and +/-inf rather than propagate them -- the values below are
hand-computed from the events written, not read back from the code.
"""
from __future__ import annotations
import argparse
import hashlib
import io
import json
import sys
import tempfile
import unittest
from contextlib import redirect_stderr, redirect_stdout
from pathlib import Path
from unittest.mock import patch
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "flowio"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
flowio = pytest.importorskip("flowio", reason="flowio skill needs flowio")
np = pytest.importorskip("numpy", reason="inspect_fcs computes statistics with numpy")
import inspect_fcs # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
NAN = float("nan")
INF = float("inf")
def write_fcs(
path: Path,
events: list[float],
channels: list[str],
optional: list[str] | None = None,
) -> Path:
"""Write a real FCS 3.1 file; `events` is the flattened row-major matrix."""
with path.open("wb") as handle:
flowio.create_fcs(
handle, events, channel_names=channels, opt_channel_names=optional
)
return path
def namespace(**overrides) -> argparse.Namespace:
"""The parsed-argument defaults `inspect_file` reads, overridable per test."""
defaults = dict(
input=Path("unused.fcs"),
output=None,
include_text=False,
include_analysis=False,
stats=False,
raw=False,
sha256=False,
max_bytes=inspect_fcs.DEFAULT_MAX_BYTES,
max_array_bytes=inspect_fcs.DEFAULT_MAX_ARRAY_BYTES,
max_datasets=inspect_fcs.DEFAULT_MAX_DATASETS,
null_channel=[],
ignore_offset_error=False,
ignore_offset_discrepancy=False,
use_header_offsets=False,
)
defaults.update(overrides)
return argparse.Namespace(**defaults)
class ArgumentValidatorTests(unittest.TestCase):
def test_nonnegative_int_admits_zero_because_zero_disables_a_limit(self) -> None:
# --max-bytes 0 is documented as "no ceiling", so zero must parse.
self.assertEqual(inspect_fcs.nonnegative_int("0"), 0)
self.assertEqual(inspect_fcs.nonnegative_int("512"), 512)
def test_nonnegative_int_refuses_a_negative_ceiling(self) -> None:
with self.assertRaises(argparse.ArgumentTypeError):
inspect_fcs.nonnegative_int("-1")
def test_positive_int_refuses_zero_and_below(self) -> None:
# --max-datasets 0 would make the dataset walk yield nothing at all.
self.assertEqual(inspect_fcs.positive_int("1"), 1)
for value in ("0", "-3"):
with self.subTest(value=value):
with self.assertRaises(argparse.ArgumentTypeError):
inspect_fcs.positive_int(value)
def test_non_numeric_limits_raise(self) -> None:
for validator in (inspect_fcs.nonnegative_int, inspect_fcs.positive_int):
with self.subTest(validator=validator.__name__):
with self.assertRaises(ValueError):
validator("lots")
class FcsFileTestCase(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
class ChannelClassificationTests(FcsFileTestCase):
"""`channel_kind` decides what each column *means* to a downstream gate."""
CHANNELS = ["FSC-A", "SSC-A", "FL1-A", "Time"]
def flow(self, null: list[str] | None = None):
path = write_fcs(
self.root / "kinds.fcs",
[1.0, 2.0, 3.0, 4.0],
self.CHANNELS,
optional=["fsc", "ssc", "CD3", "time"],
)
return flowio.FlowData(str(path), null_channel_list=null or [])
def test_scatter_time_and_fluorescence_are_told_apart(self) -> None:
flow = self.flow()
kinds = [inspect_fcs.channel_kind(flow, i) for i in range(len(self.CHANNELS))]
self.assertEqual(kinds, ["scatter", "scatter", "fluorescence", "time"])
def test_a_declared_null_channel_outranks_its_detected_kind(self) -> None:
# --null-channel exists to retire a channel; if the kind still said
# "fluorescence" the column would keep being analysed.
flow = self.flow(null=["FL1-A"])
self.assertEqual(inspect_fcs.channel_kind(flow, 2), "null")
# Nulling one channel must not reclassify the others.
self.assertEqual(inspect_fcs.channel_kind(flow, 0), "scatter")
self.assertEqual(inspect_fcs.channel_kind(flow, 3), "time")
def test_channel_records_number_parameters_from_one(self) -> None:
# FCS $PnN keywords are 1-based; the numpy column index is 0-based.
# Conflating them shifts every reported gain and range by one channel.
records = inspect_fcs.channel_records(self.flow())
self.assertEqual([r["array_index"] for r in records], [0, 1, 2, 3])
self.assertEqual([r["parameter_number"] for r in records], [1, 2, 3, 4])
self.assertEqual([r["pnn"] for r in records], self.CHANNELS)
self.assertEqual([r["pns"] for r in records], ["fsc", "ssc", "CD3", "time"])
def test_channel_records_carry_the_scaling_keywords_as_json_types(self) -> None:
# $PnE is a tuple in FlowIO; JSON cannot hold a tuple, so the record
# must convert it -- otherwise `write_report` fails on real files.
record = inspect_fcs.channel_records(self.flow())[0]
self.assertIsInstance(record["pne"], list)
self.assertEqual(record["pne"], [0.0, 0.0]) # linear scaling
self.assertEqual(record["png"], 1.0) # unity gain
json.dumps(record) # would raise if a tuple survived
class FiniteStatisticsTests(unittest.TestCase):
"""Every value here is computed by hand from the array below."""
EVENTS = np.array(
[
[1.0, NAN, 10.0],
[3.0, INF, 20.0],
[5.0, -INF, 30.0],
[NAN, NAN, 40.0],
]
)
LABELS = ["mixed", "unusable", "clean"]
def setUp(self) -> None:
self.records = inspect_fcs.finite_statistics(self.EVENTS, self.LABELS)
def test_one_record_is_emitted_per_channel_in_order(self) -> None:
self.assertEqual([r["pnn"] for r in self.records], self.LABELS)
self.assertEqual([r["array_index"] for r in self.records], [0, 1, 2])
def test_statistics_are_computed_over_the_finite_values_only(self) -> None:
mixed = self.records[0]
# Finite values are 1, 3, 5: mean 3, min 1, max 5, one NaN dropped.
self.assertEqual(mixed["finite_count"], 3)
self.assertEqual(mixed["nan_count"], 1)
self.assertEqual(mixed["minimum"], 1.0)
self.assertEqual(mixed["maximum"], 5.0)
self.assertEqual(mixed["mean"], 3.0)
def test_infinities_are_counted_by_sign_and_excluded_from_the_range(self) -> None:
unusable = self.records[1]
self.assertEqual(unusable["positive_infinity_count"], 1)
self.assertEqual(unusable["negative_infinity_count"], 1)
self.assertEqual(unusable["nan_count"], 2)
self.assertEqual(unusable["finite_count"], 0)
def test_a_channel_with_no_finite_value_reports_none_not_nan(self) -> None:
# json.dumps(..., allow_nan=False) in write_report rejects NaN, so the
# statistics must degrade to null rather than to a NaN float.
unusable = self.records[1]
for key in ("minimum", "maximum", "mean"):
with self.subTest(key=key):
self.assertIsNone(unusable[key])
json.dumps(self.records, allow_nan=False)
def test_a_fully_finite_channel_keeps_every_event(self) -> None:
clean = self.records[2]
self.assertEqual(clean["finite_count"], 4)
self.assertEqual(clean["nan_count"], 0)
self.assertEqual(clean["mean"], 25.0) # (10+20+30+40)/4
class FakeText(dict):
"""A `FlowData.text` stand-in exposing only what `iter_datasets` reads."""
class FakeFlow:
def __init__(self, nextdata: str) -> None:
self.text = FakeText(nextdata=nextdata)
class DatasetWalkTests(FcsFileTestCase):
"""The `$NEXTDATA` chain is attacker-controlled; the walk must bound it."""
def setUp(self) -> None:
super().setUp()
self.path = self.root / "chain.fcs"
self.path.write_bytes(b"x" * 1000)
def walk(self, nextdata_values: list[str], max_datasets: int = 8):
"""Run `iter_datasets` with FlowData replaced by a scripted stub."""
seen: list[int] = []
remaining = list(nextdata_values)
def fake_flow_data(path, **kwargs):
seen.append(kwargs["nextdata_offset"])
return FakeFlow(remaining.pop(0) if remaining else "0")
with patch.object(inspect_fcs, "FlowData", fake_flow_data):
datasets = list(
inspect_fcs.iter_datasets(
self.path,
max_datasets=max_datasets,
null_channel_list=[],
ignore_offset_error=False,
ignore_offset_discrepancy=False,
use_header_offsets=False,
)
)
return datasets, seen
def test_a_single_dataset_file_stops_after_one(self) -> None:
datasets, offsets = self.walk(["0"])
self.assertEqual(len(datasets), 1)
self.assertEqual(offsets, [0])
self.assertEqual(datasets[0][0], 0) # dataset_index
self.assertEqual(datasets[0][1], 0) # byte offset
def test_nextdata_is_relative_so_offsets_accumulate(self) -> None:
# FCS $NEXTDATA is relative to the start of the current dataset. Two
# hops of 100 must land at 100 then 200, not 100 twice.
datasets, offsets = self.walk(["100", "100", "0"])
self.assertEqual(offsets, [0, 100, 200])
self.assertEqual([index for index, _, _ in datasets], [0, 1, 2])
self.assertEqual([offset for _, offset, _ in datasets], [0, 100, 200])
def test_a_negative_relative_offset_is_refused(self) -> None:
with self.assertRaisesRegex(
flowio.exceptions.MultipleDataSetsError, "negative relative offset"
):
self.walk(["-8", "0"])
def test_an_offset_past_the_end_of_the_file_is_refused(self) -> None:
# The file is 1000 bytes; seeking to 5000 would read foreign memory.
with self.assertRaisesRegex(
flowio.exceptions.MultipleDataSetsError, "outside the input file"
):
self.walk(["5000", "0"])
def test_the_dataset_count_ceiling_is_enforced(self) -> None:
# A chain of 10-byte hops never terminates on its own; --max-datasets
# is the only thing that stops it.
with self.assertRaisesRegex(
flowio.exceptions.MultipleDataSetsError, r"--max-datasets limit \(3\)"
):
self.walk(["10"] * 20, max_datasets=3)
class InspectFileTests(FcsFileTestCase):
"""`inspect_file` is the whole report; its guards run before DATA loads."""
def setUp(self) -> None:
super().setUp()
# 3 events x 2 channels of known values.
self.path = write_fcs(
self.root / "sample.fcs",
[1.0, 10.0, 2.0, 20.0, 3.0, 30.0],
["FSC-A", "FL1-A"],
)
def test_the_report_describes_the_file_that_was_written(self) -> None:
report = inspect_fcs.inspect_file(namespace(input=self.path))
self.assertEqual(report["dataset_count"], 1)
self.assertEqual(report["file_name"], "sample.fcs")
self.assertEqual(report["size_bytes"], self.path.stat().st_size)
dataset = report["datasets"][0]
self.assertEqual(dataset["event_count"], 3)
self.assertEqual(dataset["channel_count"], 2)
self.assertEqual(dataset["fcs_version"], "3.1")
def test_the_array_estimate_is_events_times_channels_times_eight(self) -> None:
# float64 is 8 bytes; this estimate is what --max-array-bytes compares
# against, so an understated one defeats the ceiling.
report = inspect_fcs.inspect_file(namespace(input=self.path))
self.assertEqual(report["datasets"][0]["estimated_array_bytes"], 3 * 2 * 8)
def test_metadata_only_is_the_default_and_says_so(self) -> None:
report = inspect_fcs.inspect_file(namespace(input=self.path))
self.assertTrue(report["parse_options"]["metadata_only"])
dataset = report["datasets"][0]
self.assertNotIn("statistics", dataset)
self.assertEqual(dataset["event_semantics"], "not loaded; metadata-only inspection")
def test_stats_loads_events_and_reports_the_array_shape(self) -> None:
report = inspect_fcs.inspect_file(namespace(input=self.path, stats=True))
dataset = report["datasets"][0]
self.assertEqual(dataset["array_shape"], [3, 2])
self.assertFalse(report["parse_options"]["metadata_only"])
first, second = dataset["statistics"]
self.assertEqual((first["minimum"], first["maximum"]), (1.0, 3.0))
self.assertEqual((second["minimum"], second["maximum"]), (10.0, 30.0))
def test_raw_stats_are_labelled_as_encoded_values(self) -> None:
# The semantics string is how a reader knows whether gain/log scaling
# was applied; mislabelling it invalidates every downstream comparison.
report = inspect_fcs.inspect_file(namespace(input=self.path, stats=True, raw=True))
self.assertIn("preprocess=False", report["datasets"][0]["event_semantics"])
scaled = inspect_fcs.inspect_file(namespace(input=self.path, stats=True))
self.assertIn("uncompensated", scaled["datasets"][0]["event_semantics"])
def test_text_and_analysis_are_withheld_unless_requested(self) -> None:
# TEXT can carry patient identifiers, so it is opt-in.
default = inspect_fcs.inspect_file(namespace(input=self.path))
self.assertNotIn("text", default["datasets"][0])
self.assertGreater(default["datasets"][0]["metadata_summary"]["text_key_count"], 0)
# FlowIO normalises TEXT keywords to lowercase without the `$`.
included = inspect_fcs.inspect_file(namespace(input=self.path, include_text=True))
self.assertEqual(included["datasets"][0]["text"]["p1n"], "FSC-A")
def test_a_file_over_the_byte_ceiling_is_refused_before_parsing(self) -> None:
with self.assertRaisesRegex(ValueError, "--max-bytes"):
inspect_fcs.inspect_file(namespace(input=self.path, max_bytes=10))
def test_a_zero_byte_ceiling_disables_the_check(self) -> None:
report = inspect_fcs.inspect_file(namespace(input=self.path, max_bytes=0))
self.assertEqual(report["parse_options"]["max_bytes"], 0)
def test_the_array_ceiling_only_applies_with_stats(self) -> None:
# Metadata-only inspection allocates no array, so a tiny ceiling must
# not block it -- and must block --stats.
tiny = namespace(input=self.path, max_array_bytes=8)
inspect_fcs.inspect_file(tiny)
with self.assertRaisesRegex(ValueError, "--max-array-bytes"):
inspect_fcs.inspect_file(namespace(input=self.path, stats=True, max_array_bytes=8))
def test_a_ceiling_equal_to_the_estimate_is_accepted(self) -> None:
# The comparison is strictly-greater, so the boundary must pass.
report = inspect_fcs.inspect_file(
namespace(input=self.path, stats=True, max_array_bytes=48)
)
self.assertEqual(report["datasets"][0]["array_shape"], [3, 2])
def test_a_missing_input_raises_file_not_found(self) -> None:
with self.assertRaises(FileNotFoundError):
inspect_fcs.inspect_file(namespace(input=self.root / "absent.fcs"))
def test_a_directory_is_not_a_regular_file(self) -> None:
with self.assertRaisesRegex(ValueError, "not a regular file"):
inspect_fcs.inspect_file(namespace(input=self.root))
def test_the_checksum_is_opt_in_and_matches_the_streaming_helper(self) -> None:
without = inspect_fcs.inspect_file(namespace(input=self.path))
self.assertNotIn("sha256", without)
with_sum = inspect_fcs.inspect_file(namespace(input=self.path, sha256=True))
self.assertEqual(with_sum["sha256"], inspect_fcs.sha256_file(self.path))
def test_the_checksum_matches_hashlib_on_the_same_bytes(self) -> None:
expected = hashlib.sha256(self.path.read_bytes()).hexdigest()
self.assertEqual(inspect_fcs.sha256_file(self.path), expected)
class ReportWritingTests(FcsFileTestCase):
def test_stdout_receives_parseable_json(self) -> None:
buffer = io.StringIO()
with redirect_stdout(buffer):
inspect_fcs.write_report({"schema_version": "1.0"}, None)
self.assertEqual(json.loads(buffer.getvalue()), {"schema_version": "1.0"})
def test_an_existing_output_file_is_never_overwritten(self) -> None:
output = self.root / "report.json"
output.write_text("keep me", encoding="utf-8")
with self.assertRaises(FileExistsError):
inspect_fcs.write_report({"a": 1}, output)
self.assertEqual(output.read_text(encoding="utf-8"), "keep me")
def test_a_non_finite_number_is_refused_rather_than_written_as_nan(self) -> None:
# Bare NaN is not valid JSON; writing it would produce a file that
# every conforming parser rejects.
with self.assertRaises(ValueError):
inspect_fcs.write_report({"mean": NAN}, self.root / "nan.json")
class MainTests(FcsFileTestCase):
def setUp(self) -> None:
super().setUp()
self.path = write_fcs(self.root / "main.fcs", [1.0, 2.0], ["FSC-A", "FL1-A"])
def test_a_clean_run_exits_zero_and_writes_the_requested_file(self) -> None:
output = self.root / "out.json"
self.assertEqual(inspect_fcs.main([str(self.path), "--output", str(output)]), 0)
report = json.loads(output.read_text(encoding="utf-8"))
self.assertEqual(report["datasets"][0]["event_count"], 1)
def test_raw_without_stats_is_rejected_by_the_parser(self) -> None:
# --raw only changes how events are decoded, so it is meaningless
# without --stats; accepting it would silently do nothing.
with self.assertRaises(SystemExit) as raised:
inspect_fcs.main([str(self.path), "--raw"])
self.assertEqual(raised.exception.code, 2)
def test_writing_the_report_over_the_input_fcs_is_rejected(self) -> None:
with self.assertRaises(SystemExit) as raised:
inspect_fcs.main([str(self.path), "--output", str(self.path)])
self.assertEqual(raised.exception.code, 2)
# The input must survive intact.
self.assertGreater(self.path.stat().st_size, 0)
def test_a_missing_input_exits_two_with_a_message_not_a_traceback(self) -> None:
errors = io.StringIO()
with redirect_stderr(errors):
status = inspect_fcs.main([str(self.root / "absent.fcs")])
self.assertEqual(status, 2)
self.assertIn("inspect_fcs:", errors.getvalue())
self.assertNotIn("Traceback", errors.getvalue())
def test_a_file_that_is_not_fcs_at_all_exits_two(self) -> None:
junk = self.root / "notes.txt"
junk.write_text("this is not an FCS file", encoding="utf-8")
with redirect_stderr(io.StringIO()):
self.assertEqual(inspect_fcs.main([str(junk)]), 2)
if __name__ == "__main__":
unittest.main()

2
tests/fluidsim/test_static.py
View File

@@ -20,7 +20,7 @@ class SkillStructureTests(unittest.TestCase):
self.assertIn("\nlicense: MIT\n", text)
self.assertRegex(
text,
r"\nmetadata:\n version: \"1\.1\"\n skill-author:",
r"\nmetadata:\n version: \"\d+\.\d+\"\n skill-author:",
)
self.assertNotIn('metadata: {"version"', text)
self.assertIn('last-reviewed: "2026-07-23"', text)

2
tests/generate-image/test_scripts.py
View File

@@ -229,7 +229,7 @@ class SaveImageTests(unittest.TestCase):
class SkillDocumentTests(unittest.TestCase):
def test_frontmatter_version_matches_expected(self):
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertIn('version: "2.0"', text)
self.assertRegex(text, r'\n version: "\d+\.\d+"\n')
def test_documented_default_model_matches_the_script(self):
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")

9
tests/genomic-coordinates/test_scripts.py
View File

@@ -19,6 +19,8 @@ import unittest
from contextlib import redirect_stdout, redirect_stderr
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "genomic-coordinates"
SCRIPTS_DIR = SKILL_ROOT / "scripts"
FIXTURES = Path(__file__).resolve().parent / "fixtures"
@@ -607,7 +609,7 @@ class SkillStructureTests(unittest.TestCase):
def test_version_is_a_quoted_string(self):
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertIn('version: "1.0"', text)
self.assertRegex(text, r'\n version: "\d+\.\d+"\n')
def test_skill_md_is_within_the_line_budget(self):
lines = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8").splitlines()
@@ -650,5 +652,10 @@ class SkillStructureTests(unittest.TestCase):
self.assertNotIn(module.split(".")[0], third_party)
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
if __name__ == "__main__":
unittest.main()

2
tests/geopandas/test_static.py
View File

@@ -30,7 +30,7 @@ class SkillStructureTests(unittest.TestCase):
self.assertIn("\nallowed-tools: Read Write Bash Glob Grep\n", text)
self.assertRegex(
text,
r'\nmetadata:\n version: "1\.1"\n skill-author:'
r'\nmetadata:\n version: "\d+\.\d+"\n skill-author:'
r'.*\n last-reviewed: "2026-07-23"',
)
self.assertNotIn('metadata: {"version"', text)

7
tests/get-available-resources/test_scripts.py
View File

@@ -25,6 +25,8 @@ import detect_resources # noqa: E402
import plan_workload # noqa: E402
import snapshot_tools # noqa: E402
import skill_contract
CASES = json.loads(FIXTURES.read_text(encoding="utf-8"))
GIB = 1024**3
@@ -510,5 +512,10 @@ class PlannerTests(unittest.TestCase):
self.assertIn("stress tests or large allocations", plan["prohibited_actions"])
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
if __name__ == "__main__":
unittest.main()

555
tests/gget/test_scripts.py Normal file
View File

@@ -0,0 +1,555 @@
"""Tests for the gget workflow scripts.
Every gget module in these scripts is a network call, so all three suites patch
the gget function with `autospec=True` and assert on the call that *would* have
gone out. `autospec` is load-bearing rather than decoration: it binds each
recorded call against the installed gget signature, so a script that passes an
argument gget no longer accepts fails here instead of failing in the field --
which is how `f.write(gget.muscle(...))` (muscle returns None) and
`f.write(gget.seq(...))` (seq returns a list) were caught.
The rest is the logic the scripts own: FASTA and gene-list parsing, the
per-database enrichment sweep, the deliberate five-gene cap on expression
lookups, and the resilience contract -- one service failing must not abort the
run, but an empty gene search must stop it before any further request is made.
"""
from __future__ import annotations
import io
import os
import sys
import tempfile
import unittest
from contextlib import redirect_stdout
from pathlib import Path
from unittest.mock import patch
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "gget"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pd = pytest.importorskip("pandas", reason="gget scripts need pandas")
gget = pytest.importorskip("gget", reason="gget skill needs gget")
import batch_sequence_analysis as batch # noqa: E402
import enrichment_pipeline as pipeline # noqa: E402
import gene_analysis # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
#: Every gget module these scripts reach for. All of them are network calls.
GGET_MODULES = (
"blast", "muscle", "enrichr", "archs4", "search", "info", "seq",
"opentargets", "alphafold",
)
class WorkingDirectoryTestCase(unittest.TestCase):
"""The scripts write output relative to the working directory."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
previous = Path.cwd()
os.chdir(self.root)
self.addCleanup(os.chdir, previous)
# Every gget module is replaced up front, with `autospec` so each
# recorded call is bound against the installed gget signature. The
# default side effect is a blanket guard: a module a test forgot to
# arrange raises instead of quietly querying Ensembl, NCBI, Enrichr or
# ARCHS4 for real.
self._modules = {}
for name in GGET_MODULES:
patcher = patch.object(gget, name, autospec=True)
mock = patcher.start()
self.addCleanup(patcher.stop)
mock.side_effect = AssertionError(
f"gget.{name} was called without being stubbed"
)
self._modules[name] = mock
def stub(self, name: str, **kwargs):
"""Arrange the already-installed mock for one gget module."""
mock = self._modules[name]
# Clears the blanket guard installed in setUp. Call history needs no
# reset: every test gets its own patchers.
mock.side_effect = kwargs.pop("side_effect", None)
if "return_value" in kwargs:
mock.return_value = kwargs.pop("return_value")
self.assertEqual(kwargs, {}, "unsupported stub arguments")
return mock
def quietly(self, function, *args, **kwargs):
"""Run a chatty workflow function, swallowing its progress output."""
with redirect_stdout(io.StringIO()):
return function(*args, **kwargs)
def blast_frame(description: str = "hypothetical protein") -> pd.DataFrame:
"""A BLAST result shaped like gget.blast returns."""
return pd.DataFrame(
[{"Description": description, "Max Score": 300, "Query Coverage": "98%"}]
)
class FastaReadingTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def read(self, text: str):
path = self.root / "sequences.fasta"
path.write_text(text, encoding="utf-8")
return batch.read_fasta(path)
def test_wrapped_sequence_lines_are_joined(self) -> None:
# FASTA wraps at 60-80 columns; keeping the newlines would corrupt
# every downstream BLAST query.
records = self.read(">seq1\nMKVL\nAAPG\n")
self.assertEqual(records, [{"id": "seq1", "seq": "MKVLAAPG"}])
def test_the_final_record_is_not_dropped(self) -> None:
# The loop flushes on the next '>', so the last record needs the
# explicit flush after the file ends.
records = self.read(">a\nMK\n>b\nVL\n>c\nAA\n")
self.assertEqual([r["id"] for r in records], ["a", "b", "c"])
self.assertEqual(records[-1]["seq"], "AA")
def test_the_identifier_excludes_the_angle_bracket(self) -> None:
records = self.read(">sp|P12345| my protein\nMK\n")
self.assertEqual(records[0]["id"], "sp|P12345| my protein")
def test_an_empty_file_yields_no_records(self) -> None:
self.assertEqual(self.read(""), [])
def test_a_header_with_no_sequence_is_still_a_record(self) -> None:
# Dropping it would silently renumber every later sequence.
records = self.read(">empty\n>next\nMK\n")
self.assertEqual(records[0], {"id": "empty", "seq": ""})
self.assertEqual(len(records), 2)
def test_a_missing_trailing_newline_is_tolerated(self) -> None:
self.assertEqual(self.read(">a\nMK")[0]["seq"], "MK")
class BatchSequenceAnalysisTests(WorkingDirectoryTestCase):
def fasta(self, records: int = 2) -> Path:
path = self.root / "input.fasta"
path.write_text(
"".join(f">seq{i}\nMKVL{i}\n" for i in range(records)), encoding="utf-8"
)
return path
def test_each_sequence_is_blasted_once_with_the_documented_parameters(self) -> None:
blast = self.stub("blast", return_value=blast_frame())
self.stub("muscle")
self.quietly(
batch.analyze_sequences,
self.fasta(2),
blast_db="swissprot",
output_dir=str(self.root / "out"),
)
self.assertEqual(blast.call_count, 2)
sequences = [call.args[0] for call in blast.call_args_list]
self.assertEqual(sequences, ["MKVL0", "MKVL1"])
for call in blast.call_args_list:
# save=False keeps gget from writing files of its own next to the
# script; limit=10 is what the reported "top hit" assumes.
self.assertEqual(call.kwargs["database"], "swissprot")
self.assertEqual(call.kwargs["limit"], 10)
self.assertIs(call.kwargs["save"], False)
def test_one_csv_is_written_per_sequence_named_after_it(self) -> None:
self.stub("blast", return_value=blast_frame("kinase domain"))
self.stub("muscle")
output = self.root / "out"
self.quietly(batch.analyze_sequences, self.fasta(2), output_dir=str(output))
written = sorted(path.name for path in output.glob("*_blast.csv"))
self.assertEqual(written, ["seq0_blast.csv", "seq1_blast.csv"])
table = pd.read_csv(output / "seq0_blast.csv")
self.assertEqual(table.loc[0, "Description"], "kinase domain")
def test_a_failed_blast_does_not_abort_the_remaining_sequences(self) -> None:
# A batch run that dies on sequence 1 of 50 is useless.
blast = self.stub(
"blast",
side_effect=[RuntimeError("NCBI timed out"), blast_frame()],
)
self.stub("muscle")
output = self.root / "out"
self.quietly(batch.analyze_sequences, self.fasta(2), output_dir=str(output))
self.assertEqual(blast.call_count, 2)
self.assertFalse((output / "seq0_blast.csv").exists())
self.assertTrue((output / "seq1_blast.csv").exists())
def test_alignment_asks_gget_to_write_the_file(self) -> None:
# gget.muscle returns None, so the alignment must be requested via
# `out=`; writing the return value would raise TypeError.
self.stub("blast", return_value=blast_frame())
muscle = self.stub("muscle")
output = self.root / "out"
self.quietly(batch.analyze_sequences, self.fasta(2), output_dir=str(output))
muscle.assert_called_once()
self.assertEqual(
muscle.call_args.kwargs["out"], str(output / "alignment.afa")
)
def test_a_single_sequence_is_not_aligned(self) -> None:
# An alignment of one sequence is meaningless and wastes a request.
self.stub("blast", return_value=blast_frame())
muscle = self.stub("muscle")
self.quietly(
batch.analyze_sequences, self.fasta(1), output_dir=str(self.root / "out")
)
muscle.assert_not_called()
def test_alignment_can_be_switched_off(self) -> None:
self.stub("blast", return_value=blast_frame())
muscle = self.stub("muscle")
self.quietly(
batch.analyze_sequences,
self.fasta(3),
align=False,
output_dir=str(self.root / "out"),
)
muscle.assert_not_called()
def test_structure_prediction_stays_a_no_op_even_when_requested(self) -> None:
# The AlphaFold call is commented out on purpose (it needs `gget setup
# alphafold` and hours of compute); the flag must not silently start it.
self.stub("blast", return_value=blast_frame())
self.stub("muscle")
alphafold = self.stub("alphafold")
self.quietly(
batch.analyze_sequences,
self.fasta(2),
predict_structure=True,
output_dir=str(self.root / "out"),
)
alphafold.assert_not_called()
def test_an_empty_fasta_file_runs_no_queries(self) -> None:
(self.root / "empty.fasta").write_text("", encoding="utf-8")
blast = self.stub("blast")
muscle = self.stub("muscle")
self.quietly(
batch.analyze_sequences,
self.root / "empty.fasta",
output_dir=str(self.root / "out"),
)
blast.assert_not_called()
muscle.assert_not_called()
class GeneListReadingTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def write(self, name: str, text: str) -> Path:
path = self.root / name
path.write_text(text, encoding="utf-8")
return path
def test_a_plain_list_drops_blank_lines_and_whitespace(self) -> None:
path = self.write("genes.txt", "TP53\n\n BRCA1 \n\n")
self.assertEqual(pipeline.read_gene_list(path), ["TP53", "BRCA1"])
def test_a_csv_takes_the_first_column_and_skips_the_header(self) -> None:
# The header row must not become a gene symbol.
path = self.write("genes.csv", "gene,logFC\nTP53,2.1\nBRCA1,-1.4\n")
self.assertEqual(pipeline.read_gene_list(path), ["TP53", "BRCA1"])
def test_a_csv_read_as_text_would_keep_the_commas(self) -> None:
# Guards the suffix check: the same content named .txt is not a CSV.
path = self.write("genes.txt", "gene,logFC\nTP53,2.1\n")
self.assertEqual(pipeline.read_gene_list(path), ["gene,logFC", "TP53,2.1"])
def test_an_empty_list_file_yields_no_genes(self) -> None:
self.assertEqual(pipeline.read_gene_list(self.write("none.txt", "\n\n")), [])
def enrichr_frame(term: str = "p53 signaling pathway") -> pd.DataFrame:
"""An enrichment table shaped like gget.enrichr returns."""
return pd.DataFrame(
[
{"name": term, "adjusted_p_value": 1e-8},
{"name": "apoptosis", "adjusted_p_value": 0.002},
]
)
def archs4_frame() -> pd.DataFrame:
"""A tissue-expression table; 'lung' is the clear maximum."""
return pd.DataFrame(
[
{"tissue": "liver", "median": 3.5},
{"tissue": "lung", "median": 9.25},
{"tissue": "kidney", "median": 1.0},
]
)
class EnrichmentPipelineTests(WorkingDirectoryTestCase):
#: The five Enrichr categories the script sweeps, in order.
DATABASES = ["pathway", "ontology", "transcription", "diseases_drugs", "celltypes"]
def test_every_documented_database_is_queried_once_in_order(self) -> None:
enrichr = self.stub("enrichr", return_value=enrichr_frame())
self.stub("archs4", return_value=archs4_frame())
self.quietly(pipeline.enrichment_pipeline, ["TP53"], output_prefix="run")
self.assertEqual(
[call.kwargs["database"] for call in enrichr.call_args_list],
self.DATABASES,
)
def test_species_background_and_plotting_are_forwarded(self) -> None:
enrichr = self.stub("enrichr", return_value=enrichr_frame())
self.stub("archs4", return_value=archs4_frame())
self.quietly(
pipeline.enrichment_pipeline,
["TP53", "BRCA1"],
species="mouse",
background=["ACTB"],
output_prefix="run",
plot=False,
)
for call in enrichr.call_args_list:
self.assertEqual(call.args[0], ["TP53", "BRCA1"])
self.assertEqual(call.kwargs["species"], "mouse")
self.assertEqual(call.kwargs["background_list"], ["ACTB"])
self.assertIs(call.kwargs["plot"], False)
def test_one_csv_per_database_plus_a_summary_is_written(self) -> None:
self.stub("enrichr", return_value=enrichr_frame())
self.stub("archs4", return_value=archs4_frame())
self.quietly(pipeline.enrichment_pipeline, ["TP53"], output_prefix="run")
for database in self.DATABASES:
with self.subTest(database=database):
self.assertTrue((self.root / f"run_{database}.csv").is_file())
summary = pd.read_csv(self.root / "run_summary.csv")
self.assertEqual(len(summary), len(self.DATABASES))
self.assertEqual(summary.loc[0, "Top Term"], "p53 signaling pathway")
self.assertEqual(summary.loc[0, "Total Terms"], 2)
def test_a_database_with_no_hits_is_left_out_of_the_summary(self) -> None:
# Reporting an empty category as a result would overstate the analysis.
self.stub(
"enrichr",
side_effect=[enrichr_frame(), pd.DataFrame(), None,
enrichr_frame(), enrichr_frame()],
)
self.stub("archs4", return_value=archs4_frame())
self.quietly(pipeline.enrichment_pipeline, ["TP53"], output_prefix="run")
summary = pd.read_csv(self.root / "run_summary.csv")
self.assertEqual(len(summary), 3)
self.assertFalse((self.root / "run_ontology.csv").exists())
def test_a_failing_database_does_not_stop_the_sweep(self) -> None:
enrichr = self.stub(
"enrichr",
side_effect=[
RuntimeError("Enrichr is down"),
enrichr_frame(),
enrichr_frame(),
enrichr_frame(),
enrichr_frame(),
],
)
self.stub("archs4", return_value=archs4_frame())
self.quietly(pipeline.enrichment_pipeline, ["TP53"], output_prefix="run")
self.assertEqual(enrichr.call_count, 5)
summary = pd.read_csv(self.root / "run_summary.csv")
self.assertEqual(len(summary), 4)
def test_expression_lookups_are_capped_at_five_genes(self) -> None:
# One ARCHS4 request per gene would make a 2,000-gene list unusable.
self.stub("enrichr", return_value=enrichr_frame())
archs4 = self.stub("archs4", return_value=archs4_frame())
genes = [f"GENE{i}" for i in range(9)]
self.quietly(pipeline.enrichment_pipeline, genes, output_prefix="run")
self.assertEqual(archs4.call_count, 5)
self.assertEqual(
[call.args[0] for call in archs4.call_args_list], genes[:5]
)
for call in archs4.call_args_list:
self.assertEqual(call.kwargs["which"], "tissue")
def test_the_top_tissue_is_the_one_with_the_highest_median(self) -> None:
self.stub("enrichr", return_value=enrichr_frame())
self.stub("archs4", return_value=archs4_frame())
self.quietly(pipeline.enrichment_pipeline, ["TP53"], output_prefix="run")
expression = pd.read_csv(self.root / "run_expression.csv")
self.assertEqual(expression.loc[0, "Gene"], "TP53")
self.assertEqual(expression.loc[0, "Top Tissue"], "lung")
self.assertEqual(expression.loc[0, "Median Expression"], 9.25)
def test_an_expression_failure_leaves_the_enrichment_results_intact(self) -> None:
self.stub("enrichr", return_value=enrichr_frame())
self.stub("archs4", side_effect=RuntimeError("ARCHS4 is down"))
self.assertTrue(
self.quietly(pipeline.enrichment_pipeline, ["TP53"], output_prefix="run")
)
self.assertTrue((self.root / "run_summary.csv").is_file())
self.assertFalse((self.root / "run_expression.csv").exists())
def search_frame(ensembl_id: str = "ENSG00000141510") -> pd.DataFrame:
"""A gget.search result for TP53."""
return pd.DataFrame(
[
{
"ensembl_id": ensembl_id,
"ensembl_description": "tumor protein p53",
"gene_name": "TP53",
}
]
)
def info_frame() -> pd.DataFrame:
return pd.DataFrame(
[{"uniprot_id": "P04637", "pdb_id": "1TUP", "gene_name": "TP53"}]
)
def correlation_frame() -> pd.DataFrame:
return pd.DataFrame(
[{"gene_symbol": "MDM2", "correlation": 0.81}]
)
def diseases_frame() -> pd.DataFrame:
return pd.DataFrame(
[{"disease_name": "Li-Fraumeni syndrome", "overall_score": 0.9}]
)
class GeneAnalysisTests(WorkingDirectoryTestCase):
def stub_everything(self) -> dict:
return {
"search": self.stub("search", return_value=search_frame()),
"info": self.stub("info", return_value=info_frame()),
"seq": self.stub("seq", return_value=[">ENSG00000141510", "ATGGAG"]),
"archs4": self.stub(
"archs4", side_effect=[archs4_frame(), correlation_frame()]
),
"opentargets": self.stub(
"opentargets", side_effect=[diseases_frame(), pd.DataFrame()]
),
}
def test_the_search_is_scoped_to_the_species_and_a_single_hit(self) -> None:
stubs = self.stub_everything()
self.assertTrue(
self.quietly(gene_analysis.analyze_gene, "TP53", "homo_sapiens")
)
stubs["search"].assert_called_once()
call = stubs["search"].call_args
self.assertEqual(call.args[0], ["TP53"])
self.assertEqual(call.kwargs["species"], "homo_sapiens")
self.assertEqual(call.kwargs["limit"], 1)
def test_a_gene_that_does_not_exist_stops_before_any_other_request(self) -> None:
# Without the early return the Ensembl ID lookup would index an empty
# frame and raise instead of reporting "not found".
search = self.stub("search", return_value=pd.DataFrame())
info = self.stub("info")
self.assertFalse(
self.quietly(gene_analysis.analyze_gene, "NOT_A_GENE", "homo_sapiens")
)
search.assert_called_once()
info.assert_not_called()
def test_the_ensembl_identifier_from_the_search_drives_every_lookup(self) -> None:
stubs = self.stub_everything()
self.quietly(gene_analysis.analyze_gene, "TP53")
self.assertEqual(stubs["info"].call_args.args[0], ["ENSG00000141510"])
self.assertIs(stubs["info"].call_args.kwargs["pdb"], True)
for call in stubs["opentargets"].call_args_list:
self.assertEqual(call.args[0], "ENSG00000141510")
def test_both_sequence_forms_are_requested(self) -> None:
stubs = self.stub_everything()
self.quietly(gene_analysis.analyze_gene, "TP53")
self.assertEqual(stubs["seq"].call_count, 2)
translated = [
call.kwargs.get("translate", False) for call in stubs["seq"].call_args_list
]
self.assertEqual(translated, [False, True])
def test_the_fasta_files_hold_the_lines_gget_returned(self) -> None:
# gget.seq returns a list of FASTA lines; writing the list itself
# raises TypeError, so the join is what makes the file valid.
self.stub_everything()
self.quietly(gene_analysis.analyze_gene, "TP53")
text = (self.root / "tp53_nucleotide.fasta").read_text(encoding="utf-8")
self.assertEqual(text, ">ENSG00000141510\nATGGAG\n")
def test_an_already_joined_string_is_written_unchanged(self) -> None:
# Older gget releases returned one string; both shapes must work.
self.assertEqual(
gene_analysis.fasta_text(">x\nATG\n"), ">x\nATG\n"
)
self.assertEqual(gene_analysis.fasta_text(">x\nATG"), ">x\nATG\n")
self.assertEqual(gene_analysis.fasta_text(None), "")
def test_the_output_prefix_defaults_to_the_lowercased_gene(self) -> None:
self.stub_everything()
self.quietly(gene_analysis.analyze_gene, "TP53")
self.assertTrue((self.root / "tp53_info.csv").is_file())
def test_an_explicit_prefix_is_used_verbatim(self) -> None:
self.stub_everything()
self.quietly(gene_analysis.analyze_gene, "TP53", output_prefix="run1")
self.assertTrue((self.root / "run1_info.csv").is_file())
def test_expression_and_correlation_are_separate_archs4_queries(self) -> None:
stubs = self.stub_everything()
self.quietly(gene_analysis.analyze_gene, "TP53")
self.assertEqual(
[call.kwargs["which"] for call in stubs["archs4"].call_args_list],
["tissue", "correlation"],
)
def test_both_opentargets_resources_are_requested_with_a_limit(self) -> None:
stubs = self.stub_everything()
self.quietly(gene_analysis.analyze_gene, "TP53")
self.assertEqual(
[call.kwargs["resource"] for call in stubs["opentargets"].call_args_list],
["diseases", "drugs"],
)
for call in stubs["opentargets"].call_args_list:
self.assertEqual(call.kwargs["limit"], 10)
def test_an_empty_drug_table_is_not_written(self) -> None:
self.stub_everything() # drugs is an empty frame
self.quietly(gene_analysis.analyze_gene, "TP53")
self.assertTrue((self.root / "tp53_diseases.csv").is_file())
self.assertFalse((self.root / "tp53_drugs.csv").exists())
def test_optional_lookups_may_fail_without_failing_the_run(self) -> None:
# Steps 4-7 are wrapped in warnings on purpose: the sequences and
# annotations already retrieved are worth keeping.
self.stub("search", return_value=search_frame())
self.stub("info", return_value=info_frame())
self.stub("seq", return_value=[">x", "ATG"])
self.stub("archs4", side_effect=RuntimeError("ARCHS4 is down"))
self.stub("opentargets", side_effect=RuntimeError("Open Targets is down"))
self.assertTrue(self.quietly(gene_analysis.analyze_gene, "TP53"))
self.assertTrue((self.root / "tp53_nucleotide.fasta").is_file())
self.assertFalse((self.root / "tp53_tissue_expression.csv").exists())
if __name__ == "__main__":
unittest.main()

267
tests/hugging-science/test_scripts.py Normal file
View File

@@ -0,0 +1,267 @@
"""Tests for the Hugging Science catalogue fetcher.
The whole script is a markdown parser wrapped in a network call, so the tests
feed `parse_markdown` fixture documents directly and never fetch anything. The
parser is a small state machine over headings and bullets, and its edge cases
-- a section suffix like `## Datasets (12)`, an H3 before any H2, a bullet key
it does not recognise -- are where a catalogue silently loses entries.
"""
from __future__ import annotations
import sys
import unittest
from unittest import mock
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "hugging-science"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import fetch_catalog # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
CATALOGUE = """\
# Hugging Science
## Datasets (2)
### Protein Folding Benchmark
- **type**: dataset
- **tags**: biology, benchmark
- **huggingface**: https://huggingface.co/datasets/example/folding
- **author**: Example Lab
- **date**: 2026-01-15
A held-out set of folded structures.
Spanning two lines of prose.
### Climate Reanalysis Grid
- **type**: dataset
- **tags**: climate, earth-science
- **link**: https://huggingface.co/datasets/example/climate
## Models
### Genomics Encoder
- **type**: model
- **tags**: genomics
- **url**: https://huggingface.co/example/genomics-encoder
## Blog Posts
### Why Open Science Needs Open Weights
- **author**: A. Writer
- **date**: 2026-02-02
An argument.
"""
class ParserTests(unittest.TestCase):
def setUp(self) -> None:
self.entries = fetch_catalog.parse_markdown(CATALOGUE)
def test_every_entry_is_found(self) -> None:
self.assertEqual(
[entry.title for entry in self.entries],
[
"Protein Folding Benchmark",
"Climate Reanalysis Grid",
"Genomics Encoder",
"Why Open Science Needs Open Weights",
],
)
def test_the_count_suffix_is_stripped_from_the_section_label(self) -> None:
# "## Datasets (2)" must land in the same bucket as "## Datasets".
self.assertEqual(self.entries[0].section, "datasets")
self.assertEqual(self.entries[1].section, "datasets")
self.assertEqual(self.entries[2].section, "models")
self.assertEqual(self.entries[3].section, "blog posts")
def test_metadata_bullets_populate_the_record(self) -> None:
entry = self.entries[0]
self.assertEqual(entry.type, "dataset")
self.assertEqual(entry.tags, ["biology", "benchmark"])
self.assertEqual(entry.url, "https://huggingface.co/datasets/example/folding")
self.assertEqual(entry.author, "Example Lab")
self.assertEqual(entry.date, "2026-01-15")
def test_huggingface_link_and_url_keys_all_fill_the_url(self) -> None:
self.assertTrue(self.entries[0].url) # **huggingface**
self.assertTrue(self.entries[1].url) # **link**
self.assertTrue(self.entries[2].url) # **url**
def test_prose_after_the_bullets_becomes_the_description(self) -> None:
self.assertEqual(
self.entries[0].description,
"A held-out set of folded structures. Spanning two lines of prose.",
)
def test_an_entry_without_prose_has_an_empty_description(self) -> None:
self.assertEqual(self.entries[1].description, "")
def test_content_before_the_first_entry_is_ignored(self) -> None:
entries = fetch_catalog.parse_markdown(
"# Title\n\nSome preamble prose.\n\n### First\n- **type**: model\n"
)
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].title, "First")
def test_an_entry_before_any_section_is_labelled_unknown(self) -> None:
entries = fetch_catalog.parse_markdown("### Orphan\n- **type**: model\n")
self.assertEqual(entries[0].section, "unknown")
def test_unrecognised_bullet_keys_are_dropped_not_misfiled(self) -> None:
entries = fetch_catalog.parse_markdown(
"## Models\n\n### X\n- **license**: mit\n- **type**: model\n"
)
self.assertEqual(entries[0].type, "model")
self.assertEqual(entries[0].tags, [])
def test_empty_input_yields_no_entries(self) -> None:
self.assertEqual(fetch_catalog.parse_markdown(""), [])
self.assertEqual(fetch_catalog.parse_markdown("## Datasets\n"), [])
def test_tags_are_split_and_blank_entries_discarded(self) -> None:
entries = fetch_catalog.parse_markdown(
"### X\n- **tags**: a, , b ,\n"
)
self.assertEqual(entries[0].tags, ["a", "b"])
class FilterTests(unittest.TestCase):
def setUp(self) -> None:
self.entries = fetch_catalog.parse_markdown(CATALOGUE)
def test_no_filter_matches_everything(self) -> None:
self.assertTrue(all(e.matches_filter(None, None) for e in self.entries))
def test_each_cli_filter_value_selects_its_section(self) -> None:
expected = {"datasets": 2, "models": 1, "blogs": 1}
for kind, count in expected.items():
with self.subTest(kind=kind):
matched = [e for e in self.entries if e.matches_filter(kind, None)]
self.assertEqual(len(matched), count)
def test_the_aliases_absorb_singular_section_headings_upstream(self) -> None:
# The alias sets map one CLI value onto the section spellings the
# source markdown may use, so `--filter datasets` still works when a
# topic file writes `## Dataset`.
singular = fetch_catalog.parse_markdown("## Dataset\n\n### X\n- **type**: dataset\n")
self.assertTrue(singular[0].matches_filter("datasets", None))
blog = fetch_catalog.parse_markdown("## Blog\n\n### Y\n")
self.assertTrue(blog[0].matches_filter("blogs", None))
def test_the_cli_only_offers_the_aliased_filter_values(self) -> None:
# matches_filter falls back to an exact section match for anything that
# is not an alias key, so the CLI must not offer values outside them.
source = (SCRIPTS / "fetch_catalog.py").read_text(encoding="utf-8")
self.assertIn('choices=["datasets", "models", "blogs"]', source)
def test_kind_matching_is_case_insensitive(self) -> None:
matched = [e for e in self.entries if e.matches_filter("MODELS", None)]
self.assertEqual(len(matched), 1)
def test_tag_matching_is_a_case_insensitive_substring(self) -> None:
matched = [e.title for e in self.entries if e.matches_filter(None, "GENOM")]
self.assertEqual(matched, ["Genomics Encoder"])
def test_a_tag_also_matches_the_type_field(self) -> None:
# `matches_filter` falls back to `type` so `--tag model` still works
# on entries that carry no explicit tags.
entry = fetch_catalog.Entry(title="X", section="models", type="model")
self.assertTrue(entry.matches_filter(None, "model"))
def test_kind_and_tag_must_both_match(self) -> None:
matched = [e for e in self.entries if e.matches_filter("datasets", "genomics")]
self.assertEqual(matched, [])
def test_an_unknown_kind_matches_nothing_rather_than_everything(self) -> None:
matched = [e for e in self.entries if e.matches_filter("posters", None)]
self.assertEqual(matched, [])
class RenderTests(unittest.TestCase):
def setUp(self) -> None:
self.entries = fetch_catalog.parse_markdown(CATALOGUE)
def test_an_entry_renders_only_the_fields_it_has(self) -> None:
rendered = fetch_catalog.render_entry(self.entries[1])
self.assertIn("### Climate Reanalysis Grid", rendered)
self.assertIn("- Type: dataset", rendered)
self.assertIn("- URL: https://huggingface.co/datasets/example/climate", rendered)
self.assertNotIn("- Author:", rendered)
self.assertNotIn("- Date:", rendered)
def test_grouped_rendering_counts_each_section(self) -> None:
rendered = fetch_catalog.render_entries(self.entries)
self.assertIn("## Datasets (2)", rendered)
self.assertIn("## Models (1)", rendered)
self.assertIn("## Blog Posts (1)", rendered)
def test_ungrouped_rendering_omits_section_headers(self) -> None:
rendered = fetch_catalog.render_entries(self.entries, group_by_section=False)
self.assertNotIn("## Datasets", rendered)
self.assertIn("### Genomics Encoder", rendered)
def test_an_empty_result_says_so_rather_than_rendering_nothing(self) -> None:
self.assertEqual(fetch_catalog.render_entries([]), "(no entries matched)")
def test_a_rendered_catalogue_round_trips_through_the_parser(self) -> None:
# render_entries emits the same H2/H3 shape the parser reads, so a
# round trip must preserve every title and section.
reparsed = fetch_catalog.parse_markdown(
fetch_catalog.render_entries(self.entries)
)
self.assertEqual(
[(e.title, e.section) for e in reparsed],
[(e.title, e.section) for e in self.entries],
)
class CatalogueSourceTests(unittest.TestCase):
def test_topic_slugs_are_lowercase_hyphenated_and_sorted(self) -> None:
topics = fetch_catalog.KNOWN_TOPICS
self.assertTrue(topics)
self.assertEqual(topics, sorted(topics))
for topic in topics:
with self.subTest(topic=topic):
self.assertRegex(topic, r"^[a-z]+(-[a-z]+)*$")
def test_the_base_url_is_https(self) -> None:
self.assertTrue(fetch_catalog.BASE.startswith("https://"))
def test_requests_identify_the_skill_and_carry_a_timeout(self) -> None:
response = mock.MagicMock()
response.__enter__.return_value.read.return_value = b"ok"
with mock.patch("urllib.request.urlopen", return_value=response) as opened:
self.assertEqual(fetch_catalog.fetch(f"{fetch_catalog.BASE}/llms.txt"), "ok")
request = opened.call_args.args[0]
self.assertIn("hugging-science-skill", request.get_header("User-agent"))
self.assertEqual(opened.call_args.kwargs["timeout"], 30)
def test_network_failures_exit_with_a_message_rather_than_a_traceback(self) -> None:
import urllib.error
failures = [
urllib.error.HTTPError("u", 404, "Not Found", {}, None),
urllib.error.URLError("no route to host"),
]
for error in failures:
with self.subTest(error=type(error).__name__):
with mock.patch("urllib.request.urlopen", side_effect=error):
with self.assertRaises(SystemExit) as raised:
fetch_catalog.fetch("https://example.invalid/x")
self.assertNotIn("Traceback", str(raised.exception))
if __name__ == "__main__":
unittest.main()

2
tests/hypogenic/test_scripts.py
View File

@@ -237,7 +237,7 @@ class SafetyAndHelpTests(unittest.TestCase):
def test_skill_metadata_progressive_disclosure_and_file_set(self) -> None:
skill = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertIn("license: MIT", skill)
self.assertIn("metadata:\n version: \"1.1\"", skill)
self.assertRegex(skill, r'\nmetadata:\n version: "\d+\.\d+"\n')
self.assertLess(len(skill.splitlines()), 500)
self.assertFalse((REFERENCES / "config_template.yaml").exists())
self.assertEqual(

26
tests/hypothesis-generation/test_scripts.py
View File

@@ -3,10 +3,8 @@
from __future__ import annotations
import ast
import copy
import csv
import json
import re
import stat
import sys
import tempfile
@@ -42,6 +40,8 @@ from validate_hypothesis_schema import ( # noqa: E402
)
from validate_prediction_matrix import load_matrix, validate_matrix # noqa: E402
import skill_contract
def load_asset_json(name: str) -> dict:
return json.loads((ASSETS / name).read_text(encoding="utf-8"))
@@ -371,25 +371,10 @@ class FileSafetyAndStaticTests(unittest.TestCase):
len(rows), len({row["source_id"] for row in rows})
)
def test_markdown_local_path_references_exist(self) -> None:
markdown_files = [
SKILL_ROOT / "SKILL.md",
*(SKILL_ROOT / "references").glob("*.md"),
]
pattern = re.compile(
r"`((?:assets|references|scripts)/[A-Za-z0-9_./-]+)`"
)
for markdown in markdown_files:
for relative in pattern.findall(markdown.read_text(encoding="utf-8")):
self.assertTrue(
(SKILL_ROOT / relative).is_file(),
f"{markdown.name}: missing {relative}",
)
def test_skill_frontmatter_and_progressive_disclosure(self) -> None:
skill = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertLess(len(skill.splitlines()), 500)
self.assertIn('version: "2.1"', skill)
self.assertRegex(skill, r'\n version: "\d+\.\d+"\n')
self.assertIn("license: MIT", skill)
self.assertIn("compatibility:", skill)
self.assertNotIn("OPENROUTER", skill)
@@ -406,5 +391,10 @@ class FileSafetyAndStaticTests(unittest.TestCase):
self.assertFalse(list(SKILL_ROOT.rglob("*.pyc")))
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
if __name__ == "__main__":
unittest.main()

163
tests/infographics/test_scripts.py Normal file
View File

@@ -0,0 +1,163 @@
"""Tests for the infographics generator CLI.
Generation goes through OpenRouter, so nothing here calls it. What is testable
offline is the part that decides what the subprocess sees and what the user may
ask for: the environment allowlist (the same design as the schematic scripts --
forward a named set, never the whole parent environment) and the option
catalogue, where `--list-options` documents choices that argparse must actually
accept.
The palette presets get their own attention: this skill advertises them as
colourblind-safe, and a preset the generator does not know is a silent
downgrade to whatever it picks instead.
"""
from __future__ import annotations
import subprocess
import sys
import unittest
from pathlib import Path
from unittest import mock
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "infographics"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import generate_infographic # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
class SubprocessEnvironmentTests(unittest.TestCase):
def test_only_allowlisted_variables_are_forwarded(self) -> None:
environment = {
"PATH": "/usr/bin",
"HOME": "/home/someone",
"AWS_SECRET_ACCESS_KEY": "must-not-leak",
"SLACK_TOKEN": "also-not",
}
with mock.patch.dict("os.environ", environment, clear=True):
built = generate_infographic.build_subprocess_env(None)
self.assertEqual(set(built), {"PATH", "HOME"})
def test_the_api_key_is_injected_when_supplied(self) -> None:
with mock.patch.dict("os.environ", {"PATH": "/usr/bin"}, clear=True):
built = generate_infographic.build_subprocess_env("sk-or-test")
self.assertEqual(built["OPENROUTER_API_KEY"], "sk-or-test")
def test_an_empty_key_is_omitted_rather_than_passed_blank(self) -> None:
with mock.patch.dict("os.environ", {"PATH": "/usr/bin"}, clear=True):
for value in (None, ""):
with self.subTest(value=value):
self.assertNotIn(
"OPENROUTER_API_KEY",
generate_infographic.build_subprocess_env(value),
)
def test_the_allowlist_carries_no_credential_variables(self) -> None:
forwarded = " ".join(generate_infographic.FORWARDED_ENV_VARS).upper()
for banned in ("SECRET", "TOKEN", "PASSWORD", "API_KEY", "CREDENTIAL"):
with self.subTest(term=banned):
self.assertNotIn(banned, forwarded)
def test_the_allowlist_keeps_tls_and_proxy_settings_working(self) -> None:
forwarded = set(generate_infographic.FORWARDED_ENV_VARS)
for required in ("HTTPS_PROXY", "https_proxy", "SSL_CERT_FILE", "REQUESTS_CA_BUNDLE"):
with self.subTest(variable=required):
self.assertIn(required, forwarded)
def test_the_allowlist_matches_the_schematic_scripts(self) -> None:
# Both CLIs solve the same problem the same way; a divergence means one
# of them was updated and the other forgotten.
schematic = (
SKILL_ROOT.parent / "scientific-schematics" / "scripts" / "generate_schematic.py"
).read_text(encoding="utf-8")
for variable in generate_infographic.FORWARDED_ENV_VARS:
with self.subTest(variable=variable):
self.assertIn(f'"{variable}"', schematic)
class OptionCatalogueTests(unittest.TestCase):
CATALOGUES = {
"type": "INFOGRAPHIC_TYPES",
"style": "STYLE_PRESETS",
"palette": "PALETTE_PRESETS",
"doc-type": "DOC_TYPES",
}
def setUp(self) -> None:
self.source = (SCRIPTS / "generate_infographic.py").read_text(encoding="utf-8")
def test_every_catalogue_is_non_empty_and_free_of_duplicates(self) -> None:
for attribute in self.CATALOGUES.values():
with self.subTest(catalogue=attribute):
values = getattr(generate_infographic, attribute)
self.assertTrue(values)
self.assertEqual(len(set(values)), len(values))
def test_every_catalogued_value_is_documented_by_list_options(self) -> None:
import io
from contextlib import redirect_stdout
buffer = io.StringIO()
with redirect_stdout(buffer):
generate_infographic.list_options()
documented = buffer.getvalue()
for attribute in self.CATALOGUES.values():
for value in getattr(generate_infographic, attribute):
with self.subTest(option=value):
self.assertIn(value, documented)
def test_the_parser_offers_each_catalogue_as_choices(self) -> None:
# `choices=` wired to the catalogue means adding an option is one edit;
# a hand-copied list drifts.
for flag, attribute in self.CATALOGUES.items():
with self.subTest(flag=flag):
self.assertIn(f"choices={attribute}", self.source)
def test_the_palettes_are_the_recognised_colourblind_safe_sets(self) -> None:
self.assertEqual(
set(generate_infographic.PALETTE_PRESETS), {"wong", "ibm", "tol"}
)
def test_option_names_are_lowercase_slugs(self) -> None:
for attribute in self.CATALOGUES.values():
for value in getattr(generate_infographic, attribute):
with self.subTest(option=value):
self.assertRegex(value, r"^[a-z][a-z0-9-]*$")
class ParserTests(unittest.TestCase):
def test_help_lists_every_documented_flag(self) -> None:
result = skill_contract.cli.run_help(SCRIPTS / "generate_infographic.py")
self.assertEqual(result.returncode, 0, result.stderr)
for flag in ("--type", "--style", "--palette", "--doc-type", "--output"):
with self.subTest(flag=flag):
self.assertIn(flag, result.stdout)
def test_an_unknown_option_value_is_rejected_at_the_boundary(self) -> None:
result = subprocess.run(
[
sys.executable,
str(SCRIPTS / "generate_infographic.py"),
"a prompt",
"-o", "out.png",
"--palette", "rainbow",
],
capture_output=True,
text=True,
timeout=60,
)
self.assertNotEqual(result.returncode, 0)
self.assertIn("rainbow", result.stderr)
def test_the_generator_script_is_shipped(self) -> None:
self.assertTrue((SCRIPTS / "generate_infographic_ai.py").is_file())
if __name__ == "__main__":
unittest.main()

5
tests/iso-standards-readiness/test_scripts.py
View File

@@ -674,10 +674,5 @@ class CLITests(unittest.TestCase):
self.assertIn("CHOICE_INVALID", codes)
self.assertIn("VALUE_UNKNOWN", codes)
def test_no_bytecode_artifacts_created(self) -> None:
for path in ROOT.rglob("__pycache__"):
self.fail(f"unexpected bytecode directory: {path}")
if __name__ == "__main__":
unittest.main()

400
tests/labarchive-integration/test_scripts.py Normal file
View File

@@ -0,0 +1,400 @@
"""Tests for the LabArchives integration helpers.
Three security boundaries live in these scripts, and all three are testable
without touching LabArchives:
* the HMAC-SHA-512 request signature, which the vendor documents with a public
test vector -- so the suite is a known-answer test, not a self-consistency
check;
* the path validators in front of that signature, which refuse anything that
would sign a different route than the one actually requested;
* the `.eln` container inspector, which unpacks untrusted archives and must
reject traversal, absolute paths, and symlink members.
Nothing here uses a real credential; the only key material is the vendor's own
published dummy vector.
"""
from __future__ import annotations
import base64
import hashlib
import hmac
import io
import json
import stat
import sys
import tempfile
import unittest
import zipfile
from contextlib import redirect_stdout
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "labarchive-integration"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import entry_operations # noqa: E402
import notebook_operations # noqa: E402
import setup_config # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
class SignatureTests(unittest.TestCase):
def test_the_official_vendor_vector_reproduces_exactly(self) -> None:
vector = entry_operations._OFFICIAL_VECTOR
signature = entry_operations.create_signature(
vector["access_key_id"],
vector["api_method_input"],
int(vector["expires_ms"]),
vector["access_password"],
)
self.assertEqual(signature, vector["signature"])
def test_the_signature_is_base64_hmac_sha512_over_the_concatenation(self) -> None:
# Recomputed independently: any change to the message layout shows up
# here rather than as a 401 from the API.
expected = base64.b64encode(
hmac.new(
b"secret", b"keyidmethod12345", hashlib.sha512
).digest()
).decode("ascii")
self.assertEqual(
entry_operations.create_signature("keyid", "method", 12345, "secret"),
expected,
)
def test_every_input_changes_the_signature(self) -> None:
base = entry_operations.create_signature("k", "m", 1, "s")
variants = {
"key": entry_operations.create_signature("K", "m", 1, "s"),
"method": entry_operations.create_signature("k", "M", 1, "s"),
"expires": entry_operations.create_signature("k", "m", 2, "s"),
"secret": entry_operations.create_signature("k", "m", 1, "S"),
}
for name, signature in variants.items():
with self.subTest(changed=name):
self.assertNotEqual(signature, base)
def test_empty_inputs_are_refused_before_signing(self) -> None:
cases = [
(("", "m", 1, "s"), "Access Key ID"),
(("k", "", 1, "s"), "API method input"),
(("k", "m", 1, ""), "Access Password"),
]
for arguments, label in cases:
with self.subTest(label=label):
with self.assertRaises(setup_config.ConfigError) as raised:
entry_operations.create_signature(*arguments)
self.assertIn(label, str(raised.exception))
def test_a_negative_expiry_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "non-negative"):
entry_operations.create_signature("k", "m", -1, "s")
def test_the_signature_is_uri_encoded_for_a_query_parameter(self) -> None:
# Base64 contains '+' and '/', which are meaningful in a query string.
encoded = entry_operations.encode_eln_signature("a+b/c==")
self.assertEqual(encoded, "a%2Bb%2Fc%3D%3D")
def test_the_self_test_command_reports_a_pass_without_a_network_call(self) -> None:
class Args:
compact = True
buffer = io.StringIO()
with redirect_stdout(buffer):
code = entry_operations.command_self_test(Args())
self.assertEqual(code, 0)
payload = json.loads(buffer.getvalue())
self.assertTrue(payload["passed"])
self.assertFalse(payload["remote_request_performed"])
# The signature itself must never be printed, only a fingerprint.
self.assertNotIn(
entry_operations._OFFICIAL_VECTOR["signature"], buffer.getvalue()
)
class AuthParameterTests(unittest.TestCase):
def test_eln_parameters_carry_the_key_expiry_and_signature(self) -> None:
params = entry_operations.build_eln_auth_params(
"keyid", "secret", "entry_attachment", expires_ms=1000
)
self.assertEqual(set(params), {"akid", "expires", "sig"})
self.assertEqual(params["akid"], "keyid")
self.assertEqual(params["expires"], "1000")
self.assertEqual(
params["sig"],
entry_operations.create_signature("keyid", "entry_attachment", 1000, "secret"),
)
def test_the_password_never_appears_in_the_parameters(self) -> None:
params = entry_operations.build_eln_auth_params(
"keyid", "hunter2", "entry_attachment", expires_ms=1000
)
self.assertNotIn("hunter2", json.dumps(params))
def test_inventory_headers_sign_the_resolved_path(self) -> None:
headers = entry_operations.build_inventory_headers(
"keyid", "secret", "user", "lab", "/public/v1/items/42", expires_ms=1000
)
self.assertEqual(
headers["X-LabArchives-Signature"],
entry_operations.create_signature(
"keyid", "/public/v1/items/42", 1000, "secret"
),
)
self.assertEqual(headers["X-LabArchives-UId"], "user")
self.assertEqual(headers["X-LabArchives-LabId"], "lab")
def test_an_omitted_expiry_defaults_to_now(self) -> None:
params = entry_operations.build_eln_auth_params("k", "s", "entry_attachment")
self.assertTrue(params["expires"].isdigit())
self.assertGreater(int(params["expires"]), 1_700_000_000_000)
class ComponentValidationTests(unittest.TestCase):
def test_valid_method_names_are_accepted(self) -> None:
for value in ("entry_attachment", "users", "tree_tools", "a1_b2"):
with self.subTest(value=value):
self.assertEqual(
entry_operations.validate_eln_component(value, "method"), value
)
def test_anything_outside_the_documented_shape_is_refused(self) -> None:
for value in ("Entry", "1entry", "entry-attachment", "entry attachment", "", "entry/x"):
with self.subTest(value=value):
with self.assertRaises(setup_config.ConfigError):
entry_operations.validate_eln_component(value, "method")
class InventoryPathTests(unittest.TestCase):
def test_a_resolved_route_is_accepted(self) -> None:
for path in ("/public/v1/items", "/public/v1/items/42", "/public/v1/a.b~c"):
with self.subTest(path=path):
self.assertEqual(entry_operations.validate_inventory_path(path), path)
def test_a_query_or_fragment_is_refused(self) -> None:
# Signing a path but sending it with a query signs the wrong thing.
for path in ("/public/v1/items?page=2", "/public/v1/items#top"):
with self.subTest(path=path):
with self.assertRaisesRegex(ValueError, "query strings and fragments"):
entry_operations.validate_inventory_path(path)
def test_a_percent_encoded_path_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "unencoded relative route"):
entry_operations.validate_inventory_path("/public/v1/it%20ems")
def test_unresolved_placeholders_are_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "resolve all Inventory route placeholders"):
entry_operations.validate_inventory_path("/public/v1/items/{id}")
def test_a_path_outside_the_public_v1_prefix_is_refused(self) -> None:
for path in ("/private/v1/items", "public/v1/items", "/public/v2/items"):
with self.subTest(path=path):
with self.assertRaises(setup_config.ConfigError):
entry_operations.validate_inventory_path(path)
def test_traversal_and_ambiguous_segments_are_refused(self) -> None:
for path in ("/public/v1/../secret", "/public/v1/./items", "/public/v1//items"):
with self.subTest(path=path):
with self.assertRaises(setup_config.ConfigError):
entry_operations.validate_inventory_path(path)
def test_whitespace_and_backslashes_are_refused(self) -> None:
for path in ("/public/v1/it ems", "/public/v1/it\tems", "/public/v1\\items"):
with self.subTest(path=repr(path)):
with self.assertRaises(setup_config.ConfigError):
entry_operations.validate_inventory_path(path)
class ApiUrlTests(unittest.TestCase):
def test_every_documented_region_url_normalises_to_itself(self) -> None:
self.assertTrue(setup_config.REGIONS)
for region, values in setup_config.REGIONS.items():
with self.subTest(region=region):
url = values["eln_api_url"]
self.assertEqual(setup_config.normalize_eln_api_url(url), url)
def test_a_trailing_slash_and_mixed_case_host_are_normalised(self) -> None:
url = next(iter(setup_config.REGIONS.values()))["eln_api_url"]
host = url.removeprefix("https://").removesuffix("/api")
self.assertEqual(
setup_config.normalize_eln_api_url(f"https://{host.upper()}/api/"), url
)
def test_plain_http_is_refused(self) -> None:
url = next(iter(setup_config.REGIONS.values()))["eln_api_url"]
with self.assertRaisesRegex(ValueError, "must use https"):
setup_config.normalize_eln_api_url(url.replace("https://", "http://"))
def test_embedded_credentials_are_refused(self) -> None:
url = next(iter(setup_config.REGIONS.values()))["eln_api_url"]
host = url.removeprefix("https://").removesuffix("/api")
with self.assertRaisesRegex(ValueError, "credentials must not be embedded"):
setup_config.normalize_eln_api_url(f"https://user:pass@{host}/api")
def test_a_custom_port_query_or_fragment_is_refused(self) -> None:
url = next(iter(setup_config.REGIONS.values()))["eln_api_url"]
host = url.removeprefix("https://").removesuffix("/api")
cases = {
f"https://{host}:8443/api": "custom port",
f"https://{host}/api?x=1": "query or fragment",
f"https://{host}/api#y": "query or fragment",
}
for candidate, expected in cases.items():
with self.subTest(url=candidate):
with self.assertRaisesRegex(ValueError, expected):
setup_config.normalize_eln_api_url(candidate)
def test_a_non_allowlisted_host_is_refused_and_lists_the_alternatives(self) -> None:
with self.assertRaises(setup_config.ConfigError) as raised:
setup_config.normalize_eln_api_url("https://evil.example.invalid/api")
self.assertIn("not allowlisted", str(raised.exception))
def test_the_path_must_be_exactly_api(self) -> None:
url = next(iter(setup_config.REGIONS.values()))["eln_api_url"]
host = url.removeprefix("https://").removesuffix("/api")
with self.assertRaisesRegex(ValueError, "path must be exactly /api"):
setup_config.normalize_eln_api_url(f"https://{host}/api/v2")
def test_an_empty_url_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "is empty"):
setup_config.normalize_eln_api_url(" ")
class ContainerMemberTests(unittest.TestCase):
"""`.eln` archives come from outside; every member name is untrusted."""
def test_ordinary_member_paths_are_accepted(self) -> None:
for name in ("lamanifest.xml", "data/entry.json", "a/b/c.txt"):
with self.subTest(name=name):
self.assertIsNone(notebook_operations._member_path_error(name))
def test_traversal_and_absolute_paths_are_rejected(self) -> None:
for name in ("../escape.xml", "a/../../b", "/etc/passwd", "C:\\win\\x"):
with self.subTest(name=name):
self.assertIsNotNone(notebook_operations._member_path_error(name))
def test_backslashes_nulls_and_empty_names_are_rejected(self) -> None:
for name in ("a\\b", "a\x00b", ""):
with self.subTest(name=repr(name)):
self.assertIsNotNone(notebook_operations._member_path_error(name))
def test_redundant_but_harmless_segments_are_allowed(self) -> None:
# PurePosixPath collapses `.` and doubled separators, so these resolve
# inside the extraction root and are not treated as ambiguous. Only
# `..` actually escapes.
for name in ("./a.txt", "a/./b", "a//b"):
with self.subTest(name=name):
self.assertIsNone(notebook_operations._member_path_error(name))
def test_symlink_members_are_detected_from_the_mode_bits(self) -> None:
link = zipfile.ZipInfo("link")
link.external_attr = (stat.S_IFLNK | 0o777) << 16
self.assertTrue(notebook_operations._is_symlink(link))
regular = zipfile.ZipInfo("file")
regular.external_attr = (stat.S_IFREG | 0o644) << 16
self.assertFalse(notebook_operations._is_symlink(regular))
class ContainerLimitTests(unittest.TestCase):
def test_the_documented_limits_are_positive_and_ordered(self) -> None:
# Every limit exists to stop a zip bomb; a zero or negative one would
# disable the guard silently.
for name in (
"DEFAULT_MAX_MEMBERS",
"DEFAULT_MAX_TOTAL_BYTES",
"DEFAULT_MAX_MANIFEST_BYTES",
"DEFAULT_MAX_INDEX_BYTES",
"DEFAULT_MAX_COMPRESSION_RATIO",
):
with self.subTest(limit=name):
self.assertGreater(getattr(notebook_operations, name), 0)
self.assertLess(
notebook_operations.DEFAULT_MAX_MANIFEST_BYTES,
notebook_operations.DEFAULT_MAX_TOTAL_BYTES,
)
def _inspect(self, build) -> dict:
with tempfile.TemporaryDirectory() as directory:
archive = Path(directory) / "notebook.eln"
with zipfile.ZipFile(archive, "w") as handle:
build(handle)
return notebook_operations.inspect_container(archive)
def test_a_traversing_member_is_reported_as_an_error(self) -> None:
# The inspector reports rather than raises: the point is to hand back a
# full account of what is wrong with an archive, not to stop at the
# first problem.
report = self._inspect(lambda z: z.writestr("../escaped.xml", "<x/>"))
self.assertTrue(report["errors"])
self.assertTrue(
any("unsafe member paths" in error for error in report["errors"]),
report["errors"],
)
def test_a_symlink_member_is_reported_as_an_error(self) -> None:
def build(handle: zipfile.ZipFile) -> None:
info = zipfile.ZipInfo("link")
info.external_attr = (stat.S_IFLNK | 0o777) << 16
handle.writestr(info, "/etc/passwd")
report = self._inspect(build)
self.assertTrue(report["errors"])
def test_a_missing_manifest_is_reported(self) -> None:
report = self._inspect(lambda z: z.writestr("data/entry.json", "{}"))
self.assertTrue(report["errors"] or report["warnings"])
combined = " ".join(report["errors"] + report["warnings"])
self.assertIn(notebook_operations.MANIFEST_NAME, combined)
def test_non_positive_limits_are_refused_outright(self) -> None:
with tempfile.TemporaryDirectory() as directory:
archive = Path(directory) / "notebook.eln"
with zipfile.ZipFile(archive, "w") as handle:
handle.writestr("a.txt", "x")
for override in ("max_members", "max_total_bytes", "max_manifest_bytes"):
with self.subTest(limit=override):
with self.assertRaisesRegex(
notebook_operations.InspectionError, "must be positive"
):
notebook_operations.inspect_container(archive, **{override: 0})
def test_a_member_count_over_the_limit_is_reported(self) -> None:
with tempfile.TemporaryDirectory() as directory:
archive = Path(directory) / "notebook.eln"
with zipfile.ZipFile(archive, "w") as handle:
for index in range(5):
handle.writestr(f"file{index}.txt", "x")
report = notebook_operations.inspect_container(archive, max_members=2)
self.assertTrue(report["errors"])
def test_a_missing_file_is_refused(self) -> None:
with tempfile.TemporaryDirectory() as directory:
with self.assertRaises((notebook_operations.InspectionError, OSError)):
notebook_operations.inspect_container(Path(directory) / "absent.eln")
def test_a_non_zip_file_is_refused_rather_than_crashing(self) -> None:
with tempfile.TemporaryDirectory() as directory:
archive = Path(directory) / "notebook.eln"
archive.write_bytes(b"not a zip at all")
with self.assertRaises((notebook_operations.InspectionError, zipfile.BadZipFile)):
notebook_operations.inspect_container(archive)
def test_the_manifest_name_is_the_documented_one(self) -> None:
self.assertEqual(notebook_operations.MANIFEST_NAME, "lamanifest.xml")
class XmlHelperTests(unittest.TestCase):
def test_namespaced_tags_are_reduced_to_their_local_name(self) -> None:
self.assertEqual(notebook_operations._local_name("{urn:x}entry"), "entry")
self.assertEqual(notebook_operations._local_name("entry"), "entry")
if __name__ == "__main__":
unittest.main()

265
tests/latchbio-integration/test_scripts.py Normal file
View File

@@ -0,0 +1,265 @@
"""Tests for the Latch SDK inspector.
`inspect_latch_sdk.py` reports on whichever Latch SDK happens to be installed,
so almost none of it can be asserted against a fixed expectation. What *can* be
pinned is everything around that: the report shape, the exit-code contract, the
address scrubbing that makes two runs comparable, and the graceful degradation
when a module is missing -- which is the normal case in this repo's project
environment, where `latch` is not installed.
`inspect_symbol` is exercised against real standard-library modules rather than
mocks: `json.dumps` is a function that exists, `json.nope` is a symbol that does
not, and `no_such_module` is an import that fails. That covers all three
branches without needing the SDK.
"""
from __future__ import annotations
import json
import subprocess
import sys
import unittest
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "latchbio-integration"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import inspect_latch_sdk # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
class NormalizeReprTests(unittest.TestCase):
"""Two runs of the inspector must be diffable, so addresses are scrubbed."""
def test_object_repr_loses_its_address(self) -> None:
self.assertEqual(
inspect_latch_sdk.normalize_repr("<latch.types.LatchFile object at 0x7f9c8a1b2c40>"),
"<latch.types.LatchFile>",
)
def test_bare_addresses_are_replaced_everywhere(self) -> None:
self.assertEqual(
inspect_latch_sdk.normalize_repr("default=0xdeadBEEF, other=0x01"),
"default=0x<address>, other=0x<address>",
)
def test_text_without_an_address_is_untouched(self) -> None:
for value in ("(x: int) -> str", "", "no hex here"):
with self.subTest(value=value):
self.assertEqual(inspect_latch_sdk.normalize_repr(value), value)
def test_scrubbing_is_idempotent(self) -> None:
once = inspect_latch_sdk.normalize_repr("<A object at 0x1234abcd>")
self.assertEqual(inspect_latch_sdk.normalize_repr(once), once)
class SafeSignatureTests(unittest.TestCase):
def test_a_signature_is_returned_for_an_ordinary_function(self) -> None:
signature = inspect_latch_sdk.safe_signature(json.dumps)
self.assertIsNotNone(signature)
self.assertTrue(signature.startswith("("))
def test_objects_without_a_signature_yield_none_rather_than_raising(self) -> None:
for obj in (42, object(), None):
with self.subTest(obj=type(obj).__name__):
self.assertIsNone(inspect_latch_sdk.safe_signature(obj))
class SymbolInspectionTests(unittest.TestCase):
def test_an_available_symbol_is_reported_with_its_kind_and_signature(self) -> None:
result = inspect_latch_sdk.inspect_symbol("json", "dumps")
self.assertEqual(result["qualified_name"], "json.dumps")
self.assertTrue(result["available"])
self.assertEqual(result["kind"], "function")
self.assertIn("obj", result["signature"])
self.assertNotIn("error", result)
def test_a_missing_symbol_is_reported_without_raising(self) -> None:
result = inspect_latch_sdk.inspect_symbol("json", "definitely_not_here")
self.assertFalse(result["available"])
self.assertEqual(result["error"], "symbol not found")
def test_an_unimportable_module_becomes_the_diagnostic(self) -> None:
result = inspect_latch_sdk.inspect_symbol("latch_module_that_is_absent", "x")
self.assertFalse(result["available"])
self.assertIn("ModuleNotFoundError", result["error"])
self.assertEqual(
result["qualified_name"], "latch_module_that_is_absent.x"
)
def test_plain_symbols_carry_no_workflow_interface(self) -> None:
# `python_interface` only appears for Latch workflow objects; it must
# not be invented for ordinary callables.
self.assertNotIn(
"python_interface", inspect_latch_sdk.inspect_symbol("json", "dumps")
)
class MethodInspectionTests(unittest.TestCase):
def test_present_and_absent_methods_are_distinguished(self) -> None:
result = inspect_latch_sdk.inspect_methods(
"pathlib", "Path", ["exists", "iterdir", "not_a_method"]
)
self.assertTrue(result["exists"]["available"])
self.assertTrue(result["iterdir"]["available"])
self.assertFalse(result["not_a_method"]["available"])
self.assertIsNone(result["not_a_method"]["signature"])
def test_an_unimportable_class_reports_one_error_not_per_method(self) -> None:
result = inspect_latch_sdk.inspect_methods("absent_module", "Thing", ["a", "b"])
self.assertEqual(set(result), {"error"})
self.assertIn("ModuleNotFoundError", result["error"])
class CatalogueTests(unittest.TestCase):
def test_every_symbol_group_is_non_empty_and_latch_scoped(self) -> None:
self.assertTrue(inspect_latch_sdk.SYMBOL_GROUPS)
for group, entries in inspect_latch_sdk.SYMBOL_GROUPS.items():
with self.subTest(group=group):
self.assertTrue(entries)
for module_name, symbol_name in entries:
self.assertTrue(module_name.startswith("latch"))
self.assertTrue(symbol_name)
def test_no_symbol_is_catalogued_twice(self) -> None:
qualified = [
f"{module}.{symbol}"
for entries in inspect_latch_sdk.SYMBOL_GROUPS.values()
for module, symbol in entries
]
duplicates = sorted({name for name in qualified if qualified.count(name) > 1})
self.assertEqual(duplicates, [])
def test_method_targets_are_also_catalogued_as_symbols(self) -> None:
# A class whose methods are probed must itself be in SYMBOL_GROUPS, or
# the report cannot say whether a missing method means a missing class.
catalogued = {
f"{module}.{symbol}"
for entries in inspect_latch_sdk.SYMBOL_GROUPS.values()
for module, symbol in entries
}
for label, (module, class_name, methods) in inspect_latch_sdk.METHODS.items():
with self.subTest(target=label):
self.assertIn(f"{module}.{class_name}", catalogued)
self.assertTrue(methods)
class RequiredSymbolTests(unittest.TestCase):
def _report(self, availability: dict[str, bool]) -> dict:
return {
"symbols": {
"core": [
{"qualified_name": name, "available": available}
for name, available in availability.items()
]
}
}
REQUIRED = (
"latch.workflow",
"latch.resources.tasks.small_task",
"latch.resources.tasks.custom_task",
"latch.ldata.path.LPath",
"latch.registry.table.Table",
"latch_cli.services.launch.launch_v2.launch",
)
def test_a_fully_available_sdk_has_no_required_failures(self) -> None:
report = self._report({name: True for name in self.REQUIRED})
self.assertFalse(inspect_latch_sdk.has_required_failures(report))
def test_any_single_missing_core_symbol_is_a_failure(self) -> None:
for missing in self.REQUIRED:
with self.subTest(missing=missing):
availability = {name: True for name in self.REQUIRED}
availability[missing] = False
self.assertTrue(
inspect_latch_sdk.has_required_failures(self._report(availability))
)
def test_an_empty_report_counts_as_failing(self) -> None:
self.assertTrue(
inspect_latch_sdk.has_required_failures({"symbols": {}})
)
def test_every_required_symbol_is_one_the_script_catalogues(self) -> None:
catalogued = {
f"{module}.{symbol}"
for entries in inspect_latch_sdk.SYMBOL_GROUPS.values()
for module, symbol in entries
}
for name in self.REQUIRED:
with self.subTest(symbol=name):
self.assertIn(name, catalogued)
class ReportShapeTests(unittest.TestCase):
def test_the_report_describes_the_environment_and_every_group(self) -> None:
report = inspect_latch_sdk.build_report()
self.assertEqual(
set(report),
{"python", "platform", "latch_version", "symbols", "methods"},
)
self.assertEqual(set(report["symbols"]), set(inspect_latch_sdk.SYMBOL_GROUPS))
self.assertEqual(set(report["methods"]), set(inspect_latch_sdk.METHODS))
def test_the_report_is_json_serialisable(self) -> None:
# --json is the documented machine-readable path, so nothing in the
# report may be a non-serialisable introspection object.
json.dumps(inspect_latch_sdk.build_report())
def test_printing_a_report_without_the_sdk_does_not_raise(self) -> None:
report = inspect_latch_sdk.build_report()
import io
from contextlib import redirect_stdout
buffer = io.StringIO()
with redirect_stdout(buffer):
inspect_latch_sdk.print_text(report)
self.assertIn("Latch SDK:", buffer.getvalue())
class ExitCodeTests(unittest.TestCase):
def _run(self, *flags: str) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(SCRIPTS / "inspect_latch_sdk.py"), *flags],
capture_output=True,
text=True,
timeout=120,
)
def test_a_missing_sdk_exits_two_regardless_of_strict(self) -> None:
try:
import latch # noqa: F401
except ImportError:
pass
else:
self.skipTest("latch is installed; this asserts the not-installed path")
for flags in ((), ("--strict",), ("--json",)):
with self.subTest(flags=flags):
result = self._run(*flags)
self.assertEqual(result.returncode, 2, result.stderr)
self.assertNotIn("Traceback", result.stderr)
def test_json_output_parses_even_when_the_sdk_is_absent(self) -> None:
result = self._run("--json")
report = json.loads(result.stdout)
self.assertIn("symbols", report)
self.assertIn("latch_version", report)
def test_an_installed_sdk_exits_zero(self) -> None:
try:
import latch # noqa: F401
except ImportError:
self.skipTest("latch is not installed; run under --isolated")
result = self._run()
self.assertIn(result.returncode, (0, 1), result.stderr)
if __name__ == "__main__":
unittest.main()

108
tests/latex-posters/test_scripts.py Normal file
View File

@@ -0,0 +1,108 @@
"""Tests for the latex-posters helpers.
The two schematic scripts are byte-identical to the copies in
`scientific-schematics` and `literature-review`, so their behaviour comes from
the shared contract. What is specific here is `review_poster.sh`, the shell
helper that renders and inspects a compiled poster.
"""
from __future__ import annotations
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "latex-posters"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
SchematicTests = skill_contract.schematic.schematic_test_case(SKILL_ROOT)
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
REVIEW_SCRIPT = SCRIPTS / "review_poster.sh"
class ReviewScriptTests(unittest.TestCase):
def setUp(self) -> None:
self.text = REVIEW_SCRIPT.read_text(encoding="utf-8")
def test_it_is_executable_shell_with_a_shebang(self) -> None:
self.assertTrue(REVIEW_SCRIPT.is_file())
self.assertTrue(REVIEW_SCRIPT.stat().st_mode & 0o111, "not executable")
self.assertTrue(self.text.startswith("#!"), "no shebang")
def test_it_parses_as_bash(self) -> None:
syntax = subprocess.run(
["bash", "-n", str(REVIEW_SCRIPT)],
capture_output=True,
text=True,
timeout=30,
)
self.assertEqual(syntax.returncode, 0, syntax.stderr)
def _run(self, *args: str) -> subprocess.CompletedProcess:
return subprocess.run(
["bash", str(REVIEW_SCRIPT), *args],
capture_output=True,
text=True,
timeout=60,
cwd=SCRIPTS,
)
def test_no_argument_exits_non_zero_with_usage(self) -> None:
result = self._run()
self.assertEqual(result.returncode, 1)
self.assertIn("Usage:", result.stdout + result.stderr)
def test_a_missing_file_exits_non_zero_and_names_it(self) -> None:
result = self._run("no-such-poster.pdf")
self.assertEqual(result.returncode, 1)
self.assertIn("no-such-poster.pdf", result.stdout + result.stderr)
def test_a_missing_poppler_tool_degrades_instead_of_aborting(self) -> None:
# Deliberately no `set -e`: this is a report, and one unavailable
# inspector must not truncate the remaining checks or the manual
# checklist. Running with an empty PATH-ish environment proves it.
with tempfile.TemporaryDirectory() as directory:
poster = Path(directory) / "poster.pdf"
poster.write_bytes(b"%PDF-1.4\n%%EOF\n")
result = subprocess.run(
["bash", str(REVIEW_SCRIPT), str(poster)],
capture_output=True,
text=True,
timeout=60,
cwd=SCRIPTS,
)
self.assertEqual(result.returncode, 0)
output = result.stdout
# Every numbered section still runs, right through to the summary.
for section in ("[1]", "[2]", "[3]", "[4]", "[5]", "[6]", "[7]"):
with self.subTest(section=section):
self.assertIn(section, output)
self.assertIn("Quality Check Complete", output)
self.assertNotIn("Traceback", result.stderr)
def test_every_python_script_it_invokes_is_shipped(self) -> None:
for token in self.text.split():
if token.endswith(".py"):
with self.subTest(script=token):
self.assertTrue(
(SCRIPTS / Path(token).name).is_file(),
f"{token} is referenced but not shipped",
)
class AssetTests(unittest.TestCase):
def test_every_documented_asset_is_shipped(self) -> None:
# The skill's value is its templates; a SKILL.md that names one it does
# not ship sends the agent looking for a file that is not there.
problems = skill_contract.structure.link_problems(SKILL_ROOT)
self.assertEqual(problems, [])
if __name__ == "__main__":
unittest.main()

219
tests/liteparse/test_scripts.py Normal file
View File

@@ -0,0 +1,219 @@
"""Tests for the liteparse batch directory parser.
Parsing is liteparse's job; the script owns file discovery, serialisation, and
error containment. All three are testable with a stub parser -- and the third
matters most: a batch run over a hundred documents must not abort because one
of them is corrupt.
`parse_one` names its output from the source *stem*, so two sources whose stems
collide overwrite each other. That is pinned below as current behaviour rather
than asserted to be safe.
"""
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "liteparse"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pytest.importorskip("liteparse", reason="liteparse scripts import liteparse")
import batch_parse_dir # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
def text_item(text: str = "hello"):
return SimpleNamespace(
text=text,
x=1.0,
y=2.0,
width=3.0,
height=4.0,
font_name="Helvetica",
font_size=12.0,
confidence=0.99,
)
def parse_result(text: str = "hello"):
page = SimpleNamespace(
page_num=1, width=612.0, height=792.0, text=text, text_items=[text_item(text)]
)
return SimpleNamespace(text=text, pages=[page])
class StubParser:
"""Stands in for LiteParse; raises for any source named `broken*`."""
def __init__(self) -> None:
self.seen: list[Path] = []
def parse(self, path: Path):
self.seen.append(path)
if path.stem.startswith("broken"):
raise RuntimeError("unreadable document")
return parse_result(f"contents of {path.name}")
class ExtensionTests(unittest.TestCase):
def test_the_default_set_is_lowercase_and_dotted(self) -> None:
self.assertTrue(batch_parse_dir.DEFAULT_EXTENSIONS)
for extension in batch_parse_dir.DEFAULT_EXTENSIONS:
with self.subTest(extension=extension):
self.assertTrue(extension.startswith("."))
self.assertEqual(extension, extension.lower())
def test_the_documented_document_and_image_families_are_covered(self) -> None:
defaults = batch_parse_dir.DEFAULT_EXTENSIONS
for extension in (".pdf", ".docx", ".xlsx", ".pptx", ".csv", ".png", ".tiff"):
with self.subTest(extension=extension):
self.assertIn(extension, defaults)
class DiscoveryTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def touch(self, relative: str) -> Path:
path = self.root / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(b"x")
return path
def _found(self, **kwargs) -> list[str]:
kwargs.setdefault("recursive", False)
kwargs.setdefault("extension", None)
return [path.name for path in batch_parse_dir.iter_files(self.root, **kwargs)]
def test_only_supported_extensions_are_yielded(self) -> None:
self.touch("a.pdf")
self.touch("b.docx")
self.touch("c.exe")
self.assertEqual(self._found(), ["a.pdf", "b.docx"])
def test_matching_is_case_insensitive(self) -> None:
self.touch("SCAN.PDF")
self.assertEqual(self._found(), ["SCAN.PDF"])
def test_an_explicit_extension_narrows_the_selection(self) -> None:
self.touch("a.pdf")
self.touch("b.docx")
self.assertEqual(self._found(extension=".pdf"), ["a.pdf"])
def test_an_explicit_extension_is_matched_case_insensitively(self) -> None:
self.touch("a.PDF")
self.assertEqual(self._found(extension=".PdF"), ["a.PDF"])
def test_recursion_is_opt_in(self) -> None:
self.touch("top.pdf")
self.touch("nested/deep.pdf")
self.assertEqual(self._found(), ["top.pdf"])
self.assertEqual(sorted(self._found(recursive=True)), ["deep.pdf", "top.pdf"])
def test_directories_are_skipped(self) -> None:
(self.root / "folder.pdf").mkdir()
self.assertEqual(self._found(), [])
def test_the_order_is_deterministic(self) -> None:
for name in ("c.pdf", "a.pdf", "b.pdf"):
self.touch(name)
self.assertEqual(self._found(), ["a.pdf", "b.pdf", "c.pdf"])
class SerialisationTests(unittest.TestCase):
def test_a_result_becomes_a_json_serialisable_dictionary(self) -> None:
payload = batch_parse_dir._result_to_dict(parse_result("body text"))
json.dumps(payload)
self.assertEqual(payload["text"], "body text")
self.assertEqual(len(payload["pages"]), 1)
def test_page_geometry_and_text_items_survive(self) -> None:
page = batch_parse_dir._result_to_dict(parse_result())["pages"][0]
self.assertEqual(page["page_num"], 1)
self.assertEqual(page["width"], 612.0)
self.assertEqual(len(page["text_items"]), 1)
def test_a_text_item_keeps_its_position_font_and_confidence(self) -> None:
item = batch_parse_dir._text_item_dict(text_item("word"))
self.assertEqual(
set(item),
{"text", "x", "y", "width", "height", "font_name", "font_size", "confidence"},
)
self.assertEqual(item["text"], "word")
self.assertEqual(item["font_name"], "Helvetica")
self.assertEqual(item["confidence"], 0.99)
class ParseOneTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.output = self.root / "out"
self.output.mkdir()
self.parser = StubParser()
def test_text_output_writes_the_plain_text(self) -> None:
ok, source, message = batch_parse_dir.parse_one(
self.parser, Path("paper.pdf"), self.output, "text"
)
self.assertTrue(ok)
self.assertIn("paper.txt", message)
self.assertEqual(
(self.output / "paper.txt").read_text(encoding="utf-8"),
"contents of paper.pdf",
)
def test_json_output_writes_the_structured_result(self) -> None:
ok, _, message = batch_parse_dir.parse_one(
self.parser, Path("paper.pdf"), self.output, "json"
)
self.assertTrue(ok)
self.assertIn("paper.json", message)
payload = json.loads((self.output / "paper.json").read_text(encoding="utf-8"))
self.assertEqual(payload["text"], "contents of paper.pdf")
self.assertIn("pages", payload)
def test_a_failing_document_is_reported_not_raised(self) -> None:
# One corrupt file must not abort a hundred-document batch.
ok, source, message = batch_parse_dir.parse_one(
self.parser, Path("broken.pdf"), self.output, "text"
)
self.assertFalse(ok)
self.assertEqual(source, "broken.pdf")
self.assertIn("unreadable document", message)
self.assertEqual(list(self.output.iterdir()), [])
def test_the_output_name_comes_from_the_source_stem(self) -> None:
# Consequence: report.pdf and report.docx both write report.txt.
batch_parse_dir.parse_one(self.parser, Path("report.pdf"), self.output, "text")
batch_parse_dir.parse_one(self.parser, Path("report.docx"), self.output, "text")
self.assertEqual(
[path.name for path in self.output.iterdir()], ["report.txt"]
)
self.assertEqual(
(self.output / "report.txt").read_text(encoding="utf-8"),
"contents of report.docx",
)
def test_an_unknown_format_falls_back_to_text(self) -> None:
batch_parse_dir.parse_one(self.parser, Path("a.pdf"), self.output, "yaml")
self.assertTrue((self.output / "a.txt").is_file())
if __name__ == "__main__":
unittest.main()

267
tests/literature-review/test_scripts.py Normal file
View File

@@ -0,0 +1,267 @@
"""Tests for the literature-review result processing.
`search_databases` is pure list manipulation over records from several
databases, so it is fully testable offline -- and worth testing, because every
function here can lose papers silently. Deduplication that trusts a title
merges two distinct papers; a year filter that drops unparseable years quietly
shrinks the corpus; ranking that treats a missing citation count as zero
buries new work.
The schematic scripts this skill also ships are covered by the shared
contract, since `scientific-schematics` and `latex-posters` ship identical
copies.
"""
from __future__ import annotations
import json
import sys
import unittest
from pathlib import Path
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "literature-review"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
import search_databases # noqa: E402
SchematicTests = skill_contract.schematic.schematic_test_case(SKILL_ROOT)
RESULTS = [
{
"title": "Deep Learning for Protein Folding",
"authors": "Jumper, J. and Evans, R.",
"first_author": "Jumper",
"year": "2021",
"source": "PubMed",
"doi": "10.1038/s41586-021-03819-2",
"citations": 15000,
"journal": "Nature",
},
{
"title": "Attention Is All You Need",
"authors": "Vaswani, A.",
"first_author": "Vaswani",
"year": "2017",
"source": "arXiv",
"citations": 90000,
},
{
"title": "A Recent Preprint",
"authors": "Smith, J.",
"first_author": "Smith",
"year": "2026",
"source": "bioRxiv",
},
]
class DeduplicationTests(unittest.TestCase):
def test_records_sharing_a_doi_collapse_to_one(self) -> None:
duplicated = [
{"title": "One Version", "doi": "10.1000/x"},
{"title": "Another Title Entirely", "doi": "10.1000/X"},
]
self.assertEqual(len(search_databases.deduplicate_results(duplicated)), 1)
def test_doi_comparison_ignores_case_and_surrounding_space(self) -> None:
duplicated = [
{"title": "a", "doi": " 10.1000/X "},
{"title": "b", "doi": "10.1000/x"},
]
self.assertEqual(len(search_databases.deduplicate_results(duplicated)), 1)
def test_titles_only_deduplicate_records_that_have_no_doi(self) -> None:
# Two records with the same title but different DOIs are two papers.
distinct = [
{"title": "Same Title", "doi": "10.1000/a"},
{"title": "Same Title", "doi": "10.1000/b"},
]
self.assertEqual(len(search_databases.deduplicate_results(distinct)), 2)
same = [{"title": "Same Title"}, {"title": "same title"}]
self.assertEqual(len(search_databases.deduplicate_results(same)), 1)
def test_the_first_occurrence_is_the_one_kept(self) -> None:
records = [
{"title": "first", "doi": "10.1000/x", "source": "PubMed"},
{"title": "second", "doi": "10.1000/x", "source": "arXiv"},
]
self.assertEqual(
search_databases.deduplicate_results(records)[0]["source"], "PubMed"
)
def test_an_empty_list_deduplicates_to_an_empty_list(self) -> None:
self.assertEqual(search_databases.deduplicate_results([]), [])
def test_records_with_neither_doi_nor_title_are_all_kept(self) -> None:
records = [{"source": "a"}, {"source": "b"}]
self.assertEqual(len(search_databases.deduplicate_results(records)), 2)
class RankingTests(unittest.TestCase):
def test_citation_ranking_puts_the_most_cited_first(self) -> None:
ranked = search_databases.rank_results(RESULTS, "citations")
self.assertEqual(ranked[0]["title"], "Attention Is All You Need")
self.assertEqual(ranked[-1]["title"], "A Recent Preprint")
def test_year_ranking_puts_the_newest_first(self) -> None:
ranked = search_databases.rank_results(RESULTS, "year")
self.assertEqual(ranked[0]["year"], "2026")
self.assertEqual(ranked[-1]["year"], "2017")
def test_relevance_ranking_uses_the_relevance_score(self) -> None:
scored = [
{"title": "low", "relevance_score": 0.1},
{"title": "high", "relevance_score": 0.9},
]
self.assertEqual(
search_databases.rank_results(scored, "relevance")[0]["title"], "high"
)
def test_an_unknown_criterion_leaves_the_order_untouched(self) -> None:
self.assertEqual(
[r["title"] for r in search_databases.rank_results(RESULTS, "nonsense")],
[r["title"] for r in RESULTS],
)
def test_ranking_does_not_mutate_the_input(self) -> None:
original = [dict(record) for record in RESULTS]
search_databases.rank_results(RESULTS, "citations")
self.assertEqual(RESULTS, original)
def test_a_missing_citation_count_sorts_last_rather_than_erroring(self) -> None:
ranked = search_databases.rank_results(RESULTS, "citations")
self.assertNotIn("citations", ranked[-1])
class YearFilterTests(unittest.TestCase):
def test_both_bounds_are_inclusive(self) -> None:
filtered = search_databases.filter_by_year(RESULTS, 2017, 2021)
self.assertEqual(len(filtered), 2)
self.assertEqual({r["year"] for r in filtered}, {"2017", "2021"})
def test_each_bound_can_be_used_alone(self) -> None:
self.assertEqual(len(search_databases.filter_by_year(RESULTS, start_year=2021)), 2)
self.assertEqual(len(search_databases.filter_by_year(RESULTS, end_year=2017)), 1)
def test_no_bounds_keeps_everything(self) -> None:
self.assertEqual(len(search_databases.filter_by_year(RESULTS)), len(RESULTS))
def test_an_unparseable_year_is_kept_rather_than_silently_dropped(self) -> None:
# Losing a paper because a database returned "in press" would be worse
# than including one outside the range.
records = [{"title": "x", "year": "in press"}, {"title": "y", "year": None}]
self.assertEqual(len(search_databases.filter_by_year(records, 2020, 2026)), 2)
def test_filtering_does_not_mutate_the_input(self) -> None:
original = [dict(record) for record in RESULTS]
search_databases.filter_by_year(RESULTS, 2020, 2022)
self.assertEqual(RESULTS, original)
class SummaryTests(unittest.TestCase):
def test_the_summary_counts_records_sources_and_years(self) -> None:
summary = search_databases.generate_search_summary(RESULTS)
self.assertEqual(summary["total_results"], 3)
self.assertEqual(
summary["sources"], {"PubMed": 1, "arXiv": 1, "bioRxiv": 1}
)
self.assertEqual(summary["year_distribution"]["2021"], 1)
def test_citation_statistics_ignore_records_without_a_count(self) -> None:
summary = search_databases.generate_search_summary(RESULTS)
self.assertEqual(summary["total_citations"], 105000)
self.assertEqual(summary["avg_citations"], 105000 / 2)
def test_an_empty_corpus_summarises_to_zeroes_not_an_error(self) -> None:
summary = search_databases.generate_search_summary([])
self.assertEqual(summary["total_results"], 0)
self.assertEqual(summary["avg_citations"], 0)
self.assertEqual(summary["sources"], {})
def test_a_non_numeric_citation_count_is_skipped(self) -> None:
summary = search_databases.generate_search_summary(
[{"citations": "many"}, {"citations": 10}]
)
self.assertEqual(summary["total_citations"], 10)
def test_records_with_no_source_are_bucketed_as_unknown(self) -> None:
summary = search_databases.generate_search_summary([{"title": "x"}])
self.assertEqual(summary["sources"], {"Unknown": 1})
class FormattingTests(unittest.TestCase):
def test_json_output_round_trips(self) -> None:
rendered = search_databases.format_search_results(RESULTS, "json")
self.assertEqual(json.loads(rendered), RESULTS)
def test_markdown_lists_every_record_with_a_resolvable_doi_link(self) -> None:
rendered = search_databases.format_search_results(RESULTS, "markdown")
for record in RESULTS:
with self.subTest(title=record["title"]):
self.assertIn(record["title"], rendered)
self.assertIn("https://doi.org/10.1038/s41586-021-03819-2", rendered)
self.assertIn("**Total Results**: 3", rendered)
def test_markdown_substitutes_placeholders_for_absent_fields(self) -> None:
rendered = search_databases.format_search_results([{}], "markdown")
self.assertIn("Untitled", rendered)
self.assertIn("Unknown", rendered)
self.assertIn("N/A", rendered)
def test_bibtex_entries_are_balanced_and_keyed_by_author_and_year(self) -> None:
rendered = search_databases.format_search_results(RESULTS, "bibtex")
self.assertIn("@article{Jumper2021,", rendered)
self.assertIn("journal = {Nature},", rendered)
self.assertEqual(rendered.count("@"), 3)
self.assertEqual(rendered.count("{"), rendered.count("}"))
def test_bibtex_omits_fields_the_record_does_not_have(self) -> None:
rendered = search_databases.format_search_results([RESULTS[1]], "bibtex")
self.assertNotIn("journal =", rendered)
self.assertNotIn("volume =", rendered)
def test_bibtex_falls_back_to_a_placeholder_key(self) -> None:
rendered = search_databases.format_search_results([{}], "bibtex")
self.assertIn("@article{unknown0000,", rendered)
def test_an_unknown_format_is_refused_by_name(self) -> None:
with self.assertRaisesRegex(ValueError, "Unknown format: ris"):
search_databases.format_search_results(RESULTS, "ris")
def test_an_empty_corpus_formats_without_error(self) -> None:
for output_format in ("json", "markdown", "bibtex"):
with self.subTest(output_format=output_format):
rendered = search_databases.format_search_results([], output_format)
self.assertIsInstance(rendered, str)
class PipelineTests(unittest.TestCase):
def test_dedupe_filter_and_rank_compose_into_a_stable_corpus(self) -> None:
raw = RESULTS + [dict(RESULTS[0])] # the same paper found twice
processed = search_databases.rank_results(
search_databases.filter_by_year(
search_databases.deduplicate_results(raw), 2017, 2026
),
"citations",
)
titles = [record["title"] for record in processed]
self.assertEqual(len(titles), 3)
self.assertEqual(titles[0], "Attention Is All You Need")
class DependencyCheckTests(unittest.TestCase):
def test_the_pdf_generator_reports_missing_tooling_rather_than_crashing(self) -> None:
import generate_pdf
# check_dependencies inspects the host for pandoc/LaTeX; whatever it
# finds, it must answer rather than raise.
result = generate_pdf.check_dependencies()
self.assertIsNotNone(result)
if __name__ == "__main__":
unittest.main()

7
tests/market-research-reports/test_scripts.py
View File

@@ -36,6 +36,8 @@ from validate_competitor_matrix import ( # noqa: E402
)
from validate_evidence_ledger import _load_records, validate_records # noqa: E402
import skill_contract
def load_json(name: str) -> dict:
with (ASSETS / name).open("r", encoding="utf-8") as handle:
@@ -184,5 +186,10 @@ class ScaffoldTests(unittest.TestCase):
generate(manifest, output)
# The shared --help contract: every argparse CLI this skill ships answers --help
# without doing any work. It skips when the skill's packages are absent and runs
# for real under `python tests/run_all.py --isolated`.
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
if __name__ == "__main__":
unittest.main()

279
tests/markitdown/test_scripts.py Normal file
View File

@@ -0,0 +1,279 @@
"""Tests for the MarkItDown batch converters.
Conversion itself is MarkItDown's job. What these scripts own is everything
around it, and each piece has a failure mode worth guarding:
* `discover_files` decides what gets converted -- and must not silently pick up
audio and video, because with `markitdown[all]` those are transcribed via
Google Web Speech, sending local files off the machine.
* `output_path_for` preserves the source suffix in the output name, so
`report.pdf` and `report.docx` do not collide on `report.md`.
* `atomic_write_text` replaces files through a temporary, so an interrupted
run never leaves a half-written markdown file.
* `infer_metadata` reads `Author_Year_Title.pdf`, and a loose pattern would
attribute papers to the wrong author.
"""
from __future__ import annotations
import hashlib
import json
import sys
import tempfile
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "markitdown"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pytest.importorskip("markitdown", reason="markitdown scripts import markitdown")
import batch_convert # noqa: E402
import convert_literature # noqa: E402
import inspect_installation # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
class ExtensionTests(unittest.TestCase):
def test_extensions_are_lowercased_dotted_and_deduplicated(self) -> None:
self.assertEqual(
batch_convert.normalize_extensions(["PDF", ".pdf", "docx", " .DOCX "]),
(".docx", ".pdf"),
)
def test_blank_entries_are_dropped(self) -> None:
self.assertEqual(batch_convert.normalize_extensions(["pdf", "", " "]), (".pdf",))
def test_an_empty_selection_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "At least one file extension"):
batch_convert.normalize_extensions([])
def test_a_lone_dot_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "cannot be only"):
batch_convert.normalize_extensions(["."])
def test_the_result_is_sorted_so_runs_are_reproducible(self) -> None:
self.assertEqual(
batch_convert.normalize_extensions(["xlsx", "csv", "pdf"]),
(".csv", ".pdf", ".xlsx"),
)
def test_the_defaults_carry_no_audio_or_video_formats(self) -> None:
# Converting those with markitdown[all] would ship the file to Google
# Web Speech; they must be opt-in, never a default.
overlap = set(batch_convert.DEFAULT_EXTENSIONS) & batch_convert.EXTERNAL_SERVICE_EXTENSIONS
self.assertEqual(overlap, set())
def test_the_defaults_are_sorted_lowercase_and_dotted(self) -> None:
defaults = batch_convert.DEFAULT_EXTENSIONS
self.assertEqual(list(defaults), sorted(defaults))
for extension in defaults:
with self.subTest(extension=extension):
self.assertTrue(extension.startswith("."))
self.assertEqual(extension, extension.lower())
class DiscoveryTestCase(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
def touch(self, relative: str) -> Path:
path = self.root / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(b"x")
return path
class DiscoveryTests(DiscoveryTestCase):
def test_only_the_requested_extensions_are_returned(self) -> None:
self.touch("a.pdf")
self.touch("b.docx")
self.touch("c.mp3")
found = batch_convert.discover_files(self.root, (".pdf", ".docx"), recursive=False)
self.assertEqual([path.name for path in found], ["a.pdf", "b.docx"])
def test_matching_is_case_insensitive(self) -> None:
self.touch("SCAN.PDF")
found = batch_convert.discover_files(self.root, (".pdf",), recursive=False)
self.assertEqual([path.name for path in found], ["SCAN.PDF"])
def test_recursion_is_opt_in(self) -> None:
self.touch("top.pdf")
self.touch("nested/deep.pdf")
shallow = batch_convert.discover_files(self.root, (".pdf",), recursive=False)
self.assertEqual([path.name for path in shallow], ["top.pdf"])
deep = batch_convert.discover_files(self.root, (".pdf",), recursive=True)
self.assertEqual(sorted(path.name for path in deep), ["deep.pdf", "top.pdf"])
def test_directories_are_never_returned(self) -> None:
(self.root / "folder.pdf").mkdir()
self.assertEqual(
batch_convert.discover_files(self.root, (".pdf",), recursive=False), []
)
def test_the_order_is_deterministic(self) -> None:
for name in ("c.pdf", "a.pdf", "b.pdf"):
self.touch(name)
found = batch_convert.discover_files(self.root, (".pdf",), recursive=False)
self.assertEqual([path.name for path in found], ["a.pdf", "b.pdf", "c.pdf"])
def test_an_empty_directory_yields_nothing(self) -> None:
self.assertEqual(
batch_convert.discover_files(self.root, (".pdf",), recursive=True), []
)
class OutputPathTests(unittest.TestCase):
def test_the_source_suffix_is_kept_in_the_output_name(self) -> None:
# Without this, report.pdf and report.docx both become report.md and
# the second silently overwrites the first.
pdf = batch_convert.output_path_for(
Path("/in/report.pdf"), Path("/in"), Path("/out")
)
docx = batch_convert.output_path_for(
Path("/in/report.docx"), Path("/in"), Path("/out")
)
self.assertEqual(pdf, Path("/out/report.pdf.md"))
self.assertEqual(docx, Path("/out/report.docx.md"))
self.assertNotEqual(pdf, docx)
def test_the_directory_layout_is_mirrored(self) -> None:
self.assertEqual(
batch_convert.output_path_for(
Path("/in/2024/papers/a.pdf"), Path("/in"), Path("/out")
),
Path("/out/2024/papers/a.pdf.md"),
)
class AtomicWriteTests(DiscoveryTestCase):
def test_the_content_lands_at_the_destination(self) -> None:
path = self.root / "nested" / "out.md"
batch_convert.atomic_write_text(path, "# Title\n")
self.assertEqual(path.read_text(encoding="utf-8"), "# Title\n")
def test_no_temporary_file_survives(self) -> None:
path = self.root / "out.md"
batch_convert.atomic_write_text(path, "content")
self.assertEqual([entry.name for entry in self.root.iterdir()], ["out.md"])
def test_an_existing_file_is_replaced_wholesale(self) -> None:
path = self.root / "out.md"
batch_convert.atomic_write_text(path, "a much longer original document")
batch_convert.atomic_write_text(path, "short")
self.assertEqual(path.read_text(encoding="utf-8"), "short")
def test_non_ascii_content_survives_the_round_trip(self) -> None:
path = self.root / "out.md"
batch_convert.atomic_write_text(path, "Grüße — 日本語\n")
self.assertEqual(path.read_text(encoding="utf-8"), "Grüße — 日本語\n")
class MetadataInferenceTests(unittest.TestCase):
def test_an_author_year_title_filename_is_parsed(self) -> None:
author, year, title = convert_literature.infer_metadata(
Path("Jumper_2021_Highly_accurate_protein_structure.pdf")
)
self.assertEqual(author, "Jumper")
self.assertEqual(year, "2021")
self.assertEqual(title, "Highly accurate protein structure")
def test_a_multi_word_author_is_kept_whole(self) -> None:
author, year, _ = convert_literature.infer_metadata(
Path("Van_Der_Berg_1998_A_study.pdf")
)
self.assertEqual(year, "1998")
self.assertEqual(author, "Van Der Berg")
def test_only_plausible_years_are_accepted(self) -> None:
# 1899 and 2101 are not publication years the pattern should trust.
for stem in ("Smith_1899_Old.pdf", "Smith_2101_Future.pdf", "Smith_21_Short.pdf"):
with self.subTest(stem=stem):
author, year, title = convert_literature.infer_metadata(Path(stem))
self.assertIsNone(author)
self.assertIsNone(year)
self.assertTrue(title)
def test_an_unstructured_filename_still_yields_a_readable_title(self) -> None:
author, year, title = convert_literature.infer_metadata(
Path("some_random_notes.pdf")
)
self.assertIsNone(author)
self.assertIsNone(year)
self.assertEqual(title, "some random notes")
def test_runs_of_whitespace_are_collapsed(self) -> None:
self.assertEqual(
convert_literature.humanize_filename_component("a__b___c"), "a b c"
)
self.assertEqual(convert_literature.humanize_filename_component(" x "), "x")
class DigestTests(DiscoveryTestCase):
def test_the_digest_matches_hashlib(self) -> None:
path = self.root / "paper.pdf"
payload = b"%PDF-1.7\n" + b"x" * 5000
path.write_bytes(payload)
self.assertEqual(
convert_literature.digest_file(path), hashlib.sha256(payload).hexdigest()
)
def test_an_empty_file_hashes_to_the_empty_digest(self) -> None:
path = self.root / "empty.pdf"
path.write_bytes(b"")
self.assertEqual(
convert_literature.digest_file(path), hashlib.sha256(b"").hexdigest()
)
def test_a_file_larger_than_one_chunk_is_hashed_whole(self) -> None:
# The reader streams in 1 MiB chunks; a bug there would hash a prefix.
path = self.root / "large.pdf"
payload = bytes(range(256)) * 8192 # 2 MiB
path.write_bytes(payload)
self.assertEqual(
convert_literature.digest_file(path), hashlib.sha256(payload).hexdigest()
)
class YamlScalarTests(unittest.TestCase):
def test_ordinary_text_is_quoted(self) -> None:
self.assertEqual(convert_literature.yaml_scalar("A Title"), '"A Title"')
def test_quotes_and_colons_are_escaped_safely(self) -> None:
# An unquoted colon would end the YAML key and corrupt the front matter.
rendered = convert_literature.yaml_scalar('He said: "hi"')
self.assertEqual(json.loads(rendered), 'He said: "hi"')
def test_non_ascii_is_preserved_rather_than_escaped(self) -> None:
self.assertEqual(convert_literature.yaml_scalar("Grüße"), '"Grüße"')
class InstallationReportTests(unittest.TestCase):
def test_the_report_names_the_target_version_and_optional_extras(self) -> None:
self.assertRegex(inspect_installation.TARGET_VERSION, r"^\d+\.\d+\.\d+$")
self.assertTrue(inspect_installation.OPTIONAL_DISTRIBUTIONS)
def test_the_report_is_json_serialisable(self) -> None:
json.dumps(inspect_installation.inspect_installation())
def test_an_absent_distribution_reports_none_rather_than_raising(self) -> None:
self.assertIsNone(
inspect_installation.distribution_version("definitely-not-installed-xyz")
)
def test_an_installed_distribution_reports_its_version(self) -> None:
self.assertIsNotNone(inspect_installation.distribution_version("markitdown"))
if __name__ == "__main__":
unittest.main()

763
tests/matchms/test_scripts.py Normal file
View File

@@ -0,0 +1,763 @@
"""Tests for the matchms library-search helper.
`library_search.py` is a thin CLI over matchms, and almost everything that can
go wrong in it happens before or after the similarity calculation: the argument
validator that refuses pickle inputs and out-of-range thresholds, the
metric-name-to-class table that must cover every `--metric` choice the parser
advertises, the score-record reader that has to cope with both plain floats and
matchms' structured `(score, matches)` records, and `write_hits`, where the
top-k / min-score / min-matches decisions are actually made.
Those groups are driven directly. The metric table and the score-record reader
guard the two silent-wrong-answer failure modes -- a `--metric` the parser
accepts but `create_metric` cannot build, and a structured record read as a
scalar so `matched_peaks` is dropped. The end-to-end test then searches a
spectrum against a library containing a copy of itself, where cosine similarity
is exactly 1.0 by construction, so it pins the whole pipeline to a value that
is right independently of this code.
"""
from __future__ import annotations
import argparse
import csv
import sys
import tempfile
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "matchms"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
pytest.importorskip("matchms", reason="matchms skill needs matchms")
numpy = pytest.importorskip("numpy", reason="matchms skill needs numpy")
from matchms import similarity as matchms_similarity # noqa: E402
from matchms.similarity.BaseSimilarity import BaseSimilarity # noqa: E402
import library_search # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
#: A five-peak MGF spectrum. Five peaks is the minimum `--min-peaks` default
#: accepts, and the intensities stay above the 1% relative cutoff.
QUERY_MGF = """BEGIN IONS
TITLE=alpha
PEPMASS=350.1
CHARGE=1+
100.0000 100.0
150.0000 50.0
200.0000 30.0
250.0000 20.0
300.0000 10.0
END IONS
"""
#: Two references: a byte-for-byte copy of the query, and a spectrum sharing no
#: fragment with it.
LIBRARY_MGF = """BEGIN IONS
TITLE=alpha_copy
PEPMASS=350.1
CHARGE=1+
100.0000 100.0
150.0000 50.0
200.0000 30.0
250.0000 20.0
300.0000 10.0
END IONS
BEGIN IONS
TITLE=alpha_twin
PEPMASS=350.1
CHARGE=1+
100.0000 100.0
150.0000 50.0
200.0000 30.0
250.0000 20.0
300.0000 10.0
END IONS
BEGIN IONS
TITLE=unrelated
PEPMASS=500.2
CHARGE=1+
111.0000 100.0
161.0000 50.0
211.0000 30.0
261.0000 20.0
311.0000 10.0
END IONS
"""
def metric_choices() -> list[str]:
"""The `--metric` values the parser advertises.
argparse exposes no public accessor for a choice list, so this reads the
action table; the alternative is duplicating the list here, which is
exactly the drift the tests below exist to catch.
"""
for action in library_search.build_parser()._actions:
if action.dest == "metric":
return list(action.choices)
raise AssertionError("the parser no longer defines --metric")
def score_record(score: float, matches: int):
"""One matchms-style structured score record."""
dtype = [("Fake_score", "<f8"), ("Fake_matches", "<i8")]
return numpy.array([(score, matches)], dtype=dtype)[0]
class FakeSpectrum:
"""Stands in for a matchms Spectrum: metadata lookup and nothing else."""
def __init__(self, **metadata) -> None:
self._metadata = metadata
def get(self, key):
return self._metadata.get(key)
class FakeScores:
"""Stands in for matchms Scores with a fixed, hand-chosen ranking."""
score_names = ("Fake_score", "Fake_matches")
def __init__(self, ranked) -> None:
self.ranked = ranked
self.requested_name = None
self.requested_sort = None
def scores_by_query(self, query, name, sort):
self.requested_name = name
self.requested_sort = sort
return self.ranked
def search_namespace(**overrides) -> argparse.Namespace:
"""A namespace with every default `build_parser` would produce."""
defaults = dict(
queries=None,
references=None,
output=None,
metric="modified",
tolerance=0.02,
top_k=10,
min_score=0.0,
min_matches=0,
array_type="numpy",
max_pairs=5_000_000,
relative_intensity=0.01,
min_peaks=5,
max_peaks=None,
mz_min=None,
mz_max=None,
remove_precursor_window=None,
no_default_filters=False,
no_normalize=False,
query_id_field=None,
reference_id_field=None,
bin_width=0.001,
blink_top_k=None,
force=False,
quiet=True,
)
defaults.update(overrides)
return argparse.Namespace(**defaults)
class MetricTableTests(unittest.TestCase):
"""`create_metric` must cover every metric the parser accepts."""
def test_every_advertised_metric_builds_a_similarity_object(self) -> None:
# A choice the parser accepts but create_metric cannot build would only
# fail after both spectrum files have been loaded and processed.
for name in metric_choices():
with self.subTest(metric=name):
metric = library_search.create_metric(search_namespace(metric=name))
self.assertIsInstance(metric, BaseSimilarity)
def test_each_metric_name_selects_the_documented_matchms_class(self) -> None:
expected = {
"cosine": matchms_similarity.CosineGreedy,
"cosine-exact": matchms_similarity.CosineHungarian,
"cosine-linear": matchms_similarity.CosineLinear,
"modified": matchms_similarity.ModifiedCosineGreedy,
"modified-exact": matchms_similarity.ModifiedCosineHungarian,
"neutral-loss": matchms_similarity.NeutralLossesCosine,
"blink": matchms_similarity.BlinkCosine,
}
# Every non-flash choice is covered, so a new metric cannot be added to
# the parser without landing in this table.
self.assertEqual(
set(expected) | {"flash-entropy", "flash-cosine", "flash-modified"},
set(metric_choices()),
)
for name, cls in expected.items():
with self.subTest(metric=name):
self.assertIsInstance(
library_search.create_metric(search_namespace(metric=name)), cls
)
def test_the_three_flash_metrics_differ_in_score_type_and_matching_mode(self) -> None:
# All three build a FlashSimilarity; the distinction is entirely in the
# constructor arguments, so a copy-paste slip would silently give the
# user the wrong similarity.
configurations = {
name: library_search.create_metric(search_namespace(metric=name))
for name in ("flash-entropy", "flash-cosine", "flash-modified")
}
self.assertEqual(
{
name: (metric.score_type, metric.matching_mode)
for name, metric in configurations.items()
},
{
"flash-entropy": ("spectral_entropy", "fragment"),
"flash-cosine": ("cosine", "fragment"),
# "modified" cosine means hybrid (precursor-shifted) matching.
"flash-modified": ("cosine", "hybrid"),
},
)
def test_an_unknown_metric_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "unsupported metric"):
library_search.create_metric(search_namespace(metric="tanimoto"))
def test_blink_receives_the_intensity_and_score_cutoffs(self) -> None:
metric = library_search.create_metric(
search_namespace(
metric="blink", relative_intensity=0.02, blink_top_k=7, min_score=0.3
)
)
self.assertEqual(metric.min_relative_intensity, 0.02)
self.assertEqual(metric.top_k, 7)
# BLINK prunes sparse scores itself; passing --min-score through avoids
# materialising pairs the writer would drop anyway.
self.assertEqual(metric.sparse_score_min, 0.3)
def test_the_metric_sets_only_name_metrics_the_parser_offers(self) -> None:
advertised = set(metric_choices())
self.assertTrue(library_search.STRUCTURED_METRICS <= advertised)
self.assertTrue(library_search.PRECURSOR_METRICS <= advertised)
# Flash metrics report a score but no matched-peak count.
self.assertEqual(
library_search.STRUCTURED_METRICS
& {"flash-entropy", "flash-cosine", "flash-modified"},
set(),
)
class SuffixPolicyTests(unittest.TestCase):
def test_pickle_suffixes_are_never_also_supported(self) -> None:
# Refusing pickles is a security decision; the two sets overlapping
# would quietly re-enable arbitrary code execution on load.
self.assertEqual(
library_search.UNSAFE_PICKLE_SUFFIXES & library_search.SUPPORTED_SUFFIXES,
set(),
)
def test_every_suffix_is_lowercase_and_dotted(self) -> None:
# validate_args lowercases the suffix before the lookup, so an entry
# spelled ".MGF" here would never match.
for suffix in library_search.SUPPORTED_SUFFIXES | library_search.UNSAFE_PICKLE_SUFFIXES:
with self.subTest(suffix=suffix):
self.assertEqual(suffix, suffix.lower())
self.assertTrue(suffix.startswith("."))
class ValidationTests(unittest.TestCase):
"""Both directions: a usable invocation passes, a broken one is named."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.queries = self.root / "queries.mgf"
self.references = self.root / "library.msp"
for path in (self.queries, self.references):
path.write_text("", encoding="utf-8")
self.output = self.root / "hits.csv"
def args(self, **overrides) -> argparse.Namespace:
settings = dict(
queries=self.queries, references=self.references, output=self.output
)
settings.update(overrides)
return search_namespace(**settings)
def test_a_valid_invocation_passes_silently(self) -> None:
# Without this the rest of the class only proves the validator can say
# no -- not that it ever says yes.
library_search.validate_args(self.args())
def test_every_supported_suffix_is_accepted_in_both_positions(self) -> None:
for suffix in sorted(library_search.SUPPORTED_SUFFIXES):
with self.subTest(suffix=suffix):
queries = self.root / f"q{suffix}"
queries.write_text("", encoding="utf-8")
library_search.validate_args(self.args(queries=queries))
def test_uppercase_suffixes_are_accepted(self) -> None:
queries = self.root / "queries.MGF"
queries.write_text("", encoding="utf-8")
library_search.validate_args(self.args(queries=queries))
def test_a_missing_input_is_reported_by_role(self) -> None:
for role, key in (("query", "queries"), ("reference", "references")):
with self.subTest(role=role):
with self.assertRaisesRegex(ValueError, f"{role} file does not exist"):
library_search.validate_args(
self.args(**{key: self.root / "absent.mgf"})
)
def test_pickle_input_is_refused_with_the_reason(self) -> None:
for name in ("library.pickle", "library.pkl"):
with self.subTest(name=name):
path = self.root / name
path.write_bytes(b"")
with self.assertRaisesRegex(ValueError, "unpickling can execute code"):
library_search.validate_args(self.args(references=path))
def test_an_unsupported_suffix_lists_the_formats(self) -> None:
path = self.root / "library.csv"
path.write_text("", encoding="utf-8")
with self.assertRaisesRegex(ValueError, "MGF, MSP, mzML, mzXML, or JSON"):
library_search.validate_args(self.args(references=path))
def test_an_existing_output_needs_force(self) -> None:
self.output.write_text("", encoding="utf-8")
with self.assertRaisesRegex(ValueError, "--force"):
library_search.validate_args(self.args())
# With --force the same invocation is fine.
library_search.validate_args(self.args(force=True))
def test_a_missing_output_directory_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "output directory does not exist"):
library_search.validate_args(self.args(output=self.root / "no" / "hits.csv"))
def test_numeric_bounds_are_enforced_at_the_boundary(self) -> None:
cases = [
({"tolerance": 0.0}, "--tolerance must be positive"),
({"tolerance": -0.01}, "--tolerance must be positive"),
({"top_k": 0}, "--top-k must be positive"),
({"min_score": -0.001}, "--min-score must be between 0 and 1"),
({"min_score": 1.001}, "--min-score must be between 0 and 1"),
({"min_matches": -1}, "--min-matches cannot be negative"),
({"max_pairs": 0}, "--max-pairs must be positive"),
({"relative_intensity": 1.5}, "--relative-intensity must be between 0 and 1"),
({"min_peaks": -1}, "--min-peaks cannot be negative"),
({"max_peaks": 0}, "--max-peaks must be positive"),
({"remove_precursor_window": 0.0}, "--remove-precursor-window must be positive"),
({"bin_width": 0.0}, "--bin-width must be positive"),
({"blink_top_k": 0}, "--blink-top-k must be positive"),
]
for overrides, message in cases:
with self.subTest(**overrides):
with self.assertRaisesRegex(ValueError, message):
library_search.validate_args(self.args(**overrides))
def test_the_inclusive_ends_of_each_range_are_accepted(self) -> None:
# 0 disables the peak filters and 0/1 are legal similarity scores, so
# rejecting them would make documented invocations unusable.
library_search.validate_args(
self.args(min_score=0.0, min_peaks=0, relative_intensity=0.0)
)
library_search.validate_args(self.args(min_score=1.0, relative_intensity=1.0))
def test_an_inverted_mz_window_is_refused_but_one_sided_is_fine(self) -> None:
with self.assertRaisesRegex(ValueError, "--mz-min must be smaller"):
library_search.validate_args(self.args(mz_min=500.0, mz_max=100.0))
with self.assertRaisesRegex(ValueError, "--mz-min must be smaller"):
library_search.validate_args(self.args(mz_min=100.0, mz_max=100.0))
library_search.validate_args(self.args(mz_min=100.0))
library_search.validate_args(self.args(mz_max=500.0))
def test_min_matches_is_refused_for_metrics_that_report_no_matches(self) -> None:
# Silently ignoring --min-matches would look like a filter that ran.
with self.assertRaisesRegex(ValueError, "does not report matched-peak counts"):
library_search.validate_args(self.args(metric="flash-entropy", min_matches=3))
library_search.validate_args(self.args(metric="cosine", min_matches=3))
# A zero threshold is a no-op, so it stays legal everywhere.
library_search.validate_args(self.args(metric="flash-entropy", min_matches=0))
class ProcessingChainTests(unittest.TestCase):
"""`create_processor` decides which matchms filters run, and in what order."""
@staticmethod
def step_names(args: argparse.Namespace) -> list[str]:
steps = library_search.create_processor(args).processing_steps
return [step[0] if isinstance(step, tuple) else step for step in steps]
def test_metadata_filters_run_before_the_peak_filters(self) -> None:
# SpectrumProcessor keeps registered filters in their canonical order
# but appends unknown callables; the script expands default_filters by
# hand precisely so precursor m/z is derived before it is required.
names = self.step_names(search_namespace(metric="modified"))
self.assertLess(
names.index("add_precursor_mz"), names.index("require_precursor_mz")
)
self.assertLess(
names.index("normalize_intensities"),
names.index("select_by_relative_intensity"),
)
self.assertEqual(names[-1], "require_minimum_number_of_peaks")
def test_precursor_metrics_require_a_precursor_and_others_do_not(self) -> None:
# A modified-cosine score is meaningless without precursor m/z, so the
# requirement must switch on with the metric.
for metric in sorted(library_search.PRECURSOR_METRICS):
with self.subTest(metric=metric):
self.assertIn(
"require_precursor_mz", self.step_names(search_namespace(metric=metric))
)
self.assertNotIn(
"require_precursor_mz", self.step_names(search_namespace(metric="cosine"))
)
def test_a_precursor_window_forces_the_precursor_requirement(self) -> None:
names = self.step_names(
search_namespace(metric="cosine", remove_precursor_window=17.0)
)
self.assertIn("require_precursor_mz", names)
self.assertIn("remove_peaks_around_precursor_mz", names)
def test_the_optional_filters_are_absent_by_default(self) -> None:
names = self.step_names(search_namespace(metric="cosine"))
for absent in (
"select_by_mz",
"remove_peaks_around_precursor_mz",
"reduce_to_number_of_peaks",
):
with self.subTest(filter=absent):
self.assertNotIn(absent, names)
def test_each_switch_adds_exactly_its_own_filter(self) -> None:
cases = [
({"mz_min": 50.0}, "select_by_mz"),
({"mz_max": 900.0}, "select_by_mz"),
({"max_peaks": 50}, "reduce_to_number_of_peaks"),
]
for overrides, expected in cases:
with self.subTest(**overrides):
self.assertIn(expected, self.step_names(search_namespace(**overrides)))
def test_the_disabling_flags_actually_remove_steps(self) -> None:
self.assertNotIn(
"normalize_intensities",
self.step_names(search_namespace(no_normalize=True)),
)
self.assertNotIn(
"add_compound_name",
self.step_names(search_namespace(no_default_filters=True)),
)
def test_zero_thresholds_disable_their_filters(self) -> None:
# `--relative-intensity 0` and `--min-peaks 0` are documented as "off",
# not as "keep everything above zero".
names = self.step_names(search_namespace(relative_intensity=0.0, min_peaks=0))
self.assertNotIn("select_by_relative_intensity", names)
self.assertNotIn("require_minimum_number_of_peaks", names)
def test_the_same_chain_is_used_for_queries_and_references(self) -> None:
# Comparing differently processed collections biases every score, so
# the processor is built once from one namespace.
args = search_namespace()
self.assertEqual(self.step_names(args), self.step_names(args))
class ScoreRecordTests(unittest.TestCase):
"""matchms returns either a bare float or a structured `(score, matches)`."""
def test_the_score_field_is_the_one_ending_in_score(self) -> None:
self.assertEqual(
library_search.choose_score_fields(
("CosineGreedy_matches", "CosineGreedy_score")
),
("CosineGreedy_score", "CosineGreedy_matches"),
)
def test_a_metric_without_a_score_suffix_falls_back_to_the_first_field(self) -> None:
self.assertEqual(
library_search.choose_score_fields(("FlashSimilarity",)),
("FlashSimilarity", None),
)
def test_a_structured_record_is_read_field_by_field(self) -> None:
record = score_record(0.75, 9)
self.assertEqual(library_search.numeric_field(record, "Fake_score"), 0.75)
self.assertEqual(library_search.matched_peaks(record, "Fake_matches"), 9)
def test_a_scalar_score_is_read_directly_and_reports_no_matches(self) -> None:
self.assertEqual(library_search.numeric_field(numpy.float64(0.4), "Fake_score"), 0.4)
# Flash metrics return a scalar; inventing a match count would be a lie.
self.assertIsNone(library_search.matched_peaks(numpy.float64(0.4), "Fake_matches"))
self.assertIsNone(library_search.matched_peaks(score_record(0.4, 3), None))
def test_a_field_the_record_does_not_carry_is_not_invented(self) -> None:
self.assertIsNone(library_search.matched_peaks(score_record(0.4, 3), "Other_matches"))
class SpectrumIdentityTests(unittest.TestCase):
def test_the_preferred_field_wins_over_every_default(self) -> None:
spectrum = FakeSpectrum(feature_id="F7", spectrum_id="S1", compound_name="caffeine")
self.assertEqual(
library_search.spectrum_id(
spectrum, preferred="feature_id", index=3, prefix="query"
),
"F7",
)
def test_the_default_order_prefers_spectrum_id_over_a_name(self) -> None:
spectrum = FakeSpectrum(spectrum_id="S1", compound_name="caffeine", title="scan 4")
self.assertEqual(
library_search.spectrum_id(spectrum, preferred=None, index=3, prefix="query"),
"S1",
)
def test_an_empty_value_is_skipped_rather_than_used(self) -> None:
# An MGF with `TITLE=` yields "", which would produce a blank id column.
spectrum = FakeSpectrum(spectrum_id="", id=None, compound_name="caffeine")
self.assertEqual(
library_search.spectrum_id(spectrum, preferred=None, index=3, prefix="query"),
"caffeine",
)
def test_an_unidentified_spectrum_falls_back_to_prefix_and_index(self) -> None:
self.assertEqual(
library_search.spectrum_id(
FakeSpectrum(), preferred=None, index=0, prefix="reference"
),
"reference-0",
)
def test_a_numeric_identifier_is_stringified(self) -> None:
self.assertEqual(
library_search.spectrum_id(
FakeSpectrum(scan_number=42), preferred=None, index=0, prefix="query"
),
"42",
)
def test_missing_metadata_becomes_an_empty_cell_not_the_word_none(self) -> None:
self.assertEqual(library_search.metadata_value(FakeSpectrum(), "inchikey"), "")
self.assertEqual(
library_search.metadata_value(FakeSpectrum(precursor_mz=350.1), "precursor_mz"),
"350.1",
)
class HitWritingTests(unittest.TestCase):
"""`write_hits` owns the top-k, min-score and min-matches decisions."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.output = Path(self._temporary.name) / "hits.csv"
self.references = [
FakeSpectrum(spectrum_id="R0", compound_name="first", inchikey="AAA"),
FakeSpectrum(spectrum_id="R1", compound_name="second"),
FakeSpectrum(spectrum_id="R2", compound_name="third"),
]
self.query = FakeSpectrum(spectrum_id="Q0", precursor_mz=350.1)
def write(self, ranked, **overrides) -> list[dict[str, str]]:
args = search_namespace(metric="cosine", **overrides)
library_search.write_hits(
self.output,
scores=FakeScores(ranked),
queries=[self.query],
references=self.references,
args=args,
)
with self.output.open(encoding="utf-8", newline="") as handle:
return list(csv.DictReader(handle))
def ranked(self, *scores):
return [
(self.references[index], score_record(score, matches))
for index, (score, matches) in enumerate(scores)
]
def test_hits_are_numbered_from_one_and_carry_reference_metadata(self) -> None:
rows = self.write(self.ranked((0.9, 8), (0.8, 6)))
self.assertEqual([row["rank"] for row in rows], ["1", "2"])
self.assertEqual([row["reference_index"] for row in rows], ["0", "1"])
self.assertEqual([row["reference_id"] for row in rows], ["R0", "R1"])
self.assertEqual(rows[0]["reference_compound_name"], "first")
self.assertEqual(rows[0]["reference_inchikey"], "AAA")
# Reference 1 has no inchikey; the cell must be empty, not "None".
self.assertEqual(rows[1]["reference_inchikey"], "")
self.assertEqual(rows[0]["query_id"], "Q0")
self.assertEqual(rows[0]["metric"], "cosine")
self.assertEqual(rows[0]["score_field"], "Fake_score")
self.assertEqual(rows[0]["matched_peaks"], "8")
def test_top_k_truncates_after_the_filters_not_before(self) -> None:
# 0.4 is dropped by --min-score, so the third-ranked survivor must
# still be reported when --top-k is 2.
rows = self.write(
self.ranked((0.9, 8), (0.4, 8), (0.7, 8)), min_score=0.5, top_k=2
)
self.assertEqual([row["score"] for row in rows], ["0.9", "0.7"])
self.assertEqual([row["rank"] for row in rows], ["1", "2"])
def test_a_score_exactly_at_the_threshold_is_kept(self) -> None:
rows = self.write(self.ranked((0.5, 8)), min_score=0.5)
self.assertEqual(len(rows), 1)
rows = self.write(self.ranked((0.5, 8)), min_score=0.5000001)
self.assertEqual(rows, [])
def test_the_match_count_threshold_is_inclusive(self) -> None:
self.assertEqual(len(self.write(self.ranked((0.9, 5)), min_matches=5)), 1)
self.assertEqual(self.write(self.ranked((0.9, 4)), min_matches=5), [])
def test_a_scalar_score_survives_a_match_threshold(self) -> None:
# Metrics with no match count must not be filtered to nothing by a
# threshold they cannot answer; validate_args already refuses that
# combination, so write_hits must not silently drop the rows either.
ranked = [(self.references[0], numpy.float64(0.9))]
rows = self.write(ranked, min_matches=99)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["matched_peaks"], "")
def test_a_non_finite_score_is_dropped(self) -> None:
# A NaN comparison is neither above nor below the threshold, so it has
# to be excluded explicitly or it would rank first after sorting.
rows = self.write(self.ranked((float("nan"), 8), (0.6, 8)))
self.assertEqual([row["score"] for row in rows], ["0.6"])
self.assertEqual([row["rank"] for row in rows], ["1"])
def test_an_empty_ranking_still_writes_a_header(self) -> None:
self.assertEqual(self.write([]), [])
first_line = self.output.read_text(encoding="utf-8").splitlines()[0]
self.assertTrue(first_line.startswith("query_index,query_id"))
def test_a_reference_outside_the_library_is_an_error_not_a_blank_row(self) -> None:
stranger = FakeSpectrum(spectrum_id="X")
with self.assertRaisesRegex(RuntimeError, "outside the input library"):
self.write([(stranger, score_record(0.9, 8))])
def test_the_writer_asks_for_a_sorted_ranking_by_the_score_field(self) -> None:
scores = FakeScores(self.ranked((0.9, 8)))
library_search.write_hits(
self.output,
scores=scores,
queries=[self.query],
references=self.references,
args=search_namespace(),
)
# Ranks are assigned in iteration order, so the sort has to happen in
# matchms rather than here.
self.assertTrue(scores.requested_sort)
self.assertEqual(scores.requested_name, "Fake_score")
def test_scores_are_written_with_enough_precision_to_be_reproducible(self) -> None:
rows = self.write(self.ranked((0.123456789012, 8)))
self.assertEqual(rows[0]["score"], "0.123456789012")
class EndToEndSearchTests(unittest.TestCase):
"""A spectrum searched against a copy of itself scores exactly 1.0."""
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
self.queries = self.root / "queries.mgf"
self.references = self.root / "library.mgf"
self.queries.write_text(QUERY_MGF, encoding="utf-8")
self.references.write_text(LIBRARY_MGF, encoding="utf-8")
self.output = self.root / "hits.csv"
def args(self, **overrides) -> argparse.Namespace:
settings = dict(
queries=self.queries,
references=self.references,
output=self.output,
metric="cosine",
)
settings.update(overrides)
return search_namespace(**settings)
def rows(self) -> list[dict[str, str]]:
with self.output.open(encoding="utf-8", newline="") as handle:
return list(csv.DictReader(handle))
def test_an_identical_reference_is_reported_with_a_perfect_score(self) -> None:
self.assertEqual(library_search.run(self.args()), 0)
rows = self.rows()
# Two of the three references are copies of the query; the third shares
# no fragment, and matchms stores no entry for a zero score.
self.assertEqual(len(rows), 2)
for row in rows:
self.assertEqual(float(row["score"]), 1.0)
# All five peaks match within the 0.02 Da tolerance.
self.assertEqual(row["matched_peaks"], "5")
self.assertEqual(row["query_id"], "alpha")
self.assertEqual(row["query_precursor_mz"], "350.1")
self.assertEqual(
sorted(row["reference_id"] for row in rows), ["alpha_copy", "alpha_twin"]
)
def test_top_k_limits_the_reported_hits(self) -> None:
self.assertEqual(library_search.run(self.args(top_k=1)), 0)
rows = self.rows()
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["rank"], "1")
def test_a_match_threshold_above_the_peak_count_reports_nothing(self) -> None:
# Six matched peaks are impossible for a five-peak spectrum, so this
# proves the filter is applied rather than merely accepted.
self.assertEqual(library_search.run(self.args(min_matches=6)), 0)
self.assertEqual(self.rows(), [])
def test_the_pair_budget_stops_the_search_before_it_starts(self) -> None:
with self.assertRaisesRegex(ValueError, "above --max-pairs"):
library_search.run(self.args(max_pairs=2))
# Nothing was written, so a refused run cannot look like an empty result.
self.assertFalse(self.output.exists())
def test_a_library_that_processing_empties_is_reported(self) -> None:
# Requiring more peaks than any spectrum has leaves nothing to search.
with self.assertRaisesRegex(ValueError, "no query spectra remain"):
library_search.run(self.args(min_peaks=99))
def test_main_turns_a_validation_error_into_exit_code_two(self) -> None:
argv = [
"library_search.py",
str(self.root / "absent.mgf"),
str(self.references),
str(self.output),
]
original = sys.argv
sys.argv = argv
try:
self.assertEqual(library_search.main(), 2)
finally:
sys.argv = original
class VersionPinTests(unittest.TestCase):
def test_the_installed_version_is_reported_rather_than_assumed(self) -> None:
# `run` compares this against TARGET_VERSION to warn about drift, so it
# must be a real version string and not the "unknown" fallback.
reported = library_search.installed_matchms_version()
self.assertRegex(reported, r"^\d+\.\d+")
self.assertRegex(library_search.TARGET_VERSION, r"^\d+\.\d+\.\d+$")
def test_the_documented_target_matches_the_skill_metadata(self) -> None:
# SKILL.md tells the agent which release the script was verified on.
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertIn(library_search.TARGET_VERSION, text)
if __name__ == "__main__":
unittest.main()

2
tests/matlab/test_static.py
View File

@@ -42,7 +42,7 @@ class SkillStructureTests(unittest.TestCase):
self.assertIn("\ncompatibility: >-\n", text)
self.assertRegex(
text,
r'\nmetadata:\n version: "1\.1"\n'
r'\nmetadata:\n version: "\d+\.\d+"\n'
r' skill-author: "K-Dense Inc\."\n'
r' last-reviewed: "2026-07-23"\n',
)

236
tests/matplotlib/test_scripts.py Normal file
View File

@@ -0,0 +1,236 @@
"""Tests for the matplotlib plotting templates and style configurator.
Rendering assertions are shallow by nature, so these tests go after the things
that can actually be wrong: that every style preset contains only rcParams
matplotlib recognises (a typo'd key is silently ignored, and the figure just
looks wrong), that a saved `.mplstyle` file can be loaded back by matplotlib
itself, and that each plot helper draws onto the axes it is handed rather than
into the global current figure.
Everything runs on the Agg backend and closes its figures; a suite that leaks
figures eventually trips matplotlib's open-figure warning.
"""
from __future__ import annotations
import sys
import tempfile
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "matplotlib"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
matplotlib = pytest.importorskip("matplotlib", reason="matplotlib skill needs matplotlib")
matplotlib.use("Agg")
np = pytest.importorskip("numpy", reason="matplotlib scripts need numpy")
import matplotlib.pyplot as plt # noqa: E402
import plot_template # noqa: E402
import style_configurator # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
PLOT_BUILDERS = (
"create_line_plot",
"create_scatter_plot",
"create_bar_chart",
"create_histogram",
"create_heatmap",
"create_contour_plot",
"create_box_plot",
"create_violin_plot",
)
class FigureTestCase(unittest.TestCase):
def setUp(self) -> None:
self.addCleanup(plt.close, "all")
class StylePresetTests(unittest.TestCase):
def test_presets_exist_and_are_named_dictionaries(self) -> None:
self.assertTrue(style_configurator.STYLE_PRESETS)
for name, settings in style_configurator.STYLE_PRESETS.items():
with self.subTest(preset=name):
self.assertIsInstance(settings, dict)
self.assertTrue(settings)
def test_every_preset_key_is_a_real_rcparam(self) -> None:
# matplotlib ignores unknown rcParams silently, so a typo here is
# invisible until someone notices the figure looks wrong.
valid = set(matplotlib.rcParams)
for name, settings in style_configurator.STYLE_PRESETS.items():
unknown = sorted(set(settings) - valid)
with self.subTest(preset=name):
self.assertEqual(unknown, [])
def test_every_preset_applies_cleanly(self) -> None:
original = matplotlib.rcParams.copy()
self.addCleanup(matplotlib.rcParams.update, original)
for name, settings in style_configurator.STYLE_PRESETS.items():
with self.subTest(preset=name):
matplotlib.rcParams.update(settings)
def test_the_publication_preset_saves_at_print_resolution(self) -> None:
publication = style_configurator.STYLE_PRESETS["publication"]
self.assertGreaterEqual(publication["savefig.dpi"], 300)
self.assertEqual(publication["savefig.bbox"], "tight")
def test_every_documented_preset_is_defined(self) -> None:
import io
from contextlib import redirect_stdout
buffer = io.StringIO()
with redirect_stdout(buffer):
style_configurator.list_available_presets()
listed = buffer.getvalue()
for name in style_configurator.STYLE_PRESETS:
with self.subTest(preset=name):
self.assertIn(name, listed)
class StyleFileTests(unittest.TestCase):
def test_a_saved_style_is_loadable_by_matplotlib(self) -> None:
# The real contract: matplotlib must be able to read what we wrote.
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "custom.mplstyle"
style_configurator.save_style_file(
style_configurator.STYLE_PRESETS["publication"], str(path)
)
self.assertTrue(path.is_file())
original = matplotlib.rcParams.copy()
self.addCleanup(matplotlib.rcParams.update, original)
plt.style.use(str(path))
def test_the_file_is_commented_and_grouped(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "custom.mplstyle"
style_configurator.save_style_file(
style_configurator.STYLE_PRESETS["publication"], str(path)
)
text = path.read_text(encoding="utf-8")
self.assertIn("# Custom matplotlib style", text)
self.assertIn("# Figure", text)
self.assertIn("savefig.dpi: 300", text)
def test_sequence_values_are_written_comma_separated(self) -> None:
# `font.sans-serif` is a list; mplstyle wants `a, b`, not `['a', 'b']`.
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "custom.mplstyle"
style_configurator.save_style_file(
{"font.sans-serif": ["Arial", "Helvetica"]}, str(path)
)
text = path.read_text(encoding="utf-8")
self.assertIn("font.sans-serif: Arial, Helvetica", text)
self.assertNotIn("[", text)
def test_an_empty_style_still_writes_a_valid_file(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "empty.mplstyle"
style_configurator.save_style_file({}, str(path))
self.assertTrue(path.is_file())
original = matplotlib.rcParams.copy()
self.addCleanup(matplotlib.rcParams.update, original)
plt.style.use(str(path))
class SampleDataTests(unittest.TestCase):
def test_the_sample_data_covers_every_plot_type(self) -> None:
data = plot_template.generate_sample_data()
self.assertIsInstance(data, dict)
self.assertTrue(data)
def test_the_sample_data_is_deterministic(self) -> None:
# The templates are documentation; a figure that changes between runs
# cannot be compared against the one in the docs.
first = plot_template.generate_sample_data()
second = plot_template.generate_sample_data()
self.assertEqual(sorted(first), sorted(second))
for key, value in first.items():
with self.subTest(series=key):
if isinstance(value, np.ndarray):
np.testing.assert_allclose(value, second[key])
def test_the_preview_data_is_deterministic_too(self) -> None:
first = style_configurator.generate_preview_data()
second = style_configurator.generate_preview_data()
for key, value in first.items():
with self.subTest(series=key):
if isinstance(value, np.ndarray):
np.testing.assert_allclose(value, second[key])
class PlotBuilderTests(FigureTestCase):
def setUp(self) -> None:
super().setUp()
self.data = plot_template.generate_sample_data()
def test_every_builder_draws_onto_the_axes_it_is_given(self) -> None:
for name in PLOT_BUILDERS:
with self.subTest(builder=name):
figure, axes = plt.subplots()
getattr(plot_template, name)(self.data, ax=axes)
# Something was drawn: lines, patches, images, or collections.
drawn = (
len(axes.lines)
+ len(axes.patches)
+ len(axes.images)
+ len(axes.collections)
)
self.assertGreater(drawn, 0, f"{name} drew nothing")
plt.close(figure)
def test_every_builder_labels_its_axes(self) -> None:
# An unlabelled publication figure is a bug, not a style preference.
for name in PLOT_BUILDERS:
with self.subTest(builder=name):
figure, axes = plt.subplots()
getattr(plot_template, name)(self.data, ax=axes)
self.assertTrue(
axes.get_title() or axes.get_xlabel() or axes.get_ylabel(),
f"{name} produced an unlabelled figure",
)
plt.close(figure)
def test_builders_create_their_own_axes_when_none_is_supplied(self) -> None:
for name in PLOT_BUILDERS:
with self.subTest(builder=name):
before = len(plt.get_fignums())
getattr(plot_template, name)(self.data)
self.assertGreaterEqual(len(plt.get_fignums()), before)
plt.close("all")
def test_the_publication_style_applies_without_error(self) -> None:
original = matplotlib.rcParams.copy()
self.addCleanup(matplotlib.rcParams.update, original)
plot_template.set_publication_style()
self.assertGreaterEqual(matplotlib.rcParams["savefig.dpi"], 150)
class CompositeFigureTests(FigureTestCase):
def test_the_comprehensive_figure_renders_and_saves(self) -> None:
result = plot_template.create_comprehensive_figure()
self.assertIsNotNone(result)
with tempfile.TemporaryDirectory() as directory:
output = Path(directory) / "figure.png"
plt.savefig(output, dpi=72)
self.assertGreater(output.stat().st_size, 0)
def test_the_style_preview_renders(self) -> None:
result = style_configurator.create_style_preview(
style_configurator.STYLE_PRESETS["minimal"]
)
self.assertIsNotNone(result)
if __name__ == "__main__":
unittest.main()

394
tests/medchem/test_scripts.py Normal file
View File

@@ -0,0 +1,394 @@
"""Tests for the medchem batch-filtering script.
`filter_molecules.py` has two halves worth guarding. The first is input
handling: `load_molecules` reads four file formats, and when it drops an
unparseable SMILES it must drop the matching DataFrame row too -- otherwise
every downstream filter column is offset against the wrong molecule, which is
silent and unrecoverable. The second is the filter wrappers, which rename and
prune the columns medchem returns; a rename that stops matching would leave the
`passes_*` columns invisible to `--filter-output` and to the summary report.
Every filter is therefore exercised in both directions on molecules whose
verdict is known from the published rule rather than from this code: ethanol
satisfies Lipinski and a pentapeptide does not; caffeine carries no ChEMBL
structural alert and catechol is a textbook PAINS/ChEMBL flag. The summary
arithmetic is checked against hand-computed percentages, including the
empty-input case that would otherwise divide by zero.
"""
from __future__ import annotations
import sys
import tempfile
import unittest
from pathlib import Path
import pytest
import skill_contract
SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "medchem"
SCRIPTS = SKILL_ROOT / "scripts"
sys.path.insert(0, str(SCRIPTS))
medchem = pytest.importorskip("medchem", reason="medchem skill needs medchem")
datamol = pytest.importorskip("datamol", reason="medchem skill needs datamol")
pandas = pytest.importorskip("pandas", reason="medchem skill needs pandas")
pytest.importorskip("rdkit", reason="medchem skill needs rdkit")
pytest.importorskip("tqdm", reason="medchem skill needs tqdm")
from rdkit import Chem # noqa: E402
import filter_molecules # noqa: E402
CliHelpTests = skill_contract.cli.help_test_case(SKILL_ROOT)
# Verdicts below are fixed by the published rules, not by medchem's behaviour.
ETHANOL = "CCO"
BENZENE = "c1ccccc1"
CAFFEINE = "CN1C=NC2=C1C(=O)N(C(=O)N2C)C"
#: Benzene-1,2-diol. Catechols are the canonical PAINS class (catechol_A) and
#: are excluded by the ChEMBL common alerts and by the NIBR screening deck.
CATECHOL = "Oc1ccccc1O"
#: Phe-Leu-Lys-Asp-Glu: 5 residues, >500 Da with well over 5 H-bond donors, so
#: it violates Lipinski's rule of five.
PENTAPEPTIDE = (
"CC(C)C[C@H](NC(=O)[C@@H](N)CC1=CC=CC=C1)C(=O)N[C@@H](CCCCN)"
"C(=O)N[C@@H](CC(=O)O)C(=O)N[C@@H](CCC(=O)O)C(=O)O"
)
#: Not a parseable SMILES: lowercase q is not an element or aromatic atom.
GARBAGE_SMILES = "qqq(((1"
def molecules(*smiles):
return [datamol.to_mol(item) for item in smiles]
class TemporaryDirectoryTestCase(unittest.TestCase):
def setUp(self) -> None:
self._temporary = tempfile.TemporaryDirectory()
self.addCleanup(self._temporary.cleanup)
self.root = Path(self._temporary.name)
class LoadMoleculesTests(TemporaryDirectoryTestCase):
"""Reading four formats, and staying aligned when a SMILES fails to parse."""
def test_a_csv_is_read_with_its_extra_columns_preserved(self) -> None:
path = self.root / "input.csv"
path.write_text("smiles,compound_id\nCCO,c1\nc1ccccc1,c2\n", encoding="utf-8")
frame, mols = filter_molecules.load_molecules(path)
self.assertEqual(len(mols), 2)
self.assertEqual(list(frame["compound_id"]), ["c1", "c2"])
self.assertEqual(
[Chem.MolToSmiles(mol) for mol in mols],
[Chem.MolToSmiles(item) for item in molecules(ETHANOL, BENZENE)],
)
def test_a_named_smiles_column_is_honoured(self) -> None:
path = self.root / "input.csv"
path.write_text("structure\nCCO\n", encoding="utf-8")
frame, mols = filter_molecules.load_molecules(path, smiles_column="structure")
self.assertEqual(len(mols), 1)
self.assertEqual(list(frame.columns), ["structure"])
def test_a_tsv_is_split_on_tabs_not_commas(self) -> None:
# Reading a TSV with the comma separator would leave one column named
# "smiles\tname", and the missing-column branch would exit instead.
path = self.root / "input.tsv"
path.write_text("smiles\tname\nCCO\tethanol\n", encoding="utf-8")
frame, mols = filter_molecules.load_molecules(path)
self.assertEqual(list(frame.columns), ["smiles", "name"])
self.assertEqual(list(frame["name"]), ["ethanol"])
self.assertEqual(len(mols), 1)
def test_a_plain_text_file_treats_each_line_as_a_smiles(self) -> None:
path = self.root / "input.txt"
# Blank lines and surrounding whitespace are stripped, not parsed.
path.write_text("CCO\n\n c1ccccc1 \n\n", encoding="utf-8")
frame, mols = filter_molecules.load_molecules(path)
self.assertEqual(list(frame["smiles"]), [ETHANOL, BENZENE])
self.assertEqual(len(mols), 2)
def test_an_sdf_carries_its_properties_into_the_frame(self) -> None:
path = self.root / "input.sdf"
writer = Chem.SDWriter(str(path))
for smiles, name in ((ETHANOL, "ethanol"), (BENZENE, "benzene")):
mol = Chem.MolFromSmiles(smiles)
mol.SetProp("compound_name", name)
writer.write(mol)
writer.close()
frame, mols = filter_molecules.load_molecules(path)
self.assertEqual(len(mols), 2)
self.assertEqual(list(frame["compound_name"]), ["ethanol", "benzene"])
# A canonical SMILES column is synthesised so the CSV output always has
# a structure column, whatever the SDF happened to carry.
self.assertEqual(list(frame["smiles"]), [Chem.CanonSmiles(ETHANOL), Chem.CanonSmiles(BENZENE)])
def test_an_unparseable_smiles_drops_its_row_too(self) -> None:
# The alignment guarantee: filter results are concatenated positionally
# onto this frame, so a dropped molecule must drop its metadata row.
path = self.root / "input.csv"
path.write_text(
f"smiles,compound_id\n{ETHANOL},keep-1\n{GARBAGE_SMILES},drop\n"
f"{BENZENE},keep-2\n",
encoding="utf-8",
)
frame, mols = filter_molecules.load_molecules(path)
self.assertEqual(len(mols), 2)
self.assertEqual(len(frame), 2)
self.assertEqual(list(frame["compound_id"]), ["keep-1", "keep-2"])
# The index is reset, so positional concatenation lines up.
self.assertEqual(list(frame.index), [0, 1])
def test_a_file_of_only_bad_smiles_yields_nothing_rather_than_failing(self) -> None:
path = self.root / "input.txt"
path.write_text(f"{GARBAGE_SMILES}\n", encoding="utf-8")
frame, mols = filter_molecules.load_molecules(path)
self.assertEqual(mols, [])
self.assertEqual(len(frame), 0)
def test_a_header_only_csv_loads_zero_molecules(self) -> None:
path = self.root / "input.csv"
path.write_text("smiles\n", encoding="utf-8")
frame, mols = filter_molecules.load_molecules(path)
self.assertEqual(mols, [])
self.assertEqual(len(frame), 0)
def test_a_missing_smiles_column_exits_rather_than_guessing(self) -> None:
path = self.root / "input.csv"
path.write_text("structure\nCCO\n", encoding="utf-8")
with self.assertRaises(SystemExit) as raised:
filter_molecules.load_molecules(path)
self.assertEqual(raised.exception.code, 1)
def test_an_unsupported_extension_exits(self) -> None:
path = self.root / "input.xlsx"
path.write_bytes(b"")
with self.assertRaises(SystemExit) as raised:
filter_molecules.load_molecules(path)
self.assertEqual(raised.exception.code, 1)
def test_the_extension_check_is_case_insensitive(self) -> None:
path = self.root / "input.CSV"
path.write_text("smiles\nCCO\n", encoding="utf-8")
_, mols = filter_molecules.load_molecules(path)
self.assertEqual(len(mols), 1)
class RuleFilterTests(unittest.TestCase):
"""Lipinski and friends, verified against the published rule."""
def test_rule_of_five_accepts_a_small_molecule_and_rejects_a_pentapeptide(self) -> None:
results = filter_molecules.apply_rule_filters(
molecules(ETHANOL, PENTAPEPTIDE), ["rule_of_five"], 1
)
self.assertEqual(list(results["rule_of_five"]), [True, False])
def test_the_molecule_objects_are_not_carried_into_the_output_frame(self) -> None:
# A `mol` column of RDKit objects cannot be written to CSV.
results = filter_molecules.apply_rule_filters(molecules(ETHANOL), ["rule_of_five"], 1)
self.assertNotIn("mol", results.columns)
self.assertEqual(len(results), 1)
def test_several_rules_produce_one_column_each_plus_the_aggregates(self) -> None:
results = filter_molecules.apply_rule_filters(
molecules(ETHANOL, PENTAPEPTIDE), ["rule_of_five", "rule_of_veber"], 1
)
self.assertLessEqual({"rule_of_five", "rule_of_veber"}, set(results.columns))
# `pass_all` is what generate_summary and --filter-output key on.
self.assertIn("pass_all", results.columns)
self.assertEqual(
list(results["pass_all"]),
[
bool(results["rule_of_five"][index] and results["rule_of_veber"][index])
for index in range(2)
],
)
def test_the_rule_names_the_cli_validates_against_include_the_documented_ones(self) -> None:
# main() warns when --rules names something outside this list, so the
# list has to contain the rules SKILL.md tells the agent to ask for.
available = set(medchem.rules.RuleFilters.list_available_rules_names())
self.assertLessEqual({"rule_of_five", "rule_of_cns", "rule_of_veber"}, available)
self.assertNotIn("rule_of_nonsense", available)
def test_an_unknown_rule_is_refused_rather_than_ignored(self) -> None:
with self.assertRaisesRegex(ValueError, "rule_of_nonsense"):
filter_molecules.apply_rule_filters(molecules(ETHANOL), ["rule_of_nonsense"], 1)
class StructuralAlertTests(unittest.TestCase):
"""Alert catalogs, and the column renames the rest of the script depends on."""
def test_common_alerts_pass_caffeine_and_exclude_catechol(self) -> None:
results = filter_molecules.apply_common_alerts(molecules(CAFFEINE, CATECHOL), 1)
self.assertEqual(list(results["passes_common_alerts"]), [True, False])
self.assertEqual(list(results["common_alert_status"]), ["ok", "exclude"])
def test_common_alerts_columns_are_renamed_to_the_passes_convention(self) -> None:
# generate_summary and --filter-output only see columns starting with
# `passes_`, so medchem's generic `pass_filter`/`status` must be renamed.
results = filter_molecules.apply_common_alerts(molecules(CAFFEINE), 1)
self.assertNotIn("pass_filter", results.columns)
self.assertNotIn("status", results.columns)
self.assertNotIn("mol", results.columns)
self.assertIn("passes_common_alerts", results.columns)
def test_nibr_passes_caffeine_and_excludes_catechol(self) -> None:
results = filter_molecules.apply_nibr(molecules(CAFFEINE, CATECHOL), 1)
self.assertEqual(list(results["passes_nibr"]), [True, False])
self.assertEqual(list(results["nibr_status"]), ["ok", "exclude"])
def test_nibr_columns_are_renamed_too(self) -> None:
results = filter_molecules.apply_nibr(molecules(CAFFEINE), 1)
self.assertNotIn("pass_filter", results.columns)
self.assertNotIn("status", results.columns)
self.assertIn("passes_nibr", results.columns)
def test_the_pains_catalog_flags_catechol_and_leaves_caffeine_alone(self) -> None:
produced = list(
filter_molecules.apply_alert_catalog(molecules(CAFFEINE, CATECHOL), ["pains"], 1)
)
self.assertEqual(len(produced), 1)
name, passes = produced[0]
self.assertEqual(name, "pains")
self.assertEqual(list(passes), [True, False])
def test_each_requested_catalog_is_yielded_separately(self) -> None:
# main() pairs each yielded name with its own `passes_<name>` column, so
# the generator must not merge catalogs into one verdict.
produced = list(
filter_molecules.apply_alert_catalog(molecules(CAFFEINE), ["pains", "brenk"], 1)
)
self.assertEqual([name for name, _ in produced], ["pains", "brenk"])
for name, passes in produced:
with self.subTest(catalog=name):
self.assertEqual(len(passes), 1)
class ComplexityAndQueryTests(unittest.TestCase):
def test_the_complexity_column_records_the_metric_used(self) -> None:
# Two metrics in one run would collide on a fixed column name.
bertz = filter_molecules.apply_complexity(molecules(ETHANOL), "99", "bertz", 1)
sas = filter_molecules.apply_complexity(molecules(ETHANOL), "99", "sas", 1)
self.assertEqual(list(bertz.columns), ["passes_complexity_bertz"])
self.assertEqual(list(sas.columns), ["passes_complexity_sas"])
def test_a_generous_complexity_percentile_keeps_a_simple_molecule(self) -> None:
# Ethanol sits at the very bottom of the ZINC-15 complexity
# distribution, so a 99th-percentile ceiling cannot exclude it.
results = filter_molecules.apply_complexity(molecules(ETHANOL), "99", "bertz", 1)
self.assertEqual(list(results["passes_complexity_bertz"]), [True])
def test_a_query_can_both_admit_and_reject_molecules(self) -> None:
results = filter_molecules.apply_query(
molecules(ETHANOL, PENTAPEPTIDE), 'MATCHRULE("rule_of_five")', 1
)
self.assertEqual(list(results["passes_query"]), [True, False])
self.assertEqual(list(results.columns), ["passes_query"])
def test_a_negated_query_inverts_the_verdict(self) -> None:
results = filter_molecules.apply_query(
molecules(ETHANOL, PENTAPEPTIDE), 'NOT MATCHRULE("rule_of_five")', 1
)
self.assertEqual(list(results["passes_query"]), [False, True])
def test_a_malformed_query_raises_instead_of_admitting_everything(self) -> None:
# A parse failure that fell through would look like "all molecules
# passed" -- the worst possible outcome for a filter.
with self.assertRaises(Exception) as raised:
filter_molecules.apply_query(molecules(ETHANOL), "THIS IS NOT A QUERY", 1)
self.assertNotIsInstance(raised.exception, AssertionError)
class ChemicalGroupTests(unittest.TestCase):
def test_one_boolean_column_is_produced_per_requested_group(self) -> None:
results = filter_molecules.apply_groups(
molecules(ETHANOL, CAFFEINE), ["amino_acids", "common_organic_solvents"]
)
self.assertEqual(
list(results.columns), ["has_amino_acids", "has_common_organic_solvents"]
)
self.assertEqual(len(results), 2)
for column in results.columns:
with self.subTest(column=column):
self.assertTrue(all(isinstance(value, bool) for value in results[column]))
def test_an_unrecognised_group_reports_no_matches_rather_than_raising(self) -> None:
# Documented sharp edge: unlike --rules, group names are not validated,
# so a typo yields an all-False column instead of an error.
results = filter_molecules.apply_groups(molecules(ETHANOL), ["not_a_real_group"])
self.assertEqual(list(results["has_not_a_real_group"]), [False])
class LillyFilterTests(unittest.TestCase):
def test_a_missing_lilly_installation_yields_a_null_column_of_the_right_length(self) -> None:
# The Lilly rules are a separate conda package. When absent the script
# must still return one row per molecule, or the positional concat in
# main() would misalign every later column.
results = filter_molecules.apply_lilly(molecules(ETHANOL, CAFFEINE), 160, 1)
self.assertEqual(list(results.columns), ["passes_lilly"])
self.assertEqual(len(results), 2)
class SummaryReportTests(TemporaryDirectoryTestCase):
"""The arithmetic in `generate_summary`, against hand-computed values."""
def summarize(self, frame: "pandas.DataFrame", name: str = "results.csv") -> str:
output = self.root / name
filter_molecules.generate_summary(frame, output)
return (self.root / f"{output.stem}_summary.txt").read_text(encoding="utf-8")
def test_counts_and_percentages_are_reported_per_filter(self) -> None:
frame = pandas.DataFrame(
{
"passes_a": [True, True, False],
"passes_b": [True, False, False],
}
)
text = self.summarize(frame)
self.assertIn("Total molecules processed: 3", text)
self.assertIn("passes_a: 2 passed (66.7%)", text)
self.assertIn("passes_b: 1 passed (33.3%)", text)
# Only the first molecule clears both filters.
self.assertIn("All filters passed: 1 (33.3%)", text)
def test_the_summary_sits_beside_the_output_and_takes_its_stem(self) -> None:
frame = pandas.DataFrame({"passes_a": [True]})
self.summarize(frame, name="run_07.csv")
self.assertTrue((self.root / "run_07_summary.txt").is_file())
def test_the_aggregate_pass_all_column_is_counted_as_a_filter(self) -> None:
# apply_rule_filters emits `pass_all`, which does not start with
# `passes_`; it is named explicitly so rule runs are summarised at all.
text = self.summarize(pandas.DataFrame({"pass_all": [True, False]}))
self.assertIn("pass_all: 1 passed (50.0%)", text)
def test_a_non_boolean_filter_column_is_skipped_not_summed(self) -> None:
# apply_lilly returns None values when the Lilly rules are missing;
# summing those would raise rather than report.
frame = pandas.DataFrame(
{"passes_a": [True, False], "passes_lilly": [None, None]}
)
text = self.summarize(frame)
self.assertIn("passes_a: 1 passed (50.0%)", text)
self.assertNotIn("passes_lilly:", text)
# The all-filters line must ignore the null column rather than zero out.
self.assertIn("All filters passed: 1 (50.0%)", text)
def test_an_empty_result_set_does_not_divide_by_zero(self) -> None:
frame = pandas.DataFrame({"passes_a": pandas.Series(dtype=bool)})
text = self.summarize(frame)
self.assertIn("Total molecules processed: 0", text)
self.assertIn("passes_a: 0 passed (0.0%)", text)
def test_a_frame_with_no_filter_columns_reports_only_the_total(self) -> None:
text = self.summarize(pandas.DataFrame({"smiles": [ETHANOL, BENZENE]}))
self.assertIn("Total molecules processed: 2", text)
self.assertNotIn("All filters passed", text)
if __name__ == "__main__":
unittest.main()

Some files were not shown because too many files have changed in this diff Show More