Update security documentation and scanning process

- Clarified the security scan report generation process in README.md, emphasizing the publication of results.
- Revised `scan_skills.py` documentation to reflect changes in report generation and the removal of the validate_report.py script.
- Updated SECURITY.md to specify the scope of vulnerabilities and improved clarity on the report's purpose and limitations.
- Adjusted the security scan workflow to remove the validation step, streamlining the process while ensuring accurate reporting.
This commit is contained in:
Timothy Kassis
2026-07-26 08:16:14 -07:00
parent b01cff4cda
commit 489de4ce4c
5 changed files with 14 additions and 26 deletions

View File

@@ -45,12 +45,6 @@ jobs:
SKILL_SCAN_FULL: ${{ inputs.full_scan && '1' || '' }}
run: uv run python scan_skills.py
# Gate: a scan that contradicts the contents of skills/ must not be
# published. A non-zero exit here fails the job, so the commit step below
# is skipped and the previous report stays in place.
- name: Validate scan report
run: uv run python validate_report.py
- name: Upload report artifact
if: always()
uses: actions/upload-artifact@v4

View File

@@ -221,7 +221,7 @@ We recommend the following:
```
- **Report anything suspicious.** If you find a skill that looks malicious or behaves unexpectedly, please [open an issue](https://github.com/K-Dense-AI/scientific-agent-skills/issues) immediately so we can investigate.
Skills are scanned weekly — incrementally, so unchanged skills carry their previous findings forward, with a full rescan of everything at least every 30 days and whenever the scanner or model changes — and the results are published to [docs/security-report.md](docs/security-report.md) once they pass an automated consistency check against the repository contents. See [SECURITY.md](SECURITY.md) for our security policy, what is in scope, how to report a vulnerability privately, and how to contest a scan finding. We try to address security gaps as they arise.
Skills are scanned weekly — incrementally, so unchanged skills carry their previous findings forward, with a full rescan of everything at least every 30 days and whenever the scanner or model changes — and the results are published to [docs/security-report.md](docs/security-report.md). See [SECURITY.md](SECURITY.md) for our security policy, what is in scope, how to report a vulnerability privately, and how to contest a scan finding. We try to address security gaps as they arise.
---

View File

@@ -49,7 +49,7 @@ This repository distributes **Agent Skills**: instructions, reference material,
- Prompt-injection vectors — including content in `references/` or `assets/` that an agent is instructed to treat as authoritative
- A skill whose documented behavior materially misrepresents what its bundled code does
- Unsafe credential handling, such as instructions to place secrets where they will be committed or logged
- Vulnerabilities in this repository's own tooling (`scan_skills.py`, `scan_pr_skills.py`, `validate_report.py`) or its GitHub Actions workflows
- Vulnerabilities in this repository's own tooling (`scan_skills.py`, `scan_pr_skills.py`) or its GitHub Actions workflows
## What is out of scope
@@ -75,9 +75,7 @@ The scheduled scan runs weekly and is incremental: a skill whose package content
- **Report:** [`docs/security-report.md`](docs/security-report.md) (machine-readable companion: [`docs/security-report.json`](docs/security-report.json))
- **Workflow:** [`.github/workflows/security-scan.yml`](.github/workflows/security-scan.yml)
**How to read the report.** It is generated by automated tooling, including a language model, and is published to be useful rather than authoritative. It is not an audit, a certification, or a guarantee. A finding in the report is a prompt to review a skill, not a determination that the skill is malicious.
Before any report is published, `validate_report.py` checks it against the actual contents of `skills/` and blocks publication if the scan makes claims that cannot be true of the packages on disk — findings anchored to files that do not exist, cross-file behavior in single-file packages, or asserted script counts that the package does not have. When that check fails, the workflow fails and no report is published. This exists because a scanner that malfunctions would otherwise publish its own errors unreviewed.
**How to read the report.** It is generated by automated tooling, including a language model, and is published to be useful rather than authoritative. It is not an audit, a certification, or a guarantee. Each scan is published automatically, with no pre-publication check that its claims are consistent with the contents of `skills/`, so verify a finding against the skill itself before acting on it. A finding in the report is a prompt to review a skill, not a determination that the skill is malicious.
**If you believe a finding is wrong**, open a regular issue (false positives are not sensitive) with the skill name, the rule ID, and why the finding cannot hold. If a class of false positive originates in the scanner rather than in our configuration, we will also raise it upstream.

View File

@@ -1,8 +1,8 @@
# Security Scan Report
**Status:** awaiting regeneration — no validated report is currently published.
**Status:** awaiting regeneration — no report is currently published.
This file is the output of the automated skill scan described in the [Security Policy](../SECURITY.md#automated-skill-scanning). It is regenerated by `scan_skills.py` and published by [`.github/workflows/security-scan.yml`](../.github/workflows/security-scan.yml) only after `validate_report.py` confirms the scan's claims are consistent with the contents of `skills/`.
This file is the output of the automated skill scan described in the [Security Policy](../SECURITY.md#automated-skill-scanning). It is regenerated by `scan_skills.py` and published by [`.github/workflows/security-scan.yml`](../.github/workflows/security-scan.yml).
## Why there is no report here yet
@@ -17,10 +17,9 @@ The previous report was withdrawn rather than relocated. Verification against th
These are defects in the scan pipeline, not findings about the skills. Because the report was published automatically each week with no check on its plausibility, the errors were republished unreviewed — and because the report occupied the `SECURITY.md` filename, GitHub presented it as this project's official security policy.
Two changes address that:
One change addresses part of that: the security policy is now hand-authored at [`SECURITY.md`](../SECURITY.md), and the scan report lives here instead, so a generated report can no longer present itself as this project's official security policy.
1. The security policy is now hand-authored at [`SECURITY.md`](../SECURITY.md), and the scan report lives here instead.
2. `validate_report.py` now runs between the scan and the commit. It fails the workflow when a report anchors a finding to a file that is not present, describes cross-file behavior in a package too small to have any, or asserts a script count the package does not have. A failing scan no longer publishes anything.
The plausibility of a report's claims is not checked before publication. Treat a finding here as a prompt to inspect the skill, and verify it against the package contents before acting on it.
The withdrawn report remains in version control for reference:
@@ -30,6 +29,6 @@ git show a177179:SECURITY.md
## What replaces it
The next scheduled run regenerates this file. If it passes validation, this page is replaced by the report; if it fails, the workflow fails, this page stays as it is, and the failure output names each claim that could not be reconciled with the repository.
The next scheduled run regenerates this file and publishes it, replacing this page with the report.
Investigating the underlying pipeline defect — determining whether the confabulated inventories originate in per-skill scanner state, in the prompt assembly, or in the model's response — is tracked separately. The `venue-templates` case, where findings cite another skill's files by name, points toward contamination between skills within a single scan run rather than a per-skill parsing error.

View File

@@ -1,9 +1,9 @@
#!/usr/bin/env python3
"""Scan all skills for security issues and produce the security scan report.
Writes a human-readable report plus a machine-readable companion. The JSON is
what `validate_report.py` checks before CI is allowed to publish the report, so
the two files must always be generated together.
Writes a human-readable report plus a machine-readable companion. The two files
must always be generated together: the markdown links into the JSON for
per-skill scan dates, and the JSON is what the next run reads as its cache.
`SECURITY.md` is deliberately not written here: GitHub resolves that filename as
the repository's official security policy, which is hand-authored.
@@ -272,11 +272,9 @@ def generate_json_report(report, skill_meta: dict | None = None,
run_meta: dict | None = None) -> dict:
"""Build the machine-readable companion to the markdown report.
`validate_report.py` consumes this to sanity-check the scan before CI
publishes it, so every field it asserts on must be present here. The
per-skill `scan_metadata` and `analyzability_details` are included because
they carry the scanner's own view of each package, which is what makes
disagreements with the filesystem diagnosable.
The per-skill `scan_metadata` and `analyzability_details` are included
because they carry the scanner's own view of each package, which is what
lets a reviewer diagnose a finding that disagrees with the filesystem.
Per-skill `content_hash` and `last_scanned` are what make incremental runs
possible and honest: the hash decides whether a later run may reuse these
@@ -611,7 +609,6 @@ def main(argv=None):
print(f"\nReport written to {OUTPUT_FILE}")
print(f"Machine-readable report written to {JSON_OUTPUT_FILE}")
print("Run `python validate_report.py` before publishing.")
if __name__ == "__main__":