Compare commits
6 Commits
v2.62.0
...
feat/googl
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
df8fa36140 | ||
|
|
b0587d69a4 | ||
|
|
73f1e5dee8 | ||
|
|
a2cd6d747f | ||
|
|
0ce5914927 | ||
|
|
332e043259 |
@@ -1,103 +1,95 @@
|
||||
---
|
||||
name: parallel-web
|
||||
description: "All-in-one web toolkit powered by parallel-cli, with a strong emphasis on academic and scientific sources. Use this skill whenever the user needs to search the web, fetch/extract URL content, enrich data with web-sourced fields, or run deep research reports. Covers: web search (fast lookups, research, current info — prioritizing peer-reviewed papers, preprints, and scholarly databases), URL extraction (fetching pages, articles, academic PDFs), bulk data enrichment (adding fields to CSV/lists from the web), and deep research (exhaustive multi-source reports grounded in academic literature). Also handles setup, status checks, and result retrieval. Use this skill for ANY web-related task — even if the user doesn't mention 'parallel' or 'web' explicitly. If they want to look something up, fetch a page, enrich a dataset, investigate a topic, find academic papers, check citations, or review scientific literature, this is the skill to use."
|
||||
compatibility: Requires parallel-cli and internet access.
|
||||
required_environment_variables: [{"name": "PARALLEL_API_KEY", "prompt": "Parallel API key.", "required_for": "full functionality"}]
|
||||
metadata: {"version": "1.1", "author": "K-Dense, Inc.", "openclaw": {"primaryEnv": "PARALLEL_API_KEY", "envVars": [{"name": "PARALLEL_API_KEY", "required": true, "description": "Parallel API key."}]}}
|
||||
description: "All-in-one web toolkit powered by Google Deep Research (Vertex AI), with a strong emphasis on academic and scientific sources. Use this skill whenever the user needs to search the web, fetch/extract URL content, or run deep research reports. Covers: web search (fast lookups, research, current info — prioritizing peer-reviewed papers, preprints, and scholarly databases), URL extraction (fetching pages, articles, academic PDFs), and deep research (exhaustive multi-source reports grounded in academic literature). Also handles status checks and result retrieval by interaction ID. Use this skill for ANY web-related task — even if the user doesn't mention 'search' or 'web' explicitly. If they want to look something up, fetch a page, investigate a topic, find academic papers, check citations, or review scientific literature, this is the skill to use."
|
||||
compatibility: Requires google-genai (pip) and Vertex AI access.
|
||||
required_environment_variables: [{"name": "GOOGLE_CLOUD_PROJECT", "prompt": "GCP project ID for Vertex AI.", "required_for": "full functionality"}, {"name": "GOOGLE_APPLICATION_CREDENTIALS", "prompt": "Path to Vertex service-account JSON (or use ADC).", "required_for": "full functionality"}]
|
||||
metadata: {"version": "2.0", "author": "K-Dense, Inc.", "openclaw": {"primaryEnv": "GOOGLE_CLOUD_PROJECT", "envVars": [{"name": "GOOGLE_CLOUD_PROJECT", "required": true, "description": "GCP project ID."}, {"name": "GOOGLE_APPLICATION_CREDENTIALS", "required": false, "description": "Service-account JSON path. Not needed if using ADC."}]}}
|
||||
---
|
||||
|
||||
# Parallel Web Toolkit
|
||||
# Google Deep Research Web Toolkit
|
||||
|
||||
A unified skill for all web-powered tasks: searching, extracting, enriching, and researching — with academic and scientific sources as the default priority.
|
||||
A unified skill for all web-powered tasks: searching, extracting, and researching — with academic and scientific sources as the default priority. Powered by Google Deep Research via Vertex AI.
|
||||
|
||||
## Routing — pick the right capability
|
||||
|
||||
Read the user's request and match it to one of the capabilities below. For web search, extract, enrichment, and deep research, read the corresponding reference file for detailed instructions.
|
||||
Read the user's request and match it to one of the capabilities below. For web search, extract, and deep research, read the corresponding reference file for detailed instructions.
|
||||
|
||||
| User wants to... | Capability | Where |
|
||||
|---|---|---|
|
||||
| Look something up, research a topic, find current info | **Web Search** | `references/web-search.md` |
|
||||
| Fetch content from a specific URL (webpage, article, PDF) | **Web Extract** | `references/web-extract.md` |
|
||||
| Add web-sourced fields to a list of companies/people/products | **Data Enrichment** | `references/data-enrichment.md` |
|
||||
| Get an exhaustive, multi-source report (user says "deep research", "exhaustive", "comprehensive") | **Deep Research** | `references/deep-research.md` |
|
||||
| Install or authenticate parallel-cli | **Setup** | Below |
|
||||
| Check status of a running research/enrichment task | **Status** | Below |
|
||||
| Retrieve completed research results by run ID | **Result** | Below |
|
||||
| Retrieve a research report by interaction ID (poll after timeout) | **Poll** | Below |
|
||||
| Install dependencies or verify auth | **Setup** | Below |
|
||||
|
||||
### Decision guide
|
||||
|
||||
- **Default to Web Search** for a single lookup, research question, or "what is X?" query. It's fast and cost-effective. When the query touches a scientific or technical topic, include academic domains (see `references/web-search.md`) to surface peer-reviewed and preprint sources alongside general results.
|
||||
- **Use Web Extract** when the user provides a URL or asks you to read/fetch a specific page. Prefer this over the built-in WebFetch tool. Particularly useful for extracting full text from academic PDFs, preprint servers, and journal articles.
|
||||
- **Use Data Enrichment** when the user has **multiple entities** (a CSV, a list of companies/people/products, or even a short inline list) and wants to find or add the same kind of information for each one. The key signal is a repeated lookup across a set of items — e.g., "find the CEO for each of these companies" or "get the founding year for Apple, Stripe, and Anthropic." Even if the user doesn't say "enrich," use `parallel-cli enrich` whenever the task is the same query applied to multiple entities. Do NOT use Web Search in a loop for this — the enrichment pipeline handles batching, parallelism, and structured output automatically.
|
||||
- **Use Deep Research only** when the user explicitly asks for deep, exhaustive, or comprehensive research. It is 10-100x slower and more expensive than Web Search — never default to it. Deep research is especially valuable for literature reviews and multi-paper synthesis.
|
||||
- If `parallel-cli` is not found when running any command, follow the Setup section below.
|
||||
- **Default to Web Search** for a single lookup, research question, or "what is X?" query. It's fast and cost-effective.
|
||||
- **Use Web Extract** when the user provides a URL or asks you to read/fetch a specific page. Particularly useful for academic PDFs, preprint servers, and journal articles.
|
||||
- **Use Deep Research only** when the user explicitly asks for deep, exhaustive, or comprehensive research. It takes **15–30 minutes** — never default to it.
|
||||
- Data enrichment (batch CSV/entity lookups) is **not available** in this skill version — there is no `enrich` command in `google_research.py`. Tell the user batch enrichment is unsupported for now. For small lists (roughly <10 rows), use **web search** per entity; for large tables, ask the user to narrow the scope or split the task.
|
||||
|
||||
### Academic source priority
|
||||
|
||||
Across all capabilities, prefer academic and scientific sources when the query is technical or scientific in nature. This means:
|
||||
Across all capabilities, prefer academic and scientific sources when the query is technical or scientific in nature:
|
||||
- Peer-reviewed journal articles and conference proceedings over blog posts or news articles
|
||||
- Preprints (arXiv, bioRxiv, medRxiv) when peer-reviewed versions aren't available
|
||||
- Institutional and government sources (NIH, WHO, NASA, NIST) over commercial sites
|
||||
- Primary research over secondary summaries
|
||||
|
||||
When citing academic sources, include author names and publication year where available (e.g., [Smith et al., 2025](url)) in addition to the standard citation format. If a DOI is present, prefer the DOI link.
|
||||
When citing academic sources, include author names and publication year where available in addition to the standard citation format.
|
||||
|
||||
## Context chaining
|
||||
### Google source mix (important)
|
||||
|
||||
Several capabilities support multi-turn context via `interaction_id`. When a research or enrichment task completes, it returns an `interaction_id`. If the user asks a follow-up question related to that task, pass `--previous-interaction-id` to carry context forward automatically. This avoids restating what was already found.
|
||||
Unlike research-curated pipelines, **Google Deep Research may return mixed sources** — peer-reviewed journals alongside supplement retailers, health blogs, news sites, and product pages. This is expected API behavior, not a script error.
|
||||
|
||||
---
|
||||
When presenting Google output to the user:
|
||||
|
||||
1. **Always include a Source Quality assessment** (see capability-specific reference files for templates).
|
||||
2. **Do not treat all sources as equal evidence** — distinguish peer-reviewed / preprint / clinical-registry sources from commercial or blog sources.
|
||||
3. **Flag thin academic coverage** — if roughly less than half of cited sources are academic or institutional, tell the user and note which claims rely mainly on non-academic sources.
|
||||
4. **Prefer evidence from academic sources** when summarizing clinical or mechanistic claims.
|
||||
|
||||
URLs in Google output are often **grounding redirect wrappers** (`vertexaisearch.cloud.google.com/grounding-api-redirect/...`). Assess source type from the **citation title and domain name** shown in the `## Sources` list, not from the redirect URL string itself.
|
||||
|
||||
## Setup
|
||||
|
||||
If `parallel-cli` is not installed, install and authenticate:
|
||||
If `google-genai` is not installed, install it:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://parallel.ai/install.sh | bash
|
||||
uv pip install google-genai
|
||||
# or: pip install google-genai
|
||||
```
|
||||
|
||||
If unable to install that way, use uv instead:
|
||||
Verify auth. The script needs `GOOGLE_CLOUD_PROJECT` and either `GOOGLE_APPLICATION_CREDENTIALS` (service-account JSON path) or Application Default Credentials (ADC via `gcloud auth application-default login`).
|
||||
|
||||
Check if a `.env` file exists in the project root containing `GOOGLE_CLOUD_PROJECT` and `GOOGLE_APPLICATION_CREDENTIALS`. If so, load it before running:
|
||||
|
||||
```bash
|
||||
uv tool install "parallel-web-tools[cli]"
|
||||
dotenv -f .env run python scripts/google_research.py search "test" --fast
|
||||
```
|
||||
|
||||
Then authenticate. First, check if a `.env` file exists in the project root and contains `PARALLEL_API_KEY`. If so, load it with `dotenv`:
|
||||
If `dotenv` isn't available: `pip install python-dotenv[cli]` or `uv pip install python-dotenv[cli]`.
|
||||
|
||||
If env vars are already in the environment, run directly:
|
||||
|
||||
```bash
|
||||
dotenv -f .env run parallel-cli auth
|
||||
python scripts/google_research.py search "test" --fast
|
||||
```
|
||||
|
||||
If `dotenv` isn't available, install it with `pip install python-dotenv[cli]` or `uv pip install python-dotenv[cli]`.
|
||||
---
|
||||
|
||||
If there's no `.env` file or it doesn't contain the key, fall back to interactive login:
|
||||
## Poll a research result by interaction ID
|
||||
|
||||
Deep research uses a **two-step flow** (see `references/deep-research.md`):
|
||||
|
||||
1. `research "$QUERY" --no-wait` — starts the job and prints `interaction_id` to stderr (seconds).
|
||||
2. `poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800` — waits for the report.
|
||||
|
||||
If poll times out, research is **still running server-side**. Wait 2–3 minutes and re-run the same `poll` command. Total time may reach 30–60 minutes.
|
||||
|
||||
```bash
|
||||
parallel-cli login
|
||||
python scripts/google_research.py poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800
|
||||
```
|
||||
|
||||
Or set the key manually: `export PARALLEL_API_KEY="your-key"`
|
||||
|
||||
Verify with:
|
||||
|
||||
```bash
|
||||
parallel-cli auth
|
||||
```
|
||||
|
||||
If `parallel-cli` is not found after install, add `~/.local/bin` to PATH.
|
||||
|
||||
## Check task status
|
||||
|
||||
```bash
|
||||
parallel-cli research status "$RUN_ID" --json
|
||||
```
|
||||
|
||||
Report the current status to the user (running, completed, failed, etc.).
|
||||
|
||||
## Get completed result
|
||||
|
||||
```bash
|
||||
parallel-cli research poll "$RUN_ID" --json
|
||||
```
|
||||
|
||||
Present results in a clear, organized format.
|
||||
Report the file path and offer to read the file if the user wants to review the results.
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
# Data Enrichment
|
||||
|
||||
Enrich: $ARGUMENTS
|
||||
|
||||
## Before starting
|
||||
|
||||
Inform the user that enrichment may take several minutes depending on the number of rows and fields requested.
|
||||
|
||||
## Step 1: Start the enrichment
|
||||
|
||||
Use ONE of these command patterns (substitute user's actual data):
|
||||
|
||||
For inline data:
|
||||
|
||||
```bash
|
||||
parallel-cli enrich run --data '[{"company": "Google"}, {"company": "Microsoft"}]' --intent "CEO name and founding year" --target "output.json" --no-wait --json
|
||||
```
|
||||
|
||||
For CSV file:
|
||||
|
||||
```bash
|
||||
parallel-cli enrich run --source-type csv --source "input.csv" --target "/tmp/output.json" --source-columns '[{"name": "company", "description": "Company name"}]' --intent "CEO name and founding year" --no-wait --json
|
||||
```
|
||||
|
||||
If this is a **follow-up** to a previous research or enrichment task where you know the `interaction_id`, add context chaining:
|
||||
|
||||
```bash
|
||||
parallel-cli enrich run --data '...' --intent "..." --target "output.json" --no-wait --json --previous-interaction-id "$INTERACTION_ID"
|
||||
```
|
||||
|
||||
By chaining `interaction_id` values across requests, each follow-up automatically has the full context of prior turns — so you can enrich entities discovered in earlier research without restating what was already found.
|
||||
|
||||
**IMPORTANT:** Always include `--no-wait` so the command returns immediately instead of blocking.
|
||||
|
||||
Parse the output to extract the `taskgroup_id`, `interaction_id`, and monitoring URL. Immediately tell the user:
|
||||
- Enrichment has been kicked off
|
||||
- The monitoring URL where they can track progress
|
||||
|
||||
Tell them they can background the polling step to continue working while it runs.
|
||||
|
||||
## Step 2: Poll for results
|
||||
|
||||
Choose a short, descriptive filename based on the enrichment task (e.g., `companies-ceos`, `startups-funding`). Use lowercase with hyphens, no spaces.
|
||||
|
||||
```bash
|
||||
parallel-cli enrich poll "$TASKGROUP_ID" --timeout 540 --json --output "$FILENAME.json"
|
||||
```
|
||||
|
||||
The `--target` flag on `enrich run` does not carry over to the poll — you must pass `--output` here to save the results. Always use `--json` to get structured JSON output.
|
||||
|
||||
Important:
|
||||
- Use `--timeout 540` (9 minutes) to stay within tool execution limits
|
||||
|
||||
### If the poll times out
|
||||
|
||||
Enrichment of large datasets can take longer than 9 minutes. If the poll exits without completing:
|
||||
1. Tell the user the enrichment is still running server-side
|
||||
2. Re-run the same `parallel-cli enrich poll` command to continue waiting
|
||||
|
||||
## Response format
|
||||
|
||||
**After step 1:** Share the monitoring URL (for tracking progress).
|
||||
|
||||
**After step 2:**
|
||||
1. Report number of rows enriched
|
||||
2. Preview first few rows of the output JSON
|
||||
3. Tell user the full path to the output JSON file (`$FILENAME.json`)
|
||||
4. Share the `interaction_id` and tell the user they can ask follow-up questions that build on this enrichment
|
||||
|
||||
Do NOT re-share the monitoring URL after completion — the results are in the output file.
|
||||
|
||||
**Remember the `interaction_id`** — if the user asks a follow-up question that relates to this enrichment, use it as `--previous-interaction-id` in the next research or enrichment command.
|
||||
@@ -4,77 +4,99 @@ Research topic: $ARGUMENTS
|
||||
|
||||
## When to use (vs web search)
|
||||
|
||||
ONLY use this capability when the user explicitly requests deep/exhaustive research. Deep research is 10-100x slower and more expensive than web search. For normal "research X" requests, quick lookups, or fact-checking, use **web search** instead.
|
||||
ONLY use this capability when the user explicitly requests deep/exhaustive research. Deep research takes **15–30 minutes** (sometimes up to 60 minutes) and is significantly more expensive than web search. For normal "research X" requests, quick lookups, or fact-checking, use **web search** instead.
|
||||
|
||||
## Step 1: Start the research
|
||||
|
||||
Frame the research objective to prioritize academic literature. If the user's query is scientific or technical, prepend context to the arguments that steers toward scholarly sources — e.g., instead of `"effects of sleep deprivation"`, use `"peer-reviewed research and clinical studies on the effects of sleep deprivation"`. This nudges the research agent toward primary literature without excluding relevant non-academic sources.
|
||||
Frame the research objective to prioritize academic literature. If the user's query is scientific or technical, prepend context that steers toward scholarly sources — e.g., instead of `"effects of sleep deprivation"`, use `"peer-reviewed research and clinical studies on the effects of sleep deprivation"`.
|
||||
|
||||
Choose a descriptive filename based on the topic (e.g., `mrna-vaccine-platforms-2026`, `gut-microbiome-depression`). Use lowercase with hyphens, no spaces.
|
||||
|
||||
```bash
|
||||
parallel-cli research run "$ARGUMENTS" --processor pro-fast --no-wait --json
|
||||
```
|
||||
|
||||
If this is a **follow-up** to a previous research or enrichment task where you know the `interaction_id`, add context chaining:
|
||||
|
||||
```bash
|
||||
parallel-cli research run "$ARGUMENTS" --processor lite --no-wait --json --previous-interaction-id "$INTERACTION_ID"
|
||||
```
|
||||
|
||||
By chaining `interaction_id` values across requests, each follow-up question automatically has the full context of prior turns — so you can drill deeper into a topic without restating what was already researched. Use `--processor lite` for follow-ups since the heavy research was already done in the initial turn and the follow-up just needs to build on that context.
|
||||
|
||||
This returns instantly. Do NOT omit `--no-wait` — without it the command blocks for minutes and will time out.
|
||||
|
||||
Processor options (choose based on user request):
|
||||
|
||||
| Processor | Expected latency | Use when |
|
||||
|-----------|-----------------|----------|
|
||||
| `pro-fast` | 30s - 5 min | Default — good balance of depth and speed |
|
||||
| `ultra-fast` | 1 - 10 min | Deeper analysis, more sources (~2x cost) |
|
||||
| `ultra` | 5 - 25 min | Maximum depth, only when explicitly requested (~3x cost) |
|
||||
|
||||
Parse the JSON output to extract the `run_id`, `interaction_id`, and monitoring URL. Immediately tell the user:
|
||||
- Deep research has been kicked off
|
||||
- The expected latency for the processor tier chosen (from the table above)
|
||||
- The monitoring URL where they can track progress
|
||||
|
||||
Tell them they can background the polling step to continue working while it runs.
|
||||
|
||||
## Step 2: Poll for results
|
||||
|
||||
Choose a descriptive filename based on the topic (e.g., `ai-chip-market-2026`, `react-vs-vue-comparison`). Use lowercase with hyphens, no spaces.
|
||||
|
||||
```bash
|
||||
parallel-cli research poll "$RUN_ID" -o "$FILENAME" --timeout 540
|
||||
python scripts/google_research.py research "$ARGUMENTS" --no-wait
|
||||
```
|
||||
|
||||
Important:
|
||||
- Use `--timeout 540` (9 minutes) to stay within tool execution limits
|
||||
- Do NOT pass `--json` — the full output is large and will flood context. The `-o` flag writes results to files instead.
|
||||
- The `-o` flag generates two output files:
|
||||
- `$FILENAME.json` — metadata and basis
|
||||
- `$FILENAME.md` — formatted markdown report
|
||||
- The poll command prints an **executive summary** to stdout when the research completes. Share this executive summary with the user — it gives them a quick overview without having to open the files.
|
||||
- **Always use `--no-wait`.** This returns in seconds with an `interaction_id`. Do **NOT** run blocking `research` without `--no-wait` — it blocks for 15–30+ minutes and will time out in the coding agent's Bash tool (~2 min limit).
|
||||
- The script prints `interaction_id=ChBi...` to stderr. **Save this ID** — you need it for Step 2.
|
||||
- Immediately tell the user:
|
||||
- Deep research has been kicked off
|
||||
- Expected latency: **15–30 minutes** (up to ~60 minutes for complex academic queries)
|
||||
- You will poll for the report in Step 2
|
||||
|
||||
Tell them they can continue other work while polling runs (you may background the poll step).
|
||||
|
||||
## Step 2: Poll for results
|
||||
|
||||
```bash
|
||||
python scripts/google_research.py poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800
|
||||
```
|
||||
|
||||
Important:
|
||||
- Use `--timeout 1800` (30 minutes) per poll invocation.
|
||||
- The poll command waits synchronously, checking status every ~10s until `completed` or timeout.
|
||||
- Do **NOT** paste the full report stdout into chat — the `-o` file is the deliverable.
|
||||
|
||||
### If the poll times out
|
||||
|
||||
Higher processor tiers can take longer than 9 minutes. If the poll exits without completing:
|
||||
1. Tell the user the research is still running server-side
|
||||
2. Re-run the same `parallel-cli research poll` command to continue waiting
|
||||
A poll timeout is **NOT a failure** — research continues server-side on Google.
|
||||
|
||||
1. Tell the user the research is still running server-side.
|
||||
2. Wait **2–3 minutes**.
|
||||
3. Re-run the **same** poll command with the same `interaction_id`:
|
||||
|
||||
```bash
|
||||
python scripts/google_research.py poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800
|
||||
```
|
||||
|
||||
Total job time may reach **30–60 minutes**. Re-run poll until the command succeeds and `$FILENAME.md` exists. Do **NOT** tell the user research failed without retrying poll at least once.
|
||||
|
||||
### If poll exits before Step 1 completed
|
||||
|
||||
If you only have stderr from Step 1 and have not polled yet, use the `interaction_id` from `[research] interaction_id=...` in Step 2.
|
||||
|
||||
## What the output looks like
|
||||
|
||||
The script writes a comprehensive markdown report to `$FILENAME.md`. The report contains:
|
||||
- Full narrative with inline `[[N]](url)` citations
|
||||
- A `## Sources` section (often 50–100+ entries for deep queries)
|
||||
|
||||
There is no separate `.json` metadata file or executive summary — the markdown is the complete deliverable.
|
||||
|
||||
**Expect mixed sources.** Google deep research often includes peer-reviewed journals *and* commercial supplement sites, health blogs, news outlets, or product pages — especially in practical/clinical sections. Do not assume every source is academic.
|
||||
|
||||
## Response format
|
||||
|
||||
**After step 1:** Share the monitoring URL (for tracking progress only — it is not the final report).
|
||||
**After Step 1:** Confirm research started; share expected latency and that you saved `interaction_id`.
|
||||
|
||||
**After step 2:**
|
||||
1. Share the **executive summary** that the poll command printed to stdout
|
||||
2. Briefly assess the source quality: how many of the cited sources are peer-reviewed journals, preprints, or institutional reports vs. news articles or blog posts. If academic coverage is thin, flag this to the user and suggest a follow-up search targeting specific databases (e.g., "I can search PubMed/arXiv specifically for more primary sources on this").
|
||||
3. Tell the user the two generated file paths:
|
||||
- `$FILENAME.md` — formatted markdown report
|
||||
- `$FILENAME.json` — metadata and basis
|
||||
4. Share the `interaction_id` and tell the user they can ask follow-up questions that build on this research (e.g., "drill deeper into X" or "compare that to Y")
|
||||
**After Step 2 (report ready):**
|
||||
|
||||
Do NOT re-share the monitoring URL after completion — the results are in the files, not at that link.
|
||||
1. **Brief summary** — relay the report's opening summary section (usually `## Summary` or the first paragraph). Do **not** paste the full report into chat.
|
||||
|
||||
Ask the user if they would like to read through the files for more detail. Do NOT read the file contents into context unless the user asks.
|
||||
2. **Source Quality** — required section. Read only the `## Sources` section (and the summary if needed for context). Do **not** read the entire report into context.
|
||||
|
||||
**Remember the `interaction_id`** — if the user asks a follow-up question that relates to this research, use it as `--previous-interaction-id` in the next research or enrichment command.
|
||||
Use this template:
|
||||
|
||||
```markdown
|
||||
### Source Quality
|
||||
|
||||
- **Total sources cited:** ~N
|
||||
- **Peer-reviewed / preprint / clinical-registry** (PubMed, PMC, Nature, MDPI, Frontiers, arXiv, ClinicalTrials.gov, university domains): ~X (~Y%)
|
||||
- **Institutional / government** (NIH, WHO, .gov, .edu pages that are not journal articles): ~X
|
||||
- **News / industry / commercial / blog** (supplement retailers, product pages, health blogs, Wikipedia, YouTube): ~X (~Y%)
|
||||
|
||||
**Assessment:** [1–2 sentences — e.g. "Core mechanistic and clinical claims are grounded in peer-reviewed literature; non-academic sources appear mainly in commercial/product sections." OR "Academic coverage is thin — consider a follow-up targeted search."]
|
||||
|
||||
**Evidence note:** [If <50% academic/institutional, flag explicitly. Note which key claims rely on academic vs non-academic sources.]
|
||||
```
|
||||
|
||||
Classification rules:
|
||||
- URLs are often grounding redirects — judge from **source titles and domain names** in the `## Sources` list (e.g. `nih.gov`, `mdpi.com`, `justthrivehealth.com`).
|
||||
- Do not read all 100+ sources line-by-line if the list is huge — sample systematically (first 20, middle 20, last 20) or grep for domain patterns, then extrapolate with a clear caveat.
|
||||
- Do not invent counts — if you cannot estimate, say "approximately" and explain your sampling method.
|
||||
|
||||
3. **Report file path** — tell the user: `$FILENAME.md`
|
||||
|
||||
4. **Offer to read more** — ask if the user wants to review the full report. **Do NOT read the file into context unless the user asks** — reports are often 5,000–10,000+ words.
|
||||
|
||||
5. **`interaction_id`** — note from stderr for potential follow-up or poll recovery.
|
||||
|
||||
@@ -4,42 +4,43 @@ Extract content from: $ARGUMENTS
|
||||
|
||||
## Command
|
||||
|
||||
Choose a short, descriptive filename based on the URL or content (e.g., `vespa-docs`, `react-hooks-api`). Use lowercase with hyphens, no spaces.
|
||||
Choose a short, descriptive filename based on the URL or content (e.g., `vespa-docs`, `attention-is-all-you-need`). Use lowercase with hyphens, no spaces.
|
||||
|
||||
```bash
|
||||
parallel-cli extract "$ARGUMENTS" --json -o "$FILENAME.json"
|
||||
python scripts/google_research.py extract "$ARGUMENTS" -o "$FILENAME.md"
|
||||
```
|
||||
|
||||
Options if needed:
|
||||
- `--objective "focus area"` to focus on specific content
|
||||
- `--objective "focus area"` to focus on specific content (highly recommended for academic papers)
|
||||
|
||||
## Academic content handling
|
||||
|
||||
When extracting from academic sources (arXiv, PubMed, journal sites, conference proceedings), use `--objective` to focus on the most valuable sections:
|
||||
When extracting from academic sources (arXiv, PubMed, journal sites, conference proceedings), use `--objective` to target the most valuable sections:
|
||||
|
||||
```bash
|
||||
parallel-cli extract "$URL" --json --objective "extract abstract, methodology, key findings, and conclusions" -o "$FILENAME.json"
|
||||
python scripts/google_research.py extract "$URL" \
|
||||
--objective "extract title, authors, publication date, abstract, methodology, key findings, and conclusions" \
|
||||
-o "$FILENAME.md"
|
||||
```
|
||||
|
||||
For arXiv papers, prefer the `/abs/` URL (which has structured metadata) over the raw PDF URL when available. If the user provides a PDF link, extract it directly — parallel-cli handles PDFs.
|
||||
For arXiv papers, prefer the `/abs/` URL (which has structured metadata) over the raw PDF URL when available. The script handles both.
|
||||
|
||||
## What the output looks like
|
||||
|
||||
The script returns a structured markdown document with:
|
||||
- Title, authors, publication date (for academic papers)
|
||||
- Abstract and key findings as prose sections
|
||||
- A `## Sources` section with the source URL
|
||||
|
||||
The output is richer than raw page scraping — it is a structured synthesis of the page content.
|
||||
|
||||
## Response format
|
||||
|
||||
Return content as:
|
||||
Present the extracted content to the user. For academic papers, confirm key metadata (title, authors, venue, date) are present before presenting the full extraction.
|
||||
|
||||
**[Page Title](URL)**
|
||||
|
||||
For academic papers, include structured metadata when available:
|
||||
- **Authors:** list of authors
|
||||
- **Published:** date and venue/journal
|
||||
- **DOI:** if available
|
||||
- **Abstract:** the paper's abstract
|
||||
|
||||
Then the extracted content verbatim, with these rules:
|
||||
- Keep content verbatim - do not paraphrase or summarize
|
||||
- Parse lists exhaustively - extract EVERY numbered/bulleted item
|
||||
- Strip only obvious noise: nav menus, footers, ads
|
||||
Keep the content faithful to the source:
|
||||
- Do not paraphrase or re-summarize the extracted text
|
||||
- Preserve all facts, names, numbers, dates, quotes
|
||||
- For academic papers, preserve figure/table captions and references
|
||||
- For academic papers, preserve figure/table captions and key numerical results
|
||||
|
||||
After the response, mention the output file path (`$FILENAME.json`) so the user knows it's available for follow-up questions.
|
||||
After presenting the content, mention the output file path (`$FILENAME.md`) so the user knows it's saved for follow-up questions.
|
||||
|
||||
@@ -7,67 +7,52 @@ Search the web for: $ARGUMENTS
|
||||
Choose a short, descriptive filename based on the query (e.g., `ai-chip-news`, `react-vs-vue`). Use lowercase with hyphens, no spaces.
|
||||
|
||||
```bash
|
||||
parallel-cli search "$ARGUMENTS" -q "<keyword1>" -q "<keyword2>" --json --max-results 10 --excerpt-max-chars-total 27000 -o "$FILENAME.json"
|
||||
python scripts/google_research.py search "$ARGUMENTS" --fast -o "$FILENAME.md"
|
||||
```
|
||||
|
||||
The first argument is the **objective** — a natural language description of what you're looking for. It replaces multiple keyword searches with a single call for broad or complex queries. Add `-q` flags for specific keyword queries to supplement the objective. The `-o` flag saves the full results to a JSON file for follow-up questions.
|
||||
`--fast` uses `gemini-2.5-flash + google_search` for a ~10s synthesized response. This is the default and recommended mode for all search queries.
|
||||
|
||||
Options if needed:
|
||||
- `--after-date YYYY-MM-DD` for time-sensitive queries
|
||||
- `--include-domains domain1.com,domain2.com` to limit to specific sources
|
||||
- `--timeout N` to extend the timeout (default 600s; rarely needed with `--fast`)
|
||||
|
||||
## What the output looks like
|
||||
|
||||
The script writes a pre-synthesized markdown report directly to `$FILENAME.md` and prints the same content to stdout. The output already contains:
|
||||
- Inline citations in `[[N]](url)` format
|
||||
- A `## Sources` section listing all referenced URLs with titles
|
||||
|
||||
**Do not re-parse or re-synthesize the output.** Present it as-is — the synthesis and citation linking are already done.
|
||||
|
||||
## Academic source strategy
|
||||
|
||||
For scientific or technical queries, run **two searches** to ensure academic sources surface alongside general results:
|
||||
For scientific or technical queries, append academic context to the query string to improve source targeting:
|
||||
|
||||
1. **Academic-focused search** — append `--include-domains` with scholarly domains:
|
||||
```bash
|
||||
parallel-cli search "$ARGUMENTS" -q "<keyword1>" --json --max-results 10 --excerpt-max-chars-total 27000 --include-domains "scholar.google.com,arxiv.org,pubmed.ncbi.nlm.nih.gov,semanticscholar.org,biorxiv.org,medrxiv.org,ncbi.nlm.nih.gov,nature.com,science.org,ieee.org,acm.org,springer.com,wiley.com,cell.com,pnas.org,nih.gov" -o "$FILENAME-academic.json"
|
||||
```
|
||||
```bash
|
||||
python scripts/google_research.py search "peer-reviewed research on $ARGUMENTS" --fast -o "$FILENAME.md"
|
||||
```
|
||||
|
||||
2. **General search** — the standard command without domain restrictions, to catch relevant non-academic sources.
|
||||
You do not need to run two separate searches. Google's search synthesis already blends academic and general sources; biasing the query string is sufficient.
|
||||
|
||||
Merge results, leading with academic sources. If only one search is practical (e.g., clearly non-scientific query), skip the academic-focused search.
|
||||
|
||||
**When to use the two-search pattern:** Any query involving scientific claims, medical information, research findings, technical mechanisms, statistical data, or anything where primary literature would be more reliable than secondary reporting.
|
||||
|
||||
## Parsing results
|
||||
|
||||
Do not set `max_output_tokens` on the command execution — the output is already bounded by `--max-results` and `--excerpt-max-chars-total`. Capping output tokens will truncate the JSON and break parsing.
|
||||
|
||||
Parse the JSON from stdout. For each result, extract:
|
||||
- title, url, publish_date
|
||||
- Useful content from excerpts (skip navigation noise like menus, footers, "Skip to content")
|
||||
**Expect mixed sources** for scientific queries — Google may cite journals alongside news sites, company pages, or blogs. Always assess quality before presenting.
|
||||
|
||||
## Response format
|
||||
|
||||
**CRITICAL: Every claim must have an inline citation.** Use markdown links pulling only from the JSON output. Never invent or guess URLs.
|
||||
1. Present the synthesized markdown output to the user directly. Keep all inline citations intact.
|
||||
|
||||
For academic sources, use author-year citation style where metadata is available:
|
||||
- Academic: [Smith et al., 2025](url) or [Smith & Jones, 2024](url)
|
||||
- Non-academic: [Source Title](url)
|
||||
2. **Source Quality** — required for scientific or technical queries. Read the `## Sources` section and narrative text.
|
||||
|
||||
Synthesize a response that:
|
||||
- Leads with findings from peer-reviewed or preprint sources when available
|
||||
- Clearly distinguishes between claims backed by primary research vs. secondary reporting
|
||||
- Includes specific facts, names, numbers, dates
|
||||
- Cites every fact inline — do not leave any claim uncited
|
||||
- Organizes by theme if multiple topics
|
||||
- Notes the evidence quality (e.g., "a randomized controlled trial found..." vs. "a blog post reports...")
|
||||
```markdown
|
||||
### Source Quality
|
||||
|
||||
**End with a Sources section** listing every URL referenced, grouped by type:
|
||||
- **Peer-reviewed / preprint / institutional:** [count or estimate]
|
||||
- **News / commercial / other:** [count or estimate]
|
||||
- **Assessment:** [1 sentence — sufficient for the query? flag if mostly non-academic]
|
||||
```
|
||||
|
||||
```
|
||||
Sources:
|
||||
- URLs are grounding redirects — assess from **source titles and domain names** (e.g. "Nature", "PubMed", "arXiv", "NIH", "WHO"), not the redirect URL string.
|
||||
- If academic/institutional sources are clearly present: note briefly.
|
||||
- If coverage appears primarily non-academic: **flag it** and offer to run a query that explicitly targets scholarly sources.
|
||||
|
||||
Academic / Peer-reviewed:
|
||||
- [Smith et al., 2025 — Title of Paper](https://doi.org/...) (Nature, 2025)
|
||||
- [Jones & Lee, 2024 — Title of Paper](https://arxiv.org/...) (arXiv preprint)
|
||||
3. End by mentioning the output file path (`$FILENAME.md`) so the user knows it's available for follow-up questions.
|
||||
|
||||
Other:
|
||||
- [Source Title](https://example.com/article) (Feb 2026)
|
||||
```
|
||||
|
||||
This Sources section is mandatory. Do not omit it. If no academic sources were found, note that and explain why (e.g., the topic is too recent, not yet studied, or inherently non-academic).
|
||||
|
||||
After the Sources section, mention the output file path (`$FILENAME.json`) so the user knows it's available for follow-up questions.
|
||||
**Sources section is already included** in the script output. Do not generate a second one. If you want to highlight specific sources by type (academic vs. general), quote from the existing `## Sources` list rather than inventing new entries.
|
||||
|
||||
1169
skills/parallel-web/scripts/google_research.py
Executable file
1169
skills/parallel-web/scripts/google_research.py
Executable file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user