Compare commits

...

6 Commits

Author SHA1 Message Date
Yuhuan He
df8fa36140 chore: replace project ID with placeholder in example 2026-07-08 15:22:45 -07:00
Yuhuan He
b0587d69a4 chore(parallel-web): remove enrichment doc and debug script snapshots
Drop data-enrichment.md (no Google equivalent); document web-search fallback
in SKILL.md. Remove pre-postprocess baseline and with-postprocess copies
from the skill deploy path.
2026-07-08 10:52:43 -07:00
Yuhuan He
73f1e5dee8 feat(parallel-web): two-step deep research (--no-wait + poll)
Align Google deep research with Parallel: kick off with --no-wait, poll separately.
Add 1800s default timeout, 120s grace retry on poll timeout, and updated skill docs.
2026-07-08 09:59:03 -07:00
Yuhuan He
a2cd6d747f feat(parallel-web): Direction A — mixed-source presentation guidance for Google output
Add structured Source Quality templates in deep-research and web-search refs;
document expected mixed sources and redirect URL assessment in SKILL.md.
2026-07-07 11:30:10 -07:00
Yuhuan He
0ce5914927 feat(parallel-web): migrate skill docs from parallel-cli to google_research.py (Step 4)
Replace all parallel-cli commands and PARALLEL_API_KEY references with
google_research.py + Vertex AI equivalents:

- SKILL.md: swap env vars, setup instructions, and routing table
- web-search.md: --fast mode (gemini-2.5-flash), pre-synthesized output
- deep-research.md: synchronous command with --timeout 1800, poll recovery
- web-extract.md: google_research.py extract with --objective support

Residual: data-enrichment.md unchanged (out of scope per step2 decision log).
2026-07-01 16:59:19 -07:00
Yuhuan He
332e043259 feat(parallel-web): add google_research.py Vertex CLI (Step 2 spike)
Adds google_research.py and two baseline variants as the Vertex-backed
replacement for parallel-cli search / research / extract.  All T1–T6
parity tests pass (step2-google-deep-research-spike.md).

Co-authored-by: K-Dense AI <engineering@k-dense.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-01 16:56:47 -07:00
6 changed files with 1346 additions and 249 deletions

View File

@@ -1,103 +1,95 @@
---
name: parallel-web
description: "All-in-one web toolkit powered by parallel-cli, with a strong emphasis on academic and scientific sources. Use this skill whenever the user needs to search the web, fetch/extract URL content, enrich data with web-sourced fields, or run deep research reports. Covers: web search (fast lookups, research, current info — prioritizing peer-reviewed papers, preprints, and scholarly databases), URL extraction (fetching pages, articles, academic PDFs), bulk data enrichment (adding fields to CSV/lists from the web), and deep research (exhaustive multi-source reports grounded in academic literature). Also handles setup, status checks, and result retrieval. Use this skill for ANY web-related task — even if the user doesn't mention 'parallel' or 'web' explicitly. If they want to look something up, fetch a page, enrich a dataset, investigate a topic, find academic papers, check citations, or review scientific literature, this is the skill to use."
compatibility: Requires parallel-cli and internet access.
required_environment_variables: [{"name": "PARALLEL_API_KEY", "prompt": "Parallel API key.", "required_for": "full functionality"}]
metadata: {"version": "1.1", "author": "K-Dense, Inc.", "openclaw": {"primaryEnv": "PARALLEL_API_KEY", "envVars": [{"name": "PARALLEL_API_KEY", "required": true, "description": "Parallel API key."}]}}
description: "All-in-one web toolkit powered by Google Deep Research (Vertex AI), with a strong emphasis on academic and scientific sources. Use this skill whenever the user needs to search the web, fetch/extract URL content, or run deep research reports. Covers: web search (fast lookups, research, current info — prioritizing peer-reviewed papers, preprints, and scholarly databases), URL extraction (fetching pages, articles, academic PDFs), and deep research (exhaustive multi-source reports grounded in academic literature). Also handles status checks and result retrieval by interaction ID. Use this skill for ANY web-related task — even if the user doesn't mention 'search' or 'web' explicitly. If they want to look something up, fetch a page, investigate a topic, find academic papers, check citations, or review scientific literature, this is the skill to use."
compatibility: Requires google-genai (pip) and Vertex AI access.
required_environment_variables: [{"name": "GOOGLE_CLOUD_PROJECT", "prompt": "GCP project ID for Vertex AI.", "required_for": "full functionality"}, {"name": "GOOGLE_APPLICATION_CREDENTIALS", "prompt": "Path to Vertex service-account JSON (or use ADC).", "required_for": "full functionality"}]
metadata: {"version": "2.0", "author": "K-Dense, Inc.", "openclaw": {"primaryEnv": "GOOGLE_CLOUD_PROJECT", "envVars": [{"name": "GOOGLE_CLOUD_PROJECT", "required": true, "description": "GCP project ID."}, {"name": "GOOGLE_APPLICATION_CREDENTIALS", "required": false, "description": "Service-account JSON path. Not needed if using ADC."}]}}
---
# Parallel Web Toolkit
# Google Deep Research Web Toolkit
A unified skill for all web-powered tasks: searching, extracting, enriching, and researching — with academic and scientific sources as the default priority.
A unified skill for all web-powered tasks: searching, extracting, and researching — with academic and scientific sources as the default priority. Powered by Google Deep Research via Vertex AI.
## Routing — pick the right capability
Read the user's request and match it to one of the capabilities below. For web search, extract, enrichment, and deep research, read the corresponding reference file for detailed instructions.
Read the user's request and match it to one of the capabilities below. For web search, extract, and deep research, read the corresponding reference file for detailed instructions.
| User wants to... | Capability | Where |
|---|---|---|
| Look something up, research a topic, find current info | **Web Search** | `references/web-search.md` |
| Fetch content from a specific URL (webpage, article, PDF) | **Web Extract** | `references/web-extract.md` |
| Add web-sourced fields to a list of companies/people/products | **Data Enrichment** | `references/data-enrichment.md` |
| Get an exhaustive, multi-source report (user says "deep research", "exhaustive", "comprehensive") | **Deep Research** | `references/deep-research.md` |
| Install or authenticate parallel-cli | **Setup** | Below |
| Check status of a running research/enrichment task | **Status** | Below |
| Retrieve completed research results by run ID | **Result** | Below |
| Retrieve a research report by interaction ID (poll after timeout) | **Poll** | Below |
| Install dependencies or verify auth | **Setup** | Below |
### Decision guide
- **Default to Web Search** for a single lookup, research question, or "what is X?" query. It's fast and cost-effective. When the query touches a scientific or technical topic, include academic domains (see `references/web-search.md`) to surface peer-reviewed and preprint sources alongside general results.
- **Use Web Extract** when the user provides a URL or asks you to read/fetch a specific page. Prefer this over the built-in WebFetch tool. Particularly useful for extracting full text from academic PDFs, preprint servers, and journal articles.
- **Use Data Enrichment** when the user has **multiple entities** (a CSV, a list of companies/people/products, or even a short inline list) and wants to find or add the same kind of information for each one. The key signal is a repeated lookup across a set of items — e.g., "find the CEO for each of these companies" or "get the founding year for Apple, Stripe, and Anthropic." Even if the user doesn't say "enrich," use `parallel-cli enrich` whenever the task is the same query applied to multiple entities. Do NOT use Web Search in a loop for this — the enrichment pipeline handles batching, parallelism, and structured output automatically.
- **Use Deep Research only** when the user explicitly asks for deep, exhaustive, or comprehensive research. It is 10-100x slower and more expensive than Web Search — never default to it. Deep research is especially valuable for literature reviews and multi-paper synthesis.
- If `parallel-cli` is not found when running any command, follow the Setup section below.
- **Default to Web Search** for a single lookup, research question, or "what is X?" query. It's fast and cost-effective.
- **Use Web Extract** when the user provides a URL or asks you to read/fetch a specific page. Particularly useful for academic PDFs, preprint servers, and journal articles.
- **Use Deep Research only** when the user explicitly asks for deep, exhaustive, or comprehensive research. It takes **1530 minutes** — never default to it.
- Data enrichment (batch CSV/entity lookups) is **not available** in this skill version — there is no `enrich` command in `google_research.py`. Tell the user batch enrichment is unsupported for now. For small lists (roughly &lt;10 rows), use **web search** per entity; for large tables, ask the user to narrow the scope or split the task.
### Academic source priority
Across all capabilities, prefer academic and scientific sources when the query is technical or scientific in nature. This means:
Across all capabilities, prefer academic and scientific sources when the query is technical or scientific in nature:
- Peer-reviewed journal articles and conference proceedings over blog posts or news articles
- Preprints (arXiv, bioRxiv, medRxiv) when peer-reviewed versions aren't available
- Institutional and government sources (NIH, WHO, NASA, NIST) over commercial sites
- Primary research over secondary summaries
When citing academic sources, include author names and publication year where available (e.g., [Smith et al., 2025](url)) in addition to the standard citation format. If a DOI is present, prefer the DOI link.
When citing academic sources, include author names and publication year where available in addition to the standard citation format.
## Context chaining
### Google source mix (important)
Several capabilities support multi-turn context via `interaction_id`. When a research or enrichment task completes, it returns an `interaction_id`. If the user asks a follow-up question related to that task, pass `--previous-interaction-id` to carry context forward automatically. This avoids restating what was already found.
Unlike research-curated pipelines, **Google Deep Research may return mixed sources** — peer-reviewed journals alongside supplement retailers, health blogs, news sites, and product pages. This is expected API behavior, not a script error.
---
When presenting Google output to the user:
1. **Always include a Source Quality assessment** (see capability-specific reference files for templates).
2. **Do not treat all sources as equal evidence** — distinguish peer-reviewed / preprint / clinical-registry sources from commercial or blog sources.
3. **Flag thin academic coverage** — if roughly less than half of cited sources are academic or institutional, tell the user and note which claims rely mainly on non-academic sources.
4. **Prefer evidence from academic sources** when summarizing clinical or mechanistic claims.
URLs in Google output are often **grounding redirect wrappers** (`vertexaisearch.cloud.google.com/grounding-api-redirect/...`). Assess source type from the **citation title and domain name** shown in the `## Sources` list, not from the redirect URL string itself.
## Setup
If `parallel-cli` is not installed, install and authenticate:
If `google-genai` is not installed, install it:
```bash
curl -fsSL https://parallel.ai/install.sh | bash
uv pip install google-genai
# or: pip install google-genai
```
If unable to install that way, use uv instead:
Verify auth. The script needs `GOOGLE_CLOUD_PROJECT` and either `GOOGLE_APPLICATION_CREDENTIALS` (service-account JSON path) or Application Default Credentials (ADC via `gcloud auth application-default login`).
Check if a `.env` file exists in the project root containing `GOOGLE_CLOUD_PROJECT` and `GOOGLE_APPLICATION_CREDENTIALS`. If so, load it before running:
```bash
uv tool install "parallel-web-tools[cli]"
dotenv -f .env run python scripts/google_research.py search "test" --fast
```
Then authenticate. First, check if a `.env` file exists in the project root and contains `PARALLEL_API_KEY`. If so, load it with `dotenv`:
If `dotenv` isn't available: `pip install python-dotenv[cli]` or `uv pip install python-dotenv[cli]`.
If env vars are already in the environment, run directly:
```bash
dotenv -f .env run parallel-cli auth
python scripts/google_research.py search "test" --fast
```
If `dotenv` isn't available, install it with `pip install python-dotenv[cli]` or `uv pip install python-dotenv[cli]`.
---
If there's no `.env` file or it doesn't contain the key, fall back to interactive login:
## Poll a research result by interaction ID
Deep research uses a **two-step flow** (see `references/deep-research.md`):
1. `research "$QUERY" --no-wait` — starts the job and prints `interaction_id` to stderr (seconds).
2. `poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800` — waits for the report.
If poll times out, research is **still running server-side**. Wait 23 minutes and re-run the same `poll` command. Total time may reach 3060 minutes.
```bash
parallel-cli login
python scripts/google_research.py poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800
```
Or set the key manually: `export PARALLEL_API_KEY="your-key"`
Verify with:
```bash
parallel-cli auth
```
If `parallel-cli` is not found after install, add `~/.local/bin` to PATH.
## Check task status
```bash
parallel-cli research status "$RUN_ID" --json
```
Report the current status to the user (running, completed, failed, etc.).
## Get completed result
```bash
parallel-cli research poll "$RUN_ID" --json
```
Present results in a clear, organized format.
Report the file path and offer to read the file if the user wants to review the results.

View File

@@ -1,72 +0,0 @@
# Data Enrichment
Enrich: $ARGUMENTS
## Before starting
Inform the user that enrichment may take several minutes depending on the number of rows and fields requested.
## Step 1: Start the enrichment
Use ONE of these command patterns (substitute user's actual data):
For inline data:
```bash
parallel-cli enrich run --data '[{"company": "Google"}, {"company": "Microsoft"}]' --intent "CEO name and founding year" --target "output.json" --no-wait --json
```
For CSV file:
```bash
parallel-cli enrich run --source-type csv --source "input.csv" --target "/tmp/output.json" --source-columns '[{"name": "company", "description": "Company name"}]' --intent "CEO name and founding year" --no-wait --json
```
If this is a **follow-up** to a previous research or enrichment task where you know the `interaction_id`, add context chaining:
```bash
parallel-cli enrich run --data '...' --intent "..." --target "output.json" --no-wait --json --previous-interaction-id "$INTERACTION_ID"
```
By chaining `interaction_id` values across requests, each follow-up automatically has the full context of prior turns — so you can enrich entities discovered in earlier research without restating what was already found.
**IMPORTANT:** Always include `--no-wait` so the command returns immediately instead of blocking.
Parse the output to extract the `taskgroup_id`, `interaction_id`, and monitoring URL. Immediately tell the user:
- Enrichment has been kicked off
- The monitoring URL where they can track progress
Tell them they can background the polling step to continue working while it runs.
## Step 2: Poll for results
Choose a short, descriptive filename based on the enrichment task (e.g., `companies-ceos`, `startups-funding`). Use lowercase with hyphens, no spaces.
```bash
parallel-cli enrich poll "$TASKGROUP_ID" --timeout 540 --json --output "$FILENAME.json"
```
The `--target` flag on `enrich run` does not carry over to the poll — you must pass `--output` here to save the results. Always use `--json` to get structured JSON output.
Important:
- Use `--timeout 540` (9 minutes) to stay within tool execution limits
### If the poll times out
Enrichment of large datasets can take longer than 9 minutes. If the poll exits without completing:
1. Tell the user the enrichment is still running server-side
2. Re-run the same `parallel-cli enrich poll` command to continue waiting
## Response format
**After step 1:** Share the monitoring URL (for tracking progress).
**After step 2:**
1. Report number of rows enriched
2. Preview first few rows of the output JSON
3. Tell user the full path to the output JSON file (`$FILENAME.json`)
4. Share the `interaction_id` and tell the user they can ask follow-up questions that build on this enrichment
Do NOT re-share the monitoring URL after completion — the results are in the output file.
**Remember the `interaction_id`** — if the user asks a follow-up question that relates to this enrichment, use it as `--previous-interaction-id` in the next research or enrichment command.

View File

@@ -4,77 +4,99 @@ Research topic: $ARGUMENTS
## When to use (vs web search)
ONLY use this capability when the user explicitly requests deep/exhaustive research. Deep research is 10-100x slower and more expensive than web search. For normal "research X" requests, quick lookups, or fact-checking, use **web search** instead.
ONLY use this capability when the user explicitly requests deep/exhaustive research. Deep research takes **1530 minutes** (sometimes up to 60 minutes) and is significantly more expensive than web search. For normal "research X" requests, quick lookups, or fact-checking, use **web search** instead.
## Step 1: Start the research
Frame the research objective to prioritize academic literature. If the user's query is scientific or technical, prepend context to the arguments that steers toward scholarly sources — e.g., instead of `"effects of sleep deprivation"`, use `"peer-reviewed research and clinical studies on the effects of sleep deprivation"`. This nudges the research agent toward primary literature without excluding relevant non-academic sources.
Frame the research objective to prioritize academic literature. If the user's query is scientific or technical, prepend context that steers toward scholarly sources — e.g., instead of `"effects of sleep deprivation"`, use `"peer-reviewed research and clinical studies on the effects of sleep deprivation"`.
Choose a descriptive filename based on the topic (e.g., `mrna-vaccine-platforms-2026`, `gut-microbiome-depression`). Use lowercase with hyphens, no spaces.
```bash
parallel-cli research run "$ARGUMENTS" --processor pro-fast --no-wait --json
```
If this is a **follow-up** to a previous research or enrichment task where you know the `interaction_id`, add context chaining:
```bash
parallel-cli research run "$ARGUMENTS" --processor lite --no-wait --json --previous-interaction-id "$INTERACTION_ID"
```
By chaining `interaction_id` values across requests, each follow-up question automatically has the full context of prior turns — so you can drill deeper into a topic without restating what was already researched. Use `--processor lite` for follow-ups since the heavy research was already done in the initial turn and the follow-up just needs to build on that context.
This returns instantly. Do NOT omit `--no-wait` — without it the command blocks for minutes and will time out.
Processor options (choose based on user request):
| Processor | Expected latency | Use when |
|-----------|-----------------|----------|
| `pro-fast` | 30s - 5 min | Default — good balance of depth and speed |
| `ultra-fast` | 1 - 10 min | Deeper analysis, more sources (~2x cost) |
| `ultra` | 5 - 25 min | Maximum depth, only when explicitly requested (~3x cost) |
Parse the JSON output to extract the `run_id`, `interaction_id`, and monitoring URL. Immediately tell the user:
- Deep research has been kicked off
- The expected latency for the processor tier chosen (from the table above)
- The monitoring URL where they can track progress
Tell them they can background the polling step to continue working while it runs.
## Step 2: Poll for results
Choose a descriptive filename based on the topic (e.g., `ai-chip-market-2026`, `react-vs-vue-comparison`). Use lowercase with hyphens, no spaces.
```bash
parallel-cli research poll "$RUN_ID" -o "$FILENAME" --timeout 540
python scripts/google_research.py research "$ARGUMENTS" --no-wait
```
Important:
- Use `--timeout 540` (9 minutes) to stay within tool execution limits
- Do NOT pass `--json` — the full output is large and will flood context. The `-o` flag writes results to files instead.
- The `-o` flag generates two output files:
- `$FILENAME.json` — metadata and basis
- `$FILENAME.md` — formatted markdown report
- The poll command prints an **executive summary** to stdout when the research completes. Share this executive summary with the user — it gives them a quick overview without having to open the files.
- **Always use `--no-wait`.** This returns in seconds with an `interaction_id`. Do **NOT** run blocking `research` without `--no-wait` — it blocks for 1530+ minutes and will time out in the coding agent's Bash tool (~2 min limit).
- The script prints `interaction_id=ChBi...` to stderr. **Save this ID** — you need it for Step 2.
- Immediately tell the user:
- Deep research has been kicked off
- Expected latency: **1530 minutes** (up to ~60 minutes for complex academic queries)
- You will poll for the report in Step 2
Tell them they can continue other work while polling runs (you may background the poll step).
## Step 2: Poll for results
```bash
python scripts/google_research.py poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800
```
Important:
- Use `--timeout 1800` (30 minutes) per poll invocation.
- The poll command waits synchronously, checking status every ~10s until `completed` or timeout.
- Do **NOT** paste the full report stdout into chat — the `-o` file is the deliverable.
### If the poll times out
Higher processor tiers can take longer than 9 minutes. If the poll exits without completing:
1. Tell the user the research is still running server-side
2. Re-run the same `parallel-cli research poll` command to continue waiting
A poll timeout is **NOT a failure** — research continues server-side on Google.
1. Tell the user the research is still running server-side.
2. Wait **23 minutes**.
3. Re-run the **same** poll command with the same `interaction_id`:
```bash
python scripts/google_research.py poll "$INTERACTION_ID" -o "$FILENAME.md" --timeout 1800
```
Total job time may reach **3060 minutes**. Re-run poll until the command succeeds and `$FILENAME.md` exists. Do **NOT** tell the user research failed without retrying poll at least once.
### If poll exits before Step 1 completed
If you only have stderr from Step 1 and have not polled yet, use the `interaction_id` from `[research] interaction_id=...` in Step 2.
## What the output looks like
The script writes a comprehensive markdown report to `$FILENAME.md`. The report contains:
- Full narrative with inline `[[N]](url)` citations
- A `## Sources` section (often 50100+ entries for deep queries)
There is no separate `.json` metadata file or executive summary — the markdown is the complete deliverable.
**Expect mixed sources.** Google deep research often includes peer-reviewed journals *and* commercial supplement sites, health blogs, news outlets, or product pages — especially in practical/clinical sections. Do not assume every source is academic.
## Response format
**After step 1:** Share the monitoring URL (for tracking progress only — it is not the final report).
**After Step 1:** Confirm research started; share expected latency and that you saved `interaction_id`.
**After step 2:**
1. Share the **executive summary** that the poll command printed to stdout
2. Briefly assess the source quality: how many of the cited sources are peer-reviewed journals, preprints, or institutional reports vs. news articles or blog posts. If academic coverage is thin, flag this to the user and suggest a follow-up search targeting specific databases (e.g., "I can search PubMed/arXiv specifically for more primary sources on this").
3. Tell the user the two generated file paths:
- `$FILENAME.md` — formatted markdown report
- `$FILENAME.json` — metadata and basis
4. Share the `interaction_id` and tell the user they can ask follow-up questions that build on this research (e.g., "drill deeper into X" or "compare that to Y")
**After Step 2 (report ready):**
Do NOT re-share the monitoring URL after completion — the results are in the files, not at that link.
1. **Brief summary** — relay the report's opening summary section (usually `## Summary` or the first paragraph). Do **not** paste the full report into chat.
Ask the user if they would like to read through the files for more detail. Do NOT read the file contents into context unless the user asks.
2. **Source Quality** — required section. Read only the `## Sources` section (and the summary if needed for context). Do **not** read the entire report into context.
**Remember the `interaction_id`** — if the user asks a follow-up question that relates to this research, use it as `--previous-interaction-id` in the next research or enrichment command.
Use this template:
```markdown
### Source Quality
- **Total sources cited:** ~N
- **Peer-reviewed / preprint / clinical-registry** (PubMed, PMC, Nature, MDPI, Frontiers, arXiv, ClinicalTrials.gov, university domains): ~X (~Y%)
- **Institutional / government** (NIH, WHO, .gov, .edu pages that are not journal articles): ~X
- **News / industry / commercial / blog** (supplement retailers, product pages, health blogs, Wikipedia, YouTube): ~X (~Y%)
**Assessment:** [12 sentences — e.g. "Core mechanistic and clinical claims are grounded in peer-reviewed literature; non-academic sources appear mainly in commercial/product sections." OR "Academic coverage is thin — consider a follow-up targeted search."]
**Evidence note:** [If &lt;50% academic/institutional, flag explicitly. Note which key claims rely on academic vs non-academic sources.]
```
Classification rules:
- URLs are often grounding redirects — judge from **source titles and domain names** in the `## Sources` list (e.g. `nih.gov`, `mdpi.com`, `justthrivehealth.com`).
- Do not read all 100+ sources line-by-line if the list is huge — sample systematically (first 20, middle 20, last 20) or grep for domain patterns, then extrapolate with a clear caveat.
- Do not invent counts — if you cannot estimate, say "approximately" and explain your sampling method.
3. **Report file path** — tell the user: `$FILENAME.md`
4. **Offer to read more** — ask if the user wants to review the full report. **Do NOT read the file into context unless the user asks** — reports are often 5,00010,000+ words.
5. **`interaction_id`** — note from stderr for potential follow-up or poll recovery.

View File

@@ -4,42 +4,43 @@ Extract content from: $ARGUMENTS
## Command
Choose a short, descriptive filename based on the URL or content (e.g., `vespa-docs`, `react-hooks-api`). Use lowercase with hyphens, no spaces.
Choose a short, descriptive filename based on the URL or content (e.g., `vespa-docs`, `attention-is-all-you-need`). Use lowercase with hyphens, no spaces.
```bash
parallel-cli extract "$ARGUMENTS" --json -o "$FILENAME.json"
python scripts/google_research.py extract "$ARGUMENTS" -o "$FILENAME.md"
```
Options if needed:
- `--objective "focus area"` to focus on specific content
- `--objective "focus area"` to focus on specific content (highly recommended for academic papers)
## Academic content handling
When extracting from academic sources (arXiv, PubMed, journal sites, conference proceedings), use `--objective` to focus on the most valuable sections:
When extracting from academic sources (arXiv, PubMed, journal sites, conference proceedings), use `--objective` to target the most valuable sections:
```bash
parallel-cli extract "$URL" --json --objective "extract abstract, methodology, key findings, and conclusions" -o "$FILENAME.json"
python scripts/google_research.py extract "$URL" \
--objective "extract title, authors, publication date, abstract, methodology, key findings, and conclusions" \
-o "$FILENAME.md"
```
For arXiv papers, prefer the `/abs/` URL (which has structured metadata) over the raw PDF URL when available. If the user provides a PDF link, extract it directly — parallel-cli handles PDFs.
For arXiv papers, prefer the `/abs/` URL (which has structured metadata) over the raw PDF URL when available. The script handles both.
## What the output looks like
The script returns a structured markdown document with:
- Title, authors, publication date (for academic papers)
- Abstract and key findings as prose sections
- A `## Sources` section with the source URL
The output is richer than raw page scraping — it is a structured synthesis of the page content.
## Response format
Return content as:
Present the extracted content to the user. For academic papers, confirm key metadata (title, authors, venue, date) are present before presenting the full extraction.
**[Page Title](URL)**
For academic papers, include structured metadata when available:
- **Authors:** list of authors
- **Published:** date and venue/journal
- **DOI:** if available
- **Abstract:** the paper's abstract
Then the extracted content verbatim, with these rules:
- Keep content verbatim - do not paraphrase or summarize
- Parse lists exhaustively - extract EVERY numbered/bulleted item
- Strip only obvious noise: nav menus, footers, ads
Keep the content faithful to the source:
- Do not paraphrase or re-summarize the extracted text
- Preserve all facts, names, numbers, dates, quotes
- For academic papers, preserve figure/table captions and references
- For academic papers, preserve figure/table captions and key numerical results
After the response, mention the output file path (`$FILENAME.json`) so the user knows it's available for follow-up questions.
After presenting the content, mention the output file path (`$FILENAME.md`) so the user knows it's saved for follow-up questions.

View File

@@ -7,67 +7,52 @@ Search the web for: $ARGUMENTS
Choose a short, descriptive filename based on the query (e.g., `ai-chip-news`, `react-vs-vue`). Use lowercase with hyphens, no spaces.
```bash
parallel-cli search "$ARGUMENTS" -q "<keyword1>" -q "<keyword2>" --json --max-results 10 --excerpt-max-chars-total 27000 -o "$FILENAME.json"
python scripts/google_research.py search "$ARGUMENTS" --fast -o "$FILENAME.md"
```
The first argument is the **objective** — a natural language description of what you're looking for. It replaces multiple keyword searches with a single call for broad or complex queries. Add `-q` flags for specific keyword queries to supplement the objective. The `-o` flag saves the full results to a JSON file for follow-up questions.
`--fast` uses `gemini-2.5-flash + google_search` for a ~10s synthesized response. This is the default and recommended mode for all search queries.
Options if needed:
- `--after-date YYYY-MM-DD` for time-sensitive queries
- `--include-domains domain1.com,domain2.com` to limit to specific sources
- `--timeout N` to extend the timeout (default 600s; rarely needed with `--fast`)
## What the output looks like
The script writes a pre-synthesized markdown report directly to `$FILENAME.md` and prints the same content to stdout. The output already contains:
- Inline citations in `[[N]](url)` format
- A `## Sources` section listing all referenced URLs with titles
**Do not re-parse or re-synthesize the output.** Present it as-is — the synthesis and citation linking are already done.
## Academic source strategy
For scientific or technical queries, run **two searches** to ensure academic sources surface alongside general results:
For scientific or technical queries, append academic context to the query string to improve source targeting:
1. **Academic-focused search** — append `--include-domains` with scholarly domains:
```bash
parallel-cli search "$ARGUMENTS" -q "<keyword1>" --json --max-results 10 --excerpt-max-chars-total 27000 --include-domains "scholar.google.com,arxiv.org,pubmed.ncbi.nlm.nih.gov,semanticscholar.org,biorxiv.org,medrxiv.org,ncbi.nlm.nih.gov,nature.com,science.org,ieee.org,acm.org,springer.com,wiley.com,cell.com,pnas.org,nih.gov" -o "$FILENAME-academic.json"
```
```bash
python scripts/google_research.py search "peer-reviewed research on $ARGUMENTS" --fast -o "$FILENAME.md"
```
2. **General search** — the standard command without domain restrictions, to catch relevant non-academic sources.
You do not need to run two separate searches. Google's search synthesis already blends academic and general sources; biasing the query string is sufficient.
Merge results, leading with academic sources. If only one search is practical (e.g., clearly non-scientific query), skip the academic-focused search.
**When to use the two-search pattern:** Any query involving scientific claims, medical information, research findings, technical mechanisms, statistical data, or anything where primary literature would be more reliable than secondary reporting.
## Parsing results
Do not set `max_output_tokens` on the command execution — the output is already bounded by `--max-results` and `--excerpt-max-chars-total`. Capping output tokens will truncate the JSON and break parsing.
Parse the JSON from stdout. For each result, extract:
- title, url, publish_date
- Useful content from excerpts (skip navigation noise like menus, footers, "Skip to content")
**Expect mixed sources** for scientific queries — Google may cite journals alongside news sites, company pages, or blogs. Always assess quality before presenting.
## Response format
**CRITICAL: Every claim must have an inline citation.** Use markdown links pulling only from the JSON output. Never invent or guess URLs.
1. Present the synthesized markdown output to the user directly. Keep all inline citations intact.
For academic sources, use author-year citation style where metadata is available:
- Academic: [Smith et al., 2025](url) or [Smith & Jones, 2024](url)
- Non-academic: [Source Title](url)
2. **Source Quality** — required for scientific or technical queries. Read the `## Sources` section and narrative text.
Synthesize a response that:
- Leads with findings from peer-reviewed or preprint sources when available
- Clearly distinguishes between claims backed by primary research vs. secondary reporting
- Includes specific facts, names, numbers, dates
- Cites every fact inline — do not leave any claim uncited
- Organizes by theme if multiple topics
- Notes the evidence quality (e.g., "a randomized controlled trial found..." vs. "a blog post reports...")
```markdown
### Source Quality
**End with a Sources section** listing every URL referenced, grouped by type:
- **Peer-reviewed / preprint / institutional:** [count or estimate]
- **News / commercial / other:** [count or estimate]
- **Assessment:** [1 sentence — sufficient for the query? flag if mostly non-academic]
```
```
Sources:
- URLs are grounding redirects — assess from **source titles and domain names** (e.g. "Nature", "PubMed", "arXiv", "NIH", "WHO"), not the redirect URL string.
- If academic/institutional sources are clearly present: note briefly.
- If coverage appears primarily non-academic: **flag it** and offer to run a query that explicitly targets scholarly sources.
Academic / Peer-reviewed:
- [Smith et al., 2025 — Title of Paper](https://doi.org/...) (Nature, 2025)
- [Jones & Lee, 2024 — Title of Paper](https://arxiv.org/...) (arXiv preprint)
3. End by mentioning the output file path (`$FILENAME.md`) so the user knows it's available for follow-up questions.
Other:
- [Source Title](https://example.com/article) (Feb 2026)
```
This Sources section is mandatory. Do not omit it. If no academic sources were found, note that and explain why (e.g., the topic is too recent, not yet studied, or inherently non-academic).
After the Sources section, mention the output file path (`$FILENAME.json`) so the user knows it's available for follow-up questions.
**Sources section is already included** in the script output. Do not generate a second one. If you want to highlight specific sources by type (academic vs. general), quote from the existing `## Sources` list rather than inventing new entries.

File diff suppressed because it is too large Load Diff