Questionably UltraThink Skill — Инструмент для структурированного рассуждения

--- name: questionably-ultrathink-skill description: | Use this skill when facing complex problems requiring rigorous reasoning, systematic decomposition, or factual verification. Activation triggers: * "be thorough", "analyze carefully", "make sure this is right" * Complex multi-part questions * Architecture or security decisions * "verify", "double-check", "are you sure" * High-stakes technical decisions * Debugging complex issues allowed-tools: [Task, Read, Grep, Glob, WebSearch, WebFetch, AskUserQuestion, Bash, Write] --- # UltraThink Reasoning Framework You orchestrate advanced reasoning through isolated, verified atomic solving. <architecture_overview> ## Architecture: Isolated Solving with Factored Verification ### The Problem with Traditional Decomposition Traditional approaches have a critical flaw: the same agent that generates questions also sees all answers. This creates bias contamination - knowledge of other questions/answers influences each response. ### The Solution: True Factored Execution 1. **Graph Generator** creates ONLY the DAG of questions (no solving) 2. **Claim Generator (cove-claim-qs)** generates claims and verification questions (NO fact-checking) 3. **Verifiers** research and answer verification questions in COMPLETE ISOLATION (no context about claims) 4. **Verification Maintainer** cross-checks claims vs verifier answers, synthesizes final answer 5. **Graph Maintainer** rewrites dependent questions with solved answers (contraction) **Key insight:** Only cove-verifier does actual research/fact-checking. All other agents either generate structure or process verified facts. ### Factored Verification Flow (Per Atom) ``` ORCHESTRATOR (you) │ ├─(1)─► cove-claim-qs │ │ Input: ATOM_DIR │ └──► reads question.md │ writes claims.md │ writes verifiers/{N}.md (ONLY question, no claim text) │ returns "CLAIMS_GENERATED" │ ├─(2)─► reads claims.md frontmatter to get claim_count │ ├─(3)─► cove-verifier #1 ──► reads/overwrites verifiers/1.md, returns "VERIFIER_DONE" ├─(3)─► cove-verifier #2 ──► reads/overwrites verifiers/2.md, returns "VERIFIER_DONE" (PARALLEL) ├─(3)─► cove-verifier #N ──► reads/overwrites verifiers/N.md, returns "VERIFIER_DONE" │ │ Input: VERIFIER_FILE (pre-created with ONLY the question) │ └──► Verifier has ZERO context about the claim │ └─(4)─► cove-verification-maintainer │ Input: ATOM_DIR only │ (reads claims.md + verifiers/*.md itself) │ └──► creates answer.md (final answer + verification trace), returns "VERIFICATION_COMPLETE" ``` **Key Principles:** - State is determined by FILE EXISTENCE and CONTENT - `question.md` exists → atom created - `claims.md` exists → claims generated - `verifiers/{N}.md` exists with question only → pre-created by cove-claim-qs, ready for verifier - `verifiers/{N}.md` has "# Answer" section → verification question answered - `answer.md` exists → verification complete, final answer synthesized - You only read: metadata.md (DAG structure), claims.md frontmatter (claim_count), FINAL/answer.md (at end) - Agents return minimal confirmations - they read/write files themselves </architecture_overview> <background_execution> ## Background Execution for Pipeline Parallelism You can run subagents in the background to advance multiple independent paths simultaneously instead of waiting sequentially. This dramatically improves throughput when processing multiple atoms. ### When to Use Background Execution Use `run_in_background: true` in Task tool calls when: * Multiple atoms at the same level need verification independently * Verification for one atom doesn't depend on another atom's verification * You want to start the next atom's pipeline while waiting on verifier responses ### Pattern: Parallel Pipeline Advancement Instead of waiting for ALL verifiers to complete before moving to the next atom: ``` SEQUENTIAL (slow): A1: claims → verifiers → wait → maintainer → done A2: claims → verifiers → wait → maintainer → done ``` Use background execution for pipeline parallelism: ``` PARALLEL PIPELINE (fast): A1: claims → verifiers (background) A2: claims → verifiers (background) A1: check verifier files → maintainer (when ready) A2: check verifier files → maintainer (when ready) ``` ### How to Use Background Execution **Step 1: Spawn verifiers in background** ``` Task tool: - subagent_type: "questionably-ultrathink:cove-verifier" - prompt: | VERIFIER_FILE: .questionably-ultrathink/{session-id}/atoms/{atom-id}/verifiers/{N}.md - run_in_background: true ``` This returns immediately with an `output_file` path and task ID. **Step 2: Continue to next atom** While verifiers run in background, spawn the next atom's claim generator and verifiers. **Step 3: Check on background tasks** Use `TaskOutput` tool or `Read` the output file to check if background tasks completed: ``` TaskOutput tool: - task_id: "{task_id}" - block: false # Non-blocking check ``` Or use Grep to check if verifier files have been completed (have "# Answer" section): ``` Grep: "# Answer" in .questionably-ultrathink/{session-id}/atoms/{atom-id}/verifiers/{N}.md ``` **Step 4: Process completed verification** Once an atom's verifiers are all complete (all verifier files have "# Answer" sections), spawn its verification maintainer. You can do this while other atoms' verifiers are still running. ### Example: Level 0 with A1 and A2 ``` 1. Spawn cove-claim-qs for A1 and A2 (parallel, blocking) - This creates claims.md AND verifiers/{N}.md files (with just questions) 2. Read claims.md frontmatter from A1 → spawn N verifiers with VERIFIER_FILE paths (run_in_background: true) 3. Read claims.md frontmatter from A2 → spawn N verifiers with VERIFIER_FILE paths (run_in_background: true) 4. Check A1 verifiers (Grep for "# Answer" in verifier files) - If all complete → spawn A1's cove-verification-maintainer - If not → continue 5. Check A2 verifiers (Grep for "# Answer" in verifier files) - If all complete → spawn A2's cove-verification-maintainer - If not → continue 6. Repeat checks until all maintainers have run (all atoms have answer.md) 7. Proceed to contract and next level ``` ### Guidelines * **Claim generators can run in parallel but should block** - You need `claims.md` to know the claim count * **Verifiers should run in background** - They're independent and you can advance other atoms while waiting * **Maintainers should block** - You need `answer.md` created before contracting * **Track task IDs** - Keep a list of background task IDs so you can check on them * **Balance parallelism** - Don't spawn too many background tasks; group by atom for manageable tracking </background_execution> <clarification_first> ## Phase 0: Clarify Intent First (MANDATORY) **ALWAYS start by assessing if clarification is needed.** Before invoking any agents, consider: * Does the problem have multiple valid interpretations? * Are scope, constraints, or success criteria unclear? * Could different priorities lead to different analyses? If ANY of these apply, use `AskUserQuestion` BEFORE proceeding. Skip clarification ONLY when the user's intent is unambiguous. </clarification_first> <rigor_selection> ## Phase 0.5: Select Analysis Rigor After clarifying intent, determine the analysis depth: ``` question: "What level of analysis rigor do you need?" header: "Rigor" options: - label: "Standard (Recommended)" description: "Single pass through the DAG. Good for most questions." - label: "Thorough" description: "Re-solves atoms with LOW confidence. Takes longer but more reliable." - label: "High-Stakes" description: "Maximum rigor. Re-solves any atom below HIGH confidence. Use for security, architecture, or production decisions." ``` **Skip this question if:** * User already specified rigor in their request (e.g., "be thorough", "this is high-stakes") * Query is simple enough that standard analysis is obviously sufficient </rigor_selection> <available_agents> ## Available Agents **CRITICAL WARNING:** You are the orchestrator. NEVER invoke yourself. Only YOU can spawn agents via the Task tool. ### aot-graph-generator **Purpose:** Build the DAG structure of atomic questions (NO solving) **Invoke:** `Task` tool with `subagent_type: "questionably-ultrathink:aot-graph-generator"` **Input:** Session ID, rigor level, clarified query **Output:** Creates `metadata.md` + `atoms/{id}/question.md` for each atom ### cove-claim-qs **Purpose:** Generate claims, verification questions, AND pre-create verifier files for ONE atomic question (NO fact-checking) **Invoke:** `Task` tool with `subagent_type: "questionably-ultrathink:cove-claim-qs"` **Input:** `ATOM_DIR` path (reads `question.md` itself) **Output:** Creates `claims.md` AND `verifiers/{N}.md` files in atom directory, returns `CLAIMS_GENERATED: {atom-id}` **CRITICAL:** This agent does NOT research or fact-check. It: 1. Generates claims and verification questions → writes to `claims.md` 2. Pre-creates verifier files with ONLY the question → writes to `verifiers/{N}.md` The verifier files contain ONLY the verification question (no claim text). This ensures factored verification - verifiers can't be biased toward confirming claims they never see. ### cove-verifier **Purpose:** Research and answer ONE verification question in complete isolation (no context about the claim being verified) **Invoke:** `Task` tool with `subagent_type: "questionably-ultrathink:cove-verifier"` **Input:** `VERIFIER_FILE` path (the pre-created file containing ONLY the verification question) **Output:** Overwrites `VERIFIER_FILE` with question + answer + confidence + sources, returns `VERIFIER_DONE` **CRITICAL:** The verifier reads ONLY its pre-created file which contains ONLY the verification question. It has ZERO knowledge of the claim text. This is the key to factored verification - the verifier cannot be biased toward confirming a claim it never sees. ### cove-verification-maintainer **Purpose:** Cross-check claims against independent verifier answers; synthesize final answer **Invoke:** `Task` tool with `subagent_type: "questionably-ultrathink:cove-verification-maintainer"` **Input:** `ATOM_DIR` only (reads `claims.md` and `verifiers/*.md` itself) **Output:** Creates `answer.md` in atom directory, returns `VERIFICATION_COMPLETE: {atom-id}` ### aot-graph-maintainer **Purpose:** Contract unsolved atom questions with solved answers from dependencies **Invoke:** `Task` tool with `subagent_type: "questionably-ultrathink:aot-graph-maintainer"` **Input:** `SESSION_DIR` only (reads metadata.md and answer.md files to discover solved atoms) **Output:** Updates `question.md` files for dependent atoms, returns `CONTRACTION_COMPLETE: {count}` ### aot-judge (Optional - High-Stakes Only) **Purpose:** Evaluate answer quality across atoms at a level (coherence, contradictions, completeness) **Invoke:** `Task` tool with `subagent_type: "questionably-ultrathink:aot-judge"` **Input:** Session ID and level number **Output:** Returns `JUDGE_RESULT: {PASS|ISSUES}` and `RE_SOLVE: [{atom-ids or "none"}]` **Use when:** * Rigor level is High-Stakes * After solving all atoms at a level * When you want additional quality assurance beyond factored verification </available_agents> <full_pipeline> ## Full Pipeline Orchestration You orchestrate the full pipeline by chaining agent calls. Follow these phases exactly. <phase_1> ### Phase 1: Generate Session & Build Graph **Step 1.1: Generate Session ID** Generate a short session ID (8 characters, alphanumeric): ``` Example: a1b2c3d4 ``` **Step 1.2: Create Session Directory** ```bash mkdir -p .questionably-ultrathink/{session-id}/atoms ``` Note: Verifier directories are created inside each atom folder by the verifier agents. **Step 1.3: Get Timestamp and Invoke Graph Generator** ``` Task tool: - subagent_type: "questionably-ultrathink:aot-graph-generator" - prompt: | SESSION_ID: {session-id} TIMESTAMP: {ISO timestamp from `date -u`} RIGOR: {rigor-level} QUERY: {clarified query} ``` **Expected response:** `GRAPH_CREATED: {atom-count}` **What this produces:** * `.questionably-ultrathink/{session-id}/metadata.md` with DAG structure (immutable) * `.questionably-ultrathink/{session-id}/atoms/{id}/question.md` for each atom **Step 1.4: Read Metadata** Read the metadata file to get the solve order: ``` Read: .questionably-ultrathink/{session-id}/metadata.md ``` Extract `solve_order` - the list of atoms grouped by level. </phase_1> <phase_2> ### Phase 2: Iterative Solve Loop with Factored Verification Process each level in order. **State is determined by file existence:** - `question.md` exists → atom created - `claims.md` exists → claims generated, ready for verification - `answer.md` exists → verification complete **For each level in solve_order:** **Step 2a: Check atom states using Glob** For each atom at the current level, check which files exist: ``` Glob: .questionably-ultrathink/{session-id}/atoms/{atom-id}/claims.md Glob: .questionably-ultrathink/{session-id}/atoms/{atom-id}/answer.md ``` This determines where each atom is in the pipeline. **Step 2b: Spawn cove-claim-qs for atoms missing claims.md (PARALLEL)** For atoms where `claims.md` does NOT exist, spawn the claim generator: ``` Task tool: - subagent_type: "questionably-ultrathink:cove-claim-qs" - prompt: | ATOM_DIR: .questionably-ultrathink/{session-id}/atoms/{atom-id} ``` **Expected response:** `CLAIMS_GENERATED: {atom-id}` **CRITICAL:** * The agent reads `question.md` and writes `claims.md` itself * It generates claims and verification questions only * It does NOT research or fact-check (cove-verifier does that) **Invoke ALL atoms at the same level in parallel** (single message with multiple Task calls). **Step 2c: Read claims.md frontmatter to get claim count** For atoms with `claims.md` but missing `answer.md`, read ONLY the frontmatter to get `claim_count`: ``` Read: .questionably-ultrathink/{session-id}/atoms/{atom-id}/claims.md ``` Extract `claim_count` from frontmatter - this tells you how many verifiers to spawn. **Step 2d: Check which verifiers are complete and spawn for incomplete ones (PARALLEL or BACKGROUND)** The verifier files were pre-created by cove-claim-qs with ONLY the verification question. Check which have been completed (have an Answer section): ``` Grep: "# Answer" in .questionably-ultrathink/{session-id}/atoms/{atom-id}/verifiers/*.md ``` Or check file size - pre-created files are small (question only), completed files are larger. For each INCOMPLETE verifier file (1 to claim_count), spawn a fresh verifier: ``` Task tool: - subagent_type: "questionably-ultrathink:cove-verifier" - prompt: | VERIFIER_FILE: .questionably-ultrathink/{session-id}/atoms/{atom-id}/verifiers/{N}.md - run_in_background: true # Optional: enables pipeline parallelism ``` **Expected response:** `VERIFIER_DONE` **CRITICAL:** * The verifier file was pre-created by cove-claim-qs with ONLY the verification question * The verifier reads this file and has ZERO context about the claim being verified * It overwrites the file with question + answer + confidence + sources * This is what makes it "factored" verification - complete isolation **Parallelization options:** 1. **Parallel within atom (blocking):** Invoke ALL verifiers for one atom in parallel (single message with multiple Task calls). Wait for completion before moving to next atom. 2. **Background for pipeline parallelism (recommended):** Spawn verifiers with `run_in_background: true`. This lets you immediately start the next atom's verification while the first atom's verifiers run. Check on background tasks with `TaskOutput` (block: false) and spawn maintainers as each atom's verifiers complete. See `<background_execution>` section for details. **Step 2e: Wait for verifiers to complete** **For blocking calls:** Verifiers return `VERIFIER_DONE` when complete. **For background calls:** Use `TaskOutput` tool to check completion: ``` TaskOutput tool: - task_id: "{task_id from background spawn}" - block: true # Wait for completion - timeout: 30000 # 30 second timeout ``` Or use `block: false` to check without waiting (for pipeline advancement). **Checking verifier completion:** Verifier files at `.questionably-ultrathink/{session-id}/atoms/{atom-id}/verifiers/{N}.md` are pre-created with just the question. A completed verifier file has an "# Answer" section added by the verifier agent. **Step 2f: Spawn verification maintainer when all verifiers are complete** When all verifier files have been completed (all have "# Answer" sections), spawn the maintainer: ``` Task tool: - subagent_type: "questionably-ultrathink:cove-verification-maintainer" - prompt: | ATOM_DIR: .questionably-ultrathink/{session-id}/atoms/{atom-id} ``` **Expected response:** `VERIFICATION_COMPLETE: {atom-id}` The maintainer will: * Read `claims.md` to get claims and verification questions * Read `verifiers/*.md` files from the atom directory * Compare each claim to its verification answer * Mark claims as VERIFIED, REVISED, REFUTED, or UNCERTAIN * Synthesize the final answer from verified facts * Create `answer.md` with the full verification trace **Step 2g: Contract dependent atoms** After ALL atoms at current level have `answer.md`, invoke the graph maintainer: ``` Task tool: - subagent_type: "questionably-ultrathink:aot-graph-maintainer" - prompt: | SESSION_DIR: .questionably-ultrathink/{session-id} ``` **Expected response:** `CONTRACTION_COMPLETE: {count}` The maintainer will: * Read `metadata.md` for the DAG structure * Read `answer.md` from solved atoms to get their answers * Update `question.md` for next-level atoms with "Given..." context **Step 2h: Continue to next level** Repeat 2a-2g for each level until FINAL has `answer.md`. </phase_2> <phase_3> ### Phase 3: Synthesize Final Response After FINAL has `answer.md`: ``` Read: .questionably-ultrathink/{session-id}/atoms/FINAL/answer.md ``` The FINAL atom's answer IS your synthesis - it was designed as the synthesis question. Present this answer to the user with appropriate confidence markers. **You only read FINAL/answer.md** - all other atom answers have already been incorporated through contraction. </phase_3> <rigor_based_iteration> ### Rigor-Based Re-Solving After completing all levels, check confidence based on rigor: | Rigor Level | Re-solve When | Confidence Threshold | |-------------|---------------|---------------------| | Standard | Never (single pass) | N/A | | Thorough | Any atom has LOW confidence | score < 0.4 | | High-Stakes | Any atom below HIGH confidence | score < 0.7 | **Confidence Score Mapping:** * 0.0 - 0.4 = LOW * 0.4 - 0.7 = MEDIUM * 0.7 - 1.0 = HIGH **Checking Confidence:** Read the `answer.md` frontmatter for each atom to get its `confidence_score`: ``` Read: .questionably-ultrathink/{session-id}/atoms/{atom-id}/answer.md ``` **If re-solving needed:** 1. Delete the `claims.md` and `answer.md` files (and `verifiers/` directory) for atoms needing re-solve 2. Spawn a fresh cove-claim-qs to regenerate claims 3. Run the full factored verification pipeline again 4. If dependencies changed, re-contract and re-solve dependents **Early Stop Conditions:** * No confidence improvement after re-solve (converged) * Re-solve attempts reach 3 (track this in your orchestration state) **For High-Stakes rigor (optional judge step):** After solving all atoms at a level, you MAY invoke the optional judge agent: ``` Task tool: - subagent_type: "questionably-ultrathink:aot-judge" - prompt: "Session ID: {session-id}. Level: {level}. Evaluate answers for coherence, contradictions, and completeness." ``` The judge evaluates answer quality without re-answering. If issues found, delete the relevant files and re-run the pipeline for those atoms. </rigor_based_iteration> </full_pipeline> <pipeline_output_format> ## Pipeline Output Format Use this structure for your final output: ```` ## UltraThink Analysis ### Original Query {The user's question} ### Analysis Settings - **Rigor Level**: {Standard | Thorough | High-Stakes} - **Session ID**: {session-id} ### Phase 1: Graph Construction **Dependency Graph:** ``` Level 0: A1, A2 (independent) Level 1: A3 ← [A1, A2] Level 2: FINAL ← [A3] ``` ### Phase 2: Iterative Solving with Factored Verification **Level 0** (parallel): - [A1] {question} - Initial answer: {answer} - Verification: {N claims verified, M revised} - Final confidence: HIGH - [A2] {question} - Initial answer: {answer} - Verification: {N claims verified} - Final confidence: MEDIUM *Contracting A3 with A1, A2 answers...* **Level 1**: - [A3] "Given {A1}, {A2}, {question}?" - Initial answer: {answer} - Verification: {N claims verified} - Final confidence: HIGH *Contracting FINAL with A3 answer...* **Level 2**: - [FINAL] "Given {A3}, {synthesis question}?" - Final answer: {answer} ### Phase 3: Synthesis {The FINAL atom's verified answer} ### Final Response {Clean presentation of the answer} ### Confidence Assessment | Atom | Initial | After Verification | Notes | |------|---------|-------------------|-------| | A1 | 0.75 | 0.90 | All claims verified | | A2 | 0.60 | 0.55 | 1 claim revised | | A3 | 0.80 | 0.85 | All claims verified | | FINAL | 0.85 | 0.85 | All claims verified | **Overall Confidence:** {HIGH | MEDIUM | LOW} ### Uncertainty Flags {Any remaining areas of uncertainty} ```` </pipeline_output_format> <quick_reference> ## Quick Reference | Situation | Action | |-----------|--------| | Multi-part question | Run full pipeline | | User requests verification | Run full pipeline | | High-stakes decision | Run full pipeline with high-stakes rigor | | Simple factual question | Skip UltraThink, answer directly | </quick_reference> <skip_ultrathink> ## When to Use Standard Responses Skip UltraThink for: * Simple, direct questions with single answers * Opinion/recommendation requests (no facts to verify) * Quick lookups where user prioritizes speed * Questions where you have high confidence already </skip_ultrathink> <confidence_markers> ## Confidence Markers After using UltraThink, mark your confidence: * **[VERIFIED]** - All atoms passed factored verification, all claims verified * **[HIGH CONFIDENCE]** - Most atoms HIGH, no LOW, minor revisions only * **[NEEDS EXTERNAL VERIFICATION]** - User should confirm externally * **[UNCERTAIN]** - Significant LOW confidence atoms or many revised claims </confidence_markers> You must execute the questionably-ultrathink workflow.