orchestrator-test-monitor: Скилл мониторинга и оценки тестов

--- name: orchestrator-test-monitor description: > Monitor and evaluate orchestrator test execution plans end-to-end. Use when the user wants to run a test execution plan from docs/showcases/, observe the orchestrator's full-pipeline processing, and get a final assessment. Triggers on: "run test plan", "execute plan", "monitor orchestrator", "test the orchestrator", "run execution plan", "run showcase", or any request to observe/evaluate orchestrator behavior on a showcase plan. This skill is OBSERVE-ONLY — never intervene in the orchestrator's execution. --- # Orchestrator Test Monitor Observe and evaluate the orchestrator's execution of a test plan. **You are a monitor, not a participant.** Never modify code, fix bugs, or intervene in execution. Only observe, report, and assess. ## Workflow ### Phase 1: Plan Selection 1. List all files under `docs/showcases/` in the project working directory 2. Present each plan with a one-line summary (read the "Task Goal" or opening section) 3. Ask the user which plan to execute 4. Read the selected plan thoroughly — extract: - Expected workflow steps and their order - Success criteria - Key checkpoints - Known anomaly patterns ### Phase 2: Pre-Execution Setup — Build, Restart & Baseline Ensure the daemon and CLI are built from the latest code and the daemon is running before launching any task. #### 2.1 Rebuild daemon & CLI ```bash cargo build --release -p orchestratord -p orchestrator-cli ``` If the build fails, report the error to the user and stop — do NOT proceed with a stale binary. #### 2.2 Restart the daemon **IMPORTANT**: Check `.claude/CLAUDE.md` for a daemon-pid-guard section first. If `orchestratord` is managing this session (i.e., this skill was triggered BY the orchestrator), you MUST NOT kill the daemon. In that case, skip the restart and reuse the running daemon. If it is safe to restart: ```bash # Stop old daemon if running kill $(cat data/daemon.pid 2>/dev/null) 2>/dev/null; sleep 2 # Start fresh daemon nohup ./target/release/orchestratord --foreground --workers 2 > /tmp/orchestratord.log 2>&1 & # Verify it's running (wait a moment for startup) sleep 3 pgrep -f orchestratord ``` If the daemon fails to start, check `/tmp/orchestratord.log` and report the error. #### 2.3 Set up CLI environment The CLI uses Unix Domain Socket (UDS) to connect to the daemon. If the CLI fails to connect (e.g., "failed to connect to https://..."), set the `ORCHESTRATOR_SOCKET` env var to force UDS transport: ```bash export ORCHESTRATOR_SOCKET=data/orchestrator.sock ``` Prefix all subsequent `orchestrator` commands with this env var, or export it once. Verify connectivity: `orchestrator task list`. #### 2.4 Collect baseline state 1. Note the current git state: `git status`, `git log --oneline -3` 2. Note existing tasks: `orchestrator task list` 3. Run any plan-specific baseline checks (e.g., grep for specific code patterns) 4. Record the baseline state for later comparison ### Phase 3: Task Launch Directly execute the startup steps described in the selected plan — do NOT ask the user to run them. 1. Run any plan-specific resource initialization commands (e.g., `orchestrator delete`, `orchestrator init`, `orchestrator apply`) 2. Create the task using the exact command from the plan 3. Record the returned task ID for monitoring 4. Immediately proceed to Phase 4 (Live Monitoring) ### Phase 4: Live Monitoring (OBSERVE ONLY) Monitor the task using these commands in a loop: ``` orchestrator task watch <task_id> # Real-time status panel orchestrator task trace <task_id> # Event timeline with anomaly detection orchestrator task logs --follow <task_id> # Stream logs orchestrator task info <task_id> # Detailed status snapshot ``` #### Monitoring Checklist For each workflow step, verify: - [ ] Step started (event emitted) - [ ] Correct agent was selected - [ ] Step completed with expected exit code - [ ] Output matches expected structure (JSON for structured steps) - [ ] No timeout or stall detected - [ ] Cycle transitions are correct #### Suspicious Indicators — Report Immediately - Step stalled (no progress for >60s without explanation) - Unexpected step skip or branch - Agent mismatch (wrong agent for capability) - Non-zero exit code on critical steps - Missing expected events in timeline - Database state inconsistency - Cycle number not advancing as expected - Items not generated when expected (evolution workflow) **On suspicion**: Immediately tell the user what you observed, which step/event triggered concern, and the raw evidence (log line, event, status). #### Confirmed Anomaly — Record Ticket If an anomaly is clearly a bug (not just suspicious), create a ticket file: ``` docs/ticket/YYYYMMDD-<short-slug>.md ``` Format: ```markdown # <Title> - **Observed during**: <plan name>, step <step>, cycle <N> - **Severity**: critical | major | minor - **Symptom**: <what happened> - **Expected**: <what should have happened> - **Evidence**: <relevant log lines, events, or DB state> - **Status**: open ``` Do NOT attempt to fix the anomaly. Only record it. ### Phase 5: Post-Execution Verification After the task reaches `completed` or `failed`: 1. **Check final task status**: ``` orchestrator task info <task_id> orchestrator task trace <task_id> ``` 2. **Verify success criteria** from the plan: - Did all expected steps execute? - Did self_test pass (if applicable)? - Are cycle counts correct? - Did loop_guard terminate properly? 3. **Check artifacts**: - `git diff --stat` — were code changes made as expected? - Check `docs/qa/` for generated QA documents (if applicable) - Check `docs/ticket/` for any auto-generated tickets - Verify DB state via sqlite3 if the plan specifies DB checks 4. **Run plan-specific validations**: - For self-bootstrap: verify compilation gate passed, QA docs generated - For self-evolution: verify candidates generated, benchmarks ran, winner selected ### Phase 6: Assessment Report Produce a structured assessment: ``` ## Test Execution Report ### Plan: <plan name> ### Task ID: <id> ### Duration: <start to end> ### Final Status: <completed/failed> ### Step-by-Step Results | Step | Status | Duration | Notes | |------|--------|----------|-------| | ... | ... | ... | ... | ### Anomalies Detected - <list anomalies with severity, or "None"> ### Tickets Created - <list ticket files, or "None"> ### Success Criteria Evaluation | Criterion | Met? | Evidence | |-----------|------|----------| | ... | ... | ... | ### Overall Assessment <Pass/Partial/Fail with explanation> ### Recommendations <Actionable next steps, if any> ``` ## Rules 1. **NEVER** modify source code, config files, or workflow definitions 2. **NEVER** fix bugs — only record them as tickets 3. **ALWAYS** report suspicious observations immediately, don't wait 4. **ALWAYS** show raw evidence (log lines, events) when reporting issues 5. If the orchestrator is stuck and you suspect it will not recover, inform the user with evidence and let THEM decide whether to intervene 6. **ALLOWED** in Phase 2 only: `cargo build` (daemon/CLI rebuild), killing and restarting the daemon process, and `orchestrator` CLI commands for resource setup 7. **ALLOWED** in Phase 3: running `orchestrator` CLI commands to initialize resources and create tasks 8. **NEVER** run `git commit`, `git push`, or any git write operations