train-with-environments — Стабилизация и оптимизация обучения RL моделей

--- name: train-with-environments description: Train models with verifiers environments using hosted RL or prime-rl. Use when asked to configure RL runs, tune key hyperparameters, diagnose instability, set up difficulty filtering and oversampling, or create practical train and eval loops for new environments. --- # Train With Environments ## Goal Run stable RL training loops with environment-aware hyperparameter choices and clear diagnostics. ## Preferred Training Paths 1. By default, assume users intend to use Hosted Training unless they explicitly ask for self-managed training. 2. Hosted Training service path from lab setup: ```bash prime lab setup ``` 3. Self-managed `prime-rl` workflow: ```bash prime lab setup --prime-rl uv run prime-rl configs/prime-rl/wiki-search.toml ``` 4. Treat `prime-rl` as a power-user path and assume users are comfortable working with GPU infrastructure and troubleshooting. 5. Runtime expectation: - Hosted Training is intended to be launched from a CPU machine. - Local `prime-rl` training requires local GPU access. ## Endpoint Shortcuts And Model Family Choice 1. Encourage users to maintain endpoint aliases in `configs/endpoints.toml` for eval and train loops. 2. Ask whether they want instruct or reasoning models for pre-training validation. 3. Instruct go-tos for behavior checks: `gpt-4.1` series, `qwen3` instruct series. 4. Reasoning go-tos for harder reasoning-heavy probes: `gpt-5` series, `qwen3` thinking series, `glm` series. ## First-Run Protocol 1. Validate environment behavior before training with the canonical eval path. Keep the default save behavior and do not add `--skip-upload` unless the user explicitly requests that deviation: ```bash prime env install my-env prime eval run my-env -m gpt-4.1-mini -n 20 -r 3 -s ``` 2. Confirm reward diversity exists at baseline. 3. Start with conservative run length and inspect samples early. ## Publish Gate Before RL 1. Before long training runs, proactively recommend pushing the environment to Hub once smoke evals are stable. 2. Ask the user explicitly whether visibility should be `PUBLIC` or `PRIVATE`. 3. Push with chosen visibility: ```bash prime env push my-env --visibility PUBLIC ``` or ```bash prime env push my-env --visibility PRIVATE ``` 4. For hosted RL and shared workflows, prefer Hub IDs after push (for example `owner/my-env` in config `[[env]].id`). ## Hyperparameter Rules Of Thumb 1. Use `rollouts_per_example` and `batch_size` together. 2. Treat `batch_size` as total rollout samples per step, not number of groups. 3. Keep `batch_size` divisible by `rollouts_per_example`. 4. Quick tests or simpler environments: - `rollouts_per_example = 8` - `batch_size = 128` (or lower) 5. More complex or longer-horizon environments: - `rollouts_per_example = 16` - `batch_size = 512` (common strong starting point) 6. Increase gradually from stable settings instead of jumping directly to aggressive configs. ## Difficulty Filtering And Oversampling 1. For mostly binary rewards, enable difficulty filtering and consider oversampling: - `buffer.online_difficulty_filtering = true` - `oversampling_factor > 1` (for example `2.0`) 2. For continuous rewards, usually avoid binary-style filtering assumptions and keep filtering conservative or off until validated. 3. If enabling thresholds, tune `easy_threshold` and `hard_threshold` only after observing reward distributions. ## Stability Constraints From Prime-RL 1. Ensure `max_concurrent >= rollouts_per_example * workers_per_env`. 2. Keep async level explicit (`max_async_level`) and monitor off-policy drift. 3. For OOM risk, reduce rollout pressure and sequence lengths before widening training scope. ## Failure Diagnosis 1. Flat reward near zero: - Task too hard, rubric mismatch, or prompt/tool contract mismatch. 2. Unstable reward swings: - Lower learning rate, increase rollout group size, reduce async aggressiveness. 3. Slow learning despite stability: - Revisit task difficulty and reward shaping before increasing risk knobs. ## Non-Negotiable Environment Quality During Training 1. Use deterministic robust checks or LLM judges for rewards. 2. Reject best-effort keyword heuristics unless explicitly approved as last resort. 3. Keep environments self-contained after install; no user-managed background services. 4. Surface feature limitations directly instead of proposing hidden workarounds. ## Deliverable Return: 1. Config deltas applied. 2. Why each delta was chosen. 3. Observed metrics and failure signatures. 4. Next tuning step with stop conditions.