Improving Qwen3.8-27B-Medium on AutomationBench with GRPO

Overview

We improved Qwen3.8-27B’s AutomationBench pass@1 (avg 3) from 32.4% to 45.2% (+12.8 points) using GRPO on 587 tasks from First Nature’s workflow automation training environments. Each task is a self-contained business world with its own records, messages and policies, built separately from AutomationBench to avoid evaluation contamination.

At medium reasoning, the trained model scores within about one point of GPT-5.6 Sol (45.8%) and Claude Fable 5 (46.2%) at maximum effort on the public set (comparison).

Training data

Each task asks the agent to complete a business workflow across several SaaS applications, such as reconciling invoices against a spreadsheet. A verifier checks the final state of every application.

First Nature generates workflow automation agent environments (API / MCP) on demand: tool-use tasks across 40 applications, each calibrated to a difficulty band for the model being trained, at the scale of 10,000+ tasks per engagement. For this study we selected 587 tasks in a learnable distribution for Qwen3.8-27B.

Difficulty is set per model. For frontier models we calibrate to their pass rates, with tasks that Claude Opus 5.5 at maximum effort solves about half the time. We used all 587 for training, with none set aside for validation. No training task shares a company name or any 13-word passage with the 600 public tasks.

By domain, the training set includes 137 sales tasks (23.3%), 104 support (17.7%), 102 operations (17.4%), 89 marketing (15.2%), 81 finance (13.8%) and 74 HR (12.6%).

Training methodology

For each task, we generated eight attempts and used GRPO to update a LoRA adapter based on each attempt’s reward relative to the others. We chose medium reasoning to speed up rollouts and keep long, multi-step workflows within the context window, and used the same setting for evaluation. Training ran for 60 updates on 48 NVIDIA B300 GPUs.

Training diagnostics

Reward

Mean training reward, after penalties for incomplete attempts and response token limits, rose from 0.636 to 0.915 over 60 updates. Each update used a different batch of tasks.

Entropy

Entropy measures how spread out the model’s next-token probabilities are. It ranged from 0.42 to 0.52 nats/token, with no sustained decline over 60 updates.

Evaluation results

We evaluated the base model and the final checkpoint (update 60) on all 600 public AutomationBench tasks, three attempts each, with AutomationBench’s own harness and grader. Both models used medium reasoning, with the same prompts and generation settings.

During training, we monitored 60 public tasks every ten updates to decide when to stop. The remaining 540 tasks were not used for any training decision.

AutomationBench

The pass@1 gain has a 95% confidence interval of +10.2 to +15.5 points (paired bootstrap over tasks). On the 540 tasks not used for any training decision, it is +13.1 points.

Our audits flagged up to 146 public tasks as possibly flawed. Removing them leaves the gain between +12.7 and +14.9 points.

Every domain improved, from +5.7 points in support to +16.3 points in finance.

On the public set, the trained model at medium reasoning scored 45.2%, close to GPT-5.6 Sol (45.8%) and Claude Fable 5 (46.2%) at maximum effort.

General capability check

We also evaluated both models on six reasoning, knowledge, instruction-following and coding benchmarks with NeMo Evaluator. None changed significantly. MMLU-Pro used the same seeded sample of 3,000 questions for both models.

Kai F. (2026). Improving Qwen3.8-27B-Medium on AutomationBench with GRPO. First Nature. https://firstnature.si/blog/improving-qwen38-27b-on-automationbench/

@misc{firstnature2026automationbench,
  title        = {Improving Qwen3.8-27B-Medium on AutomationBench with GRPO},
  author       = {{Kai F.}},
  organization = {First Nature},
  year         = {2026},
  month        = oct,
  url          = {https://firstnature.si/blog/improving-qwen38-27b-on-automationbench/}
}