587 workflow tasks lift Qwen3.8-27B by 12.8 points on AutomationBench
- AutomationBench, all 600 public tasks
- 32.4%45.2%
- Change in pass@1
- +12.8 pts
- Training tasks
- 587
In short
Qwen3.8-27B solved 32.4% of AutomationBench’s 600 public tasks before training and 45.2% after, a gain of 12.8 points. The only training data was 587 workflow tasks from First Nature’s environments. Every task is its own small company, with records, messages and policies written for it, and none was derived from AutomationBench.
Run at medium reasoning, the trained 27B model lands about a point below GPT-5.6 Sol (45.8%) and Claude Fable 5 (46.2%), both run at their highest effort (see the comparison).
The tasks
An agent receives a request, such as matching a month of invoices to the payments recorded in a spreadsheet, and has to carry it out across several business applications through their APIs. When the agent stops, a verifier reads the final state of each application and decides whether the work was done.
Our environments cover 40 applications and can produce more than 10,000 tasks for one customer. Each set is tuned to the model being trained: for a frontier model we keep tasks that Claude Opus 5.5 at maximum effort solves about half the time. For a 27B model the target is lower, and we picked 587 tasks the model could sometimes, but not reliably, complete.
All 587 went into training; we kept no validation split. We checked that no training task reuses a company name from the public set or shares any 13-word passage with it.
How we trained
The model attempted every task eight times. GRPO then scored each attempt against the other seven and updated a LoRA adapter toward the better ones. We ran 60 updates on 48 NVIDIA B300 GPUs.
Training and evaluation both used medium reasoning. Long workflows fit in the context window at that setting, and rollouts finish faster.
Training curves
Reward includes penalties for unfinished attempts and for hitting the response length limit. It climbed from 0.636 at the first update to 0.915 at the sixtieth, with a fresh batch of tasks at every update.
Entropy stayed between 0.42 and 0.52 nats per token for all 60 updates. It never trended down, so the model kept exploring rather than collapsing onto a few answers.
AutomationBench results
We ran both models on all 600 public tasks, three attempts each, using AutomationBench’s official harness and grader at medium reasoning. Only the model weights differed between the two runs.
To choose when to stop, we checked 60 of the public tasks every ten updates. The other 540 played no part in any decision.
A paired bootstrap over tasks puts the pass@1 gain between +10.2 and +15.5 points at 95% confidence. Measured on the 540 untouched tasks alone, the gain is +13.1 points.
Our own review marked as many as 146 public tasks as possibly broken. Without them, the gain is between +12.7 and +14.9 points.
All six domains gained. Finance gained the most (+16.3 points) and support the least (+5.7).
Against published models
At 45.2%, the trained model sits between Gemini 3.6 Flash (45.0%) and GPT-5.6 Sol (45.8%).
Other benchmarks
To check that workflow training cost nothing elsewhere, we ran both models on six reasoning, knowledge, instruction-following and coding benchmarks through NeMo Evaluator. No score moved by a statistically significant amount. Both models saw the same seeded 3,000-question sample of MMLU-Pro.
Cite this work
Kai F. (2026). 587 workflow tasks lift Qwen3.8-27B by 12.8 points on AutomationBench. First Nature. https://firstnature.si/blog/automationbench-grpo-qwen38-27b/
@misc{firstnature2026automationbench,
title = {587 workflow tasks lift Qwen3.8-27B by 12.8 points on AutomationBench},
author = {{Kai F.}},
organization = {First Nature},
year = {2026},
month = oct,
url = {https://firstnature.si/blog/automationbench-grpo-qwen38-27b/}
}
