First Nature

Research

587 workflow tasks lift Qwen3.8-27B by 12.8 points on AutomationBench

AutomationBench, all 600 public tasks
32.4%45.2%
Change in pass@1
+12.8 pts
Training tasks
587

In short

Qwen3.8-27B solved 32.4% of AutomationBench’s 600 public tasks before training and 45.2% after, a gain of 12.8 points. The only training data was 587 workflow tasks from First Nature’s environments. Every task is its own small company, with records, messages and policies written for it, and none was derived from AutomationBench.

Run at medium reasoning, the trained 27B model lands about a point below GPT-5.6 Sol (45.8%) and Claude Fable 5 (46.2%), both run at their highest effort (see the comparison).

Base Qwen3.8-27B and the checkpoint after 60 updates, both at medium reasoning with identical prompts and sampling settings. pass@1 is the mean of three attempts per task; pass@3 counts a task as solved if any of the three succeeds. Scored by AutomationBench’s own grader.

The tasks

An agent receives a request, such as matching a month of invoices to the payments recorded in a spreadsheet, and has to carry it out across several business applications through their APIs. When the agent stops, a verifier reads the final state of each application and decides whether the work was done.

Our environments cover 40 applications and can produce more than 10,000 tasks for one customer. Each set is tuned to the model being trained: for a frontier model we keep tasks that Claude Opus 5.5 at maximum effort solves about half the time. For a 27B model the target is lower, and we picked 587 tasks the model could sometimes, but not reliably, complete.

All 587 went into training; we kept no validation split. We checked that no training task reuses a company name from the public set or shares any 13-word passage with it.

The 587 training tasks: sales 137, support 104, operations 102, marketing 89, finance 81 and HR 74.

How we trained

The model attempted every task eight times. GRPO then scored each attempt against the other seven and updated a LoRA adapter toward the better ones. We ran 60 updates on 48 NVIDIA B300 GPUs.

Training and evaluation both used medium reasoning. Long workflows fit in the context window at that setting, and rollouts finish faster.

Training curves

Mean reward per update, unsmoothed. The axis begins at 0.4.

Reward includes penalties for unfinished attempts and for hitting the response length limit. It climbed from 0.636 at the first update to 0.915 at the sixtieth, with a fresh batch of tasks at every update.

Per-token entropy in nats, unsmoothed. The axis begins at 0.3.

Entropy stayed between 0.42 and 0.52 nats per token for all 60 updates. It never trended down, so the model kept exploring rather than collapsing onto a few answers.

AutomationBench results

We ran both models on all 600 public tasks, three attempts each, using AutomationBench’s official harness and grader at medium reasoning. Only the model weights differed between the two runs.

To choose when to stop, we checked 60 of the public tasks every ten updates. The other 540 played no part in any decision.

A paired bootstrap over tasks puts the pass@1 gain between +10.2 and +15.5 points at 95% confidence. Measured on the 540 untouched tasks alone, the gain is +13.1 points.

Our own review marked as many as 146 public tasks as possibly broken. Without them, the gain is between +12.7 and +14.9 points.

Change in pass@1 by domain, in points, with 95% intervals from a paired bootstrap over tasks.

All six domains gained. Finance gained the most (+16.3 points) and support the least (+5.7).

Against published models

Public set, 600 tasks. Other models’ scores come from the AutomationBench repository at each model’s highest reasoning setting; both Qwen3.8-27B runs used medium.

At 45.2%, the trained model sits between Gemini 3.6 Flash (45.0%) and GPT-5.6 Sol (45.8%).

Other benchmarks

To check that workflow training cost nothing elsewhere, we ran both models on six reasoning, knowledge, instruction-following and coding benchmarks through NeMo Evaluator. No score moved by a statistically significant amount. Both models saw the same seeded 3,000-question sample of MMLU-Pro.

Score after training minus score before, in points, with 95% intervals from a paired bootstrap over items.

Cite this work

Kai F. (2026). 587 workflow tasks lift Qwen3.8-27B by 12.8 points on AutomationBench. First Nature. https://firstnature.si/blog/automationbench-grpo-qwen38-27b/
@misc{firstnature2026automationbench,
  title        = {587 workflow tasks lift Qwen3.8-27B by 12.8 points on AutomationBench},
  author       = {{Kai F.}},
  organization = {First Nature},
  year         = {2026},
  month        = oct,
  url          = {https://firstnature.si/blog/automationbench-grpo-qwen38-27b/}
}