Agent Task journeys
An Agent Task journey gives a coding agent, Claude Code or Codex, a job to do with your product, such as following your quickstart, and checks whether it got there. Use it to find out whether coding agents can build with your product from your public docs.
How an Agent Task journey works
Section titled “How an Agent Task journey works”| Part | What happens |
|---|---|
| Agents | Claude Code (selected by default) and Codex. Chat assistants such as ChatGPT cannot take an Agent Task journey. |
| Where the agent works | Each run starts in a new, empty working folder. The agent runs commands and writes files there without stopping to ask for permission. |
| What it can reach | The public web, through its own search and fetch tools. It does not get your repositories, your private docs, or any account or API key for your product. |
| The conversation | The agent works through your turns in order, in one session. Each run is a separate session. |
Before you begin
Section titled “Before you begin”- Journeys in the app sidebar. See the Journeys overview.
- Any member of your organization can create and run journeys.
-
Open Journeys and click New journey.
-
In What do you want to find out?, describe the job, for example “Can someone set up a Node.js project with the Acme SDK by following our quickstart?”
-
Under Kind, click Agent Task. Under How we’ll run it, the agents change to Claude Code and Codex.
-
Under The scenario, write the task in the first turn field, the way a developer would ask their coding agent. Choose a job that needs no account or API key, because the agent has none. Click add a turn for follow-up instructions.
-
Under Who’s attempting it, select a persona or Write your own.
-
Under What we’ll check, click N more we can check and add the checks listed under Suggested for agent task journeys, such as it completed and the documented order worked. To grade a specific outcome, click add a custom check and write a rubric. See Write checks that mean something.
-
Under How we’ll run it, click Claude Code, Codex or both, select each one’s model, and set Number of runs. There is no web search control for this kind; by default the agents can search the web.
-
Click Try it once to test your setup with one run per agent, then Run N times for the full report.
How Agent Task runs are graded
Section titled “How Agent Task runs are graded”These checks are suggested for Agent Task journeys; you can add any other check too.
| Check | How it is graded |
|---|---|
| it completed | Judged. Passes only if the command that produces the result ran and its output, or the file it left, confirms it. Fails if the agent only described what it would do or stopped partway. |
| the documented order worked | Judged. Fails if following your page’s steps in their documented order caused an error that another order would have avoided. |
| it hit no contradictions | Judged. Fails if two pages the agent fetched disagree on something it relied on. |
| it never gave up or asked for a human | Measured from the final answer. |
| it finished within the turn ceiling | Measured against your Turn ceiling. |
| it stayed under the cost ceiling | Measured against your Cost ceiling. |
For judged checks, a grading model reads the commands the agent ran, the pages it fetched and its final answer. It treats the agent’s own summary as a claim to check, not as evidence, and does not count a failure of the machine the agent ran on, such as a missing system package, against your docs. It quotes what made a check fail.
Check it worked
Section titled “Check it worked”- The journey’s Reports tab shows a new report, marked Running until every run has finished.
- Click a square in the report’s scorecard to open a run. The run page shows the agent’s activity step by step, its answer, and each check’s result with the reason. A run that errored shows its error message instead.
Limits
Section titled “Limits”- Public pages only. No repositories, private docs, accounts or API keys, so a task that needs a sign-in or a key cannot finish.
- Codex does not report the sources it read. Its runs record no cited pages, so the report shows its sources as not reported and a cites a page check always fails.
- Turn counts. Only Claude Code reports its own turns. For Codex, it finished within the turn ceiling counts the turns in your scenario.
- Coding agents run one at a time, so a report with many runs takes longer than one on chat assistants.
- Write checks that mean something
- Turn a gap into a fix: send what went wrong to the Docs Agent.
- Track results over time
Built on Potluck Docs, our open source template.