How results are measured
Journeys grades every answer an agent gives with checks, then counts the answers that passed. A collection uses that count to mark each buyer question You win, Split or You lose.
What happens in a run
Section titled “What happens in a run”- Someone clicks Run N times, Run now or Try it once on a journey, or Run collection on a collection, and the run is queued.
- EkLine creates one report and works through the agents you picked, one after another. If one agent fails, its runs show as errored and the others carry on.
- In each run, the agent plays the journey’s persona and receives the turns in order, in one conversation.
- Every check grades each finished run’s answer.
- EkLine writes What went wrong from the checks that failed, and the report is complete.
A collection runs each of its journeys with that journey’s own agents, models and number of runs. Its report appears when every question’s answers are in, or after 60 minutes pass without a new answer. In that case it counts only the answers collected.
How many times each question is asked
Section titled “How many times each question is asked”Each agent runs a journey as many times as its Number of runs (1, 3, 5 or 10), and every run is a separate conversation. A journey with 3 agents and 5 runs gives 15 answers. Try it once runs once per agent. The free report asks each of its 15 questions 10 times on each of ChatGPT, Claude, Perplexity and Gemini: 40 answers per question, 600 in all.
An agent can answer the same question differently each time, so a result is the share of answers that passed, not one answer.
Web search is on by default, with at most 5 searches. Only Claude applies the limit, Perplexity always searches, and Codex reports no sources. See Journey agents and checks.
How an answer is graded
Section titled “How an answer is graded”An answer is the agent’s reply to the last turn. Most checks read that reply; you appear at turn 1 reads the reply to the first turn.
- Measured checks are code. The same answer always gets the same result.
- Judged checks are graded by an AI model (
claude-sonnet-5) against the check’s rubric. The judge reads the conversation, the sources the agent reported, the persona, and your brand names and competitors. It treats the agent’s own summary as a claim to verify, and quotes the passage when it fails an answer.
Each check on each answer ends as passed, failed, or not checked. A check is not checked when it needs brand names and Company Profile has none, when a ceiling has no value, when the judge finds the evidence not enough to decide, or when the check could not run. An errored run gets no check results.
When an answer passes
Section titled “When an answer passes”In a collection, the checks on a buyer question decide which rule applies:
| The question’s checks include | An answer passes when |
|---|---|
| you are recommended, not just mentioned | That check passed. Other checks show on the report but do not decide. |
| What it says about you is accurate (the free report’s questions that name you) | That check passed. |
| Neither | Every check that ran passed, and at least one ran. |
An answer is graded when the deciding check (or, in the last row, any check) has a result other than not checked. Errored and ungraded answers are left out of the counts. They do not count as failures.
The report’s How results are decided box sums this up as “An answer passes when it meets every check on its question.” For the first two rows, read it as the one deciding check.
You win, Split and You lose
Section titled “You win, Split and You lose”For each buyer question, passed and graded answers are added up across every assistant. Then:
| Share of graded answers that passed | Result |
|---|---|
| 80% or more | You win |
| More than 20% and less than 80% | Split |
| 20% or less | You lose |
| No graded answers | No result |
For example, a question asked 40 times with 2 errored answers has 38 graded. If 31 passed (82%), it is You win.
Only collections mark questions this way. A single journey’s report shows each check against each run instead.
If a question’s deciding check is never run, for example because Company Profile has no brand names, the question gets No result. On a journey report’s scorecard, a not-checked result is a white square with an outline, and its row does not read as passing.
How “14 of 15” is counted
Section titled “How “14 of 15” is counted”The headline reads, for example, “Assistants recommend Acme in 14 of 15 buyer questions”:
- 14 is the number of questions marked You win. It counts every win, including accuracy questions, where a win means the answers about you were accurate.
- 15 is the number of questions with a result. Questions with No result are left out and shown after the headline, for example “1 more could not be judged”.
On a later run, the headline adds how the count compares with the collection’s first run, for example “2 more than on 7 Oct 2026”.
Why results change between runs
Section titled “Why results change between runs”- Agents vary. Each agent runs with its vendor’s default settings, so the same question can get a different answer each time.
- The web changes. Answers that use web search change as search results change.
- The judge varies. A judged check can grade the same evidence differently on another run.
- Models change behind their names. Each run records the model the vendor reports; the report labels this “recorded, not pinned”.
- Errors change the sample. Errored answers are left out, so the number of graded answers can differ.
A journey report compares each check with the previous report for the same agent and model, so a change of model never reads as a change in your docs. A collection report compares each question with the collection’s first run. See Track results over time.
Why did my result change when nothing changed on our side?
Section titled “Why did my result change when nothing changed on our side?”The agents, the web and the judge all vary between runs. A question near a boundary, such as 78% passing, can move between Split and You win on that variation alone. Compare the question’s passed count between runs, not only its result.
Are the questions arbitrary?
Section titled “Are the questions arbitrary?”No. The free report writes 15 questions for your company, from your website and Company Profile, in fixed slots. Ten do not name you: Your category, Your buyers, A must-have, The problem, two Use case, Your edge, two Alternatives and Do it for me. Five name you: Who you are, Pricing, Fit, Head to head and How to. You can edit every question before the report runs. In a collection you build, the questions are the journeys you choose.
Does Split mean a competitor won?
Section titled “Does Split mean a competitor won?”No. Split means some of the question’s answers passed and some did not, so nobody won it. Recommended instead of you on the report shows who was named in the answers you did not win.
Built on Potluck Docs, our open source template.