Jobs to be done: evaluator context, per-version stats and Blueprint support
T
Thomas Mueller
User story
As an agency managing many AI employees, I want Jobs to be done to evaluate with full context and roll out via Blueprints, so that success rates are trustworthy and consistent across accounts.
Problem / context
The evaluator does not see tool calls or knowledge sources, so it marks correct answers as failures. Statistics mix all versions of a job. Jobs cannot be part of a Blueprint. The 25-job limit is per account, and default jobs cannot be switched off.
Acceptance criteria
- Evaluator receives tool calls, tool results and cited knowledge.
- Stats are kept per job version, or can be reset on edit.
- Jobs are a Blueprint component.
- Limit counted per employee; default jobs can be disabled.
Log In
M

Megan Cheesbrough
Hi Thomas, can you give us a link to an example where a job was marked as a failure due to missing tool calls or knowledge sources so we can understand the expected behaviour vs the problem behaviour?
I've recorded blueprints, success rates across edits, improved control over job limits, and ability to hide global jobs as candidate roadmap work and highlighted that they would add value for large-scale use of job evaluations for high-volume partners. Our planning cycles will evaluate and prioritize candidate work and updates should feed back to this request.
T
Thomas Mueller
Megan Cheesbrough
Hi Megan, thank you, and great that Blueprints, success rates across edits, job limits and hiding global jobs are on the candidate list.
Our request (what we need, in order of impact)
The evaluator should receive tool calls, tool results and the cited knowledge snippets, not only the visible chat text.
Success rates per job version (or a reset when a job is edited), so a rewritten job does not blend old and new scores.
Jobs as a Blueprint component, so we can roll out the same jobs to every account. Today we sync them per account through the test-case API.
The 25-job limit counted per AI employee, and the ability to disable default jobs.
Examples of false failures (hotel concierge employee, chat; we can't share conversation links because they contain guest data, but we can show you a test account)
"Hand over correctly to the team" scored 0 of 6, because the evaluator counted sauna bookings and a declined late check-out as handover cases.
It cannot see that the booking tool handled them and that the house rules said no.
"Don't invent anything" marked facts from the knowledge base as "no source", because the evaluator can't see the cited knowledge.
"Answer the question the first time" marked a correct follow-up question about the travel dates as a failure.
Our workaround is to write scenarios that exclude tool actions and house rules, and to only judge what is visible in the text. That makes the jobs much weaker than they should be.
The question
Does the evaluator receive tool calls or knowledge sources today at all (and we are missing a setting), or is that not available yet? And which of the two would you ship first: per-version statistics or Blueprint support? That decides whether we keep syncing jobs by script or wait.
Thanks, Thomas
M

Megan Cheesbrough
Thomas Mueller I've captured the failure cases for the AI team and let them know that if they need an example account we can follow up with you.
I don't know the timeline for it, but Blueprint support is most likely to come ahead of per-version statistics. The user expectations of blueprint support for jobs is well-understood. But per-version statistics needs product planning for how users can understand job-versions vs employee versions and make sure they have the control they expect across versions.