Session scorecards
Agent evaluationBeta
Beam turns the activity it already records — commands, file edits, tool calls, flagged findings — into a per-session scorecard: did this agent run stay inside policy, and where did it not. Evaluation against your rules, not a benchmark of model quality.
This scores behavior against policy, not output quality — it won't tell you if a generated answer was correct, only whether the actions taken to get there were allowed.
What a session scorecard covers
Every check reads from the same activity record Beam already keeps — nothing is inferred about intent, only what the agent actually did.
- Policy checks
Pass/fail against the rules you set
Define what an agent run is allowed to touch — paths, commands, network destinations — and each session is checked against it.
- Risk score
One number from the flagged findings
Sessions with credential access, destructive commands, or permission changes score higher risk, with the evidence behind the number one click away.
- Trend
Per-agent and per-repo history
Compare scorecards across sessions to see whether a given agent, project, or team is drifting toward riskier behavior over time.
- Coverage
What the scorecard does not see
A session with no recorded activity yet, or activity outside what Beam is watching, is marked unscored rather than silently passed.
How scoring behaves
- Verdict
- A scorecard for review, not an automated gate — yet
From a session to a scorecard
- 01
An agent session runs, recorded as usual
No separate setup — evaluation reads the same activity record Beam captures for every monitored session.
- 02
Policy checks run against the record
Each rule you've defined — allowed paths, blocked commands, network destinations — is checked against what actually happened.
- 03
Flagged findings roll into a risk score
Credential access, destructive commands, and permission changes weigh into one score per session, evidence attached.
- 04
Scorecards line up into a trend
Review one session, or compare scorecards across an agent's history to catch drift before it becomes an incident.
What Beam's evaluation is and is not
Beam does
- Score a recorded session against policy rules you define
- Roll flagged findings into one risk number, with evidence attached
- Mark sessions with no coverage as unscored, not passing
- Show trend across sessions for the same agent or repo
Beam does not
- Judge whether a generated answer or output was correct
- Grade against a public benchmark or leaderboard
- Block a session automatically on a low score — yet
- Score activity Beam was never watching in the first place
Questions about agent evaluation
Does Beam evaluate the quality of an agent's output?
No. Beam's scorecard checks behavior — what commands ran, what files changed, what was flagged — against policy. It does not judge whether a generated answer, PR, or piece of code was correct; that's a different problem from the one Beam is built to solve.
What is a session risk score based on?
The same flagged findings Beam already surfaces for runtime monitoring — sensitive-data access, credential exposure, destructive commands, permission changes — rolled into one number per session, with the underlying evidence still one click away.
What happens to activity Beam wasn't watching?
It's marked unscored, not passing. A scorecard is only as complete as the activity record behind it, and Beam never fills a coverage gap with an assumed result.
Can a low score block an agent automatically?
A low scorecard does not deny a tool call on its own. Workspace policy (observe, advisory, or enforce) is a separate control that owners and admins set. Scorecards are for review.
Related
Score the sessions you already record
No separate integration — if Beam is watching an agent, that session can be scored against the policy you set.