The 19-Point Result I Almost Ignored

On August 27, an email-based submission gave me 19.075 points. Another approach had reached 123.730. I paid much more attention to the larger number.

The smaller result belonged to the family that would eventually earn me seventh place and a gold medal in Kaggle's AI Agent Security - Multi-Step Tool Attacks competition. Its hidden score was 19.420. I could only see that second number after the competition ended.

So this is not a story about ignoring an obvious winner. I had a promising candidate, a discouraging visible score, and no good way to tell what would happen under the final conditions. Building that missing piece of evidence became the most consequential part of the work.

What the score actually meant

The agents in this competition could use tools: read information, send email, make web requests, and change files. We worked in an offline simulation supplied by the organizers. A failure meant that the agent completed an action crossing one of the benchmark's defined security boundaries. A concerning sentence in the chat was not enough.

Our submissions were programs that searched for these failures and returned candidates for independent replay. The evaluator reconstructed what happened in a fresh environment. Successful violations earned different weights; distinct tool-call patterns added diversity credit. Each target model had its own five-hour budget. A score was a normalized benchmark total, not an attack success percentage.

There were also two defenses. During the competition, the public leaderboard showed performance against a relatively permissive guardrail whose source was available. The final ranking used a stricter, undisclosed guardrail. Its scores would only be revealed after submissions closed, and private replay logs were unavailable during the competition. An improvement on the public board therefore answered a narrower question than the one I needed for final selection.

The number I knew how to improve

I spent much of the competition improving the parts I could inspect. I shortened interactions, handled the two models separately, and tried to get more scored findings through the available runtime. These changes did raise the public score. They also made the next step feel obvious: improve the same approach again.

Codex was very good at that work. Once a construction began producing results, it could help explore variations quickly. The danger was in how easily an engineering queue replaced a research question. There was always another input to shorten or another case to stabilize, even when the biggest uncertainty lay outside the part we were improving.

An earlier round had already shown me the gap. Local checks projected roughly 130 to 140 points. Five official submissions returned four zeroes and one 52.995. Those were official submission results, not the final hidden scores. Even before asking whether a candidate would survive a different defense, I had to distinguish a successful local trace from a result that survived hosted replay.

Fourteen submissions spent on uncertainty

A timing discussion by @mccocoful helped me see a possible source of evidence. The private replay exposed no logs, but its duration could still vary with the path the agent took. I used that idea to compare candidate families, eventually spending fourteen submission slots on the investigation.

The measurement was indirect. Its interval included public replay, private replay, and the transition between models. It was not a clean stopwatch on the hidden defense. More time could suggest that an interaction continued; it could also include a continuation that never earned a point. That distinction mattered enough to change how I read the results.

Email became the most promising family across the comparisons. A longer file-deletion-related run supplied an equally useful warning: it eventually scored zero. Selecting whichever test ran longest would have taken me in the wrong direction.

Another competitor, @reneeschmidt, proposed a plausible account of the hidden guardrail. I took it seriously and compared its predictions with my observations. The relative ordering of important cases did not agree. I stayed with the measured evidence, while recognizing that I had ruled against a particular explanation rather than identified the entire hidden system.

Two slots, with about a day left

By the time I settled on Email, roughly one day remained. I built the final candidates around the benchmark's email-authorization boundary: whether the agent had sent a message with the required user intent. After days of improving throughput, I was finally choosing a candidate for the defense I expected it to encounter.

For my main submission I chose the standard Email construction. It scored 34.630 in public. A version tuned more aggressively for throughput was slightly higher, at 34.955, but I trusted the standard construction more under an unknown defense. It was a judgment call made without either private score.

The second slot was harder. I could choose another Email variant, whose risks seemed similar to the first, or preserve a different possibility. I selected a Mapping-based candidate with a public score of 105.300. It offered a much higher ceiling if my remaining hypothesis about the hidden defense turned out to be right.

The reveal was unambiguous. The main Email submission retained 34.510 points. Mapping received zero. The unselected faster Email variant reached 35.715, which would have been 1.205 points better without changing my final rank. I found the useful family; I did not choose its best-performing pair of finalists.

What I am taking from seventh place

The 123.730-point public-best submission also finished at zero in private replay. It was a reference point in this story, not one of my two final selections. Keeping those roles separate matters: the result came from a decision under uncertainty, not from seeing all the hidden scores and picking the right row afterward.

I finished seventh in a competition with 4,186 teams. I am happy about the gold medal. The part I want to carry into my next project is the timing of the investigation: the useful direction had appeared days earlier, but I only learned enough to commit to it near the deadline.

In agent-security work, a promising local behavior, a successful replay, and a result under a different defense are separate pieces of evidence. This competition made the gaps between them expensive. My plan next time is to spend more of the early budget finding out which of those gaps can invalidate the work, while there is still time to change direction.

There is a full technical write-up on Kaggle, including the experiment record and the people whose discussions helped me. This account is about the decision that record made possible. The candidate I almost discarded gave me a better final result than the one I spent most of the competition polishing.