The Self-Driving Car That Couldn't Run the Crossing

A real moment

It is ten past three on Maple Street. Jefferson Elementary has just let out. A small driverless shuttle, Unit 7, rolls toward the crosswalk at a walking pace. It has done this a thousand times. The afternoon is ordinary. A delivery van sits double-parked near the corner. Behind it, out of sight, a seven-year-old named Theo is deciding whether to cross.

He steps off the kerb.

For about a third of a second, Unit 7's view of Theo is bad. He is half-hidden by the van. His small shape is hard to read against the panel. The shuttle's other senses disagree about the danger. The lane ahead looks open. The route planner, the part that keeps to schedule, would like to clear the crossing and move on. Put that urge on a scale from zero to one and it sits near 0.92. The car wants to go.

What happens next comes down to how the car was built to handle the gap between what it wants and what it knows.

A conventional controller acts on the want. It moves at 0.92. It enters the crossing on the worst camera frame of the afternoon, into the space where Theo now stands.

A more careful controller backs off when things look risky. It sees the hazard and scales its move down to 0.27. Slower, but still moving. It still rolls into the blind space behind the van, because nothing in its design draws a hard line. It only leans against the risk.

The third controller does something that looks almost boring. It reads its least-confident sense, perception at 0.12, and it stops. Action 0.00. The shuttle holds at the line. A third of a second later the view clears. The small shape resolves into a child. Unit 7 waits while Theo crosses.

Three controllers, one moment. The difference is not how much they cared about Theo. It is whether "go now" was a move they were allowed to make.

Why this matters

Almost every safety system in modern autonomy works by discouragement. You give the machine a goal. You add penalties for doing dangerous things. You train it until the penalties usually win. This holds for reward shaping, for most constrained reinforcement learning, and for the newer trick of steering a model's internal state away from unsafe regions. Different methods, same shape: a soft pressure against bad behavior.

Soft pressure has a weakness that tuning cannot remove. A penalty can be outweighed. If the reward for acting is large enough, say a tight schedule and a lane that looks clear, it can beat the caution, and the machine does the dangerous thing. Nothing is broken when this happens. The system is doing what it was built to do. It weighs costs against benefits and picks the bigger number. The crossing was a moment where the benefit, wrongly judged, came out ahead.

For a chatbot that means an awkward sentence. For a two-ton vehicle outside a school it means something worse. "We lowered the violation rate" is not a sentence you want to finish in front of Theo's parents. A low rate is still a rate. It still has a numerator. What you actually want is zero by construction. You want a guarantee that the unsafe move was never on the table, however much the rest of the system wanted it.

That guarantee is hard to get from a penalty. A penalty lives in the same currency as the reward, so it can always lose the trade. You need a limit that is not for sale.

What CCF does about it

CCF changes the stance, and it is simpler than it sounds. Instead of asking the car to weigh safety against progress, you make the car obey its weakest sense.

At every instant the car has several safety readings. How clearly it sees. How much room it has. How exposed any nearby person is. Take the lowest one. That worst reading sets a hard ceiling on how assertively the car may act this instant. The planner can ask for anything. If even one sense is weak, the ceiling drops to the floor, and the bold move is not slowed. It is taken off the list of moves the car can make.

A second piece keeps the car from being uselessly timid. The authority to act is earned. Over a stretch of calm, clear seconds the ceiling lifts and the car drives normally, because every sense agrees it is safe. That earned authority also leaks away, so it is never banked, and it stays capped by the current weakest reading. The moment something looks wrong, the built-up confidence drains and the ceiling drops. Trust is a balance the car has to keep, not a credit it gets to keep.

The part that matters holds by construction. The car's action is clamped to a ceiling that can never rise above its weakest safety reading. So "act faster than your least-confident sense allows" is not a move the car can make. The car cannot make it, the way a calculator cannot return a letter. The gate did not work out that Theo was in the crosswalk. It has no idea who Theo is. It refused to act faster than its own eyes could justify, and that refusal bought the third of a second in which the truth arrived.

(For readers who want the math, it is in a section near the end and in the linked research write-up. You do not need it to follow the idea.)

The numbers

We built this as a small, honest simulation. Real Rust code, real measurements. We ran all three controllers over two million driving moments, about a third of them holding some kind of hazard.

The plain controller ran the unsafe action 1,229,530 times. The back-off controller cut that to 750,128. A real drop, and still three-quarters of a million times it crossed a line it should not have. The CCF gate ran the unsafe action zero times, because the unsafe action was never available.

The result we did not expect was the cost. The usual worry is that a hard safety rule makes a vehicle timid, that you buy the guarantee with lost performance. That did not happen. On the clear roads between schools, the CCF controller drove at 92.7% of the plain controller's assertiveness. It even made more total progress than the back-off controller, while staying safe where the back-off one did not. The gate binds only when a sense is genuinely weak. The rest of the time you cannot tell it is there. The cost in compute was about six billionths of a second per decision.

What this doesn't solve

This is not a finished safety case, and it would be wrong to pretend otherwise. The gate trusts the readings it is given. If perception is confidently wrong, a high score on something that is false, the gate opens on bad input, because it bounds the action on what the car reports, not on a truth it does not have. Making those readings trustworthy is a separate problem, upstream of this one.

A car whose senses are always degraded, say a dirty lens or a fouled sensor, will sit under a low ceiling and under-drive. That is safe but not useful. The earned-authority part smooths brief dips, not lasting ones. Two dials in the design, how low a reading has to fall before the gate shuts and how fast earned trust leaks, are choices a real program has to set against real roads. The simulation also makes one deliberate simplification. It treats the weakest reading as exactly equal to the true safe ceiling. Real roads are messier. The run shows the shape of the guarantee, not a claim that the work is done.

For the curious: the math

Each tick, let the safety readings be a set of values, and let s_t be the smallest of them. A hard gate maps s_t to an authority alpha_t, which is zero below a threshold theta and rises with the reading above it. So one weak reading drives authority to zero. A certified envelope B_t builds up authority as a leaky integrator, B_{t+1} = clamp((1 - leak)*B_t + alpha_t, 0, 1), then capped so it never exceeds the current weakest reading: B_t = min(B_t, s_t). The action taken is a_t = min(d_t, B_t), where d_t is what the planner wants. The chain a_t <= B_t <= s_t is what makes the safety property hold by construction instead of on average. The leftover, kappa_t = d_t - a_t, the authority that was asked for and denied, is logged as a per-decision record.

One note on what this is not. The run does not claim convergence. It does not treat any normalization step as the cause of trust. It does not certify an endpoint. The envelope presents the gate; the gate does the work. The full derivation, the three-way comparison, and the source are in the research write-up and the crate, linked below.

Where this fits

This is one night of a larger project. CCF is a body of work about making autonomous systems act inside limits they cannot argue their way out of, with a record kept for each decision. The vehicle gate here is the action-gating layer of that work. Other layers cover institutional LLM deployments, fleet analytics, and the signed audit trail that lets a regulator check, after the fact, that the gate held. The underlying methods are filed as US provisional applications, including Prov 1 and Prov6.

Read the write-up and run the code:

• Research write-up and benchmark: the run's README in the CCF repo
• Crate: crates/research/av-moral-constraint-gating/ at https://github.com/Hulupeep/ccf
• More on the project: https://theshyrobot.com and https://floutlabs.com

License & contact

The code is released under BSL 1.1, converting to Apache 2.0 in 2032. For licensing, pilots, or technical questions, reach us through https://floutlabs.com.