Skip to content

Conformance suite

The conformance suite is the set of 24 scripted scenarios behind every result in the paper. This page lists them, explains how a run is scored, and shows what the default pipeline does on each one.

The scenarios live in decision_testbench.SCENARIOS and are not edited. bikeped.scenarios exposes the same list, in the same order, as the suite conformance.

The 24 scenarios

The table is read from decision_testbench.SCENARIOS. Duration is the last waypoint time plus 0.5 s (get_scenario_duration), rounded to 0.1 s.

# Scenario Expected state Pedestrians Cyclists Duration (s) Braking profile
0 Safe Crossing safe 1 0 6.5 bicycle
1 Bike Approaches Ped alert 1 1 7.5 bicycle
2 High-Speed Near Miss alert 1 1 6.5 bicycle
3 Multiple VRUs alert 3 2 8.5 bicycle
4 Occluded Emergence alert 1 1 6.5 bicycle
5 Parallel Paths warning 1 1 8.5 bicycle
6 Bikes Only idle 0 2 5.5 bicycle
7 Fast Approach alert 1 1 6.5 ebike
8 Speed Variant alert 1 2 10.5 bicycle
9 Counter-Flow on Crosswalk alert 1 1 6.5 bicycle
10 Multi-Speed Converging alert 2 3 8.5 bicycle
11 Wheelchair User alert 1 1 27.2 bicycle
12 Stopped Pedestrian alert 1 1 13.5 bicycle
13 Cross-Street Cyclist alert 1 1 8.5 bicycle
14 E-Scooter alert 1 1 7.5 ebike
15 Child Pedestrian alert 1 1 13.8 bicycle
16 Overtake from Behind alert 1 1 12.5 bicycle
17 Cyclist Abort safe 1 1 8.0 bicycle
18 Dense Pedestrian Group alert 8 1 10.4 bicycle
19 Diagonal Jaywalker alert 1 1 9.5 bicycle
20 Swerving Cyclist alert 1 1 8.0 bicycle
21 Late Turn Toward Ped alert 1 1 7.5 bicycle
22 U-Turn Cyclist alert 1 1 8.5 bicycle
23 Accelerating E-Bike alert 1 1 7.5 ebike

Twenty scenarios expect alert, two expect safe, one expects warning and one expects idle.

Twenty-one of the 24 contain at least one frame that the ground-truth danger model labels as dangerous. The three that contain none are Safe Crossing, Parallel Paths and Bikes Only. In two scenarios, Occluded Emergence and Fast Approach, every danger frame is too late for a warning to help the pedestrian, so they have danger frames and no actionable ones (see Scoring).

Families

bikeped.scenarios tags each conformance scenario with a family.

Family Count Scenarios (index)
safe-crossing 3 Safe Crossing (0), Parallel Paths (5), Bikes Only (6)
standard-approach 5 Bike Approaches Ped (1), Speed Variant (8), Counter-Flow on Crosswalk (9), Cross-Street Cyclist (13), Overtake from Behind (16)
high-speed 4 High-Speed Near Miss (2), Fast Approach (7), E-Scooter (14), Accelerating E-Bike (23)
multi-agent 3 Multiple VRUs (3), Multi-Speed Converging (10), Dense Pedestrian Group (18)
edge-case 3 Occluded Emergence (4), Stopped Pedestrian (12), Diagonal Jaywalker (19)
accessibility 2 Wheelchair User (11), Child Pedestrian (15)
non-linear 4 Cyclist Abort (17), Swerving Cyclist (20), Late Turn Toward Ped (21), U-Turn Cyclist (22)

Braking profiles

Three scenarios set 'vehicle': 'ebike': Fast Approach, E-Scooter and Accelerating E-Bike. The other 21 use the default bicycle profile.

Profile Reaction time (s) Deceleration (m/s²)
bicycle 0.84 1.96
ebike 0.84 6.0

The profile is applied twice. The danger model uses it to decide whether the cyclist can still stop, and make_pipeline passes the same values to the pipeline's speed-adaptive stopping distance. The sources of the values are listed in Safety model.

Scoring

simulate_scenario steps a scenario at 30 FPS from t = 0 to its duration. Each frame gets a pipeline state and a ground-truth label with two tiers. A frame is a danger frame when the cyclist is on a collision course and cannot stop in time. It is also actionable when the time to collision is still longer than the pedestrian reaction time, so a warning can change the outcome.

Column Definition
TP Frames in alert that are actionable.
FN Actionable frames without alert.
FP Frames in alert with no danger of either tier.
TN Frames without alert and without danger.
Sens TP divided by the number of actionable frames.
Spec TN divided by the number of frames without danger.
SevFN Severity-weighted miss rate: severity summed over missed actionable frames, divided by severity summed over all actionable frames.
Lat Seconds from the first actionable frame to the first alert at or after it.
WarnBgt Smallest time to closest approach, measured at each transition into alert.
RxnAdq Share of those transitions that leave more than 1.87 s, the reaction time assumed for a distracted pedestrian.
Fatigue Fraction of all frames spent in alert.
Exp The scenario's expected_dominant. The testbench prints it and does not score against it.

Danger frames that are not actionable count on neither side: an alert during one is not a false positive, and silence during one is not a false negative.

The TOTAL row pools the frame counts over all scenarios before dividing. Its SevFN is the mean over the scenarios that have actionable frames.

Results with the default pipeline

$ python decision_testbench.py
Loaded calibration: FOV=197.90 cx=1752.7 cy=1804.5 f=1013.3 R=1750 frame=3500x3500 pad=3840 pitch=0 k1=0 k2=0
Running all scenarios with Default pipeline...

  SCENARIO RESULTS -- N=58, d=[1.9,24.8]m, v_min=0.147, k=2
  Scenario                       TP   FP   FN   TN   Sens   Spec  SevFN    Lat WarnBgt RxnAdq Fatigue       Exp
  --------------------------------------------------------------------------------------------------------------------
  Safe Crossing                   0    0    0  196  0.0% 100.0%  0.0%    N/A     N/A    N/A   0.0%      safe
  Bike Approaches Ped            86   25   14  101 86.0% 80.2% 26.3%   0.47    3.5s   100%  49.1%     alert
  High-Speed Near Miss           97    0   17   82 85.1% 100.0% 18.0%   0.47    3.3s   100%  49.5%     alert
  Multiple VRUs                 113   29    0  114 100.0% 79.7%  0.0%   0.00    3.9s   100%  55.5%     alert
  Occluded Emergence              0    0    0  164  0.0% 100.0%  0.0%    N/A    1.0s     0%  10.7%     alert
  Parallel Paths                  0    0    0  256  0.0% 100.0%  0.0%    N/A     N/A    N/A   0.0%   warning
  Bikes Only                      0    0    0  166  0.0% 100.0%  0.0%    N/A     N/A    N/A   0.0%      idle
  Fast Approach                   0    0    0  147  0.0% 100.0%  0.0%    N/A    1.6s     0%  23.0%     alert
  Speed Variant                  54   83    0  179 100.0% 68.3%  0.0%   0.00    4.3s   100%  43.4%     alert
  Counter-Flow on Crosswalk      19    0    2  114 90.5% 100.0%  9.5%   0.07    2.6s   100%  35.7%     alert
  Multi-Speed Converging         72   81    0  103 100.0% 56.0%  0.0%   0.00    4.2s   100%  59.8%     alert
  Wheelchair User                83    0   15  651 84.7% 100.0% 26.9%   0.50    4.9s   100%  14.6%     alert
  Stopped Pedestrian             33    0    1  307 97.1% 100.0%  2.9%   0.03    3.2s   100%  20.2%     alert
  Cross-Street Cyclist           51    0    1  143 98.1% 100.0%  1.9%   0.03    3.7s   100%  39.5%     alert
  E-Scooter                      15    0    1  154 93.8% 100.0%  6.2%   0.03    2.3s   100%  28.8%     alert
  Child Pedestrian               99   22   31  263 76.2% 92.3% 26.2%   0.47    4.5s   100%  29.2%     alert
  Overtake from Behind           21    0    2  287 91.3% 100.0%  8.7%   0.07    2.9s   100%  17.6%     alert
  Cyclist Abort                  69    6    0  166 100.0% 96.5%  0.0%   0.00    2.4s   100%  31.1%      safe
  Dense Pedestrian Group         83   27    0  203 100.0% 88.3%  0.0%   0.00    3.5s   100%  35.1%     alert
  Diagonal Jaywalker             82   30   12  162 87.2% 84.4% 12.2%   0.13    4.0s   100%  39.2%     alert
  Swerving Cyclist               83   26    2  130 97.6% 83.3%  4.1%   0.07    3.5s   100%  45.2%     alert
  Late Turn Toward Ped           88   32    0  106 100.0% 76.8%  0.0%   0.00    3.8s   100%  53.1%     alert
  U-Turn Cyclist                 21    0    3  110 87.5% 100.0%  7.6%   0.03    2.2s   100%  45.7%     alert
  Accelerating E-Bike            40    0    6  123 87.0% 100.0%  8.9%   0.20    3.3s   100%  40.7%     alert
  --------------------------------------------------------------------------------------------------------------------
  TOTAL                        1209  361  107 4427 91.9% 92.5%  8.4%

  Alert fatigue (mean fraction of time in alert): 31.9%

Pooled over the suite, the default parameters alert on 91.9% of actionable frames and stay silent on 92.5% of frames without danger.

This invocation feeds the pipeline the scripted positions as they are. The paper's comparison table adds the fisheye localisation error, which python decision_testbench.py --compare applies unless --no-bbox-noise is given. Both runs are described in Testbench and Reproducing the paper.

A few rows need the scoring rules to read correctly. Occluded Emergence and Fast Approach show no true positives and no misses because they have no actionable frames. The pipeline does alert in both (10.7% and 23.0% of frames), 1.0 s and 1.6 s before closest approach, which is less than the 1.87 s a distracted pedestrian needs. Cyclist Abort expects safe as its dominant state and still contains 69 actionable frames before the cyclist turns away, all of which are caught.

Dominant state

python -m bikeped.scenarios evaluate --suite conformance scores the dominant state instead of frames: the most severe state (alert > warning > safe > idle) that the pipeline holds for at least 0.3 s in total over the run. Under that rule the default pipeline matches 23 of the 24 paper labels. The one miss is Cyclist Abort, where the pipeline alerts and the paper's label is safe.

$ python -m bikeped.scenarios evaluate --suite conformance
scenario                             dur  ag danger  alert  false lead s (per interval)  dominant expected  ok
--------------------------------------------------------------------------------------------------------------
safe-crossing                        6.5   1   0.00   0.00   0.00 -                          safe     safe  yes
bike-approaches-ped                  7.5   2   3.33   3.67   0.83 -0.5,+1.6                 alert    alert  yes
...
cyclist-abort                        8.0   2   2.30   2.50   0.20 +0.1                      alert     safe  NO
...
--------------------------------------------------------------------------------------------------------------
24 scenarios, dominant state matches expected in 23; 18 of 27 danger intervals alerted no later than one frame after onset; false-alert time 12.27 s

Catalog

Scenario Family Expected Duration Agents
Safe Crossing safe-crossing
A pedestrian crosses the full crosswalk at about 2.7 m/s with no cyclist anywhere in the scene.
safe-crossing safe 6.5 s 1 person
Bike Approaches Ped bike-approaches-ped
A cyclist rides in along the road from 34 m at 8 m/s, slows to about 2 m/s and passes through the crosswalk just behind a pedestrian walking across.
standard-approach alert 7.5 s 1 person, 1 bike
High-Speed Near Miss high-speed-near-miss
A cyclist at about 7 m/s crosses the crosswalk while a slow pedestrian is in the middle of it, so the danger window is short.
high-speed alert 6.5 s 1 person, 1 bike
Multiple VRUs multiple-vrus
Three pedestrians cross in both directions while one cyclist rides along the road and a second comes in diagonally from the far side.
multi-agent alert 8.5 s 3 person, 2 bike
Occluded Emergence occluded-emergence
The cyclist is first seen 4 m from the crosswalk, as if appearing from behind an obstacle, and cuts diagonally across the path of a slow pedestrian.
edge-case alert 6.5 s 1 person, 1 bike
Parallel Paths parallel-paths
A cyclist rides across the road 3 m to the side of a crossing pedestrian, at the same speed and heading; the gap stays at 3 m.
safe-crossing warning 8.5 s 1 person, 1 bike
Bikes Only bikes-only
Two cyclists ride through the crosswalk area and nobody is on foot.
safe-crossing idle 5.5 s 2 bike
Fast Approach fast-approach
An e-bike at 12 m/s appears 20 m out and rides straight through the crosswalk while a pedestrian is on it.
high-speed alert 6.5 s 1 person, 1 ebike
Speed Variant speed-variant
A slow pedestrian crosses while a cyclist at 3 m/s and then a second at 10 m/s ride through the crosswalk on either side of the centre line.
standard-approach alert 10.5 s 1 person, 2 bike
Counter-Flow on Crosswalk counter-flow-on-crosswalk
A cyclist rides along the crosswalk itself, head-on towards a pedestrian walking the other way.
standard-approach alert 6.5 s 1 person, 1 bike
Multi-Speed Converging multi-speed-converging
Two pedestrians cross in opposite directions while three cyclists at different speeds converge on the crosswalk from different angles.
multi-agent alert 8.5 s 2 person, 3 bike
Wheelchair User wheelchair-user
A wheelchair user crosses at 0.6 m/s and is still on the crosswalk when a cyclist rides through it at speed.
accessibility alert 27.2 s 1 wheelchair, 1 bike
Stopped Pedestrian stopped-pedestrian
A pedestrian stops in the middle of the crosswalk for 4 s to look at a phone. A cyclist rides through the stopping point.
edge-case alert 13.5 s 1 person, 1 bike
Cross-Street Cyclist cross-street-cyclist
A cyclist coming from the cross street rides along the crosswalk at 6 m/s, through a pedestrian who is crossing in the same direction.
standard-approach alert 8.5 s 1 person, 1 bike
E-Scooter e-scooter
An e-scooter at about 7 m/s rides along the road through the crosswalk while a pedestrian is crossing.
high-speed alert 7.5 s 1 person, 1 escooter
Child Pedestrian child-pedestrian
A child crosses at 0.9 m/s, so is exposed on the crosswalk for longer, and a cyclist rides through at up to 8 m/s.
accessibility alert 13.8 s 1 child, 1 bike
Overtake from Behind overtake-from-behind
A cyclist rides along the crosswalk in the same direction as a pedestrian and passes through them from behind at 4 m/s.
standard-approach alert 12.5 s 1 person, 1 bike
Cyclist Abort cyclist-abort
A cyclist heads for the crosswalk, then turns away 3 m short of it and leaves, so the threat resolves.
non-linear safe 8.0 s 1 person, 1 bike
Dense Pedestrian Group dense-pedestrian-group
Eight pedestrians cross together as a tight group and a cyclist rides into it.
multi-agent alert 10.4 s 8 person, 1 bike
Diagonal Jaywalker diagonal-jaywalker
A pedestrian crosses diagonally across the road beside the marked crosswalk while a cyclist comes along the road.
edge-case alert 9.5 s 1 person, 1 bike
Swerving Cyclist swerving-cyclist
A cyclist riding along the far side of the road swerves across it towards the pedestrian on the crosswalk.
non-linear alert 8.0 s 1 person, 1 bike
Late Turn Toward Ped late-turn-toward-ped
A cyclist rides parallel to the kerb, then turns late across the crosswalk into the pedestrian.
non-linear alert 7.5 s 1 person, 1 bike
U-Turn Cyclist u-turn-cyclist
A cyclist passes a pedestrian standing on the crosswalk, makes a U-turn and comes back through them.
non-linear alert 8.5 s 1 person, 1 bike
Accelerating E-Bike accelerating-e-bike
An e-bike accelerates from 3 m/s to 8 m/s on the way to the crosswalk while a pedestrian is crossing.
high-speed alert 7.5 s 1 person, 1 ebike

The same data is available from python -m bikeped.scenarios list --suite conformance and, for one scenario, python -m bikeped.scenarios show ID.