Glossary
What is an immutable benchmark?
A benchmark is an immutable, scored run on a pinned dataset with a defined scorer roster. Once created, the dataset rows and scorer contract stay fixed so comparisons across weeks stay honest.
Why immutability matters for video
Generative video drifts — model endpoints change, prompts get tweaked, and “better” becomes subjective. An immutable benchmark freezes the inputs and scoring rules so when you re-run after a fal model update, you are diffing like against like.
Teams on fal.ai often run an experiment first, then lock a benchmark when a clip direction is approved. The benchmark becomes the link you paste in Slack, not another attachment.
Benchmark vs experiment
Use an experiment when you are still swapping tasks or models. Use a benchmark when you need a durable, linkable record with the full scorer roster attached.