Question 1

What is Frametail?

Accepted Answer

Frametail is production observability for generative video: benchmarks on fixed datasets, traces with spans beside artifacts, and experiments before you commit to scored runs. The goal is to make quality and regressions discussable with evidence instead of hallway memory.

Question 2

What is generative video evaluation?

Accepted Answer

Generative video evaluation compares AI-generated clips across model, prompt, or pipeline changes using pinned datasets, scorer contracts, and reproducible benchmark runs — not ad-hoc review in chat. Frametail is built for that workflow end to end.

Question 3

How much does Frametail cost?

Accepted Answer

Frametail has a free tier ($0/month) with 500 benchmark rows, 1,000 scorer runs, and 14-day trace retention. Pro is $100/month with higher limits and live scoring; Enterprise is custom pricing. Book a demo at frametail.com/demo.

Question 4

How long does it take to get useful signals?

Accepted Answer

Most teams connect a project, wire the SDK for traces, and run their first benchmark within a day. The slow part is rarely the product; it is agreeing on the scorer roster and which rows belong in the dataset.

Question 5

Who is the primary user?

Accepted Answer

AI product managers, applied ML engineers, and anyone accountable for a generative video surface. If you ship prompts, model settings, or scoring rubrics, Frametail is meant to sit in that loop. Our beachhead is teams on fal.ai who need tracing and benchmarks before promoting video models to production.

Question 6

Is Frametail built for fal.ai teams?

Accepted Answer

Yes. Frametail wraps the fal client for automatic tracing and sits alongside fal inference as the evaluation layer — benchmarks on pinned datasets and scorers. It is complementary to fal, not a replacement. See frametail.com/integrations/fal for setup.

Question 7

Can we export our benchmarks and traces?

Accepted Answer

Yes. Benchmarks can be exported as structured reports with full scorer outputs and metadata. Traces can be downloaded as JSON or shared via persistent links for debugging sessions outside the dashboard.

Question 8

What video generation services does Frametail support?

Accepted Answer

Frametail integrates with major video generation providers including fal.ai, OpenRouter, and custom model endpoints. The SDK wraps fal and OpenRouter clients for automatic tracing and can instrument any pipeline that emits structured outputs.

Question 9

How does Frametail compare to Langfuse?

Accepted Answer

Langfuse is strong for general LLM tracing. Frametail is purpose-built for generative video: artifact-native traces and immutable benchmarks on pinned datasets. See frametail.com/compare/frametail-vs-langfuse for a feature-by-feature comparison.

Question 10

How do I get started with Frametail?

Accepted Answer

Sign up or accept an org invite, create a project with a project key, install the SDK, and enable tracing on your first fal or custom pipeline call. Then pin a small dataset and run a pilot benchmark. Step-by-step guide: frametail.com/docs/quickstart.

Question 11

How do I install the Frametail SDK?

Accepted Answer

Add the Frametail client package to your Node or TypeScript service, initialize it with your org credentials and project key, then wrap providers such as fal or OpenRouter for automatic spans. Integration guides live at frametail.com/docs and frametail.com/integrations.

Question 12

What is a dataset in Frametail?

Accepted Answer

A dataset is a pinned collection of inputs (and optional reference outputs) you use for repeatable benchmarks. Rows stay stable so every benchmark run compares the same cases — the baseline for regression discussions.

Question 13

What is the difference between a benchmark and an experiment?

Accepted Answer

A benchmark is an immutable, scored run on a dataset with a scorer contract. An experiment compares tasks on inputs without attaching that scorer roster. Use experiments when you are still mixing models; use benchmarks when you need a linkable artifact.

Question 14

What is a scorer in Frametail?

Accepted Answer

A scorer is a rubric or automated judge that grades benchmark outputs consistently — for example, a model-based judge or a structured checklist. You attach scorers before launching a benchmark so every row gets comparable scores.

Question 15

How does live scoring relate to benchmarks?

Accepted Answer

Live scoring samples production traces under rules you configure and attaches scores to those traces. It does not silently create a benchmark. If you want a dataset-backed comparison, you still run the evaluation workflow explicitly.

Question 16

Can I use Frametail without fal.ai?

Accepted Answer

Yes. fal.ai is our primary integration path, but Frametail also supports OpenRouter and custom endpoints via the SDK. Any pipeline that emits structured outputs can be traced and benchmarked.

Question 17

How do I trace fal.ai calls automatically?

Accepted Answer

Initialize the Frametail client with your project key, call initializeFalClient on the fal SDK, and use the wrapped client for inference. Spans capture latency, inputs, outputs, and errors without manual instrumentation. Details: frametail.com/integrations/fal.

Question 18

Does Frametail integrate with OpenRouter?

Accepted Answer

Yes. Dedicated setup guides cover OpenRouter tracing patterns. See frametail.com/integrations/openrouter.

Question 19

How long are traces retained?

Accepted Answer

On the Free plan, trace retention is 14 days per month. Pro typically includes 30-day retention; Enterprise supports custom retention and export policies. See frametail.com/pricing for current limits.

Question 20

Does Frametail support SSO or SAML?

Accepted Answer

SSO and SAML are available on Enterprise plans, along with RBAC and org-level API key controls. Free and Pro are demo-led today — contact founders@frametail.com or book a demo for enterprise requirements.

Question 21

Is live scoring included on the Free plan?

Accepted Answer

No. Live scoring on production traces is included on Pro and Enterprise. Free still includes benchmarks, traces, and datasets within monthly row and scorer limits.

Question 22

Does Frametail offer a public API?

Accepted Answer

Yes. Projects, traces, benchmarks, and related resources are available via authenticated HTTP APIs using your project key. Reference documentation is at frametail.com/docs/api.

Question 23

Where is customer data stored?

Accepted Answer

Customer generations, prompts, traces, and scorer outputs stay within your organization tenancy. Frametail does not use customer video or prompts to train shared foundation models.

Question 24

Do you train on customer video?

Accepted Answer

No. Customer generations, prompts, and scorer outputs are not used to train shared foundation models. Your artifacts stay inside your tenancy boundaries.

Input	A: Subject closeup	B: Wide shot	C: Low angle
A ballet dancer performs a pirouette in a sunlit studio.	completed 8.2s	completed 8.1s	completed 8.3s
The dancer extends one arm gracefully while spinning.	completed 7.9s	completed 8.0s	completed 8.1s

Frequently asked questions

Get started today

exp_8a3f2b9c4e1d