Keyframes, not prompt boxes: how to actually direct AI video
Last week we published a benchmark: we gave the best AI video models a real production brief and reviewed the output like an actual production. They failed the review session, and they failed it in a specific way. A director gave notes, and the models had no way to receive them.
That finding raises the question this post answers. If prompting cannot get a shot through review, what can?
The prompt box is the wrong interface
A prompt describes a scene. A director describes a change. Those are different jobs, and the second one is where production actually lives. “The arm needs to extend further on frame 42” is a normal Tuesday note in an animation review. Try typing it into a prompt box. The model will give you a new shot, and the new shot will have new problems, because text is a lossy language for motion.
Animators do not think in adjectives. They think in arcs, spacing, timing, weight. A prompt box forces all of that through a keyhole, and what comes out the other side is a gamble. We have described this workflow before as a prompt box and a prayer. You pull the lever, you hope. That is not direction. That is a slot machine with a render queue.
Animation already invented the controls
Here is the part the AI conversation keeps missing: the interface for directing motion already exists. It has existed for a century.
Keyframes are the industry’s language of intent. Easing curves are how a performance gets its character. Masks define the scope of a change, so a note about one element does not destroy everything around it. Reference frames hold identity, so the character on frame 100 is the character from frame 1.
Every animator alive already speaks this language, and every pipeline already runs on it. The missing piece was never a better way to describe a shot in English. The missing piece was a model that accepts animation’s own primitives as input.
What directing AI video looks like in practice
This is the thesis we built Fossa Tether on: the comp is the prompt.
Inside After Effects, you set up null layers as a lightweight rig and keyframe them the way you would keyframe anything. Position, scale, rotation. The generation follows your motion curves, and not just the start and end points. The easing you chose survives into the output. Lock layers anchor what must not move, so the background holds while the character travels.
When a note comes in, you handle it like a note. The arm needs to extend further on frame 42: move the null on frame 42 and regenerate. The client wants one element changed: mask it, change it, and keep every other pixel. A new shot needs to grow out of an existing one: generate from any frame in your timeline instead of starting from zero.
None of that requires prompt engineering. It requires the craft skills animators already have, pointed at a model that finally listens.
Direction compounds, models converge
The gap between the top video models shrinks every month, and whichever one leads today will be caught by autumn. Interfaces are different. A pipeline where AI takes direction gets more valuable with every model release, because each better model slots into the same controllable workflow.
That is the real dividing line we see across studios right now. The teams getting production value from AI are the ones treating generation as a shot inside a pipeline, with a director in the loop and notes that land. The teams stuck in demo land are still pulling the lever.
When AI takes direction, it stops being a slot machine and becomes a department.
Animated Company is a London studio bringing AI into animation production: R&D for major studios, finished animation and VFX, and Fossa, our AI toolset for Adobe After Effects. Fossa Tether is in public beta. If you want AI video that takes direction, get in touch at hello@animatedcompany.com.