LLMs Write Good FFmpeg Commands. Now Give Them a Place to Run.

Ask a language model for an FFmpeg command and you usually get a good one. “Make this MOV a web-friendly MP4” becomes -c:v libx264 -crf 23 -c:a aac -movflags +faststart. “Cut 00:30 to 00:45” becomes -ss 30 -to 45. FFmpeg has decades of public documentation, mailing-list answers and forum threads, and models have read a lot of it.

The command was never the hard part. The hard part is running it: a machine with FFmpeg, access to the input, enough CPU and disk, and a way to report back to the model in terms it can act on. This post covers where models still go wrong and how to set up execution so those mistakes are cheap.

Where models still go wrong

These are the patterns to plan for. None of them is a reason not to let a model write commands; all of them are reasons to run the commands where errors are clear.

• Guessing at the input. The model assumes the file has audio, or is 1080p, or is H.264. A -c:a copy on a file with no audio stream, or a scale that assumes landscape on a portrait video, gives an error or a wrong output.
• Filter graph syntax. Labels, semicolons and escaping in -filter_complex are easy to get slightly wrong, more so with drawtext and its own quoting rules.
• Stream mapping. With several inputs, a missing -map lets FFmpeg pick streams by its own rules, which may not be the ones intended.
• Seek position. -ss before -i and after -i behave differently, and with -c copy the cut starts on a keyframe, not the exact time.
• Codec availability. The model writes libfdk_aac or a hardware encoder that the build does not have.
• Odd dimensions. scale=1280:-1 can give an odd height, which H.264 encoders reject. -2 is the fix.

Make execution forgiving

The way to handle these mistakes is not to stop the model from writing commands. It is to make each run cheap, safe and easy to read.

Probe first

Most input guesses go away when the model looks first. A probe job returns the facts it needs:

{ "mode": "probe", "input": { "url": "https://example.com/upload.mov" } }

The completed job has fields such as hasAudio, durationSeconds, width, height, videoCodec and audioCodec. The model writes the command after it reads them. See probe.

Arguments, not shell strings

A command string must go through a shell or a parser, which is where quoting goes wrong and where injection is possible. An argument array goes to FFmpeg one element per argument:

["-i", "{{in_1}}", "-vf", "drawtext=text='Live now':x=24:y=24:fontsize=36", "{{out_1}}"]

The model still needs FFmpeg’s own filter escaping, but there is no second layer of shell quoting, and no way for a value to become a new shell command.

Errors the model can read

When a job is refused, the model needs to know which field to fix, not just that FFmpeg exited with code 1. Xora returns errors as JSON:

{
  "httpStatus": 402,
  "error": {
    "code": "LIMIT_EXCEEDED",
    "message": "Plan credits exhausted",
    "feature": "credits",
    "retryable": false
  }
}

VALIDATION_ERROR means fix the request, and its message names the field. LIMIT_EXCEEDED means stop and tell the user the plan allowance is used up. A failed job carries an error with the stage it failed in. The model can branch on the code without parsing a log.

Retries that do not double the work

Agents retry. If a call times out, the agent cannot tell whether the job was created. An idempotencyKey on create_job makes the retry safe: the same key returns the job that already exists.

Mistakes that cost nothing

Failed and rejected Xora jobs cost nothing. A model that writes a bad filter graph and gets an error has lost a few seconds, not money, and it can try again with the error in hand.

A short system prompt for agents

If your agent has the Xora MCP server, a few lines in its instructions cover most of the above:

You can process video and audio with the Xora tools.
- Before you write an FFmpeg command for a file you have not seen, run create_job with mode "probe" and read the result.
- Prefer a recipe (call list_recipes once) when one fits. Use mode "ffmpeg" with an args array for anything else.
- Output file names use letters, numbers and underscores, plus an extension.
- Send an idempotencyKey with every create_job, and reuse it if you retry the same job.
- If a job fails, read error.code and error.message, fix the request, and try again once. Tell the user if it fails again.

Next steps

• FFmpeg for AI agents: connect the MCP server, the REST API or the agent skill.
• FFmpeg in AI agent sandboxes: why sandboxes are the wrong place for video jobs.
• FFmpeg cheat sheet: the common commands, each as an API call.

Frequently asked questions

Can ChatGPT or Claude write FFmpeg commands?

Yes. Language models write correct FFmpeg commands for most common tasks, such as converting, compressing, trimming, resizing and extracting audio. They make more mistakes with complex filter graphs, stream mapping and seeking, so it helps to check the input with ffprobe first and to run the command where errors come back in a form the model can read.

How do I let an AI agent run FFmpeg safely?

Run the commands in an isolated service, not on your own machine or server. Send the arguments as an array so there is no shell to inject into. Give the agent a key with spending limits, and let it read structured errors so it can fix a command and try again.

What should an LLM check before it writes an FFmpeg command?

The input's codecs, duration, resolution and whether it has an audio stream. With Xora, a probe job returns these fields before the agent picks a command, so it does not guess.