Visual effects produced with generative AI models — inpainting, object removal, automatic rotoscoping, and diffusion-based compositing since 2024.
AI VFX is the collective term for visual effects whose generation or processing is based on generative AI models. In contrast to classic VFX, which relies on compositing software like Nuke or After Effects, diffusion models, vision transformers, and feed-forward neural nets are used here – either as standalone tools or as plug-ins in the NLE.
Pipeline Position
AI VFX is predominantly used in post-production. Typical tasks: Object Removal (wire rigs, boom mics, reflections), inpainting behind cut-out subjects, roto without manual frame-by-frame pulling, background replacement without green screen, relighting existing shots, style transfer for entire sequences.
Tools 2026
- Runway Gen-4 / Gen-4.5 — Reference-to-Video, Multi-Shot Sequencing, native audio generation.
- Wonder Dynamics (Autodesk Flow Studio) — Markerless body tracking + automatic character replacement.
- Meta SAM 3 — Rotoscoping, 3D object generation from image/video, free via Playground.
- Beeble Switch X — Generative relighting + background swap with subject consistency.
- Netflix VOID — Open-source object removal with physics awareness, Apache-2.0.
- NVIDIA Lyra 2.0 — Single image to navigable 3D world, open weights on HuggingFace.
- Topaz Video AI — Upscaling, frame interpolation, stabilization as a pre-edit step.
- DaVinci Resolve 21 — Eight new AI tools natively in the Edit page (Reshaper, Speech Gen, IntelliSearch).
- Adobe Firefly Video Editor — Text-based editing, generative SFX, Color Mode.
On-Set Practice
The most significant change is not the tool – it's set discipline. If a background is to be replaced later, the lighting continuity must be tighter, not looser. The lighting of the subject should match the eventual compositing background, otherwise the final result will look like a green screen cutout. DPs now plan their key scenes with the eventual AI VFX workflow in mind, not just in post.
Practical Tips
- Always film reference frames – even for plates that will be completely replaced later. A 5-second reference of the lighting setup saves hours in post.
- Audio plates as well: Room tone and Walla are often forgotten during AI background swaps, then they are missing for sound continuity.
- Frame rate consistency is mandatory. AI tools compute most cleanly at 24 fps cinema standard; mixed forms with 30/60/120 fps cause tracking errors.
- Clarify output format in advance: 1080p is fine for streaming, 4K DCI for cinema masters. Some tools internally render lower resolutions and upscale – this becomes noticeable on large screens.
History
The precursors were in 2022 – Stable Diffusion made inpainting popular in Photoshop plugins. 2023 saw the release of Runway Gen-2 as the first production-ready video tool, followed by Pika 1.0. 2024 marks the leap into Hollywood pipelines: Industrial Light & Magic adopts Wonder Dynamics, ARRI integrates AI modules into its workflow tools, and with OpenAI's Sora demo, text-to-video becomes mainstream discourse. In 2025, Sora crashes (app shutdown April 26, 2026), and the market reorganizes around open-source models (Hunyuan, Wan 2.7, LTX-2) and closed API premium (Veo 3.1, Kling 3.0, HappyHorse 1.0).