Process of generating clean masks from greenscreen/bluescreen footage — algorithmically or manually separates subject from background. First station in the keying pipeline.
After shooting with a greenscreen or bluescreen, the real work in the VFX workflow begins: you need a clean matte that separates the subject from the background. This is precisely what Matte Extraction is – the process of generating this matte algorithmically or manually from your raw keying material. It is the first critical stage in the entire keying pipeline, and it's here that the decision is made whether your composite will look believable or not.
The classic method works with the color difference between the screen and the subject. Automatic keyers analyze the color channels, detect gradients in the transition area, and calculate an initial matte from them. This works quickly, but rarely perfectly – especially in difficult situations like backlight shots, fine hair, or translucent fabrics. This is where manual post-processing comes in: rotoscoping, point-cloud-based extraction, or hybrid methods where you correct machine pre-calculations by hand. Adobe After Effects, Nuke, or specialized keying software like Keylight offer you tools for this – but without your control over feathering, edge handling, and luminance separation, the matte will end up holey or too hard.
The quality of your screen recordings themselves is crucial: even lighting, minimal wrinkles in the fabric, enough distance between talent and screen – this significantly reduces your extraction effort. Many compositors say that 70 percent of matte quality is already decided during the shoot. Spill Removal is often part of the extraction: green or blue light reflected from the screen onto hair or clothing must be filtered out without damaging the subject itself.
You typically save the final matte as a separate alpha channel or a dedicated grayscale file. This matte is then manipulated, refined, and applied to your background plate in the next step – keying, compositing, color correction. A bad extraction poisons all subsequent steps. Therefore, the effort here is worthwhile: saving time on clean mattes is false economy.