Edge or boundary of a mask — defines where subject ends and background begins. Can be hard or soft; often requires manual refinement in final stages.
The matte line determines whether a mask shows clean or unclean edges in the final image. It's about the precise boundary between the isolated subject and the background—and this edge is almost never perfect on the first pass. You cut out your subject, whether by rotoscope, keying, or automatic masking, and afterwards, the line is set: sometimes hard and sharp-edged, sometimes blurred, sometimes with pixel fringes or unintentional color bleeding. That's the matte line—and you'll work on it until the flaw is no longer noticeable.
Hardness and softness of the edge are two extreme poles. A hard matte line works for geometric objects, buildings, or vehicles in parallel lighting. But for human hair, fire, or transparent objects, an overly sharp edge immediately becomes a flaw—there you need a soft, graduated boundary zone that blends with the background. In compositing, this can be adjusted via blur, falloff, or edge expansion. A classic beginner's trap: leaving the line too dark or too bright because you don't invest enough time in the color control of the edge.
On set, you only notice matte line problems later—in editing or VFX supervision. A bluescreen shot with poor lighting creates an unclean edge where green or blue color fringes remain. A rotoscope project that was worked on under time pressure leaves shaky lines where the roto artist wasn't frame-by-frame precise. Digital artifacts during keying—haloing, color spill—are also matte line problems. You see them clearly when you view the layer in isolation against a black or white background.
Practitioners reduce matte line work through smart preparation: clean keying, calibrated lighting on set, or via a rotoscope workflow with feather and edge refinement already during masking. In DaVinci or Nuke, the line can still be corrected in the final hour—with erode/dilate, color range tuning, or manual edge fixes. But that costs time. It's better if the line is set correctly on the first pass. Then the matte line is invisible, and that's the only goal.