AI Background Removal (Matting) — BackgroundRemovalBlock¶

BackgroundRemovalBlock runs an ONNX segmentation or matting model on RGBA video frames, estimates a per-pixel foreground alpha mask, and composites a replacement background into the frame. Use it for portrait matting, virtual background blur, virtual green-screen output, static background replacement, or transparent RGBA output.

graph LR;
    Source-->BackgroundRemovalBlock;
    BackgroundRemovalBlock-->VideoRendererBlock;

The block lives in VisioForge.Core.AI (VisioForge.DotNet.Core.AI), uses BackgroundRemovalSettings (which extends OnnxInferenceSettings), and has one video Input and one video Output. It uses an internal sample grabber for RGBA frames and runs inference on the pipeline streaming thread, so a slow model can throttle the pipeline. It does not raise a recognition event — the output is the processed video frame.

Models and licensing¶

Set BackgroundRemovalSettings.Model to the family that matches the supplied .onnx file. The SDK does not ship model weights in the NuGet package; your application supplies the .onnx file.

How the matting pipeline works¶

BackgroundRemovalBlock is a frame processor, not a separate source or renderer. The upstream block delivers RGBA frames to an internal sample grabber, the block runs the configured ONNX model when the frame is selected for inference, and the output frame is pushed downstream with the same video timing.

The ONNX model input size is resolved from the model metadata when the model has fixed dimensions. If the model uses dynamic dimensions, InputWidth and InputHeight from BackgroundRemovalSettings are used instead — these default to 512x512 (tuned for MODNet) and are not adjusted per-model automatically, so set them yourself for U2Net (320x320) or BiRefNet (1024x1024). The frame is resized directly to that tensor size and converted from RGBA pixels to an RGB NCHW floating-point tensor. The model family controls the normalization:

• MODNet and PPMattingV2 use RGB values normalized to -1..1.
• U2Net and BiRefNet use ImageNet-style RGB mean/std normalization.
• Custom uses the generic NormalizeTo01, InputWidth, and InputHeight settings.

After inference, the block reads the first float output as a foreground matte. The last two output dimensions are treated as matte height and width. MODNet, PPMattingV2, and Custom outputs are expected to already be in 0..1; U2Net is min/max-normalized per frame; BiRefNet raw logits are converted with sigmoid. If MaskFeatherAmount is greater than 0, the matte is blurred at model resolution before compositing.

The compositor samples that matte back onto the original frame with bilinear interpolation. For Blur, SolidColor, and Image, each pixel is blended as foreground * alpha + background * (1 - alpha). MaskThreshold can harden uncertain edges by forcing very low alpha values to background and very high values to foreground. For Transparent, the block keeps the foreground RGB pixels and writes the matte into the output alpha channel, so the visible result depends on the downstream renderer or encoder preserving RGBA alpha.

Choosing a matting model¶

Use MODNet or PPMattingV2 first for real-time portrait background replacement. They are designed for human matting and have lower input sizes than high-accuracy salient-object models. U2Net is a good fallback when the subject is not always a person, but its output is a segmentation-style matte that may need MaskFeatherAmount for soft edges. BiRefNet is the heavier quality option: it can produce better fine detail, but its typical 1024x1024 input size is much more expensive, especially on CPU. Use Custom only when your ONNX model follows a different preprocessing or output convention and you have verified those settings against the exported model.

Replacement modes¶

BackgroundRemovalSettings.ReplacementMode selects how low-alpha background pixels are replaced.

Additional matte controls:

Pipeline example¶

using VisioForge.Core.MediaBlocks;
using VisioForge.Core.MediaBlocks.AI;
using VisioForge.Core.MediaBlocks.VideoRendering;
using VisioForge.Core.Types.X.AI;

var settings = new BackgroundRemovalSettings(modelPath)
{
    Model = BackgroundRemovalModel.PPMattingV2,
    ReplacementMode = BackgroundReplacementMode.Blur,
    BlurRadius = 15f,
    MaskFeatherAmount = 2,
    Provider = OnnxExecutionProvider.Auto,
    FramesToSkip = 2,
    MaskThreshold = 0.05f,
};

var backgroundRemoval = new BackgroundRemovalBlock(settings);

var videoRenderer = new VideoRendererBlock(pipeline, videoView) { IsSync = false };

pipeline.Connect(source.Output, backgroundRemoval.Input);
pipeline.Connect(backgroundRemoval.Output, videoRenderer.Input);

await pipeline.StartAsync();

Console.WriteLine($"Active provider: {backgroundRemoval.ActiveProvider}");

Use with VideoCaptureCoreX and MediaPlayerCoreX¶

var backgroundRemoval = new BackgroundRemovalBlock(settings);

core.Video_Processing_AddBlock(backgroundRemoval); // before StartAsync (VideoCaptureCoreX)
// player.Video_Processing_AddBlock(backgroundRemoval); // before OpenAsync/PlayAsync (MediaPlayerCoreX)

await core.StartAsync();

See Using AI blocks with VideoCaptureCoreX and MediaPlayerCoreX for the full processing-block API, insertion order, and lifecycle rules shared by every video AI block.

Use cases¶

• Video conferencing and webcam apps — virtual green screen or background blur without a physical green screen.
• Livestreaming and broadcast overlays — composite a presenter over a branded background or scene.
• Virtual studio / product photography video — replace a plain backdrop with a custom scene.
• Privacy blurring — blur the background in a recording so only the foreground subject is sharp.

Troubleshooting¶

Frequently Asked Questions¶

Which matting model should I start with?¶

MODNet (the default) or PPMattingV2 for real-time portrait/webcam scenarios — both are tuned for human matting at a relatively low input size. Use BiRefNet only if you need higher fine-detail accuracy and can afford its heavier 1024x1024 input cost.

Can I use a virtual green screen without a physical one?¶

Yes — set ReplacementMode = BackgroundReplacementMode.SolidColor (it defaults to green) instead of Blur, Image, or Transparent.

Does this block require a GPU?¶

No, but a GPU execution provider reduces per-frame latency, which matters most for BiRefNet's larger input size or for high frame-rate live video.

Can I output a transparent (alpha) video instead of compositing a background?¶

Yes — set ReplacementMode = BackgroundReplacementMode.Transparent. The block writes the foreground alpha into the frame's alpha channel; your renderer, encoder, or downstream compositor must preserve RGBA to make use of it.

Demo¶

• Background Removal Demo — WPF demo with webcam, file, and RTSP sources, downloadable matting models, blur, solid color, image, and transparent replacement modes.