Real-time VisionCamera pipelines

This is the specialized companion to react-native-vision-camera. Optimize the complete path from Camera buffer to final result, not an isolated stage. Before relying on exact APIs, check installed versions against current VisionCamera docs and the consumer's official docs or source.

Choose by final consumer

Load references/interop.md only when implementing or reviewing Nitro, NativeBuffer, WebGPU, Skia, Resizer, or ArrayBuffer interop.

Hot-path invariants

1. Keep orientation and mirroring as metadata. Set enablePhysicalBufferRotation: false, then pass frame.orientation and frame.isMirrored to the consumer or apply them in the same GPU transform that scales, crops, or renders. Never rotate the Camera buffer physically.
2. Stay in one execution and memory domain. In a GPU pipeline, import once, keep preprocessing, inference, postprocessing, and rendering on the GPU, and read back only a compact result when required.
3. Prefer pixelFormat: 'native' for a verified GPU-only path. Check frame.pixelFormat and frame.hasNativeBuffer because the resolved native format may be YUV, RGB, RAW, or private.
4. Do not use getPixelBuffer(), getPlanes(), plane pixel buffers, mapped GPU buffers, or typed pixel views in the normal GPU path. CPU visibility can force synchronization or download.
5. Create and warm pipelines, shaders, samplers, model sessions, resizers, large buffers, and native processors once. Reuse them for the component or session lifetime; never allocate them per frame.
6. Draw frame-coupled overlays from the same Frame with Skia or WebGPU. Do not route per-frame geometry through React state, ordinary views, or Reanimated shared values.
7. Release every Frame, NativeBuffer, wrapper, texture, and pooled slot exactly once on every path. Release wrappers in reverse ownership order and dispose the Frame last.

Prefer same-frame processing

Keep detection, tracking, decisions, and drawing synchronous with the matching frame when they must align visually. At 60 FPS the hard interval is 16.67 ms; at 30 FPS it is 33.33 ms. Target under roughly 16 ms and 33 ms to leave scheduling margin.

"Synchronous" means same-frame dataflow, not blocking the CPU until the GPU finishes. Encode dependent GPU stages in one command graph when possible. Do not add per-frame queue.onSubmittedWorkDone(), buffer mapping, readback, or another CPU or GPU fence.

Before making work asynchronous, remove copies and readbacks, reduce input resolution or FPS, fuse passes, optimize model tensors, and reuse warmed state. Use async only when the optimized work still cannot fit the frame interval, often around 50 ms or more, and the product accepts stale results. For frame-coupled visuals, prefer simplifying the work over visible lag.

The async delivery patterns are peers:

• native Nitro work with a retained completion callback
• native Nitro work that stores completed state behind a synchronous latest-state getter
• a synchronous native method scheduled with VisionCamera's useAsyncRunner()

Every async design must bound in-flight work. Use one active task or a small fixed pool, reject or replace stale pending input, and never build an unbounded FIFO queue. dropFramesWhileBusy is an overload guard, not the architecture. With useAsyncRunner(), dispose an accepted Frame inside the task and a rejected Frame immediately.

Choose ML compute end to end

If inference feeds a same-frame Skia or WebGPU render, prefer keeping the entire path on the GPU. Crossing to an ANE, NPU, or CPU and returning geometry to the renderer is worthwhile only when end-to-end profiling proves it is faster while preserving the frame budget.

For state-only scanning, benchmark the platform runtime's available compute units. An ANE or NPU can avoid GPU contention and accelerate supported models; a CPU can win for tiny models when accelerator dispatch and transfer cost dominates. Measure input conversion, synchronization, inference, and result delivery, not inference alone. Normal React state or navigation is fine after a scan that has no frame-coupled overlay.

Development and production checks

When all native dependencies support it, use a resizable iPad-shaped Mac Catalyst or iPad-on-Mac build as a rapid iteration harness. A desktop agent can relaunch, resize, and screenshot it while using a built-in Mac camera or external UVC camera via useCameraDevice('external'). Fall back to a phone when the Mac target or required plugin is unavailable.

The Mac loop is for functional iteration, not performance prediction. Validate release builds on every production device class and representative GPUs. Test long enough to expose thermal throttling and pool leaks. Track:

• camera timestamp to matching result or presentation latency at median, p95, and p99
• dropped frames and maximum in-flight frames
• CPU and GPU time, readbacks, maps, and synchronization points
• allocations per frame, steady-state memory, sustained FPS, temperature, and power

Sample GPU timings asynchronously and sparsely enough that instrumentation does not become a synchronization point.

Authoritative references

• VisionCamera: docs index, performance, async processing, external cameras
• Rendering and compute: VisionCamera Skia, React Native WebGPU integration
• ML compute: Apple Core ML compute units, LiteRT NPU delegates