MCPInstall

axiom-avfoundation-ref

Reference — AVFoundation audio APIs, AVAudioSession categories/modes, AVAudioEngine pipelines, bit-perfect DAC output, iOS 26+ spatial audio capture, ASAF/APAC, Audio Mix with Cinematic framework

Installs
221
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/charleswiltgen/axiom --skill axiom-avfoundation-ref
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads axiom-avfoundation-ref automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

AVFoundation Audio Reference

Quick Reference

// AUDIO SESSION SETUP
import AVFoundation

try AVAudioSession.sharedInstance().setCategory(
    .playback,                              // or .playAndRecord, .ambient
    mode: .default,                         // or .voiceChat, .measurement
    options: [.mixWithOthers, .allowBluetooth]
)
try AVAudioSession.sharedInstance().setActive(true)

// AUDIO ENGINE PIPELINE
let engine = AVAudioEngine()
let player = AVAudioPlayerNode()
engine.attach(player)
engine.connect(player, to: engine.mainMixerNode, format: nil)
try engine.start()
player.scheduleFile(audioFile, at: nil)
player.play()

// INPUT PICKER (iOS 26+)
import AVKit
let picker = AVInputPickerInteraction()
picker.delegate = self
myButton.addInteraction(picker)
// In button action: picker.present()

// AIRPODS HIGH QUALITY (iOS 26+)
try AVAudioSession.sharedInstance().setCategory(
    .playAndRecord,
    options: [.bluetoothHighQualityRecording, .allowBluetoothA2DP]
)

AVAudioSession

Categories

CategoryUse CaseSilent SwitchBackground
.ambientGame sounds, not primarySilencesNo
.soloAmbientDefault, interrupts othersSilencesNo
.playbackMusic player, podcastIgnoresYes
.recordVoice recorder—Yes
.playAndRecordVoIP, voice chatIgnoresYes
.multiRouteDJ apps, multiple outputsIgnoresYes

Modes

ModeUse Case
.defaultGeneral audio
.voiceChatVoIP, reduces echo
.videoChatFaceTime-style
.gameChatVoice chat in games
.videoRecordingCamera recording
.measurementFlat response, no processing
.moviePlaybackVideo playback
.spokenAudioPodcasts, audiobooks

Options

// Mixing
.mixWithOthers          // Play with other apps
.duckOthers             // Lower other audio while playing
.interruptSpokenAudioAndMixWithOthers  // Pause podcasts, mix music

// Bluetooth
.allowBluetooth         // HFP (calls)
.allowBluetoothA2DP     // High quality stereo
.bluetoothHighQualityRecording  // iOS 26+ AirPods recording

// Routing
.defaultToSpeaker       // Route to speaker (not receiver)
.allowAirPlay           // Enable AirPlay

Interruption Handling

NotificationCenter.default.addObserver(
    forName: AVAudioSession.interruptionNotification,
    object: nil,
    queue: .main
) { notification in
    guard let userInfo = notification.userInfo,
          let typeValue = userInfo[AVAudioSessionInterruptionTypeKey] as? UInt,
          let type = AVAudioSession.InterruptionType(rawValue: typeValue) else {
        return
    }

    switch type {
    case .began:
        // Pause playback
        player.pause()

    case .ended:
        guard let optionsValue = userInfo[AVAudioSessionInterruptionOptionKey] as? UInt else { return }
        let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue)
        if options.contains(.shouldResume) {
            player.play()
        }

    @unknown default:
        break
    }
}

Route Change Handling

NotificationCenter.default.addObserver(
    forName: AVAudioSession.routeChangeNotification,
    object: nil,
    queue: .main
) { notification in
    guard let userInfo = notification.userInfo,
          let reasonValue = userInfo[AVAudioSessionRouteChangeReasonKey] as? UInt,
          let reason = AVAudioSession.RouteChangeReason(rawValue: reasonValue) else {
        return
    }

    switch reason {
    case .oldDeviceUnavailable:
        // Headphones unplugged — pause playback
        player.pause()

    case .newDeviceAvailable:
        // New device connected
        break

    case .categoryChange:
        // Category changed by system or another app
        break

    default:
        break
    }
}

AVAudioEngine

Basic Pipeline

let engine = AVAudioEngine()

// Create nodes
let player = AVAudioPlayerNode()
let reverb = AVAudioUnitReverb()
reverb.loadFactoryPreset(.largeHall)
reverb.wetDryMix = 50

// Attach to engine
engine.attach(player)
engine.attach(reverb)

// Connect: player → reverb → mixer → output
engine.connect(player, to: reverb, format: nil)
engine.connect(reverb, to: engine.mainMixerNode, format: nil)

// Start
engine.prepare()
try engine.start()

// Play file
let url = Bundle.main.url(forResource: "audio", withExtension: "m4a")!
let file = try AVAudioFile(forReading: url)
player.scheduleFile(file, at: nil)
player.play()

Node Types

NodePurpose
AVAudioPlayerNodePlays audio files/buffers
AVAudioInputNodeMic input (engine.inputNode)
AVAudioOutputNodeSpeaker output (engine.outputNode)
AVAudioMixerNodeMix multiple inputs
AVAudioUnitEQEqualizer
AVAudioUnitReverbReverb effect
AVAudioUnitDelayDelay effect
AVAudioUnitDistortionDistortion effect
AVAudioUnitTimePitchTime stretch / pitch shift

Installing Taps (Audio Analysis)

let inputNode = engine.inputNode
let format = inputNode.outputFormat(forBus: 0)

inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { buffer, time in
    // Process audio buffer
    guard let channelData = buffer.floatChannelData?[0] else { return }
    let frameLength = Int(buffer.frameLength)

    // Calculate RMS level
    var sum: Float = 0
    for i in 0..<frameLength {
        sum += channelData[i] * channelData[i]
    }
    let rms = sqrt(sum / Float(frameLength))
    let dB = 20 * log10(rms)

    DispatchQueue.main.async {
        self.levelMeter = dB
    }
}

// Don't forget to remove when done
inputNode.removeTap(onBus: 0)

Format Conversion

// AVAudioEngine mic input is always 44.1kHz/32-bit float
// Use AVAudioConverter for other formats

let inputFormat = engine.inputNode.outputFormat(forBus: 0)
let outputFormat = AVAudioFormat(
    commonFormat: .pcmFormatInt16,
    sampleRate: 48000,
    channels: 1,
    interleaved: false
)!

let converter = AVAudioConverter(from: inputFormat, to: outputFormat)!

// In tap callback:
let outputBuffer = AVAudioPCMBuffer(
    pcmFormat: outputFormat,
    frameCapacity: AVAudioFrameCount(outputFormat.sampleRate * 0.1)
)!

var error: NSError?
converter.convert(to: outputBuffer, error: &error) { inNumPackets, outStatus in
    outStatus.pointee = .haveData
    return inputBuffer
}

Bit-Perfect Audio / DAC Output

iOS Behavior

iOS provides bit-perfect output by default to USB DACs — no resampling occurs. The DAC receives the source sample rate directly.

// iOS automatically matches source sample rate to DAC
// No special configuration needed for bit-perfect output

let player = AVAudioPlayerNode()
// File at 96kHz → DAC receives 96kHz

Avoiding Resampling

// Check hardware sample rate
let hardwareSampleRate = AVAudioSession.sharedInstance().sampleRate

// Match your audio format to hardware when possible
let format = AVAudioFormat(
    standardFormatWithSampleRate: hardwareSampleRate,
    channels: 2
)

USB DAC Routing

// List available outputs
let currentRoute = AVAudioSession.sharedInstance().currentRoute
for output in currentRoute.outputs {
    print("Output: \(output.portName), Type: \(output.portType)")
    // USB DAC shows as .usbAudio
}

// Prefer USB output
try AVAudioSession.sharedInstance().setPreferredInput(usbPort)

Sample Rate Considerations

SourceiOS BehaviorNotes
44.1 kHzPassthroughCD quality
48 kHzPassthroughVideo standard
96 kHzPassthroughHi-res
192 kHzPassthroughHi-res
DSDNot supportedUse DoP or convert

iOS 26+ Input Selection

AVInputPickerInteraction

Native input device selection with live metering:

import AVKit

class RecordingViewController: UIViewController {
    let inputPicker = AVInputPickerInteraction()

    override func viewDidLoad() {
        super.viewDidLoad()

        // Configure audio session first
        try? AVAudioSession.sharedInstance().setCategory(.playAndRecord)
        try? AVAudioSession.sharedInstance().setActive(true)

        // Setup picker
        inputPicker.delegate = self
        selectMicButton.addInteraction(inputPicker)
    }

    @IBAction func selectMicTapped(_ sender: UIButton) {
        inputPicker.present()
    }
}

extension RecordingViewController: AVInputPickerInteractionDelegate {
    // Implement delegate methods as needed
}

Features:

  • Live sound level metering
  • Microphone mode selection
  • System remembers selection per app

iOS 26+ AirPods High Quality Recording

LAV-microphone equivalent quality for content creators:

// AVAudioSession approach
try AVAudioSession.sharedInstance().setCategory(
    .playAndRecord,
    options: [
        .bluetoothHighQualityRecording,  // New in iOS 26
        .allowBluetoothA2DP              // Fallback
    ]
)

// AVCaptureSession approach
let captureSession = AVCaptureSession()
captureSession.configuresApplicationAudioSessionForBluetoothHighQualityRecording = true

Notes:

  • Uses dedicated Bluetooth link optimized for AirPods
  • Falls back to HFP if device doesn't support HQ mode
  • Supports AirPods stem controls for start/stop recording

Spatial Audio Capture (iOS 26+)

First Order Ambisonics (FOA)

Record 3D spatial audio using device microphone array:

// With AVCaptureMovieFileOutput (simple)
let audioInput = AVCaptureDeviceInput(device: audioDevice)
audioInput.multichannelAudioMode = .firstOrderAmbisonics

// With AVAssetWriter (full control)
// Requires two AudioDataOutputs: FOA (4ch) + Stereo (2ch)

AVAssetWriter Spatial Audio Setup

// Configure two AudioDataOutputs
let foaOutput = AVCaptureAudioDataOutput()
foaOutput.spatialAudioChannelLayoutTag = kAudioChannelLayoutTag_HOA_ACN_SN3D  // 4 channels

let stereoOutput = AVCaptureAudioDataOutput()
stereoOutput.spatialAudioChannelLayoutTag = kAudioChannelLayoutTag_Stereo    // 2 channels

// Create metadata generator
let metadataGenerator = AVCaptureSpatialAudioMetadataSampleGenerator()

// Feed FOA buffers to generator
func captureOutput(_ output: AVCaptureOutput,
                   didOutput sampleBuffer: CMSampleBuffer,
                   from connection: AVCaptureConnection) {
    metadataGenerator.append(sampleBuffer)
    // Also write to FOA AssetWriterInput
}

// When recording stops, get metadata sample
let metadataSample = metadataGenerator.createMetadataSample()
// Write to metadata track

Output File Structure

Spatial audio files contain:

  1. Stereo AAC track — Compatibility fallback
  2. APAC track — Spatial audio (FOA)
  3. Metadata track — Audio Mix tuning parameters

File formats: .mov, .mp4, .qta (QuickTime Audio, iOS 26+)


ASAF / APAC (Apple Spatial Audio)

Overview

ComponentPurpose
ASAFApple Spatial Audio Format — production format
APACApple Positional Audio Codec — delivery codec

APAC Capabilities

  • Bitrates: 64 kbps to 768 kbps
  • Supports: Channels, Objects, Higher Order Ambisonics, Dialogue, Binaural
  • Head-tracked rendering adaptive to listener position/orientation
  • Required for Apple Immersive Video

Playback

// Standard AVPlayer handles APAC automatically
let player = AVPlayer(url: spatialAudioURL)
player.play()

// Head tracking enabled automatically on AirPods

Platform Support

All Apple platforms except watchOS support APAC playback.


Audio Mix (Cinematic Framework)

Separate and remix speech vs ambient sounds in spatial recordings:

AVPlayer Integration

import Cinematic

// Load spatial audio asset
let asset = AVURLAsset(url: spatialAudioURL)
let audioInfo = try await CNAssetSpatialAudioInfo(asset: asset)

// Configure mix parameters
let intensity: Float = 0.5  // 0.0 to 1.0
let style = CNSpatialAudioRenderingStyle.cinematic

// Create and apply audio mix
let audioMix = audioInfo.audioMix(
    effectIntensity: intensity,
    renderingStyle: style
)
playerItem.audioMix = audioMix

Rendering Styles

StyleEffect
.cinematicBalanced speech/ambient
.studioEnhanced speech clarity
.inFrameFocus on visible speakers
+ 6 extraction modesSpeech-only, ambient-only stems

AUAudioMix (Direct AudioUnit)

For apps not using AVPlayer:

// Input: 4 channels FOA
// Output: Separated speech + ambient

// Get tuning metadata from file
let audioInfo = try await CNAssetSpatialAudioInfo(asset: asset)
let remixMetadata = audioInfo.spatialAudioMixMetadata as CFData

// Apply to AudioUnit via AudioUnitSetProperty

Common Patterns

Background Audio Playback

// 1. Set category
try AVAudioSession.sharedInstance().setCategory(.playback)

// 2. Enable background mode in Info.plist
// <key>UIBackgroundModes</key>
// <array><string>audio</string></array>

// 3. Set Now Playing info (recommended)
let nowPlayingInfo: [String: Any] = [
    MPMediaItemPropertyTitle: "Song Title",
    MPMediaItemPropertyArtist: "Artist",
    MPNowPlayingInfoPropertyElapsedPlaybackTime: player.currentTime,
    MPMediaItemPropertyPlaybackDuration: duration
]
MPNowPlayingInfoCenter.default().nowPlayingInfo = nowPlayingInfo

Ducking Other Audio

try AVAudioSession.sharedInstance().setCategory(
    .playback,
    options: .duckOthers
)

// When done, restore others
try AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)

Bluetooth Device Handling

// Allow all Bluetooth
try AVAudioSession.sharedInstance().setCategory(
    .playAndRecord,
    options: [.allowBluetooth, .allowBluetoothA2DP]
)

// Check current Bluetooth route
let route = AVAudioSession.sharedInstance().currentRoute
let hasBluetoothOutput = route.outputs.contains {
    $0.portType == .bluetoothA2DP || $0.portType == .bluetoothHFP
}

Anti-Patterns

Wrong Category

// WRONG — music player using ambient (silenced by switch)
try AVAudioSession.sharedInstance().setCategory(.ambient)

// CORRECT — music needs .playback
try AVAudioSession.sharedInstance().setCategory(.playback)

Missing Interruption Handling

// WRONG — no interruption observer
// Audio stops on phone call and never resumes

// CORRECT — always handle interruptions
NotificationCenter.default.addObserver(
    forName: AVAudioSession.interruptionNotification,
    // ... handle began/ended
)

Tap Memory Leaks

// WRONG — tap installed, never removed
engine.inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { ... }

// CORRECT — remove tap when done
deinit {
    engine.inputNode.removeTap(onBus: 0)
}

Format Mismatch Crashes

// WRONG — connecting nodes with incompatible formats
engine.connect(playerNode, to: mixerNode, format: wrongFormat)  // Crash!

// CORRECT — use nil for automatic format negotiation, or match exactly
engine.connect(playerNode, to: mixerNode, format: nil)

Forgetting to Activate Session

// WRONG — configure but don't activate
try AVAudioSession.sharedInstance().setCategory(.playback)
// Audio doesn't work!

// CORRECT — always activate
try AVAudioSession.sharedInstance().setCategory(.playback)
try AVAudioSession.sharedInstance().setActive(true)

Resources

WWDC: 2025-251, 2025-403, 2019-510

Docs: /avfoundation, /avkit, /cinematic


Targets: iOS 12+ (core), iOS 26+ (spatial features) Frameworks: AVFoundation, AVKit, Cinematic (iOS 26+) History: See git log for changes

FAQ

What does the axiom-avfoundation-ref skill do?

Reference — AVFoundation audio APIs, AVAudioSession categories/modes, AVAudioEngine pipelines, bit-perfect DAC output, iOS 26+ spatial audio capture, ASAF/APAC, Audio Mix with Cinematic framework

How do I install axiom-avfoundation-ref?

Run npx skills add https://github.com/charleswiltgen/axiom --skill axiom-avfoundation-ref in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/axiom-avfoundation-ref/ (personal) or .claude/skills/axiom-avfoundation-ref/ (project).

Which AI agents can use axiom-avfoundation-ref?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is axiom-avfoundation-ref safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains axiom-avfoundation-ref?

It is published by charleswiltgen in charleswiltgen/axiom. It has 221 recorded installs.

More from charleswiltgen/axiom

View all →

axiom-vision

charleswiltgen/axiom

Use when implementing ANY computer vision feature — image analysis, pose detection, person segmentation, subject lifting, text recognition, barcode scanning.

Media & Creative1.8K installs

axiom-swiftui

charleswiltgen/axiom

Use when building, fixing, or improving ANY SwiftUI UI — views, navigation, layout, animations, gestures, debugging, iOS 26 features, iPhone Duo, view-level performance, feature architecture.

Mobile1.5K installs

axiom-networking

charleswiltgen/axiom

Use when implementing or debugging ANY network connection, API call, or socket in an Apple app. Covers URLSession, Network.framework, NetworkConnection, connection diagnostics.

Backend & APIs1.5K installs

axiom-apple-docs

charleswiltgen/axiom

Use when you need Apple's own documentation or a Swift compiler diagnostic explained rather than recalled.

Docs & Writing1.5K installs

axiom-shipping

charleswiltgen/axiom

Use when preparing an app for App Store submission, handling rejections or appeals, managing App Store Connect and Xcode Cloud, or triaging TestFlight/Sentry crash corpora. Covers checklists, metadata, privacy manifests, age ratings, export compliance.

Mobile1.4K installs

axiom-xcode-mcp

charleswiltgen/axiom

Use when connecting to Xcode via MCP, using xcrun mcpbridge or the headless mcp-server, or working with ANY Xcode MCP tool (XcodeRead, BuildProject, RunSomeTests, RenderPreview). Covers setup, tool reference, workflows, troubleshooting.

AI & Agents1.4K installs

Popular media & creative skills

View category →

hyperframes

heygen-com/hyperframes

Read first to make, edit, animate or render any video with HyperFrames (HTML rendered to video), including launch films, explainers, music videos, charts, maps, vertical clips, title cards, captions and overlays. Holds the render contract, the work loop and what each kind of video needs.

Media & Creative1.2M installs

media-use

heygen-com/hyperframes

Get or make any media for a HyperFrames video - music, sound effects, images, icons, logos, voiceover, avatar and generated video, transcripts, cutouts, colour grades and looks - with HeyGen or free local tools, and mix placed audio. Also for footage that looks dark, flat or dull, or should be stylized or hidden.

Media & Creative1.1M installs

hyperframes-registry

heygen-com/hyperframes

About 450 ready-made HyperFrames blocks and components installed as editable source - transitions, overlays, titles, kinetic type, captions, charts, map overlays, 3D, camera moves. Use it before hand-building any named visual (CRT scanlines, glitch, film grain, a chart); also for hyperframes add and catalog.

Media & Creative1M installs

hyperframes-creative

heygen-com/hyperframes

Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use hyperframes-animation.

Media & Creative916.5K installs

ai-video-generation

101-skills/superpowers

Generate AI videos with Google Veo, Seedance 2.0, HappyHorse, Wan, Grok and 40+ models via inference.sh CLI. Models: Veo 3.1, Veo 3, Seedance 2.0, HappyHorse 1.0, Wan 2.5, Grok Imagine Video, OmniHuman, Fabric, HunyuanVideo. Capabilities: text-to-video, image-to-video, reference-to-video, video editing, lipsync, avatar animation, video upscaling, foley sound. Use for: social media videos, marketing content, explainer videos, product demos, AI avatars. Triggers: video generation, ai video, text to video, image to video, veo, animate image, video from image, ai animation, video generator, generate video, t2v, i2v, ai video maker, create video with ai, runway alternative, pika alternative, sora alternative, kling alternative, seedance, happyhorse

Media & Creative881.5K installs

ai-image-generation

101-skills/superpowers

Generate AI images with GPT-Image-2, FLUX, Gemini, Grok, Seedream, Reve and 50+ models via inference.sh CLI. Models: GPT-Image-2, FLUX Dev LoRA, FLUX.2 Klein LoRA, Gemini 3 Pro Image, Grok Imagine, Seedream 4.5, Reve, ImagineArt. Capabilities: text-to-image, image-to-image, inpainting, LoRA, image editing, upscaling, text rendering. Use for: AI art, product mockups, concept art, social media graphics, marketing visuals, illustrations. Triggers: flux, image generation, ai image, text to image, stable diffusion, generate image, ai art, midjourney alternative, dall-e alternative, text2img, t2i, image generator, ai picture, create image with ai, generative ai, ai illustration, grok image, gemini image, gpt image, openai image, chatgpt image

Media & Creative880.3K installs