
axiom-avfoundation-ref
Reference — AVFoundation audio APIs, AVAudioSession categories/modes, AVAudioEngine pipelines, bit-perfect DAC output, iOS 26+ spatial audio capture, ASAF/APAC, Audio Mix with Cinematic framework
- Installs
- 221
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/charleswiltgen/axiom --skill axiom-avfoundation-ref- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads axiom-avfoundation-ref automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
AVFoundation Audio Reference
Quick Reference
// AUDIO SESSION SETUP
import AVFoundation
try AVAudioSession.sharedInstance().setCategory(
.playback, // or .playAndRecord, .ambient
mode: .default, // or .voiceChat, .measurement
options: [.mixWithOthers, .allowBluetooth]
)
try AVAudioSession.sharedInstance().setActive(true)
// AUDIO ENGINE PIPELINE
let engine = AVAudioEngine()
let player = AVAudioPlayerNode()
engine.attach(player)
engine.connect(player, to: engine.mainMixerNode, format: nil)
try engine.start()
player.scheduleFile(audioFile, at: nil)
player.play()
// INPUT PICKER (iOS 26+)
import AVKit
let picker = AVInputPickerInteraction()
picker.delegate = self
myButton.addInteraction(picker)
// In button action: picker.present()
// AIRPODS HIGH QUALITY (iOS 26+)
try AVAudioSession.sharedInstance().setCategory(
.playAndRecord,
options: [.bluetoothHighQualityRecording, .allowBluetoothA2DP]
)
AVAudioSession
Categories
| Category | Use Case | Silent Switch | Background |
|---|---|---|---|
.ambient | Game sounds, not primary | Silences | No |
.soloAmbient | Default, interrupts others | Silences | No |
.playback | Music player, podcast | Ignores | Yes |
.record | Voice recorder | — | Yes |
.playAndRecord | VoIP, voice chat | Ignores | Yes |
.multiRoute | DJ apps, multiple outputs | Ignores | Yes |
Modes
| Mode | Use Case |
|---|---|
.default | General audio |
.voiceChat | VoIP, reduces echo |
.videoChat | FaceTime-style |
.gameChat | Voice chat in games |
.videoRecording | Camera recording |
.measurement | Flat response, no processing |
.moviePlayback | Video playback |
.spokenAudio | Podcasts, audiobooks |
Options
// Mixing
.mixWithOthers // Play with other apps
.duckOthers // Lower other audio while playing
.interruptSpokenAudioAndMixWithOthers // Pause podcasts, mix music
// Bluetooth
.allowBluetooth // HFP (calls)
.allowBluetoothA2DP // High quality stereo
.bluetoothHighQualityRecording // iOS 26+ AirPods recording
// Routing
.defaultToSpeaker // Route to speaker (not receiver)
.allowAirPlay // Enable AirPlay
Interruption Handling
NotificationCenter.default.addObserver(
forName: AVAudioSession.interruptionNotification,
object: nil,
queue: .main
) { notification in
guard let userInfo = notification.userInfo,
let typeValue = userInfo[AVAudioSessionInterruptionTypeKey] as? UInt,
let type = AVAudioSession.InterruptionType(rawValue: typeValue) else {
return
}
switch type {
case .began:
// Pause playback
player.pause()
case .ended:
guard let optionsValue = userInfo[AVAudioSessionInterruptionOptionKey] as? UInt else { return }
let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue)
if options.contains(.shouldResume) {
player.play()
}
@unknown default:
break
}
}
Route Change Handling
NotificationCenter.default.addObserver(
forName: AVAudioSession.routeChangeNotification,
object: nil,
queue: .main
) { notification in
guard let userInfo = notification.userInfo,
let reasonValue = userInfo[AVAudioSessionRouteChangeReasonKey] as? UInt,
let reason = AVAudioSession.RouteChangeReason(rawValue: reasonValue) else {
return
}
switch reason {
case .oldDeviceUnavailable:
// Headphones unplugged — pause playback
player.pause()
case .newDeviceAvailable:
// New device connected
break
case .categoryChange:
// Category changed by system or another app
break
default:
break
}
}
AVAudioEngine
Basic Pipeline
let engine = AVAudioEngine()
// Create nodes
let player = AVAudioPlayerNode()
let reverb = AVAudioUnitReverb()
reverb.loadFactoryPreset(.largeHall)
reverb.wetDryMix = 50
// Attach to engine
engine.attach(player)
engine.attach(reverb)
// Connect: player → reverb → mixer → output
engine.connect(player, to: reverb, format: nil)
engine.connect(reverb, to: engine.mainMixerNode, format: nil)
// Start
engine.prepare()
try engine.start()
// Play file
let url = Bundle.main.url(forResource: "audio", withExtension: "m4a")!
let file = try AVAudioFile(forReading: url)
player.scheduleFile(file, at: nil)
player.play()
Node Types
| Node | Purpose |
|---|---|
AVAudioPlayerNode | Plays audio files/buffers |
AVAudioInputNode | Mic input (engine.inputNode) |
AVAudioOutputNode | Speaker output (engine.outputNode) |
AVAudioMixerNode | Mix multiple inputs |
AVAudioUnitEQ | Equalizer |
AVAudioUnitReverb | Reverb effect |
AVAudioUnitDelay | Delay effect |
AVAudioUnitDistortion | Distortion effect |
AVAudioUnitTimePitch | Time stretch / pitch shift |
Installing Taps (Audio Analysis)
let inputNode = engine.inputNode
let format = inputNode.outputFormat(forBus: 0)
inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { buffer, time in
// Process audio buffer
guard let channelData = buffer.floatChannelData?[0] else { return }
let frameLength = Int(buffer.frameLength)
// Calculate RMS level
var sum: Float = 0
for i in 0..<frameLength {
sum += channelData[i] * channelData[i]
}
let rms = sqrt(sum / Float(frameLength))
let dB = 20 * log10(rms)
DispatchQueue.main.async {
self.levelMeter = dB
}
}
// Don't forget to remove when done
inputNode.removeTap(onBus: 0)
Format Conversion
// AVAudioEngine mic input is always 44.1kHz/32-bit float
// Use AVAudioConverter for other formats
let inputFormat = engine.inputNode.outputFormat(forBus: 0)
let outputFormat = AVAudioFormat(
commonFormat: .pcmFormatInt16,
sampleRate: 48000,
channels: 1,
interleaved: false
)!
let converter = AVAudioConverter(from: inputFormat, to: outputFormat)!
// In tap callback:
let outputBuffer = AVAudioPCMBuffer(
pcmFormat: outputFormat,
frameCapacity: AVAudioFrameCount(outputFormat.sampleRate * 0.1)
)!
var error: NSError?
converter.convert(to: outputBuffer, error: &error) { inNumPackets, outStatus in
outStatus.pointee = .haveData
return inputBuffer
}
Bit-Perfect Audio / DAC Output
iOS Behavior
iOS provides bit-perfect output by default to USB DACs — no resampling occurs. The DAC receives the source sample rate directly.
// iOS automatically matches source sample rate to DAC
// No special configuration needed for bit-perfect output
let player = AVAudioPlayerNode()
// File at 96kHz → DAC receives 96kHz
Avoiding Resampling
// Check hardware sample rate
let hardwareSampleRate = AVAudioSession.sharedInstance().sampleRate
// Match your audio format to hardware when possible
let format = AVAudioFormat(
standardFormatWithSampleRate: hardwareSampleRate,
channels: 2
)
USB DAC Routing
// List available outputs
let currentRoute = AVAudioSession.sharedInstance().currentRoute
for output in currentRoute.outputs {
print("Output: \(output.portName), Type: \(output.portType)")
// USB DAC shows as .usbAudio
}
// Prefer USB output
try AVAudioSession.sharedInstance().setPreferredInput(usbPort)
Sample Rate Considerations
| Source | iOS Behavior | Notes |
|---|---|---|
| 44.1 kHz | Passthrough | CD quality |
| 48 kHz | Passthrough | Video standard |
| 96 kHz | Passthrough | Hi-res |
| 192 kHz | Passthrough | Hi-res |
| DSD | Not supported | Use DoP or convert |
iOS 26+ Input Selection
AVInputPickerInteraction
Native input device selection with live metering:
import AVKit
class RecordingViewController: UIViewController {
let inputPicker = AVInputPickerInteraction()
override func viewDidLoad() {
super.viewDidLoad()
// Configure audio session first
try? AVAudioSession.sharedInstance().setCategory(.playAndRecord)
try? AVAudioSession.sharedInstance().setActive(true)
// Setup picker
inputPicker.delegate = self
selectMicButton.addInteraction(inputPicker)
}
@IBAction func selectMicTapped(_ sender: UIButton) {
inputPicker.present()
}
}
extension RecordingViewController: AVInputPickerInteractionDelegate {
// Implement delegate methods as needed
}
Features:
- Live sound level metering
- Microphone mode selection
- System remembers selection per app
iOS 26+ AirPods High Quality Recording
LAV-microphone equivalent quality for content creators:
// AVAudioSession approach
try AVAudioSession.sharedInstance().setCategory(
.playAndRecord,
options: [
.bluetoothHighQualityRecording, // New in iOS 26
.allowBluetoothA2DP // Fallback
]
)
// AVCaptureSession approach
let captureSession = AVCaptureSession()
captureSession.configuresApplicationAudioSessionForBluetoothHighQualityRecording = true
Notes:
- Uses dedicated Bluetooth link optimized for AirPods
- Falls back to HFP if device doesn't support HQ mode
- Supports AirPods stem controls for start/stop recording
Spatial Audio Capture (iOS 26+)
First Order Ambisonics (FOA)
Record 3D spatial audio using device microphone array:
// With AVCaptureMovieFileOutput (simple)
let audioInput = AVCaptureDeviceInput(device: audioDevice)
audioInput.multichannelAudioMode = .firstOrderAmbisonics
// With AVAssetWriter (full control)
// Requires two AudioDataOutputs: FOA (4ch) + Stereo (2ch)
AVAssetWriter Spatial Audio Setup
// Configure two AudioDataOutputs
let foaOutput = AVCaptureAudioDataOutput()
foaOutput.spatialAudioChannelLayoutTag = kAudioChannelLayoutTag_HOA_ACN_SN3D // 4 channels
let stereoOutput = AVCaptureAudioDataOutput()
stereoOutput.spatialAudioChannelLayoutTag = kAudioChannelLayoutTag_Stereo // 2 channels
// Create metadata generator
let metadataGenerator = AVCaptureSpatialAudioMetadataSampleGenerator()
// Feed FOA buffers to generator
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {
metadataGenerator.append(sampleBuffer)
// Also write to FOA AssetWriterInput
}
// When recording stops, get metadata sample
let metadataSample = metadataGenerator.createMetadataSample()
// Write to metadata track
Output File Structure
Spatial audio files contain:
- Stereo AAC track — Compatibility fallback
- APAC track — Spatial audio (FOA)
- Metadata track — Audio Mix tuning parameters
File formats: .mov, .mp4, .qta (QuickTime Audio, iOS 26+)
ASAF / APAC (Apple Spatial Audio)
Overview
| Component | Purpose |
|---|---|
| ASAF | Apple Spatial Audio Format — production format |
| APAC | Apple Positional Audio Codec — delivery codec |
APAC Capabilities
- Bitrates: 64 kbps to 768 kbps
- Supports: Channels, Objects, Higher Order Ambisonics, Dialogue, Binaural
- Head-tracked rendering adaptive to listener position/orientation
- Required for Apple Immersive Video
Playback
// Standard AVPlayer handles APAC automatically
let player = AVPlayer(url: spatialAudioURL)
player.play()
// Head tracking enabled automatically on AirPods
Platform Support
All Apple platforms except watchOS support APAC playback.
Audio Mix (Cinematic Framework)
Separate and remix speech vs ambient sounds in spatial recordings:
AVPlayer Integration
import Cinematic
// Load spatial audio asset
let asset = AVURLAsset(url: spatialAudioURL)
let audioInfo = try await CNAssetSpatialAudioInfo(asset: asset)
// Configure mix parameters
let intensity: Float = 0.5 // 0.0 to 1.0
let style = CNSpatialAudioRenderingStyle.cinematic
// Create and apply audio mix
let audioMix = audioInfo.audioMix(
effectIntensity: intensity,
renderingStyle: style
)
playerItem.audioMix = audioMix
Rendering Styles
| Style | Effect |
|---|---|
.cinematic | Balanced speech/ambient |
.studio | Enhanced speech clarity |
.inFrame | Focus on visible speakers |
| + 6 extraction modes | Speech-only, ambient-only stems |
AUAudioMix (Direct AudioUnit)
For apps not using AVPlayer:
// Input: 4 channels FOA
// Output: Separated speech + ambient
// Get tuning metadata from file
let audioInfo = try await CNAssetSpatialAudioInfo(asset: asset)
let remixMetadata = audioInfo.spatialAudioMixMetadata as CFData
// Apply to AudioUnit via AudioUnitSetProperty
Common Patterns
Background Audio Playback
// 1. Set category
try AVAudioSession.sharedInstance().setCategory(.playback)
// 2. Enable background mode in Info.plist
// <key>UIBackgroundModes</key>
// <array><string>audio</string></array>
// 3. Set Now Playing info (recommended)
let nowPlayingInfo: [String: Any] = [
MPMediaItemPropertyTitle: "Song Title",
MPMediaItemPropertyArtist: "Artist",
MPNowPlayingInfoPropertyElapsedPlaybackTime: player.currentTime,
MPMediaItemPropertyPlaybackDuration: duration
]
MPNowPlayingInfoCenter.default().nowPlayingInfo = nowPlayingInfo
Ducking Other Audio
try AVAudioSession.sharedInstance().setCategory(
.playback,
options: .duckOthers
)
// When done, restore others
try AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)
Bluetooth Device Handling
// Allow all Bluetooth
try AVAudioSession.sharedInstance().setCategory(
.playAndRecord,
options: [.allowBluetooth, .allowBluetoothA2DP]
)
// Check current Bluetooth route
let route = AVAudioSession.sharedInstance().currentRoute
let hasBluetoothOutput = route.outputs.contains {
$0.portType == .bluetoothA2DP || $0.portType == .bluetoothHFP
}
Anti-Patterns
Wrong Category
// WRONG — music player using ambient (silenced by switch)
try AVAudioSession.sharedInstance().setCategory(.ambient)
// CORRECT — music needs .playback
try AVAudioSession.sharedInstance().setCategory(.playback)
Missing Interruption Handling
// WRONG — no interruption observer
// Audio stops on phone call and never resumes
// CORRECT — always handle interruptions
NotificationCenter.default.addObserver(
forName: AVAudioSession.interruptionNotification,
// ... handle began/ended
)
Tap Memory Leaks
// WRONG — tap installed, never removed
engine.inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { ... }
// CORRECT — remove tap when done
deinit {
engine.inputNode.removeTap(onBus: 0)
}
Format Mismatch Crashes
// WRONG — connecting nodes with incompatible formats
engine.connect(playerNode, to: mixerNode, format: wrongFormat) // Crash!
// CORRECT — use nil for automatic format negotiation, or match exactly
engine.connect(playerNode, to: mixerNode, format: nil)
Forgetting to Activate Session
// WRONG — configure but don't activate
try AVAudioSession.sharedInstance().setCategory(.playback)
// Audio doesn't work!
// CORRECT — always activate
try AVAudioSession.sharedInstance().setCategory(.playback)
try AVAudioSession.sharedInstance().setActive(true)
Resources
WWDC: 2025-251, 2025-403, 2019-510
Docs: /avfoundation, /avkit, /cinematic
Targets: iOS 12+ (core), iOS 26+ (spatial features) Frameworks: AVFoundation, AVKit, Cinematic (iOS 26+) History: See git log for changes
FAQ
What does the axiom-avfoundation-ref skill do?
Reference — AVFoundation audio APIs, AVAudioSession categories/modes, AVAudioEngine pipelines, bit-perfect DAC output, iOS 26+ spatial audio capture, ASAF/APAC, Audio Mix with Cinematic framework
How do I install axiom-avfoundation-ref?
Run npx skills add https://github.com/charleswiltgen/axiom --skill axiom-avfoundation-ref in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/axiom-avfoundation-ref/ (personal) or .claude/skills/axiom-avfoundation-ref/ (project).
Which AI agents can use axiom-avfoundation-ref?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is axiom-avfoundation-ref safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains axiom-avfoundation-ref?
It is published by charleswiltgen in charleswiltgen/axiom. It has 221 recorded installs.
More from charleswiltgen/axiom
axiom-vision
charleswiltgen/axiom
Use when implementing ANY computer vision feature — image analysis, pose detection, person segmentation, subject lifting, text recognition, barcode scanning.
axiom-swiftui
charleswiltgen/axiom
Use when building, fixing, or improving ANY SwiftUI UI — views, navigation, layout, animations, gestures, debugging, iOS 26 features, iPhone Duo, view-level performance, feature architecture.
axiom-networking
charleswiltgen/axiom
Use when implementing or debugging ANY network connection, API call, or socket in an Apple app. Covers URLSession, Network.framework, NetworkConnection, connection diagnostics.
axiom-apple-docs
charleswiltgen/axiom
Use when you need Apple's own documentation or a Swift compiler diagnostic explained rather than recalled.
axiom-shipping
charleswiltgen/axiom
Use when preparing an app for App Store submission, handling rejections or appeals, managing App Store Connect and Xcode Cloud, or triaging TestFlight/Sentry crash corpora. Covers checklists, metadata, privacy manifests, age ratings, export compliance.
axiom-xcode-mcp
charleswiltgen/axiom
Use when connecting to Xcode via MCP, using xcrun mcpbridge or the headless mcp-server, or working with ANY Xcode MCP tool (XcodeRead, BuildProject, RunSomeTests, RenderPreview). Covers setup, tool reference, workflows, troubleshooting.
Popular media & creative skills
hyperframes
heygen-com/hyperframes
Read first to make, edit, animate or render any video with HyperFrames (HTML rendered to video), including launch films, explainers, music videos, charts, maps, vertical clips, title cards, captions and overlays. Holds the render contract, the work loop and what each kind of video needs.
media-use
heygen-com/hyperframes
Get or make any media for a HyperFrames video - music, sound effects, images, icons, logos, voiceover, avatar and generated video, transcripts, cutouts, colour grades and looks - with HeyGen or free local tools, and mix placed audio. Also for footage that looks dark, flat or dull, or should be stylized or hidden.
hyperframes-registry
heygen-com/hyperframes
About 450 ready-made HyperFrames blocks and components installed as editable source - transitions, overlays, titles, kinetic type, captions, charts, map overlays, 3D, camera moves. Use it before hand-building any named visual (CRT scanlines, glitch, film grain, a chart); also for hyperframes add and catalog.
hyperframes-creative
heygen-com/hyperframes
Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use hyperframes-animation.
ai-video-generation
101-skills/superpowers
Generate AI videos with Google Veo, Seedance 2.0, HappyHorse, Wan, Grok and 40+ models via inference.sh CLI. Models: Veo 3.1, Veo 3, Seedance 2.0, HappyHorse 1.0, Wan 2.5, Grok Imagine Video, OmniHuman, Fabric, HunyuanVideo. Capabilities: text-to-video, image-to-video, reference-to-video, video editing, lipsync, avatar animation, video upscaling, foley sound. Use for: social media videos, marketing content, explainer videos, product demos, AI avatars. Triggers: video generation, ai video, text to video, image to video, veo, animate image, video from image, ai animation, video generator, generate video, t2v, i2v, ai video maker, create video with ai, runway alternative, pika alternative, sora alternative, kling alternative, seedance, happyhorse
ai-image-generation
101-skills/superpowers
Generate AI images with GPT-Image-2, FLUX, Gemini, Grok, Seedream, Reve and 50+ models via inference.sh CLI. Models: GPT-Image-2, FLUX Dev LoRA, FLUX.2 Klein LoRA, Gemini 3 Pro Image, Grok Imagine, Seedream 4.5, Reve, ImagineArt. Capabilities: text-to-image, image-to-image, inpainting, LoRA, image editing, upscaling, text rendering. Use for: AI art, product mockups, concept art, social media graphics, marketing visuals, illustrations. Triggers: flux, image generation, ai image, text to image, stable diffusion, generate image, ai art, midjourney alternative, dall-e alternative, text2img, t2i, image generator, ai picture, create image with ai, generative ai, ai illustration, grok image, gemini image, gpt image, openai image, chatgpt image