iOS Keyboard Extension Limitations

When building iOS custom keyboards with voice/audio features, these are the hard limitations discovered through the PolyVoice project.

🔴 Hard Limitations (Cannot be worked around)

1. Microphone Access — DISALLOWED

Keyboard extensions cannot access the microphone.

• AVAudioRecorder will fail with permission error
• SFSpeechRecognizer is unavailable
• No Siri integration from keyboard context

Why: Apple security model — keyboards run in sandbox and could keylog audio.

2. Open Other Apps — BLOCKED

Keyboards cannot programmatically open the main app or any other app.

• UIApplication.shared.open() returns false
• URL schemes don't work (myapp://)
• ExtensionContext.open() not available

Why: Prevents malicious keyboards from launching apps without user consent.

3. Memory Limit — ~50MB

Keyboard extensions have strict memory limits (~30-60MB).

• App terminated silently if exceeded
• No crash log, just disappears
• Heavy audio processing = instant death

Mitigation: - Record at 16kHz mono (not 44.1kHz) - Use 32kbps bitrate max - Immediate file cleanup after processing - 60-second max recording hard limit

4. No Persistent Storage

UserDefaults unavailable, only App Groups.

• Standard UserDefaults doesn't persist
• Must use UserDefaults(suiteName: "group.com.company.app")
• Requires App Group capability in both targets

5. Network Requires "Full Access"

API calls fail without user enabling "Allow Full Access" in Settings.

• User must explicitly enable: Settings → General → Keyboard → [Keyboard Name] → Allow Full Access
• Most users won't do this
• Cannot prompt or explain from keyboard UI effectively

🟡 Partial Workarounds (User friction)

The "Open App" Workaround

Goal: Let user tap a button to open main app for recording.

Attempt:

// This does NOT work
extensionContext?.open(URL(string: "myapp://record")!)

Reality: Must use UIApplication.shared.open() outside extension context, but keyboards can't call this.

The Manual Switch Pattern

What actually works (with friction):

1. User taps button in keyboard → Shows alert: "Open PolyVoice to record?"
2. User manually switches to main app (Home button, swipe, etc.)
3. Main app detects active session (via App Groups / shared state)
4. Main app auto-records on appear
5. Auto-stops on silence (2 seconds)
6. Auto-copies to clipboard
7. User manually switches back to target app
8. Keyboard auto-pastes on reappear

User flow:

Keyboard → Tap mic → [Manual: Switch to app] → App auto-records → 
[Manual: Switch back] → Keyboard auto-pastes

Friction points: - Two manual app switches - Context switching breaks flow - Users forget to return - Clipboard may be overwritten

🟢 Alternative Architectures

Option 1: Share Extension (Better for Audio)

Use Share Sheet instead of keyboard.

• Full app capabilities
• Can record audio
• Can process and return text

Limitation: Not a keyboard — user must open share sheet per text field.

Option 2: Full App Mode

Don't use keyboard extension — use main app only.

• User opens app
• Records dictation
• Copies result
• Switches to target app
• Pastes manually

Benefit: No memory limits, full mic access, reliable.
Cost: More friction than keyboard.

Option 3: Siri Shortcuts Integration

Provide Siri Shortcuts for voice-to-text.

• "Hey Siri, dictate with PolyVoice"
• Returns text to current app
• Fully supported by Apple

Limitation: Not instant, requires Siri setup.

📊 Decision Matrix

Approach	Mic Access	Memory	User Friction	Apple Approved
Keyboard extension	❌ No	⚠️ 50MB	Low (if no audio)	✅ Yes
Keyboard + audio workaround	❌ No	⚠️ 50MB	🔴 High	✅ Yes
Share extension	✅ Yes	✅ Full	🟡 Medium	✅ Yes
Full app only	✅ Yes	✅ Full	🟡 Medium	✅ Yes
Siri Shortcuts	✅ Yes	✅ Full	🟡 Medium	✅ Yes

🎯 Recommendation

For voice dictation/AI transcription:

1. Don't build a keyboard extension — the limitations make it frustrating
2. Use Share Extension — Apple-supported, full capabilities
3. Or full app — simplest to build, most reliable
4. Add Shortcuts — for power users who want speed

For non-audio keyboards (emoji, translation, etc.):

Keyboard extension works great. Just avoid audio features.

📚 References

• Apple's official docs: https://developer.apple.com/documentation/uikit/keyboards_and_input/creating_a_custom_keyboard
• Custom Keyboard Programming Guide (WWDC sessions)
• PolyVoice project learnings (~/Projects/polyvoice-keyboard/)