Overview
- 1. About us
- 2. Sonoport Studio
- 3. Dynamic Sound Models
- 4. Porting to Web Audio
- 5. Problems & Solutions
- 6. Audio Worker
- 7. Summary
About Me
Chinmay Pendharkar
@ntt / notthetup
- we work with Interactive Dynamic Audio for Multimedia
- we're building a Smart Sound Exchange Platform
- we're located in Singapore & UK
2. Sonoport Studio
- First application for the Smart Sound Exchange
- Interactive Audio Authoring Platform
- Core : Dynamic Sound Engine
-
- Collection of Sonoport Dynamic Sound Models
-
- Runtime + Utilities
2. Sonoport Studio
Demo Booth
- Check out Sonoport Studio at Demo/Poster Session #2
- Tuesday afternoon at MOZILLA
- 15.00 - 17.00
3. Sonoport Dynamic Sound Models
- Algorithms that make sound
- Method to start/stop sounds
- Parameters to change the sounds in real-time
- Texture based or purely Synthesized
3. Sonoport Dynamic Sound Models
- Based on Lonce Wyse's work in Java
- Moved to Flash (starting with Flash 10)
- January 2014 - Starting migration to Web Audio
4. Porting to Web Audio - Aims
- Port all existing functionality of Flash version
- Keep API Consistency with Flash version
- Leverage on Web Audio for 'plumbing'
- Leverage the composibilty of Web Audio
3 Problems : 3 Solutions Hacks!
Problem #1 : Can't create custom Parameters in Web Audio
Problem #1 - Higher Level Nodes
- All Dynamic Sound Models are Higher Level Nodes
- Higher Level Nodes to leverage composibility of Web Audio
- Internally composed of multiple AudioNodes
Problem #1 - Parameters
- Higher Level Nodes need Parameters
- Custom params should have similar interface as AudioParam
- Custom params may be mapped to internal AudioNodes
- Bonus : Custom Params should support Automation
Solution #1: SPAudioParam
Wrap around AudioParams, expose similar interface to AudioParams
SPAudioParam - Two Approaches
Wrapped SPAudioParam
- Map SPAudioParam to underlying AudioParam
- Use getters/setters to pass along values
- Mapping/normalization on the fly
- Pass along Automation method calls
Mocked SPAudioParam
- Plain old JS Object
- Fakes Automation using setInterval (URGH!)
- For edge case without underlying AudioParam
Solution #1: SPAudioParam
- Mostly used Wrapped SPAudioParam approach
- Slight change in API for Parameters but free automation!!
- Mocked SPAudioParam approach handy in edge case scenarios
SPAudioParam
/*Create Model*/
var looper = new Looper (context, "/audio.wav");
/*Play*/
looper.play();
/*Set SPAudioParam*/
looper.easeIn.value = 0.3;
looper.easeOut.value = 0.7;
/* Automate SPAudioParam*/
looper.playSpeed.exponentialRampToValueAtTime(2,10);
Problem #2 : Can't schedule some operations in Web Audio
Problem #2 - Queues
- Scheduling is key, but can't un-schedule in Web Audio
- Keep a Queue, schedule/unschedule on the Queue
- Queue executes 'Events' close to the scheduled time
- Common Queue Events : Start, Stop, Change Param
Problem #2 - Polyphony
- Polyphonic Models with multiple voices
- Typical archtecture : fixed number / pool of voices
- Queue assigns buffers to a voice at the scheduled time
Problem #2 - Polyphony
- New Queue Event : SetSource <- not schedulable!
- Can't use setInterval for scheduling SetSource either
- Executing SetSource at earlier could override another voice
Solution #2: Fire and Forget Voices
Create new voices as and when needed
Solution #2 - Fire and Forget Voices
- Architecture : new BufferNode everytime a voice is needed
- Ensure GC captures all the used up BufferNodes
- Test no leaks using memory profiling and heap dumps
- Tested to upto ~60x3 Nodes created per second
Problem #3 : Can't pause an AudioBufferSourceNode
Problem #3 - Buffers
- AudioBufferSourceNode can only be played once
- Pause/Play needs the position at which it was paused
- No way to know current playbackPosition
- Time based tracking only works for static playbackRate
- Tracking through automated playbackRate is tedious and inaccurate
Solution #3: SPAudioBufferSourceNode
Wrap around AudioBufferSourceNode, and use a 'Scope' track playback
Solution #3 - SPAudioBufferSourceNode
- Create a mirror AudioBuffer with index as value
- Create a CounterBufferNode from this AudioBuffer
- Connect this CounterBufferNode to a ScriptProcessorNode
- onprocessaudio stores the last value of the inputBuffer
- Perform every action (start, stop, automation..) on both AudioBufferSourceNodes
SPAudioBufferSourceNode Implementation
/*Create Nodes*/
var counterNode_ = audioContext.createBufferSource();
var scopeNode_ = audioContext.createScriptProcessor( 256, 1, 1 );
/*Connect Nodes*/
counterNode_.connect(scopeNode_);
/*Create Counter Buffer*/
counterNode_.buffer = createCounterBuffer( buffer.buffer );
function createCounterBuffer( audioBuf ) {
var array = new Float32Array( buffer.length );
var audioBuf = audioContext.createBuffer( 1, buffer.length, 44100 );
for ( var index = 0; index < buffer.length; index++ ) {
array[ index ] = index;
}
audioBuf.getChannelData( 0 ).set( array );
}
/*Remember last element in inputBuffer*/
scopeNode_.onaudioprocess = function savePosition( processEvent ) {
var inputBuffer = processEvent.inputBuffer.getChannelData( 0 );
lastPos = inputBuffer[ inputBuffer.length - 1 ] || 0;
}
Solution #3 - SPAudioBufferSourceNode
- Stored value exposed as playbackPosition property
- playbackPosition is read only
- Accurate to nearest ScriptNode buffer size
- Not perfect buy good enogh for pausing/playing
Porting to WebAudio - Summary
- Overcame most issues we faced while porting
- Most solutions worked for our specific use case
- Your mileage may vary based on what you're doing
- There are creative ways to implement most use cases
Problem #1 - Parameters
- AudioWorkers have an addParameter method!
- Create a-rate parameters accessible inside AudioWorker!
- Should solve all issues with creating custom Parameters
Problem #2 - Scheduling
- AudioWorkers enables sample accurate scheduling
- Queues can runs inside an AudioWorker
- Load operations can be scheduled accurately
Problem #3 - Buffers
- No playbackPosition property on AudioBufferSourceNode
- .. but AudioBufferSourceNode could be reimplmented inside AudioWorker!
- No idea of performance untill we get AudioWorker
7. Summary
- Hacking around Web Audio was fun
- Performance vs Functionality tradeoff
- Might not work perfectly in all scenarios
- AudioWorker should make life easier
- Would love to know other ways of solving such problems!