I would like to thank all my supervisors at The University of Edinburgh who encouraged me and continually challenged me. When I would show some completed piece of this project to Dr. Martin Parker, Dr. Julian Rawlinson, or Dr. Tom Mudd, they would always have some ‘little idea’ to add on. I tried them all, spent sleepless nights on some, and was always better for it.
A huge thank you to Carla Sayer(poetry) and Martina Maclachlan(sung vocals) for providing amazing vocal samples to store in various parts of Playing In The Waves.
Thank you to every fellow Sound Design MSc student to tried out early builds of the project and gave me feedback.
CSS by Markdown CSS, ‘Splendor’ by John Otander
This dissertation is primarily a practical work. The main focus was building an audio application that would run in a computer’s internet browser. Please experience the final product before reading this paper at https://cpmdude89.github.io/playingInTheWaves/playingInTheWaves.html. By clicking on the title of the page upon arrival, a popup window will display further information summarizing this work.
This document explains why I built what I built, and a technical look into some of the interesting aspects of the finished product. Because web audio is a relatively new field and the technological limitations I encountered directly stem from its history, I document how browser audio came to be what it is today.
Playing In The Waves is a fun, interactive sound experience intended to educate the user on certain fundamental aspects of digital audio and provide an accessible platform for experimenting with the user's immediate sonic surroundings. It is a website and an audio application focused on on leveraging the relatively new tools that makes complex digital audio processing possible in the browser.
For my final project, I set out to learn the fundamentals of JavaScript and web audio through creating a product that could be immediately usable and reached on any device by a URL. What does web audio mean? What can a web audio application do that no other kind of audio software can do? What are the limits of web audio technology and what unique technical considerations to keep in mind when making a web audio program? These are the questions I set out to answer by diving into the web audio community online, learning the basics, and creating an actual product.
The first chapter of paper will cover my philosophical motivations behind the design choices in Playing In The Waves, and what influences led me to those choices. The second chapter will explain some of the reasons how web audio is unique and why it suits my project goals. The third chapter will cover the specifics of those design choices, and how they were implemented with the use of JavaScript libraries.
This dissertation assumes the reader understands the basics of DSP to save room for more discussion on the basics of web audio. For an overview on the libraries I used, refer to the Appendix A. Appendix B has a few inspiring examples of current web audio and how they have influenced this project. Appendix C contains tutorials I made to get past some tricky quirks of these libraries as well.
As a child growing up in San Diego, California, my parents would often take me to the Fleet Science Center in Balboa Park. All the exhibits in the Fleet are interactive and encourage visitors to experience for themselves a sense of wonder while playing with science. I spent many carefree afternoons running around with all the other kids, being chased by our parents as we ran under and climbed over science exhibits designed just for that purpose.
My father fostered in me a passion for science and science fiction. When I was a teenager, he gifted me a copy of Issac Asimov's Foundation. It is a story of a mathematical genius trying to save a future galactic human empire from succumbing to an impending Dark Age. Early in the book, the hero is asked how he will accomplish this task, and he responds:
By saving the knowledge of the race. The sum of human knowing is beyond any one man... But, if we now prepare a giant summary of all knowledge, it will never be lost"
(Asimov, 28)
The concept of saving the human race from its worst impulses through promotion of science alone is a concept that I still naively cling to. I believe that science and more importantly, the critical thinking that comes with the scientific method, is a force for good in the world.
I also believe that science is an extremely difficult thing to communicate to many people. Sometimes, people don't know they are interested in a scientific topic until it is presented to them in a way that is inspiring or exciting. I don't think the science I was exposed to at the Fleet Science Center was cheapened in any way by encouraging the visitors to 'play' with it. My life was enriched by being exposed to new ways of thinking, and I was more open to it because I was having fun at the same time.
Eloquent science communication unites people by revealing a glimpse of a cosmos that is bigger than all of us. In the introduction to his famous book/television series, Cosmos, Carl Sagan writes:
It is dedicated to the proposition that the public is far more intelligent than it has generally been given credit for; that the deepest scientific questions on the nature and origin of the world excite the interests and passions of enormous numbers of people.
(Sagan, 14)
I grew up watching the Cosmos program and anything else Carl Sagan would present on TV, and am deeply inspired by him in my creative practice. I share the belief that the mysteries of the universe are endlessly fascinating and should be shared.
... it turns out we are connected - not in the personal, small-scale unimaginative fashion that the astrologers pretend, but in the deepest ways, involving the origin of matter, the habitability of the Earth, the evolution and destiny of the human species...
(Sagan, 65)
Carl Sagan's beautiful prose humanizes science and imparts a sense of profundity that inspires. The ability to communicate the reasons why science is so interesting while at the same time educating an audience on the science itself is an important balance to strike. In this way I see The Fleet Science Center's focus on fun and Carl Sagan's inspirational expression as achieving the same goal.
Early in semester 1 during Tom Mudd's Creative Coding for Sound class, he demonstrated what amplitude modulation on a sine tone looks like on a Max MSP Spectrogram. The modulation frequency started out low, around 1 Hz, to just hear the overall volume fluctuate and see the lone bump on the visualizer move up and down. When the modulating LFO's frequency increased, I saw and heard another tone emerge. A second bump emerge from the fundamental and another pitch slowly grew into existence.
Maybe it is the sound design equivalent of a party trick, but it seemed like pure magic to me. During this demonstration, I could not hear when the tone started or stopped. It's as if it was always there, and turning the amplitude modulation frequency up from 1Hz to 50Hz just increases the volume up on a sound that was there the whole time. It is like creating something from nothing.
I know there is a more technical explanation for amplitude modulation, but I am more interested the sensation of discovering something new from something familiar. Playing In The Waves is not a project exploring all the different kinds of amplitude modulation, but a project exploring the magic feeling of experiencing amplitude modulation for the first time.
Last semester, my final project submission for Interactive Sound Design played with the idea of hearing stories told by people in their own voice. I was able to interview friends, family, and classmates and record their stories for me to inset into my virtual world. Once the project was completed I showed three of my fellow Sound Design MSc students the project and played each person's recorded story back to them. Although it is a small sample size, I was struck that everyone's reaction was a surprised sense of joy. Even if a little embarrassed at hearing their own voice, there were lots of laughs had and what I observed to be moments of pure joy.
Every page in Playing In The Waves makes the user's own voice available to experiment on and play with. It makes the science behind basic audio processes more real and hopefully more fun.
Accessibility is a word with lots of different meanings. I made a web audio application that is 'accessible' by requiring the lowest amount of effort to 'reach' or interface directly with it. Not everyone has a DAW or even a computer to run it on, but almost every device has a browser. Internet browsers are everywhere, come pre-installed on almost all computers and mobile devices, and most importantly: are free.
The primary design principle in Playing In The Waves is maintaining simplicity. On every page of my web application, the user is always no more than two clicks of the mouse away from generating an interesting sound. Even if the user doesn't understand some component, randomly interacting with the page will result in sounds and effects that are designed to pull the user in and learn by experimenting. There just aren't enough buttons to get lost in or break the program.
Screen-readers help unlock internet browsing for visually-impaired individuals. These are programs that scan the computer screen and read aloud a description of what is on the page. Making websites compatible with screen reader technology can be an easy win for web developers to reach a wider audience. Considering I am creating a primarily audio focused web application, it is the logical next step to make a web audio program easily accessible to this demographic.
p5.js is a creative coding library that creates a 'canvas' on the webpage, which does not store the kind of data that a screen reader can understand. Adding the p5.js object textOutput() to the top of the draw() loop, will help screen readers understand what is going on on the page.
I hope that many different kinds of people will find different things about Playing In The Waves to enjoy. Each 'stop' on the Tour is designed to be entertaining and simple enough to capture the attention of someone totally new to audio software and might otherwise be intimidated by lots of buttons and sliders onscreen. The Playground is a much deeper program than any one part of the Tour and is there to provide more control and stimulation to someone already comfortable with a DAW. Anyone who is interested in sound generally should find something to enjoy in this project.
Web audio technology today looks nothing like it did 10 years ago.
It is no secret that computer technology, especially software, moves fast. It takes a lot of effort to keep up with new trends as companies compete with each other to constantly output new products that iterate and change the way things were done previously. When I started working on this project, I assumed that web audio would be just more audio programming like I had done with Max MSP or C++ and JUCE. I was wrong. Web audio technology is extremely new, and after more than a decade of stagnation, has recently exploded in both power and usability. I expect the field of web audio to expand in credibility and demand in the coming years as the uptake of the technology by audio programmers increases.
In 1995, NetScape released its next generation browser: Navigator 2.0. With this release came the <embed> tag, that would allow the user to 'embed' a piece of media into the webpage. This tag supported many different file types, including image and video, as well as audio.
Later in 1997, Internet Explorer 4.0 was released, and supported a new Microsoft-proprietary HTML tag called <bgsound>, standing for 'background sound'. It was designed to just play or loop an audio file in the background of the webpage, without any control from the user or being able to turn it off. It was designed to only ever work on Internet Explorer, which is obsolete now, but if a developer even thinks about using <bgsound> they need only look at the W3C doc page to see what the rest of the internet thinks about it. In the 'examples' section of the page, there is only a single sentence saying:
"No, really. Don't use it."
(W3C Wiki, 2010)
Despite this, <bgsound> gets credit for being the first audio-specific HTML element created.
The most popular browser plug-in was Future Splash Animator, which would later become Adobe Flash Player. The memory of being constantly bothered to update Flash Player on your computer is likely a painful one. From the mid 1990's to the late 2000's plug-ins like Flash, ActiveX, Quicktime, and Shockwave were the primary way to watch videos, listen to music, or play games on the browser. In 2007, Apple released the first version of the iPhone, but it didn't support Flash. This was the beginning of the end for browser plug-ins. In 2010, Steve Jobs wrote 'Thoughts on Flash', an open letter explaining why Apple was moving all its products away from supporting Flash, siting lax security, no touch support, and a desire to focus the most recent version of HTML.
Though the operating system for the iPhone, iPod and iPad is proprietary, we strongly believe that all standards pertaining to the web should be open. Rather than use Flash, Apple has adopted HTML5, CSS and JavaScript--all open standards. Apple's mobile devices all ship with high performance, low power implementations of these open standards. HTML5, the new web standard that has been adopted by Apple, Google and many others, lets web developers create advanced graphics, typography, animations and transitions without relying on third party browser plug-ins (like Flash). HTML5 is completely open and controlled by a standards committee, of which Apple is a member.
(Jobs, 2010)
HTML5, which would become the standard version of HTML later in 2014, provided browsers with newer and better tools for handling audio and video with the <audio> and <video> tags.
With Apple signaling a new era, new technologies began appearing in the 2010's that would become the foundation for modern web audio. In 2011, the first version of the Web Audio API was released by Mozilla (makers of Firefox), and is the bedrock technology that Playing In The Waves is based on. In 2015, ES6, the next version big update to JavaScript was released, and allowed for the creation of 'classes'. This enabled real object-oriented programming for web developers and is how most other kinds of audio software is structured. In 2014, p5.sound and Tone.js, the two web audio libraries I studied for this project were first published on GitHub. These libraries provide a friendly wrapper around The Web Audio API and lowers the minimum amount of web programming experience it requires to make a web audio application.
All the above technologies have been iterated and improved significantly since their initial release, along with newer and more powerful browsers to develop for. Now in 2021, web audio tools have a small but dedicated online community to learn from and interact with. There are interesting examples of audio programming that are idiomatic to the web space, like the autosampler and the Chrome Music Lab. The web audio field is beginning to compare in power and capability to other forms of audio programming, but still has a few unique technical idiosyncrasies that separate it from more traditional audio software.
All web audio starts with the browser. Sending a request to the server, which could be anywhere in the world, and getting a data packet back is too long of a process to happen mid-audio stream. So we are stuck doing all our digital processes in the browser, which is not specifically designed for the purpose. Web browsers present a wide range of inconsistencies across the various brands, devices, and versions. Please refer to my second post in my short-lived blog, where I first ran into the problem of different browsers interpreting the same JavaScript code in very different ways.
The Web Audio API is an impressive piece of technology, and lists a wide variety of use cases. However, in the official recommendation by the W3C, they admit:
...modern desktop audio software can have very advanced capabilities, some of which would be difficult or impossible to build with this system. Apple’s Logic Audio is one such application which has support for external MIDI controllers, arbitrary plug-in audio effects and synthesizers, highly optimized direct-to-disk audio file reading/writing, tightly integrated time-stretching, and so on.
(W3C, 2021)
The fact that they list 'highly optimized direct-to-disk' audio processes as one of the features that the Web Audio API can't replicate is a clue that running in the browser creates a processing bottleneck.
Another, less obvious obstacle that all web audio programmers must overcome is lack of access to a sample-accurate clock.
Desktop audio applications like Apple's Logic are installed directly to the computer's hard drive, which gives them direct access to the computer's extremely accurate, internal clock. Considering the high sample rates audio software must keep up with, and a single sample out of place may result in an audible glitch, having a precise clock to rely on is critical. Programs on the internet do not have direct access to a computer's internal clock, as any internet program exists between the push and pull of the browser and the server it's communicating with.
All programs on the browser run on the JavaScript clock, which has an accuracy only to the nearest millisecond. This may seem accurate to us humans, but audio processing must run at sample rate which is usually no slower than 44100 Hz. So by the time a single millisecond has gone by, an audio program must run its processes 44 (.1) times. This kind of latency doesn't matter when simply triggering audio on/off or changing the volume, but even basic audio programming processes like shaping an amplitude envelope need to run at audio sample rate.
This problem was circumvented first by Adobe because Flash was a plug-in, downloaded to the computer hard drive and therefore directly accessible to the device's internal clock. The Web Audio API overcomes the inaccuracy of the JavaScript clock by digging down past the browser and accessing the system internal clock and putting a JavaScript wrapper around it. The Web Audio Clock is returned to the browser as a 32-bit float, or a 'double', which is accurate to 15 decimal points. Much of The Web Audio API's complex audio processing would not be possible without this critical component.
As this whole chapter shows, web audio technology is an extremely new field. I had no idea that web audio was such a young field of work when I began, but now the challenge of working in a field that is changing so fast is very exciting to me. While I was working on this project, on 17 June 2021, the W3C confirmed the Web Audio API, the foundational technology my project is based on, as an 'official standard'. While this is mostly a formality, it is a sign that web audio technology is both new and growing.
Being a self-taught learner for years before attending University of Edinburgh, I have benefitted from a world-wide community of programmers and educators that were kind enough to share their knowledge with the world for free. In its current form, web audio only has a few good, free online resources to rely on. I have the opportunity to share what I have learned and contribute to a growing body of knowledge. Please refer to the Problems Overcome for specific problems I encountered with Tone.js and p5.sound that I did not find easy solutions in the documentation for and had to solve on my own.
As the autosampler demonstrates, there is so much sonic potential in our day-to-day environment. By having quick and easy access to manipulate our immediate sonic surroundings, new depths can be found in sounds that are taken for granted. A clap, a dog barking across the street, our own breath, can be the foundation of interesting new sounds. Knowing what a clap or your own voice sounds modulated by some process can also bring a new appreciation for the acoustic space we take for granted. I want to give a set of quick-access tools for anyone to record whats going on around them and experiment with them.
As with everything, there is a trade off. I have spent large amounts of this chapter discussing the drawbacks of web audio, but there is one major and undeniable benefit to making a web audio application: reach. It is much easier to look up a URL than download a plug-in or an indie game. Playing In The Waves will remain free to access and can be reached on any device with an internet browser. It is accessible on a phone without having to download yet another app. I wanted to make a project that would appeal to many different kinds of people, and spread knowledge and joy.
In short, my design goal for this project was to make something approachable to a wide range of people, but also be capable of generating complex and interesting sounds. To this end, Playing In The Waves is split into two parts: the Tour and the Playground. The Tour is a friendly, visually stimulating experience aimed at people who are less familiar with DSP concepts. The Tour then culminates in the Playground, which is aimed at more experienced audio software users. The Playground is also available from the Playing In The Waves homepage.
Each page, except for the phone version, in Playing In The Waves has the ability to hit a ‘record’ button and record the audio output from the page, then download it as a .webm file. This provides a very basic was to ‘bounce’ out new sounds for the user to keep or use in other projects. Saving the audio output is a standard function all audio software has, and web audio should be no different.
I hope the visual design of each page encourages the user to try things to see what happens and learn by doing. However I understand not everybody learns in a trial-and-error way, so each page will have a popup link to access a more detailed explainer on how the page itself works and the basics of the underlying audio processes at work.
The Tour section is a 4-part walkthrough of 4 aspects of digital audio that I find particularly interesting. Each 'stop' on the Tour is designed to encourage playful experimentation with large interactive objects featured prominently, and rewards the user with interesting sounds within one or two clicks. Explanatory information is initially withheld from the user to set up exciting surprises as they interact with the program, but can be reached by clicking on the title of the page.
The first stop on the Tour focuses on one of the oldest and most famous ways of manipulating audio: changing the rate of play. The primary feature on this page is a large 2-dimensional graph in the middle of the page. The x-axis controls the audio rate, with negative values reversing the audio. The y-axis controls the pitch shift of the audio, giving the user easy access to experiment with the relationship between time shifting and pitch shifting.
There is a smaller red box that represents the value '1.0' in all directions, which results in no time stretching on the x-axis, or one octave up or down on the y-axis. There are deliberately no numbers explaining this fact on the graph itself, instead the numbers are continuously updated on the side for the user to keep track of.
The audio manipulation in this sketch is made possible by using the p5.js function map() to scale the dimensions of one half of the x-axis of the graph to between 0.0 and 1.51, which is then used to continuously update the Tone.Player and the Tone.PitchShifter. Surprisingly, the values are being updated out of the draw()loop, so are not being modulated at audio rate, but do not cause audible pops or glitches. Because Tone.Player will not accept a negative play rate value, there are actually two Player() objects playing at the same time, with one's buffer reversed. Which side of the '0' the user's mouse is on the graph's x-axis determines if the forward or backward Tone.Player's is audible. To facilitate this, I extended my custom class SamplerButton to create the ForwardsAndBackwardsSamplerButton. This allows the user to record into a single buffer, which is then copied over to two Tone.Player()s, one of which will be reversed.
I considered placing the amplitude modulation page first in the Tour because of the story in Chapter 2 and my personal fascination with the effect. Maybe that's why this is the most complex 'stop' on the Tour. In the end I decided it's better to start the Tour with such a classic audio effect like time shifting, but this is still my favorite page.
Just like the other 'stops' in the Tour, the user is able to record directly to the page via a SamplerButton, or trigger preloaded samples. However, Tour: Amplitude Modulation is the only 'stop' that has the option to trigger a 'TEST TONE'. Since my interest was initially sparked by watching amplitude modulation via a single oscillator, I thought it appropriate to include one here.
Activating the 'TEST TONE' button will bring up a warning about volume while changing wave types. The sawtooth and square waves produce extremely interesting visuals but are much louder and abrasive than the sine or triangle waves. The OscScope window is based on amplitude, so automatically scaling down the volume would diminish the depiction of the signal in the visualizer. I tried to find a compromise by scaling down the sawtooth and square waves a little bit and adding a bold volume warning. That said, experimenting with higher modulation frequencies on various wave types results in the most interesting visuals in all of Playing In The Waves.
Clicking on the 'ACTIVATE AMP MOD' button will engage the modulating LFO, which will trigger the first appearance of a LFOVisualizer. Seeing the LFO ball move up and down in sync with the growing and diminishing of the audio signal in the OscScope provides a very clear visual of the amplitude modulation process at low frequencies.
Activating the amplitude modulation signal will also bring up a horizontal slider to change the rate of the LFO. The amplitude modulation has two ranges, a high range and a low range. Different modulation frequency ranges have different flavors, and being able to zoom in on the 0-20Hz range emphasizes this point. In 'HIGH FREQ' mode, the user is then able to sweep through multiple frequency ranges and try to find the sweet spot where an oscillating volume magically transforms into a new, independent sound.
Even though this is the most cluttered user interface, it is balanced by being able to toggle chunks on and off. This implicitly tells the user what certain buttons like are connected to. This subconscious conversation with the user communicates the systems at work in the program without having to explicitly spell it out, but of course, the full explanation is always a click away.
The delay stop on the Tour functions in a very similar way to the previous stop. There is the SamplerButton and the preloaded samples to generate sound, but no test tone. Amplitude modulation is an audio process that clearest to hear in long, sustained sounds, but delay is most obvious with short, impactful sounds.
At first the page only shows buttons to interact with, sound activating buttons on the left and delay controls on the right. Two OscScopes are positioned under their sound sources, and appear as their corresponding sound is activated. By connecting the appearance and positioning of each sound visualizer with each sound source, the relationship between the fundamental and delay line are made clear implicitly.
One of the most impactful experiences sounds in the entire project is when the delay time LFO approaches 0.0 and the LFOVisualizer is about to reach the bottom of the track. The delay suddenly sounds metallic and the two OscScopes temporarily sync. Sometimes it looks like the two independent OscScopes are connect into one single window as the sound originates on the left and smoothly moves right across the entire screen.
This page has the simplest design, but was the most difficult technical puzzle to solve. I still don't feel like I have figured out the 'right' way to accomplish what I want from this 'stop'. My intention was to visualize the process of granulation synthesis, but I fell a little short of that grandiose aim. The end result is still very engaging and I imagine will be a new way of experiencing sound for many users.
I settled for a smooth way to visualize cutting up, or 'slicing', an audio file and looping the slices. These 'audio slices' can be as small as 0.01 seconds long and can be slid by the user across an audio file. In a similar way as hearing a tone emerge out of an amplitude modulated audio file, hearing something like a tone emerge out of an extremely small loop is a fascinating experience I want to share with the users of this project.
I still do not understand the solution I achieved in this 'stop'. For a more detailed description of my journey getting dynamically-sized loops to sound good in the project as a whole, refer to this appendix tutorial. What is happening on this page is best understood from looking at the code, from lines 146 to 183 in TourAudioSlicer.js.
I had initially named this page Tour: Granulation, but because there is no grain overlap or randomization present in the program I eventually decided to name it 'Audio Slicing'. The other limitation in this program is the presence of only a single slice. I did copy my code from the draw() loop into a class named GranulationSlicer and tried to instantiate multiple 'slicers' in a single script, but it ran too slow. It worked in principle, but something about the processing it takes to constantly calculate the loop size was too much for the draw() loop. The loop lines' movement was jerky and often the clip would play all the way through instead of looping a specific 'slice'. Sometimes this happens anyway in the final version, just refresh the page if it does.
The Playground is visually utilitarian, but has a lot of sonic potential. This is the part of the project designed specifically for more experienced users. Aside from having easy access to a bank of effects, the Playground's playful aspect comes in the form of being able to map effect parameter controls to the mouse's position on screen.
The first thing the user will likely notice is the three horizontal rows of buttons. Each 'track' on the Playground is an instance of two classes, the SamplerButton and the PlaygroundControls. The PlaygroundControls class manages the effect rack set on each track and communicates back with the draw() loop of PITW_Playground.js for effect parameter mouse tracking.
Six out of the eight available effects on each track also are, by default, being modulated by LFOs to make the output a little more interesting. For example, the delay's delay time is being affected by an LFO to bring out the pitch effect that the user might have explored earlier in the Tour. Presenting the user with a non-static set of effects was a choice made for two main reasons.
You’ll see I wear only gray or blue suits,... I’m trying to pare down decisions. I don’t want to make decisions about what I’m eating or wearing. Because I have too many other decisions to make.
(Obama, 2012)
If the user wants more control on the effects or to stop the default LFO, there is a system in place to map an effect's parameter to the mouse's position on screen. This system is designed to give the user agency over the shape of the sound, but in a less precise and more playful way. The user can select pink for x-axis control, or yellow for y-axis control, so they can have multiple effects changing in interesting ways while moving the mouse diagonally across the screen.
The final bit of control the user has over the effects is the ability to 'freeze' in place their selected mouse position/effect parameter by pressing the spacebar. I hope that by shaping a sound in such a visual way it can open new avenues of sonic experimentation to experienced sound designers and whet the appetite for even more detailed control for beginner sound designers.
The Playground makes the same trade off that p5.sound makes between control and accessibility. p5.js and p5.sound are designed to be easy to use, but not to enable every possible permutation of digital sound manipulation. p5.sound's design lowers the bar of effort it takes to just experiment with its library of tools by herding the user into making connections in specific ways. It's ironic that I then used the more powerful, less accessible library of the two (Tone.js) to make a more accessible, less directly controllable audio program.
During the entire 3-month development process, whenever I would send a link of an early build of Playing In The Waves, the friend or family member who received the link would open it on their phone. Then tell me the website I sent them is broken. Most people expect any website that works on a computer will also work on a phone. For a professional web developer accommodating multiple platforms is a standard part of the process, but for me this posed a significant problem.
All the size variables in the code are some percentage of the width or height of the screen it is displayed on. In this way, the program is dynamic and scales whatever device it is accessed by. The main issue is computer screens, which I have been developing for primarily, are horizontal and smart phone screens are vertical. The Playground looks squashed sideways on a phone screen, making the buttons too small to accurately push. For this reason I made a phone version of the Playground, which gives a usable option for users who primarily interface with the internet on a mobile device.
The phone version of the Playground is a much simpler design, and has only two tracks. The effects available were chosen to be the most interesting to a new user and most audible on a phone speaker. It can be assumed that if a user is accessing the phone version, they are likely not to be using headphones so the audio effects must be immediately audible and interesting on low-quality speakers. At least with this program, recording a sound to hear it slowed down or backwards is only a URL away for most people with a smart phone.
This is by far the most lines of code I have ever written in a single project and in is the biggest programming project I have ever worked on by myself. Now, at the end of this journey all I can say is 'it works!' After hours and hours of trial and error, reading the same documentation page over and over again, and fixing countless bugs, it all works. For the most part.
I hope my tutorials in Appendix C can help future web audio programmers as well. I have benefitted for years by the free sharing of knowledge on the internet, and this is the first time I have made a concerted effort to give back to that space. Part of writing this dissertation was to provide some foundational understanding and context to web audio technology so others can teach it in the future.
There were a few things that did not get solved by the time I submit this project.
The Phone Version is a sub-par compromise to allow mobile device users the same access as computer users to this program. Sometimes, especially on Android phones the Phone Version glitches out or doesn't allow the user to record in. Ideally the whole site would be built from the group up with HTML and CSS, which would scale much nicer between different screen types.
This project focused much more on the technical aspects of web audio than the actual sounds it produces. The reverb, playback rate stretching, and pitch shifting all sound very digital as they are just default effect nodes from Tone.js. When lots of sounds are happening at once, there is still some clipping. This can be mitigated somewhat by user controls, like the volume sliders in the Playground, but so much time was devoted to getting everything working that there just was not enough time to polish the sounds themselves.
There is a lot work to do going forward. A complete rebuild of this project without p5.js would be the first step. This would save a lot of processing power and would help the program work better across all devices. Completing this project opens the door for lots of other web audio ideas, like focusing on generative processes or real-time sound recognition.
Web audio is an exciting space to work in and is only getting more powerful. While it can never keep up with more traditional kinds of audio software, it does have reach and speed of access that other programs can't get close to. I hope this program can go on to bring a little audio magic into the world.
The Audio Working Group. “Audio Working Group.” Accessed August 1, 2021. https://www.w3.org/2011/audio/.
Asimov, Isaac. Foundation. 1953, London: Weidenfield & Nicolson Ltd.
Borgeat, Patrick. “Audio in the Web 02: The Past and Forgotten,” cappel:nord. July 21, 2021. www.cappel-nord.de/b/audio-in-the-web-02-the-past-and-forgotten/
Bouchard, David. "Week 6 - Part 1 - Tone.Js, Drones". YouTube. Accessed July 15, 2021. https://www.youtube.com/watch?v=ddVrGY1dveY&t=2093s&ab_channel=DavidBouchard.
Duvander, Adam. “The History of IMG and EMBED Tags | WIRED.” Accessed August 1, 2021. https://www.wired.com/2008/09/history-of-img-and-embed-tags/.
Fisher, Tyler. “Audio in the Browser: Horrors and Joys." Accessed July 20, 2021. https://source.opennews.org/articles/audio-browser/.
Github. “Tone.Js Docs.” Accessed June 15, 2021. https://tonejs.github.io/docs/14.7.77/index.html.
Github. “Global and Instance Mode · Processing/P5.Js Wiki.” p5.sound, Accessed July 15, 2021. https://github.com/processing/p5.js.
Kearney-Volpe, Claire. “P5 Accessibility.” Processing Foundation (blog), Medium. Accessed August 12, 2021. https://medium.com/processing-foundation/p5-accessibility-115d84535fa8.
Lewis, Micheal. "Obama's Way".Vanity Fair September, 12, 2012. Accessed August 18, 2021. https://www.vanityfair.com/news/2012/10/michael-lewis-profile-barack-obama
LSU STEM Pathways. “Sound and Music :: Educational Material for LSU Programming Digital Media.” LSU Stem Pathways Course. Accessed July 15, 2021.
https://pdm.lsupathways.org/6_resources/7_soundandmusic/.
Max 7 Documentation. “MSP Delay Tutorial 4: Flanging” Accessed July 25, 2021. https://docs.cycling74.com/max8/tutorials/06_synthesischapter03.
Max 8 Documentation. “MSP Synthesis Tutorial 3: Using Amplitude Modulation” Accessed July 25, 2021. https://docs.cycling74.com/max8/tutorials/06_synthesischapter03.
MDN Web Docs. “Hit Regions and Accessibility" Accessed August 12, 2021. https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Hit_regions_and_accessibility.
Nast, Condé. “The Rise and Fall of Flash, the Annoying Plugin That Shaped the Modern Web.” Wired UK. Accessed July 20, 2021. https://www.wired.co.uk/article/history-of-macromedia-flash.
Negus, Chris. Internet Explorer 4 Bible. Foster City, CA : IDG Books Worldwide, 1997. http://archive.org/details/internetexplorer00negu.
The Processing Foundation. “Reference | p5.js.” Accessed June 15, 2021. https://p5js.org/reference/.
The Processing Foundation. "Sound Library Reference | p5.js.” Accessed June 15, 2021. https://p5js.org/reference/#/libraries/p5.sound.
The Processing Foundation. “Learn | p5.Jjs.” j5.js Reference. Accessed August 12, 2021. https://p5js.org/learn/p5-screen-reader.html.
Sagan, Carl. Cosmos. London: Abacus, 1995.
Shankland, Stephen. “Steve Jobs’ Letter Explaining Apple’s Flash Distaste.” CNET. Accessed August 1, 2021. https://www.cnet.com/news/steve-jobs-letter-explaining-apples-flash-distaste/.
Shiffman, Daniel. “Code! Programming with P5.Js Video Playlist - YouTube.” Accessed June 14, 2021. https://www.youtube.com/playlist?list=PLRqwX-V7Uu6Zy51Q-x9tMWIv9cueOFTFA
Shiffman, Daniel. "P5.Js Sound Tutorial Video Playlist - YouTube.” The Coding Train. Accessed June 14, 2021. https://www.youtube.com/playlist?list=PLRqwX-V7Uu6aFcVjlDAkkGIixw70s7jpW
Shiffman, Daniel. “Topics of JavaScript/ES6-ES8 Video Playlist - YouTube.” The Coding Train. Accessed June 14, 2021. https://www.youtube.com/playlist?list=PLRqwX-V7Uu6YgpA3Oht-7B4NBQwFVe3pr
Tokui, Nao. “(Example) Tone.Js / Microphone Input to Player.” Codepen. Accessed July 15, 2021. https://codepen.io/naotokui/details/abBvBmW.
Tramte, Dan. "Browser Noise: Web Audio Tutorials". YouTube. The Audio Programmer. Accessed July 15, 2021. https://www.youtube.com/playlist?list=PLLgJJsrdwhPywJe2TmMzYNKHdIZ3PASbr
W3C. “Elements/Bgsound - HTML Wiki.” Accessed August 1, 2021. https://www.w3.org/html/wiki/Elements/bgsound.
W3C. “Web Audio API, W3C Recommendation” Accessed July 20, 2021. https://www.w3.org/TR/webaudio/.
W3C. “HTML5 Differences from HTML4.” Accessed August 1, 2021. https://www.w3.org/TR/html5-diff/.
W3Schools. “JavaScript ES6.” Accessed June 17, 2021. https://www.w3schools.com/js/js_es6.asp.
W3C. “Web Audio API Brings Audio Design to the Web as It Becomes a World Wide Web Consortium (W3C) Recommendation." Accessed August 1, 2021. https://www.w3.org/2021/06/pressrelease-webaudio.html.en.
“Web Audio Modules | Community Site.” Accessed August 12, 2021. https://www.webaudiomodules.org/.
Wilson, Chris “A Tale of Two Clocks - Scheduling Web Audio with Precision - HTML5 Rocks.” HTML5 Rocks - A resource for open web HTML5 developers. Accessed July 20, 2021. https://www.html5rocks.com/en/tutorials/audio/scheduling/.
Web Design Museum. “Internet Explorer 4.0" Accessed August 1, 2021. https://www.webdesignmuseum.org/old-software/web-browsers/internet-explorer-4-0.
Web Design Museum. “Netscape Navigator 2.0" Accessed August 1, 2021. https://www.webdesignmuseum.org/old-software/web-browsers/netscape-navigator-2-0.
WHATWG. “HTML Living Standard.” Accessed July 16, 2021. https://html.spec.whatwg.org/multipage/introduction.html#history-2.
Manipulating The Web Audio API in conjunction with HTML and CSS is complicated, and beyond my current skills. Thankfully there are many open-source web audio libraries that make that task a little easier.
Here is my method for standing up and testing all my projects made with p5.js and Tone.js. This method works because:
I downloaded Node Package Manager (npm), and Node.js Local Server to run my websites off my personal computer. There are a lot of resources that recommend against installing Node.js with the sudo keyword, but this is the method I used because it was the easiest, fastest, and I am the only user on my private machine. I also am using VS Code, with the 'Live Server' extension by Ritwick Dey.
Getting all the JavaScript files and libraries working together on the same HTML file can be tricky. My HTML files all are very bare as all the processing is done in JavaScript files. So the order that the <script> tags are loaded in is important.
If using p5.sound, then the first 3 JavaScript files should be:
<script src="p5/p5.js"></script> <script src="p5/p5.sound.js"></script> <script src="PITW_Classes_p5sound.js"></script>
If using Tone.js, then the first 3 JavaScript files should be loaded like this:
<script src="p5/p5.js"></script> <script src="https://unpkg.com/tone@14.7.58/build/Tone.js"></script> <script src="PITW_Classes_Tone.js"></script>
You'll notice that there are two 'PITW' libraries. These are my two custom class libraries, separated by sound library. Every HTML file I submit as part of the final project has Tone.js, so all are set up like the second example. Also Tone.js has to be loaded in via external link to stay out of the 'Global Mode' p5.js is set up in.
I am not a seasoned web developer. I learned the basics of JavaScript during the development of this project and I had a secret weapon: p5.js. On its own website, p5.js is described as:
a JavaScript library for creative coding, with a focus on making coding accessible and inclusive for artists, designers, educators, beginners, and anyone else!
(The Processing Foundation, 2021)
p5.js is much more than just a library. It is a set of tools to make 3D graphics, animation, HTML elements, and can be used to make all the components of a simple website. This is how I was able to make all of Playing In The Waves coding exclusively in JavaScript, using HTML only to link the necessary files together.
p5.js was developed by The Processing Foundation to be an entry point for people, like me, new to writing JavaScript code, or even for artists who have never written a single line of code and want to put a creative idea on a web page. Writing JavaScript code in p5.js is much simpler than writing inline JavaScript in an HTML file, which frees up someone like me to focus purely on the sound design. It is also home to the first web audio library I studied to develop Playing In The Waves
The readme on Github for p5.sound claims it:
"brings the Processing approach to Web Audio as an addon for p5.js."
(The Processing Foundation, 2021)
p5.sound takes the code design principles of p5.js and applies them to audio programming in JavaScript. Oscillators, FFT analysers, audio recorders, and some basic effects are all present in p5.sound.
I would guess that p5.sound is the most approachable and simplest web audio library available. It brings p5.js's accessibility principles to The Web Audio API, but comes with the tradeoff in flexibility. Because p5.sound was developed as just an addition to an entirely graphics-focused library, it is not the most powerful web audio tool available. It is running through the p5.js engine and is also standing on top of several other technologies, so any audio processing has to go through several layers of abstractions to complete whatever task it needs to. One of the libraries p5.sound uses and abstracts away to deliver its simplicity and ease of use is Tone.js, the other and more powerful web audio library I used for this project.
Tone.js a high-performance web audio library, made by Yotam Mann, that sits on top of The Web Audio API. The readme on Tone.js's GitHub page describes the library as:
"a Web Audio framework for creating interactive music in the browser. The architecture of Tone.js aims to be familiar to both musicians and audio programmers creating web-based audio applications... Additionally, Tone provides high-performance building blocks to create your own synthesizers, effects, and complex control signals."
(Mann, 2021)
Tone.js is the other side of the trade off between ease of use vs flexibility. Tone.js provides a powerful wrapper around the Web Audio API to schedule events with sample-accurate precision and create a timeline that can run parallel alongside another other processes. Tone.js has a well-organized, but much less friendly documentation page compared to p5.sound, with fewer examples to study. It was a significant time investment for me to understand how to accomplish even basic operations, but the effort was well worth it. Many critical components of the final version of Playing In The Waves rely on Tone.js functionality that does not exist in p5.sound.
p5.sound actually is standing on Tone.js to some degree. The p5.sound readme gives credit to this fact:
"p5.sound is built with a few modules (Clock, TimelineSignal, and signal math components) from Tone.js..."
(The Processing Foundation, 2021)
Tone.Clock is one of the most powerful components of the library. So powerful that p5.sound is built on top of it. As detailed in Chapter 2, having access to an accurate clock is a facet of audio programming that, for web audio programmers, is a hurdle to overcome. The Tone.Clock documentation page explains the object as:
"A sample accurate clock which provides a callback at the given rate. While the callback is not sample-accurate (it is still susceptible to loose JS timing), the time passed in as the argument to the callback is precise."
(Mann, 2021)
While Tone.js can't fix the fact that JavaScript is not a high-performance language like C++ is, it is an excellent tool for web audio programmers.
For me, the biggest disadvantage of p5.sound is the lack of direct access to the audio buffer array values. Coming from my Audio Programming class, which was focused on the JUCE framework and C++, I took for granted the ability to write a for loop to iterate over an array of audio data. Tone.js does allow the developer this access and has multiple ways to retrieving and manipulating the raw audio data. This access was critical in solving the issue in Appendix C, and was a breakthrough that cleared the way for more interesting audio effect functionality in the final version.
Tone.SignalTone.js's 'Signal' data type allows calculations to be processed at audio rate and gives the ability to modulate object parameters with audio signals. The above quote from p5.sound's readme also mentions Tone.js's 'signal math' as a component that both libraries use. p5.sound does allow things like amplitude and frequency modulation, but Tone.js just allows more flexibility in general to perform audio rate processes. Tone.js's delay nodes will accept an audio signal from an LFO to modulate its delay time. To accomplish this in p5.sound, you would have to update the delay time in the draw() loop and smooth the values as you go. This is a much more processing-heavy approach and sucks power away from other potentially interesting sound design components.
Tone.LFOTone.js has a dedicated LFO object for controlling audio processes via other audio signals and its output can easily be scaled to different min/max values. p5.sound only has an oscillator object, and scaling the min/max output values is very sensitive. Refer to Appendix C, tutorial 6 to read more about this specific issue.
The class-based, multi-track design I implemented in the 'Playground', was not possible with p5.sound. I made track and effect rack classes in both p5.sound and Tone.js, both running on a p5.js script. Having multiple tracks with multiple, different effects running simultaneously created many audible glitches in p5.sound. You could say the entire Playground component of Playing In The Waves is an example of something Tone.js is capable of that p5.sound is not.
Credit where credit is due. There are actually three examples of baked-in functionality that exist in p5.sound's objects that are not easily accessible in Tone.js.
p5.SoundFileYou can get a file to play backwards easily by passing a negative number as an argument to the p5.SoundFile().rate() method, similar to the [~groove] object in Max MSP. Tone.js's main audio file handler object, Tone.Player(), will not accept a number less than zero to its playback rate parameter. It is possible set the Tone.Player.reverse boolean to true, which would flip the buffer around, but will result in an audible pop if done in real-time. This is why I had to make the ForwardsAndBackwardsSamplerButton class.
p5.Reverb has a boolean to flip its buffer and create a backwards reverb. This is a really cool effect that does not exist in an easily accessible way in Tone.js's reverb nodes. I'm sure it's possible to create this effect with Tone.js by manually iterating over the Tone.Reverb's return signal's audio buffer, but I did not have time to try to find a solution to this problem. It's a shame because that would have been a really nice effect to include in the 'Playground'.
Both Tone.js and p5.sound allow the user to record the audio output of the webpage then download the returned file. p5.sound will return a .wav file, which is much nicer than to use than the .webm file type Tone.js returns. Using only Tone.js there is no option to convert its download file to a more usable file type like .wav or .mp3.
After talking to Professor Jules Rawlinson about potentially working in web audio for my final project, he kindly pointed me to the autosampler early in the project developement. This web application showed me early on how web audio can achieve a level of accessibility that no other audio program I have used can reach. It is a program that with the click of one button (after asking permission to your device's microphone) will begin to record your surroundings and play back in multiple, layered loops. These loops are distorted by simple buffer manipulation techniques like reversing the audio or adjusting the playback rate, but create lots of new and interesting sounds. The transparency of the interface and depth of sonic complexity achieved from just the sound coming in from an open window proved to me that many of my initial ideas were possible.
Leimma and Apotome are two of the most advanced web audio tools I have come across. Leimma is a source to experimenting with different kinds of tuning systems and Apotome is a generative web audio system that can be routed through Leimma. The sound quality generated by these two web audio applications is extremely high and are a very powerful and flexible sstem for sonic exploration. Leimma and Apotome represent the high water mark for technical sophistication and musical usability out of any web audio application I have used.
The official W3C press release recommending the use of The Web Audio API on 17 June 2021 mentions Leimma and Apotome in a list of high profile applications that already use The Web Audio API like Spotify, SoundCloud, and Ableton. What an endorsement!
In communicating with the team behind behind the technical aspects of Leimma and Apotome, (Tero Parviainen, email communication on 02, August, 2021) I learned they use Tone.js synth objects in some of the sound generation for both tools. However most of the synthesizers used in Leimma and Apotome were developed using Audio Web Modules. Web Audio Modules is a way to write C++ code that is then compiled into an assembly language that a browser can understand. Link below for more information on how 'WAM's work.
The Chrome Music Lab is a child-friendly, interactive series of activities that provides a bright and colorful means of learning the basics of sound. In their own words:
"Chrome Music Lab is a website that makes learning music more accessible through fun, hands-on experiments."
(Chrome Music Lab, 2021)
This website has more of a museum feel, and finds very interesting ways to visualize different aspects of sound. This is a great example of making education fun and appealing to young minds, but is still stimulating to more experienced sound designers. The 'Voice Spinner', 'Spectrogram', and 'Oscillators' exhibits are stand outs on this platform.
<>
As the name would suggest, Chrome Music Lab was made my a Google-backed team, which explains why the interface is so clean and dynamic. Further down in the 'About' section, the website mentions both The Web Audio API and Tone.js as technologies foundational to the creation of the Chrome Music Lab.
Similar in many ways to the Chrome Music Lab, Ableton has a series of playful and educational inter-actives designed to educate an audience on the basics of synthesis. Unlike the Chrome Music Lab, the Ableton Learning Synths site is focused almost entirely on synthesizer concepts.
This series of tutorials was also very influential to me during the development of Playing In The Waves. The box/oscillator example mentioned above is a technique used in quite a few web audio applications, including on p5.sound's documentation page. Mapping one effect to the x-axis of a rectangle and another effect to the y-axis is a core principle of Tour: Play Rate and The Playground.
In fact, there are quite a lot of similarities between my project and the Ableton Learning Synths page that I did not realize until later in the development process. The overall structure of the Ableton site takes the user through a series of accessible and interactive tutorials before dumping that user out into a much more complex page called a 'Playground'. Ableton Learning Synths is almost entirely music focused and Playing In The Waves centers on pure sound and DSP concepts. Every single page in Playing In The Waves allows the user to record directly from their device microphone and apply that recording to my tutorials, while Ableton uses oscillators to generate all their sample sounds.
setup() and draw()?setup() and draw() functions are the two halves of the p5.js system that is easy to understand and draw graphics to the screen.
setup()setup() is a function that runs one time right as the program loads. It is the place where the user can initialize all their objects and set their initial parameters. Any objects or calculations that might need to accessed later in the program should be setup in setup().
draw()Understanding the draw() loop is essential for understanding p5.js, and is where the majority of program actually happens. I refer to it as 'The draw() Loop' because it is a giant loop that runs at 60 frames per second for as long as the program is open. By running at a high enough frame rate to simulate motion, graphics and animations can run smoothly in the draw() loop. Because it renders from top to bottom, objects at the top of the draw() loop will be rendered behind things at the bottom of the draw() loop. For example, it is always a good idea for the first line of the draw() loop to call the background() function.
The draw() loop is generally the biggest performance bottleneck in the project. Running too many visual and audio processes based in the draw() loop will result in audible glitches and artifacts, so be careful. When using Tone.js this is not as much of a problem, but it will create a bottleneck with p5.sound if you're not careful.
In above example, you will notice the oscButton does not exist at all in the draw() loop, but can be manipulated in real time. This is because you can create HTML elements with p5.js, that can have listeners and callback functions outside of the draw() loop. These objects are not being called or checked at 60 fps, so do not use as much valuable processing power. createButton() is one of these p5/HTML objects, and any function that is prefixed by create is a p5/HTML element. Using lots of p5.Elements is a strategy I use in every script to save on processing power. This is also why the buttons are so bland looking.
This project has tested my capabilities as an audio programmer and my patience. There have been quite a few issues that came up that had no ready-made solution for at all. I feel that the knowledge gained in this chapter is extremely valuable to me, and to any new web audio programmer. I wish I could send this chapter back in time to myself 3 months ago.
The first tip I have to give is more of a practical one than a technical one. When activating a p5.SoundRecorder or Tone.Recorder to begin recording, it will also record the mouse click, especially if the user input device is the computer (laptop) speaker. Just use setTimeout() to give yourself 100 - 200 milliseconds between the actual mouse click and the start of recording. I have 120 ms of space hard-coded in my SamplerButton class in this space.
setTimeout(recorder.record(soundFile), 120);
setTimeout(recorder.start(soundFile), 120);
Getting the SamplerButton class working the way I wanted it to was the longest running technical issue in this entire development process. As it turns out, it's really difficult to create a looping audio file player that has an amplitude envelope that will shape itself dynamically in JavaScript. Tone.Player has a boolean to set to true to loop, so that was the easy part. However getting newly recorded user audio to loop without clicking at the beginning and (especially) the end was very tricky. I had moved on to Tone.js by the point I was ready to tackle this issue, so I do not have a solution for this problem in p5.sound.
The Tone.Player also has built in fadeIn and fadeOut properties to set automatic fade times on either side of the audio file. These act like little built in ADSR attack and releases, but like ADSR they need to be triggered. When a Tone.Player.loop is set to true and the audio file is looping, the built in fades are inactive. The fades must be triggered by a .start() or a .stop() command to the Player.
Tone.js has a tool called a Loop which will repeatedly callback a function given as an argument at a given 'interval'. Knowing I had to call .start() and .stop() on the Player to trigger the built-in fade times, I made a Tone.Loop that did just that and had the Loop.interval dynamically set to the length of the recorded audio file. This worked pretty well, but would create runtime bugs where the Loop would be triggered on time, but the Player would not play. There would usually be a click at the end of the audio file every third or fourth repeat despite the built in fades triggering. I suspect that inaccurate JavaScript timing, even bolstered by the Tone.js and Web Audio API's Audio Clock, was throwing things off.
Finally the solution was a suggestion given to me by my professor Tom Mudd. He suggested using a simple method we had learned in his Audio Programming class: using a for loop to iterate over part of the audio buffer array and manually scale up or down the ends of the buffer. Thankfully, Tone.js allows direct access to the audio buffer to accomplish this. Tone.js does have its own audio buffer object, the ToneAudioBuffer, but so the Player via its .buffer attribute. The ToneAudioBuffer has a method called getChannelData(), which returns the actual array of digital audio data, provided you pass in the channel number (in most cases will be either 0 or 1 for stereo left and right).
By just using a for loop immediately after recording user input, this also freed up a lot of processing power that was being taken up by repeated calls to a Tone.Loop. Before I successfully implemented this solution, the Playground couldn't have any reverb because it was too heavy on processor power. The below code streamlined every part of Playing In The Waves and was a huge breakthrough in the development process.
SamplerButton line 68):setTimeout(() => { let array = this.player.buffer.getChannelData(0); let max = 10000; for (let i = 0; i < max; i++) { array[i] = array[i] * (i/max); this.player.buffer.getChannelData(0)[this.player.buffer.getChannelData(0).length - (1 + i)] = this.player.buffer.getChannelData(0)[this.player.buffer.getChannelData(0).length - (1 + i)] * (i / max); } array = this.player.buffer.getChannelData(1); for (let i = 0; i < max; i++) { array[i] = array[i] * (i/max); this.player.buffer.getChannelData(1)[this.player.buffer.getChannelData(1).length - (1 + i)] = this.player.buffer.getChannelData(1)[this.player.buffer.getChannelData(1).length - (1 + i)] * (i / max); } }, 300);
Part of the reason I didn't try this sooner was because I wasn't sure if the for loops would go fast enough to be unnoticeable. This is why I use setTimeout to give 300ms of space to process audio before it can be used. It may not need this much time but 0.3 seconds is not a long time for the user to wait after recording to play back and is not generally perceptible.
The only instance of recording user input and playing it back without the above listed method is in Tour: Audio Slicing. In that case, through lots of trial and error, chanced on a solution. I suspect that the fact this method repeatedly calls a Tone.Loop is why I was not able to turn this into a class and instantiate two versions of this on the same script while still having a functional web audio experience. To be clear, I do not understand why this works. I tested Tone.Player.fadeOut and know it only is triggered when called by player.stop(), but this method does not do that. Without the code set up like this, the 'Audio Slicer' would create audible clicks at the end of every loop, which is hard to listen to when the loop is only 100 ms long. Take a look at 'TourAudioSlicer.js' line 289 to see how this is implemented.
One of the major benefits of using Tone.js instead of p5.sound is the Tone.LFO object. Where p5.sound does not have a dedicated LFO object and is very inflexible when it comes to scaling audio output into a usable range, Tone.js provides a ready-made LFO that has easily adjustable max/min attributes to set the output range. This was one of the main reasons I switched over to Tone.js from p5.sound in the first place, but that was before I tried to implement amplitude modulation with Tone.js.
I also want to remind the reader here that Tone.js volume objects deal in decibels instead of normal range: 0.0 - 1.0. Tone.js documentation claims there is a constant called -Infinity that functionally works as zero volume when passed into a volume node. I have not been able to get it to work, so i just use 0 as full volume, and -100 as no volume.
All objects in Tone.js that return an audio signal must be given the .start() command before they actually generate that audio signal. The exception to this rule seems to be if a Tone.LFO is connected to a Tone.Volume. Volume is just a useful, but not necessary volume node that I use to plug all my objects into in order to keep my amplitude processes separate from any other processes.
The output from a Volume node that has an inactive LFO connected to it sounds modulated, as though it is receiving an audio signal from an oscillator. Even before the LFO.start() function has fired. This must be some kind of bug in the Tone.js code, but there is a way around it. In setup(), set the phase of the LFO to 90. I am not sure why this works, but I think it's because 90 degrees of a sine wave is the top of the wave, which is 1. So the LFO is just multiplying it's connected target by 1.
In setup() of Tour: Amp Mod:
ampModLFO.phase = 90;
When applying an LFO to a Tone.Volume or a Player.volume, there seems to be an issue if the target (carrier) volume node is not already at -100 or some other functionally silent decibel amount. I set up my amplitude modulating LFOs to have a .min of -100 and a .max of 0, so before I apply the LFO signal, I turn the target volume node down to -100. This is must have something to do with multiplying the signals together and not getting a result that sounds like amplitude modulation. Whatever the reason is, I suggest turning the target volume node down to whatever the LFO's .min value is just beforehand.
triggerAmpMod() from PlaygroundControlsif (this.ampModActive) { this.player.volume.rampTo(-100, 0.1); this.ampModLFO.amplitude.rampTo(1, 0.1); this.ampModLFO.start("+0.1"); }
As the above example shows, I hedge my bets by keeping the LFO's amplitude at 0 before I need it. The .phase = 90 is the most important part, but other safeguards to keep the LFO silent until you need it can't hurt.
Speaking of the above example, a quick tip I wanted to leave here is to make use of rampTo() whenever possible. It provides a sample rate speed way of smoothly modulating any signal value. The first argument is the target value, and the second argument is the ramp time. The Tone.js documentation says which object attributes are numbers or signals. Anything that is a signal should be changed with rampTo(). The rampTo() function is actually present on the Tone.js 'Examples' page, linked below, but is not present on the API documentation page.
My library of custom p5.js/Tone.js classes are relatively narrowly suited to the designs of the programs in Playing In The Waves, but there are three that I think have the most reusability for other developers: SamplerButton, OscScope, and LFOVisualizer.
SamplerButton and OscScope are largely derived from other tutorials, all linked in the code comments in the PITW_Classes.js file. The LFOVisualizer class is the only reusable class that was an original trick that might not be obvious on first glance.
Even though Tone.js does allow direct access to the audio buffer, I have not been able to find a way to get the real-time phase value of an LFO. As shown above, you can set it easily enough, but getting the phase as a return value to scale and track with an animation is not straight forward. In LFOVisualizer, a Tone.Waveform returns an array of amplitude values, of which the very first is scaled from normal range (0.0 - 1.0) to the dimensions of the vertical track and those values are passed to the p5.js circle object's y-axis position.
It's important to note that this process is not happening at sample rate. This is being called from inside the draw() loop, as Tone.Waveform and p5.FFT both only return a snapshot in time. I have not tried to instantiate more than one of these objects at a time in a script, but as far as I can tell, it does not cause any slowdown despite taking detailed snapshots of the audio signal at 60 frames per second.
As mentioned in the 'Web Audio' chapter, JavaScripts major update, ES6, allowed for the creation of classes in JavaScript. Being able to make several instances of a class was critical to getting the Playground to work, but getting Tone.js to work inside a class took some trial and error. Here's a few tips I picked up from that process that I wish I had a few months ago.
Scope in JavaScript gets a little weird inside class declarations. One of the major things to deal with in the regard is the this. prefix. Any variable that is scoped to more than just a single function, must have the this. prefix. I went to test some newly-written code and had to go back to add a this. to a variable being compared inside an if statement more times than I can count. If any new JavaScript user is running into bugs while using classes, always check this FIRST.
this. Example:if (this.reverbActive) { this.player.connect(this.verb); } else { this.player.disconnect(this.verb); }
Uncaught (in promise) ReferenceError: [object] is not defined
=> Arrow FunctionI don't fully understand why, but using callback functions inside JavaScript classes require the use of the => arrow function. JavaScript already allows for 'anonymous functions' which are unnamed functions stored inside variables or callback functions. These are functions which are generally defined then immediately used. An anonymous function is traditionally declared with the function keyword. ES6 introduced the => symbol to as a syntactic shortcut to having write out the word 'function' for an anonymous function.
Arrow functions also seem to facilitate using callback functions inside a JavaScript class. I mentioned eariler that scope can get a little weird in a JavaScript class, so I think this is a way to deal with this issue. The => symbol is used to create an anonymous function that can be definted then and there, or reference another function in the same class.
=> Anonymous Function Example (line 90: TourAmpMod.js):fftButton.mousePressed(() => { scope.fftActive = scope.fftActive ? scope.fftActive = false : scope.fftActive = true; })
=> Callback Function Example (line 593: PITW_Classes.js):this.delayButton.mousePressed(() => this.triggerDelay());
Uncaught (in promise) TypeError: Cannot read property 'html' of undefined
My last tip is a warning about the scale() function in p5.sound. scale() is a function that 'scales' the audio signal coming from an oscillator into a useful range. It basically fulfills the same role map() is used for, but to work on audio signals instead of numbers. scale() seems to be the only way to have a functioning LFO in a p5.sound script.
The issue is scale() does something to the audio signal that can not be undone, and attempting to call scale() on an oscillator more than once will result totally stop the program, forcing a restart. This is not obvious because scale() is not a function that is applied to the oscillator, it is a function that seems to act as a filter between the oscillator and target. scale() works fine and does its job, but be aware if you want to change the range a p5.Oscillator is outputting, it is best to just create a new instance of p5.Oscillator and apply the new scale() to it. JavaScript is a 'garbage-collected' language, and will free up memory allocated to objects that are no longer referenced.
Uncaught TypeError: Cannot read property 'length' of undefined