NSynth
Machine learning audio synthesizer
From Wikipedia, the free encyclopedia
NSynth (a portmanteau of "Neural Synthesis") is a WaveNet-based autoencoder for synthesizing audio, outlined in a paper in April 2017.[1]
| NSynth: Neural Audio Synthesis | |
|---|---|
| Original authors | Google Brain, Deep Mind, Magenta |
| Initial release | 6 April 2017 |
| Written in | Python |
| Type | Software synthesizer |
| License | Apache 2.0 |
| Website | magenta |
| Repository | github |
Overview
The model generates sounds through a neural network based synthesis, employing a WaveNet-style autoencoder to learn its own temporal embeddings from four different sounds.[2][3] Google then released an open source hardware interface for the algorithm called NSynth Super,[4] used by notable musicians such as Grimes and YACHT to generate experimental music using artificial intelligence.[5][6] The research and development of the algorithm was part of a collaboration between Google Brain, Magenta and DeepMind.[7]
Technology
Dataset
The NSynth dataset is composed of 305,979 one-shot instrumental notes featuring a unique pitch, timbre, and envelope, sampled from 1,006 instruments from commercial sample libraries.[8] For each instrument the dataset contains four-second 16 kHz audio snippets by ranging over every pitch of a standard MIDI piano, as well as five different velocities.[9] The dataset is made available under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.[10]
Machine learning model
A spectral autoencoder model and a WaveNet autoencoder model are publicly available on GitHub.[11] The baseline model uses a spectrogram with fft_size 1024 and hop_size 256, MSE loss on the magnitudes, and the Griffin-Lim algorithm for reconstruction. The WaveNet model trains on mu-law encoded waveform chunks of size 6144. It learns embeddings with 16 dimensions that are downsampled by 512 in time.[12]
NSynth Super
| NSynth Super | |
|---|---|
 NSynth Super Front Panel | |
| Manufacturer | Google Brain, Google Creative Lab |
| Dates | 2018 |
| Technical specifications | |
| Synthesis type | Neural Network Sample-based synthesis |
| Input/output | |
| Left-hand control | Pitch bend, ADSR |
| External control | MIDI |
In 2018 Google released a hardware interface for the NSynth algorithm, called NSynth Super, designed to provide an accessible physical interface to the algorithm for musicians to use in their artistic production.[13][14]
Design files, source code and internal components are released under an open source Apache License 2.0,[15] enabling hobbyists and musicians to freely build and use the instrument.[16] At the core of the NSynth Super there is a Raspberry Pi, extended with a custom printed circuit board to accommodate the interface elements.[17]
Influence
Despite not being publicly available as a commercial product, NSynth Super has been used by notable artists, including Grimes and YACHT.[18][19]
Grimes reported using the instrument in her 2020 studio album Miss Anthropocene.[5]
YACHT announced an extensive use of NSynth Super in their album Chain Tripping.[20]
Claire L. Evans compared the potential influence of the instrument to the Roland TR-808.[21]
The NSynth Super design was honored with a D&AD Yellow Pencil award in 2018.[22]