Author(s): Dimitre Oliveira Deep Learning Applying deep learning to classify audio with Tensorflow Source: https://www.tensorflow.org/tutorials/audio/simple_audio We have seen a lot of recent advances in deep learning related to vision and language fields, it is intuitive to understand why CNN performs very well on images, with pixel’s local correlation, and how sequential models like RNNs or transformers also perform very well on language, with its sequential nature, but what about audio? What are the different models and processes that are used to deal with audio data? This article will show you how to solve a simple problem in audio classification. You will also learn the Tensorflow code and some common methods. Disclaimer: This code is based upon my work for the “Rainforest Connection Species Audio Detection” Kaggle competition. However, for demonstration purposes I will be using the “Speech Commands” dataset. Waveforms Audio files are usually stored in “.wav”, which is commonly known as waveforms. If we look at one of the waveform samples, we’ll see that the x-axis represents the time while the y-axis the normalized signal ampltude. Intuitively, one could model this data to be a regular series, such as e.g. Stock price forecasting can be performed using a RNN model. However, since the audio files are in the “.wav” format, it is possible to convert these waveform samples into spectrums. Spectrograms A spectrum is an image representation that shows the frequency range of the waveform signal. It can also be used to show the frequency distribution. The spectrogram of the waveform representation we just saw is shown below. The x-axis represents the sample time, while the y-axis the frequency Speech Commands Use Case. To simplify this tutorial we’ll be using the Speech Commands dataset. This dataset contains one-second audio clips that contain spoken words such as “down”, “go”, and “left”, and includes “no”, and “right”, and “stop”, and “yes”. Processing audio with Tensorflow. Now that you have an understanding of the process, it is time to see how the code is implemented. The “Speech Commands” dataset is used to simplify the tutorial. This dataset contains one-second audio clips with spoken words like: “down”, “go”, and “left”, and these audio clips can be edited to include the following audio commands: Audio processing using Tensorflow. Loading the data def load_dataset(filenames): dataset = tf.data.Dataset.from_tensor_slices(filenames) return dataset The load_dataset function will be responsible for loading the .wav files and converting them into a Tensorflow dataset. Extracting waveform and label commands = np.array(tf.io.gfile.listdir(str(data_dir)))commands = commands[commands != ‘README.md’] def decode_audio(audio_binary): audio, _ = tf.audio.decode_wav(audio_binary) return tf.squeeze(audio, axis=-1) def get_label(filename): label = tf.strings.split(filename, os.path.sep)[-2] label = tf.argmax(label == commands) return label def get_waveform_and_label(filename): label = get_label(filename) audio_binary = tf.io.read_file(filename) waveform = decode_audio(audio_binary) return waveform, label After loading the .wav files we need to decode them, this can be done using the tf.audio.decode_wav function, it will turn the .wav files into float tensors. The next step is to get the label commands from the files. In this particular case, the file paths can be used to obtain the labels. After that, you will need to decode the files using the tf.audio.decode_wav function. It will convert the.wav files into float tensors. Here is an example:First, we get a file path like this one: “data/mini_speech_commands/up/50f55535_nohash_0.wav” Then we extract the text after the second “/”, in this case, the label is UP, finally, we use the commands list to one-hot encode the labels. Commands: [‘up’ ‘down’ ‘go’ ‘stop’ ‘left’ ‘no’ ‘yes’ ‘right’] label = “up” After one-hot encode: Label = [1, 0, 0, 0, 0, 0, 0, 0] Converting waveforms to spectrograms Next, we use the commands list to one-hot encode the labels. The function tf.signal.stft converts the file into the time frequency domain. We then apply the operator tf.abs to reduce the signal phase and preserve the magnitude. You should note that some parameters of the tf.signal.stft function, such as frame_length, frame_step and min_padding, will have an impact on the generated spectrum. I won’t go into detail about tuning them, but this video will. def get_spectrogram(waveform, padding=False, min_padding=48000): waveform = tf.cast(waveform, tf.float32) spectrogram = tf.signal.stft(waveform, frame_length=2048, frame_step=512, fft_length=2048) spectrogram = tf.abs(spectrogram) return spectrogram def get_spectrogram_tf(waveform, label): spectrogram = get_spectrogram(waveform) spectrogram = tf.expand_dims(spectrogram, axis=-1) return spectrogram, label Transform spectrograms into RGB images The final step is to transform the spectrograms into RGB images, this step is optional, but here we will be using a model pre-trained on the ImageNet dataset, and this model requires input images with 3 channels, otherwise, you could keep the spectrograms with only one channel. def prepare_sample(spectrogram, label): spectrogram = tf.image.resize(spectrogram, [HEIGHT, WIDTH]) spectrogram = tf.image.grayscale_to_rgb(spectrogram) return spectrogram, label Combining all together HEIGHT, WIDTH = 128, 128AUTO = tf.data.AUTOTUNE def get_dataset(filenames, batch_size=32): dataset = load_dataset(filenames) dataset = files_ds.map(get_waveform_and_label, num_parallel_calls=AUTO) dataset = dataset.map(get_spectrogram_tf, num_parallel_calls=AUTO) dataset = dataset.map(prepare_sample, num_parallel_calls=AUTO) dataset = dataset.shuffle(256) dataset = dataset.repeat dataset = dataset.batch(batch_size) dataset = dataset.prefetch(AUTO) return dataset Bringing all together we have the get_dataset function that takes the filenames as inputs and after going through all the steps described above, returns a Tensorflow dataset with RGB spectrograms images and its labels. The model def model_fn(input_shape, N_CLASSES): inputs = L.Input(shape=input_shape, name=’input_audio’) base_model = efn.EfficientNetB0(input_tensor=inputs, include_top=False, weights=’imagenet’) x = L.GlobalAveragePooling2D(base_model.output) x = L.Dropout(.5)(x) output = L.Dense(N_CLASSES, activation=’softmax’,name=’output’)(x) model = Model(inputs=inputs, outputs=output) return model Our model will have an EfficientNetB0 backbone, and at its top, we have added a GlobalAveragePooling2D followed by a Dropout, with a final Dense layer that will do the actual multi-class classification. Although it may have a limited dataset, EfficientNetB0 has a decent level of accuracy for a light and fast model. Training model = model_fn((None, None, CHANNELS), N_CLASSES) model.compile(optimizer=tf.optimizers.Adam, loss=losses.CategoricalCrossentropy, metrics=[metrics.CategoricalAccuracy])model.fit(x=get_dataset(FILENAMES), steps_per_epoch=100, epochs=10) The training code is very standard for a Keras model, so you probably won’t find anything new here. You should now have an understanding of how deep learning works with audio files. While it’s not the best way to do this, it does make it easier and offers a better performance trade-off. You might also want to consider transformers if you plan to model audio. You can also truncate the or pad your waveforms with additional processing steps. This is useful in situations where the sample lengths are different or you need to remove a small portion of the sample. The code for this can be found in the references section. References: Simple audio recognition. Rainforest -Audio classification Tensorflow starter Rainforest -Audio Classification TF Improving A Gentle Introduction To Audio Classification with Tensorflow originally appeared in . Medium users are responding by highlighting the story and commenting on it. Published via
A Gentle Introduction to Audio Classification With Tensorflow
