How to Perform Voice Gender Recognition using TensorFlow in Python

Learn how to build a deep learning model that is able to detect and recognize your gender just by your voice tone using Tensorflow framework in Python.
Abdou Rockikz · 15 min read · Updated jun 2020 · Machine Learning


Gender recognition by voice is a technique in which you can determine the gender category of a speaker by processing speech signals, in this tutorial, we will be trying to classify gender by voice using TensorFlow framework in Python.

Gender recognition can be useful in many fields, including automatic speech recognition, in which it can help improve the performance of these systems. It can also be used in categorizing calls by gender, or you can add it as a feature to a virtual assistant that is able to distinguish the talker's gender.

Here is the table of contents:

Preparing the Dataset

We won't be using raw audio data, since audio samples can be in any length and can be problematic in terms of noise. As a result, we need to perform some kind of feature extraction before we feed it into the neural network.

Feature extraction is always the first phase of any speech analysis task, it basically takes an audio of any length as an input, and outputs a fixed length vector that is suitable for classification. Some examples of feature extraction methods are the MFCC and Mel Spectrogram.

We'll be using Mozilla's Common Voice Dataset, it is a corpus of speech data read by users on the Common Voice website, its purpose is to enable the training and testing of automatic speech recognition. However, after I took a look at the dataset, most of the samples are actually labeled in genre column. Therefore, we can extract these labeled samples and perform gender recognition.

Here is what I did to prepare the dataset for gender recognition:

  • First, I only filtered the samples that are labeled in genre field.
  • After that, I balanced the dataset so that the number of female samples are equal to male samples, this will help the neural network to not overfit on a particular gender.
  • Finally, I've used Mel Spectrogram extraction technique to get a vector of the length 128 from each voice sample.

You can take a look at the prepared dataset for this tutorial in this repository.

To get started, install the following libraries using pip:

pip3 install numpy pandas tqdm sklearn tensorflow pyaudio librosa

To follow along, open up a new notebook and import the modules we gonna need:

import pandas as pd
import numpy as np
import os
import tqdm
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Dropout
from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard, EarlyStopping
from sklearn.model_selection import train_test_split

Now to get the gender of each sample, there is a CSV metadata file (check it here) that links each audio sample's file path to its appropriate gender:

df = pd.read_csv("balanced-all.csv")
df.head()

Here is how it looks like:

                                filename  gender
0  data/cv-other-train/sample-069205.npy  female
1  data/cv-valid-train/sample-063134.npy  female
2  data/cv-other-train/sample-080873.npy  female
3  data/cv-other-train/sample-105595.npy  female
4  data/cv-valid-train/sample-144613.npy  female

Let's see how the dataframe ends:

df.tail()

Output:

                                    filename gender
66933  data/cv-valid-train/sample-171098.npy   male
66934  data/cv-other-train/sample-022864.npy   male
66935  data/cv-valid-train/sample-080933.npy   male
66936  data/cv-other-train/sample-012026.npy   male
66937  data/cv-other-train/sample-013841.npy   male

Let's see the number of samples of each gender:

# get total samples
n_samples = len(df)
# get total male samples
n_male_samples = len(df[df['gender'] == 'male'])
# get total female samples
n_female_samples = len(df[df['gender'] == 'female'])
print("Total samples:", n_samples)
print("Total male samples:", n_male_samples)
print("Total female samples:", n_female_samples)

Output:

Total samples: 66938
Total male samples: 33469
Total female samples: 33469

Perfect, a large number of balanced audio samples, the following function loads all the files into a single array, we don't need any generation mechanism as it fits the memory (since each audio sample is only the extracted feature with the size of 1KB):

def load_data(vector_length=128):
    """A function to load gender recognition dataset from `data` folder
    After the second run, this will load from results/features.npy and results/labels.npy files
    as it is much faster!"""
    # make sure results folder exists
    if not os.path.isdir("results"):
        os.mkdir("results")
    # if features & labels already loaded individually and bundled, load them from there instead
    if os.path.isfile("results/features.npy") and os.path.isfile("results/labels.npy"):
        X = np.load("results/features.npy")
        y = np.load("results/labels.npy")
        return X, y
    # read dataframe
    df = pd.read_csv("balanced-all.csv")
    # get total samples
    n_samples = len(df)
    # get total male samples
    n_male_samples = len(df[df['gender'] == 'male'])
    # get total female samples
    n_female_samples = len(df[df['gender'] == 'female'])
    print("Total samples:", n_samples)
    print("Total male samples:", n_male_samples)
    print("Total female samples:", n_female_samples)
    # initialize an empty array for all audio features
    X = np.zeros((n_samples, vector_length))
    # initialize an empty array for all audio labels (1 for male and 0 for female)
    y = np.zeros((n_samples, 1))
    for i, (filename, gender) in tqdm.tqdm(enumerate(zip(df['filename'], df['gender'])), "Loading data", total=n_samples):
        features = np.load(filename)
        X[i] = features
        y[i] = label2int[gender]
    # save the audio features and labels into files
    # so we won't load each one of them next run
    np.save("results/features", X)
    np.save("results/labels", y)
    return X, y

The above function is responsible for reading that CSV file and loading all audio samples in a single array, this will take some time the first time you run it, but it will save that bundled array in results folder, which will save us time in the second run.

Now this is a single array, but we need to split our dataset into training, testing and validation sets, the below function is doing that:

def split_data(X, y, test_size=0.1, valid_size=0.1):
    # split training set and testing set
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=7)
    # split training set and validation set
    X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=valid_size, random_state=7)
    # return a dictionary of values
    return {
        "X_train": X_train,
        "X_valid": X_valid,
        "X_test": X_test,
        "y_train": y_train,
        "y_valid": y_valid,
        "y_test": y_test
    }

We're using sklearn's train_test_split() convenient function, which will shuffle our dataset and splits it into training and testing sets, we then run it again on training set to get the validation set. Let's use these functions:

# load the dataset
X, y = load_data()
# split the data into training, validation and testing sets
data = split_data(X, y, test_size=0.1, valid_size=0.1)

Now this data dictionary contains everything we need to fit our model, let's build the model then!

Building the Model

For this tutorial, we are going to use a deep feed-forward neural network with 5 hidden layers, it isn't the perfect architecture, but it does the job so far:

def create_model(vector_length=128):
    """5 hidden dense layers from 256 units to 64, not the best model."""
    model = Sequential()
    model.add(Dense(256, input_shape=(vector_length,)))
    model.add(Dropout(0.3))
    model.add(Dense(256, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(128, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(128, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(64, activation="relu"))
    model.add(Dropout(0.3))
    # one output neuron with sigmoid activation function, 0 means female, 1 means male
    model.add(Dense(1, activation="sigmoid"))
    # using binary crossentropy as it's male/female classification (binary)
    model.compile(loss="binary_crossentropy", metrics=["accuracy"], optimizer="adam")
    # print summary of the model
    model.summary()
    return model

We're using a 30% dropout rate after each fully connected layer, this type of regularization will hopefully prevent overfitting on the training dataset.

An important thing to note here is we're using a single output unit (neuron) with a sigmoid activation function in the output layer, the model will output the scalar 1 (or close to it) when the audio's speaker is a male, and female when it's closer to 0.

Also, we're using binary cross entropy as the loss function, as it is a special case of categorical cross entropy when we only have 2 classes to predict. Let's use this function to build our model:

# construct the model
model = create_model()

Training the Model

Now that we have built the model, let's train it using the previously loaded dataset:

# use tensorboard to view metrics
tensorboard = TensorBoard(log_dir="logs")
# define early stopping to stop training after 5 epochs of not improving
early_stopping = EarlyStopping(mode="min", patience=5, restore_best_weights=True)

batch_size = 64
epochs = 100
# train the model using the training set and validating using validation set
model.fit(data["X_train"], data["y_train"], epochs=epochs, batch_size=batch_size, validation_data=(data["X_valid"], data["y_valid"]),
          callbacks=[tensorboard, early_stopping])

We defined two callbacks that will get executed after the end of each epoch:

  • The first is the tensorboard, we gonna use it to see how the model goes during the training in terms of loss and accuracy.
  • The second callback is early stopping, this will stop the training when the model stops improving, I've specified a patience of 5, which means it will stop training after 5 epochs of not improving, setting restore_best_weights to True will restore the optimal weights that was recorded during the training and assign it to the model weights.

Let's save this model:

# save the model to a file
model.save("results/model.h5")

Here is my output:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                (None, 256)               33024     
_________________________________________________________________
dropout (Dropout)            (None, 256)               0
_________________________________________________________________
dense_1 (Dense)              (None, 256)               65792
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0
_________________________________________________________________
dense_2 (Dense)              (None, 128)               32896
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_3 (Dense)              (None, 128)               16512
_________________________________________________________________
dropout_3 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_4 (Dense)              (None, 64)                8256
_________________________________________________________________
dropout_4 (Dropout)          (None, 64)                0
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 65
=================================================================
Total params: 156,545
Trainable params: 156,545
Non-trainable params: 0
_________________________________________________________________
Train on 54219 samples, validate on 6025 samples
Epoch 1/100
54219/54219 [==============================] - 8s 143us/sample - loss: 0.5514 - accuracy: 0.7651 - val_loss: 0.3807 - val_accuracy: 0.8508
Epoch 2/100
54219/54219 [==============================] - 5s 93us/sample - loss: 0.4159 - accuracy: 0.8326 - val_loss: 0.3464 - val_accuracy: 0.8536
Epoch 3/100
54219/54219 [==============================] - 5s 93us/sample - loss: 0.3860 - accuracy: 0.8466 - val_loss: 0.3112 - val_accuracy: 0.8744
<..SNIPPED..>
Epoch 16/100
54219/54219 [==============================] - 5s 96us/sample - loss: 0.2864 - accuracy: 0.8936 - val_loss: 0.2387 - val_accuracy: 0.9087
Epoch 17/100
54219/54219 [==============================] - 5s 95us/sample - loss: 0.2824 - accuracy: 0.8945 - val_loss: 0.2464 - val_accuracy: 0.9110
Epoch 18/100
54219/54219 [==============================] - 6s 103us/sample - loss: 0.2887 - accuracy: 0.8920 - val_loss: 0.2406 - val_accuracy: 0.9074
Epoch 19/100
54219/54219 [==============================] - 5s 95us/sample - loss: 0.2822 - accuracy: 0.8939 - val_loss: 0.2435 - val_accuracy: 0.9080
Epoch 20/100
54219/54219 [==============================] - 5s 96us/sample - loss: 0.2813 - accuracy: 0.8957 - val_loss: 0.2567 - val_accuracy: 0.8993
Epoch 21/100
54219/54219 [==============================] - 5s 89us/sample - loss: 0.2759 - accuracy: 0.8962 - val_loss: 0.2442 - val_accuracy: 0.9112

As you can see, the model training stopped at epoch 21, and reached about 0.2387 loss and almost 91% validation accuracy (this was on epoch 16).

Testing the Model

Since the model now is trained and the weights are optimal, let's test it using our testing set we created earlier:

# evaluating the model using the testing set
print(f"Evaluating the model using {len(data['X_test'])} samples...")
loss, accuracy = model.evaluate(data["X_test"], data["y_test"], verbose=0)
print(f"Loss: {loss:.4f}")
print(f"Accuracy: {accuracy*100:.2f}%")

Check this out:

Evaluating the model using 6694 samples...
Loss: 0.2405
Accuracy: 90.95%

Amazing, we've reached 91% accuracy on samples that the model never seen before! That is great!

If you open up tensorboard (using the command: tensorboard --logdir="logs"), you'll see loss and accuracy curves similar to this:

Binary crossentropy loss during trainingModel accuracy during the training

The blue curve is the validation set, whereas the orange is training set, you can see the loss is decreasing over time and the accuracy is increasing, that's exactly what we expected!

Testing the Model with your own Voice

I know this is the exciting part, I've made a script that records your voice until you stop speaking (you can speak in any language though) and save it to a file, then extract features from that audio and feed it to the model to retrieve results:

import librosa
import numpy as np

def extract_feature(file_name, **kwargs):
    """
    Extract feature from audio file `file_name`
        Features supported:
            - MFCC (mfcc)
            - Chroma (chroma)
            - MEL Spectrogram Frequency (mel)
            - Contrast (contrast)
            - Tonnetz (tonnetz)
        e.g:
        `features = extract_feature(path, mel=True, mfcc=True)`
    """
    mfcc = kwargs.get("mfcc")
    chroma = kwargs.get("chroma")
    mel = kwargs.get("mel")
    contrast = kwargs.get("contrast")
    tonnetz = kwargs.get("tonnetz")
    X, sample_rate = librosa.core.load(file_name)
    if chroma or contrast:
        stft = np.abs(librosa.stft(X))
    result = np.array([])
    if mfcc:
        mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=40).T, axis=0)
        result = np.hstack((result, mfccs))
    if chroma:
        chroma = np.mean(librosa.feature.chroma_stft(S=stft, sr=sample_rate).T,axis=0)
        result = np.hstack((result, chroma))
    if mel:
        mel = np.mean(librosa.feature.melspectrogram(X, sr=sample_rate).T,axis=0)
        result = np.hstack((result, mel))
    if contrast:
        contrast = np.mean(librosa.feature.spectral_contrast(S=stft, sr=sample_rate).T,axis=0)
        result = np.hstack((result, contrast))
    if tonnetz:
        tonnetz = np.mean(librosa.feature.tonnetz(y=librosa.effects.harmonic(X), sr=sample_rate).T,axis=0)
        result = np.hstack((result, tonnetz))
    return result

The above function is the function that is responsible for loading the audio file and extracting features from it, the below lines of code will use argparse module to parse an audio file path passed from the command line and make inference on it:

import argparse
parser = argparse.ArgumentParser(description="""Gender recognition script, this will load the model you trained, 
                                    and perform inference on a sample you provide (either using your voice or a file)""")
parser.add_argument("-f", "--file", help="The path to the file, preferred to be in WAV format")
args = parser.parse_args()
file = args.file
# construct the model
model = create_model()
# load the saved/trained weights
model.load_weights("results/model.h5")
if not file or not os.path.isfile(file):
    # if file not provided, or it doesn't exist, use your voice
    print("Please talk")
    # put the file name here
    file = "test.wav"
    # record the file (start talking)
    record_to_file(file)
# extract features and reshape it
features = extract_feature(file, mel=True).reshape(1, -1)
# predict the gender!
male_prob = model.predict(features)[0][0]
female_prob = 1 - male_prob
gender = "male" if male_prob > female_prob else "female"
# show the result!
print("Result:", gender)
print(f"Probabilities::: Male: {male_prob*100:.2f}%    Female: {female_prob*100:.2f}%")

This won't work if you execute it, as record_to_file() method isn't defined (you can check the full script code here), but this helps me explain the code.

We're using argparse module to parse the file path passed from command lines, if the file isn't passed (using --file or -f parameter), the script will start recording using your default microphone.

We then create the model and load the optimal weights we trained before, and then we extract the features of that audio file passed (or recorded) and we use model.predict() to get the resulting predictions, here is an example:

$ python test.py --file "test-samples/16-122828-0002.wav"

Output:

Result: female
Probabilities:     Male: 20.77%    Female: 79.23%

And indeed, that sample that was grabbed from LibriSpeech dataset is a female! Again, get the code here.

Related: How to Play and Record Audio in Python

Conclusion

Now you have a lot of options to further make the model more accurate, one is trying to come up with another model architecture, you can also use Convolution or recurrent nets and see the results! I can expect you reach more than 95% accuracy, if you did so, please don't hesitate to share it with us in the comments below!

You can also download the original dataset from Kaggle and use another feature extraction technique such as MFCC using the provided extract_feature() function, and then you can compare the results.

The whole tutorial material is located in this repository, check it out!

Finally, there is a similar tutorial in which you'll learn how to recognize emotions from speech as well, check it out!

Learn also: How to Convert Text to Speech in Python

Happy Learning ♥

View Full Code
Sharing is caring!



Read Also





Comment panel

   
Comment system is still in Beta, if you find any bug, please consider contacting us here.