How to Use Whisper: A Free Speech-to-Text AI Tool by OpenAI

Whisper is automatic speech recognition (ASR) system that can understand multiple languages. It has been trained on 680,000 hours of supervised data collected from the web.

Whisper is developed by OpenAI. It’s free and open source.

Speech processing is a critical component of many modern applications, from voice-activated assistants to automated customer service systems. This tool will make it easier than ever to transcribe and translate speeches, making them more accessible to a wider audience. OpenAI hopes that by open-sourcing their models and code, others will be able to build upon their work to create even more powerful applications.

Whisper can handle transcription in multiple languages, and it can also translate those languages into English.

In this tutorial, we’ll get started using Whisper in Google Colab. We’ll quickly install it, and then we’ll run it with one line to transcribe an mp3 file. We won’t go in-depth, and we want to just test it out to see what it can do.

Quick Video Demo

This is a short demo showing how we’ll use Whisper in this tutorial.

Sidenote: AI art tools are developing so fast it’s hard to keep up.

We set up a newsletter called tl;dr AI News.

In this newsletter we distill the information that’s most valuable to you into a quick read to save you time. We cover the latest news and tutorials in the AI art world on a daily basis, so that you can stay up-to-date with the latest developments.

Check tl;dr AI News

Using Whisper For Speech Recognition Using Google Colab

[powerkit_alert type=”info” dismissible=”false” multiline=”false”]Google Colab is a cloud-based service that allows users to write and execute code in a web browser. Essentially Google Colab is like Google Docs, but for coding in Python.

You can use Google Colab on any device, and you don’t have to download anything. For a quick beginner friendly intro, feel free to check out our tutorial on Google Colab to get comfortable with it.[/powerkit_alert]

If you don’t have a powerful computer or don’t have experience with Python, using Whisper on Google Colab will be much faster and hassle free. For example, on my computer (CPU I7-7700k/GPU 1660 SUPER) I’m transcribing 30s in a few minutes, whereas on Google Colab it’s a few seconds.

Open a Google Colab Notebook

First, we’ll need to open a Colab Notebook. To do that, you can just visit this link https://colab.research.google.com/#create=true and Google will generate a new Colab notebook for you.

Alternatively, you can go anywhere in your Google Drive > Right Click (in an empty space like you want to create a new file) > More > Google Colaboratory. A new tab will open with your new notebook. It’s called Untitled.ipynb but you can rename it anything you want.

Enable GPU

Next, we want to make sure our notebook is using a GPU. Google often allocates us a GPU by default, but not always.

To do this, in our Google Colab menu, go to Runtime > Change runtime type.

Next, a small window will pop up. Under Hardware accelerator there’s a dropdown. Make sure GPU is selected and click Save.

Install Whisper

Now we can install Whisper. (You can also check install instructions in the official Github repository).

To install it, just paste the following lines in a cell. To run the commands, click the play button at the left of the cell or press Ctrl + Enter. The install process should take 1-2 minutes.

!pip install git+https://github.com/openai/whisper.git 
!sudo apt update && sudo apt install ffmpeg

[powerkit_alert type=”info” dismissible=”false” multiline=”false”]
Note: We’re prefixing every command with ! because that’s how Google Colabs works when using shell scripts instead of Python. If you’re using Whisper on your computer, in a terminal, then don’t use the ! at the beginning of the line.
[/powerkit_alert]

Upload an Audio File

Now, we can upload a file to transcribe it. To do this, open the File Browser at the left of the notebook by pressing the folder icon.

Now you can press the upload file button at the top of the file browser, or just drag and drop a file from your computer and wait for it to finish uploading.

Run Whisper to Transcribe Speech to Text

Next, we can simply run Whisper to transcribe the audio file using the following command. If this is the first time you’re running Whisper, it will first download some dependencies.

!whisper "Rick Astley - Never Gonna Give You Up Official Music Video.mp3"

In less than a minute, it should start transcribing.

When it’s finished, you can find the transcription files in the same directory in the file browser:

Using Whisper Models

Whisper comes with multiple models. You can read more about Whisper’s models here.

By default, it uses the small model. It’s faster but not as accurate as a larger model. For example, let’s use the medium model.

We can do this by running the command:

!whisper AUDI_FILE --model medium

In my case:

!whisper "Rick Astley - Never Gonna Give You Up Official Music Video.mp3" --model medium

The result is more accurate when using the medium model than the small one.

Whisper Command-Line Options

You can check out all the options you can use in the command-line for Whisper by running !whisper -h in Google Colab:

Conclusion

In this tutorial, we covered the basic usage of Whisper by running it via the command-line in Google Colab. This tutorial was meant for us to just to get started and see how OpenAI’s Whisper performs.

You can easily use Whisper from the command-line or in Python, as you’ve probably seen from the Github repository. We’ll most likely see some amazing apps pop up that use Whisper under the hood in the near future.

Useful Resources & Acknowledgements

• The GitHub Repository for Whisper – https://github.com/openai/whisper. It has useful information on Whisper, as well as some nice examples of using Whisper from the command-line, or in Python.
• OpenAI Whisper – MultiLingual AI Speech Recognition Live App Tutorial – https://www.youtube.com/watch?v=ywIyc8l1K1Q. A very useful intro to Whisper, as well as a great demo on how to use it with a simple Web UI using Gradio.
• Hacker News Thread – https://news.ycombinator.com/item?id=32927360. You can find some great insights in the comments.