Methods: Assessing AI speech-to-text accuracy

We compared three leading speech recognition application programming interfaces (APIs) – Amazon Transcribe, Google Cloud Speech-to-Text, and IBM Watson Speech to Text – to human transcriptionists on a number of criteria:

• Accuracy: The rate at which the solution makes mistakes in transcribing uttered words, measured as the Word Error Rate (WER [Transcript, Reference] = [Substitutions + Deletions + Insertions] / Words in Reference).
• First-hypothesis latency: The time between the utterance of a word and the output of text.
• Stable-hypothesis latency: The time between the utterance of a word and the output of correct text.
• Cost: The fee for use of the associated service.

To evaluate automatic transcription performance, we fed each API over 1,500 sample phrases from a test set made available by Texas Instruments and the Massachusetts Institute of Technology (TIMIT). We compared the results to the reference transcriptions included with the test set and measured latency. Ultimately, we decided against adjusting transcription timings for round-trip time (RTT) since RTT made up a relatively small portion of overall latency in every case.

To establish a baseline for human transcription performance, we drew and generalized results from multiple academic sources.

Conclusion: AI speech-to-text accuracy is comparable to humans

Each API achieved a level of accuracy and latency suitable for real-time captioning. The latency of Amazon’s API was a bit higher than IBM’s and Google’s engines, but the three are comparable when it comes to accuracy and cost. We also tested each engine for noise resilience (transcription accuracy in the presence of noise) and found that audio equipment quality, microphone placement, and other factors are essential for acceptable performance.

What does all this mean in practical terms? These APIs are ready for use in live event scenarios – but how can organizations actually leverage them?

This would require developing:

• An automatic speech recognition edge agent to capture and stream audio data to the cloud
• A digital signage platform and agent to receive, render, and display transcriptions
• A Web portal or mobile application to accommodate users who are seated far from in-room monitors or who have visual impairments or vision loss

And so on. The other, less burdensome option is to use an off-the-shelf dedicated automatic transcription device.

Get the best of automatic transcription technology today

Powered by Google’s advanced speech recognition technology, LiveScrypt features professional audio inputs (XLR, TRS) so you can capture crystal-clear audio that’s conducive to high AI speech-to-text accuracy. LiveScrypt also includes HDMI and SDI inputs to capture embedded audio, a built-in screen for configuration, and a QR code system for easy streaming, simplifying setup and making for fewer points of failure.

1. Hire a transcriptionist

In the past, hiring a professional transcriptionist was the only option for captioning in real time. This approach involves hiring an individual who listens to proceedings (on site or remotely) and transcribes on the fly.

There are advantages to human transcriptionists. A human can listen closer to someone who is speaking softly and still be able to discern what they mean, while an AI-based system may not be able to reach the same level of precision. Some medical or legal events may require the transcriptionist to carry certain professional certifications. For example, a certified health documentation specialist credential indicates the transcriptionist understands clinical health terminology and can apply that knowledge to correct commonly confused medical terms to ensure an accurate medical record. In other instances, an experienced professional may be able to parse industry-specific terminology or slang that some automatic transcription solutions may struggle with.

But human transcriptionists are also highly variable in quality and reliability. Someone transcribing a single 20-minute speech could be highly accurate, but that accuracy rate could change if you’re asking them to transcribe four hours of lectures. Similarly, that transcriptionist could be taken out of commission by inclement weather, unexpected illness or personal emergencies. Finally, not all transcriptionists carry the equipment needed to share captions in real time. In addition to booking someone with gear that can connect to AV equipment, it will likely be on you to find a way to ensure transcripts can be shared with your audience in real-time.

Price:

Highly variable, with prices ranging from $90 to $180 an hour, with experienced or credentialed transcriptionists coming in at the higher end of the scale. Transcriptionists may bill you at an overtime rate for longer events, further increasing costs.

Bottom line:

While there are definitely places where a human transcriptionist is required, the high price point can be prohibitive. High demand for real-time transcription services only continues to drive that price up, and it may mean a professional transcriptionist isn’t available on the day or time of your event.

2. Buy a hardware solution

A relatively new entry to the speech-recognition market, hardware solutions provide a simplified real-time transcription option. A hardware device includes a way to capture audio live, convert that speech to text, and share that transcription with guests. Typically, these devices connect directly to a local audio source to ensure a clean audio feed and also include some kind of standardized video output to share the transcription on a monitor in real time.

A dedicated hardware solution also removes possible points of failure present in an AI-transcription solution that relies on a computer or mobile device. A dedicated hardware transcriber will not suffer from a blue screen of death, receive unexpected text messages during an important presentation, or require the same ongoing care and maintenance that other solutions might.

A purpose-built hardware solution will also include extra features depending on the hardware developer. LiveScrypt, Epiphan Video’s own dedicated automatic real-time transcription solution, supports transcription in over 20 languages and dialects, and includes additional features like profanity filters and text scaling to ensure visibility on connected monitors.

These solutions have a higher initial investment cost, expressed as the cost of hardware itself. This cost may be steep for some, but organizations and people in regular need of transcription break even quickly.

Examples include a college or university intending to caption several lectures a day, or a convention planner who wants to transcribe dozens of speakers at each event they stage. Even after the cost of hardware is accounted for, the per-hour cost of transcription remains far below the cost of professional transcriptionists for those organizations and groups.

Price:

Variable. People and organizations buying into a hardware solution will need to pay for the hardware itself as well as the ongoing subscription costs of transcription. However, the cost of these services still remain far below the cost of hiring a transcriptionist , and value-for-money increases the more the hardware is used.

3. Build a cloud-based transcription solution

Services like Google Speech-to-Text, Amazon Transcribe, and IBM Watson Speech to Text all use very similar technology to convert speech into text. In brief, automatic transcription services take a digital audio signal, breaks that signal into smaller segments of sound, and compares those segments (also called phonemes) to an existing database of language sounds. When a match is found, the service then determines what word those phonemes are constructing, and returns a result as text.

The process typically requires a lot of computing power, which is why these services use cloud computing to deliver quick results. The accuracy of AI transcription services is often comparable to human typists, with the gap between the two narrowing daily.Consistency and lack of downtime are a couple of the clear benefits of using a cloud-based automatic transcription service. While it’s unreasonable to expect a human to transcribe hours of speech without a break, that kind of task is well-suited for AI-powered speech recognition services. Modern AI-based transcription services can yield word error rates low enough for real-time event captioning.

The cost of these services is also significantly lower than working with a professional transcriptionist, making it attractive for longer events with many hours of speech to be transcribed and for organizations staging multiple live events throughout a year.

The low price also means it’s possible to offer real-time captioning from end to end. A conference organizer using a professional transcriptionist may be forced to limit captioning to one or two keynote speeches for budgetary reasons. But for a fraction of that price, an automatic transcription service could caption the entire event – from the keynote presentation to the final word.

But cloud services also require a degree of computer competency that is beyond many organizations. These services provide a way for digital audio to be converted to text, but a coder is required to develop a program which interacts with that cloud service. That process takes time, and will require testing, patching, and updating as problems emerge.

You will also need some form of local console that can convert an analog audio signal to a digital one, send that signal to the cloud, and receive your transcription. This can be some kind of personal computer, though the general-purpose nature of these presents a few challenges. This includes unplanned system updates and unknowing bystanders interfering with the transcribing to charge their phones.

Most personal computers also do not natively have a way to receive professional audio feeds, such as through XLR connections. It’s possible to add that capacity using expansion cards or an external sound card, but this adds complexity to the job. Any computer enthusiast will tell you that care and maintenance of a computer – even a single-purpose one – is an ongoing task.

Price:

Among the most affordable options, prices range from  96 an hour for Google Text-to-Speech to $1.44 an hour for Amazon Transcribe. Price can also come down with volume. IBM, for instance, offers discounted rates for users who need to transcribe over 250,000 minutes, 500,000 minutes, or one million minutes of speech.

You will also need a computer to send audio to the cloud, receive the transcription, and share it with your audience. While some organizations may have that gear handy, building a PC for this purpose increases cost.

Bottom line:

Low cost makes this an attractive option, but cloud services still rely on you to find a way to gather audio and share the transcription live. The added complications associated with sourcing a local console capable of doing this may make this option unappealing for people and organizations looking for a simple way to add live transcription to an event.

4. Find a speech-to-text app

While phone-based speech-recognition apps have many effective uses, they’re limited by the hardware they’re tied to. Smartphones and tablets are limited by storage and processing capacity, while microphone audio on a smart device can be variable in quality. This may mean their best applications are typically in one-on-one interviews, or small meetings rather than a large lecture, in a hall where the speaker may be far from the transcribing phone.

App-based solutions are also dependent on the app developer to add functionality and resolve issues. More popular apps will be responsive to user needs with developers rolling out regular updates, but an app developed by an independent firm or individual user could see updates stopped, or be abandoned completely.

Users will also need a way to share the transcript. Smartphones and tablets capable of using these apps are not typically designed with audiences in mind; sending the transcript to a large screen will require additional setup. Plus, solutions that rely on a smartphone are vulnerable to unexpected phone calls, instant messages, and software updates.