A Google engineer thought he'd have to stop coding because of a health condition that makes it incredibly painful to type. Now, he programs with his voice and helps others learn to do it, too.
- Google senior staff software engineer James Stout programs with his voice because of a health condition he has called Ehlers-Danlos syndrome, which makes it painful for him to type and makes him more prone to muscular injuries.
- Stout started voice programming while working at Google's self-driving unit Waymo, when his condition took a turn for the worse.
- Stout had already started experimenting with using his voice to use a computer, and his manager allowed him to do his programming with voice as well.
- Now, Stout writes a blog called Hands-Free Coding, which shows how to program without hands.
- Six months ago, Stout joined Google's Accessibility team to work on voice access features, as well as other features to help people with disabilities.
- Read more about the tech industry's shift to the voice platform in BI Prime's special report.
When James Stout, senior staff software engineer at Google, writes code, he's not typing away at his keyboard. Instead, he speaks into a microphone, saying commands such as "studly test rule, leap studly mapping, reap tab key."
At his desk, there's a monitor, an eye tracker, a microphone, and a keyboard with two concave sections. Below his desk, there's a foot pedal he steps on, which makes numbers appear on the screen to mark each line.
Stout, who has been working at Google for 10 years, has this setup because it's difficult for him to type. He has Ehlers-Danlos syndrome, which affects connective tissues. This rare condition affects between 1 in 2,500 to 1 in 5,000 people, and makes people more prone to getting injured, including muscular injuries and repetitive strain injury.
Today, Stout works as an engineer on the accessibility team to help users more easily use Google's products. For example, Stout is working on features that help people use their voice to navigate apps in a natural way. And on the side, Stout writes a blog called Hands-Free Coding, where he writes about how to to make voice coding work.
"I solved my problems, and now it's time to go help other people who are where I was," Stout told Business Insider. "I got these comments on this blog from people saying it had saved their careers. If this is what a weekend project in the space of accessibility can do, what would happen if I can devote my full-time job to accessibility?"
As products like Amazon Alexa and Apple Siri catch on with consumers, providing a convenient, hands-free way to hear the latest sports scores or order groceries, Stout's experience shows the potential for voice-based computing to improve lives in more fundamental ways. Stout's work creating his own voice-based system saved his career and could open the door to even broader innovations to help people overcome physical limitations.
Watch Stout program with his voice:
Still, there was a time when Stout feared he would lose his long-term career as a software engineer because it was hard for him to type. However, he says, throughout all this, Google and his co-workers have been "nothing but supportive," and his manager allowed him to start programming with his voice.
"I was mostly worried it would be too slow to be the engineer I always wanted to be," Stout said. "I could either lose my job, I wouldn't be able to get enough done or that I wouldn't be able to realize my full potential."
'As a software engineer, this was terrifying'
Stout started at Google on the infrastructure team, then joined Waymo, the company's self-driving car project. But while he was there, his condition took a turn for the worse. He was already accustomed to his fingers getting sore while he coded, but now a sharp pain would shoot up his arm simply by tapping a key.
"As a software engineer, this was terrifying," Stout said.
Although Stout had EDS his entire life, it did not become a major problem until he started typing regularly. When he was younger, he noticed he had loose joints and could stretch his finger back far. But in college he noticed that he got injured more often, dislocating a shoulder and knee cap, for example.
The condition also affects any other activities in which Stout uses his hands, such as cooking, driving, using a cell phone, and biking. Now, Stout uses an ElliptiGO, or an elliptical bicycle, to get to work.
Stout worried that he would lose his career as a software engineer because he was no longer able to type. He visited several doctors, and one even told him that he should consider a career change.
"I chose computer science because it was my passion," Stout said. "I loved what I did. I loved being a software engineer. That was pretty devastating. At some level, I had to ignore his advice. You hear those words ringing in your head as things get worse and worse. It creates a doubt that's hard to shake until you've actually made it out on the other side."
'Why am I on my last leg here?'
Despite what the doctor said, Stout started experimenting on the side with using voice to browse the Internet and write code. Stout says he was motivated by a talk by the developer Tavis Rudd, who gave a talk about coding in Python with his voice. Rudd even said that he codes faster when he uses a combination of his voice and his hands.
"I wondered, why am I on my last leg here?" Stout said. "It helped a lot to see a video of someone who made it on the other side. There's a difference between seeing someone do that versus doing it yourself."
Eventually, Stout asked his manager if he could spend one day a week using voice to type instead. In response, Stout says his manager said, "You can take time off and take as many days as you need to do this. This is your health, this matters more than anything else." Then, one day a week turned into five.
"I am extremely thankful to him for getting this," Stout said. "To have my manager give me that sense of security and say, 'do whatever you need to do' was huge."
When working with his computer, Stout uses voice recognition software called Dragon, which can detect speech and turn it into text. Still, it has limitations. Stout says that while it's good for writing emails or typing in Microsoft Word, it's not built for writing code.
Initially, Stout says, getting used to coding with his voice was slow, but he started adding accessibility features and writing his own software that would help him program with his voice. For example, he added a feature that allows him to jump to different points on the page.
He also created special commands that would make coding easier. When writing programs, developers often use left and right parentheses. But saying "left parentheses" or "right parentheses" out loud is mouthful. Instead, Stout says the words "leap" and "reap."
"There were a lot of periods where I would get really frustrated at the computer," Stout said. "I would just have to take a step back and say, relax, find this peaceful place in your mind."
It took him about three months to get his voice coding capabilities to about half his original coding speed. That progress was enough to give him confidence that programming with his voice could actually work.
"For me, it was a transformative moment because it was a realization that these technical improvements could have a life-sized impact on me," Stout said. "It could be the difference between losing my career and keeping my career. Naturally, I thought, I know there's a lot of people out there who are in that really scared state. How can I help them out?"
'What I wanted to do was something radical and dramatic'
Meanwhile, Stout was blogging on the side about using voice programming. Soon, readers started commenting about how his blog saved their careers.
For example, a reader told him that his 12-year-old sister has a condition that would prevent her from typing. Stout video chatted with this reader, who said he wanted to learn how to use voice to type so that he could teach his sister how to do the same.
When Stout saw the impact of his blog, he thought, why not focus his career on this? As a result, six months ago, he decided to leave Waymo and join Google's Accessibility team.
There was another reason Stout was excited about joining this team: Google's expertise in machine learning, a type of artificial intelligence programming that allows computers to learn on their own. Although he has more experience with voice access, he sees potential for using machine learning to help people with other disabilities.
"Machine learning offers us the ability to have a deeper understanding of what's going on," Stout said. "That's what I'm excited about at Google. I felt limited to what I wanted to do on weekends. What I wanted to do was something radical and dramatic."
Outside of work, he's trying to make as many of Dragon's features available in open source, or making it free for anyone to use, download, or modify it.
Since Dragon is proprietary software, the online community is trying to replace pieces of Dragon with open source software. That way, anyone can download and use it, and people can easily improve it.
"I plan to continue to share what I do, continue to blog about the useful things I discover," Stout said. "I hope that the work I do at Google, not designed for coders but for general audience will benefit everyone including coders. I hope some of the ambitious ideas I'm excited about applying to machine learning can also inspire work in the coding space and inspire ideas there."
'That's what motivates me to address this'
Today, Stout says there are certain tasks he can perform faster with his voice than by typing, such as writing emails and coding. However, visual tasks like drawing a picture in PhotoShop or organizing a PowerPoint presentation are more difficult.
Another aspect that's challenging, Stout says, is that it can be difficult to plan what he's working on and use his voice at the same time because they use the same part of the brain. In comparison, people can often easily multi-task when typing, as it's easier to think about one thing and to type something different at the same time.
Stout hopes that in the future, he can design a voice system that minimizes distraction.
"That's something that is very subtle and hard to really master until you've been through this yourself and you really had to struggle with," Stout said. "It think it is possible to design a system that does not have this overhead."
Still, Stout says, in general, he has become more productive. Not only does he code faster, using voice also pushes him to rethink every aspect of his job.
"It forced me to be conscious of every little keystroke I take," Stout said. "I question, when am I being inefficient? What could I have done more quickly? How could I reduce how much stuff I needed to do?"
Finally, he no longer fears for his long-term career, but he still sees places where there's potential to improve.
"I figured out how to organize myself and how to do my best but I still see so many places where I can be that much faster or that much less cognitive load," Stout said. "I still see places where I'm held back, and that's what motivates me to address this."
- Read more about the platform shift to voice computing:
- These 10 'voice-first' startups are building apps for smart speakers, cars and watches that will completely change how we use computers
- Hackers are inventing clever ways to trick the microphones in smart speakers, and it's opening a 'new world of dangers'
- In-house venture funds at Amazon and Google are leading the charge into the voice-first revolution and pouring millions of dollars into startups
- 2 Amazon execs discuss plans to make a smarter Alexa that can anticipate your needs and stay ahead of Google in the voice wars