- OpenAI has launched features that let ChatGPT "hear" and "see."
- ChatGPT users can upload images and ask questions based on the pictures.
OpenAI has launched some impressive new features for its viral chatbot, ChatGPT.
The company announced in late September that the AI-powered bot was getting a new voice and some "eyes."
Users signed up for the subscription model, ChatGPT Plus will be able to start interacting "in a back-and-forth conversation" using the voice feature, the company said.
Plus users will also be able to upload images to the chatbot and ask questions based on the pictures. It's unclear when the image and voice features will roll out to basic users.
ChatGPT's new image feature is easy enough to use.
On the web version, a picture icon will appear on the left-hand side of the text bar which users can click on to upload a picture to the chatbot. If users have the ChatGPT mobile app they can also take a photo on their smartphones and circle whatever they want the chatbot to zero in on.
Once uploaded users can ask questions about the picture.
For example, I sent ChatGPT a picture of some — slightly random —ingredients and asked for recipe suggestions. The bot came back with four suggestions within minutes, all of which appeared at least edible at first glance despite including a few cupboard staples not featured in the image.
The feature has other practical uses. For example, the bot was able to understand my imperfect handwriting and type up some hand-written notes. It also did a good job at identifying objects in the wild and offered some advice to bring my dying plant back to life.
But the new feature can be used for more than kitchen inspiration and horticultural advice.
AI startup founder Mckay Wrigley shared another example of the new image feature. In a video posted on X, Wrigley showed how ChatGPT could write code from a whiteboarding session.
Wrigley uploaded a picture of a whiteboard that depicted website instructions in plain English and asked the bot to generate the code. The model turned the instructions into working code.
While the feature has obvious uses, it also raises privacy questions. For example, Wired's Reece Rogers reminded users to avoid uploading personal, sensitive photos to ChatGPT when trying out the image feature.
The bot has also got some new guardrails. OpenAI limited the chatbot from answering questions that identify humans.
When I showed it a picture of Mexican artist Frida Kahlo it refused to identify the image. However, after a little probing, it eventually did so after citing a label in the top right-hand corner.
"I am programmed to avoid identifying real people based on images for privacy reasons," ChatGPT told me. "However, the artwork you provided is labeled."