GPT-4o, where “o” stands for omni, surpasses GPT-4 Turbo in several ways. The disruptive innovation can respond to audio prompts in real time, mimicking human conversation speeds.

It was also built with superior understanding of languages other than English, along with audio and visual data, having double the speed and half the cost of GPT-4 Turbo within the API. The model brings increased accessibility through the free tier of ChatGPT and higher message limits for Plus users.

Unlike previous models requiring separate processes for audio, GPT-4o removes this information loss, allowing it to grasp issues like tone, background noise, and multiple speakers.

The model can generate a wider range of outputs, including laughter, singing, and emotional expression.

OpenAI built GPT-4o with great capabilities like creating stories based on visual prompts, generating characters from descriptions, and even transforming text into unique fonts.

This shows the model’s ability to process information across different mediums, and also leverage that understanding for creative tasks.

OpenAI emphasizes safety measures implemented throughout GPT-4o’s development. These include data filtering and post-training refinements. Additionally, voice outputs are subject to safety protocols.

OpenAI acknowledges limitations in all modalities and welcomes feedback to further refine the model. The company is releasing GPT-4o’s capabilities iteratively.

Text and image functionalities are now available in ChatGPT. Audio capabilities, including voice mode with GPT-4o, will roll out in the coming weeks for Plus users.

Developers can access GPT-4o’s text and vision capabilities in the API. Access to audio and video features will be granted to a select group of partners initially.

ChatGPT, which was released just a year ago, quickly gained traction, accumulating an estimated 100 million monthly users within its first two months. Altman highlighted that over two million developers are now utilizing the platform, including 92% of Fortune 500 companies.

One of the key announcements at the event was the introduction of GPT-4 Turbo, the next-generation model built with a 128k context window, enabling it to comprehend the equivalent of more than 300 pages of text in a single prompt. This advanced version of the model is not only more capable but also cost-effective, with prices reduced to a third for input tokens and half for output tokens compared to GPT-4. GPT-4 Turbo, having knowledge up to April 2023, is available for preview to paying developers and is set to be released in the coming weeks.

Additionally, OpenAI unveiled the Assistants API, facilitating developers in building assistive AI applications with specific goals. The API includes features such as Code Interpreter, Retrieval, and function calling, streamlining the process of creating high-quality AI apps. OpenAI introduced enhancements in function calling accuracy, improved instruction following, and JSON mode for precise output generation.

GPT-4 Turbo’s capabilities were expanded to include vision support, enabling it to process images and perform tasks such as generating captions and analyzing real-world images. OpenAI has also integrated DALL·E 3, a technology allowing developers to generate images and designs programmatically. OpenAI also introduced a text-to-speech API, enabling the generation of high-quality speech from text, providing six preset voices to choose from.

To empower developers further, OpenAI announced an experimental access program for GPT-4 fine-tuning, aiming to offer a tailored experience for specific domains. Additionally, a Custom Models program was introduced, allowing select organizations to work closely with OpenAI researchers for highly customized solutions.

The event was graced by a surprise appearance from Microsoft’s CEO, Satya Nadella, emphasizing the partnership between the two tech giants. Nadella praised OpenAI’s zeal for safety and empowerment, to which Altman responded by stressing the importance of gradual and careful deployment in AI.