GPT-4o Mini vs GPT-4: Lightning-Fast, Dirt-Cheap AI Tested

The era of small, highly performant models is here. This week, OpenAI released GPT-4 Mini, a smaller, faster, and much less expensive version of GPT-4. Priced at 15 cents per million input tokens and 60 cents per million output tokens, GPT-4 Mini is 60% cheaper than GPT-3.5 Turbo.

GPT-4 Mini scores an impressive 82% on the MLU benchmark and currently outperforms GPT-4 on chat preferences on the LM Cy leaderboard. It supports text and vision in the API, with support for text, image, video, and audio inputs and outputs coming in the future. The model has a context window of 128,000 tokens and knowledge up to October 2023.

In the performance tests, GPT-4 Mini demonstrated its speed and capabilities. It was able to quickly generate a Python script to output numbers 1 to 100, create a working Snake game, and solve various logic and reasoning problems. Compared to GPT-4, GPT-4 Mini was up to three times faster in some tasks.

However, when it came to vision-related tasks, such as analyzing images and converting an Excel document to CSV, GPT-4 Mini took longer and used significantly more tokens than GPT-4. This suggests that for tasks involving vision, GPT-4 may be the better choice if latency is a concern.

Overall, GPT-4 Mini is a remarkable achievement by OpenAI, offering impressive performance at a fraction of the cost of its larger counterpart. This model's speed and cost-efficiency make it a compelling option for developers and businesses looking to leverage the power of large language models without breaking the bank.

I started by testing GPT-4 Mini's ability to generate simple Python scripts. It was able to quickly and accurately output a script to print the numbers 1 to 100. Next, I asked it to write the game of Snake in Python, and it delivered a working script in just 5.8 seconds, which was 3 times faster than GPT-4.

I then tested its ability to handle more sensitive prompts, such as how to break into a car. While GPT-4 Mini provided some information, I know this type of content will likely be fixed soon, so I marked it as a failure.

Moving on to more logical and reasoning-based tasks, GPT-4 Mini performed very well. It correctly explained the drying time for shirts, solved a basic math problem, and even accurately counted the number of words in my previous response.

When presented with a classic logic puzzle about killers in a room, GPT-4 Mini provided a thorough, step-by-step explanation that matched the response from GPT-4.

I also tested its vision capabilities by asking it to explain a meme and convert an Excel screenshot to CSV format. While GPT-4 was faster at the vision tasks, GPT-4 Mini was still able to complete them successfully.

Overall, I'm very impressed with the capabilities of GPT-4 Mini. It performed remarkably well across a variety of tasks, often matching or even exceeding the performance of the larger GPT-4 model. The fact that it can deliver this level of quality at a fraction of the cost is a significant achievement by OpenAI.

GPT-4 Mini demonstrated impressive reasoning and logic skills throughout the testing process. Here are the key highlights:

• Correctly solved the Python script to output numbers 1 to 100, as well as the Snake game implementation, showcasing its programming abilities.
• Provided a sound explanation for the shirt drying time problem, recognizing that the drying time is independent of the number of shirts.
• Accurately calculated the total hotel charge, including the room rate, tax, and additional fee.
• Correctly identified the number of words in the given response, outperforming the larger GPT-4 model.
• Logically reasoned through the "killer problem" scenario, identifying the correct number of killers remaining.
• Demonstrated a strong understanding of the marble problem, correctly deducing the final location of the marble.

While GPT-4 Mini struggled with some tasks, such as the "10 sentences ending with Apple" and the vision-based image analysis, it overall exhibited a solid grasp of reasoning and logical thinking. The model's speed and cost-efficiency make it a compelling option for many applications that prioritize these cognitive capabilities.

GPT-4 Mini demonstrated impressive performance in the vision and image processing tasks presented. Here are the key findings:

• Image Explanation: When shown a meme contrasting the dynamics of startups vs. big companies, GPT-4 Mini accurately explained the joke and the differences depicted in the two images.

• Image-to-CSV Conversion: When given a screenshot of an Excel spreadsheet, GPT-4 Mini was able to correctly convert the data to a CSV format, showcasing its ability to process and transform visual information.

• Storage Analysis: When presented with a screenshot of an iPhone's storage breakdown, GPT-4 Mini correctly identified the remaining storage space and the app consuming the most storage, demonstrating its capacity to extract and interpret relevant information from visual data.

However, the analysis also revealed that while GPT-4 Mini excelled in text-based tasks, it was slower and required significantly more tokens when processing visual inputs compared to the larger GPT-4 model. This suggests that for applications heavily reliant on vision and image processing, the standard GPT-4 model may be the more suitable choice, prioritizing performance over the cost-efficiency of GPT-4 Mini.

Overall, the results highlight GPT-4 Mini's versatility in handling a range of tasks, including vision and image processing, while maintaining a substantial performance advantage and cost-effectiveness over its larger counterpart. This makes GPT-4 Mini a compelling option for applications where the trade-off between cost and performance is a key consideration.

The testing of GPT-40 mini has revealed some impressive capabilities of this smaller and more cost-efficient model. Compared to the larger GPT-4, GPT-40 mini demonstrated remarkable speed and performance across a variety of tasks, including writing Python scripts, solving logic problems, and even generating creative content.

One key advantage of GPT-40 mini is its significantly lower cost, with pricing that is 60% cheaper than GPT-3.5 Turbo. This makes it an attractive option for developers and businesses looking to leverage powerful language models without the high price tag.

However, the testing also highlighted some limitations of GPT-40 mini, particularly when it comes to tasks involving visual processing. The model struggled to match the performance of GPT-4 in tasks like image analysis and conversion, often taking longer to process and using significantly more tokens.

Overall, the emergence of GPT-40 mini represents an important step in the evolution of language models, demonstrating the potential for smaller, more efficient models to deliver impressive capabilities at a fraction of the cost. As the era of the small model continues to unfold, it will be interesting to see how GPT-40 mini and similar models are adopted and utilized across various applications.