Magic Wands Don’t Exist: The Jagged Frontier of AI

Author(s): Pawel Rzeszucinski, PhD

Originally published on Towards AI.

The onset of generative AI (genAI) and the spectacular capabilities of large language models (LLMs) has opened the door to a new era of technological possibilities. I agree — some things that these tools do are truly jaw-dropping. However, I started to observe what I had seen in the past with previous waves of new technologies — some managers, executives, and board members believe that genAI is a magic wand capable of solving all organizational problems. While we are experiencing a tremendous leap forward and it’s extremely exciting to witness these changes in person, the applicability of these tools is far from universal.

Imagine the introduction of calculators. They revolutionized how we perform calculations, making complex computations faster and more accessible. Yet, they did not fundamentally change the way, for lack of a better example, buildings are constructed. Calculators assist in specific tasks but do not directly impact the construction process. Similarly, genAI can significantly aid in certain areas but remains out of reach in others.

The Concept of the Jagged Frontier of AI

The term “jagged frontier of AI” aptly describes the uneven capabilities of AI. Coined by researchers at Harvard Business School, it highlights how nondeterministic AI performance can be at times: excelling at tasks we think should not be within their reach, yet failing on tasks that human common sense suggests should be straightforward for the AI​​.

A great example was given by Ethan Mollick in his blog post on the subject:

“[Consider asking AI to] write a sonnet and an exactly 50 word poem — are actually on different sides of the wall. The AI is great at the sonnet, but, because of how it conceptualizes the world in tokens, rather than words, it consistently produces poems of more or less than 50 words.”

This inconsistency is the hallmark of the jagged frontier, where AI’s capabilities are marked by peaks of excellence and valleys of inadequacy.

The Importance of Hands-On Experimentation

To navigate the jagged frontier, hands-on experimentation with LLMs is crucial. Since there is no comprehensive map of AI capabilities, only exploration can reveal its true colors. The cliché that practice makes perfect could not be more accurate in this situation. By engaging directly with these models, users can better understand their strengths and weaknesses.

Experimentation is not just about running models but involves iterating on tasks, understanding the nuances of AI responses, and identifying patterns in performance. This process allows practitioners to discover practical applications of LLMs, showcasing where they can truly add value and where they might fall short. For example, while an LLM might excel in generating coherent text for content creation, it might not perform as well in tasks requiring precise numerical predictions without substantial training and fine-tuning. Only trial and error and give us such insights.

Analyzing the AI Techniques Heat Map

While there is no universal map of AI capabilities, some organizations attempt to chart where AI excels and where it falls short. A very useful recent resource is a heat map of AI capabilities produced by Gartner, which provides insights into the suitability of different AI techniques for various application types. This heat map helps in understanding the areas where specific AI models, including generative AI, perform well and where they may not be the best fit.

Generative Models

• High Suitability (H): generative models excel in creative and interpretative tasks. For example, they can generate realistic text/images, engage in natural language conversations, and uncover patterns in large datasets.
• Medium Suitability (M): while generative models perform well in these areas, they are not the best. They can classify data, provide recommendations, and automate processes to a reasonable extent but may not always be the most efficient/accurate choice.
• Low Suitability (L): genAI can struggle with tasks requiring precise and structured outputs, such as making accurate predictions, strategic planning, and real-time decision-making.

Non-Generative Machine Learning

These include countless algorithms from large families of approaches that were developed in the past decades e.g. Support Vector Machines, k-nearest Neighbors, Decision Trees, Logistic Regression

• H: non-generative AI excel in making accurate predictions and detecting anomalies, crucial for systems requiring high reliability.
• M: these models are effective in classifying data and making informed decisions but might need supplementary techniques to achieve optimal results.
• L: non-generative models lack the creative and adaptive capabilities required for generating content and engaging in natural conversations.

Optimization and Simulation

Can include methods like Linear Programming, Genetic Algorithms, and Markov Decision Processes.

• H: ideal for strategic planning and decision-making processes, providing robust frameworks for autonomous operations.
• M: these techniques are beneficial in enhancing the efficiency of recommendations and automation but may require integration with other AI methods for full effectiveness.
• L: the least effective in predicting outcomes and creating new content, as they rely on predefined parameters and simulations.

Rules/Heuristics and Graphs

We’re talking methods like Expert Systems, Decision Rules, Graph Databases, Knowledge Graphs

• H: rules and graphs are excellent for structured tasks like knowledge extraction and anomaly detection, where predefined rules can be highly effective.
• M: these techniques can aid decision-making and automation but may not adapt well to dynamic, unstructured data.
• L: their rigidity makes them less suitable for dynamic and creative tasks, where flexibility and learning from data are crucial.

Conclusion

Generative AI and LLMs represent a groundbreaking advancement in technology, poised to impact many areas of society. However, they are not a panacea for all problems. The jagged frontier of AI underscores the importance of understanding where these models excel and where they fall short. By leveraging the strengths of different AI techniques and acknowledging their limitations, we can harness their full potential and navigate the complex landscape of AI innovation. Since there is no cookbook or one-size-fits-all solution, an attitude of continuous experimentation is the best way to stay on top of the latest developments. And these are changing fast — remember, the AI we see today is the weakest AI we will ever see.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI