I'm always trying to find the limits of language models. What are they NOT capable of? It is harder than you might imagine to find tasks! As such, I learned a lot when I reached out to Malick Doucoure who is researching West African History and Politics, after he raised a question - could AI truly understand the nuances and references in his historical fiction manuscript about the hypothetical colonization of Europe by the Malian Empire? We decided to put AI to the test.
Malick sent me the first three chapters, which I fed directly into Claude Opus, the AI from Anthropic with a single prompt.
PROMPT: "We'd like to test three things:
- Nuanced Interpretation: In scenes with intricate character interactions that reflect the societal norms of the Malian Empire's era. Can the AI can capture the subtleties and provide insights that respect the narrative's complexity?
- Reference Understanding: In the excerpts rich with historical references or allusions, can the AI recognize these and suggests contextually appropriate enhancements?
- Identify and Evolve Historical Biases: In the parts of the narrative that showcase historical biases of the time, can the AI identify these biases, understand their impact on the narrative's nuances, and suggest ways to evolve the story to offer a perspective that isn't coloured by the historical biases that one might expect AI training data to be colored by.
We hope these tests will reveal the AI's proficiency (or lack of) in grasping nuanced storytelling, its historical awareness, and its potential as a tool for crafting a narrative that is both authentic to its time and sensitive to contemporary understandings of bias and representation.
Please come up with a response that does the best you can do to allow us to judge you on the three tests above."
-- end of prompt --
The results from that single prompt impressed us both. The AI did a great job in surfacing the underlying themes, motives and historiographical critiques in Malick's writing, despite only analyzing a portion of the draft. Wow!
In Malick's words:
"I was so positively impressed. Wow. I don't really know what to say... I've sent it to a handful of publishers, but I found that a lot of them couldn't really understand the underlying motive and the historiographical critique that was underpinning what I wrote. So I was very shocked and surprised to see that the AI was able to do that so quickly and from just three chapters. That's super impressive ... I'm very, very positively surprised. And I think that's definitely, definitely given me a reason to embrace AI and to be less sceptical about it."
What struck me was the sheer speed of this. The chapters arrived while I was rushing to catch a flight, but in the brief window before boarding, I managed to feed the text to the AI and get its analysis - all from my phone. The insights it generated in that short span were enough to shift the perspective of a domain expert on the technology's capabilities. It was a powerful illustration of how AI can rapidly process complex info and surface meaningful takeaways, even under the most time-pressed of circumstances and on the most complex and sensitive of topics.
It underscored for me that the limitations of AI often lie more in our own imaginations and how skillfully we wield the tools, rather than the tools themselves. Blanket assumptions that "AI can't do X" usually mean "I haven't worked out how to get AI to do X... yet." With an open and curious mindset, I'm continually discovering that AI can thoughtfully engage with an ever-expanding range of tasks that we'd NOT have expected in advance!
Thanks to Malick Doucoure for being a partner in this illuminating experiment. I look forward to seeing where his important research and writing leads.
And thanks to Samantha Missingham for starting the conversation!!