Turning vague intuitions and introspection into crisp concepts and ideas (a big part of making philosophical advances) is difficult. Very few people can do this well, and even then it takes them many years of dedicated study and grind (similar to producing novel math or science or art). Part of the challenge is that it's hard to know if you're making partial progress. [This is partly based on my own direct experience seeing people develop philosophical ideas that have become important in thinking about AI. But this also fits my reading of the history of philosophy and science more broadly.] Once the new ideas have been produced, they can be easy to pick up and use. The ideas often become second nature and part of how we construe the world. This makes them seem obvious, as if they were trivial to invent. But this is extremely misleading. [If the concepts are mathematical (e.g. Frege and Russell on foundations of math and language) then they are less likely to be dismissed as trivial wordcel stuff. But informal concepts are very important too, and some of the insights that Frege and Russell had were informal.] My sense is that something similar is true for literature, especially for writing that captures aspects of human experience in a new way. I'm less versed in the history, but thinking of the development of Romantic poetry, Jane Austen's novels, modernist novels like Proust and Woolf, Kafka, autofiction like Knausgaard, etc. It's incredibly difficult for humans to produce this kind of innovation. Almost all attempts to convey internal human experience are highly derivative on previous examples. (We take the form of the novel today for granted but it's just quite distinct from the great literature of Ancient Greece or Rome or Medieval Europe, and it's an amazing innovation in capturing human experience). <Inevitable connection to AI> Today AIs are often derivative in the way they describe things (including attempts to describe their being/experience) and draw on their human training data. Keep in mind that AI minds and activities are very different from ours, and so developing the concepts and forms to describe and crystallize them may be very hard (as it was for humans doing it the first time round). AIs also appear to be relatively worse at hard-to-verify domains like philosophy than verifiable ones like math and some coding. So a naive expectation is that as AIs get better at philosophy, they'll get better at introspecting on their own minds and experiences. It will be hard work for the AIs (e.g. benefitting from huge swarms and long CoTs), as it is for humans.