Lasting memories
Lasting Memories – how we work around limited context window
5 April 2024
An RPG campaign could last hundreds of hours. Some of the most famous D&D games have been running for years. My favourite Acquisitions Incorporated started in 2008 and is still going strong.
With Casual Legends, a role-playing game powered by AI, our goal is to make it very easy to get into the game and enjoy a short story but then to be able to scale it to an epic with many characters, places, and an overarching story—a full campaign.
A short story is easy, but a campaign is very hard, and it has to do with the architecture of large language models (LLMs), specifically the limited context window.
When a model generates the next token, it has to put it in context and calculate based on all the surrounding tokens: say, the word charm could refer to a magical effect or jewellery, depending on the context. Understanding the context is crucial for generating the next sentence that makes sense.
The context window size defines how coherent a model is over time. At the time of my writing this, OpenAI gpt4-turbo featuring 128K tokens of context and Calude-3 models 200K. In practice, many platforms like Poe or Coze reduce the context window to just a few thousand tokens to manage the costs. It is ok for short questions and answers, but for a game like ours, we need to use most of what is available and extend beyond the limitations.
One way we do it is by summarizing. In Epic Legends, we take the conversation sliding out of the context window, summarise it, and add the summary back into the context. This way, the overall story is remembered.
On top of that, every time players encounter a new character, a record in a database is created, storing details of the character sheet. From there on, every mention of the characters retrieves the data from the database and adds it to the context so that the model has exact details — the state of the game world.
In the background, we generate an image for every character, location, place, or item or assign a pre-generated one to make navigating the world easier and help players remember what is out there.
With this approach, the game's world starts undefined with only a few big strokes: your character at the centre, the setting, and the first quest. Over time, the world becomes more defined as you interact again with places and characters you encountered before, find how they relate, and establish a broader story.
We are very early in building up this system, and it will take a lot of iteration to do it right.
We are still figuring out how to introduce more interesting turns of events, challenges, villains, and themes. I will probably share more about themes in our next blog post.
Meanwhile, join our Discord community and start your story to remember today.