The ballad of LLM agents: philosophical reasoning for chemistry

Lately, it feels like we can't go more than a week without a new version of an LLM or foundation model dropping. New versions of Claude, new Gemini, new Qwen, new GPT, new DeepSeek, new Llama, new Mistral, new Grok, and more. There are now so many highly capable models, with such similar functionality, and they're releasing so frequently, that I for one have trouble telling the release notes apart. Every announcement feels and looks the same. More parameters, more training data, bigger context windows, more reasoning, blah blah blah. Always the biggest, best model ever...right?

So for me personally, these press releases have started to matter less and less. And I care less and less about how the model producer is choosing to describe their network or its capabilities. What I care about instead are the numbers.

If you scroll down to the bottom of most announcements, you'll encounter a table at the end. That table describes how this particular model performs on a standard, independent, often community-driven benchmark. Usually compared to its predecessors, and other major competitors. To figure out how "good" a model is, you just need to find the row for the benchmark you care about, and see how this new model stacks up on it.

And for the most part, this system is great. It's an objective, convenient, and standardized way to help us all contextualize a model's release. And to figure out, despite the PR talk surrounding it, if this version is just a small improvement, or if it's truly a step-change moment. That being said, those tables are never exhaustive. The model developer will often just showcase the numbers for the largest, most popular benchmarks, and then throw in a bunch of others that make the model look good. And even those smaller benchmarks are often general-purpose, or wide categories like programming, scientific reasoning, or general mathematical problem-solving.

It would be a mistake to think those are the only ones. The only benchmarks that exist, or the only ones that matter. In reality, there are many more. Hundreds or thousands, even. From broad general-purpose ones, to smaller, more exotic ones that test the model's abilities in a very narrow or specialized domain. And everything in between. One of those lesser-known benchmarks is called ChemBench. It tests the model on hundreds of open-ended chemistry questions across physical, analytical, general, inorganic, and organic chemistry. One question might ask the model to combine thermochemical equations to calculate a reaction enthalpy. Another might ask it to count electrons in a molecule, or even predict how many NMR peaks a molecule should produce based on symmetry and chemical equivalence. And to be clear: this test is hard. It's designed to evaluate if the model can do more than recognize chemistry-flavored text, and test whether or not it can carry out quantitative, multi-step chemical reasoning. One of the reasons you don't often see this bench reported is that even the newest, most capable flagship models score terribly on it out of the box. At the 1% error threshold (meaning the numerical answers need to be within 1% of the true value to be considered correct) the base GPT-4o model scores about 57%, GPT-5 around 86%, and GPT-5.1 only 51.0%. All of these models fail frequently, and they do so for a myriad of reasons. They miss calculations, they choose the wrong intermediate quantity, they lose track of units, they apply the right rule in the wrong place, they round or extract the wrong final value, or they just guess. And end up producing answers that sound chemically plausible but are far outside the required tolerance.

But this sub-par performance is not for lack of trying. Many, if not most, flagship models have plenty of chemistry text and worked examples in their training set. Plenty of exposure to textbooks and problem sets, and plenty of examples of stoichiometry and thermodynamic calculations, of titrations and spectroscopy, and of oxidation states and reaction mechanisms. But despite that, they still just don't do very well at this. At all.

So the question is: "can they do better?" Not "can they be fine-tuned to do better"? But can the models, as they are right now, unchanged from their current state, be prompted and instructed in such a way that better unlocks that knowledge. Better enables them to connect to the relevant parts of their training corpus, and generate more coherent, accurate solutions to these types of problems.

That's exactly what the authors of today's paper are trying to do. In it, they're showcasing a new prompt-level reasoning enhancement framework, that they claim can improve model performance on ChemBench by between 4 and 11 percentage points, and in one case even up 21 points. But it's not a RAG system. It's not fine-tuning, and it's not tool use. It's just a system-prompt wrapper. A series of instructions that wrap around the incoming question and then constrain and steer the model's reasoning process in such a way that it performs better on these types of questions. On today's episode we'll walk through how it works. Let's dive in.

To understand what the authors have created here, we need to brush up on a little background knowledge. Not about model training, not about NLP, not even about machine learning, no. We need to brush up on the canon of classical philosophy. Specifically: Socrates, Aristotle, Descartes, Hegel, Hume, Plato, and Kant. Why? Because the authors built seven different versions of their framework, and modeled each version after one of those thinkers.

Bear with me, these are going to be horrible oversimplifications. But here are the main things you need to know about those seven philosophers:

1. Socrates is the philosopher of questioning. He does not begin by declaring an answer; he begins by interrogating the assumptions underneath the question. What exactly do we mean? What must be true for this answer to follow? Where could the reasoning fail?
2. Aristotle is the philosopher of categories, causes, and first principles. He wants to know what kind of thing something is, what properties define it, what causes explain it, and how a particular case fits into a larger structure.
3. Plato is the philosopher of abstraction. He pushes away from messy particulars and toward ideal forms, symmetries, and general conceptual patterns.
4. Descartes is the philosopher of methodical doubt and decomposition. He breaks a problem into simpler parts, solves those parts in order, and tries not to trust anything until it has been rebuilt from a secure starting point.
5. Hume is the empiricist. He is skeptical of grand claims that outrun the evidence, and he prefers observed regularities, examples, analogies, and cautious probabilistic inference.
6. Kant is the philosopher of rules, conditions, and constraints. He asks what premises have to be in place before a judgment is valid, and he emphasizes consistency with governing principles.
7. Hegel is the philosopher of contradiction and synthesis. He looks for tensions inside a claim, sets opposing ideas against each other, and tries to resolve them into a more developed understanding.

Again, these are cartoon versions of very complicated thinkers, but you get the point. Besides, the authors here are not trying to recreate the entire history of philosophy. They are trying to turn recognizable archectypal reasoning habits into operational prompts.

So, how do those personalities translate into prompting and model behavior? Well, let's zoom out to the architecture. At a bird's eye view, the authors' system looks like this. You start with a normal base LLM, then prepend a fixed system prompt before an incoming chemistry question. That system prompt does not add new facts, retrieve documents, call calculators, browse the web, or modify the model's weights. It simply defines a reasoning policy.

These are just hardcoded instructions that say, in effect, "when you answer this question, reason according to this structured style." And then the actual ChemBench question is passed in as the user prompt. And at that point, the model then generates its answer normally. It's just that, under the hood, its generation process is now being conditioned by that system-level instruction.

When the answer comes back, the authors then parse it, compare it to the ChemBench reference value, calculate the relative error, and ask whether the answer falls within 1%.

The authors call each system-prompt an "agent" but that's really not the right term for what they're doing here. It's better thought of as a personality or skin. There's no agentic behavior to speak of. It is just the base model plus a system prompt in the style of a particular philosopher. And the only thing that changes from one personality to another is the reasoning scaffold that wraps around the question. For example:

• The Socrates version tells the model to clarify the problem, test assumptions, ask itself follow-up questions, and check whether its reasoning actually supports the answer. The theory being that this kind of self-interrogation can keep the model from jumping too quickly to the first plausible-looking answer.
• The Descartes version tells the model to decompose the problem into simple steps, proceed deductively, and rebuild the solution from stable premises. The theory being that this should help when the task is really a sequence of calculations hiding inside a paragraph.
• And so on for each of the philosophers.

Once the authors had all seven personalities prepped, it was time to test them. To pair them with base models and see if any of the personalities could help them improve on this particular bench. Specifically they tested GPT-4o, GPT-5, and GPT-5.1. And for each, the authors ran the same ChemBench subset through the base model alone and then through each of the seven philosopher wrappers. That means every question was tested repeatedly: once with no philosopher prompt, and then once for each philosopher, and this was repeated for each of the base models. They used identical inference settings across runs, a 2000-token budget, low temperature, and a fixed nucleus sampling. This way the comparison would mostly reflect the system prompt rather than randomness in decoding. They also avoided allowing the system to adapt between questions. There was no feedback loop, no memory, and no multi-turn repair process. Each question was a fresh single-turn test of what that specific model+philosopher could produce.

So what happened? Well, the headline result is that the philosopher prompts usually helped, but not evenly.

• For GPT-4o, (whose base performance was only 57%) every philosophy agent made a difference. Hume was the best overall, pushing it to 68%, while Aristotle and Kant both reached 67%.
• For GPT-5, the starting point was much higher: remember the base model already scored 86%, so there was less room to improve. But still, Kant pushed it up to nearly 91%, and Aristotle and Descartes landed around 89%. Hegel and Plato on the other hand slightly underperformed the base model.
• And for GPT-5.1, whose baseline was the weakest (51%), the gains were the largest. Socrates raised performance to 73%, Descartes reached 63%, and Aristotle reached 58%.

And both the task-level results and domain-level results followed this same kind of uneven pattern. Big improvements in some areas, flatlines in others, and backsliding in a few. So the result was not "philosophy wrappers make the models better". It was more specific than that: that particular reasoning prompts helped particular models on particular kinds of chemistry problems, but not necessarily others.

But why? Why did Socrates work so well for GPT-5.1, but less for GPT-5? And why were Descartes and Kant more effective on calculation-heavy and rule-bound problems than Plato and Hegel? Well, honestly, we don't exactly know. But here's a theory: these prompts work because they change the model's default trajectory through the problem. The base model may already contain a lot of chemistry knowledge, but without structure it can grab the wrong fragment, stop too early, or produce an answer that sounds like chemistry but isn't mathematically sound. A philosophy-personality prompt adds an external reasoning scaffold to that process. It tells the model what kind of mental posture to adopt before it starts generating tokens. Socrates slows the model down and forces it to interrogate the task. Descartes forces decomposition. Kant forces rule-checking. Hume forces grounding in examples and evidence. None of that gives the model any new knowledge. It just forces the model to use its existing knowledge in a more disciplined way. But then why are the gains uneven? Well, a strong model like GPT-5 may already do a lot of that internally, so the prompt only helps at the margins. A weaker or less stable model like GPT-5.1 may benefit more because the prompt supplies the kind of structure that the model was not reliably supplying for itself. Then, add in the fact that the utility of the scaffold depends on the shape of the task. And you can end up in a situation where a calculation problem benefits from Descartes in a way that an empirical pattern-recognition problem does not. And a rule-bound problem might benefit from Kant, while a Hegel-style prompt might make it backslide.

So what can we learn from this paper? Narrowly, that philosophy-inspired system prompts can substantially improve performance on the numerical subset of ChemBench, especially for weaker baselines and for tasks where the prompt style matches the problem structure. But more broadly, and more importantly, this study shows that in some cases prompt design and instruction structure can be just as important as model size and training data. And that while we all love to see the latest benchmark numbers and new high scores, those numbers don't always tell us the whole story. In some cases the model we're examining may have far more knowledge, reasoning capacity, and reliability to give, but only if they are prompted and scaffolded the right way. If their relevant knowledge and reasoning habits are encouraged and their tendencies to shortcut and guess are constrained. To put it another way: this paper shows us that the model you've been waiting for, for your particular use case, might already exist. You just need to figure out a way to unlock its performance. Want to go deeper? Make sure you download the paper.