Welcome to the 19th issue of Agents in Practice.
📬 Agents in Practice is a weekly newsletter on agentic AI research and applications. Subscribe here →
How reliably can LLMs judge novelty?
Researchers from Ai2 and the Hebrew University of Jerusalem studied whether LLMs can reliably judge the novelty of research ideas. The authors used peer reviews to identify human-authored ideas with strong agreement about their novelty. They tested six models on ideas from ICLR 2026 papers and ideas generated by LLMs, varying the prompts, reasoning budget, and access to related papers.
The researchers found that models are susceptible to how the model is prompted to judge novelty. Given a human-authored-and-reviewed idea and a generated idea, GPT-5.4 preferred the human idea 93.5% of the time when it was told reviewers had found one idea novel and the other not. However, asking only which idea was novel lowered this to 40.9%, changing its verdict on more than half of the same pairs. Prompt changes also flipped roughly a quarter to a third of verdicts when comparing ideas from two AI generators.
The authors also write that providing related papers or more reasoning did not consistently improve judgments.
Personal Thoughts
Novelty is such a difficult concept to evaluate. Even human reviewers often disagree whether a paper is sufficiently novel, especially when the outcome seems “obvious” in retrospect. Using ICLR reviews seems like a nice way to proxy human perception of novelty, but I wonder how much of this will be contaminated by reviewers using LLMs.
Read more
Do personalized skills help coding agents?
Researchers from UMass Amherst and OpenRefinery.ai studied whether coding agents benefit from skills that capture a developer’s preferences. Their method used pattern matching to build an initial skill file, then an LLM to refine it, keeping rules supported by at least two separate sessions. They replayed tasks with an LLM simulating developer follow-ups and another LLM scoring completion. Only one of the eight model pairings showed a statistically significant improvement in task scores with personalized skills.
In a separate comparison using one model setup, a generic skill built from multiple developers’ conversations raised average task scores by 3.11 points out of 100 compared with using no skill (65.27 → 68.38). However, the difference between generic (68.38) and personalized (67.64) skills was not statistically significant. Skills also did not consistently reduce follow-up requests from the simulated developer.
Personal Thoughts
This is a continuation of my anecdote from two weeks ago. It seems like building personalized skills often leads to local optima and becomes a maintenance burden. Yet, in some areas personalization is still important. Keeping personalization lightweight yet consistent across different LLMs seems like an interesting problem, both for researchers and product engineers.
Read more
One-liners
- On the model side, Anthropic released Claude Fable 5.2 and Mythos 5.2 (Nov 9), and Google released Gemini 4 Hydrogen (Nov 12). Google also made Gemini 4 Carbon (Nov 10) generally available, after the limited access covered two weeks ago.
- Databricks introduced a new version of Genie (Nov 11), an agent for long-running enterprise work.
Subscribe to Agents in Practice
Get new issues of Agents in Practice — a newsletter summarizing exciting new research and applications in agentic AI — delivered to your inbox.