JetBrains, the maker of IntelliJ IDEA, PyCharm and Kotlin with more than 12.5 million active users, has revealed in a conversation with Anthropic how the company evaluates new models and when it deploys Claude Fable 5 over predecessor models. For CTOs, the report provides concrete figures on benchmark results and a structured approach to model evaluation in production coding environments.
Vladislav Tankov, CTO of JetBrains, describes an evaluation pipeline for language models that has grown over several years. The company tests new models against large, private evaluation sets, including its own monorepo, to check whether models maintain their public benchmark scores on real-world tasks as well. To this end, JetBrains maintains separate leaderboards for best quality, lowest cost per task, and fastest execution.
In testing, Claude Fable 5 achieved a Python pass rate of 44.3 percent compared to 28.2 percent for Opus 4.8, a jump of 16 percentage points. In direct comparison, Claude Fable 5 solved 18 Python tasks on which Opus 4.8 failed, while losing only two in the other direction. The model also required around 22 percent fewer steps to reach a solution. On Java tasks, Opus 4.8 repeatedly attempted to pull in external resources that are of little help in the JetBrains environment, whereas Claude Fable 5 worked directly with the existing code.
According to Tankov, this translates into a clear division of roles in practice: Opus is regarded as a reliable default worker for tasks where the solution path is known. Claude Fable 5 is used when deeper reasoning is required or when the solution path itself is unclear — for example, in implementing a complex rich-text editor component that JetBrains had unsuccessfully attempted multiple times over the years, which succeeded with Claude Fable 5 on nearly the first attempt. Another use case is long-running agentic coding: an agent receives specifications in text and image form and implements complex IDE-like applications from them, with the specifications in some cases generated by the agent itself from existing apps — a method JetBrains also uses for porting applications between runtimes, frameworks, or programming languages.
On the topic of security and data handling, Tankov makes clear that JetBrains relies on Anthropic’s red-teaming and safety measures rather than working on the model itself. The company’s own contribution lies in systematically securing the infrastructure and the deployment harness surrounding the model. Security is also one of the most important use cases for Claude Fable 5 at JetBrains: the company runs white-box tests against its own products to uncover vulnerabilities.
At JetBrains, Claude Fable 5 achieved a Python pass rate of 44.3 percent compared to 28.2 percent for Opus 4.8 and requires around 22 percent fewer solution steps, while for data security JetBrains relies on Anthropic’s red-teaming and its own infrastructure hardening.
Source: claude.com · Published August 12, 2026
Lumi AI News — AI-assisted curation pursuant to Art. 50 EU AI Act. Paraphrasing and classification by Lumi News Pipeline v1.8.3.