Introduction
Just over a year ago, we asked a question that was still genuinely open: can AI agents design and implement drug-discovery pipelines? The bar was not whether a language model could describe active learning, recommend a molecular representation, or generate an isolated function. It was whether an autonomous system could receive a complex scientific problem, choose a strategy, implement it, learn from intermediate results, and produce a competitive final answer.
The distinction matters because only the second kind of system adds capacity. Drug discovery is constrained not only by experimental cost but by the amount of expert attention available for each problem, and that attention is finite and unevenly distributed: some disease areas attract substantial research investment, while neglected infectious diseases, rare conditions, and important areas of women’s health receive far less. Autonomous systems that can perform substantial parts of the computational discovery process, with human scientists providing oversight and judgment, could expand the range of scientific problems that are practical to pursue.
Testing that possibility requires more than another molecular-prediction benchmark. It requires a task in which the agent must decide what to do, not merely produce predictions from a predefined model and dataset.
1.1 The DO Challenge
Most LLM benchmarks isolate a narrow capability. The DO Challenge instead tests an end-to-end scientific decision process under constrained resources. At a high level, the agent must use a limited number of score queries to identify the best candidates in a corpus of one million molecular poses. It must decide what to measure, how to represent the data, what models to build, how to allocate its label budget, and how to use feedback from earlier submissions. It must then implement and execute that strategy.
Those decisions are consequential because the search space is large and the measurement budget is limited. The released dataset contains one million candidate poses, produced through a filtering and docking process. The process began with 500,000 molecules sampled from the Enamine REAL database and filtered to molecules with 25–30 heavy atoms. After docking against PDB 6G3C - the JAK2-V617F pseudokinase domain - 200,000 molecules were retained, each with its five best predicted binding poses. A pose is a proposed orientation of a molecule inside the protein pocket; five poses for each of 200,000 molecules produce the one-million-structure corpus.
Each candidate pose has a hidden DO Score, which combines predicted binding to the target with penalties for predicted binding to three ADMET-related counter-targets. The agent starts with no labels. It may request exact scores for at most 100,000 poses, or 10% of the corpus, in one batch or across several adaptive rounds. Each submission must contain exactly 3,000 pose IDs, and the agent may submit three times. The benchmark score is the fraction of the hidden global top 1,000 poses recovered in a submission. A score of 33.6%, for example, means that the submission contained 336 of those 1,000 poses.[1]
After each submission, the agent receives only one aggregate score, not a hit flag for every pose. The feedback is sparse, but it is not useless. When all 3,000 submitted poses have already been measured, a score of N/1,000 reveals that the top N of those poses by measured DO Score are true global hits. The paper’s reference baselines exploit this inference, so a submission can function as an experiment rather than merely as a final answer.
Success therefore requires more than fitting a predictive model. The system must decide what to measure, how to represent 3D structures, which model to train, how to validate it, how to allocate a fixed label budget, and how to use one submission to improve the next. The task description adds a particularly consequential constraint: the hidden score depends on absolute atomic coordinates and is sensitive to translation and rotation. A representation that centers or aligns every molecule can therefore erase information the benchmark explicitly says matters.
The benchmark was evaluated under two conditions: a ten-hour development-and-submission limit and an unrestricted condition.
1.2 Deep Thought
Deep Thought is a general-purpose multi-agent system for software engineering, machine learning, and scientific research. Its components come in two forms: simple agents that own one well-defined task, and agent groups in which a manager delegates work, integrates intermediate results, and compresses long interaction histories into observations that fit within context limits.
The core component is the Software Engineer agent group. It moves through planning, exploration, coding, and review. Every group member may inspect code, but only the Software Engineer may modify or execute it or call research tools. The Reviewer checks correctness, compliance, and completeness and is explicitly instructed not to design an alternative implementation. An ML Engineer is available but optional.
Around that core sit an Evaluation and Installation group, which creates isolated Conda environments, resolves dependencies, executes code, and categorizes failures; an optional Scientist agent that can provide a blueprint; and an optional Research group that searches, summarizes, ranks, and critiques external information. Figure 1 maps these components and highlights the two roles that received new models in the 2026 controlled rerun.
1.3 What the 2025 evaluation found
The human comparison came from the DO Challenge 2025, an open competition. More than 40 teams applied, 24 were selected based on machine-learning and software-engineering experience, and 20 competed. Every team received the same ten-hour window and computational environment, including an NVIDIA H200 GPU, the one million unlabeled structures on local disk, and a Python client for requesting labels and submitting predictions.
AI coding assistants were permitted and commonly used, so the comparison is best read as a comparison of complete problem-solving workflows rather than unaided humans versus AI.
The best competition team scored 16.4%. Separately, two machine-learning experts with backgrounds in drug discovery and no prior knowledge of the task built reference solutions: 33.6% under the ten-hour limit and 77.8% without a time limit.[1]
Deep Thought’s best timed result was 33.5%, essentially level with the timed human expert’s score and roughly twice the best competition score.
The 2025 study also included two distinct ablations. The first varied the foundation model used for the Software Engineer, measuring how much performance depended on the underlying model. The two panels in Figure 2a and Figure 2b separate this model ablation from the human and competition-team reference conditions. This distinction matters: a 77.8% unrestricted expert result is not directly comparable to a ten-hour competition score, but it provides the strongest recorded benchmark reference.
The second ablation varied the agent composition while holding the task fixed. The Software Engineer alone peaked at 11.5%, while adding a Reviewer raised the peak to 32.3%. Adding an ML Engineer also helped, but less, and using all three roles reduced performance. Full results are reported in the agent-composition ablation in Section 4.2.4 of the 2025 paper.[1] In other words, performance did not improve simply by adding more agents; the specific composition of the system mattered.
Taken together, these results established the core possibility: an AI system could autonomously assemble a scientifically meaningful pipeline and compete with human participants under similar time constraints. They also established clear limitations. Runs were unreliable, workflows were inefficient, and the best autonomous result remained far below the strongest unrestricted expert solution.
Results
Building on the 2025 evaluation, this work investigates two questions: how much performance improves when newer models run in the legacy Deep Thought harness, and how performance shifts when deployed in their native command-line harnesses, Claude Code and Codex CLI. The core quantitative comparison analyzes 15 protocol-matched runs shown in Figure 3 — 11 of which produced scored autonomous submissions, while four failed to submit.
Overall, the newer models substantially raised benchmark performance, but the gains depended on the execution harness. The results therefore separate two effects: the increase in capability associated with newer models, and the extent to which the execution harness converted that capability into a completed, submitted pipeline.
2.1 Overall benchmark performance
New models clearly translated their increased capabilities into a higher performance ceiling. Figure 3 places every comparable run on the same scale, and shows the sequence of submissions.
Deep Thought configurations follow an engineer → reviewer naming convention; runs without a Reviewer are labeled solo. The two external harnesses, Claude Code and Codex, run claude-opus-5 and gpt-5.6-sol, respectively—the same models used as Software Engineers inside Deep Thought. Notably, all six claude-opus-5 runs exceeded Deep Thought’s best 2025 score, and three surpassed the 77.8% unrestricted human benchmark.
2.2 Model generation and harness dependence
Holding the harness fixed. To isolate the effect of model capability, we held the legacy Deep Thought harness fixed and changed only the model. In four protocol-matched runs, claude-opus-5 scored 53.4 – 92.2%, with a median of 65.9%. This represents a substantial improvement under the same harness. At first glance, however, the wide spread among claude-opus-5 runs suggests that this higher performance was not yet reliable from run to run. gpt-5.6-sol showed a different pattern: across four attempts, it never requested labels and never submitted a result. One run nevertheless generated code that scored 51.1% when we later executed it manually without modification, indicating that the failure was not simply an inability to produce a viable solution.
Holding the model fixed. To isolate the effect of the harness, we then held the model fixed. With claude-opus-5, the four legacy Deep Thought runs scored 53.4 – 92.2%, while Claude Code scored 82.1% and 87.3%. The difference was even more striking with gpt-5.6-sol. Both Codex CLI runs used the full label budget, made all three allowed submissions, and scored 68.5% and 74.6%. For gpt-5.6-sol, then, the harness difference was not merely in the quality of the final ranking, but in whether a viable solution was executed and submitted at all.
Taken together, the experiments show two empirical patterns:
- Newer models can reach substantially higher benchmark scores when the harness is held fixed.
- The harness can materially affect whether model capability is converted into a completed autonomous submission.
Both conclusions have limits. The experiment does not separate improved reasoning from possible exposure to the public benchmark, and each comparison contains only two to four observations. We return to benchmark exposure in Section 3.2 and to harness interpretation in Section 3.3.
2.3 Strategies used by high-scoring runs
The high-scoring runs independently converged on the same reframing. The benchmark looked like molecular property prediction, but its scoring rule made it a different problem: budgeted retrieval of an extreme tail. The goal was not to predict every DO Score accurately. It was to spend 100,000 measurements so that as many as possible of the hidden top 1,000 structures became known.
The 92.2% run, using claude-opus-5 as the Software Engineer and gpt-5.6-sol as the Reviewer**,** implemented exactly this idea: a multi-resolution ensemble of sparse ridge models over docking-typed voxel channels at 1.0, 1.25, and 1.5 Å. After a random pilot of 6,000 structures, it acquired labels in five adaptive rounds of 20,000, 20,000, 21,000, 20,000, and 12,997 structures. The hit rate in each new batch rose from 0.10% to 2.03%, a roughly twentyfold enrichment.
Three decisions separated this run from weaker versions of the same basic approach:
• Chemically meaningful atom types. Pharmacophore or docking atom types outperformed bare elements, raising recall@5k from 0.677 to 0.790 in the ablation.
• A tail-focused learning target. Optimizing the extreme high-score region rather than ordinary least squares raised recall from 0.591 to 0.707.
• An early diagnostic submission. At 46,000 measurements, the run used its first submission to identify the exact global top-1,000 cutoff, τ = 0.011021. Progress could then be monitored by exact hit counts rather than an estimated quantile.
The clearest warning was a metric mismatch. Gradient boosting achieved better Spearman correlation than the winning field model, 0.542 versus 0.506, but much worse tail recall, 0.364 versus 0.545. Optimizing the intuitive global metric would have sacrificed roughly half the eventual benchmark score.
The claude-opus-5→claude-opus-5 run reached a similar solution by another route: multi-scale trilinear voxel occupancy with ridge regression, where 148,049 possible columns collapsed to about 25,070 non-empty features. It identified a practical parsing trap - 16.8% of files use the V3000 format rather than V2000 - preserved the fixed coordinate frame, and rejected its own rigid-pose hypothesis using MD5 and inertia-tensor fingerprints. After a self-imposed two-hour timeout stopped execution, the run resumed without spending labels twice.
2.4 Submission trajectories and execution failures
The original 2025 analysis found little evidence that agents used submission feedback to guide later attempts. Because each submission returns a score, the benchmark is inherently sequential: that feedback can inform changes to the predictive pipeline, candidate ranking, or selection strategy before the next submission. Earlier systems, however, largely treated submissions as independent attempts rather than as opportunities for iterative refinement.
The 2026 runs show a different pattern. Most directly comparable runs with verified scores improved consistently across submissions: 9 of 11 never declined, and 8 improved after every attempt. Claude Code and Codex were especially consistent, improving with every successive scored submission. For example, Codex #2 increased from 26.2% to 68.5% over successive attempts. Together, these results suggest that newer agents are beginning to use benchmark feedback as a signal for iterative refinement rather than treating each submission as an independent trial.
The four runs in Figure 3 produced no score. All used gpt-5.6-sol inside the legacy Deep Thought harness. Every run generated substantial code; none autonomously executed a benchmark pipeline.
These cases separate code generation from scientific execution. An agent can produce highly competitive code yet still fail the benchmark because it never takes responsibility for executing the full workflow. The bottleneck was not necessarily model capability, but the surrounding harness and prompting: task ownership was ambiguous, execution was poorly orchestrated, and failures were not reliably detected and recovered from.
Discussion
3.1 Performance gains and changing failure modes
The rise in the best score from 33.5% to 92.2% came with a qualitative change in how the agents failed. In 2025, agents sometimes stated a decisive constraint—such as the dependence on absolute coordinates described in Section 1.1 — and then erased it during implementation by choosing translation- or rotation-invariant representations. They also tended to treat submissions as isolated outputs rather than as steps in a sequential experiment.
Both failure modes were less evident in the 2026 pipelines. Every scored pipeline preserved absolute geometry, and one run explicitly tested and rejected a competing pose hypothesis. Nine of 11 scored runs had non-decreasing submission trajectories, with several traces showing changes to acquisition or ranking between submissions. The strongest systems also framed the task more precisely as extreme-tail retrieval under a limited label budget, rather than general-purpose regression over one million structures, and allocated labels accordingly. Together, these behaviors indicate a more consistent translation of problem understanding into implementation and iterative experimentation than in 2025[1].
These gains are consistent with increased agent capability, but they are not a clean estimate of capability alone. The benchmark and several solution strategies were public before the evaluated models were released, creating a possible exposure confound.
3.2 Memorization and benchmark exposure
The benchmark description and example solutions have been public since April 2025, and every model evaluated here postdates that release. This raises an obvious concern: could the stronger results reflect prior exposure to the benchmark rather than improved problem-solving capability? As a first probe, we supplied only the paper’s title and abstract and asked what the models knew. They consistently recognized the study and volunteered surrounding details, but they failed to state how the benchmark score was computed or reconstruct the task specification. Instead, they treated those details as gaps rather than recalling them.
This offers little evidence of explicit memorization, but it cannot rule out training-data influence. A model may internalize a useful strategy without remembering its source, and it cannot audit its own training data. Exposure could also come from discussions, derivative analyses, or published solution code rather than the paper itself.
We therefore looked for a second, indirect test: whether knowledge of the benchmark and its key constraints was sufficient to explain the performance gains we observed. The claude-opus-4.5 runs provide a useful comparison because they also postdate the benchmark release but achieved much lower performance. In three runs under the same legacy harness, claude-opus-4.5 correctly identified the fixed coordinate frame, rejected a conventional QSAR framing, and used all three submissions, yet scored only 27.3-30.7%. This suggests that recognizing the benchmark structure and key task constraints was not, by itself, sufficient to produce the much higher scores later achieved by claude-opus-5. The difference is therefore consistent with a genuine capability improvement, although this comparison cannot determine how much of the gain came from improved reasoning versus benchmark exposure or other changes between the models.
3.3 What the harness comparison can - and cannot - show
The legacy traces pointed to a concrete execution problem. In the two Opus runs in Figure 4 that included a Reviewer, review began only after label acquisition and all submissions were complete, leaving no opportunity for the Reviewer to affect the benchmark outcome. All four gpt-5.6-sol attempts in the legacy harness likewise failed to carry their proposed pipelines through autonomous execution. This pattern is consistent with failure modes reported in prior work. Kim et al. found that collaboration can become counterproductive for capable models on tasks requiring tightly coupled sequential reasoning[2], while Cemri et al. identified coordination errors, duplicated work, context loss, and weak verification as recurring sources of failure in multi-agent systems[3]. The DO Challenge has exactly this sequential structure: agents must repeatedly acquire labels, fit a model, execute a strategy, evaluate the result, and update the next step.
At the same time, the stronger performance of Claude Code and Codex CLI suggested a practical follow-up: could a lightly modernized Deep Thought harness produce comparable results with the same models? We call this configuration Deep Thought v2.
Both models completed the benchmark autonomously in both of their v2 runs. claude-opus-5 scored 89.2% and 91.3%, compared with 82.1% and 87.3% in Claude Code. gpt-5.6-sol scored 70.7% and 74.8%, compared with 68.5% and 74.6% in Codex CLI. The gpt-5.6-sol contrast is especially informative because none of four legacy-harness attempts submitted autonomously; the values in Figure 4 come from later manual evaluation of the artifacts.
Within this small sample, v2 matched the native CLIs and removed the execution failure observed with gpt-5.6-sol in the legacy harness. The result does not show that single-agent systems are universally superior, nor does it identify a causal contribution from each prompt or orchestration change. It does show that the underlying model capability was sufficient, and that relatively modest, task-agnostic changes to ownership and execution were enough to convert that capability into successful autonomous execution.
Conclusions
The DO Challenge looks very different one year later. The best autonomous run recovered 922 of the hidden top 1,000 structures, and five runs exceeded the previous unrestricted human result of 77.8%. Newer models clearly raised the performance ceiling, but the results also show that capability alone was not enough: the legacy Deep Thought harness sometimes failed to translate strong model output into autonomous execution. The clearest example was gpt-5.6-sol, which failed to submit in four legacy-harness attempts but completed the benchmark in both Codex CLI and Deep Thought v2. Together, these results suggest that both model capability and the execution system around it determine autonomous performance.
The benchmark is also beginning to saturate at the observed frontier, and its public history limits what can be inferred about general scientific capability. The next question is therefore whether these systems can perform reliably on new problems where the answer is not already available and success must be established prospectively.
We are now testing that transfer from closed optimization to prospective science. Deep Origin has developed a next-generation AI Scientist with state-of-the-art computational tools[4] and integrated wet-lab capabilities. The system is being deployed in a real drug-discovery setting through the ARIA-funded endometriosis program. Public reports describe the targets identified by the AI Scientist,[5] and ongoing updates are available on X (@DOAIScientist).[6]
References
[1] Smbatyan, K., Ghukasyan, T., Aghajanyan, T., et al. (2025). Can AI Agents Design and Implement Drug Discovery Pipelines? Deep Origin. arXiv preprint arXiv:2504.19912. arxiv.org/abs/2504.19912. Benchmark: doi.org/10.5281/zenodo.15296510.
[2] Kim, Y., Gu, K., Park, C., et al. (2026). Capable language models can outgrow the benefits of collaboration. Nature Machine Intelligence, 8, 1157-1172. doi.org/10.1038/s42256-026-01268-y.
[3] Cemri, M., Pan, M. Z., Yang, S., et al. (2025). Why Do Multi-Agent LLM Systems Fail? NeurIPS 2025 Datasets and Benchmarks Track. arXiv preprint arXiv:2503.13657. arxiv.org/abs/2503.13657.
[4] Petrosyan, G., Altunyan, V., Ghukasyan, T., et al. (2026). Overcoming the accuracy-generalization tradeoff in docking and scoring for prospective virtual screening. bioRxiv preprint 2026.08.03.742480. doi.org/10.64898/2026.08.03.742480.
[5] Deep Origin. AI Scientist: Endometriosis Target Discovery. deeporigin.com/ai-scientist/endometriosis-targets; Deep Origin. AI Scientist Research Logs 8-9: MIF receptor selection and validation-set construction. Deep Origin on LinkedIn; Advanced Research and Invention Agency. ARIA Corporate Plan 2026, p. 31. aria.org.uk.
[6] Deep Origin. AI Scientist (@DOAIScientist). X. x.com/DOAIScientist.
