Evaluation

T3 — Mutant perturbation prediction. 3D MERFISH (500-gene panel) · predicts expression + 3D coordinates.

Task

Participants will predict a held-out knockout: the expression and 3D position of the cells in a mutant embryo, given wild-type development plus one observed knockout to learn the shape of a perturbation response from.

A conditional knockout line is months of work per gene, so no lab can make one for every gene it wants to understand. The three knockouts here are ordered by difficulty: training and validation use very specific genes, while the hidden test gene is broadly expressed and its effect correspondingly diffuse.

Scoring is on the perturbation effect, not the absolute state. Predicting no change at all — simply returning the matched wild type — already reaches 0.956 absolute pseudobulk correlation, so nothing but the wild-type-to-mutant change carries any information about whether a model understood the knockout.

Two parallel pairs. On the left a wild-type embryo and, after an arrow, the observed knockout it becomes, both filled. On the right the same wild type and, after an arrow, an empty violet outline where the unseen gene's mutant would be. — The same question asked twice. One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

The same question asked twice. One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

Splits

Validation and test ground truth are both withheld. A validation submission returns a leaderboard score, not the answers — those are released only when the final test set is.

Train: Mab21l2 KO @ E9.5; A clear phenotype from a very specific gene.
Validation: Gata4 KO @ E8.75; Also specific, at a different stage. Scored through the leaderboard; the answers are not distributed.
Test: β-catenin KO @ E8.75; Broadly expressed, so its effect is diffuse — the hardest of the three. Hidden until the competition closes.
Reference: Matched WT @ E8.75 and E9.5; Every DE metric is computed against the matched wild type, not against a preceding stage.

Scored on the perturbation effect (WT→KO delta), not absolute expression: doing nothing already scores 0.956 absolute pseudobulk Pearson, so only the change counts. Full task page →