Evaluation

T3. Mutant perturbation prediction. 3D MERFISH (500-gene panel) · predicts expression + 3D coordinates.

Task

Participants will predict a held-out knockout: the expression and 3D position of the cells in a mutant embryo, given wild-type development plus one observed knockout to learn the shape of a perturbation response from.

A conditional knockout line is months of work per gene, so no lab can make one for every gene it wants to understand. The three knockouts here are ordered by difficulty: training and validation use very specific genes, while the hidden test gene is broadly expressed and its effect correspondingly diffuse. Validation answers become available by explicit opt-in during P3, as for the other tasks. Access permanently ends P2 submission eligibility; hidden test answers remain private.

Scoring is on the perturbation effect, not the absolute state. Predicting no change at all, simply returning the matched wild type, already reaches 0.956 absolute pseudobulk correlation, so nothing but the wild-type-to-mutant change carries any information about whether a model understood the knockout.

Task 3 from the competition proposal: 3D MERFISH point clouds of a wild-type embryo and a β-catenin knockout side by side, with dashed boxes marking the held-out knockouts used for validation and for the hidden test. — One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

Splits

During P3, eligible participants can explicitly request the P2 validation answers for model development. Receiving access or submitting a first final P3 entry permanently stops further P2 submissions: for the entire ordinary team, or the individual Yukon member. Access does not consume a P3 attempt. The live Validation leaderboard remains separate from the immutable Frozen P2 standings captured at the P3 opening cutoff. Hidden test ground truth is never distributed.

Train: Mab21l2 KO @ E9.5; A clear phenotype from a very specific gene.
Validation: Gata4 KO @ E8.75; Also specific, at a different stage. Scored through the leaderboard until the final phase, when participants can explicitly request the answers and permanently end their P2 submission eligibility.
Test: β-catenin KO @ E8.75; Broadly expressed, so its effect is diffuse: the hardest of the three. Hidden until the competition closes.
Reference: Matched WT @ E8.75 and E9.5; Every DE metric is computed against the matched wild type, not against a preceding stage.

A mutant embryo still looks mostly like a wild-type one: hand back the matched wild type unchanged and absolute expression already correlates at 0.956. So the score is computed on the difference between wild type and mutant, and a submission earns nothing for reproducing the wild type it was given. Only for moving the genes the knockout actually moves. Full task page →