Evaluation

Requirements for a valid file · Task 3

The starter kit's Task 3 tutorial ↗ walks through building a prediction file end to end, and score_h5ad.py applies every check below to a local file before you ever upload it. A file that breaks one of these comes back as a validation error naming what was wrong — not as a low score.

• var_names must be exactly the task gene panel — 500 genes — in panel order. The check is element by element; a mismatch reports the expected and received counts plus the first missing and extra names. If you have the same genes in a different order, pass --allow-reorder and the scorer reindexes for you.
• .X must be a 2D cells × genes matrix, finite, and non-negative. Sparse and dense both load — the scorer densifies and casts to float32 either way — so float32 is worth writing yourself only to avoid a surprise in your own pipeline.
• Not caught by validation —Values must already be log-normalised. A raw count matrix is non-negative and finite, so it passes every check and is then scored as though it were on the log scale. Nothing will tell you; the score will simply be wrong. Do not submit counts that are normalised but not log-transformed either.
• obs["celltype"] is optional. Cells with no label are read as NA, and no label in your file affects how it is typed: the scorer trains its own probe on the held-out truth and applies it identically to every entry.
• The number of cells is free. There is no minimum, no cap, and no correspondence to the target — every metric is distributional, so a submission need not preserve cell identity or count.
• obsm["spatial_3D"] is required, with shape cells × 3 or more and no NaN or infinity. Only the first three columns are read.
• Coordinates may be in any frame. Every spatial metric is invariant to translation and rotation, so a submission is never asked to register itself to an atlas — with one acknowledged blind spot, laterality: a mirrored embryo scores identically to a correct one.
• Predict the mutant embryo itself, not its difference from wild type. Returning the matched wild type unchanged already reaches 0.956 absolute pseudobulk correlation, so the absolute state carries almost no information about whether a model understood the knockout.
• Knock-outs ship as two replicates and the scorer picks the larger one to compare against. You submit one file per target knockout; which replicate it is scored on is not yours to choose.

File contract · Task 3

Format

AnnData .h5ad, .X + obsm["spatial_3D"]

Condition

The held-out mutant, at its stated stage

Genes

500-gene MERFISH panel, in panel order

Coordinates

(n, 3) — any frame

Labels

Never submitted

What the scorer reads

A submission is one AnnData file. This is the layout the scorer opens it expecting for Task 3; anything not listed is free.

AnnData object with n_obs × n_vars = <your cells> × 500
    X       float32, log-normalised, finite, non-negative
    var     index = the 500-gene MERFISH panel, in panel order
    obsm    'spatial_3D'  (n_obs, 3) float32, finite — any frame
    obs     'celltype' optional and ignored

What is deliberately not constrained

• Cell count. Nothing requires n_pred = n_true — a genuine growth or proliferation model may predict a different number of cells than the target has. Where a count difference would confound a comparison, both point clouds are subsampled to a shared size first.
• Coordinate frame. Every spatial metric is invariant to translation and rotation, and outside the laterality blind spot to reflection, so no registration to the atlas is expected.
• Cell ordering and identity. No metric assumes predicted cell i corresponds to target cell i.

Score a file locally

The starter kit exposes the same scoring path the task runners use, standalone — no baseline model involved. It loads the task’s real target itself, validates your file against the expected gene panel and order, and prints the full metric panel as JSON.

python score_h5ad.py --task T3 --target-ko gata4 --input pred.h5ad

• Optimising absolute expression loses to the floor by construction — the floor is the matched wild type, which is already almost right in absolute terms.
• Copying the training knockout’s delta onto the test gene recovers a share of the effect and gets the direction wrong; the panel scores direction explicitly.