Evaluation

Requirements for a valid file · Task 2

The starter kit's Task 2 tutorial ↗ walks through building a prediction file end to end, and score_h5ad.py applies every check below to a local file before you ever upload it. A file that breaks one of these comes back as a validation error naming what was wrong — not as a low score.

• var_names must be exactly the task gene panel — 500 genes — in panel order. The check is element by element; a mismatch reports the expected and received counts plus the first missing and extra names. If you have the same genes in a different order, pass --allow-reorder and the scorer reindexes for you.
• .X must be a 2D cells × genes matrix, finite, and non-negative. Sparse and dense both load — the scorer densifies and casts to float32 either way — so float32 is worth writing yourself only to avoid a surprise in your own pipeline.
• Not caught by validation —Values must already be log-normalised. A raw count matrix is non-negative and finite, so it passes every check and is then scored as though it were on the log scale. Nothing will tell you; the score will simply be wrong. Do not submit counts that are normalised but not log-transformed either.
• obs["celltype"] is optional. Cells with no label are read as NA, and no label in your file affects how it is typed: the scorer trains its own probe on the held-out truth and applies it identically to every entry.
• The number of cells is free. There is no minimum, no cap, and no correspondence to the target — every metric is distributional, so a submission need not preserve cell identity or count.
• obsm["spatial_3D"] is required, with shape cells × 3 or more and no NaN or infinity. Only the first three columns are read.
• Coordinates may be in any frame. Every spatial metric is invariant to translation and rotation, so a submission is never asked to register itself to an atlas — with one acknowledged blind spot, laterality: a mirrored embryo scores identically to a correct one.
• One file per setting. The heart and embryo settings are scored separately — score each with --setting heart or --setting embryo, and submit them as separate entries.

File contract · Task 2

Format

AnnData .h5ad, .X + obsm["spatial_3D"]

Genes

500-gene MERFISH panel, in panel order

Coordinates

(n, 3) — any frame

Cells

Need not match the target count

Setting

heart or embryo — scored separately

What the scorer reads

A submission is one AnnData file. This is the layout the scorer opens it expecting for Task 2; anything not listed is free.

AnnData object with n_obs × n_vars = <your cells> × 500
    X       float32, log-normalised, finite, non-negative
    var     index = the 500-gene MERFISH panel, in panel order
    obsm    'spatial_3D'  (n_obs, 3) float32, finite — any frame
    obs     'celltype' optional and ignored

What is deliberately not constrained

• Cell count. Nothing requires n_pred = n_true — a genuine growth or proliferation model may predict a different number of cells than the target has. Where a count difference would confound a comparison, both point clouds are subsampled to a shared size first.
• Coordinate frame. Every spatial metric is invariant to translation and rotation, and outside the laterality blind spot to reflection, so no registration to the atlas is expected.
• Cell ordering and identity. No metric assumes predicted cell i corresponds to target cell i.

Score a file locally

The starter kit exposes the same scoring path the task runners use, standalone — no baseline model involved. It loads the task’s real target itself, validates your file against the expected gene panel and order, and prints the full metric panel as JSON.

python score_h5ad.py --task T2 --setting heart --input pred.h5ad

• Coordinates are per-embryo local and not registered across time points, so no shared frame can be assumed between input stages — and none is required of the output.
• Regressing coordinates against an MSE target optimises absolute position in a frame the metrics deliberately ignore, while leaving the relational structure they do score unconstrained.