Research demo · Project B · non-clinical
Tissue images × gene activity
Pathology foundation models · HEST-1k
NCBI10 Public HEST-1k sample H-Optimus-0 UNI2-h Research only

Can AI ‘see’ tissue the same way genes describe it?

We took one public tissue slide (NCBI10), cut it into 138 small image patches, ran two pathology foundation models, and compared their view of the tissue with measured gene activity.

HEST-1k NCBI10 — H-Optimus-0 vs UNI2-h geometry, alignment, and morphology–expression coupling

For everyone: pathologists look at stained tissue under a microscope; modern labs can also measure thousands of genes in small spots on that same tissue. Here we ask whether two large AI models, trained on millions of pathology images, notice the same patterns — and whether those patterns line up with genes.

For specialists: dual foundation-model tile embeddings (1536-d) on Visium-aligned H&E, with representation geometry (PCA/UMAP), cross-model congruence (cosine, linear CKA, k-NN, Mantel, ARI/NMI), and morphology–transcriptome coupling. Hover or tap any i for plain + technical definitions.

Key findings

What this sample shows — written twice: first for everyone, then for specialists.

  1. The two AI models agree on structure, not on raw numbers. On average their paired spot scores only match at cosine 0.030, but a geometry test (CKA=0.78) shows they still organize similar-looking spots similarly. Think of two languages describing the same neighborhood map. See chart →
    mean paired cosine=0.0295; linear CKA=0.7796; Mantel ρ(model distances)=0.681 (p=0.00e+00); k-NN@10 agreement=0.457.
  2. Image patterns show moderate coupling to gene activity. Spots that look similar under the microscope tend to have somewhat similar gene profiles (coupling score ≈ 0.47 on a −1…1 scale). Morphology does not fully determine expression here. See chart →
    Mantel Spearman ρ between H-Optimus-0 pairwise distances and expression-PC (1–5) distances = 0.466 (p=0.00e+00, n_pairs=9453).
  3. The main image axis tracks how many RNA molecules were captured. The strongest image pattern (PC1) correlates with library size (r=0.66). Cellularity and technical capture efficiency can dominate unsupervised maps — always check this. See chart →
    Pearson r(H-Optimus-0 PC1, total UMI)=0.657 (p=2.09e-18). Consider regressing library size before biological claims.
  4. Embeddings are not just ‘where on the slide’. Similarity in AI space only weakly follows physical distance on the tissue (score ≈ 0.11). The models encode appearance, not pure x/y coordinates — good for 138 Visium spots here. See chart →
    Mantel ρ(H-Optimus-0 distances, Euclidean spatial)=0.109 (p=2.54e-26).
  5. Unsupervised tissue domains partially match across models. When we auto-group spots into 4 look-alike regions, the two models agree more than chance (agreement index ARI=0.48; 1.0 would be identical partitions). See chart →
    k-means k=4; Adjusted Rand Index=0.4841 between H-Optimus-0 and UNI2-h labels.
Tissue spots
n_spots
138
Genes measured
n_genes
15997
AI fingerprint size
embedding_dim
1536
Raw score match
mean cosine
0.0295
Shared mental map
linear CKA
0.7796
Local neighborhood match
NN@10 agreement
0.457
Region agreement
cluster ARI
0.484
Avg RNA per spot
mean UMI
6260.9

0. Why pathology foundation models — and why not “just use Grok”?

The short version: these models exist to measure what tissue looks like at scale. Chat models exist to reason and write. You usually want both — in different seats of the same lab stack.

For everyone

A whole-slide image can be tens of thousands of pixels on a side — far too large to “chat” with directly. Labs cut the slide into thousands of small tiles and need a repeatable score for each tile (how similar is this region to that one? does it look inflamed? tumor-like?).

UNI2-h and H-Optimus-0 are specialist vision models trained on enormous collections of pathology slides. Their job is not to write a report; it is to turn each tile into a list of numbers (an embedding) that later math can use — clustering, nearest-neighbor search, predicting labels with a tiny linear model, linking to gene expression (this dashboard).

A general model like Grok 4.5 (or other frontier LLMs) is brilliant at reading papers, drafting analysis plans, writing the Python that builds this page, and explaining CKA in plain English. It is the wrong tool to embed every Visium-aligned H&E tile on a multi-gigapixel WSI with the stability and domain priors pathology benchmarks demand.

For specialists

Pathology FMs are typically ViT / DINOv2-style self-supervised models trained on 10⁵–10⁶+ WSIs (Mass-100K lineage for UNI; Bioptimus multi-cohort slides for H-Optimus). Outputs are fixed-width vectors (here 1536-d) used as features for linear probes, kNN, MIL (ABMIL/TransMIL), retrieval, and multimodal fusion.

Evaluation is not chat win-rate. Common protocols:

  • Patch / ROI classification — few-shot linear probe or kNN on frozen embeddings
  • Slide-level MIL — aggregate tiles for cancer subtype, grade, biomarker status
  • Retrieval — nearest slides/tiles by embedding distance
  • HEST-1k-style tasks — predict expression / gene programs from morphology; compare FMs under a shared tile grid
  • Mutation / survival proxies — weakly supervised signals from H&E (cohort-dependent)

Frontier LLMs (Grok, GPT, Claude, Gemini, …) dominate language, tool use, and code benchmarks. Multimodal LLMs can discuss a single uploaded micrograph but do not replace a tile-grid embedding service with documented stain/resolution assumptions and offline batch throughput.

When to use which

  • Use UNI2-h / H-Optimus to featurize H&E tiles, compare morphology spaces, train cheap probes, retrieve look-alike regions, fuse with spatial omics.
  • Use Grok 4.5 (or similar LLMs) to design the study, critique QC, write pipelines, interpret metrics for stakeholders, draft figure legends.
  • Use both in a loop: FM embeddings → quantitative plots → LLM-assisted narrative and next experiments (this repo’s workflow).
  • Do not treat chat answers as a substitute for held-out cohort evaluation or pathologist review.

Licenses matter: UNI2-h is research / non-commercial (CC-BY-NC-ND lineage); H-Optimus-0 access is gated by Bioptimus terms. General LLMs have their own API terms — different compliance surface.

Capability map

Job Pathology FM
UNI2-h · H-Optimus-0
Frontier LLM
e.g. Grok 4.5
Classic vision
ImageNet ResNet/ViT
Turn every tile into a stable vector Primary tool — built for frozen 1536-d features Poor fit — chat/multimodal APIs are not WSI batch embedders Works, but large domain shift from photos → H&E
Few labels, many slides Strong — linear probes / kNN on frozen emb. Can suggest methods; does not replace the probe train loop Weaker transfer; needs more labels or fine-tuning
Link morphology ↔ spatial genes (HEST-style) Designed for this class of benchmark Can help interpret correlations; cannot produce tile features alone Baseline only; usually loses to pathology-pretrained FMs
Whole-slide diagnosis / MIL Standard backbone under ABMIL/TransMIL etc. Not a slide-level classifier out of the box Older pipelines; often surpassed by pathology FMs
Write code, critique stats, explain to non-experts No language interface Primary tool — reasoning + prose + tools No language interface
Literature Q&A, protocol design Out of scope Primary tool Out of scope
Typical public benchmarks Patch classification & retrieval; MIL slide tasks; HEST-1k ST benchmarks; internal mutation/survival probes MMLU-class knowledge, coding, agentic tool use, LMSYS-style chat arenas — not HEST leaderboards ImageNet / COCO lineage — wrong domain unless fine-tuned
What “good” looks like Higher probe AUC/F1, better retrieval mAP, stronger gene-prediction R² under fixed tiles Correct reasoning, lower hallucination, better code that reproduces analyses Top-1 accuracy on natural images (mostly irrelevant here)

This demo does not re-run public leaderboards; it shows a reproducible dual-FM audit on one HEST sample. For absolute SOTA claims, see the UNI/UNI2 papers (Mahmood Lab), Bioptimus H-Optimus model cards, and the HEST-1k benchmark suite — always check the exact task, freeze vs fine-tune, and cohort leakage rules.

Why two pathology models?

If only one FM is used, every downstream plot inherits that model’s quirks (stain sensitivity, texture bias, collapse modes). Running H-Optimus-0 and UNI2-h on the same tiles lets us ask: is a pattern morphology-real or model-specific? High CKA / Mantel with low paired cosine (as on this sample) is the classic signature of shared geometry, different coordinate systems — evidence that the signal is not a single network’s arbitrary rotation.

Where Grok-class models still win

Frontier LLMs remain the best interface for scientific work about these embeddings: choosing metrics, writing this dashboard, stress-testing claims, and translating ARI=0.48 for a non-technical audience. Think of pathology FMs as instruments (like a sequencer) and Grok 4.5 as the scientist-operator who designs the experiment and tells the story — not as competitors for the same leaderboard.

1. How the experiment works

Plain path: tissue photo → small squares → AI fingerprints → compare to genes.

Data

HEST-1k pairs H&E WSIs with Visium spatial transcriptomics. We use sample NCBI10:

  • WSI pyramid: 7 levels, level-0 8073×8593
  • Microns/pixel: 0.8519
  • Spot diameter (level-0 px): 131
  • Expression matrix: 138 × 15997 (sparsity 0.8102)

Tile extraction

For each Visium centroid, a square crop of side ≈ spot diameter is taken from the WSI and resized to 224×224 for both models (standard FM input). Coordinates are stored in level-0 pixel space for spatial plotting.

  • H-Optimus-0: ImageNet-style mean/std specialized for H&E
  • UNI2-h: model pretrained_cfg transform via timm
  • Inference: CPU, batch-wise, torch.inference_mode()

Analysis metrics

  • Linear CKA — representation similarity invariant to isotropic scaling / orthogonal transform
  • k-NN agreement — local neighborhood overlap in cosine space
  • Mantel (Spearman) — rank correlation of pairwise distance matrices (models, space, expression)
  • k-means + silhouette / ARI / NMI — discrete partition concordance
  • Expression PCs — log1p(CP10k) on top-variance genes, then PCA

H-Optimus-0

OrganizationBioptimus
Parameters~1.1B
ArchitectureViT (DINOv2-style)
Output dim1536
HF repobioptimus/H-optimus-0
Mean L2 norm30.995
PC1 / cum. PC1–517.86% / 44.8%
Silhouette (k=4)0.229

UNI2-h

OrganizationMahmood Lab / Harvard-BWH
Parameters681M
ArchitectureViT-H/14 DINOv2 + reg tokens
Output dim1536
HF repoMahmoodLab/UNI2-h
Mean L2 norm13.72
PC1 / cum. PC1–518.84% / 42.48%
Silhouette (k=4)0.182

2. Is the data healthy enough to trust?

Quality checks on gene capture and basic image–model agreement per spot.

Top genes by total counts

Mitochondrial % is zero for all spots in this HEST export (gene annotation flag), so mito QC is non-informative here. Sparsity = fraction of zero entries in the count matrix.

3. Where on the tissue do patterns live?

Each dot is one measurement spot — click any point to open its H&E tile. Try different colors.

Why do we see four separate blobs?

Those are four separate pieces of tissue sitting on one microscope slide (not four random “AI clusters”). The lab only measured genes inside tissue, so empty glass has no dots. Within each piece, dots are Visium spots — small locations where RNA was captured.

UMI counts

Plain: roughly “how much RNA did we catch here?” More yellow = more molecules counted at that spot (often denser or more active tissue, or better capture).

Tech: total unique molecular identifiers / library size per Visium spot.

Model agreement

Plain: do the two AIs give a similar fingerprint for the same tile? Near 0 = different numeric scales (common); higher = more alike. Compare with CKA on the whole set for “same map geometry.”

Tech: per-spot cosine similarity between H-Optimus-0 and UNI2-h embeddings.

Axes on this map

X / Y = position on the slide photo (pixels of the thumbnail). Color = the metric you pick in the dropdown — not a third spatial axis.

Colorbar on the right: low (purple) → high (yellow) for continuous metrics.

X/Y: location on the H&E photo. Color: currently UMI counts (how much RNA).

Selected spot

Click a marker on the map (or on UMAP / 3D plots) to load that region’s H&E tile and a plain-language readout.

4. What patterns do the AI models notice?

2D maps of AI fingerprints. Prefer the 3D views to rotate and explore. Every plot below has axis notes under the figure.

X: principal component index (PC1 = strongest direction of variation). Y: % of embedding variance explained. Tall first bars = one dominant pattern (often overall cellularity/stain).

X: number of PCs kept. Y: cumulative % variance. Faster rise = fewer axes capture most of the signal.

X/Y: UMAP axes (unitless layout — only relative nearness matters). Color: UMI (RNA amount). Nearby points = tiles the H-Optimus-0 model treats as similar-looking.

X/Y: UMAP axes for UNI2-h (not aligned to H-Optimus-0 axes). Color: UMI. Compare shape of the cloud to the other model, not absolute X/Y values.

X: PC1 (largest linear direction in H-Optimus-0 space). Y: PC2. Color: UMI. If yellow tracks PC1, the main image axis is tied to library size.

X/Y: UNI2-h PC1 vs PC2. Same reading as left: continuous color = RNA amount, not “class labels.”

X: pairwise distance between spots in embedding space (after unit-normalizing). Y: how many spot-pairs fall in that bin. Tight peak = spots look more similar to each other overall.

Geometric reading

A steep PCA spectrum (large PC1 fraction) implies a dominant global mode — often stain intensity, tissue vs background residual, or bulk cellularity. Flatter spectra suggest more multi-factor morphological diversity.

UMAP is for visualization only; cluster labels and Mantel tests are computed in the original 1536-d (or PCA) spaces. Pairwise distance histograms characterize manifold concentration.

L2-norm differences across models are expected — compare structure (CKA, Mantel, NN), not raw norms.

4b. Explore the patterns in 3D

Drag to rotate · scroll to zoom · change color with the dropdown. Axis notes are on each card.

Axes UMAP1–3: unitless layout of H-Optimus-0 fingerprints (only distances matter). Color: dropdown metric. Click a point to open its H&E tile.

Axes UMAP1–3: UNI2-h layout (not the same coordinates as H-Optimus-0). Color: same dropdown — compare cloud shape, not absolute XYZ.

X/Y/Z: PC1–PC3 (largest linear directions). Ordered by variance: PC1 carries the most structure.

X/Y/Z: UNI2-h PC1–PC3. Same interpretation as left; axis signs can flip between models.

X/Y/Z: gene-expression PCs (not image). Color by morphology to see if RNA programs track look-alike groups.

X/Y: real slide position (pixels). Z: usually UMI (RNA height). Four towers = four tissue pieces; taller peaks = more RNA.

X: H-Optimus-0 PC1. Y: UNI2-h PC1. Z: model agreement. High Z = AIs more aligned on that spot.

Reading 3D embeddings

  • UMAP 3D — neighborhood sketch for humans; not used for Mantel/CKA math.
  • PCA 3D — linear, ordered axes (PC1 ≥ PC2 ≥ PC3).
  • Spatial relief — geography of the slide with RNA (or other metric) as height.
  • Color is independent of the three position axes — always check the colorbar title.

Drag to rotate · scroll to zoom · click points for tiles.

5. Do the two AI models agree?

Side-by-side tests of whether H-Optimus-0 and UNI2-h see the same structure.

Interpretation guide

  • Linear CKA ≈ 1: representations related by linear transform; ≈ 0: unrelated.
  • Mean cosine on paired spots can be low even when geometry is similar after rotation — prefer CKA / Mantel for geometry.
  • NN agreement: local retrieval consistency (important for spot-neighborhood methods).
  • Mantel ρ (models): global ranking of pairwise dissimilarities agrees.
  • ARI / NMI: discrete cluster partitions agree beyond chance.

Low paired cosine with moderate CKA is common when models live in different cones of Rd but share relative structure.

6. Do image patterns match gene patterns?

Comparing microscope appearance with measured RNA programs.

Expression features: log1p(counts-per-10k) on the 2,000 highest-variance genes, then PCA. Heatmaps show Pearson r between morphology PC1–5 and expression PC1–5. Strong off-diagonal or PC1–PC1 coupling would indicate that the dominant morphological axis tracks the dominant transcriptional program (e.g., proliferation, immune infiltrate, stroma).

7. Auto-discovered tissue regions

Four look-alike groups per model, painted back onto the tissue.

8. What this does not prove (and what to try next)

Findings framing

With n=138 spots on a single sample, this is a methodological quickstart, not a population-scale benchmark. The value is in the analysis recipe: dual-FM embedding, multi-metric alignment, and explicit morphology–expression tests that can scale to full HEST-1k cohorts.

Limitations

Natural extensions

References (selected)

Generated by scripts/build_dashboard.py · analytics snapshot also written to outputs/NCBI10_phd_analytics.json · research / non-clinical use only.