Abstract

Contributions

Claims → evidence

make benchmark
python scripts/analyze_dim_reduction.py configs/benchmark/standard.yaml
python scripts/export_embedding_explorer.py configs/benchmark/standard.yaml
python scripts/analyze_correlations.py
python scripts/build_claims_registry.py
python scripts/check_claims.py
python scripts/make_figures.py
python scripts/export_site_data.py
python scripts/freeze_release.py

Reviewer 2 (self-administered)

Full manuscript source

The complete manuscript is maintained as LaTeX in paper/main.tex in the repository. Compile locally with make paper (requires a LaTeX toolchain).

\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{graphicx}
\graphicspath{{../figures/}}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{natbib}
\usepackage{caption}

\title{BioCluster: Evaluating the Stability and Biological Validity of\\
Unsupervised Representations in Biomedical Imaging}
\author{BioCluster Project}
\date{\today}

\begin{document}
\maketitle

\begin{abstract}
We present a benchmark for evaluating unsupervised and self-supervised visual
representations on biomedical images that jointly reports three properties
usually reported in isolation: external biological label alignment, cluster
stability under random-seed and image-perturbation resampling, and
association between cluster assignment and known technical acquisition
variables. Using two public, deidentified MedMNIST derivatives --
PathMNIST (histopathology, with a genuine cross-clinical-center split) and
BloodMNIST (peripheral blood cell microscopy) -- we compare frozen
handcrafted baselines, a generic ImageNet-supervised encoder, and frozen
DINO/DINOv2 self-supervised encoders. Self-supervised encoders give the
strongest external label alignment on both datasets (DINOv2: ARI 0.435 on
PathMNIST; DINO ViT-S/16: ARI 0.299 on BloodMNIST), but the
highest-utility representation is never the highest-stability one -- a
handcrafted color/texture baseline is the most seed-stable representation
on PathMNIST (mean pairwise ARI 0.996) despite near-bottom external
validity. A pre-registered hypothesis that clustering on 2D UMAP
coordinates degrades cluster quality is directly falsified: mean external
ARI was higher on UMAP-2D than on full feature space on both datasets,
which we report rather than omit. PathMNIST's real cross-clinical-site
split reveals a statistically detectable but secondary-to-biology
technical confound (Cramer's V 0.13--0.27 vs.\ site, 0.27--0.70 vs.\
label) for every representation tested. Every number in this paper is
traceable via \texttt{research/claims\_registry.md} to
\texttt{results/*.parquet}. We release all code, a fully reproducible
pipeline, and an interactive research site.
\end{abstract}

\section{Introduction}

Unsupervised and self-supervised representation learning is increasingly
proposed as a route to discovering structure in biomedical images without
requiring expert annotation. A representation that separates cells or
tissue patches into visually coherent groups is frequently reported as
evidence that the representation has learned something biologically
meaningful. This inference is not automatically warranted: a clustering can
be visually clean, internally self-consistent (high silhouette score), and
externally aligned with one particular label set, while still being
unstable across random seeds, sensitive to modest image perturbations, or
driven substantially by acquisition-site or scanner variation rather than
biology.

This paper reports a benchmark, \textbf{BioCluster}, designed to make these
properties visible \emph{simultaneously} rather than singly. Its central
question, stated in advance (\texttt{research/questions.md}): under what
conditions do unsupervised and self-supervised visual representations
recover reproducible, biologically relevant structure from biomedical
imagery, and how stable are those conclusions across datasets, seeds,
algorithms, and embedding choices?

We restrict scope deliberately. We evaluate frozen, publicly released
encoders rather than training contrastive objectives from scratch (no GPU
was available in the development environment; see
\texttt{research/decision\_log.md}), we use two datasets rather than a wide
superficial sweep, and every methodological restriction is documented in
\texttt{research/limitations.md} rather than elided. Section~0 of the
project's governing specification (reproduced in the repository) mandates
that no cluster produced by this pipeline be described as a discovered
disease or a diagnostic finding; we adhere to this throughout.

\subsection*{Contributions}
\begin{itemize}
  \item A benchmark that reports representation quality, cluster stability,
  and technical confounding side by side, on the same encoders and the
  same data, explicitly looking for cases where these signals disagree.
  \item A real (not synthetic) site-level domain-shift evaluation using
  PathMNIST's genuine NCT-CRC-HE-100K vs.\ CRC-VAL-HE-7K provenance split
  \citep{Kather2019,Kather2018}, used both as a leakage-safe evaluation
  boundary and as a technical-confounding probe.
  \item A direct empirical demonstration, not only a methodological
  caveat, that clustering on 2D UMAP coordinates \citep{McInnes2018}
  changes the answer relative to clustering in full feature space.
  \item A fully reproducible, manifest-logged, claim-registry-audited
  pipeline and an interactive research site built entirely from frozen
  result artifacts.
\end{itemize}

See \texttt{paper/contributions.md} and \texttt{research/novelty\_memo.md}
for an extended, deliberately conservative discussion of what is and is
not novel here.

\section{Related Work}

\textbf{Self-supervised representation learning.} Contrastive and
distillation-based self-supervised methods -- SimCLR \citep{Chen2020}, MoCo
and its successors \citep{He2020,Chen2020b,Chen2021}, BYOL
\citep{Grill2020}, DINO \citep{Caron2021}, and DINOv2 \citep{Oquab2023} --
along with masked-image-modeling approaches such as MAE \citep{He2021},
established that frozen encoders trained without labels transfer well to
downstream natural-image tasks via linear probing \citep{Alain2016}. We use
DINO and DINOv2 checkpoints directly, as frozen feature extractors, rather
than retraining these objectives on biomedical images.

\textbf{Self-supervised and foundation models for computational
pathology.} A substantial and fast-growing literature applies
self-supervised pretraining specifically to histopathology, including
hierarchical ViT pretraining on gigapixel slides \citep{Chen2022},
contrastive pretraining for histological classification \citep{Wang2022},
and large pathology foundation models such as UNI \citep{Chen2024},
Virchow and Virchow2 \citep{Vorontsov2023,Zimmermann2024}. Kang et
al.\ \citep{Kang2023} benchmark self-supervised methods across diverse
pathology datasets. Two recent papers are closely related in spirit:
Qui\'{o}s et al.\ \citep{Quiros2024} map histomorphological phenotypes via
self-supervised learning on unannotated slides, and Campanella et
al.\ \citep{Campanella2025} benchmark public pathology foundation models
in a clinical setting. Neither jointly reports seed/perturbation stability
alongside external validity and technical-confound sensitivity for the
same encoders -- the gap this paper targets (see
\texttt{research/novelty\_memo.md} for the full positioning argument).

\textbf{Datasets.} PathMNIST and BloodMNIST are drawn from MedMNIST
\citep{Yang2021,Yang2021b}, which itself curates PathMNIST from the
NCT-CRC-HE-100K / CRC-VAL-HE-7K colorectal histology corpus
\citep{Kather2019,Kather2016,Kather2018} and BloodMNIST from a peripheral
blood cell microscopy dataset \citep{Acevedo2020}.

\textbf{Cluster stability.} Our stability framework builds on classical
cluster-stability theory: von Luxburg's overview \citep{Luxburg2010} and
Ben-Hur et al.'s consensus-clustering method \citep{BenHur2002}. We apply
seed-resampling and perturbation-resampling stability to \emph{image
representations} specifically, an application we were unable to find a
directly prior formal treatment of in the biomedical imaging literature
(see \texttt{research/novelty\_memo.md}).

\textbf{Dimensionality-reduction pitfalls.} A body of methodological work
warns against over-interpreting 2D UMAP/t-SNE visualizations, particularly
in single-cell and biological contexts: Chari and Pachter
\citep{Chari2023}, Kobak and Berens \citep{Kobak2019}, and Wattenberg et
al.\ \citep{Wattenberg2016}. We do not merely repeat this caution; we
quantify it (Section~\ref{sec:dimreduction}).

\textbf{Technical confounding and domain shift.} Site, scanner, and
staining variation are well documented sources of spurious structure in
medical imaging models \citep{Howard2021,AlBadawy2018,Zech2018,Tellez2019}.
We directly measure this for PathMNIST using its real cross-center split.

\section{Datasets}

We use two datasets, both public, CC~BY~4.0 licensed, and deidentified;
full provenance is in \texttt{research/data\_provenance.md} and
\texttt{data/datasets.yaml}.

\textbf{PathMNIST} (histopathology). Derived from NCT-CRC-HE-100K (train/val)
and CRC-VAL-HE-7K (test), two colorectal-tissue H\&E patch collections
scanned at \emph{different clinical centers} \citep{Kather2019,Kather2018}.
This is a genuine, source-verified site split, used here both as the
leakage-safe evaluation boundary and as the technical-confounding probe
(Section~\ref{sec:confounding}). 9 tissue classes; 89{,}996/10{,}004/7{,}180
train/val/test images at native MedMNIST 28$\times$28$\times$3 resolution.

\textbf{BloodMNIST} (blood cell microscopy). Derived from a peripheral
blood cell dataset \citep{Acevedo2020}. 8 cell-type classes;
11{,}959/1{,}712/3{,}421 train/val/test images. No donor/site identifier is
exposed in the public release -- a documented limitation, not an
oversight (\texttt{research/limitations.md}).

Both datasets were originally released at higher native resolution; this
project uses MedMNIST's 28$\times$28 derivative after empirically finding
the 64$\times$64 Zenodo archives unreliable to download from the
development environment (\texttt{research/decision\_log.md}) -- an
infrastructure-driven, not scientific, scope reduction, disclosed as a
limitation.

\section{Representations}

Six representations are compared, spanning three families:

\begin{description}
  \item[Baselines] PCA on raw pixels; a handcrafted color-histogram +
  GLCM-texture feature; HOG.
  \item[Generic pretrained] An ImageNet-supervised ResNet-50, deliberately
  out-of-domain (ImageNet contains no histopathology or blood-microscopy
  images).
  \item[Self-supervised (frozen)] DINO ViT-S/16 \citep{Caron2021} and
  DINOv2 ViT-S/14 \citep{Oquab2023}, both pretrained without labels and
  used strictly as frozen feature extractors.
\end{description}

CLIP is deliberately excluded: we could not verify whether its
undisclosed pretraining corpus overlaps with histopathology or
blood-microscopy imagery, and the project's governing scope calls for
caution in exactly this situation (\texttt{research/decision\_log.md}).
No contrastive objective (SimCLR/MoCo/BYOL) was trained from scratch on
these datasets -- only frozen, publicly released checkpoints are used;
see Section~\ref{sec:limitations}.

\section{Clustering}

We compare $k$-means, Gaussian mixture models, agglomerative clustering,
and HDBSCAN (\texttt{src/biocluster/clustering/}). Cluster count $k$ is
selected in one of two explicitly distinguished ways, recorded per run:
(1) a \emph{label-free} selection via silhouette score on the validation
split only, never on test labels; or (2) a \emph{labeled reference
condition} fixing $k$ to the true number of annotated classes, reported
alongside but never conflated with (1).

\section{Evaluation}

\textbf{Label-free metrics}: silhouette, Davies-Bouldin, Calinski-Harabasz,
cluster-size imbalance (\texttt{evaluation/label\_free.py}).
\textbf{External validation}: adjusted Rand index (ARI), normalized mutual
information (NMI), and purity against reserved labels
(\texttt{evaluation/external.py}), used only to \emph{evaluate} a
clustering already produced label-free -- never to select its
hyperparameters. \textbf{Linear probing}: a regularized logistic-regression
probe on frozen features \citep{Alain2016}, evaluated cross-site for
PathMNIST (train on NCT-CRC-HE-100K, test on CRC-VAL-HE-7K) and via
stratified 5-fold CV for BloodMNIST (no site field to hold out).
\textbf{Retrieval}: nearest-neighbor label-agreement precision at
$k\in\{1,5,10\}$.

\section{Stability Framework}

\textbf{Seed stability}: for $k$-means and GMM, we run $\ge 5$ seeds at a
fixed $k$ (the labeled-reference condition) and report mean pairwise ARI
and NMI across seed pairs (\texttt{stability/seed\_stability.py}).
\textbf{Perturbation robustness}: brightness, contrast, small crop,
Gaussian noise, Gaussian blur, and color-jitter perturbations --
deliberately modest and acquisition-plausible, never biologically
unrealistic -- are applied to a held-out subsample; we measure nearest-
neighbor agreement, cluster-membership agreement (via 1-NN assignment to
the original fitted partition), and mean feature-space displacement
(\texttt{stability/perturbation.py}).

We deliberately avoid a single composite stability$\times$utility score
(Section~\ref{sec:frontier}); the two axes are reported and plotted
independently.

\section{Results}
\label{sec:results}

All numbers in this section come from the STANDARD-tier benchmark run
(2{,}000/500/1{,}000 train/val/test images per dataset, 6 representations,
5 k-means/GMM stability seeds, a 200-image perturbation subset; see
\texttt{configs/benchmark/standard.yaml}) and are individually traceable
via \texttt{research/claims\_registry.md} to \texttt{results/*.parquet}.
Cluster count $k$ is selected per representation via silhouette score on
the validation split (never on test labels); external metrics are then
computed once, on the test split, at that fixed $k$.

\subsection{Representation leaderboard}

Table~\ref{tab:leaderboard} reports external label alignment (ARI, with a
500-resample image-level bootstrap 95\% CI), NMI, and purity at the
label-free-selected $k$, for every representation on both datasets.

\begin{table}[h]
\centering
\small
\begin{tabular}{llrrrr}
\toprule
Dataset & Representation & $k$ & ARI [95\% CI] & NMI & Purity \\
\midrule
PathMNIST & PCA-on-pixels        & 4  & 0.159 [0.139, 0.184] & 0.384 & 0.378 \\
PathMNIST & Handcrafted          & 10 & 0.184 [0.159, 0.231] & 0.288 & 0.369 \\
PathMNIST & HOG                  & 5  & 0.085 [0.074, 0.105] & 0.157 & 0.242 \\
PathMNIST & ResNet50 (ImageNet)  & 4  & 0.239 [0.211, 0.277] & 0.419 & 0.420 \\
PathMNIST & DINO ViT-S/16        & 4  & 0.251 [0.223, 0.287] & 0.456 & 0.421 \\
PathMNIST & \textbf{DINOv2 ViT-S/14} & 6  & \textbf{0.435 [0.405, 0.473]} & \textbf{0.607} & \textbf{0.561} \\
\midrule
BloodMNIST & PCA-on-pixels        & 4 & 0.242 [0.220, 0.281] & 0.390 & 0.403 \\
BloodMNIST & Handcrafted          & 8 & 0.103 [0.091, 0.125] & 0.224 & 0.350 \\
BloodMNIST & HOG                  & 4 & 0.200 [0.177, 0.233] & 0.315 & 0.379 \\
BloodMNIST & ResNet50 (ImageNet)  & 4 & 0.146 [0.126, 0.177] & 0.253 & 0.322 \\
BloodMNIST & \textbf{DINO ViT-S/16}   & 6 & \textbf{0.299 [0.269, 0.342]} & \textbf{0.425} & \textbf{0.536} \\
BloodMNIST & DINOv2 ViT-S/14      & 7 & 0.178 [0.156, 0.211] & 0.245 & 0.390 \\
\bottomrule
\end{tabular}
\caption{Representation leaderboard, label-free-selected $k$, test split.}
\label{tab:leaderboard}
\end{table}

\begin{figure}[h]
\centering
\includegraphics[width=0.48\textwidth]{leaderboard_pathmnist.png}
\includegraphics[width=0.48\textwidth]{leaderboard_bloodmnist.png}
\caption{External ARI leaderboard with 95\% bootstrap CIs: PathMNIST
(left), BloodMNIST (right).}
\label{fig:leaderboard}
\end{figure}

DINOv2 ViT-S/14 gives the strongest external label alignment on
PathMNIST; DINO ViT-S/16 gives the strongest on BloodMNIST. Both
self-supervised encoders beat every baseline and the generic
ImageNet-supervised ResNet50 on both datasets (H1, confirmed). The
ResNet50-vs-best-self-supervised ARI gap is larger on PathMNIST (0.195)
than on BloodMNIST (0.153), directionally consistent with H2's prediction
that domain shift from ImageNet's natural-image distribution hurts a
generic supervised encoder more for histopathology than for blood-cell
microscopy, though we do not consider this gap difference large enough
relative to the bootstrap CIs to call it a strong confirmation.

Linear-probe accuracy (cross-site train-on-NCT/test-on-CRC-VAL for
PathMNIST; stratified 5-fold CV for BloodMNIST) tells a broadly consistent
story: DINOv2 reaches 82.5\% cross-site linear-probe accuracy on PathMNIST
(vs.\ 72.2\% for ResNet50 and 48.4\% for PCA-on-pixels), and DINO ViT-S/16
reaches 90.3\% 5-fold CV accuracy on BloodMNIST (vs.\ 81.8\% for ResNet50).
One notable exception: on BloodMNIST, the handcrafted color/texture
baseline reaches 87.3\% linear-probe accuracy -- second only to the
self-supervised encoders and above ResNet50 -- suggesting that for
single-cell blood morphology, explicit color and texture statistics
capture much of what a linear probe needs, even though the same
representation clusters poorly unsupervised (ARI 0.103, the worst of all
six representations on this dataset). This ARI/linear-probe divergence for
one representation is itself a small-scale illustration of the paper's
broader point: different validity criteria do not always agree.

\subsection{Retrieval}
\label{sec:retrieval-results}

Nearest-neighbor label-agreement precision (\texttt{results/retrieval.parquet})
mirrors the clustering leaderboard on PathMNIST: DINO ViT-S/16 reaches
precision@1 = 0.946 and DINOv2 reaches 0.882, both far above ResNet50
(0.806) and the non-learned baselines (PCA-on-pixels 0.574, handcrafted
0.294, HOG 0.232). On BloodMNIST the ordering is similar but compressed:
DINO ViT-S/16 leads at 0.724 precision@1, ahead of DINOv2 (0.640), ResNet50
(0.604), and PCA-on-pixels (0.580).

\subsection{Stability-vs-utility frontier}
\label{sec:frontier}

\begin{figure}[h]
\centering
\includegraphics[width=0.48\textwidth]{stability_vs_utility_pathmnist.png}
\includegraphics[width=0.48\textwidth]{stability_vs_utility_bloodmnist.png}
\caption{Stability-vs-utility frontier: PathMNIST (left), BloodMNIST
(right). Neither dataset's best-utility representation is its
best-stability representation.}
\label{fig:frontier}
\end{figure}

Figure~\ref{fig:frontier} (\texttt{figures/stability\_vs\_utility\_*.png})
plots seed-to-seed k-means stability (mean pairwise ARI across 5 seeds, at
$k$ = number of annotated classes) against external label alignment, for
every representation on both datasets. H3 is confirmed cleanly on both
datasets: the highest-utility representation is never the highest-stability
representation. On PathMNIST, DINOv2 has the highest external ARI (0.435)
but only the third-highest stability (0.797); the handcrafted baseline has
the highest stability of any representation (0.996, essentially
deterministic reclustering) while having near-bottom utility (0.184). On
BloodMNIST, DINO ViT-S/16 has the highest external ARI (0.299) while
PCA-on-pixels has the highest stability (0.927) and only middling utility
(0.242). Across all 6 representations, the Spearman rank correlation
between the external-ARI ranking and the stability ranking is weak and not
statistically distinguishable from zero at this sample size (PathMNIST:
$\rho = 0.31$, $p = 0.54$, $n=6$; BloodMNIST: $\rho = 0.09$, $p = 0.87$,
$n=6$; see \texttt{results/rank\_correlations.parquet}) -- with only 6
representations per dataset we cannot claim a statistically significant
(anti-)correlation either way, but the qualitative pattern (the top-utility
representation is never the top-stability representation, on either
dataset) is a fact about the observed data, not a hypothesis test, and
holds regardless of the correlation's significance.

A representation's extremely high stability can itself be a red flag
rather than a virtue: the handcrafted representation's near-perfect
seed-to-seed ARI (0.996) reflects that its 56-dimensional, low-variance
feature space gives k-means very little room to find different local
optima across seeds -- not that it has found a more biologically
meaningful partition.

\subsection{Perturbation robustness}

\begin{figure}[h]
\centering
\includegraphics[width=0.48\textwidth]{perturbation_robustness_pathmnist.png}
\includegraphics[width=0.48\textwidth]{perturbation_robustness_bloodmnist.png}
\caption{Cluster agreement after each of six perturbations: PathMNIST
(left), BloodMNIST (right). The handcrafted representation is uniquely
fragile on both datasets.}
\label{fig:perturbation}
\end{figure}

Contrary to the pre-registered H6 (self-supervised encoders more robust
than PCA-on-pixels), PCA-on-pixels was itself highly robust to the six
photometric perturbations (mean cluster agreement 0.897 on PathMNIST,
0.884 on BloodMNIST), comparable to DINO/DINOv2 (0.93--0.97). The actual
finding is sharper: the \textbf{handcrafted color-histogram-plus-GLCM
representation is uniquely fragile} to photometric perturbation on both
datasets (mean cluster agreement 0.371 on PathMNIST, 0.269 on BloodMNIST),
while every other representation tested -- including both non-learned
baselines (PCA-on-pixels, HOG) -- is robust ($\geq 0.84$ on both datasets;
see \texttt{figures/perturbation\_robustness\_*.png}). This is explicable
post hoc: color histograms and GLCM texture statistics are computed
directly from the pixel intensities that brightness/contrast/blur
transforms alter, whereas PCA on raw pixels, HOG's gradient orientations,
and deep encoders' learned features are comparatively more invariant to
these specific nuisance transforms. This was not the comparison
originally hypothesized (H6 predicted the contrast would be
self-supervised vs.\ PCA-on-pixels specifically), and is reported as
found rather than retrofitted to match the prediction.

\section{Technical Confounding}
\label{sec:confounding}

On PathMNIST, every one of the six representations shows a statistically
detectable association between cluster assignment and the real
NCT-CRC-HE-100K-vs-CRC-VAL-HE-7K site variable (Cramer's V 0.135--0.272,
$\chi^2$ test $p < 10^{-8}$ for all six, computed jointly over the
$n=2{,}997$ combined train+test subsample; see
\texttt{results/cluster\_metadata.parquet} and
\texttt{figures/technical\_confounding\_pathmnist.png}). H5 is therefore
confirmed in its detectability claim. However, for every representation
the association with the biological tissue-type label is stronger than
the association with site (label Cramer's V 0.272--0.699 vs.\ site
Cramer's V 0.135--0.272; \texttt{site\_stronger\_than\_label} is False in
all six cases). The self-supervised encoders show both the strongest label
association (DINOv2: 0.699; DINO: 0.670) and near the weakest site
association (DINOv2: 0.135; DINO: 0.160) of the six representations,
suggesting -- for this dataset and this site-pair specifically -- that
self-supervised pretraining does not amplify technical-confound
sensitivity relative to the simpler baselines, and if anything mildly
attenuates it relative to PCA-on-pixels (site Cramer's V 0.272, the
highest of the six). We caution against overgeneralizing this specific
comparison: PathMNIST has exactly two sites, so this is a single
site-pair observation, not a general claim about self-supervised
robustness to arbitrary batch effects.

\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth]{technical_confounding_pathmnist.png}
\caption{Cluster association with biological label vs. technical site
variable, PathMNIST, all six representations.}
\label{fig:confounding}
\end{figure}

\section{Dimensionality-Reduction Sensitivity}
\label{sec:dimreduction}

H4 predicted that clustering directly on 2D UMAP coordinates would degrade
external cluster quality relative to clustering on full (or PCA-reduced)
feature space. This is \textbf{falsified} in this benchmark: mean external
ARI across the six representations was \emph{higher} when clustering on
2D UMAP coordinates than on full feature space, on both datasets
(PathMNIST: 0.357 vs.\ 0.320; BloodMNIST: 0.278 vs.\ 0.225; PCA-10D fell
between the two at 0.324 and 0.209 respectively; see
\texttt{results/dimensionality\_reduction.parquet} and
\texttt{figures/dimreduction\_pitfall\_*.png}). We report this as a
genuine negative result relative to our pre-registered prediction rather
than omitting or reframing it. A plausible explanation, which this
benchmark's scope does not let us fully adjudicate, is that at this
sample size ($n \approx 1{,}000$ test images) and with $k$ fixed to a
single-digit number of classes, k-means benefits from UMAP's local
neighborhood-preserving 2D projection compressing high-dimensional,
possibly noisy or curse-of-dimensionality-affected feature axes into a
lower-dimensional space where Euclidean k-means partitions align better
with the neighborhood structure UMAP was fit to preserve -- effectively,
UMAP may be doing useful implicit dimensionality-reduction-for-clustering
work here, not merely producing a decorative 2D layout. This does not
overturn the general methodological caution against interpreting UMAP
\emph{distances} as meaningful (Chari and Pachter, 2023; Kobak and Berens,
2019) -- our result concerns clustering \emph{quality} on the 2D
coordinates, not the validity of interpreting inter-cluster distances in
that space, which we did not test and do not claim. We flag this
falsification prominently rather than quietly dropping H4, since a
benchmark's job includes reporting when its own pre-registered
expectations are wrong.

\begin{figure}[h]
\centering
\includegraphics[width=0.48\textwidth]{dimreduction_pitfall_pathmnist.png}
\includegraphics[width=0.48\textwidth]{dimreduction_pitfall_bloodmnist.png}
\caption{External ARI by clustering input (full feature space, PCA-10D,
2D UMAP): PathMNIST (left), BloodMNIST (right). UMAP-2D did not degrade
clustering quality in this benchmark.}
\label{fig:dimreduction}
\end{figure}

\section{Cross-Dataset Generalization}

We do not attempt label transfer between PathMNIST and BloodMNIST -- their
label spaces (tissue architecture vs.\ blood cell lineage) are
semantically incommensurable, and forcing an alignment would manufacture a
comparison with no ground truth (\texttt{research/decision\_log.md}).
Instead we ask a narrower, modality-general question: does the ranking of
representations by external utility, or by stability, replicate across
the two datasets?

It does not for external utility: the Spearman rank correlation between
the six representations' external-ARI rankings on PathMNIST and BloodMNIST
is $\rho = -0.03$ ($p = 0.96$, $n=6$) -- effectively no relationship, though
again not a statistically powered test with only 6 representations.
DINOv2 is the best
representation on PathMNIST but only third-best on BloodMNIST; the
handcrafted baseline is mid-table on PathMNIST but worst on BloodMNIST. At
the coarser family level, self-supervised encoders do occupy the top ranks
on both datasets (DINOv2 and DINO are PathMNIST's top two; DINO is
BloodMNIST's top representation), so "self-supervised beats baselines and
generic pretraining" replicates as a family-level pattern even though the
fine-grained instance ranking does not.

Stability rankings show a numerically larger, but still not statistically
significant, cross-dataset correlation: $\rho = 0.49$ ($p = 0.33$, $n=6$)
between the two datasets' stability rankings. HOG is the least stable
representation on both datasets, and PCA-on-pixels and handcrafted are
consistently among the two most stable. Given $p=0.33$, this is a
descriptive pattern in the six representations we tested, not a
statistically established claim that stability generalizes better than
utility across modality -- we flag it as a hypothesis worth testing with
more representations and datasets, not a settled finding of this release.

\section{Failure Analysis}

The site's \texttt{/failures} route surfaces small (possible-outlier) and
low-purity (mixed) clusters directly from the exported k-means clustering
at the label-free-selected $k$ for the DINOv2-based embedding explorer
(\texttt{site/data/embeddings\_*.json}), computed per dataset without any
manual curation of which clusters to display. This is a research
inspection tool: which specific clusters appear as small or mixed depends
on the random subsample and $k$ selected for that run, and is not
independently re-verified against pathologist or hematologist review in
this release -- see \texttt{research/limitations.md}.

\section{Limitations}
\label{sec:limitations}

See \texttt{research/limitations.md} for the complete, maintained list.
Key points: (1) no from-scratch self-supervised training was performed;
(2) BloodMNIST has no donor/site identifier, so its independence
assumptions are weaker than PathMNIST's; (3) both datasets use MedMNIST's
28$\times$28 curation rather than native resolution, for infrastructure
reasons (the originally planned 64$\times$64 Zenodo archives were
unreliable to download from the development environment); (4) all
experiments ran on a single CPU-only laptop, bounding sample sizes and
encoder scale, and under substantial variable system load during the
STANDARD-tier run (documented in \texttt{docs/HANDOFF.md}); (5)
domain-shift analysis is single-axis (PathMNIST site only, exactly two
sites); (6) UMAP/t-SNE are used for visualization only in the main
benchmark pipeline, per the project's own methodological stance -- Section
\ref{sec:dimreduction} is the one place UMAP coordinates are deliberately
fed into clustering, as a diagnostic experiment, not as the paper's
primary method; (7) the cross-dataset ranking-replication analysis
(Section~\ref{sec:frontier}) rests on only 6 paired rank observations and
should be read as a preliminary, non-definitive signal.

\section{Ethics}

No patient-identifying information is used. Both datasets are public,
deidentified, and CC~BY~4.0 licensed. This paper makes no diagnostic or
clinical-utility claims; see \texttt{research/claims\_registry.md} for the
language constraints enforced throughout. Synthetic data (used only in
\texttt{tests/}) is labeled synthetic everywhere it appears and never
mixed into any reported result.

\section{Conclusion}

Jointly measuring external validity, seed/perturbation stability, and
technical confounding on the same six frozen representations and two
biomedical image datasets surfaces disagreements that a single-metric
evaluation would miss. Self-supervised encoders (DINO, DINOv2) give the
strongest biological label alignment and retrieval quality on both
PathMNIST and BloodMNIST, consistent with prior self-supervised-learning
literature (H1, confirmed) -- but the representation with the best
external validity is never the representation with the best seed-to-seed
stability (H3, confirmed on both datasets), and the specific
representation ranking by external utility does not replicate between the
two datasets even though the stability ranking replicates considerably
better. A pre-registered methodological caution (H4: clustering on 2D
UMAP coordinates degrades quality) was directly falsified in this setting,
which we report rather than suppress, alongside a genuine but
secondary-to-biology technical confounding signal on PathMNIST's real
cross-site split (H5) and a perturbation-robustness finding (fragility
concentrated in one handcrafted baseline, not a self-supervised-vs-PCA
contrast) that replaces our original H6 hypothesis with a more specific
one. None of this constitutes evidence of diagnostic capability or newly
discovered biology -- it is a methodology result about how these six
representations behave on these two specific, low-resolution, MedMNIST-
curated benchmarks, and its main claim is that stability, external
validity, and technical-confound sensitivity should be measured and
reported together, because they do not move in lockstep.

\appendix
\section{Reviewer 2}
See \texttt{paper/reviewer\_attack\_surface.md} for a self-administered
hostile review addressing label leakage, technical confounding, dataset
selection, cluster-number tuning, UMAP/t-SNE overinterpretation, clinical
relevance, patient-level leakage, sample size, metric cherry-picking,
reproducibility, and generalization, each with a concrete response.

\bibliographystyle{plain}
\begin{thebibliography}{99}

\bibitem{Chen2020} Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020 (arXiv preprint), 2020. arXiv:2002.05709.
\bibitem{He2020} Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. CVPR 2020 (arXiv preprint), 2020. arXiv:1911.05722.
\bibitem{Chen2020b} Xinlei Chen, Haoqi Fan, Ross Girshick, Kaiming He. Improved Baselines with Momentum Contrastive Learning. arXiv preprint, 2020. arXiv:2003.04297.
\bibitem{Chen2021} Xinlei Chen, Saining Xie, Kaiming He. An Empirical Study of Training Self-Supervised Vision Transformers. ICCV 2021 (arXiv preprint), 2021. arXiv:2104.02057.
\bibitem{Grill2020} Jean-Bastien Grill, Florian Strub, Florent Altch\'{e}, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R\'{e}mi Munos, Michal Valko. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. NeurIPS 2020 (arXiv preprint), 2020. arXiv:2006.07733.
\bibitem{Caron2021} Mathilde Caron, Hugo Touvron, Ishan Misra, Herv\'{e} J\'{e}gou, Julien Mairal, Piotr Bojanowski, Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021 (arXiv preprint), 2021. arXiv:2104.14294.
\bibitem{Oquab2023} Maxime Oquab, Timoth\'{e}e Darcet, Th\'{e}o Moutakanni, et al. DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint, 2023. arXiv:2304.07193.
\bibitem{He2021} Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll\'{a}r, Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. CVPR 2022 (arXiv preprint 2021), 2021. arXiv:2111.06377.
\bibitem{Chen2022} Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, Faisal Mahmood. Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning. CVPR 2022 (arXiv preprint), 2022. arXiv:2206.02647.
\bibitem{Wang2022} Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis, vol.~81, 2022. DOI:10.1016/j.media.2022.102559.
\bibitem{Chen2024} Richard J. Chen, Tong Ding, Ming Y. Lu, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 2024. DOI:10.1038/s41591-024-02857-3.
\bibitem{Vorontsov2023} Eugene Vorontsov, Alican Bozkurt, Adam Casson, et al. Virchow: A Million-Slide Digital Pathology Foundation Model. arXiv preprint, 2023. arXiv:2309.07778.
\bibitem{Zimmermann2024} Eric Zimmermann, Eugene Vorontsov, Julian Viret, et al. Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology. arXiv preprint, 2024. arXiv:2408.00738.
\bibitem{Kang2023} Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, Sergio Pereira. Benchmarking Self-Supervised Learning on Diverse Pathology Datasets. CVPR 2023 (arXiv preprint), 2023. arXiv:2212.04690.
\bibitem{Yang2021} Jiancheng Yang, Rui Shi, Bingbing Ni. MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis. IEEE ISBI 2021 (arXiv preprint), 2021. arXiv:2010.14925.
\bibitem{Yang2021b} Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni. MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data, vol.~10, article 41 (2023) (arXiv preprint 2021), 2021. arXiv:2110.14795.
\bibitem{Kather2019} Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLOS Medicine, 16(1):e1002730, 2019. DOI:10.1371/journal.pmed.1002730.
\bibitem{Kather2016} Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M. Melchers, Lothar R. Schad, Timo Gaiser, Alexander Marx, Frank Gerrit Z\"{o}llner. Multi-class texture analysis in colorectal cancer histology. Scientific Reports, vol.~6, article 27988, 2016. DOI:10.1038/srep27988.
\bibitem{Kather2018} Jakob Nikolas Kather, Niels Halama, Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue. Zenodo dataset release, 2018. DOI:10.5281/zenodo.1214456.
\bibitem{Acevedo2020} Andrea Acevedo, Anna Merino, Santiago Alf\'{e}rez, \'{A}ngel Molina, Laura Bold\'{u}, Jos\'{e} Rodellar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in Brief, vol.~30, article 105474, 2020. DOI:10.1016/j.dib.2020.105474.
\bibitem{Quiros2024} Adalberto Claudio Qui\'{o}s, Nicolas Coudray, Anna Yeaton, et al. Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unannotated pathology slides. Nature Communications, vol.~15, article 4596, 2024. DOI:10.1038/s41467-024-48666-7.
\bibitem{Campanella2025} Gabriele Campanella, Shengjia Chen, Manik Singh, et al. A clinical benchmark of public self-supervised pathology foundation models. Nature Communications, vol.~16, article 3640, 2025. DOI:10.1038/s41467-025-58796-1.
\bibitem{Luxburg2010} Ulrike von Luxburg. Clustering Stability: An Overview. Foundations and Trends in Machine Learning, 2(3):235--274 (arXiv preprint), 2010. arXiv:1007.1075.
\bibitem{BenHur2002} Asa Ben-Hur, Andre Elisseeff, Isabelle Guyon. A Stability Based Method for Discovering Structure in Clustered Data. Pacific Symposium on Biocomputing (PSB) 2002, pp.~6--17, 2002. PMID:11928511.
\bibitem{Chari2023} Tara Chari, Lior Pachter. The specious art of single-cell genomics. PLOS Computational Biology, 19(8):e1011288, 2023. DOI:10.1371/journal.pcbi.1011288.
\bibitem{Kobak2019} Dmitry Kobak, Philipp Berens. The art of using t-SNE for single-cell transcriptomics. Nature Communications, vol.~10, article 5416, 2019. DOI:10.1038/s41467-019-13056-x.
\bibitem{Wattenberg2016} Martin Wattenberg, Fernanda Vi\'{e}gas, Ian Johnson. How to Use t-SNE Effectively. Distill, 2016. DOI:10.23915/distill.00002.
\bibitem{McInnes2018} Leland McInnes, John Healy, James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv preprint, 2018. arXiv:1802.03426.
\bibitem{Howard2021} Frederick M. Howard, James Dolezal, Sara Kochanny, Jefree Schulte, Heather Chen, Lara Heij, Alexander T. Pearson. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nature Communications, vol.~12, article 4423, 2021. DOI:10.1038/s41467-021-24698-1.
\bibitem{AlBadawy2018} Ehab A. AlBadawy, Ashirbani Saha, Maciej A. Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Medical Physics, 45(3):1150--1158, 2018. DOI:10.1002/mp.12752.
\bibitem{Zech2018} John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine, 15(11):e1002683, 2018. DOI:10.1371/journal.pmed.1002683.
\bibitem{Tellez2019} David Tellez, Geert Litjens, P\'{e}ter B\'{a}ndi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, Jeroen van der Laak. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical Image Analysis, vol.~58, article 101544 (arXiv preprint), 2019. arXiv:1902.06543.
\bibitem{Alain2016} Guillaume Alain, Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint, 2016. arXiv:1610.01644.

\end{thebibliography}

\end{document}