BioCluster

Methodology

Methods, questions, and limitations

Research Questions

Central question

Under what conditions do unsupervised and self-supervised visual representations recover reproducible, biologically relevant structure from biomedical imagery, and how stable are those conclusions across datasets, seeds, algorithms, and embedding choices?

Subquestions and how BioCluster answers each

  1. Do self-supervised representations outperform handcrafted and generic pretrained features? Measured via external label alignment (ARI/NMI/purity against reserved labels) and linear-probe accuracy, compared across: PCA-on-pixels, color/ texture handcrafted features, HOG, generic ImageNet-pretrained ResNet50, and frozen self-supervised encoders (DINO ViT-S/16, DINOv2 ViT-S/14).

  2. Are discovered clusters reproducible across random seeds? Measured via consensus clustering and pairwise co-clustering agreement (ARI/NMI between seed pairs) for k-means and GMM (the stochastic-init algorithms in scope).

  3. Do cluster assignments remain stable under image perturbation? Measured via nearest-neighbor and cluster-membership agreement before/ after modest nuisance transforms (brightness, contrast, small crop, Gaussian noise, Gaussian blur, color jitter).

  4. How much does cluster quality depend on dimensionality-reduction choices? Clustering is run both on full-dimensional (whitened) features and on PCA-reduced features at multiple target dimensionalities; label-free and external metrics are compared across this choice.

  5. Do apparently clean UMAP/t-SNE visualizations correspond to actual separability? UMAP/t-SNE are used ONLY for visualization, never as clustering input. The embedding explorer displays an explicit warning to this effect, and the analysis separately reports whether visual UMAP separation agrees with label-free clustering metrics computed in the original feature space.

  6. Can representations transfer across datasets or acquisition sites? PathMNIST's own construction gives a real, pre-existing site split: the training/validation pool (NCT-CRC-HE-100K) and the test pool (CRC-VAL-HE-7K) were scanned at two different clinical centers. This is used as a genuine (not synthetic) domain-shift evaluation for the histopathology dataset. Cross-dataset transfer between PathMNIST and BloodMNIST is not attempted for label alignment, since the two datasets have disjoint, non-comparable label spaces (tissue type vs. blood cell type) — see research/decision_log.md for why this was ruled out rather than forced.

  7. Which imaging artifacts dominate latent structure? For PathMNIST, cluster assignment is tested for association with the known source-site variable (NCT vs. CRC-VAL). For BloodMNIST, no scanner/ site metadata is available in the public derived dataset (see research/data_provenance.md), so this axis is documented as untestable for that dataset rather than assumed absent.

  8. How should unsupervised biomedical discoveries be validated without circularly using labels? Clustering hyperparameters (k, min_cluster_size, etc.) are selected using only label-free criteria (silhouette / Davies-Bouldin on a held-out validation split) or fixed a priori to the known number of annotated classes as a documented, labeled reference condition — never by scanning over label-aligned metrics and picking the best. This distinction is recorded per-run in the benchmark manifest (see src/biocluster/utils/manifest.py).

Explicit non-goals

  • No diagnostic claims. No cluster is described as a "disease" or "diagnosis."
  • No training of self-supervised encoders from scratch (SimCLR/MoCo/BYOL) in this release — see research/decision_log.md for the compute-vs-rigor tradeoff that led to using frozen pretrained/self-supervised encoders instead. This is recorded as a documented scope limitation, not hidden.

Hypotheses

Stated in advance of running the full benchmark (H1-H6 fixed before make benchmark was run on the STANDARD config; see research/decision_log.md for the timestamp/commit this was frozen at). Where results contradict a hypothesis, this file is NOT edited after the fact — contradictions are reported in results/ and discussed in paper/main.tex Section "Results" and "Negative Results," not hidden.

  • H1. Frozen self-supervised encoders (DINO/DINOv2) will achieve higher external label alignment (ARI/NMI vs. reserved tissue/cell-type labels) than PCA-on-pixels and handcrafted texture/color baselines, on both datasets.

  • H2. Frozen self-supervised encoders will achieve higher external label alignment than a generic ImageNet-supervised ResNet50 on PathMNIST (histopathology is visually far from ImageNet's natural-image distribution), but the gap will be smaller on BloodMNIST (cell morphology is closer to ImageNet's typical object/texture statistics).

  • H3. The representation with the highest external label alignment will not be the representation with the highest seed-to-seed cluster stability. I.e., stability and external utility are expected to partially dissociate — this dissociation, if found, is the paper's central empirical claim (see research/novelty_memo.md).

  • H4. Cluster quality (label-free and external) will degrade substantially when clustering directly on 2D UMAP/t-SNE coordinates compared to clustering on full/PCA-reduced feature space, demonstrating that visually "clean" 2D embeddings are not a reliable proxy for clustering input.

  • H5. On PathMNIST, at least one representation will show a statistically detectable association between cluster assignment and the NCT-vs-CRC-VAL source-site variable, independent of tissue-type label agreement — i.e., some fraction of latent structure is site-driven, not purely biological.

  • H6. Cluster assignments will be more robust to modest photometric perturbations (brightness/contrast/noise/blur/color jitter) for self-supervised encoders than for the PCA-on-pixels baseline, because self-supervised pretraining objectives are explicitly built for augmentation invariance.

Falsification conditions

Each hypothesis is considered falsified (not "inconclusive") if the 95% bootstrap CI of the relevant metric difference crosses zero, computed at the image level with the appropriate resampling unit (site-level for PathMNIST cross-site comparisons; see research/decision_log.md on why BloodMNIST bootstrapping is image-level, not patient-level).

Verdicts (STANDARD-tier run, 2026-09-08)

Recorded once, after make benchmark + scripts/analyze_dim_reduction.py + scripts/build_claims_registry.py ran to completion. Every number below has a row in research/claims_registry.md. This section is appended, not a rewrite of the hypotheses above.

  • H1 -- CONFIRMED on both datasets. On PathMNIST, DINOv2 ViT-S/14 reaches external ARI 0.435 (95% CI [0.405, 0.473]) vs. the best baseline (handcrafted, ARI 0.184) and PCA-on-pixels (ARI 0.159). On BloodMNIST, DINO ViT-S/16 reaches ARI 0.299 vs. the best baseline (PCA-on-pixels, ARI 0.242).

  • H2 -- WEAKLY CONFIRMED, smaller effect than predicted. The best-self-supervised-vs-ResNet50 ARI gap is 0.196 on PathMNIST (0.435 vs. 0.239) and 0.153 on BloodMNIST (0.299 vs. 0.146) -- the PathMNIST gap is larger as predicted, but the difference between the two gaps is modest and both bootstrap CIs are wide enough that this should be read as directionally consistent, not a strong confirmation.

  • H3 -- CONFIRMED on both datasets; this is the paper's cleanest finding. On PathMNIST, the highest-utility representation (DINOv2, ARI 0.435) is not the highest-stability representation (handcrafted, mean pairwise k-means ARI 0.996 vs. DINOv2's 0.797). On BloodMNIST, the highest-utility representation (DINO ViT-S/16, ARI 0.299) is not the highest-stability representation (PCA-on-pixels, mean pairwise ARI 0.927 vs. DINO's 0.684). Across all 6 representations, the Spearman rank correlation between external-ARI ranking and seed-stability ranking is weak and not statistically distinguishable from zero at n=6 (PathMNIST: rho=0.31, p=0.54; BloodMNIST: rho=0.09, p=0.87; see results/rank_correlations.parquet); regardless of significance, the qualitative fact that the top-utility representation is never the top-stability representation holds on both datasets -- a representation being the most reproducible under reclustering is not a proxy for it being the most biologically aligned.

  • H4 -- FALSIFIED (contrary to prediction). Clustering directly on 2D UMAP coordinates did NOT degrade external ARI relative to full feature space -- mean external ARI across representations was higher on UMAP-2D than on full feature space on both datasets (PathMNIST: 0.357 vs. 0.320; BloodMNIST: 0.278 vs. 0.225; see results/dimensionality_reduction.parquet and figures/dimreduction_pitfall_*.png). This is reported as a genuine negative result relative to the pre-registered prediction, not suppressed: at this sample size (n~1000 test images) and this k range, a 2D UMAP embedding did not measurably hurt k-means clustering quality, which complicates a blanket "never cluster on UMAP coordinates" claim -- see paper/main.tex Section "Dimensionality-Reduction Sensitivity" for the caveated discussion of why this might be sample-size- or cluster-shape-dependent rather than a general result.

  • H5 -- PARTIALLY CONFIRMED. Every representation shows a statistically detectable cluster-vs-site association on PathMNIST (Cramer's V 0.13-0.27, p < 1e-8 for all six representations via chi-squared test on n=2997 combined train+test images). However, for every representation the label association is stronger than the site association (Cramer's V 0.27-0.70 vs. label), so site_stronger_than_label is False in all six cases -- technical confounding is real and detectable in this benchmark, but secondary to biological signal for every representation tested.

  • H6 -- NOT CONFIRMED AS STATED; a more specific finding replaces it. Self-supervised encoders were not more perturbation-robust than the PCA-on-pixels baseline -- PCA-on-pixels was itself highly robust (mean cluster agreement across 6 perturbations: 0.898 on PathMNIST, 0.884 on BloodMNIST), comparable to DINO/DINOv2 (0.93-0.97). The real finding is that the handcrafted color-histogram-plus-GLCM-texture representation is uniquely fragile to photometric perturbation on both datasets (mean cluster agreement 0.371 on PathMNIST, 0.269 on BloodMNIST) while every other representation tested -- including the two non-learned baselines (PCA-on-pixels, HOG) -- is robust (>=0.84 on both datasets). This makes sense post hoc (color histograms and GLCM statistics are directly computed from the pixel intensities that brightness/contrast/blur transforms alter) but was not the comparison originally hypothesized, and is reported as such.

Additional finding not pre-registered: cross-dataset ranking replication

Because H3 motivated comparing rankings, we also checked whether the external-ARI ranking of the 6 representations replicates between PathMNIST and BloodMNIST: it does not (Spearman rho = -0.03, p=0.96, n=6 -- effectively no rank correlation). The stability ranking shows a numerically larger but still non-significant correlation (rho = 0.49, p=0.33, n=6): HOG is the least stable representation on both datasets, and PCA-on-pixels/handcrafted are consistently among the most stable. This suggests cluster stability may be a more modality-general property of a representation than external label alignment is, at least at this sample size -- a hypothesis for future work, not a claim this release can fully support on two datasets.

Limitations

This file is maintained honestly and is expected to grow. Nothing here should be "fixed" by removing the limitation from this file — only by actually resolving the underlying issue (and then noting the resolution with a date).

  1. No from-scratch self-supervised training. SimCLR/MoCo/BYOL are evaluated only insofar as their frozen, publicly-released pretrained checkpoints (DINO, DINOv2) are used; this project does not train a contrastive objective from scratch on these datasets. See research/decision_log.md.

  2. BloodMNIST has no donor/patient identifiers. All BloodMNIST leakage/independence analysis assumes image-level independence, which is weaker than the patient-level guarantee available for PathMNIST. If donor-level correlation exists among visually similar cells from the same blood draw, BloodMNIST confidence intervals are anti-conservative (too narrow). This is disclosed wherever BloodMNIST bootstrap CIs are reported.

3a. Native 28x28 resolution, not 64x64. Originally planned at 64x64 to preserve more morphological texture (see research/decision_log.md), but downgraded to the native MedMNIST 28x28 resolution because the 64px Zenodo archives repeatedly failed to download reliably from this environment. This is an infrastructure constraint, not a scientific choice, and it caps how much fine-grained texture any representation in this project — including the handcrafted HOG/texture baselines — can possibly recover. Re-running at 64x64 or native full resolution is listed as future work.

  1. Two datasets, both from MedMNIST. Both datasets are pre-processed, center-cropped, and resized by the MedMNIST curators rather than accessed at full native resolution from the original source archives. This trades fidelity for tractability and reproducibility on commodity hardware. Full-resolution NCT-CRC-HE-100K (224px, ~100k patches) is noted as a future-work direction, not attempted here.

  2. CPU-only compute. All experiments in this release ran on a single laptop CPU (no GPU available in this environment). This bounds the STANDARD-tier sample sizes and the largest feasible pretrained encoder (ViT-S rather than ViT-L/ViT-g). Runtime is logged per-run in the benchmark manifest so a reader can judge feasibility on their own hardware.

  3. Only two datasets, both derived from MedMNIST's curation pipeline. Any bias MedMNIST's curators introduced (e.g., their center-crop / resize policy) is inherited by every representation and cluster result in this project equally, and cannot be distinguished from a genuine dataset property.

  4. Domain-shift analysis is single-axis. The PathMNIST site-shift analysis (NCT-CRC-HE-100K vs. CRC-VAL-HE-7K) is the only real multi-site comparison available; no equivalent exists for BloodMNIST. Cross-dataset (PathMNIST <-> BloodMNIST) transfer is deliberately not attempted for label alignment, since the label spaces are incommensurable (see research/decision_log.md) — only representation-ranking replication is tested across datasets, not label transfer itself.

  5. UMAP/t-SNE are visualization-only by design, per the project's own methodological stance (Section 12/30 of the brief). This means the site's Embedding Explorer cannot be used to eyeball "how separable" the true high-dimensional structure is; label-free clustering metrics computed in the original feature space are the authoritative separability measure, and the UI says so explicitly.

  6. Clustering hyperparameter selection is samples-of-one per representation-dataset pair for the STANDARD-tier release (a single label-free-selected k per representation, not a full grid search across every conceivable k), due to CPU compute budget. The chosen k/algorithm per run is always recorded in the manifest so this is auditable, and an EXTENDED config tier is provided (configs/benchmark/extended.yaml) for users with more compute who want a fuller sweep.

  7. No clinical validation whatsoever. Nothing in this repository should be read as evidence of diagnostic capability. See research/claims_registry.md.