QUEST
QUEry-based virtual STaining: multiplex immunofluorescence predicted from H&E, with dynamic output panel. The model reads a frozen encoding of the H&E plus a semantic embedding of each marker name.
Quick start
import matplotlib.pyplot as plt
from quest import QuestGenerator
he = plt.imread("assets/example_he.png")[..., :3] # (224, 224, 3)
mif = QuestGenerator("quest-semantic")([he], ["PanCK", "CD68", "DAPI"])
Tutorials
We prepared tutorials from virtual staining to downstream applications.
| Tutorial | What for |
|---|---|
| 📦 00_data | what to download, where it goes, and the format of each cohort |
| 🔬 01_predict_mif | predict a panel from one H&E patch, and score it against the real MIF |
| 🧫 02_cell_typing | cell types from a virtual stained expression |
| 📐 03_distance_entropy | cell-cell distance and neighbourhood entropy from based on cell typing |
| 🗺️ 04_cn_annotation | cell states and cellular neighbourhoods discovered from H&E alone |
| 🔎 05_retrieval | search a stained archive with an H&E query |
| 📈 06_patient_aggregation | patches to a patient survival prediction, and which patch contribution analyses |
Set the two constants at the top of tutorials/_common.py, get the data with Tutorial 0, then explore them step-by-step.
Weights
| repo | |
|---|---|
| yandrewl/QUEST | three QUEST models, cell typer, the Eva marker table |
| yandrewl/Eva | Eva_model.ckpt, the MIF foundation encoder the retrieval benchmark embeds in |
| MahmoodLab/UNI2-h | the frozen H&E encoder |
| model | |
|---|---|
quest-semantic | marker queries from a semantic embedding of the marker's name |
quest-id | the same architecture with a fixed learned per-marker table instead |
quest-eva | a masked autoencoder that inpaints the marker channels of a partly observed stack |
Check quest.zoo for detailed description of model architectures and configurations.
Data quick start
huggingface-cli login first.
import shutil
from huggingface_hub import snapshot_download, hf_hub_download
snapshot_download("yandrewl/QUEST", local_dir="checkpoints", # weights, 1.5 GB
allow_patterns=["*.ckpt", "*.npz"])
snapshot_download("yandrewl/QUEST-tutorial-data", repo_type="dataset", # cohorts, 1.7 GB
local_dir="data",
allow_patterns=["crc-metu/*", # 73 MB tutorial 6
"bog-86337/*", # 36 MB tutorial 4
"stanford-pc/*"]) # 1.6 GB tutorials 1, 5
shutil.copy(hf_hub_download("yandrewl/Eva", "Eva_model.ckpt"), # tutorial 5
"checkpoints/Eva_model.ckpt")
from questkit import cohort # tutorials 2, 3
cohort.fetch_pathocell(["reg016_B", "reg032_B"]) # ~350 MB / region
QUEST_DATA points elsewhere if you keep the cohorts outside the repo. Tutorial 0 has the file formats.
License
Code, weights and staged cohorts are released under
CC BY-NC-ND 4.0. See LICENSE.