Mini Course

A hands-on mini course on multimodal large language models and their clinical applications

From architecture to fine-tuning, inference, and evaluation with MedGemma 1.5.

Tianhao (Terry) Fu1,2  ·  Jun Ma2

1Faculty of Applied Science and Engineering, University of Toronto  ·  2Princess Margaret Cancer Centre & AI Hub, University Health Network

Open in Colab Course Repository Take the Quiz

About this mini course

Multimodal LLMs combine images, text, video, and audio in a single model - and they are rapidly becoming practical tools in clinical workflows. This mini course walks through the complete pipeline for a medical image-text-to-text task: understanding how MLLMs align modalities, dissecting the MedGemma 1.5 architecture, then preparing the FLARE-MLLM-2D dataset, fine-tuning with QLoRA, running inference, and evaluating report generation with CRIMSON and GREEN scores.

Every hands-on step runs in the linked interactive Jupyter notebook on Colab. This GitHub repository contains the course website; the executable notebook workflow remains on Colab.

≈ 90 minutes end to end ≈ 35 minutes of compute, excluding initial downloads 8 modules Level: graduate (MSc / PhD) Format: self-paced, Colab notebook

Learning objectives

By the end of this mini course you should be able to:

  1. Explain why modality alignment is the central problem in multimodal LLMs, and contrast the four architectural families that solve it - LLaVA, BLIP-2, Flamingo, and Kosmos.Module 01
  2. Trace how MedGemma 1.5 turns a medical image into visual tokens a language model can read, naming the role of MedSigLIP, the multimodal projector, and the Gemma 3 decoder.Module 02
  3. Convert a raw medical imaging dataset into Hugging Face supervised fine-tuning format, and describe the preprocessing choices that shape what the model learns.Module 03
  4. Fine-tune a 4B vision-language model with QLoRA on a single GPU, and explain what low-rank adapters and 4-bit quantization each contribute.Module 04
  5. Run inference with the base and fine-tuned checkpoints and compare their generated reports side by side.Module 05
  6. Evaluate generated reports with CRIMSON and GREEN, compute both by hand on a worked example, and articulate what each score does and does not capture.Module 06

Prerequisites

What you should know

  • Comfortable reading and running Python in a notebook
  • Basic PyTorch: tensors, a training loop, moving work to a GPU
  • Deep learning fundamentals - what attention is, what fine-tuning does
  • Helpful but not required: prior exposure to Hugging Face transformers
  • No radiology background is assumed. Clinical concepts are introduced where they matter

What you need to have

  • A Google account for Colab, or a local Jupyter environment
  • A Hugging Face account, plus accepted data terms for FLARE-MLLM-2D and access to the MedGemma weights
  • A GPU runtime with ≥ 40 GB VRAM (A100 or H100 class). The free Colab T4 has 16 GB and will run out of memory during fine-tuning
  • Roughly 15 GB when using only the notebook’s report-generation subset. Allow about 50 GB to cache the complete 35.6 GB FLARE collection together with model weights and checkpoints

Course outline

01 BackgroundIntroduction to MLLM + clinical applications Jump →
02 Model Architecture: MedGemma 1.5Detailed description of the model architecture Jump →
03 Data PreparationPrepare the FLARE-MLLM-2D dataset for fine-tuning Jump →
04 Fine-tuningFine-tune MedGemma 1.5 4B on the preprocessed dataset Jump →
05 InferenceInfer with the base and fine-tuned MedGemma 1.5 4B Jump →
06 EvaluationEvaluate MedGemma 1.5 4B’s performance on report generation using CRIMSON score and GREEN score Jump →
07 ConclusionSummary of what we have covered in the mini course Jump →
08 QuizTest your knowledge with the provided NotebookLM Jump →

Module 01

Background

Introduction to MLLM + clinical applications.

Introduction to MLLM

MLLM emphasizes the alignment of latent spaces between different modalities. The building blocks are familiar - an image encoder and a text decoder - and the interesting question is how to combine encoders and decoders of different modalities so they can operate as one model.

Based on the input and output modalities, we can classify MLLMs into categories like the ones on the right. Hugging Face also uses these categories.

MLLM task categories

  • Audio-text-to-text
  • Image-text-to-text
  • Image-text-to-image
  • Image-text-to-video
  • Video-text-to-text
  • Any-to-any

A vision encoder and a language model do not naturally understand one another:

  • Separate pretrainingThe two components are usually trained independently, using different data and objectives.
  • Different latent spacesEach component learns its own representation space, so their coordinate systems are not automatically compatible.
  • Uninterpretable featuresA patch embedding from the vision encoder has no inherent meaning to a language model that never encountered it during training.
  • The alignment problemUnlike a jointly trained encoder–decoder such as a UNet, an MLLM needs an explicit bridge between its pretrained vision and language components. The approaches below provide different forms of that bridge.

LLaVA-style

Align vision features to an existing LLM

+Simple projector, cheap to train, reuses a full off-the-shelf LLM
-Every visual token sits in the input sequence, so long/many images inflate context length

BLIP-2-style

Compress vision through learned queries

+Fixed, small number of visual tokens regardless of image size or resolution
-Extra Q-Former to train, and compression can lose fine-grained visual detail

Flamingo-style

Inject vision through cross-attention

+Handles many interleaved images/video without growing the text sequence
-Requires splicing new cross-attention layers into the LLM, more invasive and harder to adapt with lightweight fine-tuning

Kosmos-style

Train a unified multimodal autoregressive model

+One shared representation space by design, flexible across input/output modalities
-Most data- and compute-hungry option, can't just bolt onto an existing pretrained LLM
General MLLM architecture: modality encoder, input projector, LLM backbone, output projector, modality generator
The general anatomy of an MLLM: modality encoders project non-text inputs into the LLM backbone; projectors and generators map latents back out to other modalities.

Figure credit: NVIDIA. (n.d.). Multimodal large language models. NVIDIA Glossary. Retrieved from nvidia.com/en-eu/glossary/multimodal-large-language-models  ·  Extended reading: Multimodal Large Language Models - NVIDIA

Clinical applications of MLLMs

MLLMs are powerful in a clinical workflow for perception, reasoning, documentation, triage, and patient-facing support.

Below are five concrete clinical applications (CAs), each illustrating one way an MLLM slots into practice.

Five clinical applications

CA1Report generation

In a report generation task, the MLLM:

MLLM caption generation process producing a bronchoscopy examination report with human revision

Figure credit: Luo, X., Huang, X., Liang, X. et al. Towards Automated Reporting: A Bronchoscopy Report Dataset for Enhancing Multimodality Large Language Models. Sci Data 13, 339 (2026). doi.org/10.1038/s41597-026-06692-8

CA2Longitudinal comparison

In a longitudinal comparison task, the MLLM:

Comparison of MLLM radiology answers with and without a prior study

Figure credit: Zhang, X., Meng, Z., Lever, J., & Ho, E. S. (2025, July). Libra: Leveraging temporal images for biomedical radiology analysis. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 17275–17303). doi.org/10.48550/arXiv.2411.19378

CA3Multi-class classification

In a multi-class classification task, the MLLM outputs one label for several classes. It is the same as naive classification, but with outputs in the form of text - we use string parsers to convert the textual class ids into integers. Some common clinical examples are BI-RADS category, tumor subtype, disease stage, and dermatology diagnosis category.

Photo of a tabby cat
→
Vision Encoder
“What animal is shown in the image? Answer in a single {class id} only: 0 for cat, 1 for dog, and 2 for owl.”
→
Tokenizer+ embedding
→
LLM
→
0

Figure credit: Wikipedia - Cat (Cat_August_2010-4.jpg)

CA4Multi-label classification

In a multi-label classification task, the MLLM outputs multiple labels at once. Common clinical examples include chest X-ray findings: edema, consolidation, atelectasis, cardiomegaly, and pleural effusion.

Multi-class versus multi-label classification examples with one-hot and multi-hot label vectors

Figure credit: Sharma, G. (2021, February 7). Multi-label classification. Medium; Analytics Vidhya. medium.com/analytics-vidhya/multi-label-classification

CA5Regression

In a regression task, the MLLM outputs a continuous value. It is the same as naive regression, but with outputs in the form of text - we use string parsers to convert the strings into floats. Some common clinical examples are ejection fraction, tumor size, organ volume, lab value prediction, risk score, and survival time.

Photo of a tabby cat
→
Vision Encoder
“What percentage of the image area does the cat occupy?”
→
Tokenizer+ embedding
→
LLM
→
0.43

Figure credit: Wikipedia - Cat (Cat_August_2010-4.jpg)

Course setup

In this mini course, we are going to use MedGemma 1.5 4B as an example. The MedGemma family consists of LLaVA-style vision-language models (VLMs) designed for image-text-to-text tasks, specialized in medical images.

Throughout the mini course, we will be learning hands-on examples with MedGemma 1.5, covering a complete pipeline for the report generation task using the FLARE-MLLM-2D dataset.

The hands-on pipeline

  • Data preparation
  • Fine-tuning
  • Inference
  • Evaluation

Dataset: FLARE-MLLM-2D · Model: MedGemma 1.5 4B

Module 02

Model architecture

Detailed description of the MedGemma 1.5 model architecture.

MedGemma overview

MedGemma collection: MedGemma 1.5 4B, MedGemma 27B, and the MedSigLIP vision encoder across 2D imaging, text, and advanced imaging

Figure credit: Sellergren, A., Gao, C., Mahvar, F., Kohlberger, T., Jamil, F., Traverse, M., ... & Golden, D. (2026). MedGemma 1.5 technical report. arXiv preprint arXiv:2604.05081. doi.org/10.48550/arXiv.2604.05081

MedGemma 1.5

3D CT Scan
→
2D Slices
→
MedSigLIPEncoder
→
Gemma 3Decoder
→
Text Outputs

Inside MedSigLIPEncoder-only Vision Transformer (ViT) · SigLIP-400M tuned on medical data

896×896 imageresized to the encoder’s fixed input resolution
↓
Patch embedding14×14 pixel patches → 64×64 = 4,096 patch tokens + position embeddings
↓
× 27 blocks
Bidirectional self-attention16 heads · width 1,152 · every patch attends to every patch - no causal mask
↓
MLPGELU · hidden 4,304 · LayerNorm around each sub-layer
↓
Average pooling4,096 patch tokens → 256 visual tokens
↓
Multimodal projectornorm + linear map into Gemma 3’s 2,560-d embedding space
↓
256 visual tokens per image“pan & scan” may add extra crops, each encoded the same way

MedSigLIP is the 400M-parameter SigLIP image encoder further trained on medical image–text pairs with SigLIP’s sigmoid contrastive loss, so its visual features are aligned with medical language before the LLM ever sees them.

Inside Gemma 3 (4B)Decoder-only Transformer LLM · generates text autoregressively

Input sequencetext tokens (262K SentencePiece vocab) with visual tokens spliced in at each image position
↓
Token embeddingsdmodel = 2,560 · shared with the output layer
↓
× 34 blocks
Grouped-query attention + RoPE8 query heads share 4 KV heads · head dim 256 · RMSNorm before & after
↓
GeGLU feed-forwardhidden dimension 10,240

5 local sliding-window attention layers (1,024-token window) for every 1 global layer → a 128K-token context at manageable KV-cache cost

↓
Final RMSNorm + LM headprobability distribution over the 262K vocabulary
↓
Next tokenappended to the sequence and fed back in until the report is complete

Attention is causal over text - each token sees only its past - but all visual tokens belonging to the same image attend to each other bidirectionally.

MedGemma 1.5 4B is based on Gemma 3 with the same general architecture, using a 400M MedSigLIP vision encoder as the visual front end and a decoder-only Transformer LLM as the text generator. Images are normalized to 896×896 and encoded into 256 visual tokens per image.

The key improvement between MedGemma 1 and MedGemma 1.5 is the long context window, with which we are able to feed multiple uniformly sampled slices (up to 85, modeled as a time sequence) to represent a 3D volume. However, in this mini course, we only deal with 2D images.

The encoder runs at a fixed 896×896 input and pools its 4,096 patch tokens down to 256 visual tokens. For images that are large or far from square, an optional “pan & scan” pass crops additional windows and encodes each one the same way, trading extra tokens for effective resolution.

Cited: Sellergren, A., Gao, C., Mahvar, F., Kohlberger, T., Jamil, F., Traverse, M., ... & Golden, D. (2026). MedGemma 1.5 technical report. arXiv preprint arXiv:2604.05081. doi.org/10.48550/arXiv.2604.05081

Module 03

Data preparation

Prepare the FLARE-MLLM-2D dataset for fine-tuning.

1

Download FLARE-MLLM-2D Time varies by subset and bandwidth

FLARE-MLLM-2D is a gated multimodal dataset for the MICCAI FLARE challenge. The course workflow selects its report-generation portion rather than downloading unrelated modalities. The complete collection is 35.6 GB, so a full download needs substantially more time and storage.

2

Preprocess ≈ 5 minutes

Since we use Hugging Face’s transformers as the backend of the pipeline, we want to convert the dataset format into Hugging Face’s supervised fine-tuning (SFT) records’ format.

Try it out yourself!

The executable data-preparation workflow is in the interactive Colab notebook. The GitHub repository contains the supporting course website.

Open in Colab GitHub

Module 04

Fine-tuning

Fine-tune MedGemma 1.5 4B on the preprocessed dataset.

Fine-tuning protocols

Even with foundation models trained on massive amounts of data, it is still extremely common that the application dataset is out of distribution (OoD). To adapt to the target domain, we need to apply fine-tuning. There are multiple ways to perform fine-tuning; in this mini course, we will focus on parameter-efficient fine-tuning (PEFT) - specifically QLoRA.

LoRA: frozen weight matrix W0 plus trainable low-rank matrices A and B
LoRA keeps W₀ frozen and learns a low-rank update AB; QLoRA applies the same adapters on top of 4-bit quantized base weights.

Figure credit: Görner, M. (2025, March 13). Are you still using LoRA to fine-tune your LLM? Towards Data Science. towardsdatascience.com/are-you-still-using-lora-to-fine-tune-your-llm

Method references: Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.  ·  Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs.

3

Run fine-tuning ≈ 15 minutes, depending on GPU

In this mini course, we will fine-tune for only 0.25 epochs to save time.

Try it out yourself!

Run fine-tuning in the interactive Colab notebook with a GPU runtime providing ≥ 40 GB VRAM (A100 or H100 class) - see Prerequisites for the full setup. The GitHub repository contains the supporting course website.

Open in Colab GitHub

Module 05

Inference

Infer with the base and fine-tuned MedGemma 1.5 4B.

4

Run inference ≈ 5 minutes, depending on GPU

Now let us infer on 16 samples (to save time) in the validation set, with both the base and the fine-tuned model.

Try it out yourself!

Run inference in the interactive Colab notebook with a GPU runtime providing ≥ 40 GB VRAM (A100 or H100 class) - see Prerequisites for the full setup. The GitHub repository contains the supporting course website.

Open in Colab GitHub

Module 06

Evaluation

Evaluate MedGemma 1.5 4B’s performance on report generation using CRIMSON score and GREEN score.

Evaluation metrics

CRIMSON Score

  • Scope. Chest X-ray reports. The official package defaults to the open MedGemmaCRIMSON judge, which can run locally without sending reports to an external API. GPT-5.2 is available as an optional API-backed judge.
  • Stage 1 - extract and weight. Pull every abnormal finding from both reports (normal findings are excluded, so reporting style cannot inflate the score), then assign each finding a clinical significance weight w(f) from a rubric written with attending cardiothoracic radiologists. Patient age and indication feed this call: aortic calcification is expected/benign at 75, actionable at 25.
  • Stage 2 - classify. Discrepancies fall into false findings (hallucinations), missing findings (omissions), and attribute errors on matched findings across eight dimensions: location/laterality, severity/extent, morphology, measurement, certainty, under-interpretation, over-interpretation, and temporal comparison. Each attribute error is weighted 0.5 if significant, 0.0 if negligible - wrong laterality is significant, “apical” vs “lateral” in one lobe is not.
  • Stage 3 - score. Severity-weighted credit minus weighted false positives, normalized by the reference report’s total significance. Range is (-1, 1]: 1 is perfect, 0 means no more useful than submitting a normal template, and negative means a radiologist would rather start from a blank template than edit this report.
  • Validation. Against clinically significant error counts from six board-certified radiologists on ReXVal: Kendall’s τ = 0.61-0.71, Pearson’s r = 0.71-0.84. Also released with two new benchmarks, RadJudge and RadPref.

Cited: Baharoon, M., Heintz, T., Raissi, S., Alabbad, M., Alhammad, M., AlOmaish, H., Kim, S. E., Banerjee, O., & Rajpurkar, P. (2026). CRIMSON: A clinically-grounded LLM-based metric for generative radiology report evaluation. arXiv preprint arXiv:2603.06183. doi.org/10.48550/arXiv.2603.06183

GREEN Score

  • Scope. Radiology reports broadly. The released judge is an open-source 7B-parameter model, fine-tuned on ~100k reference/candidate report pairs drawn from six chest X-ray corpora (MIMIC-CXR, MIMIC-PRO, CandidPTX, PadChest, BIMCV-COVID19, OpenI).
  • Step 1 - generate the error notation. The judge reads both reports and emits structured text: a list of matched findings plus counts for six error categories - (a) false report of a finding, (b) missing a finding, (c) wrong anatomic location, (d) wrong severity, (e) mentioning a comparison absent from the reference, (f) omitting a comparison to a prior study. Each error is also tagged significant or insignificant.
  • Step 2 - parse. Counts are pulled out of that text with regular expressions (parse_error_counts in the green-score package); per-category counts and matched-finding counts land in a result dataframe.
  • Step 3 - score. Matched findings over matched findings plus significant errors, giving a value in [0, 1]. Insignificant errors are reported but kept out of the score. An error-free report with at least one matched finding scores 1; omissions, false findings, and other significant discrepancies increase the denominator and lower the score.
  • Validation. On ReXVal, GREEN’s significant-error count lands within 1.54 of the average radiologist’s - close to inter-expert disagreement - and it returns a natural-language explanation naming each error, so it doubles as model feedback.

Cited: Ostmeier, S., Xu, J., Chen, Z., Varma, M., Blankemeier, L., Bluethgen, C., Michalson, A. E., Moseley, M., Langlotz, C., Chaudhari, A. S., & Delbrouck, J.-B. (2024). GREEN: Generative radiology report evaluation and error notation. In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 374–390). Association for Computational Linguistics. doi.org/10.18653/v1/2024.findings-emnlp.21

How commonly used report-generation metrics compare

Report-generation metrics inspect different signals, from exact wording to clinical error severity. This comparison is included for context; the hands-on notebook computes only GREEN and CRIMSON.

Conceptual comparison of general-purpose and radiology-specific metrics
Metric What it compares What it is useful for Main limitation
BLEULexical Exact candidate/reference n-gram precision, with a brevity penalty Fast, reproducible comparison of wording overlap Penalizes valid paraphrases and can barely react when one negation reverses a diagnosis
ROUGE-LLexical Longest common subsequence between candidate and reference Coverage and ordering of shared text Measures surface similarity rather than factual or clinical correctness
BERTScoreSemantic Token-level similarity using contextual embeddings Recognizing paraphrases that use different wording A general-language embedding match is not a direct test of findings, polarity, or clinical significance
CheXbert F1Clinical labels Agreement on 14 observations extracted from chest X-ray reports Checking high-level presence, absence, and uncertainty of common findings Narrow chest X-ray label set; does not fully represent location, severity, measurements, or rare findings
RadGraph F1Clinical graph Overlap of extracted radiology entities and relations Comparing findings together with anatomy and selected relations Depends on an information-extraction model and returns overlap rather than a direct explanation of clinical harm
RadCliQComposite A learned combination of BLEU, BERTScore, CheXbert similarity, and RadGraph F1 Predicting radiologist-assessed chest X-ray report error burden better than its individual components Produces an opaque aggregate without identifying specific errors; raw RadCliQ variants are lower-is-better
GREENLLM judge Matched findings plus six categories of significant and insignificant errors Clinically aware scoring with a readable error explanation Requires judge-model inference and inherits the judge’s scope and failure modes
CRIMSONLLM judge Finding matches and attribute errors weighted by patient-specific clinical significance Distinguishing minor errors from consequential omissions and hallucinations Currently centered on chest X-ray reports and requires a configured judge plus clinical context

Comparison framing follows the motivation and related-work discussion in Ostmeier et al. (2024), GREEN; descriptions are synthesized from the original metric papers linked above.

One report pair, eight metric outputs

A single negation reverses the clinical meaning while leaving most words unchanged.

Clinical context

Adult patient; pleural effusion is a clinically relevant finding.

Ground-truth report

Pleural effusion present.

Model-generated report

Pleural effusion not present.

Outputs for the same ground truth and generated report
Metric Example output What the output reveals
BLEU 0.75 The score remains high because three of the four words overlap; the clinically decisive negation has little effect.
ROUGE-L 0.57 The shared word sequence dominates even though the diagnosis is reversed.
BERTScore 0.75 Contextual similarity drops, but the two nearly identical sentences still appear semantically close.
CheXbert F1 0.00 for the sole positive target finding The extracted pleural-effusion label changes from positive to negative. A full 14-label micro- or macro-F1 also depends on how the remaining labels are aggregated.
RadGraph F1 0.50 The finding entity overlaps, but its presence status does not, so the graph match receives only partial credit.
RadCliQ Higher / worse error estimate RadCliQ combines several component scores. There is no portable single-case number because v0/v1, checkpoints, normalization, and component implementations affect the output; raw variants are lower-is-better.
GREEN 0.00
0 matched findings; 1 significant error		 The explanation identifies that pleural effusion is positive in the reference but negative in the generated report.
CRIMSON 0.00
missing weighted finding; no earned credit		 Assuming pleural effusion is the only clinically weighted reference finding, its omission contributes to the denominator but earns no credit. Exact wording and weight depend on patient context and judge configuration.

Reading the numbers. BLEU, ROUGE-L, BERTScore, RadGraph F1, and GREEN values are the published outputs for this pair in Figure 1 of the GREEN paper. CheXbert shows the finding-level F1 implied by the positive-to-negative label disagreement. The RadCliQ and CRIMSON rows are explicitly configuration-aware rather than fabricated point estimates.

One case, two candidate reports

CONTEXT: 78-year-old, dyspnea.
REFERENCE: Moderate left pleural effusion. Aortic atherosclerosis.

Clinically correct, reworded

There is a moderate effusion in the left pleural space. Atherosclerotic aorta.

Shares few exact n-grams with the reference, so BLEU and ROUGE-L rank this report lower than the one on the right.

GREEN

  • Matched findings: 2 (effusion, atherosclerosis)
  • Significant errors: 0

2 / (2 + 0) → 1.00

CRIMSON

  • Effusion · actionable, not urgent · w = 0.5 · matched, no attribute errors → credit 0.5
  • Atherosclerosis at 78 · expected/benign · w = 0 · contributes nothing either way
  • False findings: none, so Efalse = 0

(0.5 − 0) / Wref = 0.5 / 0.5 → 1.00

Clinically wrong, near-verbatim

No left pleural effusion. Aortic atherosclerosis.

Reuses the reference’s exact wording, so surface metrics score it higher - a one-word negation that inverts the diagnosis barely moves them.

GREEN

  • Matched findings: 1 (atherosclerosis)
  • Category (b), missing a finding present in the reference: the effusion → 1 significant error

1 / (1 + 1) → 0.50

CRIMSON

  • Effusion · w = 0.5 · missing, so it earns no credit
  • Atherosclerosis · w = 0 · matched, but worth nothing at this age
  • False findings: none, so Efalse = 0

(0 − 0) / 0.5 → 0.00

How GREEN aggregates
range [0, 1] · 0 if nothing matched
GREEN = matched findingsmatched findings + Σ significant errors

Errors come from six categories: false finding, missing finding, wrong location, wrong severity, a comparison absent from the reference, and an omitted prior-study comparison. The judge tags each one significant or insignificant, and only significant errors reach the denominator.

How CRIMSON aggregates
range (-1, 1] · 0 = normal-template baseline

Every abnormal finding carries a rubric weight w, set with the patient’s age and indication in view:

1.00 urgent 0.50 actionable 0.25 not actionable 0.00 expected/benign
C = Σmatched i wi · wiwi + Eattr,i

C is the credit the candidate earns. Each matched finding contributes its own weight wi, scaled by a partial-credit factor that equals 1 when every attribute is right and shrinks as significant attribute errors (0.5 each) accumulate. Because wi sits in both numerator and denominator, one attribute error costs proportionally less on an urgent finding than on a minor one.

S = C − EfalseWref

S is the raw score: net credit, C minus the weight of hallucinated findings Efalse, over Wref - the total significance available in the reference. Missing findings are penalized implicitly: they count toward Wref but earn no credit. Below zero the score is squashed by −A/(1+A), A = Efalse − C, so it approaches -1 asymptotically however many false findings pile up.

Why n-gram overlap fails and what replaces it. A one-word negation flips the diagnosis while preserving almost every n-gram, so BLEU and ROUGE-L prefer the wrong report; both LLM-based metrics reason over findings instead and separate the two. They then part ways on aggregation: GREEN counts significant errors against matched findings, whereas CRIMSON weights each finding by clinical consequence - giving the failed report 0.00, its “no better than a normal template” baseline, rather than the 0.50 that counting alone yields.
5

Run evaluation ≈ 5 minutes, depending on GPU

Let us evaluate the inference outputs using CRIMSON score and GREEN score.

Expected results

The evaluation cell prints a per-sample table and a mean CRIMSON and GREEN score for both the base and the fine-tuned checkpoint, so you can compare them directly. Here is what to look for when your run finishes:

  • A visible change in report style. This is the clearest effect of even a quarter epoch. The base model tends to produce long, hedged, general-purpose descriptions; the fine-tuned model produces shorter reports that imitate the terse structure of the FLARE reference reports.
  • Low absolute scores for both models. Both metrics are demanding - they reward correct findings and penalize hallucinated ones, so scores far below 1 are normal and not a sign that something went wrong.
  • CRIMSON can go negative where GREEN cannot. If a model invents findings that are not in the reference, weighted false positives can exceed earned credit, which is exactly what a score below zero is meant to signal.
  • The two metrics need not agree. A report that misses one benign finding barely moves CRIMSON but still costs GREEN a matched finding. Disagreement is informative, not a bug.

Do not read these numbers as a benchmark result. This run fine-tunes for 0.25 epochs and evaluates on 16 validation samples, both chosen so the notebook finishes in minutes. At that scale the difference between the base and fine-tuned means is well within noise, and a re-run can reverse the ordering. Evaluator configuration also matters: the official CRIMSON package defaults to MedGemmaCRIMSON, while an optional API-backed judge may introduce additional run-to-run variability. Treat the output as a demonstration that the pipeline is wired correctly end to end. For a result worth reporting, train at least one full epoch, evaluate the entire validation split, and record the exact evaluator and decoding settings.

Try it out yourself!

Run evaluation in the interactive Colab notebook with a GPU runtime providing ≥ 40 GB VRAM (A100 or H100 class) - see Prerequisites for the full setup. The GitHub repository contains the supporting course website.

Open in Colab GitHub

Module 07

Key takeaways

  1. MLLMs combine multiple modalities such as images and text; the core challenge is modality alignment, since vision encoders and language models do not naturally share the same latent representation space.
  2. Different architectures solve alignment differently - LLaVA-style projection, BLIP-2 learned queries, Flamingo-style cross-attention, and Kosmos-style unified modeling - enabling clinical tasks from report generation and longitudinal comparison to classification and regression.
  3. The course’s main example, MedGemma 1.5 4B, is a LLaVA-style model that pairs a MedSigLIP vision encoder with a Gemma 3 decoder, turning medical images into visual tokens the LLM uses to generate text.
  4. The hands-on workflow uses the FLARE-MLLM-2D dataset, downloaded, preprocessed, and converted into Hugging Face supervised fine-tuning format.
  5. Fine-tuning uses QLoRA - training small low-rank adapters on 4-bit weights - and results are evaluated with CRIMSON and GREEN scores that check radiology reports for clinical errors, missing findings, and severity-weighted mistakes.

Module 08

Quiz

Test your learning outcomes

A NotebookLM has been prepared with the course materials - quiz yourself on everything covered in this mini course.

Open the NotebookLM Quiz

Acknowledgements

This course builds on datasets, models, software, and educational resources created by the wider medical AI and open-source communities.

Data and models

  • The hands-on exercises use the report-generation portion of the FLARE-MLLM-2D dataset from FLARE-MedFM and the MICCAI FLARE initiative. The gated collection is licensed under CC BY-NC 4.0 and is not redistributed here. Its IU X-Ray source is described by Demner-Fushman et al. (2016), Preparing a collection of radiology examinations for distribution and retrieval.
  • The course uses MedGemma 1.5 4B as its example model. Learners must accept and comply with Google’s Health AI Developer Foundations terms of use before accessing the weights.
  • The GREEN discussion identifies MIMIC-CXR, MIMIC-PRO, CandidPTX, PadChest, BIMCV-COVID19, and OpenI because they were used by the original GREEN authors. This course does not redistribute those datasets.

Existing software and code

  • The notebook workflow uses Hugging Face transformers conventions for model loading and supervised fine-tuning. Evaluation builds on the official green-score and crimson-score codebases.
  • The website’s image-slot.js and tweaks-panel.jsx helpers are based on the Omelette starter scaffold included with the original site prototype; minicourse-tweaks.jsx supplies course-specific settings.
  • The development interface loads React, ReactDOM, and Babel Standalone, while the typography uses Google Fonts. These third-party resources remain subject to their original licenses and terms.

Model and evaluation references

Models, architectures, and methods

  1. Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual Instruction Tuning (LLaVA). arXiv:2304.08485.
  2. Li, J., Li, D., Savarese, S., & Hoi, S. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597.
  3. Alayrac, J.-B., et al. (2022). Flamingo: A Visual Language Model for Few-Shot Learning. arXiv:2204.14198.
  4. Huang, S., et al. (2023). Language Is Not All You Need: Aligning Perception with Language Models (Kosmos-1). arXiv:2302.14045.
  5. Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid Loss for Language Image Pre-Training (SigLIP). arXiv:2303.15343.
  6. Gemma Team. (2025). Gemma 3 Technical Report. arXiv:2503.19786.
  7. Sellergren, A., et al. (2026). MedGemma 1.5 Technical Report. arXiv:2604.05081.
  8. Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
  9. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.

Evaluation metrics

  1. Baharoon, M., et al. (2026). CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation. arXiv:2603.06183.
  2. Ostmeier, S., et al. (2024). GREEN: Generative Radiology Report Evaluation and Error Notation. Findings of EMNLP 2024, 374–390.
  3. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. ACL 2002, 311–318.
  4. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, 74–81.
  5. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. ICLR 2020.
  6. Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A. Y., & Lungren, M. P. (2020). CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. EMNLP 2020, 1500–1519.
  7. Jain, S., et al. (2021). RadGraph: Extracting Clinical Entities and Relations from Radiology Reports. NeurIPS Datasets and Benchmarks 2021.
  8. Yu, F., et al. (2023). Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation (RadGraph F1 and RadCliQ). Patterns, 4(9), 100802.

Use of AI in creating this tutorial

  • Claude Design was used to create the initial draft of the course website, providing a starting point for its layout and presentation.
  • NotebookLM is used to create the course quiz from the tutorial materials.
  • ChatGPT was used to refine portions of the tutorial content for clarity, structure, and presentation.

The instructors reviewed and edited the AI-assisted outputs and remain responsible for the final tutorial. AI-generated material is not treated as a primary source; the papers, datasets, and software cited throughout the course provide the underlying references.

Figures and research claims are credited beside the relevant material throughout the course. Their inclusion does not transfer ownership; reuse remains subject to each original source’s terms.

Instructors

Tianhao (Terry) Fu

Faculty of Applied Science and Engineering, University of Toronto · Princess Margaret Cancer Centre & AI Hub, University Health Network

Jun Ma

Princess Margaret Cancer Centre & AI Hub, University Health Network