Results
Generated by brainssl report from the run directories; do not edit by hand.
Tumour segmentation (MSD Task01, 96 held-out test scans)
| Run |
Labels |
Dice WT |
Dice TC |
Dice ET |
Dice mean |
HD95 WT |
HD95 TC |
HD95 ET |
| ft_ours_5pct |
5% |
0.838 |
0.727 |
0.776 |
0.780 |
15.5 |
12.5 |
7.2 |
| ft_unet_scratch_5pct |
5% |
0.850 |
0.788 |
0.772 |
0.803 |
14.9 |
11.0 |
14.7 |
| ft_bilateral_scratch_10pct |
10% |
0.877 |
0.795 |
0.787 |
0.820 |
13.2 |
9.3 |
13.4 |
| ft_ours_10pct |
10% |
0.885 |
0.802 |
0.802 |
0.830 |
9.7 |
9.6 |
12.5 |
| ft_unet_mim_10pct |
10% |
0.872 |
0.787 |
0.807 |
0.822 |
14.9 |
11.0 |
9.7 |
| ft_unet_scratch_10pct |
10% |
0.872 |
0.799 |
0.790 |
0.821 |
15.3 |
10.5 |
13.8 |
| ft_ours_25pct |
25% |
0.894 |
0.829 |
0.828 |
0.850 |
9.4 |
11.5 |
7.7 |
| ft_unet_scratch_25pct |
25% |
0.879 |
0.827 |
0.825 |
0.844 |
12.2 |
12.0 |
7.8 |
| ft_ours_100pct |
100% |
0.899 |
0.853 |
0.840 |
0.864 |
7.0 |
6.7 |
7.2 |
| ft_unet_scratch_100pct |
100% |
0.889 |
0.837 |
0.816 |
0.847 |
9.2 |
7.5 |
11.3 |
Paired per-case comparison with the U-Net trained from scratch on the same scans (mean difference, 95% bootstrap interval, one-sided Wilcoxon signed-rank p):
| Labels |
Dice mean |
Dice WT |
Dice TC |
Dice ET |
scans better / worse |
| 5% |
-0.023 [-0.044, -0.003], p 1.000 |
-0.011 [-0.025, +0.002], p 1.000 |
-0.061 [-0.090, -0.035], p 1.000 |
+0.004 [-0.028, +0.041], p 0.967 |
26 / 70 |
| 10% |
+0.009 [-0.002, +0.020], p 0.190 |
+0.013 [+0.003, +0.023], p 0.010 |
+0.003 [-0.015, +0.021], p 0.766 |
+0.011 [-0.000, +0.023], p 0.055 |
55 / 41 |
| 25% |
+0.007 [-0.002, +0.015], p 0.010 |
+0.015 [+0.003, +0.029], p 0.027 |
+0.002 [-0.012, +0.015], p 0.148 |
+0.004 [-0.010, +0.014], p 0.026 |
55 / 41 |
| 100% |
+0.017 [+0.006, +0.031], p <0.001 |
+0.010 [+0.002, +0.022], p 0.002 |
+0.016 [+0.006, +0.027], p 0.007 |
+0.024 [+0.007, +0.048], p 0.001 |
65 / 31 |
First round of segmentation runs (10,000 pretraining steps)
The matrix above was run a second time after the first round showed no gain at 10% of the labels. Pretraining was lengthened to 30,000 steps on the strength of validation scans alone (see the method page); the test scores of the first round are kept here, unchanged.
| Run |
Labels |
Dice WT |
Dice TC |
Dice ET |
Dice mean |
HD95 WT |
HD95 TC |
HD95 ET |
| ft_ours_5pct |
5% |
0.868 |
0.780 |
0.776 |
0.808 |
15.3 |
9.0 |
14.0 |
| ft_unet_scratch_5pct |
5% |
0.850 |
0.788 |
0.772 |
0.803 |
14.9 |
11.0 |
14.7 |
| ft_bilateral_scratch_10pct |
10% |
0.877 |
0.795 |
0.787 |
0.820 |
13.2 |
9.3 |
13.4 |
| ft_ours_10pct |
10% |
0.878 |
0.785 |
0.786 |
0.817 |
13.9 |
9.8 |
13.3 |
| ft_unet_mim_10pct |
10% |
0.876 |
0.792 |
0.810 |
0.826 |
12.2 |
10.4 |
9.6 |
| ft_unet_scratch_10pct |
10% |
0.872 |
0.799 |
0.790 |
0.821 |
15.3 |
10.5 |
13.8 |
| ft_ours_25pct |
25% |
0.891 |
0.829 |
0.832 |
0.851 |
11.1 |
12.1 |
4.3 |
| ft_unet_scratch_25pct |
25% |
0.879 |
0.827 |
0.825 |
0.844 |
12.2 |
12.0 |
7.8 |
| ft_ours_100pct |
100% |
0.895 |
0.848 |
0.836 |
0.860 |
11.1 |
7.1 |
7.8 |
| ft_unet_scratch_100pct |
100% |
0.889 |
0.837 |
0.816 |
0.847 |
9.2 |
7.5 |
11.3 |
Paired per-case comparison with the U-Net trained from scratch on the same scans (mean difference, 95% bootstrap interval, one-sided Wilcoxon signed-rank p):
| Labels |
Dice mean |
Dice WT |
Dice TC |
Dice ET |
scans better / worse |
| 5% |
+0.005 [-0.005, +0.015], p 0.163 |
+0.019 [+0.006, +0.033], p 0.032 |
-0.008 [-0.023, +0.007], p 0.874 |
+0.004 [-0.007, +0.016], p 0.322 |
53 / 43 |
| 10% |
-0.004 [-0.012, +0.004], p 0.804 |
+0.006 [-0.003, +0.016], p 0.384 |
-0.014 [-0.029, -0.001], p 0.997 |
-0.004 [-0.013, +0.004], p 0.578 |
44 / 52 |
| 25% |
+0.007 [-0.001, +0.018], p 0.086 |
+0.011 [+0.001, +0.025], p 0.196 |
+0.002 [-0.008, +0.013], p 0.248 |
+0.008 [-0.010, +0.033], p 0.320 |
54 / 42 |
| 100% |
+0.013 [+0.003, +0.027], p 0.023 |
+0.006 [-0.003, +0.019], p 0.283 |
+0.011 [+0.002, +0.022], p 0.039 |
+0.021 [+0.006, +0.045], p 0.001 |
52 / 44 |
Autism vs control from resting-state connectomes (ABIDE I, 871 subjects)
Noise-aware model at the C fixed from development runs (highest mean development AUC (split seeds 0-2), ties to the smaller C); baselines at the C that does best on these splits. Mean ± standard deviation over 5 repeats of 10-fold cross-validation.
| Model |
Atlas |
C |
AUC |
Accuracy |
Balanced acc. |
Sensitivity |
Specificity |
| Noise-aware (ours) |
ensemble |
|
0.776 ± 0.004 |
0.696 |
0.684 |
0.524 |
0.844 |
| Noise-aware (ours) |
aal |
0.0003 |
0.738 ± 0.005 |
0.683 |
0.674 |
0.550 |
0.797 |
| Noise-aware (ours) |
cc200 |
0.0001 |
0.767 ± 0.005 |
0.696 |
0.684 |
0.531 |
0.838 |
| Noise-aware (ours) |
cc400 |
0.0003 |
0.768 ± 0.005 |
0.705 |
0.695 |
0.560 |
0.829 |
| Noise-aware (ours) |
dosenbach160 |
0.0001 |
0.698 ± 0.005 |
0.631 |
0.617 |
0.419 |
0.815 |
| Noise-aware (ours) |
ho |
0.0001 |
0.743 ± 0.005 |
0.674 |
0.661 |
0.486 |
0.837 |
| Tangent space + LR |
ensemble |
|
0.769 ± 0.004 |
0.698 |
0.687 |
0.534 |
0.840 |
| Tangent space + LR |
aal |
0.0003 |
0.733 ± 0.004 |
0.668 |
0.658 |
0.522 |
0.794 |
| Tangent space + LR |
cc200 |
0.0003 |
0.763 ± 0.005 |
0.702 |
0.693 |
0.569 |
0.817 |
| Tangent space + LR |
cc400 |
0.003 |
0.757 ± 0.004 |
0.693 |
0.684 |
0.562 |
0.807 |
| Tangent space + LR |
dosenbach160 |
0.0001 |
0.687 ± 0.005 |
0.627 |
0.608 |
0.358 |
0.859 |
| Tangent space + LR |
ho |
0.0003 |
0.738 ± 0.005 |
0.674 |
0.663 |
0.524 |
0.803 |
| Log-Euclidean + LR |
ensemble |
|
0.769 ± 0.004 |
0.701 |
0.690 |
0.544 |
0.836 |
| Log-Euclidean + LR |
aal |
0.0003 |
0.730 ± 0.004 |
0.671 |
0.662 |
0.529 |
0.794 |
| Log-Euclidean + LR |
cc200 |
0.0003 |
0.762 ± 0.004 |
0.691 |
0.682 |
0.556 |
0.808 |
| Log-Euclidean + LR |
cc400 |
0.003 |
0.763 ± 0.005 |
0.699 |
0.691 |
0.573 |
0.809 |
| Log-Euclidean + LR |
dosenbach160 |
0.0001 |
0.693 ± 0.004 |
0.634 |
0.616 |
0.380 |
0.853 |
| Log-Euclidean + LR |
ho |
0.0003 |
0.735 ± 0.005 |
0.673 |
0.662 |
0.527 |
0.798 |
| Correlation + LR |
ensemble |
|
0.757 ± 0.007 |
0.689 |
0.690 |
0.698 |
0.682 |
| Correlation + LR |
aal |
0.0003 |
0.715 ± 0.007 |
0.666 |
0.667 |
0.680 |
0.653 |
| Correlation + LR |
cc200 |
0.0003 |
0.753 ± 0.005 |
0.680 |
0.680 |
0.691 |
0.670 |
| Correlation + LR |
cc400 |
0.0001 |
0.750 ± 0.007 |
0.691 |
0.692 |
0.703 |
0.681 |
| Correlation + LR |
dosenbach160 |
0.0003 |
0.669 ± 0.009 |
0.614 |
0.616 |
0.643 |
0.590 |
| Correlation + LR |
ho |
0.001 |
0.743 ± 0.006 |
0.680 |
0.680 |
0.684 |
0.676 |
| MLP (cc200) |
cc200 |
|
0.738 ± 0.006 |
0.678 |
0.673 |
0.613 |
0.733 |
| BrainNetCNN (cc200) |
cc200 |
|
0.723 ± 0.007 |
0.663 |
0.659 |
0.594 |
0.723 |
| Brain Network Transformer (cc200) |
cc200 |
|
0.646 ± 0.015 |
0.612 |
0.608 |
0.554 |
0.662 |
Noise-aware ensemble against each baseline on the 50 shared splits (corrected resampled t-test, one-sided):
| Baseline |
ΔAUC |
p |
splits won / lost |
ΔBalanced acc. |
p |
| Correlation + LR, ensemble |
+0.020 |
0.037 |
39 / 11 |
-0.006 |
0.635 |
| Log-Euclidean + LR, ensemble |
+0.007 |
0.044 |
36 / 11 |
-0.006 |
0.753 |
| Tangent space + LR, ensemble |
+0.007 |
0.069 |
33 / 16 |
-0.003 |
0.644 |
| MLP (cc200) |
+0.039 |
0.001 |
44 / 6 |
+0.011 |
0.282 |
| BrainNetCNN (cc200) |
+0.053 |
<0.001 |
46 / 4 |
+0.025 |
0.063 |
| Brain Network Transformer (cc200) |
+0.132 |
<0.001 |
50 / 0 |
+0.076 |
<0.001 |
Leave one site out (20 held-out sites)
| Model |
AUC |
Accuracy |
Balanced acc. |
| Noise-aware (ours), ensemble |
0.758 |
0.696 |
0.686 |
| Tangent space + LR, ensemble |
0.756 |
0.691 |
0.684 |
| Log-Euclidean + LR, ensemble |
0.753 |
0.685 |
0.678 |
| Correlation + LR, ensemble |
0.755 |
0.677 |
0.678 |