BIOSIGNAL / INDEPENDENT EVALUATION
Test the evidence.
Know the limits.
Evaluate your model’s predictions against reference labels. Compare a baseline, inspect calibration, and measure the cost of referring uncertain results for review.
Research demo. Imported predictions are evaluated locally, without uploading files. This workspace does not process sequences or run a biological classifier. The bundled example is entirely synthetic.
01 / Define the study
Use predictions generated without fitting on the test samples. Assign participants and related samples to a single partition. Batch IDs must be unique across partitions; this check cannot detect undisclosed participant overlap.
Six columns: sample_id, group, split, label, probability, baseline_probability. Use pseudonymous IDs, probabilities of reference class 1, and validation/test splits (train optional). Maximum 2 MB / 2,000 samples. Baseline must predict the same outcome on the same samples.
Load an example or your prediction file to begin.
02 / Fix the decision criteria
Both models use this threshold. Deferral uses distance of the primary model’s score from that threshold; the same retained subset is used for the baseline comparison. This heuristic is not an uncertainty guarantee.
Only validation results appear before locking. Loading any file starts a new analysis; it does not create an unseen test set.
03 / Examine the results
| Approach | Accuracy | Sensitivity | Specificity | Coverage | FP / FN | Brier |
|---|
Brier score measures probability error; lower is better. It reflects both calibration and discrimination. Deferral sensitivity/specificity concern retained samples only; withheld cases remain unresolved.
Calibration / reliability
Ten fixed bins; dot size reflects sample count. Dashed line = perfect agreement. Empty bins are omitted.
Review burden versus retained errors
Descriptive margin sweep. Do not select a new threshold using test results.
Batch bootstrap · primary model with deferral
400 seeded whole-batch resamples, percentile 95% intervals conditional on the fitted model. They do not include retraining uncertainty or establish transportability. Sparse batches/classes may make intervals unstable.
Batch-level review
| Batch | Samples | Positive labels | Accuracy | Coverage after review |
|---|
Methods & next validation step
Developed by Prem Kumar Perumal. The evaluation code is deterministic and uses no AI-generated interpretation. Imported metrics describe supplied labels and predictions; they do not demonstrate outbreak detection or diagnostic validity.
Read the study protocol ↗ · Probability calibration reference ↗ · Public metagenomic dataset candidate ↗
curatedMetagenomicData is a candidate for a future separately trained microbiome benchmark. No public biological dataset has been incorporated into this release. Dataset-level permissions and suitability require review.
