Ranked track

Ranks combine evaluated checkpoint quality with device and network costs calculated by the common calibrated simulator.

Protocol A · Quality

Select the best evaluated checkpoint by highest accuracy, then lower loss, lower simulated wall time, and earlier step/record. Rank accuracy (higher is better) and loss (lower is better) separately. Record the system costs alongside checkpoint quality.

Protocol B · Efficiency

Use the three published target levels per model/dataset from the paper's Protocol B tables (Appendix B). At the first evaluated checkpoint reaching each target, rank cumulative time, communication, energy and peak memory separately; lower is better. The registered policy fixes the target levels.

Protocol C · Robustness

Compare each registered perturbation with its independent nominal baseline. Rank the absolute deltas of accuracy, time, communication, energy and memory separately; smaller variation is better. Seven datasets require 63 runs (seven nominal plus 56 perturbed runs). Score the 56 perturbation runs against their seven nominal baselines.

Metric rank → points

Rank 1 earns 10 points, rank 2 earns 9, through rank 10 earning 1; ranks above 10 earn 0. Equal metric values share a competition rank (for example, 1, 1, 3). Reference methods are excluded from competitive ranks.

Protocol score

For each active model/dataset/metric comparison (and B target or C condition), let n be the number of valid, non-reference ranked methods. The protocol score is sum(n * points) / sum(n) across these comparisons. The denominator is shared by all methods: a missing result, OOM or unreached target contributes zero to an active comparison. A comparison with no valid ranked results contributes no weight. Per-model scores use the same rule restricted to that model.

Final score

The configured protocol weights are currently A:B:C = 1:1:1: final = (A + B + C) / 3. A missing protocol contributes zero; weights are not redistributed. The API reports four decimal places; the page displays two. Ties at API precision are ordered by method name; equal-looking two-decimal values can still differ. Methods without any scored comparison are not assigned a ranked result.

Reference baselines and publication

Local Only and Centralized provide historical accuracy references and are excluded from competitive scoring. Centralized supports A/B only. Submit one complete evaluated ZIP per protocol, model and method. Only completed / published indicates successful publication.