From local training to a leaderboard result

Ranks combine evaluated checkpoint quality with calibrated simulations of device and network costs. Use the common Evaluator for comparable system costs.

This guide runs FedAvg on one allocated CUDA GPU. Virtual clients keep separate data and adapter states, train sequentially, and aggregate on the same machine. Train on your machine, then upload the evaluated results.

No CUDA GPU? You can still run real single-host CPU training, Logger, checkpoint evaluation and a development ZIP using integrations/pytorch/README.md in the downloaded tools. Use the CUDA virtual-client runner for formal submissions. Start with Appendix A of the full instruction.

1. Download & install
2. Pinned models & data
3. Choose A, B or C
4. Train + evaluate + package
5. Submit
6. Inspect results

Formal submissions use the full pinned datasets and --for-submission.

1 · Register and save your submission token

Open Register, request an email verification code and enter it within ten minutes. Save the submission token privately; it is displayed once. Use it on Submit or in the API's X-Submission-Token header. Re-verifying the same email replaces the token and revokes the previous one without changing team ownership.

Use your registration token for account access and uploads. Keep it private and out of code and ZIP files. You can register before or after training.

2 · Choose a pinned model and prepare the seven datasets

All A/B/C protocols support Qwen2.5-0.5B, Gemma 3-270M and Gemma 3-1B: 63 protocol–model–dataset assignments. Each upload covers one protocol, one model, one method and all seven datasets.

Use the model and dataset preparation guide to obtain the approved edgeflowertune-20261020-v1 assets. Use its --reference model commands and seven-dataset reconstruction commands. Each data folder must contain the pinned train.csv and validation.csv. Download the models and datasets using the pinned revisions in that guide.

Use the registered sequence lengths: 128 for BoolQ and 64 for all other datasets, with effective batch size 8. The runner skips overlong training rows and retains every validation row, truncating overlong evaluation prompts.

3 · Download the matching code and install

Extract both ZIPs fully. Keep the original ZIP files together. The GitHub Code link is for source browsing; use these matching downloads for this guide.

python3 participant_toolkit/verify_download_pair.py \
  --manifest /absolute/path/to/downloads/release-pair.json \
  --directory /absolute/path/to/downloads

Linux / CUDA training environment

Run these commands from the extracted participant-tools folder. Use a Python version supported by the tested ML dependencies; the tested environment uses Python 3.13 and PyTorch 2.8.0 with CUDA 12.8. Choose a PyTorch build compatible with your GPU driver.

cd /absolute/path/to/EdgeFlowerTune_Participant_Tools
export EFT_ROOT="$PWD"
python3 -m venv .venv-training
source .venv-training/bin/activate
python participant_toolkit/install_tools.py --env-dir "$VIRTUAL_ENV"
python -m pip install -r integrations/pytorch/requirements-tested.txt
python -m pip install flwr==1.27.0 PyYAML==6.0.2

Installation must report INSTALLATION COMPLETE. All five command checks passed. Keep the extracted folder in place. If you already have a training environment, activate it and run the same installer with --env-dir "$VIRTUAL_ENV".

Save the current policy before training

export EDGEFLOWERTUNE_URL="https://edgeflowertune.ai"
mkdir -p "$EFT_ROOT/policy"
export SUBMISSION_CONFIG_FILE="$EFT_ROOT/policy/server-config.json"
export PROTOCOL_POLICY_FILE="$EFT_ROOT/policy/protocol-policy.json"
curl -fsS "$EDGEFLOWERTUNE_URL/api/submissions/config" -o "$SUBMISSION_CONFIG_FILE"
curl -fsS "$EDGEFLOWERTUNE_URL/api/submissions/protocol-policy" -o "$PROTOCOL_POLICY_FILE"
python participant_toolkit/verify_benchmark_release.py \
  --protocol-policy "$PROTOCOL_POLICY_FILE" --server-config "$SUBMISSION_CONFIG_FILE"

Save the policy snapshot with each experiment and use it throughout training, evaluation and packaging. Connect using the standard HTTPS port.

4 · Configure the FedAvg experiment

Set these paths to the extracted training source and prepared assets. Select your allocated GPU and set the client count, seed, rounds and local steps.

export TRAINING_ROOT=/absolute/path/to/EdgeFlowerTune
export MODEL=qwen25_05b
export MODEL_DIR=/absolute/path/to/Qwen2.5-0.5B
export REFERENCE_DATA_ROOT=/absolute/path/to/reference-data
export METHOD_ID="fedavg_$(python -c 'import uuid; print(uuid.uuid4().hex[:12])')"
export OUTPUT_A=/absolute/path/to/new-fedavg-A
export OUTPUT_B=/absolute/path/to/new-fedavg-B
export OUTPUT_C=/absolute/path/to/new-fedavg-C
# Set CUDA_VISIBLE_DEVICES, CLIENTS, SEED, ROUNDS and LOCAL_STEPS for your experiment.
: "${CUDA_VISIBLE_DEVICES:?Set your allocated GPU index or UUID}"
: "${CLIENTS:?Set at least 2 virtual clients}"
: "${SEED:?Set a nonnegative random seed}"
: "${ROUNDS:?Set the number of training rounds}"
: "${LOCAL_STEPS:?Set local optimizer updates per round}"

METHOD_ID identifies your submission; --methods selects its algorithm. Generate it once and keep it for all A/B/C runs of the same method; save it with your run settings. The name fedavg itself is a reserved baseline ID.

Custom IDs use 3–64 lowercase letters/digits with single underscores between segments, for example team_a_fedavg_v1. Uppercase, hyphens, leading/trailing underscores and reserved baseline aliases fail preflight before training. The generated ID above already follows these rules.

Keep one configuration for the compared conditions. FedProx requires at least two local updates. To use another model, change MODEL to gemma3_270m or gemma3_1b and provide its matching pinned MODEL_DIR.

Other selectors are fedprox, hetero (HeteroLoRA), split (cut-0 SplitLoRA), centroid (Centralized) and local (Local Only). Each formal suite runs one method. Use a distinct submission ID for each algorithm. Centralized and Local Only are local reference analyses: omit --submission-id for those runs. Centralized supports A/B.

Protocol A · Quality

Run FedAvg on all seven datasets. For each dataset, select the highest-accuracy evaluated checkpoint, with lower loss used to break ties. Protocol A ranks the selected checkpoints by accuracy and loss, producing seven quality results.

python integrations/pytorch/run_reference_matrix.py \
  --training-root "$TRAINING_ROOT" \
  --model "$MODEL" --model-dir "$MODEL_DIR" \
  --data-root "$REFERENCE_DATA_ROOT" --output-dir "$OUTPUT_A" \
  --methods fedavg --datasets all \
  --clients "$CLIENTS" --seed "$SEED" \
  --rounds "$ROUNDS" --local-steps "$LOCAL_STEPS" \
  --protocol A --protocol-policy "$PROTOCOL_POLICY_FILE" \
  --submission-id "$METHOD_ID" --for-submission --preflight

Output: $OUTPUT_A/edgeflowertune_protocol_A.zip.

Protocol B · Efficiency

Run FedAvg on all seven datasets and retain their evaluated checkpoint curves. The Evaluator selects the first checkpoint reaching each registered accuracy target, producing 21 target results. Unreached targets remain unscored.

python integrations/pytorch/run_reference_matrix.py \
  --training-root "$TRAINING_ROOT" \
  --model "$MODEL" --model-dir "$MODEL_DIR" \
  --data-root "$REFERENCE_DATA_ROOT" --output-dir "$OUTPUT_B" \
  --methods fedavg --datasets all \
  --clients "$CLIENTS" --seed "$SEED" \
  --rounds "$ROUNDS" --local-steps "$LOCAL_STEPS" \
  --protocol B --protocol-policy "$PROTOCOL_POLICY_FILE" \
  --submission-id "$METHOD_ID" --for-submission --preflight

Output: $OUTPUT_B/edgeflowertune_protocol_B.zip.

Protocol C · Robustness

For each of the seven datasets, run one nominal baseline and eight registered perturbations: 63 runs in total. Protocol C publishes 56 condition results with changes from their nominal baselines. The runner applies the dropout schedule to client participation and uses the common simulation for device and network costs.

python integrations/pytorch/run_reference_matrix.py \
  --training-root "$TRAINING_ROOT" \
  --model "$MODEL" --model-dir "$MODEL_DIR" \
  --data-root "$REFERENCE_DATA_ROOT" --output-dir "$OUTPUT_C" \
  --methods fedavg --datasets all \
  --clients "$CLIENTS" --seed "$SEED" \
  --rounds "$ROUNDS" --local-steps "$LOCAL_STEPS" \
  --protocol C --protocol-policy "$PROTOCOL_POLICY_FILE" \
  --submission-id "$METHOD_ID" --for-submission --preflight

Output: $OUTPUT_C/edgeflowertune_protocol_C.zip.

6 · Check the complete protocol ZIP

The runner records training events, evaluates saved checkpoints, calculates system costs and packages the suite. Check ledger.json for successful stages, then upload the final protocol ZIP.

export PROTOCOL_PACKAGE="$OUTPUT_B/edgeflowertune_protocol_B.zip"
python -m zipfile -t "$PROTOCOL_PACKAGE"

For A or C, select the corresponding output ZIP. SUBMISSION_CONFIG_FILE enables the runner's final upload-capacity check. Contact the organizer if a complete suite exceeds the upload limit.

python participant_toolkit/pack_protocol.py \
  --protocol B --model "$MODEL" \
  --input-dir /absolute/path/to/seven-completed-dataset-runs \
  --output /absolute/path/to/new-protocol-B.zip \
  --protocol-policy "$PROTOCOL_POLICY_FILE" \
  --server-config "$SUBMISSION_CONFIG_FILE"

Optional local publication check

python -m pip install -r webapp/requirements.txt
bash examples/qwen25_boolq/run_isolated_publication_demo.sh

7 · Submit the complete package

1. Open Submit and enter your private email-issued token.
2. Select the same protocol and model as the package; supply your method name and description.
3. Upload the complete protocol ZIP: A has seven dataset runs, B seven checkpoint curves, C 63 independent runs.
4. Follow the verification status. A suite publishes atomically; incomplete or mismatched evidence is rejected.

For API uploads, include submission_scope=protocol-model, model_id, the protocol and the complete ZIP.

The server checks the package’s model, dataset, run identity, checkpoints and policy version.

8 · Inspect results

Only completed / published means publication has finished. Open the matching board from the Leaderboard menu, find your method and inspect its scores and coverage. Track progress in Recent Submissions.

Protocol scores contribute to the final ranking at the configured weights, currently 1:1:1. Missing protocols contribute zero. Protocol B ranks results at the registered target levels.

Scoring rules

Ranked track

Ranks combine evaluated checkpoint quality with device and network costs calculated by the common calibrated simulator.

Protocol A · Quality

Select the best evaluated checkpoint by highest accuracy, then lower loss, lower simulated wall time, and earlier step/record. Rank accuracy (higher is better) and loss (lower is better) separately. Record the system costs alongside checkpoint quality.

Protocol B · Efficiency

Use the three published target levels per model/dataset from the paper's Protocol B tables (Appendix B). At the first evaluated checkpoint reaching each target, rank cumulative time, communication, energy and peak memory separately; lower is better. The registered policy fixes the target levels.

Protocol C · Robustness

Compare each registered perturbation with its independent nominal baseline. Rank the absolute deltas of accuracy, time, communication, energy and memory separately; smaller variation is better. Seven datasets require 63 runs (seven nominal plus 56 perturbed runs). Score the 56 perturbation runs against their seven nominal baselines.

Metric rank → points

Rank 1 earns 10 points, rank 2 earns 9, through rank 10 earning 1; ranks above 10 earn 0. Equal metric values share a competition rank (for example, 1, 1, 3). Reference methods are excluded from competitive ranks.

Protocol score

For each active model/dataset/metric comparison (and B target or C condition), let n be the number of valid, non-reference ranked methods. The protocol score is sum(n * points) / sum(n) across these comparisons. The denominator is shared by all methods: a missing result, OOM or unreached target contributes zero to an active comparison. A comparison with no valid ranked results contributes no weight. Per-model scores use the same rule restricted to that model.

Final score

The configured protocol weights are currently A:B:C = 1:1:1: final = (A + B + C) / 3. A missing protocol contributes zero; weights are not redistributed. The API reports four decimal places; the page displays two. Ties at API precision are ordered by method name; equal-looking two-decimal values can still differ. Methods without any scored comparison are not assigned a ranked result.

Reference baselines and publication

Local Only and Centralized provide historical accuracy references and are excluded from competitive scoring. Centralized supports A/B only. Submit one complete evaluated ZIP per protocol, model and method. Only completed / published indicates successful publication.

Protocol B scoring and reference accuracies

Protocol B keeps three registered accuracy targets per model/dataset. At each target, the server selects the first evaluated checkpoint that reaches or exceeds it, then ranks cumulative time, communication, energy and peak memory separately: lower cost is better. One checkpoint curve supplies all three comparisons. Seven dataset curves produce 21 target results; an unreached target has no cost or rank and contributes no points to an otherwise active scoring item.

Use the three registered accuracy targets from the paper's Protocol B tables (Appendix B), defined by protocol_b_targets and target_level. Policy: protocol-b-paper-tables-20261022-v1. Qwen / BoolQ targets are 71%, 75%, 78%; ARC-Easy targets are 75%, 76%, 77%. All 63 targets are listed in instruction section 3.1.

Local Only and Centralized provide historical accuracy references. Both reference methods are excluded from competitive scoring. Protocol B uses the registered targets listed above.

Accuracy references (%) for each model and dataset, with Local Only followed by Centralized.

OOM marks a run that exceeded the evaluated edge client pool’s memory capacity. Its accuracy reference is unavailable. See section 3.1 of the full instruction.

Custom training and native MF integration

For custom training code, see the full instruction’s Logger, Quality Runner and Evaluator sections and participant_toolkit/PROTOCOL_SUBMISSION.md. For native MF builds, see integrations/mobilefinetuner/README.md. Real-device timing and sensor hooks are described in integrations/pytorch/README.md.

Troubleshooting