# Model and dataset preparation

Download the training source, model weights and datasets separately. The participant-tools ZIP contains the website tools and adapters; the matching runtime ZIP contains `EdgeFlowerTune/`.

| Website ID | Upstream weights | Variant |
|---|---|---|
| qwen25_05b | [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) | Base |
| gemma3_270m | [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) | Instruction-tuned |
| gemma3_1b | [google/gemma-3-1b-pt](https://huggingface.co/google/gemma-3-1b-pt) | Pretrained |

Gemma may require accepting Google's terms on Hugging Face. Use your own account
and environment-based credentials; never put tokens into commands or archives.

## Approved release and original-reference identities

Use the pinned models, data splits and preprocessing in release **edgeflowertune-20261020-v1**. The catalog at `participant_toolkit/reference_assets.v1.json` and `/downloads/reference-assets.json` lists model/tokenizer files and the fourteen reference CSV hashes. Save `benchmark_release.v1.json` with your policy snapshot and download manifest.

| Model | Immutable matching download revision | File verification |
|---|---|---|
| Qwen2.5-0.5B | `060db6499f32faf8b98477b0a26969ef7d8b9987` | Seven weight/config/tokenizer file hashes verified. |
| Gemma 3-270M IT | `ac82b4e820549b854eebf28ce6dedaf9fdfa17b3` | Archived download revision and nine file hashes recorded. |
| Gemma 3-1B PT | [`fcf18a2a879aab110ca39f8bffbccd5d49d8eb29`](https://huggingface.co/google/gemma-3-1b-pt/tree/fcf18a2a879aab110ca39f8bffbccd5d49d8eb29) | Eight model/config/tokenizer file hashes verified. |

All three models now have immutable matching download revisions:

```bash
python participant_toolkit/prepare_public_assets.py model \
  --model qwen25_05b --reference --output /absolute/path/to/new-assets/Qwen2.5-0.5B
python participant_toolkit/prepare_public_assets.py model \
  --model gemma3_270m --reference --output /absolute/path/to/new-assets/gemma-3-270m-it
python participant_toolkit/prepare_public_assets.py model \
  --model gemma3_1b --reference --output /absolute/path/to/new-assets/gemma-3-1b-pt
```

An existing copy can also be checked offline without downloading it again:

```bash
python participant_toolkit/prepare_public_assets.py verify-reference-model \
  --model gemma3_1b --root /absolute/path/to/existing/gemma-3-1b-pt
```

The reference download verifies each file and writes `asset-lock.json`. Use a new output directory for each download. The paired runtime source provides the matching training software.

## Seven-dataset reference reconstruction

All fourteen train/validation CSVs were reconstructed from newly downloaded
public sources on 2026-10-19 and matched the preserved reference SHA-256 values
exactly. The original export logic and five HF revision records were recovered.
PIQA and SocialIQA use hash-pinned official ZIPs instead of Git revisions.
Full revisions, source/output checksums and ordered source-ID hashes are in
`participant_toolkit/reference_assets.v1.json`.

| Dataset ID / output folder | Source / configuration | Revision prefix or archive | Train / validation rows |
|---|---|---|---:|
| `boolq` | [aps/super_glue / boolq](https://huggingface.co/datasets/aps/super_glue/tree/3de24cf8022e94f4ee4b9d55a6f539891524d646/boolq) | `3de24cf8022e` | 9,427 / 3,270 |
| `qnli` | [nyu-mll/glue / qnli](https://huggingface.co/datasets/nyu-mll/glue/tree/bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c/qnli) | `bcdcba79d07b` | 104,743 / 5,463 |
| `piqa` | [Official PIQA ZIP](https://storage.googleapis.com/ai2-mosaic/public/physicaliqa/physicaliqa-train-dev.zip) | Archive SHA-256 in catalog | 16,113 / 1,838 |
| `hellaswag` | [Rowan/hellaswag / default](https://huggingface.co/datasets/Rowan/hellaswag/tree/218ec52e09a7e7462a5400043bb9a69a41d06b76) | `218ec52e09a7` | 39,905 / 10,042 |
| `socialiqa` | [Official SocialIQA ZIP](https://storage.googleapis.com/ai2-mosaic/public/socialiqa/socialiqa-train-dev.zip) | Archive SHA-256 in catalog | 33,410 / 1,954 |
| `arce` | [allenai/ai2_arc / ARC-Easy](https://huggingface.co/datasets/allenai/ai2_arc/tree/210d026faf9955653af8916fad021475a3f00453/ARC-Easy) | `210d026faf99` | 2,251 / 570 |
| `winogrande` | [allenai/winogrande / winogrande_xl](https://huggingface.co/datasets/allenai/winogrande/tree/01e74176c63542e6b0bcb004dcdea22d94fb67b5/winogrande_xl) | `01e74176c635` | 40,398 / 1,267 |

Run from the participant-tools root with Python 3.10+; no model/tokenizer or GPU
is needed. The script uses full pinned revisions, not the table's short labels:

```bash
python -m pip install huggingface_hub pyarrow
python participant_toolkit/prepare_reference_datasets.py \
  --dataset all --output /absolute/path/to/new-reference-data
python participant_toolkit/prepare_public_assets.py verify \
  --manifest /absolute/path/to/new-reference-data/reference-data-lock.json \
  --root /absolute/path/to/new-reference-data
```

Data preparation was tested with Python 3.13.5, huggingface_hub 2.2.0 and pyarrow 26.0.0. Verify the output hashes after preparation.

Replace `all` with a dataset ID to reconstruct one pair. Each output folder contains `train.csv` and `validation.csv`; `reference-data-lock.json` records the complete preparation. Source and output hashes are checked before publishing the folders. Use a new directory when retrying a failed preparation.

For preserved CSVs in existing locations:

```bash
python participant_toolkit/prepare_public_assets.py verify-reference-data \
  --dataset boolq --train /path/to/original-train.csv \
  --validation /path/to/original-validation.csv
```

### Split, sample order and labels

Use upstream **train** for updates and **validation** for evaluation. PIQA and SocialIQA use their `dev` split as `validation`. Preserve source row order and the six reference CSV columns: `sample_id,question,choices_json,answer,subject,num_choices`.

| Dataset | Question / options | Answer mapping |
|---|---|---|
| BoolQ | Passage then question; yes/no | label 1 → A (yes), 0 → B (no) |
| QNLI | Question, sentence and entailment question; yes/no | label 0 → A (yes), 1 → B (no) |
| PIQA | Goal; sol1/sol2 | labels 0/1 → A/B |
| HellaSwag | ctx; endings in source order | labels 0–3 → A–D |
| SocialIQA | Context then question; answerA/B/C | labels 1–3 → A–C |
| ARC-Easy | Question; options sorted by numeric/alphabetic source labels | Locate answerKey after sorting; retain three- and five-option rows too |
| WinoGrande XL | Sentence with `_`; option1/option2 | answers 1/2 → A/B |

Keep native `sample_id` as metadata, **not a unique key**. HellaSwag has 39,905
training rows but 32,599 distinct native IDs, and 10,042 validation rows but
9,609 distinct native IDs. Do not deduplicate or rewrite the reference CSV.
For quality labels and predictions use the same stable
`<dataset-id>/<split>/<zero-based-row-index>`, e.g. `hellaswag/validation/0`.
The CSV hash and row index fix identity; the lock also records source-ID order.
For BoolQ map A/B to true/false only if the target quality schema requires it.

### Client partitions and runtime preprocessing

The preserved historical Python loader assigns raw row index `i` to client
`i % N` **before** tokenizer-specific filtering. Centralized uses all rows;
Local Only keeps separate client states. Changing N, row order or filtering
order changes partitions. The retained October 18 replay uses seed 7 and ten
Local Only clients; other device configs use three, five or eight clients.
These are not universal defaults. The short example's seed 42, two clients and
64/16 subsets are development settings. Record the actual client count,
ordered membership and training seed; the CSV export itself uses no RNG.

The historical loader constructs lettered prompts ending in `Answer: `, trains
on a space-prefixed answer letter, masks prompt/EOS labels, and may shorten
BoolQ passages or skip overlength training examples. Evaluation has its own
fixed-length handling.
Use the registered sequence lengths: 128 for BoolQ and 64 for other datasets. Preserve the six-column reference CSVs for the matrix runner. The short Qwen example uses its own `question,A,B,C,D,answer` schema.

The reference matrix entry point now makes those bindings in each prepared
`task.json` and saved checkpoint, and verifies model/data hashes at reload.
Its explicit run settings and fixed-context policy are documented in
[REFERENCE_MATRIX.md](integrations/pytorch/REFERENCE_MATRIX.md). Use the complete reference matrix for formal submissions.

## Public development assets

```bash
python -m pip install huggingface_hub datasets pyarrow transformers
python participant_toolkit/prepare_public_assets.py model \
  --model qwen25_05b --output /absolute/path/to/new-assets/Qwen2.5-0.5B
python participant_toolkit/prepare_public_assets.py boolq \
  --output /absolute/path/to/new-assets/boolq
```

Each command resolves a revision BEFORE downloading and writes `asset-lock.json`
with the immutable commit and per-file SHA-256. To repeat, pass
`--revision COMMIT_FROM_THE_LOCK`, not a moving main. Other model choices are
`gemma3_270m` and `gemma3_1b`.

```bash
python participant_toolkit/prepare_public_assets.py verify \
  --manifest /absolute/path/to/new-assets/boolq/asset-lock.json \
  --root /absolute/path/to/new-assets/boolq
export QWEN_MODEL_DIR=/absolute/path/to/new-assets/Qwen2.5-0.5B
export BOOLQ_TRAIN_ARROW=/absolute/path/to/new-assets/boolq/super_glue-train.arrow
export BOOLQ_VALIDATION_ARROW=/absolute/path/to/new-assets/boolq/super_glue-validation.arrow
export DATA_DIR=/absolute/path/to/new-assets/boolq-prepared
python examples/qwen25_boolq/prepare_mobilefinetuner_data.py \
  --model-dir "$QWEN_MODEL_DIR" --train-arrow "$BOOLQ_TRAIN_ARROW" \
  --validation-arrow "$BOOLQ_VALIDATION_ARROW" --output-dir "$DATA_DIR"
```

BoolQ uses public train for updates and validation for quality. Conversion
defaults to a small development subset, not an official split. Set limits to
zero only when a full public development split is intended.

## Official release scope

Formal submissions use the release’s pinned models, tokenizer files, dataset sources, splits and export recipe. The [reference runtime](integrations/pytorch/REFERENCE_MATRIX.md) prepares each task with the registered context length and evaluates saved checkpoints. Set client count, seed, rounds and local steps explicitly.

Prepare the three models at the revisions listed above, verify their file hashes and save the model/data locks with the run. For formal submission, use the approved release, matching ZIP pair, full reference datasets and saved server policy.

The verifier also accepts organizer manifests containing a `files` mapping
from relative filenames to SHA-256. Preserve model/data locks with results.