> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.toloka.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Label, train, and evaluate

> Turn completed Data Labeling results into a fine-tuning dataset, keep a golden set separate, and score predictions from your trained model locally.

This walkthrough connects Data Labeling and Train for a support-ticket classifier. You will export completed labels, keep a deterministic golden set outside Training, fine-tune on the remaining rows, and calculate exact-match accuracy from predictions produced by the trained model.

```text theme={null}
pipeline export
  ├─ completed rows selected for training → training.jsonl → Toloka Training
  └─ completed rows selected for evaluation
       ├─ prompts → your inference tooling → predictions.jsonl
       └─ hidden labels → local scorer
```

> [!NOTE]
>
> Toloka Training produces a Model Version that you download. Deploying it and generating predictions are jobs for the inference tooling you operate. The scorer on this page is a local Python script, not a built-in Toloka Evaluate product.

## Before you start

You need:

* A Data Labeling pipeline with a text input named `ticket_text` and a resolved string output named `intent`.
* At least 12 completed, labeled items for the small split in this walkthrough: 2 for local evaluation and at least 10 to upload for Training. Exact duplicate prompts stay together, so you need more rows if a duplicate group leaves fewer than 10 to upload. A production evaluation needs a larger set that represents the data you expect to see.
* The Toloka CLI installed, signed in, and set to a project where your credentials can create and upload datasets, create Trainings, and deploy Training Runs. For browser login, use a team with **Full access**; [switch teams](https://docs.platform.toloka.ai/docs/core/teams-and-balance.md#switch-teams) if needed. For an API key, check [CLI permissions](https://docs.platform.toloka.ai/docs/cli.md#permissions).
* A funded team balance. Cost estimation is free; starting a Training Run is billed.
* Python 3.9 or later for the local conversion and scoring scripts.
* Your own compatible serving stack for the downloaded model files.

This walkthrough uses supervised fine-tuning. The conversation data shape also works for gisting, but gisting additionally needs a prompt file and produces a different model artifact. Follow [Compress a prompt with gisting](https://docs.platform.toloka.ai/docs/train/guides/gist.md) if that is your chosen method.

**1. Label the examples**

Build and run a pipeline that receives `ticket_text` and returns `intent` from its End node. For example, `intent` might be one of `question`, `complaint`, or `praise`.

[Get your first labeled dataset](https://docs.platform.toloka.ai/docs/data/get-started/first-labeled-dataset.md) covers the complete pipeline cycle. Check that each completed item's `intent` is `question`, `complaint`, or `praise` and correctly labels its `ticket_text`.

**2. Export one finished pipeline run**

Open the run's **Items** tab and create an export as described in [Export your labeled results](https://docs.platform.toloka.ai/docs/data/guides/export-results.md). Download the JSON file after the export reaches `completed`.

Create the export after the run finishes: the file will not include results produced later. The converter uses each entry's `index`, `status`, `input.ticket_text`, and `output.intent`:

```json theme={null}
{
  "index": 41,
  "status": "completed",
  "input": { "ticket_text": "I was charged twice for one subscription." },
  "output": { "intent": "complaint" }
}
```

The conversion ignores all status values other than `completed`. See [Export format](https://docs.platform.toloka.ai/docs/data/reference/export-format.md) for the complete entry contract.

**3. Create training and evaluation partitions**

Follow [Prepare training data from labels](https://docs.platform.toloka.ai/docs/train/guides/prepare-training-data.md) to save `prepare_label_export.py`. Then run:

```bash theme={null}
python3 prepare_label_export.py labeled-results.json \
  --input-field ticket_text \
  --label-field intent \
  --evaluation-count 2 \
  --output-dir prepared
```

The converter assigns exact duplicate prompts to the same partition and makes the selection deterministic. It writes:

| File | Contains | Destination |
| - | - | - |
| `prepared/training.jsonl` | User messages and final assistant labels | Upload to Training |
| `prepared/evaluation-inputs.jsonl` | Evaluation indexes and user messages | Your inference tooling |
| `prepared/evaluation-gold.jsonl` | Evaluation indexes and expected labels | Keep local for scoring |

Open all three files. Confirm that the indexes in the two evaluation files match each other and that the training rows contain assistant labels while the evaluation inputs do not. The training file deliberately omits source indexes; use the converter's printed counts to inspect the partition rather than adding indexes or golden labels to a model request.

**4. Upload only the training rows**

Create a dataset in the selected project and upload the training partition:

```bash theme={null}
toloka dataset create support-ticket-training
toloka dataset upload support-ticket-training ./prepared/training.jsonl
toloka dataset list
```

Copy the `ds.…` ID printed by `dataset list`. Uploads append, so run the upload once unless you deliberately want duplicate examples.

Each uploaded line has the supported message shape:

```json theme={null}
{"messages":[{"role":"user","content":"I was charged twice for one subscription."},{"role":"assistant","content":"complaint"}]}
```

`evaluation-inputs.jsonl` has no assistant answer, so it is not a training dataset. `evaluation-gold.jsonl` is not uploaded at all.

**5. Create and price the fine-tuning run**

List the current Base models, create an SFT Training, and calculate a free cost estimate:

```bash theme={null}
toloka base-model list
toloka training create "Support ticket classifier" \
  --base-model qwen3-4b \
  --dataset ds.YOUR_DATASET_ID
toloka training cost trg.YOUR_TRAINING_ID --json > estimate.json
```

Copy the `trg.…` Training ID from `training create`. The cost command validates the chat rows and saves the estimate to `estimate.json`. Open that file, require `status: "ready"`, and review `maximumCharge` before continuing. `training create` creates an 80% train, 10% validation, 10% test split for this new dataset and freezes a slice. With the minimum 10 uploaded rows, expect 8 training rows and 1 validation row; 1 test row stays out of learning. All three parts remain separate from the local golden set.

> [!WARNING]
>
> Starting a Training Run places a hold on the team balance and starts billed work. Use the saved estimate you reviewed with `--estimate-file` so the run uses that approved maximum charge, rather than calculating a new estimate at launch. If the saved estimate is rejected or expires, calculate and review a new one before retrying.

[Fine-tune a model](https://docs.platform.toloka.ai/docs/train/guides/fine-tune.md) explains the settings, states, billing behavior, and failure messages in detail.

**6. Run Training and download the Model Version**

When you accept the saved estimate, start one Training Run with that estimate and follow it to completion:

```bash theme={null}
toloka training run trg.YOUR_TRAINING_ID --estimate-file estimate.json
toloka training logs trn.YOUR_RUN_ID --follow
toloka training status trn.YOUR_RUN_ID
toloka training metrics trn.YOUR_RUN_ID
```

Copy the `trn.…` ID from `training run`. A completed run publishes a Model Version. List and download it:

```bash theme={null}
toloka model versions "Support ticket classifier"
toloka model version download mdv.YOUR_VERSION_ID \
  --out ./support-ticket-classifier
```

The SFT result is a LoRA adapter Model Version. Your serving stack must load it with the matching Base model. Use the downloaded version for the next step, not the unchanged Base model by itself.

Training metrics such as validation loss and validation token accuracy describe results on the validation part. They do not replace the task-level exact-match evaluation below.

**7. Generate predictions with your serving stack**

Deploy the downloaded model files with tooling you operate.

For every line in `prepared/evaluation-inputs.jsonl`:

1. Send only the `messages` value to the deployed trained model.
2. Record the model's actual response without looking up the matching golden label.
3. Write the response with the same `index` to `predictions.jsonl`.

For two evaluation items, the result has this shape:

```json theme={null}
{"index":41,"prediction":"complaint"}
{"index":87,"prediction":"question"}
```

There must be exactly one prediction for every evaluation index. Do not hand-correct a response before writing it; the purpose is to measure what the deployed Model Version actually returned.

**8. Score the actual predictions locally**

Save the following as `score_predictions.py`:

```python theme={null}
import argparse
import json
from pathlib import Path


def load_values(path, value_field):
    values = {}
    with path.open(encoding="utf-8") as source:
        for line_number, line in enumerate(source, start=1):
            if not line.strip():
                continue
            row = json.loads(line)
            if not isinstance(row, dict):
                raise SystemExit(
                    f"{path}:{line_number} is not a JSON object."
                )
            index = row.get("index")
            value = row.get(value_field)
            if not isinstance(index, int):
                raise SystemExit(
                    f"{path}:{line_number} has no integer index."
                )
            if not isinstance(value, str):
                raise SystemExit(
                    f"{path}:{line_number} has no string {value_field}."
                )
            if index in values:
                raise SystemExit(f"{path} repeats index {index}.")
            values[index] = value.strip()
    return values


parser = argparse.ArgumentParser()
parser.add_argument("gold", type=Path)
parser.add_argument("predictions", type=Path)
args = parser.parse_args()

gold = load_values(args.gold, "expected")
predictions = load_values(args.predictions, "prediction")
if not gold:
    raise SystemExit("The golden set is empty.")
missing = sorted(set(gold) - set(predictions))
unexpected = sorted(set(predictions) - set(gold))
if missing or unexpected:
    raise SystemExit(
        f"Prediction indexes do not match gold. "
        f"Missing: {missing}; unexpected: {unexpected}"
    )

mismatches = [
    {
        "index": index,
        "expected": gold[index],
        "prediction": predictions[index],
    }
    for index in sorted(gold)
    if predictions[index] != gold[index]
]
correct = len(gold) - len(mismatches)
print(
    f"Exact-match accuracy: {correct}/{len(gold)} "
    f"= {correct / len(gold):.1%}"
)
for mismatch in mismatches:
    print(json.dumps(mismatch, ensure_ascii=False))
```

Run it against the hidden answers and the predictions produced in the previous step:

```bash theme={null}
python3 score_predictions.py \
  ./prepared/evaluation-gold.jsonl \
  ./predictions.jsonl
```

If one of two predictions is correct, it prints:

```text theme={null}
Exact-match accuracy: 1/2 = 50.0%
{"index": 87, "expected": "praise", "prediction": "question"}
```

The script refuses to score a partial or mismatched set of indexes. For a real decision, use enough held-out examples for every important class; two rows only demonstrate the mechanics.

## Interpret the result without contaminating the holdout

Exact match is suitable here because each expected answer is one of a small set of labels. A generative task needs a metric and review process appropriate to its output; do not treat token accuracy from Training as a substitute for that task metric.

Keep the same local evaluation set when comparing Model Versions so the comparison is like for like. If you inspect its answers repeatedly and change prompts, labels, or training data to improve those exact examples, it has become a development set. Create a new untouched golden set before making a final quality claim.

To add more labeled data, run the labeling and export cycle again, remove duplicates across cycles using a stable source identifier, and create a fresh deterministic split. Never move the current golden answers into training while continuing to report results on that same set.

## Related

* [Prepare training data from labels](https://docs.platform.toloka.ai/docs/train/guides/prepare-training-data.md)
* [Fine-tune a model](https://docs.platform.toloka.ai/docs/train/guides/fine-tune.md)
* [Training Run states](https://docs.platform.toloka.ai/docs/train/concepts/how-training-works.md#training-run-states)
* [Train your first model](https://docs.platform.toloka.ai/docs/train/get-started/first-trained-model.md)
* [Export your labeled results](https://docs.platform.toloka.ai/docs/data/guides/export-results.md)
* [Export format](https://docs.platform.toloka.ai/docs/data/reference/export-format.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.