Label, train, and evaluate

This walkthrough connects Data Labeling and Train for a support-ticket classifier. You will export completed labels, keep a deterministic golden set outside Training, fine-tune on the remaining rows, and calculate exact-match accuracy from predictions produced by the trained model.

pipeline export
  ├─ completed rows selected for training → training.jsonl → Toloka Training
  └─ completed rows selected for evaluation
       ├─ prompts → your inference tooling → predictions.jsonl
       └─ hidden labels → local scorer

Toloka Training produces a Model Version that you download. Deploying it and generating predictions are jobs for the inference tooling you operate. The scorer on this page is a local Python script, not a built-in Toloka Evaluate product.

Before you start

You need:

A Data Labeling pipeline with a text input named ticket_text and a resolved string output named intent.
At least 12 completed, labeled items for the small split in this walkthrough: 2 for local evaluation and at least 10 to upload for Training. Exact duplicate prompts stay together, so you need more rows if a duplicate group leaves fewer than 10 to upload. A production evaluation needs a larger set that represents the data you expect to see.
The Toloka CLI installed, signed in, and set to a project where your credentials can create and upload datasets, create Trainings, and deploy Training Runs. For browser login, use a team with Full access; switch teams if needed. For an API key, check CLI permissions.
A funded team balance. Cost estimation is free; starting a Training Run is billed.
Python 3.9 or later for the local conversion and scoring scripts.
Your own compatible serving stack for the downloaded model files.

This walkthrough uses supervised fine-tuning. The conversation data shape also works for gisting, but gisting additionally needs a prompt file and produces a different model artifact. Follow Compress a prompt with gisting if that is your chosen method.

Label the examples

Build and run a pipeline that receives ticket_text and returns intent from its End node. For example, intent might be one of question, complaint, or praise.Get your first labeled dataset covers the complete pipeline cycle. Check that each completed item’s intent is question, complaint, or praise and correctly labels its ticket_text.

Export one finished pipeline run

Open the run’s Items tab and create an export as described in Export your labeled results. Download the JSON file after the export reaches completed.Create the export after the run finishes: the file will not include results produced later. The converter uses each entry’s index, status, input.ticket_text, and output.intent:

{
  "index": 41,
  "status": "completed",
  "input": { "ticket_text": "I was charged twice for one subscription." },
  "output": { "intent": "complaint" }
}

The conversion ignores all status values other than completed. See Export format for the complete entry contract.

Create training and evaluation partitions

Follow Prepare training data from labels to save prepare_label_export.py. Then run:

python3 prepare_label_export.py labeled-results.json \
  --input-field ticket_text \
  --label-field intent \
  --evaluation-count 2 \
  --output-dir prepared

The converter assigns exact duplicate prompts to the same partition and makes the selection deterministic. It writes:

File	Contains	Destination
`prepared/training.jsonl`	User messages and final assistant labels	Upload to Training
`prepared/evaluation-inputs.jsonl`	Evaluation indexes and user messages	Your inference tooling
`prepared/evaluation-gold.jsonl`	Evaluation indexes and expected labels	Keep local for scoring

Open all three files. Confirm that the indexes in the two evaluation files match each other and that the training rows contain assistant labels while the evaluation inputs do not. The training file deliberately omits source indexes; use the converter’s printed counts to inspect the partition rather than adding indexes or golden labels to a model request.

Upload only the training rows

Create a dataset in the selected project and upload the training partition:

toloka dataset create support-ticket-training
toloka dataset upload support-ticket-training ./prepared/training.jsonl
toloka dataset list

Copy the ds.… ID printed by dataset list. Uploads append, so run the upload once unless you deliberately want duplicate examples.Each uploaded line has the supported message shape:

{"messages":[{"role":"user","content":"I was charged twice for one subscription."},{"role":"assistant","content":"complaint"}]}

evaluation-inputs.jsonl has no assistant answer, so it is not a training dataset. evaluation-gold.jsonl is not uploaded at all.

Create and price the fine-tuning run

List the current Base models, create an SFT Training, and calculate a free cost estimate:

toloka base-model list
toloka training create "Support ticket classifier" \
  --base-model qwen3-4b \
  --dataset ds.YOUR_DATASET_ID
toloka training cost trg.YOUR_TRAINING_ID --json > estimate.json

Copy the trg.… Training ID from training create. The cost command validates the chat rows and saves the estimate to estimate.json. Open that file, require status: "ready", and review maximumCharge before continuing. training create creates an 80% train, 10% validation, 10% test split for this new dataset and freezes a slice. With the minimum 10 uploaded rows, expect 8 training rows and 1 validation row; 1 test row stays out of learning. All three parts remain separate from the local golden set.

Starting a Training Run places a hold on the team balance and starts billed work. Use the saved estimate you reviewed with --estimate-file so the run uses that approved maximum charge, rather than calculating a new estimate at launch. If the saved estimate is rejected or expires, calculate and review a new one before retrying.

Fine-tune a model explains the settings, states, billing behavior, and failure messages in detail.

Run Training and download the Model Version

When you accept the saved estimate, start one Training Run with that estimate and follow it to completion:

toloka training run trg.YOUR_TRAINING_ID --estimate-file estimate.json
toloka training logs trn.YOUR_RUN_ID --follow
toloka training status trn.YOUR_RUN_ID
toloka training metrics trn.YOUR_RUN_ID

Copy the trn.… ID from training run. A completed run publishes a Model Version. List and download it:

toloka model versions "Support ticket classifier"
toloka model version download mdv.YOUR_VERSION_ID \
  --out ./support-ticket-classifier

The SFT result is a LoRA adapter Model Version. Your serving stack must load it with the matching Base model. Use the downloaded version for the next step, not the unchanged Base model by itself.Training metrics such as validation loss and validation token accuracy describe results on the validation part. They do not replace the task-level exact-match evaluation below.

Generate predictions with your serving stack

Deploy the downloaded model files with tooling you operate.For every line in prepared/evaluation-inputs.jsonl:

Send only the messages value to the deployed trained model.
Record the model’s actual response without looking up the matching golden label.
Write the response with the same index to predictions.jsonl.

For two evaluation items, the result has this shape:

{"index":41,"prediction":"complaint"}
{"index":87,"prediction":"question"}

There must be exactly one prediction for every evaluation index. Do not hand-correct a response before writing it; the purpose is to measure what the deployed Model Version actually returned.

Score the actual predictions locally

Save the following as score_predictions.py:

import argparse
import json
from pathlib import Path


def load_values(path, value_field):
    values = {}
    with path.open(encoding="utf-8") as source:
        for line_number, line in enumerate(source, start=1):
            if not line.strip():
                continue
            row = json.loads(line)
            if not isinstance(row, dict):
                raise SystemExit(
                    f"{path}:{line_number} is not a JSON object."
                )
            index = row.get("index")
            value = row.get(value_field)
            if not isinstance(index, int):
                raise SystemExit(
                    f"{path}:{line_number} has no integer index."
                )
            if not isinstance(value, str):
                raise SystemExit(
                    f"{path}:{line_number} has no string {value_field}."
                )
            if index in values:
                raise SystemExit(f"{path} repeats index {index}.")
            values[index] = value.strip()
    return values


parser = argparse.ArgumentParser()
parser.add_argument("gold", type=Path)
parser.add_argument("predictions", type=Path)
args = parser.parse_args()

gold = load_values(args.gold, "expected")
predictions = load_values(args.predictions, "prediction")
if not gold:
    raise SystemExit("The golden set is empty.")
missing = sorted(set(gold) - set(predictions))
unexpected = sorted(set(predictions) - set(gold))
if missing or unexpected:
    raise SystemExit(
        f"Prediction indexes do not match gold. "
        f"Missing: {missing}; unexpected: {unexpected}"
    )

mismatches = [
    {
        "index": index,
        "expected": gold[index],
        "prediction": predictions[index],
    }
    for index in sorted(gold)
    if predictions[index] != gold[index]
]
correct = len(gold) - len(mismatches)
print(
    f"Exact-match accuracy: {correct}/{len(gold)} "
    f"= {correct / len(gold):.1%}"
)
for mismatch in mismatches:
    print(json.dumps(mismatch, ensure_ascii=False))

Run it against the hidden answers and the predictions produced in the previous step:

python3 score_predictions.py \
  ./prepared/evaluation-gold.jsonl \
  ./predictions.jsonl

If one of two predictions is correct, it prints:

Exact-match accuracy: 1/2 = 50.0%
{"index": 87, "expected": "praise", "prediction": "question"}

The script refuses to score a partial or mismatched set of indexes. For a real decision, use enough held-out examples for every important class; two rows only demonstrate the mechanics.

Interpret the result without contaminating the holdout

Exact match is suitable here because each expected answer is one of a small set of labels. A generative task needs a metric and review process appropriate to its output; do not treat token accuracy from Training as a substitute for that task metric. Keep the same local evaluation set when comparing Model Versions so the comparison is like for like. If you inspect its answers repeatedly and change prompts, labels, or training data to improve those exact examples, it has become a development set. Create a new untouched golden set before making a final quality claim. To add more labeled data, run the labeling and export cycle again, remove duplicates across cycles using a stable source identifier, and create a fresh deterministic split. Never move the current golden answers into training while continuing to report results on that same set.

Start

Data Labeling

Train

Workflows

Admin

Before you start

Interpret the result without contaminating the holdout

​Before you start

​Interpret the result without contaminating the holdout

​Related

Before you start

Interpret the result without contaminating the holdout

Related