Build Your Annotation Taxonomy Once, Not Three Times

A team starts a labeling project with a 12-class taxonomy. Three months in, they discover class 7 is actually two classes. They split it. Re-label affected samples. Six months in, two classes really should merge. Relabel. Nine months in, the schema needs an attribute (color of vehicle, severity of defect). Re-label.

By the end, the same dataset has been labeled three times. The fix isn't to predict your taxonomy perfectly up front — it's to design for evolution.

Why taxonomies always change

Two reasons. First, you discover edge cases that don't fit your initial categories. The 'pedestrian' class needs to split into 'standing', 'walking', 'wheelchair user' because your model behaves differently on each. Second, downstream model requirements shift. The product manager decides the model should also predict severity, not just type. Both happen, predictably, in every long-running labeling project.

Design rules that survive evolution

Hierarchies, not flat lists. 'Vehicle / Car / Sedan' lets you split or merge sub-classes without breaking the parent class. 'Defect' as a top-level class with sub-types lets you add new defect types in month 6 without disturbing existing labels.

Attributes as a separate dimension. Color, size, condition, occlusion percentage — these are properties of an annotation, not part of the class hierarchy. Tools that conflate the two force you to multiply classes (redcar, blue-car, green-car) which is its own disaster.

Reserved 'unsure' and 'other' classes. Annotators encountering an ambiguous case need somewhere to put it. 'Other' becomes the source of your next taxonomy revision — when it grows past 5%, you know there's an unmet class waiting to be defined.

Version-control everything

Treat taxonomy like code. Every change has a version number, a date, a reason, and a migration policy for existing labels. Without this, six months later you'll have annotations under three subtly different schemas and no way to reconcile them.

Tools that don't version-control the schema force you to manage this with spreadsheets, which is where it falls apart at scale.

The migration policy is more important than the change

When taxonomy v2 ships, define what happens to v1 labels. Three options. Forward-migrate — automatically map old labels to new schema where possible, flag ambiguous cases for human review. Relabel — invalidate old labels and label affected samples again (expensive but cleanest). Coexist — keep v1 labels labeled v1, label new data with v2, accept inconsistency in the combined dataset (sometimes the right call for very stable old data).

The wrong move is to not decide, which is what most teams do, ending up with implicit chaos.

Pilot before committing

Before formalizing a taxonomy, pilot-label 200-500 representative samples with the proposed schema. Calibrate two annotators independently. Compare. If inter-annotator agreement is below 70% on the initial schema, your taxonomy is ambiguous — revise before scaling.

This 1-day exercise saves months of misalignment downstream.

Where the platform helps

Intellabel's class definition and taxonomy features are version-controlled by default, with explicit migration paths between schema versions. When you bump v1 to v2, the platform helps you decide what happens to existing labels — forward-migrate, re-label, or coexist — instead of leaving you to manage it in a spreadsheet.

Taxonomies that evolve cleanly are not the result of perfect foresight. They're the result of treating taxonomy as a first-class versioned asset, not a checkbox at project kickoff.

From Labeling to Structured AI Data Pipelines