Video Annotation & Object Tracking

Temporally consistent video datasets — where an object keeps the same identity from the frame it enters to the frame it leaves, through occlusion and back again.

Video is not a stack of independent images

The most common failure in video annotation is treating each frame as a separate labelling task. Do that and the geometry may be perfect on every individual frame while the dataset is still useless for tracking, because object identities are not stable across them. A model trained on identity-inconsistent data learns to re-detect rather than to track.

Temporal consistency is therefore the primary quality criterion, not a secondary one. An object that leaves the frame and returns should be reasoned about explicitly — whether it takes a new identity or resumes its previous one is a specification decision that has to be made once and applied everywhere.

What we annotate in video

Our video work covers:

  • Multi-object tracking — persistent identities across frames, with documented handling of entry, exit, occlusion, and re-entry.
  • Frame-level object detection — boxes, polygons, or masks applied per frame where tracking is not required.
  • Action and activity recognition — labelled temporal segments marking where an action begins and ends.
  • Event segmentation — boundary marking for scene changes, state transitions, and process steps.
  • Video classification — whole-clip or segment-level tagging.
  • Object interpolation review — machine-interpolated frames checked and corrected by a human rather than accepted on trust.

Keyframes, interpolation, and where it breaks

Annotating every frame of long-form video by hand is rarely justified. The standard approach is to annotate keyframes and interpolate between them, which is efficient and accurate as long as motion between keyframes is close to linear.

It stops being accurate exactly where the interesting behaviour is: sharp direction changes, acceleration, partial occlusion, and objects that deform. We set keyframe density from observed motion rather than a fixed interval, increase it around those events, and review interpolated spans rather than shipping them unchecked. Where a project needs guaranteed per-frame accuracy, we say so and price it as per-frame work instead of presenting interpolation as equivalent.

Quality assurance for temporal data

Every delivery passes through a layered review rather than a single pass. Annotators work to a written specification agreed before production starts; a second reviewer checks a defined proportion of each batch; and a final quality gate samples the batch independently against the same specification. Disagreements are not silently overwritten — they are resolved against the guideline, and where the guideline turns out to be ambiguous, the guideline is amended and the affected items are reworked.

That last step matters more than it sounds. Most systematic annotation error is not carelessness; it is a specification that failed to anticipate a real case. Treating every disagreement as a possible guideline defect is what keeps error rates from plateauing partway through a project.

Video adds failure modes that per-frame checks do not catch. We review for identity switches between objects, track fragmentation where one object is split into several tracks, and drift where a box gradually loses its subject across a long span. These are checked by playing the annotated sequence back, because they are visible in motion and effectively invisible frame by frame.

Frame rate and sampling

Not every project needs every frame. Sampling to a lower rate often produces an equally capable model at a fraction of the annotation cost — but the right rate depends on how fast your subjects move relative to the frame, and choosing it badly quietly destroys fast events.

We recommend a rate against your actual footage rather than a default, and flag where a proposed rate would drop events the model is meant to learn.

Data handling

Client data is encrypted in transit and at rest, access is limited to the specialists assigned to your project, and every member of the annotation workforce works under a signed confidentiality agreement. Where a dataset carries additional handling constraints, those are agreed in writing before any data is transferred.

Common questions

What is the difference between video annotation and frame-by-frame image annotation?

Frame-by-frame annotation labels each image independently, so object identity is not preserved between frames. Video annotation maintains a persistent identity for each object across its whole appearance, which is what tracking models need. Geometrically perfect per-frame labels are still unusable for tracking if identities are inconsistent.

Do you annotate every frame?

Only where the project requires it. Usually we annotate keyframes and interpolate, with keyframe density set from actual motion and increased around direction changes and occlusion. Interpolated spans are reviewed rather than accepted unchecked, and we quote per-frame work as per-frame work when that is what is needed.

How do you handle an object that leaves the frame and comes back?

That is a specification decision made before production: either the object resumes its original identity or it takes a new one. Whichever is chosen is applied consistently across the dataset and recorded with the delivery, because mixed handling teaches the model contradictory behaviour.

Discuss your video annotation project

Tell us what you are building and what your data has to support. We will come back with a specification, a pilot scope, and a realistic timeline.

Start a conversation