Skip to content

Creating Ultralytics YOLO Models

Introduction

The VisionCapsules® AI Marketplace can create a capsule from an Ultralytics YOLO model without any capsule code. It downloads the weights from Ultralytics, converts them to OpenVINO on your own server, fills in one of the capsule templates published in the ultralytics directory of the Capsule Zoo, and packages the result into the open vision capsule format (a .cap file) in My VisionCapsules.

This page covers:

  • how to download and package a model in the AI Marketplace;
  • what each kind of model reports, and which template it uses;
  • how to use the templates directly, to build a capsule yourself;
  • how the vocabulary, the default class names and the classes option of a capsule work;
  • what the number of baked-in classes costs for open-vocabulary models (YOLO-World and YOLOE);
  • how a prompt-free YOLOE capsule gets faster;
  • which Ultralytics models are not packaged yet.

Note

AOTU does not distribute these capsules. They are built on your machine from the Ultralytics weights, which are licensed under the Ultralytics AGPL-3.0 License. The AI Marketplace asks you to accept the license before it packages a model. The server needs internet access to download the weights, and the packaging service must be running next to the marketplace (the dialog shows a warning if it is not).

Download and Package a Model

  1. Open the AI Marketplace.
  2. In Explore VisionCapsules, choose Ultralytics Public in the source menu. The list shows the weights that Ultralytics publishes, grouped by family (YOLO26, YOLO11, YOLOv8, ...). Use the model version menu to show one family.
  3. Click the download icon at the end of a row, or tick several rows and click Download & Package Selected.
  4. The dialog Package Model into Open Vision Capsules Format opens for the first model, filled in from the model you chose. The source, model version, task, format, weights file and input size are fixed; you can change the capsule name and version. When you chose several models, each one gets its own step, one after another. OK goes to the next model; it is enabled once you have accepted the license and the fields are valid. Cancel drops that model only. Depending on the model, the step shows more fields after the common ones:

    Model What the step shows
    Detection and segmentation The model's own class names, read from the weights and shown read-only, and how many there are (Maximum Vocabulary classes). The default class names are optional and editable: the capsule's classes option starts with them (empty: every class), and can be changed at run time.
    Oriented boxes (-obb) The same class names. The default class names are greyed out: the classes option is not available for these models.
    Pose (-pose) The same, and the keypoint names, which start with the 17 COCO person keypoints (nose, left_eye, ... right_ankle) and can be edited. Empty means the same names for a model with 17 keypoints, and numbered names (keypoint_1, ...) for another number.
    Classification (-cls) The model's own 1,000 ImageNet class names, read-only, and the default class names (optional, editable: the classes the attribute can take; empty: all 1,000). Then the classes of the detections to classify (for example person) and the name of the attribute to set.
    YOLO-World and YOLOE The vocabulary (it starts with the 80 COCO classes; up to the maximum number of classes, 100 by default) and the default class names. See Vocabulary and Default Class Names.
    YOLOE prompt-free (-pf) The same two boxes. The vocabulary is fixed (4,585 classes) and read-only; you choose the default class names.

    The class names are read when the step opens. The weights are downloaded for that, and the first time the converter is installed, which takes minutes. You can press OK without waiting; the weights are downloaded again when packaging.

  5. OK on the last model starts the download and the packaging of all the models. The dialog then shows the progress of each model (downloading, converting, testing, packaging). Each model appears under the name it will have in My VisionCapsules, for example Detector Yolo26s Seg.

  6. When a model is done, it is in My VisionCapsules, named detector_<model> (classifier_<model> for classification), for example detector_yolo26n. Load it with the right arrow or Run Selected, and enable it on a stream like any other capsule.

The first time, the converter (PyTorch and Ultralytics, about 1.5 GB) is installed. YOLO-World and YOLOE also download a text encoder (350 to 600 MB). This can take several minutes; later models are much faster.

To package a model that is not in the list, for example one you trained yourself, click Package Model in My VisionCapsules and provide the .pt file, the OpenVINO files (.xml and .bin) or an ONNX file. The form has the same fields as the dialog above, and you fill them in yourself.

Capsule Name and Version

The capsule name is generated from the model (for example detector_yolo26s_seg); a number is added if the name is already taken. The Version is a whole number from 1: it is the capsule's own version, written in its capsule.py (version = 1) and reported by BrainFrame. The capsule format requires an integer there. In My VisionCapsules, the version column shows the model family and the version, for example YOLOE | v1, like the v0.32.0 | v1.0 of the capsules in the AOTU list, which show the release and the release version.

What the Capsule Reports

Task What each detection contains
Object detection A class name, a bounding box and a confidence.
Instance segmentation A polygon that outlines the object, derived from its mask. The mask_threshold option tunes the outline.
Pose estimation A bounding box, and the keypoints in the extra data keypoints, as [x, y, confidence] in frame pixels.
Oriented boxes A rotated rectangle (its four corners), and its angle in the extra data.
Image classification An attribute set on the detections of another capsule. A classifier capsule follows a detector.

The Capsule Templates

The packager chooses the template from the output of the exported model, so one template serves every model that exports in the same way.

Models Task Template
YOLOv3u, YOLOv5u, YOLOv8, YOLOv9, YOLO11, YOLO12, YOLO-World, YOLOE (all) Detect yolov5_anchor_free
YOLOv5 (original; ONNX or OpenVINO files only) Detect yolov5_anchor
YOLO26, YOLOv10 Detect yolo_end2end
RT-DETR Detect rtdetr
YOLOv8, YOLOv9, YOLO11, YOLOE (all), FastSAM Segment segment
YOLO26 Segment segment_end2end
YOLOv8, YOLO11 Pose pose
YOLO26 Pose pose_end2end
YOLOv8, YOLO11 Oriented boxes obb
YOLO26 Oriented boxes obb_end2end
YOLOv8, YOLO11, YOLO26 Classify classify

The *_end2end templates are for models that already remove duplicate boxes, so their capsules have no iou_threshold option. The other templates run non-maximum suppression (NMS) in the capsule.

These variants use the same templates as their family:

  • the Open Images V7 (-oiv7) and 1280 px (x6, -pose-p6) YOLOv8 models;
  • the Objects365 (-objv1-150, -objv1-seg), distilled and text (-text) YOLO26 models;
  • the demo models of Ultralytics Solutions, which are listed under their YOLO version: anpr-demo-model (a YOLO11 nano license plate detector) and solutions_ci_parking_model (a YOLOv8 nano model trained on VisDrone).

FastSAM is built on YOLOv8-seg and works in its "segment everything" mode: every object gets the class object.

Using the Templates Yourself

The templates are in the ultralytics/templates directory of the Capsule Zoo:

templates/
  common/         capsule.py, backend.py, preprocess.py, nms.py and meta.conf: every capsule gets these
  <template>/     the files for one model output (decode.py, ...): they replace the ones in common/

segment_end2end, pose_end2end and obb_end2end build on segment, pose and obb. To make a capsule by hand:

  1. Copy common/ and one template into a new capsule directory.
  2. Fill in the placeholders (__CAPSULE_NAME__, the class names, and so on).
  3. Add the OpenVINO model.xml and model.bin.
  4. Package the capsule as described in Creating an OpenVINO Object Detector Capsule.

The README.md in that directory lists the output of each template and explains how to add the classes option to a template of your own.

Choosing the Classes of a Capsule

A capsule made from a detection, segmentation or classification model has a classes option, which selects the classes the capsule reports. The marketplace sets its starting value when it packages the model, and you can change it in BrainFrame at any time. This section explains the terms and what the option does.

Vocabulary and Default Class Names

Two terms are used for the class names of such a capsule:

  • Vocabulary: all the classes the capsule can report. Nothing outside it can be reported.
  • Default class names: the starting text of the capsule's classes option. They decide what the capsule reports until someone changes the option; they are not a limit.

What they are depends on the model:

  • YOLO-World and text-prompt YOLOE. You write the vocabulary, one class per line. It starts with the 80 COCO classes, and can hold up to the maximum number of classes: 100 by default, and up to 1,000 (the dialog warns that more classes make the capsule slower). The default class names start with person, car, truck, bus, bicycle and motorcycle. Every default class name must be in the vocabulary, and neither box can be empty.
  • Prompt-free YOLOE. The vocabulary is the model's own list of 4,585 class names, which cannot be changed. You choose the default class names, which must be in that list.
  • The other models (for example YOLOv8, YOLO11, YOLO26 and the classifiers). The vocabulary is the model's own list of class names (80 for COCO, 1,000 for ImageNet), read from the weights. The default class names are optional; empty means every class. When you package your own file in Package Model, the list is the one you type in the form, or the names stored in the .pt file.

When you choose a .pt file in Package Model, the dialog reads its class names in the background (the first time, it installs the converter). A prompt-free YOLOE file shows its 4,585 names, read-only; for any other file you edit the vocabulary, which starts with the 80 COCO classes.

The capsule gets the vocabulary as its classes, and its classes option starts with the default class names. In BrainFrame the option can be changed to any names of the vocabulary; empty means every class of the vocabulary.

The option replaces the default class names; it does not narrow them. For a capsule packaged with the default class names person, bench, backpack, setting the option to dog reports dogs, as long as dog is in the vocabulary (one of the 80 COCO classes, or typed into the vocabulary box; every name of the vocabulary of a prompt-free model). A name that is not in the vocabulary reports nothing.

The Classes Option

The classes option holds the class names to report, separated by commas (for example person, car). Empty means every class of the capsule. Capsules of pose and oriented box models and of RT-DETR do not have the option. You change it in the capsule options in BrainFrame, without packaging again, and the detections follow at once.

  • What it can choose: the vocabulary the capsule was packaged with. A name that is not in the vocabulary is skipped, and the capsule's log says so. Names are matched ignoring upper and lower case, and a space and an underscore are the same (the ImageNet names have underscores, for example great_white_shark).
  • The starting value is the default class names typed in the dialog.
  • How it is applied: in the decode, selected_classes() in selection.py turns the text into class numbers, and decode() takes the best class among those only (the "filter before" of the experiment below).
  • End-to-end models. YOLO26 and YOLOv10 name one class per box inside the model, so the option only drops the boxes of the other classes: a box the model names person is never reported as bicycle. For this reason YOLOE-26 is packaged with its raw head and not the end-to-end one, so the option can move a box from person to bicycle.
  • Classifiers. The option chooses which classes the attribute can take, not which detections to report. The best of the selected classes is set, with its own probability (the model's probability of that class, not shared out again among the selected ones), so the confidence can be small when the picture does not look like any of them. The min_confidence option then decides whether the attribute is set. With no name known to the model, no attribute is set.
  • Processing time. For the models with their own class names the option is a filter, not a speed-up, because their class head is small. On one image, a YOLO11 nano detector took about 85 ms per frame with every class and about 90 ms with person only, and a YOLO26 nano segmentation capsule about 180 ms either way (the same machine, busy, so the difference is within the noise of the measurement). Only a prompt-free YOLOE capsule gets faster, because its class head is cut down to the selected classes (see Prompt-Free Models: Processing Time).

Open-Vocabulary Models: How Many Classes to Bake In

YOLO-World and the text-prompt YOLOE models are open-vocabulary: you give them class names, and they detect those classes. The converter turns every name into a text embedding and bakes the embeddings into the model, so the capsule needs no text encoder at run time. The classes you type when packaging are the classes the model knows.

This section measures what the number of baked classes costs. Use it to decide how many classes to bake in, and how many of them to expose in the capsule's classes option.

Note

The classes of the vocabulary are baked into the model, and the capsule's classes option selects among them at run time (see The Classes Option). The measurements below show what the size of the vocabulary costs.

Test Setup

Item Value
Models YOLO-World v2 small (yolov8s-worldv2) and YOLOE v8 small (yoloe-v8s-seg, boxes only)
Baked classes 3 (person, car, chair), 80 (the COCO classes), 300, 500, 1,000, 2,000 and 4,000 (the 80 COCO classes plus names from the YOLOE prompt-free vocabulary)
Template yolov5_anchor_free: its decode() takes the best class of every candidate box
Data The 128 images of COCO128, scored on person, car and chair
Accuracy method Letterboxed 640 x 640 input, confidence threshold 0.001, class-aware NMS at IoU 0.7, at most 300 detections per image, AP averaged over IoU 0.5 to 0.95
Processing time method One image at a time, the median of 125 frames, OpenVINO 2024.6 on a 20-core CPU with default threading
Conversion Ultralytics 8.4.214, the version the converter installed at the time of the test

This is a small sample, one size of each model and one machine. Treat the numbers as a guide: differences of about 0.02 AP are within what a sample of this size can show.

Processing Time

Model time is one OpenVINO inference, up to the point where the raw output has been copied out. Decode time is the template's decode(): the best class of each of the 8,400 candidate boxes, the confidence threshold and the box conversion. Pre-processing is identical in every case, and NMS runs only on the few boxes above the threshold. Neither is included.

Model Baked classes Model time Decode time Total
YOLO-World v2 s 3 72 ms 0.2 ms 72 ms
80 84 ms 1 ms 85 ms
300 174 ms 7 ms 181 ms
500 166 ms 11 ms 177 ms
1,000 354 ms 39 ms 394 ms
2,000 466 ms 126 ms 592 ms
4,000 367 ms 65 ms 432 ms
YOLOE v8 s 3 126 ms 0.4 ms 126 ms
80 126 ms 1.8 ms 128 ms
300 155 ms 5 ms 160 ms
500 154 ms 10 ms 164 ms
1,000 231 ms 38 ms 269 ms
2,000 330 ms 112 ms 443 ms
4,000 254 ms 76 ms 330 ms

The rows for 3, 80 and 4,000 classes were measured first, on an idle machine. The rows for 300 to 2,000 classes were measured later, one at a time, on the same machine while BrainFrame was running, so they are not comparable with the first rows: this is why the 4,000-class rows are faster than the 2,000-class rows. Compare the 300 to 2,000 rows with each other, where the time grows with the number of classes.

With 4,000 classes the model output is about 135 MB per frame, and copying it out is part of the model time.

Accuracy

The table gives mAP50 over the three test classes, with mAP50-95 in brackets.

  • Filter after takes the best of all the baked classes for each box, then drops the boxes whose best class is not one you want.
  • Filter before looks only at the scores of the classes you want.
Model Baked classes Filter after Filter before
YOLO-World v2 s 3 0.45 (0.31) 0.45 (0.31)
80 0.48 (0.33) 0.49 (0.33)
300 0.42 (0.27) 0.46 (0.31)
500 0.40 (0.26) 0.45 (0.30)
1,000 0.34 (0.21) 0.45 (0.30)
2,000 0.28 (0.17) 0.45 (0.30)
4,000 0.22 (0.13) 0.44 (0.29)
YOLOE v8 s 3 0.47 (0.29) 0.47 (0.29)
80 0.47 (0.29) 0.47 (0.29)
300 0.47 (0.29) 0.47 (0.29)
500 0.47 (0.29) 0.47 (0.29)
1,000 0.47 (0.29) 0.47 (0.29)
2,000 0.47 (0.29) 0.47 (0.29)
4,000 0.47 (0.29) 0.47 (0.29)

Packaging Time and Size

Model 3 classes 80 classes 4,000 classes
YOLO-World v2 s 8 s, 25.5 MB 15 s, 25.6 MB 389 s, 29.6 MB
YOLOE v8 s 8 s, 23 MB 12 s, 23 MB 406 s, 26 MB

The capsule size for 300, 500, 1,000 and 2,000 classes is 25.9, 26.1, 26.6 and 27.6 MB for YOLO-World, and 23.1, 23.3, 23.7 and 24.4 MB for YOLOE.

The packaging time is dominated by the text encoder, at about 0.1 s per name on one CPU core. The capsule size hardly changes.

What the Results Show

  • The cost follows the number of baked classes, not the number you report. The model scores every baked class on every frame, and the decode takes the best of all of them. Selecting fewer classes in the classes option does not make a text-prompt model faster. (A prompt-free model is different: see Prompt-Free Models: Processing Time.)
  • The cost grows with the list. From 3 to 80 classes it added 12 ms per frame on YOLO-World and nothing measurable on YOLOE. From 500 to 1,000 classes the frame time rose by 1.6 times (YOLOE) to 2.2 times (YOLO-World), and at 2,000 it was about 3.3 times (YOLO-World) and 2.7 times (YOLOE) the 500-class time. At 4,000 classes the frame time was about 6 times (YOLO-World) and 2.6 times (YOLOE) that of 3 classes, and packaging took minutes. Whether that matters depends on your frame rate, your hardware and the number of streams.
  • The list can change the accuracy. For YOLO-World, 80 classes scored slightly higher than 3. A likely reason is that other everyday classes in the list absorb objects that are not yours, which would otherwise be taken for one of yours. A larger list also lowered YOLO-World a little even with the filter before (0.44 against 0.45), probably because its neck uses all the class embeddings; this cause was not tested. YOLOE was not affected in this test.
  • Do not take the best class over a larger list than the one you report. With 4,000 baked classes and the filter after, YOLO-World lost most of its person detections (AP50 0.73 to 0.10), because a similar name such as man or pedestrian won those boxes and they were then dropped. Choosing the best class among only the classes you want (filter before) restores it to 0.68. The loss starts early: YOLO-World's person AP50 was 0.73 with 3 classes, 0.66 with 300 and 0.41 with 1,000 under the filter after, and 0.70 to 0.71 with the filter before. The capsule's classes option uses the filter before, so a large vocabulary only hurts the accuracy when the option is left empty (every class).

Recommendations

  • Bake in the minimum. Bake in only the classes your use case needs. Every baked class is scored on every frame, whether or not the classes option has switched it off. The dialog starts with the 80 COCO classes, which cost about 12 ms per frame on YOLO-World and nothing measurable on YOLOE; the maximum is 1,000 classes, where the frame time was about 390 ms (YOLO-World) and 270 ms (YOLOE) on the test machine.
  • Set the classes option instead of leaving it empty. Empty reports every class of the vocabulary, and with a large vocabulary YOLO-World's accuracy drops (see above). Naming the classes you want avoids it.
  • In a capsule of your own, offer only the classes that users need to switch, because each one is baked in and carries the cost above, and use only the columns of the selected classes in decode() to take the best class.
  • Decide for your own use case. The tables are a starting point. Measure your model on your hardware with the classes you intend to use.
  • For a very large vocabulary, use a prompt-free YOLOE model, whose vocabulary is 4,585 classes. Its model is bigger than a text-prompt one, but its capsule cuts the class head down to the classes the classes option selects, so a capsule that reports a few classes is not much slower than a text-prompt one (see Prompt-Free Models: Processing Time).
  • To add a class that is not baked in, package the model again. The text encoder is not part of the capsule.

Prompt-Free Models: Processing Time

A prompt-free YOLOE model scores every box against all of its 4,585 classes. Its output is 4,621 values per box (4 for the box, 4,585 class scores and 32 for the mask) for 8,400 boxes: 155 MB for every frame. This is where its time goes, measured on one 640 x 640 image with yoloe-26s-seg-pf on the CPU (OpenVINO 2024.6, the version of BrainFrame's core container):

Stage Time per frame
Letterbox and input 2 ms
The model about 340 to 500 ms
Decode, a few classes selected 0.2 ms
Decode, every class (empty classes option) about 100 ms with the faster decode (about 320 ms with the standard one)
Non-max suppression 1 ms
Masks (12 objects) 8 to 13 ms

The model is the cost. Its class head, a matrix product and a bias for each of three scales, followed by transposes, a join and a sigmoid, makes about half of it. The capsule uses two techniques to be faster, with the same detections:

  • A faster decode for the empty option: the best score of each box is a maximum along the class rows of the output, without transposing it, and the class is looked up only for the boxes above the threshold. This took the decode from about 320 ms to about 100 ms.
  • Cutting the class head. When the classes option selects 1 to 1,000 classes, the capsule builds a copy of the model whose class head is cut to those classes, compiles it in the background (0.7 to 1.3 s) and runs it when it is ready. Until then the frames run on the model that is active, so the results never change, only the processing time. An empty option, a name that is not in the vocabulary or more than 1,000 classes use the full model. Up to three cut models are kept, and the same classes in another order share one. If a cut fails, the capsule logs a warning and goes on with the full model.

On the model alone, one inference took 504 ms with the full head, 260 ms with the head cut to 6 classes and 268 ms cut to 64 classes (leaving out the final sigmoid made no real difference).

The capsule with and without the two techniques, on the real model, alternating the two capsules over three rounds (24 steady frames per run). The machine was busy running BrainFrame (a load average of about 15 on 20 cores), so only the comparison between the columns counts:

classes option Without the techniques, median (minimum) With the techniques, median (minimum) Gain
person about 550 ms (470 to 505) about 350 ms (250 to 270) 1.6 times, 1.9 times at the minimum
person, bus, stop sign, glasses about 560 ms (460 to 500) about 375 ms (255 to 305) 1.5 times, 1.8 times at the minimum
empty (all 4,585 classes) about 765 ms (670 to 705) about 675 ms (540 to 625) 1.15 times (the faster decode only)

A run on a less busy machine gave 160 to 315 ms per frame with the cut against 350 to 480 ms without it.

The detections do not change. A sequence of seven changes of the option in one running capsule (person; bus, stop sign; person; empty; glasses,person; a name that does not exist; person) gave exactly the same detections as the capsule without the cut at every step. The first two or three frames after a change run on the model that was active, while the new one is built.

Limits:

  • The capsule keeps the full model and up to three cut models compiled in memory (the memory was not measured).
  • The cut was tried on the CPU only.
  • After a change of the option, the capsule needs a few seconds before it is faster.

Models That Are Not Packaged Yet

These weights appear in the Ultralytics Public list of the AI Marketplace, greyed out, with the reason in the description. They are not packageable yet. This lists what has not been built, not what cannot be done. Some of them (SAM, SAM 2, MobileSAM and YOLO-NAS) are not Ultralytics models: Ultralytics hosts the weights and supports them in its library. What each one needs:

Models What is missing
yolo11n-grayscale A template for a one-channel input. The converter reads the model, but the capsule fails to run it because the templates feed three color channels. The template would convert the frame to grayscale first.
YOLOv8 -human models, YOLO26 -reid and -s3d A converter that can read them. The Ultralytics version that the converter installs (8.4.214 when this was tested) does not have the model classes these files need (HumanModel, ReidModel, the s3d module). With a version that does, they may fit an existing template (not checked).
YOLO26 -depth and -sem (semantic segmentation) A decision on how a capsule reports them. The converter refuses these tasks today. A depth map or a class per pixel has no BrainFrame output type of its own; semantic segmentation could be reported as polygons per class.
YOLO26 -stereo3ddet A way to give the capsule two images, since it detects from a stereo pair. Not evaluated yet.
SAM, SAM 2 and MobileSAM A capsule that takes prompts, for example the boxes of another capsule, and returns masks. Not evaluated yet.
YOLO-NAS A converter for the library it comes from, and a template for its output. Not evaluated yet.

Models Without Weights in the Releases

The list in the AI Marketplace is built from the weight files (.pt) that Ultralytics publishes in the releases of its assets repository. The marketplace reads the releases from GitHub and keeps the result for six hours, so new weights, and new releases, appear without a software update. If GitHub cannot be reached, it shows the last copy it has, or the list that shipped with the software.

The Ultralytics models page also describes models for which Ultralytics publishes no weights in these releases: YOLOv4, YOLOv6, YOLOv7, SAM 3, and YOLO27 (marked "Coming Soon"). They are not in the list. To use your own copy of such a model, export it to OpenVINO or ONNX and add it with Package Model. That has not been tried for these models.