Creating Ultralytics YOLO Models¶
Introduction¶
The VisionCapsules® AI Marketplace can create a capsule from an
Ultralytics YOLO model without any capsule code. It downloads the weights from
Ultralytics, converts them to OpenVINO on your own server, fills in one of the capsule templates published in the
ultralytics directory of the Capsule Zoo, and packages the result into the open vision capsule format
(a .cap file) in My VisionCapsules.
This page covers:
- how to download and package a model in the AI Marketplace;
- what each kind of model reports, and which template it uses;
- how to use the templates directly, to build a capsule yourself;
- how the vocabulary, the default class names and the
classesoption of a capsule work; - what the number of baked-in classes costs for open-vocabulary models (YOLO-World and YOLOE);
- how a prompt-free YOLOE capsule gets faster;
- which Ultralytics models are not packaged yet.
Note
AOTU does not distribute these capsules. They are built on your machine from the Ultralytics weights, which are licensed under the Ultralytics AGPL-3.0 License. The AI Marketplace asks you to accept the license before it packages a model. The server needs internet access to download the weights, and the packaging service must be running next to the marketplace (the dialog shows a warning if it is not).
Download and Package a Model¶
- Open the AI Marketplace.
- In Explore VisionCapsules, choose Ultralytics Public in the source menu. The list shows the weights that Ultralytics publishes, grouped by family (YOLO26, YOLO11, YOLOv8, ...). Use the model version menu to show one family.
- Click the download icon at the end of a row, or tick several rows and click Download & Package Selected.
-
The dialog Package Model into Open Vision Capsules Format opens for the first model, filled in from the model you chose. The source, model version, task, format, weights file and input size are fixed; you can change the capsule name and version. When you chose several models, each one gets its own step, one after another. OK goes to the next model; it is enabled once you have accepted the license and the fields are valid. Cancel drops that model only. Depending on the model, the step shows more fields after the common ones:
Model What the step shows Detection and segmentation The model's own class names, read from the weights and shown read-only, and how many there are (Maximum Vocabulary classes). The default class names are optional and editable: the capsule's classesoption starts with them (empty: every class), and can be changed at run time.Oriented boxes ( -obb)The same class names. The default class names are greyed out: the classesoption is not available for these models.Pose ( -pose)The same, and the keypoint names, which start with the 17 COCO person keypoints ( nose,left_eye, ...right_ankle) and can be edited. Empty means the same names for a model with 17 keypoints, and numbered names (keypoint_1, ...) for another number.Classification ( -cls)The model's own 1,000 ImageNet class names, read-only, and the default class names (optional, editable: the classes the attribute can take; empty: all 1,000). Then the classes of the detections to classify (for example person) and the name of the attribute to set.YOLO-World and YOLOE The vocabulary (it starts with the 80 COCO classes; up to the maximum number of classes, 100 by default) and the default class names. See Vocabulary and Default Class Names. YOLOE prompt-free ( -pf)The same two boxes. The vocabulary is fixed (4,585 classes) and read-only; you choose the default class names. The class names are read when the step opens. The weights are downloaded for that, and the first time the converter is installed, which takes minutes. You can press OK without waiting; the weights are downloaded again when packaging.
-
OK on the last model starts the download and the packaging of all the models. The dialog then shows the progress of each model (downloading, converting, testing, packaging). Each model appears under the name it will have in My VisionCapsules, for example
Detector Yolo26s Seg. - When a model is done, it is in My VisionCapsules, named
detector_<model>(classifier_<model>for classification), for exampledetector_yolo26n. Load it with the right arrow or Run Selected, and enable it on a stream like any other capsule.
The first time, the converter (PyTorch and Ultralytics, about 1.5 GB) is installed. YOLO-World and YOLOE also download a text encoder (350 to 600 MB). This can take several minutes; later models are much faster.
To package a model that is not in the list, for example one you trained yourself, click Package Model in
My VisionCapsules and provide the .pt file, the OpenVINO files (.xml and .bin) or an ONNX file. The form has
the same fields as the dialog above, and you fill them in yourself.
Capsule Name and Version¶
The capsule name is generated from the model (for example detector_yolo26s_seg); a number is added if the name is
already taken. The Version is a whole number from 1: it is the capsule's own version, written in its capsule.py
(version = 1) and reported by BrainFrame. The capsule format requires an integer there. In My VisionCapsules, the
version column shows the model family and the version, for example YOLOE | v1, like the v0.32.0 | v1.0 of the
capsules in the AOTU list, which show the release and the release version.
What the Capsule Reports¶
| Task | What each detection contains |
|---|---|
| Object detection | A class name, a bounding box and a confidence. |
| Instance segmentation | A polygon that outlines the object, derived from its mask. The mask_threshold option tunes the outline. |
| Pose estimation | A bounding box, and the keypoints in the extra data keypoints, as [x, y, confidence] in frame pixels. |
| Oriented boxes | A rotated rectangle (its four corners), and its angle in the extra data. |
| Image classification | An attribute set on the detections of another capsule. A classifier capsule follows a detector. |
The Capsule Templates¶
The packager chooses the template from the output of the exported model, so one template serves every model that exports in the same way.
| Models | Task | Template |
|---|---|---|
| YOLOv3u, YOLOv5u, YOLOv8, YOLOv9, YOLO11, YOLO12, YOLO-World, YOLOE (all) | Detect | yolov5_anchor_free |
| YOLOv5 (original; ONNX or OpenVINO files only) | Detect | yolov5_anchor |
| YOLO26, YOLOv10 | Detect | yolo_end2end |
| RT-DETR | Detect | rtdetr |
| YOLOv8, YOLOv9, YOLO11, YOLOE (all), FastSAM | Segment | segment |
| YOLO26 | Segment | segment_end2end |
| YOLOv8, YOLO11 | Pose | pose |
| YOLO26 | Pose | pose_end2end |
| YOLOv8, YOLO11 | Oriented boxes | obb |
| YOLO26 | Oriented boxes | obb_end2end |
| YOLOv8, YOLO11, YOLO26 | Classify | classify |
The *_end2end templates are for models that already remove duplicate boxes, so their capsules have no iou_threshold
option. The other templates run non-maximum suppression (NMS) in the capsule.
These variants use the same templates as their family:
- the Open Images V7 (
-oiv7) and 1280 px (x6,-pose-p6) YOLOv8 models; - the Objects365 (
-objv1-150,-objv1-seg), distilled and text (-text) YOLO26 models; - the demo models of Ultralytics Solutions, which are listed under their YOLO version:
anpr-demo-model(a YOLO11 nano license plate detector) andsolutions_ci_parking_model(a YOLOv8 nano model trained on VisDrone).
FastSAM is built on YOLOv8-seg and works in its "segment everything" mode: every object gets the class object.
Using the Templates Yourself¶
The templates are in the ultralytics/templates directory of the Capsule Zoo:
templates/
common/ capsule.py, backend.py, preprocess.py, nms.py and meta.conf: every capsule gets these
<template>/ the files for one model output (decode.py, ...): they replace the ones in common/
segment_end2end, pose_end2end and obb_end2end build on segment, pose and obb. To make a capsule by hand:
- Copy
common/and one template into a new capsule directory. - Fill in the placeholders (
__CAPSULE_NAME__, the class names, and so on). - Add the OpenVINO
model.xmlandmodel.bin. - Package the capsule as described in Creating an OpenVINO Object Detector Capsule.
The README.md in that directory lists the output of each template and explains how to add the classes option to a
template of your own.
Choosing the Classes of a Capsule¶
A capsule made from a detection, segmentation or classification model has a classes option, which selects the classes
the capsule reports. The marketplace sets its starting value when it packages the model, and you can change it in
BrainFrame at any time. This section explains the terms and what the option does.
Vocabulary and Default Class Names¶
Two terms are used for the class names of such a capsule:
- Vocabulary: all the classes the capsule can report. Nothing outside it can be reported.
- Default class names: the starting text of the capsule's
classesoption. They decide what the capsule reports until someone changes the option; they are not a limit.
What they are depends on the model:
- YOLO-World and text-prompt YOLOE. You write the vocabulary, one class per line. It starts with the 80 COCO
classes, and can hold up to the maximum number of classes: 100 by default, and up to 1,000 (the dialog warns that more
classes make the capsule slower). The default class names start with
person,car,truck,bus,bicycleandmotorcycle. Every default class name must be in the vocabulary, and neither box can be empty. - Prompt-free YOLOE. The vocabulary is the model's own list of 4,585 class names, which cannot be changed. You choose the default class names, which must be in that list.
- The other models (for example YOLOv8, YOLO11, YOLO26 and the classifiers). The vocabulary is the model's own list
of class names (80 for COCO, 1,000 for ImageNet), read from the weights. The default class names are optional; empty
means every class. When you package your own file in Package Model, the list is the one you type in the form, or the
names stored in the
.ptfile.
When you choose a .pt file in Package Model, the dialog reads its class names in the background (the first time, it
installs the converter). A prompt-free YOLOE file shows its 4,585 names, read-only; for any other file you edit the
vocabulary, which starts with the 80 COCO classes.
The capsule gets the vocabulary as its classes, and its classes option starts with the default class names. In
BrainFrame the option can be changed to any names of the vocabulary; empty means every class of the vocabulary.
The option replaces the default class names; it does not narrow them. For a capsule packaged with the default class names
person, bench, backpack, setting the option to dog reports dogs, as long as dog is in the vocabulary (one of the 80
COCO classes, or typed into the vocabulary box; every name of the vocabulary of a prompt-free model). A name that is not
in the vocabulary reports nothing.
The Classes Option¶
The classes option holds the class names to report, separated by commas (for example person, car). Empty means every
class of the capsule. Capsules of pose and oriented box models and of RT-DETR do not have the option. You change it in the
capsule options in BrainFrame, without packaging again, and the detections follow at once.
- What it can choose: the vocabulary the capsule was packaged with. A name that is not in the vocabulary is skipped,
and the capsule's log says so. Names are matched ignoring upper and lower case, and a space and an underscore are the
same (the ImageNet names have underscores, for example
great_white_shark). - The starting value is the default class names typed in the dialog.
- How it is applied: in the decode,
selected_classes()inselection.pyturns the text into class numbers, anddecode()takes the best class among those only (the "filter before" of the experiment below). - End-to-end models. YOLO26 and YOLOv10 name one class per box inside the model, so the option only drops the boxes of
the other classes: a box the model names
personis never reported asbicycle. For this reason YOLOE-26 is packaged with its raw head and not the end-to-end one, so the option can move a box frompersontobicycle. - Classifiers. The option chooses which classes the attribute can take, not which detections to report. The best of
the selected classes is set, with its own probability (the model's probability of that class, not shared out again
among the selected ones), so the confidence can be small when the picture does not look like any of them. The
min_confidenceoption then decides whether the attribute is set. With no name known to the model, no attribute is set. - Processing time. For the models with their own class names the option is a filter, not a speed-up, because their
class head is small. On one image, a YOLO11 nano detector took about 85 ms per frame with every class and about 90 ms
with
persononly, and a YOLO26 nano segmentation capsule about 180 ms either way (the same machine, busy, so the difference is within the noise of the measurement). Only a prompt-free YOLOE capsule gets faster, because its class head is cut down to the selected classes (see Prompt-Free Models: Processing Time).
Open-Vocabulary Models: How Many Classes to Bake In¶
YOLO-World and the text-prompt YOLOE models are open-vocabulary: you give them class names, and they detect those classes. The converter turns every name into a text embedding and bakes the embeddings into the model, so the capsule needs no text encoder at run time. The classes you type when packaging are the classes the model knows.
This section measures what the number of baked classes costs. Use it to decide how many classes to bake in, and how many
of them to expose in the capsule's classes option.
Note
The classes of the vocabulary are baked into the model, and the capsule's classes option selects among them at
run time (see The Classes Option). The measurements below show what the size of the
vocabulary costs.
Test Setup¶
| Item | Value |
|---|---|
| Models | YOLO-World v2 small (yolov8s-worldv2) and YOLOE v8 small (yoloe-v8s-seg, boxes only) |
| Baked classes | 3 (person, car, chair), 80 (the COCO classes), 300, 500, 1,000, 2,000 and 4,000 (the 80 COCO classes plus names from the YOLOE prompt-free vocabulary) |
| Template | yolov5_anchor_free: its decode() takes the best class of every candidate box |
| Data | The 128 images of COCO128, scored on person, car and chair |
| Accuracy method | Letterboxed 640 x 640 input, confidence threshold 0.001, class-aware NMS at IoU 0.7, at most 300 detections per image, AP averaged over IoU 0.5 to 0.95 |
| Processing time method | One image at a time, the median of 125 frames, OpenVINO 2024.6 on a 20-core CPU with default threading |
| Conversion | Ultralytics 8.4.214, the version the converter installed at the time of the test |
This is a small sample, one size of each model and one machine. Treat the numbers as a guide: differences of about 0.02 AP are within what a sample of this size can show.
Processing Time¶
Model time is one OpenVINO inference, up to the point where the raw output has been copied out. Decode time is the
template's decode(): the best class of each of the 8,400 candidate boxes, the confidence threshold and the box
conversion. Pre-processing is identical in every case, and NMS runs only on the few boxes above the threshold. Neither
is included.
| Model | Baked classes | Model time | Decode time | Total |
|---|---|---|---|---|
| YOLO-World v2 s | 3 | 72 ms | 0.2 ms | 72 ms |
| 80 | 84 ms | 1 ms | 85 ms | |
| 300 | 174 ms | 7 ms | 181 ms | |
| 500 | 166 ms | 11 ms | 177 ms | |
| 1,000 | 354 ms | 39 ms | 394 ms | |
| 2,000 | 466 ms | 126 ms | 592 ms | |
| 4,000 | 367 ms | 65 ms | 432 ms | |
| YOLOE v8 s | 3 | 126 ms | 0.4 ms | 126 ms |
| 80 | 126 ms | 1.8 ms | 128 ms | |
| 300 | 155 ms | 5 ms | 160 ms | |
| 500 | 154 ms | 10 ms | 164 ms | |
| 1,000 | 231 ms | 38 ms | 269 ms | |
| 2,000 | 330 ms | 112 ms | 443 ms | |
| 4,000 | 254 ms | 76 ms | 330 ms |
The rows for 3, 80 and 4,000 classes were measured first, on an idle machine. The rows for 300 to 2,000 classes were measured later, one at a time, on the same machine while BrainFrame was running, so they are not comparable with the first rows: this is why the 4,000-class rows are faster than the 2,000-class rows. Compare the 300 to 2,000 rows with each other, where the time grows with the number of classes.
With 4,000 classes the model output is about 135 MB per frame, and copying it out is part of the model time.
Accuracy¶
The table gives mAP50 over the three test classes, with mAP50-95 in brackets.
- Filter after takes the best of all the baked classes for each box, then drops the boxes whose best class is not one you want.
- Filter before looks only at the scores of the classes you want.
| Model | Baked classes | Filter after | Filter before |
|---|---|---|---|
| YOLO-World v2 s | 3 | 0.45 (0.31) | 0.45 (0.31) |
| 80 | 0.48 (0.33) | 0.49 (0.33) | |
| 300 | 0.42 (0.27) | 0.46 (0.31) | |
| 500 | 0.40 (0.26) | 0.45 (0.30) | |
| 1,000 | 0.34 (0.21) | 0.45 (0.30) | |
| 2,000 | 0.28 (0.17) | 0.45 (0.30) | |
| 4,000 | 0.22 (0.13) | 0.44 (0.29) | |
| YOLOE v8 s | 3 | 0.47 (0.29) | 0.47 (0.29) |
| 80 | 0.47 (0.29) | 0.47 (0.29) | |
| 300 | 0.47 (0.29) | 0.47 (0.29) | |
| 500 | 0.47 (0.29) | 0.47 (0.29) | |
| 1,000 | 0.47 (0.29) | 0.47 (0.29) | |
| 2,000 | 0.47 (0.29) | 0.47 (0.29) | |
| 4,000 | 0.47 (0.29) | 0.47 (0.29) |
Packaging Time and Size¶
| Model | 3 classes | 80 classes | 4,000 classes |
|---|---|---|---|
| YOLO-World v2 s | 8 s, 25.5 MB | 15 s, 25.6 MB | 389 s, 29.6 MB |
| YOLOE v8 s | 8 s, 23 MB | 12 s, 23 MB | 406 s, 26 MB |
The capsule size for 300, 500, 1,000 and 2,000 classes is 25.9, 26.1, 26.6 and 27.6 MB for YOLO-World, and 23.1, 23.3, 23.7 and 24.4 MB for YOLOE.
The packaging time is dominated by the text encoder, at about 0.1 s per name on one CPU core. The capsule size hardly changes.
What the Results Show¶
- The cost follows the number of baked classes, not the number you report. The model scores every baked class on
every frame, and the decode takes the best of all of them. Selecting fewer classes in the
classesoption does not make a text-prompt model faster. (A prompt-free model is different: see Prompt-Free Models: Processing Time.) - The cost grows with the list. From 3 to 80 classes it added 12 ms per frame on YOLO-World and nothing measurable on YOLOE. From 500 to 1,000 classes the frame time rose by 1.6 times (YOLOE) to 2.2 times (YOLO-World), and at 2,000 it was about 3.3 times (YOLO-World) and 2.7 times (YOLOE) the 500-class time. At 4,000 classes the frame time was about 6 times (YOLO-World) and 2.6 times (YOLOE) that of 3 classes, and packaging took minutes. Whether that matters depends on your frame rate, your hardware and the number of streams.
- The list can change the accuracy. For YOLO-World, 80 classes scored slightly higher than 3. A likely reason is that other everyday classes in the list absorb objects that are not yours, which would otherwise be taken for one of yours. A larger list also lowered YOLO-World a little even with the filter before (0.44 against 0.45), probably because its neck uses all the class embeddings; this cause was not tested. YOLOE was not affected in this test.
- Do not take the best class over a larger list than the one you report. With 4,000 baked classes and the filter
after, YOLO-World lost most of its
persondetections (AP50 0.73 to 0.10), because a similar name such asmanorpedestrianwon those boxes and they were then dropped. Choosing the best class among only the classes you want (filter before) restores it to 0.68. The loss starts early: YOLO-World'spersonAP50 was 0.73 with 3 classes, 0.66 with 300 and 0.41 with 1,000 under the filter after, and 0.70 to 0.71 with the filter before. The capsule'sclassesoption uses the filter before, so a large vocabulary only hurts the accuracy when the option is left empty (every class).
Recommendations¶
- Bake in the minimum. Bake in only the classes your use case needs. Every baked class is scored on every frame,
whether or not the
classesoption has switched it off. The dialog starts with the 80 COCO classes, which cost about 12 ms per frame on YOLO-World and nothing measurable on YOLOE; the maximum is 1,000 classes, where the frame time was about 390 ms (YOLO-World) and 270 ms (YOLOE) on the test machine. - Set the
classesoption instead of leaving it empty. Empty reports every class of the vocabulary, and with a large vocabulary YOLO-World's accuracy drops (see above). Naming the classes you want avoids it. - In a capsule of your own, offer only the classes that users need to switch, because each one is baked in and carries
the cost above, and use only the columns of the selected classes in
decode()to take the best class. - Decide for your own use case. The tables are a starting point. Measure your model on your hardware with the classes you intend to use.
- For a very large vocabulary, use a prompt-free YOLOE model, whose vocabulary is 4,585 classes. Its model is bigger
than a text-prompt one, but its capsule cuts the class head down to the classes the
classesoption selects, so a capsule that reports a few classes is not much slower than a text-prompt one (see Prompt-Free Models: Processing Time). - To add a class that is not baked in, package the model again. The text encoder is not part of the capsule.
Prompt-Free Models: Processing Time¶
A prompt-free YOLOE model scores every box against all of its 4,585 classes. Its output is 4,621 values per box (4 for the
box, 4,585 class scores and 32 for the mask) for 8,400 boxes: 155 MB for every frame. This is where its time goes,
measured on one 640 x 640 image with yoloe-26s-seg-pf on the CPU (OpenVINO 2024.6, the version of BrainFrame's core
container):
| Stage | Time per frame |
|---|---|
| Letterbox and input | 2 ms |
| The model | about 340 to 500 ms |
| Decode, a few classes selected | 0.2 ms |
Decode, every class (empty classes option) |
about 100 ms with the faster decode (about 320 ms with the standard one) |
| Non-max suppression | 1 ms |
| Masks (12 objects) | 8 to 13 ms |
The model is the cost. Its class head, a matrix product and a bias for each of three scales, followed by transposes, a join and a sigmoid, makes about half of it. The capsule uses two techniques to be faster, with the same detections:
- A faster decode for the empty option: the best score of each box is a maximum along the class rows of the output, without transposing it, and the class is looked up only for the boxes above the threshold. This took the decode from about 320 ms to about 100 ms.
- Cutting the class head. When the
classesoption selects 1 to 1,000 classes, the capsule builds a copy of the model whose class head is cut to those classes, compiles it in the background (0.7 to 1.3 s) and runs it when it is ready. Until then the frames run on the model that is active, so the results never change, only the processing time. An empty option, a name that is not in the vocabulary or more than 1,000 classes use the full model. Up to three cut models are kept, and the same classes in another order share one. If a cut fails, the capsule logs a warning and goes on with the full model.
On the model alone, one inference took 504 ms with the full head, 260 ms with the head cut to 6 classes and 268 ms cut to 64 classes (leaving out the final sigmoid made no real difference).
The capsule with and without the two techniques, on the real model, alternating the two capsules over three rounds (24 steady frames per run). The machine was busy running BrainFrame (a load average of about 15 on 20 cores), so only the comparison between the columns counts:
classes option |
Without the techniques, median (minimum) | With the techniques, median (minimum) | Gain |
|---|---|---|---|
person |
about 550 ms (470 to 505) | about 350 ms (250 to 270) | 1.6 times, 1.9 times at the minimum |
person, bus, stop sign, glasses |
about 560 ms (460 to 500) | about 375 ms (255 to 305) | 1.5 times, 1.8 times at the minimum |
| empty (all 4,585 classes) | about 765 ms (670 to 705) | about 675 ms (540 to 625) | 1.15 times (the faster decode only) |
A run on a less busy machine gave 160 to 315 ms per frame with the cut against 350 to 480 ms without it.
The detections do not change. A sequence of seven changes of the option in one running capsule (person; bus, stop sign;
person; empty; glasses,person; a name that does not exist; person) gave exactly the same detections as the capsule
without the cut at every step. The first two or three frames after a change run on the model that was active, while the
new one is built.
Limits:
- The capsule keeps the full model and up to three cut models compiled in memory (the memory was not measured).
- The cut was tried on the CPU only.
- After a change of the option, the capsule needs a few seconds before it is faster.
Models That Are Not Packaged Yet¶
These weights appear in the Ultralytics Public list of the AI Marketplace, greyed out, with the reason in the description. They are not packageable yet. This lists what has not been built, not what cannot be done. Some of them (SAM, SAM 2, MobileSAM and YOLO-NAS) are not Ultralytics models: Ultralytics hosts the weights and supports them in its library. What each one needs:
| Models | What is missing |
|---|---|
yolo11n-grayscale |
A template for a one-channel input. The converter reads the model, but the capsule fails to run it because the templates feed three color channels. The template would convert the frame to grayscale first. |
YOLOv8 -human models, YOLO26 -reid and -s3d |
A converter that can read them. The Ultralytics version that the converter installs (8.4.214 when this was tested) does not have the model classes these files need (HumanModel, ReidModel, the s3d module). With a version that does, they may fit an existing template (not checked). |
YOLO26 -depth and -sem (semantic segmentation) |
A decision on how a capsule reports them. The converter refuses these tasks today. A depth map or a class per pixel has no BrainFrame output type of its own; semantic segmentation could be reported as polygons per class. |
YOLO26 -stereo3ddet |
A way to give the capsule two images, since it detects from a stereo pair. Not evaluated yet. |
| SAM, SAM 2 and MobileSAM | A capsule that takes prompts, for example the boxes of another capsule, and returns masks. Not evaluated yet. |
| YOLO-NAS | A converter for the library it comes from, and a template for its output. Not evaluated yet. |
Models Without Weights in the Releases¶
The list in the AI Marketplace is built from the weight files (.pt) that Ultralytics publishes in the releases of its
assets repository. The marketplace reads the releases from GitHub and
keeps the result for six hours, so new weights, and new releases, appear without a software update. If GitHub cannot be
reached, it shows the last copy it has, or the list that shipped with the software.
The Ultralytics models page also describes models for which Ultralytics publishes no weights in these releases: YOLOv4, YOLOv6, YOLOv7, SAM 3, and YOLO27 (marked "Coming Soon"). They are not in the list. To use your own copy of such a model, export it to OpenVINO or ONNX and add it with Package Model. That has not been tried for these models.