DETR: End-to-End Object Detection With Transformers
- Nicolas Carion*
- Francisco Massa*
- Gabriel Synnaeve
- Nicolas Usunier
- Alexander Kirillov
- Sergey Zagoruyko
*denotes equal contribution
Abstract
We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at this URL.
ECCV Talk
Visualizing attention
Since DETR is an attention-based model, we can visualize what the network is looking at to make predictions. In the image above, we overlay the attention heat-map in the same color as the box it corresponds to. We notice that DETR focuses on the extremities of the objects, to be able to accurately predict the extent of the bounding boxes.
Main results
| Model | Backbone | AP | AP small | AP medium | AP large | FPS |
|---|---|---|---|---|---|---|
| Detr | Resnet-50 | 42.0 | 20.5 | 45.8 | 61.1 | 28 |
| Detr-DC5 | Resnet-50 | 43.3 | 22.5 | 47.3 | 61.1 | 12 |
| Detr | Resnet-101 | 43.5 | 21.9 | 48.0 | 61.8 | 20 |
| Detr-DC5 | Resnet-101 | 44.9 | 23.7 | 49.5 | 62.3 | 10 |
We report the Average Precision (AP) on the coco 2017 validation set. All models are trained for 500 epochs, and timings are obtained on a V100 with torchscript and a batch size of 1. Refer to the paper for additional details.
Extending to panoptic segmentation
DETR can be naturally extended to panoptic segmentation, proving its generality. We obtain a performance of 45.6 PQ on the coco validation set.
Demo
Plotly is hosting a live demo of DETR. Go check it out, or play with it below.