COCO is a large-scale dataset for object detection, segmentation, and image captioning. It contains bounding boxes, segmentations, keypoints, and captions for images. The COCO API can be used to access and visualize different types of annotations like bounding boxes, segmentations, keypoints, and captions for images in the COCO dataset.

1. COCO

2. COCO
segmentation
Bounding box
Keypoits
Captions
coco
COCO is a large-scale object detection, segmentation, and captioning dataset. COCO has
several features:
So powerful!!

3. $ sudo apt-get install aria2
$ aria2c -c http://msvocds.blob.core.windows.net/annotations-1-0-3/instances_train-
val2014.zip
$ aria2c -c http://msvocds.blob.core.windows.net/coco2014/train2014.zip
$ aria2c -c http://msvocds.blob.core.windows.net/coco2014/val2014.zip
$ aria2c -c http://msvocds.blob.core.windows.net/coco2014/test2014.zip
Download COCO dataset (2014)

4. Download COCO dataset (2017)
$ sudo apt-get install aria2
$ aria2c -c http://msvocds.blob.core.windows.net/annotations/instances_train-
val2017.zip
$ aria2c -c http://images.cocodataset.org/zips/train2017.zip
$ aria2c -c http://images.cocodataset.org/zips/val2017.zip
$ aria2c -c http://images.cocodataset.org/zips/test2017.zip

5. COCO annotations

6. Install cocoapi
https://github.com/cocodataset/cocoapi
Install cocoapi using command-line commands:
$ sudo pip3 install pycocotools
Test:
$ python3
>> import pycocotools
Install COCO API before using COCO dataset.

7. Use cocoapi
(instances)
"categories": [
{
"supercategory": "person",
"id": 1,
"name": "person"
},…]
instances_val2017.json
['licenses', 'info', 'images', 'categories', 'annotations']
"images": [
{
"license": 4,
"file_name": "000000397133.jpg",
"coco_url":
"http://images.cocodataset.org/val2017/000000397133.jpg",
"height": 427,
"width": 640,
"date_captured": "2013-11-14 17:02:52",
"flickr_url":
"http://farm7.staticflickr.com/6116/6255196340_da26cf2c9e_z.jpg",
"id": 397133
},…]
"annotations": [
{
"segmentation": [
[
510.03,
423.01,
510.45,
423.01...]
],
"area": 702.10574,
"iscrowd": 0,
"image_id": 289343,
"bbox": [
473.07,
395.93,
38.65,
28.67
],
"category_id": 18,
"id": 1768
},…]
(2)
(1)
(3)
https://zhuanlan.zhihu.com/p/29393415

8. Use cocoapi (instances)
from matplotlib import pyplot as plt
from matplotlib.patches import Polygon
from skimage import io
from pycocotools.coco import COCO
import numpy as np
import os
annFile = "data/coco2017/annotations/instances_val2017.json"
root = "data/coco2017/val2017/"
coco = COCO(annFile)
catIds = coco.getCatIds(catNms=['person', 'dog', 'skateboard'])
imgIds = coco.getImgIds(catIds=catIds)
images = coco.loadImgs(imgIds[0:3])

9. Use cocoapi (instances)
for idx, image in enumerate(images, 1):
plt.subplot(1, 3, idx)
I = io.imread(image['coco_url'])
plt.axis('off')
plt.imshow(I)
plt.show()

10. Use cocoapi (instances)
for idx, image in enumerate(images, 1):
plt.subplot(1, 3, idx)
I = io.imread(image['coco_url'])
plt.axis('off')
plt.imshow(I)
annIds = coco.getAnnIds(imgIds=image['id'], catIds=catIds, iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
plt.show()

11. Use cocoapi (instances)
for idx, image in enumerate(images, 1):
# image
ax = plt.subplot(1, 3, idx)
I = io.imread(image['coco_url'])
plt.imshow(I)
plt.axis('off')
# segment
annIds = coco.getAnnIds(imgIds=image['id'], catIds=catIds, iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
# bbox
for ann in anns:
bbox_x, bbox_y, bbox_w, bbox_h = ann['bbox']
poly = [[bbox_x, bbox_y], [bbox_x, bbox_y+bbox_h],
[bbox_x+bbox_w, bbox_y+bbox_h], [bbox_x+bbox_w, bbox_y]]
np_poly = np.array(poly).reshape((4,2))
ax.add_patch(Polygon(np_poly, linestyle='--', facecolor='none', edgecolor="red", linewidth=2))
plt.show()

12. Use cocoapi (instances)

13. Use cocoapi
(keypoints)
person_keypoints_val2017.json
['licenses', 'info', 'images', 'categories', 'annotations']
"images": [
{
"license": 4,
"file_name": "000000397133.jpg",
"coco_url":
"http://images.cocodataset.org/val2017/000000397133.jpg",
"height": 427,
"width": 640,
"date_captured": "2013-11-14 17:02:52",
"flickr_url":
"http://farm7.staticflickr.com/6116/6255196340_da26cf2c9e_z.jpg",
"id": 397133
},…]
"categories": [
{
"supercategory": "person",
"id": 1,
"name": "person"
},…]
"annotations": [
{
"segmentation": [
[
125.12,
539.69,...]
],
"num_keypoints": 10,
"area": 47803.27955,
"iscrowd": 0,
"keypoints": [
162,
551,
2,...],
"image_id": 425226,
"bbox": [
73.35,206.02,
300.58,372.5
],
"category_id": 1,
"id": 183126
},

14. Use cocoapi (keypoints)
• visibility == 0 that keypoint not in the image.
• visibility == 1 that keypoint is in the image BUT
not visible namely maybe behind of an object.
• visibility == 2 that keypoint looks clearly. not
hidden.
"annotations": [
{
"segmentation": [
[
125.12,
539.69,...]
],
"num_keypoints": 10,
"area": 47803.27955,
"iscrowd": 0,
"keypoints": [
162,
551,
2,...],
"image_id": 425226,
"bbox": [
73.35,206.02,
300.58,372.5
],
"category_id": 1,
"id": 183126
},

15. Use cocoapi (keypoints)
annFile = "data/coco2017/annotations/person_keypoints_val2017.json"
coco = COCO(annFile)
catIds = coco.getCatIds(catNms=['person']) # the images icludes all categories
imgIds = coco.getImgIds(catIds=catIds)
images = coco.loadImgs(imgIds[10])
image = images[0]

16. Use cocoapi (keypoints)
# image
ax = plt.gca()
I = io.imread(image['coco_url'])
plt.imshow(I)
plt.axis('off')
# keypoints/segmentation
annIds = coco.getAnnIds(imgIds=image['id'], catIds=catIds, iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
# bbox
for ann in anns:
bbox_x, bbox_y, bbox_w, bbox_h = ann['bbox']
poly = [[bbox_x, bbox_y], [bbox_x, bbox_y+bbox_h],
[bbox_x+bbox_w, bbox_y+bbox_h], [bbox_x+bbox_w, bbox_y]]
np_poly = np.array(poly).reshape((4,2))
ax.add_patch(Polygon(np_poly, linestyle='--', facecolor='none', edgecolor="red", linewidth=2))
plt.show()

17. Use cocoapi (keypoints)

18. Use cocoapi (caption)
"annotations": [
{
"image_id": 179765,
"id": 38,
"caption": "A black Honda motorcycle parked in front of a garage."
},…]
captions_val2017.json
['licenses', 'info', 'images', 'annotations']

19. Use cocoapi (caption)
annFile = "data/coco2017/annotations/captions_val2017.json"
coco = COCO(annFile)
imgIds = coco.getImgIds()
images = coco.loadImgs(imgIds[0])
image = images[0]
# image
ax = plt.gca()
I = io.imread(image['coco_url'])
plt.imshow(I)
plt.axis('off')
# keypoints/annotations
annIds = coco.getAnnIds(imgIds=image['id'], iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
plt.show()

20. Use cocoapi (caption)
A person kitesurfing over the waves of the ocean's shore.
a kite surfer is doing a flying trick over some water
A man is flying up in the air and having fun.
A guy is waterboarding in the ocean on a windy day.
A person kite boarding in rough seas near the shoreline.