Deep learning dataset: COCO

1. COCO

2. COCO
segmentation
Bounding box
Keypoits
Captions
coco
COCO is a large-scale object detection, segmentation, and captioning dataset. COCO has
several features:
So powerful!!

3. $ sudo apt-get install aria2
$ aria2c -c http://msvocds.blob.core.windows.net/annotations-1-0-3/instances_train-
val2014.zip
$ aria2c -c http://msvocds.blob.core.windows.net/coco2014/train2014.zip
$ aria2c -c http://msvocds.blob.core.windows.net/coco2014/val2014.zip
$ aria2c -c http://msvocds.blob.core.windows.net/coco2014/test2014.zip
Download COCO dataset (2014)

4. Download COCO dataset (2017)
$ sudo apt-get install aria2
$ aria2c -c http://msvocds.blob.core.windows.net/annotations/instances_train-
val2017.zip
$ aria2c -c http://images.cocodataset.org/zips/train2017.zip
$ aria2c -c http://images.cocodataset.org/zips/val2017.zip
$ aria2c -c http://images.cocodataset.org/zips/test2017.zip

5. COCO annotations

6. Install cocoapi
https://github.com/cocodataset/cocoapi
Install cocoapi using command-line commands:
$ sudo pip3 install pycocotools
Test:
$ python3
>> import pycocotools
Install COCO API before using COCO dataset.

7. Use cocoapi
(instances)
"categories": [
{
"supercategory": "person",
"id": 1,
"name": "person"
},…]
instances_val2017.json
['licenses', 'info', 'images', 'categories', 'annotations']
"images": [
{
"license": 4,
"file_name": "000000397133.jpg",
"coco_url":
"http://images.cocodataset.org/val2017/000000397133.jpg",
"height": 427,
"width": 640,
"date_captured": "2013-11-14 17:02:52",
"flickr_url":
"http://farm7.staticflickr.com/6116/6255196340_da26cf2c9e_z.jpg",
"id": 397133
},…]
"annotations": [
{
"segmentation": [
[
510.03,
423.01,
510.45,
423.01...]
],
"area": 702.10574,
"iscrowd": 0,
"image_id": 289343,
"bbox": [
473.07,
395.93,
38.65,
28.67
],
"category_id": 18,
"id": 1768
},…]
(2)
(1)
(3)
https://zhuanlan.zhihu.com/p/29393415

8. Use cocoapi (instances)
from matplotlib import pyplot as plt
from matplotlib.patches import Polygon
from skimage import io
from pycocotools.coco import COCO
import numpy as np
import os
annFile = "data/coco2017/annotations/instances_val2017.json"
root = "data/coco2017/val2017/"
coco = COCO(annFile)
catIds = coco.getCatIds(catNms=['person', 'dog', 'skateboard'])
imgIds = coco.getImgIds(catIds=catIds)
images = coco.loadImgs(imgIds[0:3])

9. Use cocoapi (instances)
for idx, image in enumerate(images, 1):
plt.subplot(1, 3, idx)
I = io.imread(image['coco_url'])
plt.axis('off')
plt.imshow(I)
plt.show()

10. Use cocoapi (instances)
for idx, image in enumerate(images, 1):
plt.subplot(1, 3, idx)
I = io.imread(image['coco_url'])
plt.axis('off')
plt.imshow(I)
annIds = coco.getAnnIds(imgIds=image['id'], catIds=catIds, iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
plt.show()

11. Use cocoapi (instances)
for idx, image in enumerate(images, 1):
# image
ax = plt.subplot(1, 3, idx)
I = io.imread(image['coco_url'])
plt.imshow(I)
plt.axis('off')
# segment
annIds = coco.getAnnIds(imgIds=image['id'], catIds=catIds, iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
# bbox
for ann in anns:
bbox_x, bbox_y, bbox_w, bbox_h = ann['bbox']
poly = [[bbox_x, bbox_y], [bbox_x, bbox_y+bbox_h],
[bbox_x+bbox_w, bbox_y+bbox_h], [bbox_x+bbox_w, bbox_y]]
np_poly = np.array(poly).reshape((4,2))
ax.add_patch(Polygon(np_poly, linestyle='--', facecolor='none', edgecolor="red", linewidth=2))
plt.show()

12. Use cocoapi (instances)

13. Use cocoapi
(keypoints)
person_keypoints_val2017.json
['licenses', 'info', 'images', 'categories', 'annotations']
"images": [
{
"license": 4,
"file_name": "000000397133.jpg",
"coco_url":
"http://images.cocodataset.org/val2017/000000397133.jpg",
"height": 427,
"width": 640,
"date_captured": "2013-11-14 17:02:52",
"flickr_url":
"http://farm7.staticflickr.com/6116/6255196340_da26cf2c9e_z.jpg",
"id": 397133
},…]
"categories": [
{
"supercategory": "person",
"id": 1,
"name": "person"
},…]
"annotations": [
{
"segmentation": [
[
125.12,
539.69,...]
],
"num_keypoints": 10,
"area": 47803.27955,
"iscrowd": 0,
"keypoints": [
162,
551,
2,...],
"image_id": 425226,
"bbox": [
73.35,206.02,
300.58,372.5
],
"category_id": 1,
"id": 183126
},

14. Use cocoapi (keypoints)
• visibility == 0 that keypoint not in the image.
• visibility == 1 that keypoint is in the image BUT
not visible namely maybe behind of an object.
• visibility == 2 that keypoint looks clearly. not
hidden.
"annotations": [
{
"segmentation": [
[
125.12,
539.69,...]
],
"num_keypoints": 10,
"area": 47803.27955,
"iscrowd": 0,
"keypoints": [
162,
551,
2,...],
"image_id": 425226,
"bbox": [
73.35,206.02,
300.58,372.5
],
"category_id": 1,
"id": 183126
},

15. Use cocoapi (keypoints)
annFile = "data/coco2017/annotations/person_keypoints_val2017.json"
coco = COCO(annFile)
catIds = coco.getCatIds(catNms=['person']) # the images icludes all categories
imgIds = coco.getImgIds(catIds=catIds)
images = coco.loadImgs(imgIds[10])
image = images[0]

16. Use cocoapi (keypoints)
# image
ax = plt.gca()
I = io.imread(image['coco_url'])
plt.imshow(I)
plt.axis('off')
# keypoints/segmentation
annIds = coco.getAnnIds(imgIds=image['id'], catIds=catIds, iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
# bbox
for ann in anns:
bbox_x, bbox_y, bbox_w, bbox_h = ann['bbox']
poly = [[bbox_x, bbox_y], [bbox_x, bbox_y+bbox_h],
[bbox_x+bbox_w, bbox_y+bbox_h], [bbox_x+bbox_w, bbox_y]]
np_poly = np.array(poly).reshape((4,2))
ax.add_patch(Polygon(np_poly, linestyle='--', facecolor='none', edgecolor="red", linewidth=2))
plt.show()

17. Use cocoapi (keypoints)

18. Use cocoapi (caption)
"annotations": [
{
"image_id": 179765,
"id": 38,
"caption": "A black Honda motorcycle parked in front of a garage."
},…]
captions_val2017.json
['licenses', 'info', 'images', 'annotations']

19. Use cocoapi (caption)
annFile = "data/coco2017/annotations/captions_val2017.json"
coco = COCO(annFile)
imgIds = coco.getImgIds()
images = coco.loadImgs(imgIds[0])
image = images[0]
# image
ax = plt.gca()
I = io.imread(image['coco_url'])
plt.imshow(I)
plt.axis('off')
# keypoints/annotations
annIds = coco.getAnnIds(imgIds=image['id'], iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)
plt.show()

20. Use cocoapi (caption)
A person kitesurfing over the waves of the ocean's shore.
a kite surfer is doing a flying trick over some water
A man is flying up in the air and having fun.
A guy is waterboarding in the ocean on a windy day.
A person kite boarding in rough seas near the shoreline.