This talk I gave on applying Deep Learning Techniques on 3D Point Clouds

1. Deep$Learning$
on
3D$Point$Clouds
SK#Reddy#
Chief#Product#Officer#AI
skreddy99
skreddy99

2. Confidential2
ImageNet(Large(Scale(Visual(Recognition(Challenge((ILSVRC)(winners

3. Confidential3 https://arxiv.org/pdf/1808.01462.pdf

4. Confidential4 https://arxiv.org/pdf/1808.01462.pdf

5. Confidential5
O"CNN%&2017
https://arxiv.org/pdf/1712.01537.pdf
Left:<the<original<3D<shape.<Middle:<the<voxelized 3D<
shape.<Right:<the<octree<representation<with<normals
sampled<at<the<finest<leaf<octants
2D<quadtree<illustration<of<octree<structure

6. Confidential6
OctNet&'2017
https://arxiv.org/pdf/1611.05009.pdf
Network=Architecture=Semantic=3D=Point=Cloud=
Labeling
Hybrid=GridKOctree=Data=Structure.=This=example=illustrates=a=
hybrid=gridKoctree=consisting=of=8=shallow=octrees=indicated=by=
different=colors.=Using=2=shallow=octrees=in=each=dimension=
with=a=maximum=depth=of=3=leads=to=a=total=resolution=of========
voxels
Bit=Representation.=Shallow=octrees=can=be=efficiently=encoded=
using=bitKstrings.=Here,=the=bitKstring=1=01010000=00000000=
01010000=00000000=01010000=0...=defines=the=octree=in=(a).=The=
corresponding=tree=is=shown=in=(b).=The=color=of=the=voxels=
corresponds=to=the=split=level

7. Confidential7
PointNet()2017
https://arxiv.org/pdf/1612.00593.pdf

8. Confidential8
Deep$Kd'Network.$2017
https://arxiv.org/pdf/1704.01222.pdf
A<kd>tree<built<on<the<point<cloud<of<eight<points<(left),<
and<the<associated<Kd>network<built<for<classification<
(right)
The<architecture<for<parts<segmentation<(individual<point<classification)<
for<the<point<cloud

9. Confidential9
PointNet++)*2017
https://arxiv.org/pdf/1706.02413.pdf
(a) MultiAscaleCgroupingC(MSG)FC
(b) MultiresolutionCgroupingC(MRG)
MNISTCdigitCclassification

10. Confidential10
SPLATNet)*2018*
https://arxiv.org/pdf/1802.08275.pdf
Bilateral*Convolution*Layer Left:<splatting<the<input<points<(orange)<onto<
the<lattice<corners<(black)C<Middle:<The<extent<of<a<filter<on<the<lattice<with<a<s<=<2<
neighborhood<(white<circles),<for<reference<we<show<a<Gaussian<filter,<with<its<
values<color<coded.<The<general<case<has<a<free<scalar/vector<parameter<per<
circle.<Right:<The<result<of<the<convolution<at<the<lattice<corners<(black)<is<projected<
back<to<the<output<points<(blue).<

11. Confidential11
MRTNet'(2018
https://arxiv.org/pdf/1807.03520.pdf

12. Confidential12
SqueezeSeg:.CNNs.with.Recurrent.CRF.for.Real8Time.Road8
Object.Segmentation.from.3D.LiDAR.Point.CloudE.2017
https://arxiv.org/pdf/1710.07368.pdf
Conditional.Random.Field.(CRF).as.an.RNN.layer

13. Confidential13
PointSeg-.Sep.2018
https://arxiv.org/pdf/1807.06288.pdf
The.pipeline.of.PointSeg..Raw.LiDAR.point.cloud.is.projected.to.a.multichannel.range.
image..After.the.network.processing,.the.predicted.pointLwise.label.will.be.projected.back.
to.the.original.space.
Source.code:.https://github.com/ywangeq/PointSeg

14. Confidential14
LMNet:.Real0time.Multiclass.Object.Detection.on.CPU.using.3D.LiDAR@.2018
https://arxiv.org/pdf/1805.04902.pdf
IMPLEMENTED.DILATED.LAYERS

15. Confidential15
DeLS%3D:.Deep.Localization.and.
Segmentation.with.a.3D.Semantic.Map;.2018
https://arxiv.org/pdf/1805.04949.pdf

16. Confidential16
Spherical CNN for/3D/Point/Clouds5/2018
https://arxiv.org/pdf/1805.07872.pdf
Classification/performance/on/ModelNetszz

17. Confidential17
https://arxiv.org/pdf/1808.06840.pdf
Example(result(of(our(FCPN(on(semantic(voxel(labeling(and(captioning(on(an(Tango(3D(
reconstruction(/(point(cloud:((a)(3D(reconstruction((not(used),((b)(Input(point(cloud((c)(Output(
semantic(voxel(prediction.(
Visualization(of(the(semantic(segmentation(on((a)(a(depth(image,((b)(a(2.4m×2.4m×2.4m(partial(
reconstruction((c)(an(entire(reconstruction(of(a(hotel(suite.(Please(note(that(each(of(these(outputs(
are(predicted(in(a(single(shot(from(the(same(network(trained(with(2.4m(× 2.4m(× 2.4m(volumes(
Fully%Convolutional-Point-Network-Architecture Aug=2018=
(Source=code:=Not=yetD=https://github.com/drethage/fullyGconvolutionalGpointGnetwork)

18. Confidential18
Data$sets
• ScanNet:2Indoor2Scenes
• ShapeNet:23D2shapes
• Princeton2ModelNet
• KITTI2(Code2available2for2some)
• Pascal3D+2(….in2the2wild)
• Pascal2VOC
• SUNCG2(room2models)
• SceneNet (RGBMD2room2models)
• CV2Lab2datasets2with2Code2
(http://cvgl.stanford.edu/resources.html)
• Collective2Activity2Dataset2(old)
• Monocular2Multiview2Object2Tracking2with2
3D2Aspect2Parts2(???)
• Stanford22DM3DMSemantics2Dataset2(2DM
3DMS)2(**)
• Stanford2Drone2Dataset
• ObjectNet3D
List$of$datasets:
1.2http://clickdamage.com/sourcecode/cv_datasets.php
Other$Data$sets
• TOSCA2(nonMrigid2dataset)
• FAUST2(highMres2human2scans)
• SUN3D
• NYC3DCars
• LabelMe3D2(old2dataset)
• CVLab (MultiMview2car2dataset)
• MIT2Street2Scenes
• Daimler2Pedestrian2Datasets
• Caltech2Pedestrian2Detection2Benchmark
• Robust2MultiMPerson2Tracking2from2Mobile2Platforms
• FlyingThings3D2(Synthetic)
• BU4DFE23D2

19. Confidential19
PVNet:"A"Joint"Network"of"Point"Cloud"and"Multi6View"for"
3D"Shape"Recognition"(Aug"2018)

20. Confidential20 https://arxiv.org/pdf/1806.01411.pdf
FlowNet3D:;Scene;Flow;in;3D;PCs(June;2018);

21. Confidential21
YOLO3D Aug02018
Sample0of0the0output0shown0in03D0and0projected0on0the0top0view0map0

22. Confidential22
PointFlowNet 3D.Scene.Flow.Estimation.from.PCs.(Sep.2018)
https://arxiv.org/pdf/1806.02170.pdf

23. Confidential23
Point&Cloud&Library Modules

24. Confidential24
https://github.com/kzampog/cilantro;
Cilantro
library;for;point;cloud;data;processing;
Bundled;with:;
• nanoflann (for;fast;kdAtree;queries);
• Spectra;(ARPACKAinspired;library;for;large;
scale;eigen;decompositions;
• Qhull (for;convex;hull;and;halfA space;
intersection;computations);
• tinyply (for;PLY;format;geometry;I/O);
• External;dependencies:;
• Eigen;(linear;algebra;library);
• Pangolin;(lightweight;OpenGL;viewport;
manager;and;video;I/O;abstraction;library)

25. Confidential25
Thank&you
SK&Reddy
Chief&Product&Officer&AI
skreddy99
skreddy99