Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Depth in Depth #iOSDC

2,605 views

Published on

iOSDC Japan 2018でのプレゼンテーションスライドです。

## 概要

原始のiPhoneからカメラは搭載されていましたが、深度センサが搭載されたのは比較的最近のことです。カメラやGPSが、デジタルの世界と我々が生きる現実世界を繋ぐ重要な役割を担い、アプリ開発者に多くの創造性を与えてくれたのと同様に、「奥行き」がわかるようになったというのはアプリ開発の次元がひとつ増えたようなものです。本トークではiOSでの深度の扱いについて、取得方法だけではなく、細かいパラメータの意味やMetalでの処理まで詳解します。

## アジェンダ

## アジェンダ

- 深度の種類とセンサのしくみ
- 深度データを取得する
- 深度データを使ってみる
- できること・できないこと事例集

Published in: Mobile

Depth in Depth #iOSDC

  1. 1. Depth in Depth Shuichi Tsutsumi @shu223 iOSDC 2018
  2. 2. • Fyusion Inc. @ H-1B • Senior Principal Software Engineer • → • @ • 8 IoT Metal ARKit NFC etc...
  3. 3. • GitHub: shu223 • : Qiita Note • :
  4. 4. Depth in Depth
  5. 5. 1 RGB
  6. 6.
  7. 7. : •
  8. 8. : 3D • 3D •
  9. 9. : 30 ※
  10. 10. • iOS API • •
  11. 11. • • • •
  12. 12. • • • •
  13. 13. Disparity iPhone 7/8 Plus, X
  14. 14. Focal Length
  15. 15. Disparity
  16. 16. Depth Disparity
  17. 17. Depth Disparity • Depth(z) = Disparity(d)
  18. 18. Time of Flight (ToF) • Depth •
  19. 19. TrueDepth iPhone X
  20. 20. • Proximity sensor ToF
  21. 21. • Dot projector ( ) 3
  22. 22. • Infrated camera ( ) →
  23. 23. • • • •
  24. 24. AVDepthData • • • @available(iOS 11.0, *)
  25. 25. AVDepthData var depthDataMap: CVPixelBuffer { get } • • CVPixelBuffer Core Image Metal
  26. 26. iOS • 1: • 2: • 3: ARKit
  27. 27. 1: • ImageIO • URL Photos
  28. 28. URL CGImageSource let source = CGImageSourceCreateWithURL(url as CFURL, nil)
  29. 29. CGImageSource Dictionary let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex( source, 0, kCGImageAuxiliaryDataTypeDisparity // ) as? [String : AnyObject] // info CFDataRef // CGImageMetadataRef
  30. 30. Dictionary AVDepthData let depthData = try! AVDepthData(fromDictionaryRepresentation: info)
  31. 31. 2: • AVFoundation •
  32. 32. AVCaptureDevice .builtInDualCamera, .builtInTrueDepthCamera let device = AVCaptureDevice.DiscoverySession( deviceTypes: [.builtInDualCamera], mediaType: .video, position: .back).devices.first
  33. 33. var supportedDepthDataFormats: [AVCaptureDevice.Format] • AVCaptureDevice.Format • AVCaptureDevice activeDepthDataFormat
  34. 34. AVCaptureDepthDataOutput let depthOutput = AVCaptureDepthDataOutput() captureSession.addOutput(depthOutput) depthOutput.setDelegate( self, callbackQueue: dataOutputQueue) • AVCaptureVideoDataOutput AVCaptureAudio
  35. 35. AVCaptureDepthDataOutputDelegate didOutput AVDepthData func depthDataOutput( _ output: AVCaptureDepthDataOutput, didOutput depthData: AVDepthData, timestamp: CMTime, connection: AVCaptureConnection)
  36. 36. 3: ARKit ARFrame capturedDepthData var capturedDepthData: AVDepthData? { get }
  37. 37. ⚠ • 1: • 2: • 3: ARKit
  38. 38. • • • •
  39. 39. • chroma
  40. 40. CIBlendWithMask Parameters inputImage inputBackgroundImage inputMaskImage • • CIFilter Metal
  41. 41. OK!?
  42. 42. • • CIFilter CIBlendWithMask • Metal MTKView
  43. 43.
  44. 44.
  45. 45. 2 • AVCaptureMetadataOutput • +offset cutOff • • cutOff • cutOff
  46. 46. • • "Holes"
  47. 47. • • Holes RGB
  48. 48. isFilteringEnabled • AVCaptureDepthDataOutput • • "Holes" RGB
  49. 49. • •
  50. 50. Portrait Matte
  51. 51. Portrait Matte • • iOS 12
  52. 52. Portrait Matte
  53. 53. • CGImageSource • PortraitEffectsMatte • AVDapthData AVPortraitEffectsMatte
  54. 54. guard let info = CGImageSourceCopyAuxiliaryDataInfoAtIndex( self, 0, kCGImageAuxiliaryDataTypePortraitEffectsMatte ) as? [String : AnyObject] else { return } let matte = AVPortraitEffectsMatte(fromDictionaryRepresentation: info)
  55. 55. Portrait Matte • • • iOS 12
  56. 56. • • • •
  57. 57. • / / ARKit • Dual / TrueDepth / • / • iOS 12 / 11 / • / /
  58. 58. Photoshop • • Portrait Matte
  59. 59. • Portrait Matte •
  60. 60. • Portrait Matte • Disprity • •
  61. 61. ARKit • OK • NG • ARFaceTrackingConfiguration • ARWorldTracking capturedDepth nil
  62. 62. • • • •
  63. 63. • Twitter: https://twitter.com/shu223 • GitHub: https://github.com/shu223 • : http://shu223.hatenablog.com/
  64. 64. AVCaptureDataOutputSynchronizer
  65. 65. // synchronizer = AVCaptureDataOutputSynchronizer( dataOutputs: [videoOutput, depthOutput])
  66. 66. AVCaptureDataOutputSynchronizerDelegate didOutput AVCaptureSynchronizedDataCollection • → AVCaptureSynchronizedDepthData • → AVDepthData
  67. 67. // AVCaptureSynchronizedDepthData if let syncedDepthData = synchronizedDataCollection.synchronizedData(for: depthOutput) as? AVCaptureSynchronizedDepthData, !syncedDepthData.depthDataWasDropped { // AVDepthData let depthData = syncedDepthData.depthData ... }
  68. 68. Metal 3D
  69. 69. • • z • • RGB

×