Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

スタディサプリを支えるデータ分析基盤 ~設計の勘所と利活用事例~

13,216 views

Published on

デブサミ2017【17-B-5】の講演資料です。
http://event.shoeisha.jp/devsumi/20170216/session/1291/

Published in: Technology
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

スタディサプリを支えるデータ分析基盤 ~設計の勘所と利活用事例~

  1. 1. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  2. 2. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  3. 3. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  4. 4. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  5. 5. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  6. 6. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  7. 7. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  8. 8. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  9. 9. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  10. 10. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  11. 11. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  12. 12. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  13. 13. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  14. 14. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  15. 15. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  16. 16. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  17. 17. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  18. 18. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  19. 19. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  20. 20. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  21. 21. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  22. 22. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  23. 23. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  24. 24. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  25. 25. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  26. 26. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  27. 27. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  28. 28. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  29. 29. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. • ( https://ring.education/ ) - - https://ring.education/research/2016_hs_al_experiment_report_02/
  30. 30. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  31. 31. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  32. 32. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  33. 33. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  34. 34. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  35. 35. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  36. 36. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  37. 37. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  38. 38. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  39. 39. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  40. 40. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. id grade_id gender school_id AAA 001 female 0000X BBB 002 male 0000Y CCC 005 male 0000Z id label 001 002 003 id label 0000X X 0000Y Y 0000Z Z id grade gender school AAA female X BBB male Y CCC male Z
  41. 41. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  42. 42. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  43. 43. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  44. 44. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  45. 45. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  46. 46. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  47. 47. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. Rails Kinesis Lambda TD SDK web JS app native web Single Page Application
  48. 48. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  49. 49. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. /(^o^)
  50. 50. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  51. 51. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  52. 52. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  53. 53. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  54. 54. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. embulk-input-mongodb by @hakobera embulk-input-postgresql embulk-parser-json embulk-filter-expand_json embulk-filter-hash by @kamatama41 embulk-filter-mask by @beniyama embulk-output-td
  55. 55. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  56. 56. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  57. 57. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  58. 58. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. Embulk Hive / Presto
  59. 59. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  60. 60. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. class DailyVideoRatingReport(luigi_td.Query): type = 'hive' database = LuigiConfig().main_db report_database = LuigiConfig().report_db source = ‘queries/hive/daily/history/daily_video_rating_report.sql’ jst_date = luigi.DateParameter(default=DateTimeUtil.get_jst_date()) def requires(self): return [ DailyUserMasterCreate(self.jst_date), DailyContentsMasterCreate(self.jst_date) ]
  61. 61. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. def run(self): TdUtil.drop_partition( table=‘video_rating_report’, to_jst_date=self.jst_date ) super(DailyVideoRatingReport, self).run() def output(self): return luigi_td.ResultTarget( '{0}/DailyVideoRatingReport-{1}.txt'.format( LuigiConfig().output_path, self.jst_date.strftime('%Y-%m-%d') ), 'td://@/reports/video_rating_report?mode=append' )
  62. 62. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  63. 63. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. +video_rating_report: +partial_delete: td_partial_delete>: video_rating_report database: reports from: ${moment(session_time).utc().add("days", -1).unix()} to: ${moment(session_time).utc().unix()} +insert_records: td>: queries/reporting/video_rating_report_insert.sql database: reports engine: presto insert_into: video_rating_report
  64. 64. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. +fetch_records: td>: queries/reporting/video_ratings_fetch.sql database: reports engine: presto store_last_results: true +slack_notification: http>: https://hooks.slack.com/services/XXXXXXXX method: POST content: text: “Average rating is ${td.last_results.average_ratings}” link_names: true content_format: json
  65. 65. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  66. 66. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  67. 67. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  68. 68. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  69. 69. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  70. 70. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. web console
  71. 71. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  72. 72. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  73. 73. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  74. 74. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  75. 75. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  76. 76. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  77. 77. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  78. 78. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  79. 79. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  80. 80. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  81. 81. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  82. 82. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.
  83. 83. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved.

×