Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

Published in: Technology
  • Be the first to comment


  1. 1. WHAT / WHY / HOW WE’RE ENGINEERING AT SMARTSTUDY Park Hyunwoo / @lqez 2013-05
  2. 2. • We started at Jun, 2010. • We do engineering for products and services, not for playing with breeding-edge technologies. • 26 freaks in this ship. And 14 of them are hackers. (May 2013) We are “Service Freak”
  3. 3. This slide describes: What we do? and Why we do it? and How we did it and will do it?
  4. 4. So WHAT
  5. 5. WHAT What we’ve created. } • Mobile Apps • Log Crawler • Log Analyzer • Push Solution • Web-based tools • Open Source
  6. 6. But WHY
  7. 7. WHY •Mobile Apps • Log Crawler • Log Analyzer • Push Solution • Web-based tools • Open Source Mobile application markets are fair.Anybody can upload their apps and deploy globally. : We’ve developed and deployed 300+ apps in mobile era.
  8. 8. WHY • Mobile Apps •Log Crawler • Log Analyzer • Push Solution • Web-based tools • Open Source We knew that the success of business is up to keep eyes on the behavior of users. : We adapt technologies and experiences what we learned from 10y+ online game services.
  9. 9. WHY • Mobile Apps • Log Crawler •Log Analyzer • Push Solution • Web-based tools • Open Source a Bunch of logs has no meaning. We have to examine them in various ways via visualization. : We polish logs into Data, and again to Information, at last, we’ll get Insight from it.
  10. 10. WHY • Mobile Apps • Log Crawler • Log Analyzer •Push Solution • Web-based tools • Open Source In mobile market, push message works as same as email. Even more powerful. : We build redundant storing & massive reservation system.
  11. 11. WHY • Mobile Apps • Log Crawler • Log Analyzer • Push Solution •Web-based tools • Open Source Web is the best platform to serve tools without a headache of updating, and runs on every machines. : We stand on HTML5 with high-productivity framework like Django.
  12. 12. WHY • Mobile Apps • Log Crawler • Log Analyzer • Push Solution • Web-based tools •Open Source Open source is not only codes, but also culture. : We uses open sources and open our codes to the world.
  13. 13. Then HOW
  14. 14. HOW/ Mobile apps
  15. 15. HOW/ Mobile apps OpenGLNativeWeb
  16. 16. HOW/ Mobile apps OpenGLNativeWeb
  17. 17. OpenGLNativeWeb HOW/ Mobile apps
  18. 18. HOW/ Mobile apps OpenGLNativeWebThere are no boundary or no fixed rules. We just choose the best fit architecture.
  19. 19. HOW/ Mobile apps iOS / Android Native API OpenGL SMARTSTUDY Library Cocos2d-x C/C++ NDKPython HTML Javascript CSS Mobile Apps and Services
  20. 20. HOW/ Mobile apps Developer Z iOS Android Developer M iOS Android Web Developer B Android Windows Developer H iOS Android Bada
  21. 21. HOW/ Mobile apps Developer Z iOS Android Developer M iOS Android Web Developer B Android Windows Developer H iOS Android Bada We develop on various platforms, so Every developer are needed to work on two or more platforms.
  22. 22. HOW/ Mobile apps • We’ve focused on each app and content, not on making platform. • Release apps on various mobile platforms even on Windows Store and Smart TV. • Each developer has their own ‘Main role’ but also ‘Sub role’ to expand experiences. GOOD
  23. 23. HOW/ Mobile apps • We have own platform-independent data format and service models, but do not have own rich middleware or library. • Too many apps to be serviced. • Pressure of multi-platform experience. BAD
  24. 24. HOW/ Mobile apps • We have lots of codebase on each apps. • If we can refine them into loose-tied libraries, it’ll be good to prevent making a ‘Wheels’ again. • Document it and Share it. TO-BE
  25. 25. HOW/ Log Crawler & Analyzer
  26. 26. HOW/ Log Crawler & Analyzer • Traditional RDB + NoSQL solution • We uses MySQL InnoDB / MyISAM with MongoDB to store & analysis logs.
  27. 27. HOW/ Log Crawler & Analyzer 230logs/sec 20,000,000logs/day about 10GB/day
  28. 28. HOW/ Log Crawler & Analyzer over 3 billion logs till now.
  29. 29. Server Apache httpd mod_php Server Redis Apache httpd mod_php Predis Server Apache httpd mod_php PredisPredis
  30. 30. ReplicaSet Secondary Secondary Primary Redis
  31. 31. PV / UV DAU / MAU PU / ARPU NRU / RR ReplicaSet
  32. 32. See also: Redis, MongoDB and MySQL Crawl and Analysis log in Mobile Application Service Link : Video :
  33. 33. HOW/ Log Crawler & Analyzer GOOD • Fault tolerance log crawler. • Client stores logs in own local storage when it failed to send logs by any reason. (i.e. server problem, network unreachable) • Redis can hold logs while MongoDB have problems while upgrading or replicating.
  34. 34. HOW/ Log Crawler & Analyzer GOOD • MongoDB can hold various types of log without creating tables for each log type. • And good bye long ‘ALTER TABLE’. • MySQL is very familiar for us.
  35. 35. HOW/ Log Crawler & Analyzer BAD • Storing entire logs in only one collection is not a good idea. • Lock strangle us every each second. • Copying logs from MongoDB to MySQL needs logs of time.
  36. 36. HOW/ Log Crawler & Analyzer • Divide logs into small pieces in MongoDB. • Use aggregation framework added in 2.1. • Use shorter key name. • Add more memory and SSD. • There are (still) many global locks. Use more physical instances for each purpose. TO-BE
  37. 37. HOW/ Push Solution
  38. 38. HOW/ Push Solution • Support APNS, GCM and C2DM. • Collect ~200 tokens per every second. • Storing 20M+ tokens in MongoDB. • Adapt a similar architecture of the Log crawling system.
  39. 39. HOW/ Push Solution • Stores tokens with version of app, language, timezone and other meaningful device information without personal’s. • With reservation system, we can send messages to the targeted users. • We sent ~30,000,000 msgs per month. • via Urban Airship = $3,000 per month.
  40. 40. HOW/ Push Solution TO-BE • Current token database is isolated from other application database. • To make better user experience, combination of two databases is needed. • Timezone awareness.
  41. 41. HOW/ Web-based tools
  42. 42. HOW/ Web-based tools • Web - actually, web browser is a good platform to provide rich toolkits rapidly. • We use Python and PHP as backend and mainly use Django and Flask. • And also use jQuery, Bootstrap and Google Chart library for frontend. (as like as you)
  43. 43. See also: Creating a comic service Link : Video :
  44. 44. See also: Do not use Django as like as SMARTSTUDY Link : as-like-as-smartstudy
  45. 45. HOW/ Web-based tools TO-BE • Currently, there is no rule for Authentication and Authorization. (some uses just LDAP) • So,We’re building standard A-A library via OAuth2.
  46. 46. HOW/ Open Source
  47. 47. HOW/ Open Source • Open source is not only code or license. • It is a culture for hackers. • We’re using open source projects for our service widely. • We also recommend to make in-house library or middleware as open source.
  48. 48. HOW/ Open Source • Can’t enumerate all in here, • Linux, OpenLDAP, Redmine, Cacti, Nginx, MySQL, MongoDB, watch, ImageMagicK, ffmpeg, libx264, OpenSSH, awk, sed, find, Python, wget, Celery, PHP, tar, gzip, curl, zlib, iptables, Django, Flask,Werkzeug, vim, fabric, eventlet, Sentry, django-debug-toolbar, Bootstrap, jQuery, Postgres, Redis, uWSGI...
  49. 49. HOW/ Open Source • And open our sources, even they’re tiny. • npk : Neat package library • • django-project-layout • • FormEncode-Jinja2 •
  50. 50. HOW/ Open Source • And we gave back to Open source by another way - Open a hands-on session. • Django In Action,Aug 2012 at SMARTSTUDY Seoul HQ. • 40 developers learned from the installation of Python to Django in 4H. •
  51. 51. HOW/ Open Source TO-BE • Open more codes to community. • Open more sessions, especially hands-on sessions. It’s very rare in Korea. • Be more sponsors for developer conferences. • Contribute to open source projects.
  52. 52. AND • Using AWS CloudFront with S3 • • Post-mortem of Making a comic service • (in Korean) • Using Sentry for Django applications • (in Korean) • Test environment for mobile devices • (in Korean)
  54. 54. Good Product not great or elegance platforms.
  55. 55. Content rules platform, Time rules talent.
  56. 56. So,
  57. 57. Our most important value is
  58. 58. YOU, AND I
  59. 59. We, Human do make it works. not code or turing machines.