In this technical but code-free sharing in Taipei.py, I shared several project experience in KKBOX Group, which mostly are related to Python programming language. It's about how we face and solve challenges by Python. It's our experience to share with the world.

1. Python Experience in KKBOX Group
Shuen-Huei (Drake) Guan
#Taipei.py (It was originally planned for PyCon TW)
04/21/2016

2. Let’s conduct some surveys first.
pyKKBOX? pyUIA? Mass?

3. @drakeguan
- Technical Director, KKStream
- Sr. Principal Engineer, KKBOX
- Volunteer, PyCon Taiwan

4. “This talk is about a compilation of tasks and projects, done in
Python, in KKBOX Group. Each project has its own
challenges and context to deal with. What you can take home
are some stories and experience.”
Warning: it’s just our own experience, not suitable to yours!

5. pyKKBOX, 2011

6. KKBOX “一起聽”

7. KKBOX “一起聽”
● An user can broadcast what he/she is listening to others.
● Multiple users are then able to subscribe to his/her station, listening to the
same song simultaneously.
● A major/highlighted KKBOX feature developed in 2011.

8. Challenges by “一起聽”
● iOS (client) co-works with Windows (broadcaster) only.
● There is even no broadcasters to listen to!
● In academia, we may call this “ill-posed (optimization) problem”.

9. pyKKBOX, https://github.com/KKBOX/pyKKBOX

10. pyKKBOX
● It is a lite-weight, not fully functional client.
● It is initially just a bot, which can login, and do broadcasting.
● It should be easy because we are iOS client team, not server team.
● It’s a proof of concept.

11. API Testing by pyKKBOX
● Now we have bots:
○ A tool to do stress testing.
○ A tool to provide radio-like automatic service.
● And then, more APIs are in.
● Test cases for KKBOX API!
● Hourly testing on Jenkins CI.

12. Playlist Auto-Sync
● CPL (Cloud PlayList) synchronization.
● The sync is not done based on data (playlist), but JSON-based commands!
● In 3 months, the API’s version changed 5 times.
● n-to-m synchronization makes it even worse.
● pyKKBOX is, again, used to prototype even though technical spec is not
finalized.

13. Toy Project: Photomosaic by pyKKBOX

14. Album covers by pyKKBOX
Yes. That’s me.

15. Photomosaic
● It’s based on KKBOX 2014 workshop group photo.
● 1,500 album covers are used.
● Final photomosaic resolution: 14k x 6k.
● I didn’t read the official KKBOX API yet. Just self-introspection of pyKKBOX in
IPython is enough.
● Coding time: less than half hour.

16. pyKKBOX facts
● Initiated in 2011, to ~2013.
● Two main contributors (Welly Liu and @zonble), but with ~10 contributors then.
● Just the builtin Python modules (including PyObjC) are used.
“Another weapon for app developers;
prototyping, or even trash-able code, is a great fit here;
being a glue, a common language among diversified developers if you’re doing
multi-platform product development;
Python speaks plain language.”
● Hmm, we have no further plans yet. :p

17. pyUIA, 2013

18. Testing Goals & Reports

19. Testing for Playlist Auto-Sync
● Easy to manage and organize playlists across various devices.
● Multiple devices, mostly in different platforms, are involed.

20. One Test Case (iOS > Android)
1. On iOS, create a new playlist.
2. On Android, check if the playlist
created in step 1 appears.

21. A (Pseudo) Test Script that Works!

22. Abstraction of Layers for Testing

23. pyUIA, https://github.com/imsardine/pyuia

24. pyUIA, https://github.com/imsardine/pyuia

25. pyUIA facts
● Initiated in 2014 and it keeps evolving.
● It doesn’t take much time because it’s just simply works. v0.3 is released on
June 2015. http://www.slideshare.net/jeremykao92/pyuia-03
● 3 developers contribute their talents.
● Python packages: selenium, robotframework and Appium-Python-Client.
“Somehow, we just want to implement automated tests in Pythonic way!”
● 120 Release Acceptance Test (RAT) are on pyUIA, ~20% cases.

26. Video Encoding System, 2014

27. Video Streaming Service
● KKBOX forms a team, Video Development, to build video streaming services
in Japan. They are Videopass and Animepass.
● More than 1,000,000 subscription users.
● More than 30,000 videos, not that many but resource-consuming.

28. Challenges for Video Encoding System
● Given a couple of thousands of videos;
● given ~150 videos per day;
● given a scalable number of encoders on EC2 instances;
● assuming the workflow for each video can be different;
● how to make a robust system to handle this challenge in consideration of
○ just 1.5 developer(s)
○ everything can be broken
○ computing resource is expensive (c3.8xlarge, c4.8xlarge, …)
○ it’s a vivid world~

29. One Simplified Audio Encoding Workflow

30. Another Simplified Encoding Workflow

31. Render Farm Idea from VFX/Animation Industry
Ed Catmull, Pixar
Pixar’s Alfred

33. http://bit.ly/1rnR0Ht

34. Mass, https://github.com/kkbox/mass

35. Mass, https://github.com/kkbox/mass
● Job script as the central data to control the workflow, instead of fixed coding
logics in workers.
● Focus on data, the workflow and real work, than underneath system or
infrastructure.
● Leverage AWS SWF (or Apache Kafka, Mesos, or other message brokers) to
provide robust, fault-tolerant and fully automatic distributed system.

36. Mass facts
● Initiated in 2014, and it keeps rocking~
● Just 2 mid-level Python developers (they both have presented in PyCon TW).
● Python packages: arrow, awscli, boto, click, clint, flask, filechunkio, flask,
fluent-logger, isodate, logbook, parmap, requests, sh.
“We hope to polish the open-sourced Mass much sexy and useful in near
future.”
● 3,000,000-minute videos got encoded through it.
● Mass once finished 3,000 videos encoding within one day without any issue
while 800 encoders on 100 c3.8xlarge instances were on duty.

37. RDC Toolbox, 2015

38. KKBOXers demands more data-driven facts/info

39. One example of request ticket from production team

44. Someone can learn from others’ queries!

45. “透過Python你幾乎可以做到任何事情, 剩下的就只是
你不知道要這樣做而已, 但是用戶的使用方式會教你
這件事。”
--- RDC Toolbox main developer

46. RDC Toolbox facts
● Initiated in Oct 2015, still quite young.
● 1.5 developer(s) with many contributes then.
● Python packages: flask, pandas, psycopg2, redis, requests, slacker, sqlparse.
“python 的強大社群
想架站的時候有 flask
想處理資料的時候有 pandas
想用創意的方式來log有 slacker
切測試環境的時候有 virtualenv
如果不使用python, 我應該沒辦法在一個人和這麼短的時間內完成 prototype”
● 10,000,000,000 metering data to deal with.
● 10,000 queries conducted by (mostly) non-developers.
● Lots of future plan. We are building music streaming specific BI tools.

47. AWS Lambda CI/CD
It would get shared in PyCon TW 2016

48. Finally, let’s do some surveys again,
but in KKBOX Group (just for fun).

49. What programming languages have you used
to contribute your talents in KKBOX Group?

51. What programming languages are you
HAPPY to work with in KKBOX Group?

53. What programming languages are you planning
to learn for future challenges in KKBOX Group?

55. Thank you!