The document summarizes the development of a scratch card game for LINE NOW by a team in one month. Key points: - The team developed a scratch card reward game for an upcoming Moon Festival campaign in LINE NOW within one month using agile methods. - They created an algorithm to randomly generate scratch cards that ensured prizes would be distributed evenly over time to avoid issues of running out of prizes or complaints from users. - The architecture included LIFF and microservices with Redis to store card data, Kafka for events, and deployment to Kubernetes. - Challenges included performance testing of the card generation and redemption APIs, scaling to handle campaign traffic peaks, and recovering when Kafka went down on the last

2. LINE NOW Scratch Card
From Nothing to Production in one month

3. About me
• Julian Shen
• Backend dev @ LINE Taiwan
• Dev lead of LINE NOW Project

4. QA
Julian Hsueh-
Min
Yu-mei Alex
Denny Shiang-Yu Deann
Johnny Bryan Kay Ting-yu
Team
• 4 Back-ends, 3 Front-ends, 4 QA
• Love to use new stuffs
• Java/node.js/Scala/Kotlin/Go
• Spring/Finagle
• Docker/K8S/Rancher
• Mongo/Redis/Mysql/Kafka
• Vue.js
• SOA/Micro services
• Willing to change and take challenges
• Running as a scrum team
• What‘s LINE NOW?
• Beacon/Location & Events
• We’re hiring

5. It’s a story about moonlight
?
• We planned to develop a new reward
type - [Scratch Card] for LINE NOW
• We want this to be “different”
• There was a campaign for Moon Festival
also like to use Scratch Card as reward
• Delayed due to design
• Need to complete it in one month

6. Agile?
Quality
Resources
Schedule Scope
Don’t do things with infinite scope if you only have limited resources and schedule

7. What exactly LINE NOW Scratch Card
is?
• A game to dispatch
rewards (scratch & get)
• LINE Points
• Coupon
• Stickers
• A tool for campaign
• Build with LIFF
(https://developers.line.me
/en/docs/liff/overview/)

8. What we try to solve?
• Problems with traditional “lucky draw” game design
• “Probability” means anything is possible
• Number of prizes is fixed but probability is not
• Prizes can be consumed too quick or too slow
• Need monitoring and adding prizes anytime
• User complaints
• User should get the prize but did not received
• User shouldn’t get a prize but he claims he should

9. Traditional Lucky Draw games
Prize 1 (0.1%) Prize 3 (10%)Prize 2 (5%)
Prize 4 (15%) Prize 6 (20%) Prize 7 (30%)
End user
Pick one based on probability
Other (No
prize)
In fact
Prize 1 (1) Prize 2 (50)
Prize 3 (100) Prize 4 (200)
Prize 6 (400)
Prize 7
(1000)
Prize count

10. Card Generation Algorithm (Bryan’s
Algorithm)
Card
Generator
publish
Card
Card
Card
Card
Card
Card
Card
Card
Card
Card
Card pool (Redis)
Draw card api
generate
• From observation of real
world scratch card
• Card generator also
generate the rule for draw
(to make sure prize
dispatching is even)
• Tile permutation is
rendered at generation
time

11. Architecture
Campaign
system
End user
LIFF
Frontend
Scratch
API
Reward
agent
Send reward through chat bot
draw
Get card
Draw information
Redeem
Generator
Meta
DB
CMS
Kafka
connect
Operator
Scratch/redeem
Customer Care
Admin
tool

12. BTW, system are deployed to a K8S
cluster
Rancher 2.0

13. Challenges in front-end development
Scratch traces
• Screen shots are logged several
times during user play
• To reduce the traffic
• Last result is saved at local
storage
• Only send “diff”
• Performance
• Calculating diff might block
main thread
• User experience
• How to make user feel like
scratching a real card

14. Challenges in back-end development
• Identify possible peaks (Moon Festival campaign)
• Marketing events and press exposure might generate boost traffic
• Banner promotion might generate boost traffic
• There is a special promotion period (buy 1 sticker get 2 scratch cards)
• Performance concerns
• Use REDIS as the main data source during campaign
• Data is written asynchronously
• Traffic is not easy to calculate precisely always
• API should be able to do horizontal scaling
• REDIS should be able to store millions of cards
• Monitoring is necessary (but we haven’t have time)
• Failure prevention
• REDIS could be the SPOF
• Data should be able to reconstruct from somewhere else (Kafka)

15. Testing
• Focus on what we most concern
• Card generation
• Check data in REDIS to make sure each card generated by generator should
follow the rule
• Traffic
• Load testing Scratch Card API
• K6 (https://k6.io/)
• Storage size
• Calculate data size for each item in REDIS
• Generate millions cards to REDIS to verify
• Kafka connect worker was down once during testing
• Cannot find scratch event in admin tool
• Recoverable

16. Sometimes shit just happens
• Kafka was down at the last day of Moon festival campaign
• No scratch events coming in
• Not possible to redeem
• Why?
• We use self-hosted Kafka on our own K8S (to save time to
deploy)
• Run out of storage space
• How we recovered?
• This situation is not in our expectation
• Redeem events are the most important
• 狡兔真的真的要有三窟 (reconstruct redeem events from REDIS)

17. Q&A
Thank you for your attention