Samsung drives and sees success within the smart phone world. Perforce is their platform of choice for Continuous Delivery for their world-wide development teams because of it's scalability and distributed set-up. Learn all about the software engineering environment behind Samsung's cutting edge smart phone products.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Supporting Android-based Platform Development in Samsung
1. #
Kanggil Lee
Senior Software Engineer
Samsung Electronics
2. #
Kanggil Lee is the senior S/W Engineer in Samsung Electronics.
He administrates ALM systems, especially Perforce servers in
Mobile Communications Business.
He is in charge of deploying Perforce to globally distributed
Samsung's R&D centers as well as HQ. He managed to
configure the world largest transactional and 24/7 sleepless
Perforce server. Before joining Perforce team, he used to work
as an administrator of IBM Rational products in Samsung.
3. #
• Perforce at Samsung
• Optimizing Perforce Replication
• Lockless Reads
• Monitoring and Maintenance
• Future
5. #
• All mobile projects are in Perforce, and most are
android platform.
• We have 15 master servers and provides 19
perforce services.
• ~30 overseas R&D centers.
• Most of our users use P4V. ( > 80%)
6. #
• Primary Server(Android 4.4.x ~)
• 4 CPUs(physically 32cores), 1.5TB Memory, Linux(RHEL)
• Metadata, Journal and Logs on Flash Arrays
• Depot on Spinning disk(15k, RAID1+0, DAS)
• > 7TB metadata(reclaim space up to 500GB/week)
• > 10,000 users
• ~ 2.6 million submitted changelists since Nov. 2013
• ~ 8 million commands/day
• ~ 6.5k commits/day
7. #
• Other Perforce Servers( ~Android 4.3.x, etc.)
• ~ 14 smaller servers, with between 10GB-3TB metadata
each
• Metadata, Journal and Logs on Flash Arrays or Spinning
disk(15k, RAID1+0, DAS)
• Depot on Spinning disk(15k, RAID1+0, DAS)
• 5 Read-Only Replicas
• > 30 Build-Farm Replicas worldwide
• 90 proxies worldwide
9. #
• ~ 5 hours replication lag on overseas build replicas
– ~ 25GB journal / hour on our primary server
– ~ 500ms latency (HQ <-> Brazil)
• What we have done
– Filter db.have by using –T flag
• db.have is over 90% of each journal
– Add QoS rules for perforce traffic
– Expand network bandwidth
– Set net.keep.x variables
10. #
• ~ 35 mins replication lag on all build replicas
– ~ 4GB journal / each p4 populate command
– Break builds(Proof/Release)
• What we have done
– Set rpl=2 in order to profile database locking activity
– Filter db.integed by using –T flag
• Most lock time is due to holding write-lock on db.integed
• < 30s (70x faster)
12. #
• H/W upgrade is essential.
– Flash Arrays, Fast CPU, Huge Memory, 10G NIC etc.
• However, It can Not guarantee best performance
all the time due to lock contention.
– Sync commands block major write commands
• dm-CommitSubmit, shelve, unshelve, submit, edit, populate
– Integ commands block more write commands
• dm-SubmitChange, revert, shelve, dm-CommitSubmit,
change, submit, edit, resolve, reopen, add, sync, delete
13. #
• Upgraded 2013.3 and set db.peeking=3
– Upgraded all replica servers in April.
• Reduce replication lag
– Upgraded our primary server on Jun. 29th
• Eliminate lock contention caused by sync and integ
commands.
• Shows significant performance improvements
19. #
• Run P4HealthCheck every 60 seconds
– check connection, p4d status, p4broker status
– send email and SMS if status is not ok.
• Run Replica Gap checker every 10 minutes
– check replication lag
– send email if gap is over 10 minutes.
30. #
• Upgrade our primary server in October
– 4 CPUs(physically 40cores), 3.0TB memory
• P4d/P4broker upgrades (r13.3 -> r14.x)
• Shared Archive or Depot on Flash Arrays(dedup)
• Server consolidation
– 14 other servers -> 7 other servers
– Setup read-only replicas for all perforce services
Write waiting 시간 1532% 개선 (247.01ms -> 15.13ms)
Write held 시간 3318% 개선 (586.01ms -> 17.14ms)
Submit 커맨드의 97%가 1초 이내 처리 완료
다른 커맨드도 1초 이내에 완료 된 커맨드 비율이 5~20% 증가하였음
Submit 처리 시간은 1136% 개선(3.09s -> 0.25s)
Shelve 처리 시간은 87% 개선(3.77s -> 2.02s)
Unshelve 처리 시간은 84% 개선(5.51s -> 2.99s)
Edit 처리 시간은 279% 개선 (1.44s -> 0.38s)
Revert 처리 시간은 68% 개선 (5.16s -> 3.07s)
Submit 처리 시간은 1136% 개선(3.09s -> 0.25s)
Shelve 처리 시간은 87% 개선(3.77s -> 2.02s)
Unshelve 처리 시간은 84% 개선(5.51s -> 2.99s)
Edit 처리 시간은 279% 개선 (1.44s -> 0.38s)
Revert 처리 시간은 68% 개선 (5.16s -> 3.07s)