Kanggil Lee is a Senior Software Engineer at Samsung Electronics, managing Perforce servers for mobile communications. He oversees the deployment and optimization of Perforce across Samsung's global R&D centers, dealing with significant user numbers and replication challenges. Upgrade efforts have been made to enhance performance and reduce replication lag, alongside implementing extensive monitoring and maintenance strategies.

1. #
Kanggil Lee
Senior Software Engineer
Samsung Electronics

2. #
Kanggil Lee is the senior S/W Engineer in Samsung Electronics.
He administrates ALM systems, especially Perforce servers in
Mobile Communications Business.
He is in charge of deploying Perforce to globally distributed
Samsung's R&D centers as well as HQ. He managed to
configure the world largest transactional and 24/7 sleepless
Perforce server. Before joining Perforce team, he used to work
as an administrator of IBM Rational products in Samsung.

3. #
• Perforce at Samsung
• Optimizing Perforce Replication
• Lockless Reads
• Monitoring and Maintenance
• Future

4. #

5. #
• All mobile projects are in Perforce, and most are
android platform.
• We have 15 master servers and provides 19
perforce services.
• ~30 overseas R&D centers.
• Most of our users use P4V. ( > 80%)

6. #
• Primary Server(Android 4.4.x ~)
• 4 CPUs(physically 32cores), 1.5TB Memory, Linux(RHEL)
• Metadata, Journal and Logs on Flash Arrays
• Depot on Spinning disk(15k, RAID1+0, DAS)
• > 7TB metadata(reclaim space up to 500GB/week)
• > 10,000 users
• ~ 2.6 million submitted changelists since Nov. 2013
• ~ 8 million commands/day
• ~ 6.5k commits/day

7. #
• Other Perforce Servers( ~Android 4.3.x, etc.)
• ~ 14 smaller servers, with between 10GB-3TB metadata
each
• Metadata, Journal and Logs on Flash Arrays or Spinning
disk(15k, RAID1+0, DAS)
• Depot on Spinning disk(15k, RAID1+0, DAS)
• 5 Read-Only Replicas
• > 30 Build-Farm Replicas worldwide
• 90 proxies worldwide

8. #

9. #
• ~ 5 hours replication lag on overseas build replicas
– ~ 25GB journal / hour on our primary server
– ~ 500ms latency (HQ <-> Brazil)
• What we have done
– Filter db.have by using –T flag
• db.have is over 90% of each journal
– Add QoS rules for perforce traffic
– Expand network bandwidth
– Set net.keep.x variables

10. #
• ~ 35 mins replication lag on all build replicas
– ~ 4GB journal / each p4 populate command
– Break builds(Proof/Release)
• What we have done
– Set rpl=2 in order to profile database locking activity
– Filter db.integed by using –T flag
• Most lock time is due to holding write-lock on db.integed
• < 30s (70x faster)

11. #

12. #
• H/W upgrade is essential.
– Flash Arrays, Fast CPU, Huge Memory, 10G NIC etc.
• However, It can Not guarantee best performance
all the time due to lock contention.
– Sync commands block major write commands
• dm-CommitSubmit, shelve, unshelve, submit, edit, populate
– Integ commands block more write commands
• dm-SubmitChange, revert, shelve, dm-CommitSubmit,
change, submit, edit, resolve, reopen, add, sync, delete

13. #
• Upgraded 2013.3 and set db.peeking=3
– Upgraded all replica servers in April.
• Reduce replication lag
– Upgraded our primary server on Jun. 29th
• Eliminate lock contention caused by sync and integ
commands.
• Shows significant performance improvements

14. #
130,000
120,000
110,000
100,000
90,000
80,000
70,000
60,000
50,000
40,000
30,000
20,000
10,000
0
1 2 4 8 16 32 64 128 256 512
Commands affected
Commands delayed
Feb.
April
July
(sec)

15. #
• Top 20 commands which blocked other commands
on the primary server for the last 30days.

16. #
700
650
600
550
500
450
400
350
300
250
200
150
100
50
0
Feb. JUN JULY
time(ms)
P4 DB Write lock Activity
write-wait
write-held
100
95
90
85
80
75
70
65
60
55
50
Feb. JUN JULY
%
Command ratio (lapse < 1s)
submit

17. #
13
12.5
12
11.5
11
10.5
10
9.5
9
8.5
8
7.5
7
6.5
6
5.5
5
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
Feb. JUN JULY
time(sec)
Avg. Lapse
submit
shelve
unshelve
edit
revert

18. #

19. #
• Run P4HealthCheck every 60 seconds
– check connection, p4d status, p4broker status
– send email and SMS if status is not ok.
• Run Replica Gap checker every 10 minutes
– check replication lag
– send email if gap is over 10 minutes.

20. #
• Monitoring Perforce Severs

21. #
• Monitoring Perforce DB Lock

22. #
• Profiling locking activities
– run a query and select any fields you are interested in

23. #
• Monitoring Perforce Commands

24. #
• Monitoring Perforce Sync command

25. #
• Monitoring Errors(max-values)

26. #
• Maintain perforce db.* files, journal and logs
– create and restore checkpoint files/week
– db rebuild/week
– unload clients and labels/day
– rotate and replay journals(logs)/hour
– recover the database
• Set up Replica servers
• Verify depots/day

27. #

28. #

29. #

30. #
• Upgrade our primary server in October
– 4 CPUs(physically 40cores), 3.0TB memory
• P4d/P4broker upgrades (r13.3 -> r14.x)
• Shared Archive or Depot on Flash Arrays(dedup)
• Server consolidation
– 14 other servers -> 7 other servers
– Setup read-only replicas for all perforce services

31. #
Kanggil Lee
kanggil.lee@samsung.com