1. Solaris/Linux Performance
Measurement, Tools and Tuning
Adrian Cockcroft, acockcroft@netflix.com
May 1, 2009
2009 5/1/09 Page 1

2. Abstract
• This course focuses on the measurement sources and tuning
parameters available in Unix and Linux, including TCP/IP
measurement and tuning, workload analysis, complex storage
subsystems, and with a deep dive on advanced Solaris metrics
such as microstates and extended system accounting.
• The meaning and behavior of metrics is covered in detail.
Common fallacies, misleading indicators, sources of
measurement error and other traps for the unwary will be
exposed.
• Free tools for Capacity Planning are covered in detail in a
different slide deck, interleaved for this event.
• Updated slide decks live at http://www.slideshare.net/adrianco
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 2

3. Sources
• Adrian Cockcroft
– Sun Microsystems 1988-2004, Distinguished Engineer
– eBay Research Labs 2004-2007, Distinguished Engineer
– Netflix 2007, Director - Web Engineering – Personalization Systems
– CMG 2007 Michelson Award Winner for lifetime contribution to
computer measurement
– Note: I am a Netflix employee, but this material does not refer to and
is not endorsed by Netflix. It is based on the author's work over the
last 20+ years.
• Books by the author
– Sun Performance and Tuning, Prentice Hall, 1994, 1998 (2nd Ed)
– Resource Management, Prentice Hall, 2000
– Capacity Planning for Internet Services, Prentice Hall, 2001
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 3

4. Contents
• Capacity Planning and Performance Definitions
• Workload Characteristics and Analysis
• Implications of Virtualization and Cloud Computing
• Metric collection interfaces
• Free Tools for capacity planning (separate slide deck)
• CPU - measurement issues and virtualization
• Network - Internet Servers and TCP/IP essentials
• Memory – The memory-go-round, Swap space instrumentation
• Disks - virtualization, SSDs, filesystems, simple disks and RAID
• Quick tips and Recipes
• References
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 4

5. Definitions
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 5

6. Capacity Planning Definitions
• Capacity
– Resource utilization and headroom
• Planning
– Predicting future needs by analyzing historical data and
modeling future scenarios
• Performance Monitoring
– Collecting and reporting on performance data
• Unix/Linux (apologies to users of OSX, HP-UX, AIX etc.)
– Emphasis on Solaris and Linux
– Much of the discussion is independent of the OS
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 6

7. Measurement Terms and Definitions
• Bandwidth - gross work per unit time [unattainable]
• Throughput - net work per unit time
• Peak throughput - at maximum acceptable response time
• Utilization - busy time relative to elapsed time [can be misleading]
• Queue length - number of requests waiting
• Service time - time to process a unit of work after waiting
• Response time - time to complete a unit of work including waiting
• Key Performance Indicator (KPI) – a measurement you have
decided to watch because it has some business value
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 7

8. Service Level Agreements (SLA)
• Behavioral goals for the system in terms of KPIs
• Response time target
– Rule of thumb: Estimate 95th percentile response time as
three times mean response time
– e.g. if SLA says 1 second response, measured average
should be less than 333ms
• Utilization Target (a proxy for Response Time)
– Specified as a minimum and maximum
– Minimum utilization target to keep costs down
– Maximum utilization target for good response times and
capacity headroom for future workload fluctuations
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 8

9. Capacity Planning Requirements
• We care about CPU, Memory, Network and Disk resources, and
Application response times
• We need to know how much of each resource we are using
now, and will use in the future
• We need to know how much headroom we have to handle
higher loads
• We want to understand how headroom varies, and how it relates
to application response times and throughput
• The application workload must be characterized so we can
understand and manage system behaviours
• We want to be able to find the bottleneck in an under-performing
system
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 9

10. Workloads
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 10

11. Workload Characteristics: One by One
Constant Workloads
• e.g. Numerical computation, compute intensive batch
• Trivial to model, utilization and duration define the work
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 11

12. Simple Random Arrivals
• Random arrival of transactions with fixed mean service time
– Little’s Law: QueueLength = Throughput * Response
– Utilization Law: Utilization = Throughput * ServiceTime
• Complex models are often reduced to this model
– By averaging over longer time periods since the formulas only
work if you have stable averages
– By wishful thinking (i.e. how to fool yourself)
• e.g. Unix Load Average is actually CPU Queue Length
– Throughput up a little, load average up a lot = slow system
– So load average is a proxy metric for response time
– High load average per CPU implies slow response times
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 12

13. Mixed random arrivals of transactions
with stable mean service times
• Think of the grocery store checkout analogy
– Trolleys full of shopping vs. baskets full of shopping
– Baskets are quick to service, but get stuck behind trolleys
– Relative mixture of transaction types starts to matter
• Many transactional systems handle a mixture
– Databases, web services
• Consider separating fast and slow transactions
– So that we have a “10 items or less” line just for baskets
– Separate pools of servers for different services
– Don’t mix OLTP with DSS queries in databases
• Performance is often thread-limited
– Thread limit and slow transactions constrains maximum throughput
– Throughput = Queue / ResponseTime
• Model using analytical solvers like PDQ
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 13

14. Load dependent servers – non-stable
mean service times
• Mean service time increases at high throughput
– Due to non-scalable algorithms, lock contention
– System runs out of memory and starts paging or frequent GC
• Systems have “tipping points”
– Hysteresis means they don’t come back when load drops
– This is why you have to kill catatonic systems
• Model using simulation tools like Hyperformix, Opnet
– Behaviour is non-linear and hard to model
– Practical option is to avoid tipping points
– Best designs shed load to be stable at the limit
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 14

15. Self-similar / fractal workloads – bursty
rather than random
• Self-similar
– Looks “random” at close up, stays “random” as you zoom out
– Work arrives in bursts, transactions aren’t independent
– Bursts cluster together in super-bursts, etc.
• Network packet streams tend to be fractal
• Common in practice, too hard to model
– Probably the most common reason why your model is wrong!
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 15

16. State Dependent Services
• Personalized services that store user history
– Transactions for new users are quick
– Transactions for users with lots of state/history are slower
– As user base builds state and ages you get into lots of
trouble…
• Social Networks, Recommendation Services
– Facebook, Flickr, Netflix, Pandora, Twitter etc.
• “Abandon hope all ye who enter here”
– Not tractable to model, repeatable tests are tricky
– Long fat tail response time distribution and timeouts
– Excessively long service times for some users
– Solutions: careful algorithm design, lots of caching
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 16

17. Workload Modelling Survivalism
• Simplify the workload algorithms
– move from hard or impossible to simpler models
– use caching and pre-compute to get constant service times
• Stand further away
– averaging is your friend – gets rid of complex fluctuations
• Minimalist Models
– most models are far too complex – the classic beginners error…
– the art of modelling is to only model what really matters
• Don’t model details you don’t use
– model peak hour of the week, not day to day fluctuations
– e.g. “Will the web site survive next Sunday night?”
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 17

18. Metrics
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 18

19. Measurement Data Interfaces
• Several generic raw access methods
– Read the kernel directly
– Structured system data
– Process data
– Network data
– Accounting data
– Application data
• Command based data interfaces
– Scrape data from vmstat, iostat, netstat, sar, ps
– Higher overhead, lower resolution, missing metrics
• Data available is always platform and release specific…
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 19

20. Reading kernel memory - kvm
• The only way to get data in very old Unix variants
• Use kernel namelist symbol table and open /dev/kmem
• Solaris wraps up interface in kvm library
• Advantages
– Still the only way to get at some kinds of data
– Low overhead, fast bulk data capture
• Disadvantages
– Too much intimate implementation detail exposed
– No locking protection to ensure consistent data
– Highly non-portable, unstable over releases and patches
– Tools break when kernel moves between 32 and 64bit address
support
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 20

21. Structured Kernel Statistics - kstat
• Solaris 2 introduced kstat and extended usage in each release
• Used by Solaris 2 vmstat, iostat, sar, network interface stats, etc.
• Advantages
– The recommended and supported Solaris metric access API
– Does not require setuid root commands to access for reads
– Individual named metrics stable over releases
– Consistent data using locking, but low overhead
– Unchanged when kernel moves to 64bit address support
– Extensible to add metrics without breaking existing code
• Disadvantages
– Somewhat complex hierarchical kstat_chain structure
– State changes (device online/offline) cause kstat_chain rebuild
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 21

22. Kernel Trace - TNF, Dtrace, ktrace
• Solaris, Linux, Windows and other Unixes have similar features
– Solaris has TNF probes and prex command to control them
– User level probe library for hires tracepoints allows
instrumentation of multithreaded applications
– Kernel level probes allow disk I/O and scheduler tracing
• Advantages
– Low overhead, microsecond resolution
– I/O trace capability is extremely useful
• Disadvantages
– Too much data to process with simple tracing capabilities
– Trace buffer can overflow or cause locking issues
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 22

23. Dtrace – Dynamic Tracing
• One of the most exiting new features in Solaris 10, rave reviews
• Book: quot;Solaris Performance and Toolsquot; by Richard McDougall
and Brendan Gregg
• Advantages
– No overhead when it is not in use
– Low overhead probes can be put anywhere/everywhere
– Trace data is correlated and filtered at source, get exactly the
data you want, very sophisticated data providers included
– Bundled, supported, designed to be safe for production
systems
• Disadvantages
– Solaris specific, but being ported to BSD/Linux
– No high level tools support yet
– Yet another (awk-like) scripting language to learn
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 23

24. Hardware counters
• Solaris cpustat for X86 and UltraSPARC pipeline and cache counters
• Solaris busstat for server backplanes and I/O buses, corestat for
multi-core systems
• Intel Trace Collector, Vampir for Linux
• Most modern CPUs and systems have counters
• Advantages
– See what is really happening, more accurate than kernel stats
– Cache usage useful for tuning code algorithms
– Pipeline usage useful for HPC tuning for megaflops
– Backplane and memory bank usage useful for database servers
• Disadvantages
– Raw data is confusing, lots of architectural background info
needed
– Most tools focus on developer code tuning
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 24

25. Configuration information
• System configuration data comes from too many sources!
– Solaris device tree displayed by prtconf and prtdiag
– Solaris 8 adds dynamic configuration notification device picld
– SCSI device info using iostat -E in Solaris
– Logical volume info from product specific vxprint and metastat
– Hardware RAID info from product specific tools
– Critical storage config info must be accessed over ethernet…
– Linux device tree in /proc is a bit easier to navigate
• It is very hard to combine all this data!
• DMTF CIM objects try to address this, but no-one seems to use them…
• Free tool - Config Engine: http://www.cfengine.org
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 25

26. Application instrumentation Examples
• Oracle V$ Tables – detailed metrics used by many tools
• Apache logging for web services
• ARM standard instrumentation
• Custom do-it-yourself and log file scraping
• Advantages
– Focussed application specific information
– Business metrics are needed to do real capacity planning
• Disadvantages
– No common access methods
– ARM is a collection interface only, vendor specific tools, data
– Very few applications are instrumented, even fewer have support
from performance tools vendors
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 26

27. Kernel values, tunables and defaults
• There is often far too much emphasis on kernel tweaks
– There really are few “magic bullet” tunables
– It rarely makes a significant difference
• Fix the system configuration or tune the application instead!
• Very few adjustable components
– “No user serviceable parts inside”
– But Unix has so much history people think it is like a 70’s car
– Solaris really is dynamic, adaptive and self-tuning
– Most other “traditional Unix” tunables are just advisory limits
– Tweaks may be workarounds for bugs/problems
– Patch or OS release removes the problem - remove the tweak
Solaris Tunable Parameters Reference Manual (if you must…)
– http://docs.sun.com/app/docs/doc/817-0404
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 27

28. Process based data - /proc
• Used by ps, proctool and debuggers, pea.se, proc(1) tools on Solaris
• Solaris and Linux both have /proc/pid/metric hierarchy
• Linux also includes system information in /proc rather than kstat
• Advantages
– The recommended and supported process access API
– Metric data structures reasonably stable over releases
– Consistent data using locking
– Solaris microstate data provides accurate process state timers
• Disadvantages
– High overhead for open/read/close for every process
– Linux reports data as ascii text, Solaris as binary structures
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 28

29. Network protocol data
• Based on a streams module interface in Solaris
• Solaris 2 ndd interface used to configure protocols and interfaces
• Solaris 2 mib interface used by netstat -s and snmpd to get TCP stats etc.
• Advantages
– Individual named metrics reasonably stable over releases
– Consistent data using locking
– Extensible to add metrics without breaking existing code
– Solaris ndd can retune TCP online without reboot
– System data is often also made available via SNMP prototcol
• Disadvantages
– Underlying API is not supported, SNMP access is preferred
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 29

30. Tracing and profiling
• Tracing Tools
– truss - shows system calls made by a process
– sotruss / apitrace - shows shared library calls
– prex - controls TNF tracing for user and kernel code
– snoop/tcpdump – network traces for analysis with wireshark
• Profiling Tools
– Compiler profile feedback using -xprofile=collect and use
– Sampled profile relink using -p and prof/gprof
– Function call tree profile recompile using -pg and gprof
– Shared library call profiling setenv LD_PROFILE and gprof
• Accurate CPU timing for process using /usr/proc/bin/ptime
• Microstate process information using pea.se and pw.se
10:40:16 name lwmx pid ppid uid usr% sys% wait% chld% size rss pf
nis_cachemgr 5 176 1 0 1.40 0.19 0.00 0.00 16320 11584 0.0
jre 1 17255 3184 5743 11.80 0.19 0.00 0.00 178112 110336 0.0
sendmail 1 16751 1 0 1.01 0.43 0.00 0.43 18624 16384 0.0
se.sparc.5.6 1 16741 1186 9506 5.90 0.47 0.00 0.00 16320 14976 0.0
imapd 1 16366 198 5710 6.88 1.09 1.02 0.00 34048 29888 0.1
dtmail 10 16364 9070 5710 0.75 1.12 0.00 0.00 102144 94400 0.0
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 30

31. Free Tools
(See Separate Slide Deck)
http://www.slideshare.net/adrianco
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 31

32. Headroom
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 32

33. What would you say if you were asked:
How busy is that system?
A: I have no idea…
A: 10%
A: Why do you want to know?
A: I’m sorry, you don’t understand your question….
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 33

34. Headroom Estimation
• CPU Capacity
– Relatively easy to figure out
• Network Usage
– Use bytes not packets/s
• Memory Capacity
– Tricky - easier in Solaris 8
• Disk Capacity
– Can be very complex
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 34

35. Headroom
• Headroom is available usable resources
– Total Capacity minus Peak Utilization and Margin
– Applies to usr+sysRAM, Net, Disk and OS
CPU, CPU for Peak Period
100
Margin
90
80
Headroom
70
60
CPU %
50
40 Utilization
30
20
10
0
Time
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 35

36. Utilization
• Utilization is the proportion of busy time
• Always defined over a time interval
OnCPU Scheduling for Each CPU
Mean CPU Util
OnCPU and
0.56
usr+sys CPU for Peak Period
100
0
90
80 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
70
Microseconds
60
CPU %
50
40
Utilization
30
20
10
0
Time
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 36

37. Response Time
• Response Time = Queue time + Service time
• The Usual Assumptions…
– Steady state averages
– Random arrivals
– Constant service time
– M servers processing the same queue
• Approximations
– Queue length = Throughput * Response Time (Little's Law)
– Utilzation = Throughput * Service Time (utilization law)
– Response Time = Service Time / (1 - UtilizationM)
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 37

38. Response Time Curves
The traditional view of Utilization as a proxy for response time
Systems with many CPUs can run at higher utilization levels, but degrade more
rapidly when they run out of capacity
Headroom margin should be set according to a response time target.
Response Time Curves R = S / (1 - (U%)m)
10.00
Response Time Increase Factor
9.00
8.00
One CPU
7.00
Two CPUs
6.00
Four CPUs
5.00 Eight CPUs
Headroom 16 CPUs
4.00
margin 32 CPUs
3.00 64 CPUs
2.00
1.00
0.00
0 10 20 30 40 50 60 70 80 90 100
Total System Utilization %
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 38

39. So what's the problem with Utilization?
• Unsafe assumptions! Complex adaptive systems are not simple!
• Random arrivals?
– Bursty traffic with long tail arrival rate distribution
• Constant service time?
– Variable clock rate CPUs, inverse load dependent service time
– Complex transactions, request and response dependent
• M servers processing the same queue?
– Virtual servers with varying non-integral concurrency
– Non-identical servers or CPUs, Hyperthreading, Multicore, NUMA
• Measurement Errors?
– Mechanisms with built in bias, e.g. sampling from the scheduler clock
– Platform and release specific systemic changes in accounting of interrupt time
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 39

40. Variable Clock Rate CPUs
• Laptop and other low power devices do this all the time
– Watch CPU usage of a video application and toggle mains/battery power….
• Server CPU Power Optimization - AMD PowerNow!™
– AMD Opteron server CPU detects overall utilization and reduces clock rate
– Actual speeds vary, but for example could reduce from 2.6GHz to 1.2GHz
– Changes are not understood or reported by operating system metrics
– Speed changes can occur every few milliseconds (thermal shock issues)
– Dual core speed varies per socket, Quad core varies per core
– Quad core can dynamically stop entire cores to save power
• Possible scenario:
– You estimate 20% utilization at 2.6GHz
– You see 45% reported in practice (at 1.2GHz)
– Load doubles, reported utilization drops to 40% (at 2.6GHz)
– Actual mapping of utilization to clock rate is unknown at this point
• Note: Older and quot;low powerquot; Opterons used in blades fix clock rate
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 40

41. Virtual Machine Monitors
• VMware, Xen, IBM LPARs etc.
– Non-integral and non-constant fractions of a machine
– Naiive operating systems and applications that don't expect this
behavior
– However, lots of recent tools development from vendors
• Average CPU count must be reported for each measurement
interval
• VMM overhead varies, application scaling characteristics may
be affected
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 41

42. Threaded CPU Pipelines
• CPU microarchitecture optimizations
– Extra register sets working with one execution pipeline
– When the CPU stalls on a memory read, it switches registers/threads
– Operating system sees multiple schedulable entities (CPUs)
• Intel Hyperthreading
– Each CPU core has an extra thread to use spare cycles
– Typical benefit is 20%, so total capacity is 1.2 CPUs
– I.e. Second thread much slower when first thread is busy
– Hyperthreading aware optimizations in recent operating systems
• Sun “CoolThreads”
– quot;Niagaraquot; SPARC CPU has eight cores, one shared floating point unit
– Each CPU core has four threads, but each core is a very simple design
– Behaves like 32 slow CPUs for integer, snail like uniprocessor for FP
– Overall throughput is very high, performance per watt is exceptional
– Niagara 2 has dedicated FPU and 8 threads per core (total 64 threads)
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 42

43. Measurement Errors
• Mechanisms with built in bias
– e.g. sampling from the scheduler clock underestimates CPU usage
– Solaris 9 and before, Linux, AIX, HP-UX “sampled CPU time”
– Solaris 10 and HP-UX “measured CPU time” far more accurate
– Solaris microstate process accounting always accurate but in Solaris 10
microstates are also used to generate system-wide CPU
• Accounting of interrupt time
– Platform and release specific systemic changes
– Solaris 8 - sampled interrupt time spread over usr/sys/idle
– Solaris 9 - sampled interrupt time accumulated into sys only
– Solaris 10 - accurate interrupt time spread over usr/sys/idle
– Solaris 10 Update 1 - accurate interrupt time in sys only
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 43

44. CPU time measurements
• Biased sample CPU measurements
– See 1998 Paper quot;Unix CPU Time Measurement Errorsquot;
– Microstate measurements are accurate, but are platform and tool specific.
Sampled metrics are more inaccurate at low utilization
• CPU time is sampled by the 100Hz clock interrupt
– sampling theory says this is accurate for an unbiased sample
– the sample is very biased, as the clock also schedules the CPU
– daemons that wakeup on the clock timer can hide in the gaps
– problem gets worse as the CPU gets faster
• Increase clock interrupt rate? (Solaris)
– set hires_tick=1 sets rate to 1000Hz, good for realtime wakeups
– harder to hide CPU usage, but slightly higher overhead
• Use measured CPU time at per-process level
– microstate accounting takes timestamp on each state change
– very accurate and also provides extra information
– still doesn’t allow for interrupt overhead
– Prstat -m and the pea.se command uses this accurate measurement
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 44

45. More CPU Measurement Issues
• Load average differences
– Just includes CPU queue (Solaris)
– Includes CPU and Disk (Linux) – which is a broken metric
• Wait for I/O is a misleading subset of idle time
– Metric removed in Solaris 10 – always zero
– Ignore it in all other Unix/Linux releases
– Only makes sense on uni-processor systems
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 45

46. How to plot Headroom
• Measure and report absolute CPU power if you can get it…
• Plot shows headroom in blue, margin in red, total power tracking day/
night workload variation, plotted as mean + two standard deviations.
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 46

47. “Cockcroft Headroom Plot”
• Scatter plot of response time
(ms) vs. Throughput (KB) from
iostat metrics
• Histograms on axes
• Throughput time series plot
• Shows distributions and shape
of response time
• Fits throughput weighted
inverse gaussian curve
• Coded using quot;Rquot; statistics
package
• Blogged development at
http://perfcap.blogspot.com/search?q=chp
2009 Solaris/Linux Performance Measurement and Tuning 5/1/09 Slide 47

48.