This document discusses using Oracle Advanced Queueing (AQ) to build a centralized and scalable retailing platform. It provides an overview of using AQ for message-oriented middleware to allow asynchronous communication between a centralized database and distributed point-of-sale systems. The architecture allows the point-of-sale systems to continue operating independently if the central network connection is unavailable. Tuning efforts focused on reducing wait times for enqueueing and dequeueing messages to maintain responsive performance as the system scales.

1. Adventures in
Building a
Centralized and
Scalable
Retailing Platform
using
Advanced Queueing
Kurt Van Meerbeeck

2. 24/7 ABCSRPUAQ - AGENDA
Agenda
The Ego part
- All about AXI & me
The Business part
- Retailing and IT
The Infrastructure part
- Diving into the architecture
The Geeky part
- The adventures
Questions

3. The Ego Part
All about AXI & me

4. 24/7 INTRODUCTION – ALL ABOUT ME
Kurt Van Meerbeeck
Oracle DBA - AXI NV/BV
- Backup & recovery internals (jDUL/DUDE)
- Oracle IAS architectures
Working with
- Oracle related products since ’97
- Java since ‘96 (jdk 1.0.1)
kvmb@axi.be

5. 24/7 INTRODUCTION – ALL ABOUT AXI
AXI NV founded in 1970 – AXI BV in 1989
Long term Oracle partner (20+ years)
- Partner of the Year 2008 (The Netherlands)
Hitting all cilinders of the IT technology stack
CUSTOMERS
TECHNOLOGY PARTNERS
RETAIL HEALTH PUBLIC TRADE,
SERVICE
SHAREHOLDERS
& INDUSTRY
Sector software and software projects
Discovery Suite for financial and administrative management
24/7 Integrated Technology Services
ICT Systems - infrastructure
PERSONNEL

6. The business part
Retailing & IT

7. 24/7 RETAILING & IT –RETAIL BUSINESS IS A SCALABLE BUSINESS
Simple local store
Front-office – POS
Backoffice

8. 24/7 RETAILING & IT –RETAIL BUSINESS IS A SCALABLE BUSINESS
Expanding local store – scale up
Front-office – POS
Backoffice

9. 24/7 RETAILING & IT –RETAIL BUSINESS IS A SCALABLE SCALABLE
Expanding local store
Front-office – POS
Backoffice

10. 24/7 RETAILING & IT –RETAIL BUSINESS IS A SCALABLE BUSINESS
Typical POS solution
Decentralised
Scalable
However ...
Hard to manage
Backup/recovery
Failures
Software updates
Business Reporting
KPI
Replication
High TCO

11. 24/7 RETAILING & IT –RETAIL BUSINESS IS A SCALABLE BUSINESS
Imagine hundreds of stores
Using their own data silo’s
... Yet it is still the most
common store architecture

12. 24/7 RETAILING & IT –CHANGES IN THE IT LANDSCAPE
Trends in the IT landscape
- Consolidation & virtualisation
- Decentral to central computing to cloud computing
- Service Oriented Infrastructure
- Character-based to C/S to 3tier to grid
- Affordable communication lines
Trends in retailing
- Big players competing each other (Netherlands)
- Profit margins under pressure
- (near) real-time information needs

13. 24/7 RETAILING & IT –CHANGES IN THE IT LANDSCAPE
The retailing industry is catching-up !
And is moving towards centralised and integrated store
solutions
New challenges

14. 24/7 RETAILING & IT –RETAIL BUSINESS IS A SCALABLE BUSINESS
The obvious counterpart
Centralised datastore
POS
Backoffice
Solution needs to be
Highly scalable
Highly available
i.e. flexible

15. 24/7 RETAILING & IT – CUSTOMER CASE - INTERGAMMA
Case study – Intergamma
GAMMA & KARWEI stores
DIY market leader in the Benelux
Number of stores : 350
Number of POS : 1500
Number of backoffice users : 600
Number of portable scan devices : 1200

16. The Infrastructure part
Diving into the architecture

17. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
RS solution relies heavely on message-oriented middleware
(MOM)
Allows applications to connect by distributing messages
Typically built around a queueing infrastructure
- IBM MQSeries, MSMQ, Tibco, Oracle AQ
Decoupling in time
- Sender (producer) and receiver (consumer) do not need to interact with
the queue at the same time
Receive/store/send and keep track of messages
- guaranteed delivery

18. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Oracle Advanced Queueing (AQ)
Oracle’s implementation of message-oriented middleware
- But within a database
Persistent storage – IOT
Aynchronous communication
Q’s
IOT

19. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Oracle Advanced Queueing (AQ)
- Point-to-point
enqueue dequeue
dequeue enqueue
- Publish/subscribe – broadcast - multicast
subscribe
publish
subscribe
publish publish

20. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Oracle Advanced Queueing (AQ)
- Message Propagation
Fan-out
Funnel-in
Sqlnet (dblinks)

21. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Oracle features using Advanced Queueing (AQ)
- Oracle Streams
- CDC (change data capture)
enqueue dequeue enqueue
dequeue
transactions
parse transactions
Redo generation
archives

22. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Replication using Advanced Queueing (AQ)
- No log mining needed
- Optimize payload for network
Optimise for bandwidth
enqueue dequeue enqueue
dequeue
app
transactions
transactions
Redo generation

23. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Browser
Http Web-based
Management Node
sqlnet
Oracle Grid Control
Oracle IAS - MID
F5 BigIP
LB
Oracle IAS - MID
Windows Embedded
Oracle IAS - MID Private Oracle XE – AQ – WS - .Net
Oracle IAS - INF nework
Oracle 10/11g
RDBMS EE
Symbol HHTerminal
PocketBrowser - APEX

24. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
The obvious question is
What if the central site is unavailable ?
(network failure)
HQ
DR Store <n>
Frontoffice must be able to run stand-alone !
Selling of items must not stop !
Oracle 10g/11g EE RDBMS Oracle 10g XE RDBMS
HQ
AQ Store <n>
DR
sqlnet AQ

25. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
The obvious question is
What if the central site is unavailable?
(network failure)
HQ • Backoffice offline (http)
• HHT offline (http)
• POS Web Service offline (http)
DR • POS available Store <n>
• Oracle XE – stores all items and prices
• AQ stores messages until network is available
Oracle 10g/11g EE RDBMS Oracle 10g XE RDBMS
HQ
AQ Store <n>
DR
AQ

26. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Architectural choices - Scale-out vs scale-up
Oracle IAS - INF
Scale-up
Oracle RDBMS
Capacity-on-demand
Oracle RDBMS
- Add cpu’s
Oracle IAS - INF - Add memory
Oracle EM 10gR3
Provisioning pack
Scale-out : add nodes
Oracle RDBMS Oracle RDBMS Oracle RDBMS

27. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Architectural choices
8cpu/32Gb (on demand)
Scale-up - example
LPAR 0.5cpu/2Gb LPAR
IBM pSeries
- Hardware based virtualisation LPAR 0.5cpu/2Gb LPAR
- (Dynamic) LPAR
- Capacity-On-demand LPAR 1cpu/2Gb LPAR
- Oracle licenses (!)
- initial cost might be high
LPAR 2cpu/2Gb LPAR
LPAR 8cpu/16Gb LPAR

28. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Architectural choices
Enterprise hardware vs commodity hardware
- Commodity hardware does not equal cheap hardware
- Healthy mix
Oracle VM – adds a new dimention to scalable architectures
- scale up and out on commodity hardware almost transparently
- Allows hard partitioning
AS0 AS1 AS2
CPU1
4cores
AS0 AS1
RAC RAC RAC
CPU0 0 1 2
4cores
RAC RAC
0 1

29. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Testing AXI*RS on (ML 464754.1)
- Commodity hardware
- Oracle RDBMS 10g Real Application Cluster
- Oracle Enterprise Linux
- Oracle VM – Xen based virtualisation
- Oracle VM – supports RAC
- Linux based LB (VIPS+ldirector)
- Lower initial costs
Oracle Unbreakable Linux support program
- Enterprise-class support for the whole stack
Scale-out : add nodes
Oracle RDBMS Oracle RDBMS Oracle RDBMS
Scale-up : Oracle VM

30. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
Application will scale near-linear because of Advanced Queueing
- Bind queues to specific RAC nodes
Oracle RDBMS Oracle RDBMS Oracle RDBMS

31. 24/7 A DIVE INTO THE ARCHITECTURE – AXI RETAIL SOLUTIONS
The challenge
- Know when to scale – measure – capacity planning
- Be prepared - provisioning
- Make your solution scale with the hardware - tuning
Good tools
- Grid Control – Automatic Workload Repository – Active Session History
- Management – tuning/troubleshooting
- Hobbit (Big Brother)
- Alerting
- Easy customization Service Desk Automatisation
- Capacity Planner

32. 24/7 SCALING THE SYSTEM – KNOWING WHEN TO SCALE - ADVANCED QUEUEING
Measure, so you can manage
OS Queues : 01/11/2006 - 01/09/2008
35
30
25
20
15
Runqueue
10
5 Waitqueue
0
Paging : 01/11/2006 - 01/09/2008
450
400
350
300
250
200
150 PI
100 PO
50
0
Physical CPU's used by LPAR : 01/11/2006 - 01/09/2008
12
10
8
# CPUs
6
4
PHYS CPU
2
0

33. 24/7 SCALING THE SYSTEM – KNOWING WHEN TO SCALE - ADVANCED QUEUEING
Sessions : IGDBP 01/11/2006 00:00 - 01/09/2008 00:00
7000
6000
5000
4000
3000
Tot. Sessions
2000
1000 Act. Sessions
0
Waits ms/s : IGDBP 01/11/2006 00:00 - 01/09/2008 00:00
7000
6000
5000
4000
3000
2000 Waits ms/s
1000
0
logical IO/min : IGDBP 01/11/2006 00:00 - 01/09/2008 00:00
30000000
25000000
20000000
15000000
10000000
Logical IO/min
5000000
0
# Users: IGDBP 01/11/2006 00:00 - 01/09/2008 00:00
6000
5000
4000
3000
WAP
2000
IRSII_WAP_FO
1000
HTMLDB_PUBLIC_USER
0

34. The Geeky part
(the adventures)

35. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Store to central site
- R=W+S
- Response time = Wait time + Service time
Oracle Oracle
dedicated dedicated Oracle J0x
process process process POS App
S Wdq Wq Weq Wnet Wdq Wq Weq
10g EE RDBMS network 10g XE RDBMS
Response time

36. 24/7
R < 1min
0
200
400
600
800
1000
0:00 1200
0:56
1:52
2:48
3:44
4:40
5:36
6:32
7:28
8:24
9:20
10:16
11:12
12:08
13:04
14:00
14:56
15:52
16:48
17:44
18:40
19:36
20:32
THE ADVENTURES – TUNING RESPONSE TIME
21:28
22:24
23:20
enqueue/min
processed/min

37. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Wdq / Weq – time spend enqueueing/dequeueing
- Contention on queue table
- Waits on ITL slots (TX enq)
- Hot spots/blocks
Spread load over multiple queues
- Max 255 POS/queue
- Lowers arrival rate
- Lowers Wq
Increase ITL slots
- Initrans/maxtrans

38. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Wdq / Weq – time spend enqueueing/dequeueing
- More dequeueing/processing procs -> high OS runqueue
- Bind queue on RAC instance

39. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Wdq / Weq – time spend enqueueing/dequeueing
- Contention on AQ metadata tables
- TX enqueue locks on AQ$_PROPAGATION_STATUS
- Update AQ$_PROPAGATION_STATUS same record over and over
- Record identified by data objectid of remote queue (XE)
- Make sure objectid of all remote queues are unique (drop/recreate)
- RAC : alter table rebuild minimize records_per_block

40. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Wdq / Weq – time spend enqueueing/dequeueing
- QMON space management on ASSM tablespaces
- Issue propagating from central database to POS
- Manual coalesce/shrink IOT
- Serious impact on queue operations (LIO->CPU)

41. 24/7 SCALING THE SYSTEM – KNOWING WHEN TO SCALE - ADVANCED QUEUEING
Weq/Wdq/S - Logfile sync waits
- LGWR can lose CPU before it has exhausted its fair time slice
- Bind to CPU – or – renice – or – make non-preemptive
- Use RAC
Redo copy/allocation latch
commit
commit
Redo
commit Log
buffer
P/W
LFS wait
Semaphore lgwr
Queue
Redo logfile

42. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Tune service time - Partitioning – divide and conquer
- For managability – ILM
- Range partition / List subpartition
- Problem – timestamp not part of PK (global index from hell)
- 50 tables x 7y x 4Q x 350 stores = 490000 (sub)partitions
- SQL plan – partition iterator – impact LIO
- For performance – partition pruning – contention elimination (RAC)
- List partition on store number – part of PK – local partitioned indexes
- 50 tables x 350 stores = 17500 partitions – partition manager
- ILM
- SQL plan – partition pruning

43. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Oracle dedicated and shared server architecture
- Healthy mix shared and dedicated server
- 11g Database Resident Connection Pool (however with 10g XE?)
User process
SERVER=DEDICATED
Dedicated server SERVER=SHARED
Shared server Dispatcher

44. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Tuning service time - SQL Plan stability
- ‘we’ve changed nothing – the system is slow now’
Optimizer trends
- Oracle 7-8 – CBO gaining grounds – plan stability
- Oracle 8i
- Optimizer_index_cost_adj + optimizer_index_caching
- Default parallel query
- Oracle 9i
- CPU costing (dbms_stats.gather_system_stats)
- Oracle 10g
- automatic statistics (stale/ for all columns auto/for all columns repeat)
- bind peeking/histograms
Oracle 11g – SQL Plan Management
11g Intelligent Cursor Sharing

45. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Tuning service time - SQL Plan stability
- ‘we’ve changed nothing – the system is slow now’
Oracle 11g – SQL Plan Management (SPM)
Compile
Compile Execute
GB
NL
HJ Plan Acceptable
SQL log
Plan history
Plan baseline
GB
GB
NL GB
HJ

46. 24/7 THE ADVENTURES – TUNING RESPONSE TIME
Tuning service time - SQL Plan stability
- ‘we’ve changed nothing – the system is slow now’
Bind variables and partitioned tables
Table (store)
4
4
p
p
p
2
2
Partitions = Natural histogram
p3
p1
Bind variable is peeked on hard parse
Plan for p4 might not be ideal for p1
10g : disable bind peeking
11g : Adaptive Intelligent Cursor Sharing

47. 24/7
CTWR
Bctr.dbf
0
0,2
0,4
0,6
0,8
1
1,2
1,4
1,6
RMAN
01-11-2006 06
26-11-2006 18
16-12-2006 12
06-01-2007 00
25-01-2007 18
14-02-2007 12
06-03-2007 06
26-03-2007 00
14-04-2007 18
04-05-2007 12
24-05-2007 06
13-06-2007 00
02-07-2007 18
22-07-2007 12
11-08-2007 06
31-08-2007 00
20-09-2007 00
09-10-2007 18
29-10-2007 12
18-11-2007 06
backupset
08-12-2007 00
backupset
27-12-2007 18
- 10g Block Change Tracking (BCT)
16-01-2008 12
05-02-2008 06
25-02-2008 00 Database growth
- RMAN incremental updated backups
15-03-2008 18
04-04-2008 12
24-04-2008 06
14-05-2008 00
02-06-2008 18
22-06-2008 12
12-07-2008 06
01-08-2008 00
20-08-2008 18
THE ADVENTURES – TUNING RESPONSE TIME
- Apply incremental backupset to backup copy
RMAN
Tot.Size [Tb]
Tuning service time – make backup really fast

48. 24/7 CUSTOMER RESPONSE
In the end ...
Simon Vreeke, CTO, IG
[ AXI*RS offers our stores a 100% available solution – able to process all
our transactions and more. For our customers, the process is quick,
correct and secure...]
PlusRetail

49. Questions
www.axi.be
www.axi.nl