The document outlines Suning's evolution as a leading commercial enterprise in China, highlighting its diverse business operations, including retail and cloud services. It details Suning's cloud journey, challenges faced in deployment and orchestration, and the technical infrastructure supporting its cloud services. The document emphasizes the need for integrated tools for runtime orchestration and management while addressing complexity in resource allocation and application scalability.

1. Xiaobin Zhang, zhangxbk@cnsuning.com
Long Jin, jinlongb@cnsuning.com
Qiming Teng, tengqim@cn.ibm.com

2. • Suning Overview
• Suning OpenStack Journey
• Suning Cloud Workload
• Lessons Learnt
• Wishlists
2

3. • Basic Information
• Established in 1990
• The largest commercial enterprise in China
• Top 3 Chinese private enterprises
• The 50th among the Top 500 enterprises in China
• Business Lines
• retail, logistics, supply chain, real estates, investments, ...
• By the end of 2012
• Suning has stores in 700+ cities in China and other countries.
• The total number of staff is 180,000
• 4 R&D centers: Beijing, Shanghai, Nanjing, Silicon Valley
• Brand value of $ 13 B, annual revenue $ 37 B
3
$ 175 Billion
43.9% YTY Growth
1H 2014, China

4. • Opportunities
• Improve Efficiency, Collaboration Paradigms and Business Models
• Traditional e-Commerce  O2O (Online-to-Offline)  Cloudified Whole Value Chain
4
PERSONALIZED
SHOPPING
EXPERIENCES
EFFICIENT
MERCHANDISING AND
SUPPLY NETWORK
TRANSFORM AND
OPTIMIZE OPERATIONS
OPERATING
EFFICIENCY
REVENUE
GROWTH
ECOSYSTEM
DOMINATION

5. Suning Private Cloud
• Multiple Data Centers
• 1000s of Hosts
• 10x1000s of Virtual Machines
• Rich & Customized Middleware
• Automated Deployment / Operation
• Workflow Consolidation
5
Suning Public Cloud
• Cloud Server
• VPC
• Shared, Object Storage
• Cloud Database
• Fast Deployment
• Monitoring and Billing

6. • The journey starts since early 2013
• single deployment -> multi-region deployment across data centers
• R & D workloads testing -> Internet/production workload
6
Domain Status
Compute • 256 GB memory, 4 GB NICs, 64 cores; Windows/Linux guests
Network • Isolated network for admin, data and storage; OVS bonding; HW LB
Storage • LVM and GlusterFS resource pool with QoS support; Cinder multi-backend
Container • Docker resource pool and docker repo with HA enabled
Deployment • Cobbler, Puppet
Management • 3 nodes HA setup for controllers; RabbitMQ cluster
Monitoring • Proprietary resource/service monitoring tools, guest agents for data collection
• RabbitMQ portal and LogStash
Optimization • Resource scheduling for standalone, clustered and layered applications

7. • 100+ applications of diverse characteristics
• Mixed CPU-intensive and I/O Intensive workload:
• CPU-intensive, long-hour duration mobile application compilation and building
• huge storage and volume (800G ~ 1T)
• search engine compilation
• big data analytics, e.g. sentiment analysis
• thumbnail generation
• Different software stacks for Internet applications
• Apache + JBoss + MySQL
• IHS + WAS + DB2
• Others
7

8. 8
Web /
Frond-End
AppServer /
Middle-Tier
Database /
Back-End
• Optionally Clustering
• Optionally Auto-Scaling
• Dispatch to different hosts,
regions, networks, ...
• Optionally Clustering
• Optionally Auto-Scaling
• Schedule to different hosts,
regions, networks, ...
• Short upgrade cycle: 1-4
weeks (not whole system)
• Optionally Active/Passive
• Dispatch to different hosts,
networks, regions, ...
Dynamic Discovery
Live Registration
Request Granularity
Dynamic Discovery
Live Registration
Transaction Dispatching

9. • A component/service may play different roles
• Apache: web-server and/or reverse-proxy and/or load-balancer
• JBoss: front-end, back-end or both
• Master agent, Host agent, JBoss instances
• Service discovery and registration is complex
• IT requirements like SSH key, service user, password, directory, package repository…
• Legacy script and automation tools (taking in or discard)
• Workload distribution has to be planned ahead
• traditional process forking is not acceptable on a virtualized platform
• VMs become the management unit on cloud
• tuning specs: quota, profile, application characteristics
• scaling VMs instead of forking new processes
9

10. • An orchestrator sitting above compute, storage and network
• Template based VM provisioning, aka. stack creation
• Heat’s auto-scaling solution is valuable for Suning's Internet applications
• A standardized approach of cloud application deployment
• and orchestration?
10
milk powder: 5 million cans milk: 100 containers promotion season: 3 days

11. • Standard images and software packages
• Post-launch configuration
• creation of user accounts
• key distribution and revocation
• VM roles assignment
• package update or upgrade
• middleware install and configuration
• application install and configuration
• monitoring tools install and configuration
• service discover and registration ???
• .......
11

12. • Deployment and Orchestration
• Heat based deployment only covers part of the story
• Cloud-init only concerns with the initial deployment
• What we need is an integrated end-to-end tool chain
that covers runtime/maintenance orchestration as well
12
• Orchestration is not thoroughly tested in community (e.g. Auto-scaling)
• involves Heat, Ceilometer, Nova, Keystone...
• rolling-update may not work as expected
• scaling out may jump from one to many directly
• ceilometer alarm evaluator may not work
• ...
• fixing these is not an easy job

13. • Triggers for scaling
• network metrics (packets processed, bytes transferred) sounds interesting, but
• CPU and memory are still the primary bottlenecks
• Scaling may be triggered with combination factors of CPU, memory, disk I/O and/or
network I/O with customized algorithm
• Rolling update is of critical importance
• ensure a given number of instances are always online when performing updates
• Deletion policy from resource groups
• sometimes the newest members are preferred to oldest members, considering that
• old members may have state cached, may have proved to be stable, ...
• Fast detection and fast scaling (seconds level)
13

14. • Availability
• Storage Reliability/Availability
• Hard disk errors are common
• VM High-Availability (aka. the "Pets" story)
• it doesn't seem like a single project mission
• host failure, network failure, storage failure, guest failure, application failure ...
• may need to get Nova, Ceilometer (Zaqar?), Heat, Keystone to work together
• AutoScaling
• Semi-AutoScaling (Scale at a given point in time)
• Smarter VM placement, aka. Global Scheduling
• e.g. 3 Apache server per host is okay, but 9 Apache per host is risky
• VM placement is mainly concerned with service availability
• Scaling across availability zones, across regions
14

15. • Application Profile and Management
• Each application has a unique architecture where some components are reusable
• Most components are capable of playing different roles (e.g. front-end vs back-end)
• domain role, slave role, host role, etc.
• Combinations are difficult to predict and manage
• Solum? Murano?
• Provider Templates?
+ promote template reusability
+ facilitate fine granularity version control
- difficult to reference resources (attributes) from outer/inner templates
- difficult to get dependencies done right
- Tools to standardize Heat template collections
15

16. • Configurable frequency for Heat engine calls
• mostly from os-xxx-config
• may need a short interval during bootup, then switch to a longer interval
• Tools and guidance for the establishment of standard workflows
• need to abstract away common features and parameters
• need to simplify the deployment, management process
• need to adapt to new technologies
• e.g. transition from using shared disk volumes to use storage cloud
16

17. Thank You!
17