Intel IT OpenStack Practice
Shuquan Huang, Intel IT Engineer Computing

Intel Information Technology
Intel Confidential — ...
Agenda
• About Me
• Intel with OpenStack
• The way we do Cloud
• OpenStack Practice
• Integration with Intel Validation Cl...
Huang, Shuquan
• Join Intel since 2011
• Software Engineer, Certified Scrum Master
• Leads Intel IT Engineer Computing Ope...
Helping Fuel Innovation—and Opportunities
11.1%

X.org

9.3%

Eclipse

GNU

JQuery

Webkit
4.9%

Red Hat

Intel

SUSE

4.2...
Intel®
Enables OpenStack* Cloud Deployments

Contributions

•
•
•
•

Across OpenStack projects
Open Source Tools
Top 10 co...
Intel Contributions* to OpenStack
User Interface (Horizon)

Monitoring/Metering (Ceilometer)

Expose Enhancements

Metrics...
OpenStack – Intel IT Convergence Platform

Silicon Design
Cloud

Validation Cloud

Hosting Cloud

OpenStack

Existing Infr...
The Way we do Cloud
Our Developers

Snapshot
Improvement

Local Git

Bug Fixed &
new Features

Deployment Solution Cloud D...
Continuous Delivery
Continuous Delivery (CD) is a pattern language used in software development to
automate and improve th...
Problem Statement
• Configuration is easy to lost.
• It takes long time to get the feedback after developers make some
cus...
Solution
• Refer to http://ci.openstack.org/ and build our own system.
• Configuration as Code.
• Version Control: Code, C...
Challenges
• Installation by source code.
• Environment cleanup before each build.
• Live upgrade
• Rebuild totally
• Conf...
Deployment Solution
Rapid deployment different topologies OpenStack from bare metal

Intel Information Technology
Intel Co...
Deployment Solution
• Open Source Solution:
• PXE boot up
• puppet/dodai deployment

• Fast Deployment
• Deploy full stack...
Tempest Launcher
Have tempest more stable and visible to verify cloud functionalities

Intel Information Technology
Intel ...
Problem Statement & Objective
• Problem Statement:

• Result is not user unfriendly and exception may happen during
runnin...
Problem 1
Dirty test environment when previous test teardown function is
skipped for unexpected errors

testFunc()

setUp(...
Problem 2
Test script requires test pre-environment

TEST PASS

Test A

0
2 Available Networks

Test B

TEST FAIL

Test A ...
Problem 3
Cannot provide fully customized or prioritized test execution plan
……
when run test in bunch. It’s hard to run t...
Problem 4
Test result and error information is separated. XML report with ‘-with-xunit’ can’t import as test report in exc...
Solution
Provide a launcher script to address these problems

Start

Read XML test
plan file

Clean test
environment (Erro...
Result
• Customized test plan can be made on demand
• Minimize the impact of test failure to follow up test cases

• Excel...
Snapshot Improvement
Reduce the time of original OpenStack Snapshot

Intel Information Technology
Intel Confidential – for...
Problem Statement & Objective
• Snapshot in OpenStack
• Takes 20 mins for the snapshot of a 20GB VM
• Fast Snapshot in Ope...
Solution
• Analysis each step of original snapshot.
• Reduce the time consuming stage.
• Optimized the workflow to fit our...
Fast Snapshot Diagram
Nova

IMG0

copy

IMG1

Glance
IMG0
IMG1
IMG2

IMG2

Linked
VM Instances
IMG0
VM1

VM2

VM3

VM2

VM...
Fast Snapshot Diagram
Nova

Glance

IMG0

IMG0

IMG1

IMG1

IMG2

IMG2

IMG0
VM1

VM Instances
IMG0
VM1

VM2

VM3

VM2

co...
Fast Snapshot Diagram
Nova

IMG0
IMG1

Glance
IMG0

IMG0

copy

VM1

IMG1

IMG0
VM1

IMG2

IMG2

VM Instances
IMG0
VM1

VM...
Fast Snapshot Diagram
Nova

IMG0
IMG1

Glance
IMG0

IMG0

IMG1

VM1

IMG0
VM1

IMG2

IMG2

copy and resize
VM Instances
IM...
Cloud Data Analysis
Operational Intelligence in OpenStack: Listen to your cloud data

Intel Information Technology
Intel C...
Problem Statement
• Machine data provides insights into cloud status and plays a fundamental
role in the cloud administrat...
Solution
Web Portal

• Data aggregation

Other Tools compatible
with REST API

API Services

• CephFS

Hadoop on OpenStack...
Challenges
• Enable MapReduce in CephFS
• Hadoop 1.1.x & Ceph 0.61.8
• Add Hadoop-cephfs plugin to $HADOOP_HOME/lib
• Crea...
Result
• As an operational intelligence helper in our day-to-day cloud
operation work help administrator analyze unreadabl...
One-Stop-Shop Admin Portal

Intel Information Technology
Intel Confidential – for internal use only

35
One-Stop-Shop Admin Portal
• VMs and Hosts status
• Cluster Monitor & Alarm

• CPU, Memory, Disk, Network
• Log Browser
• ...
Monitoring & Alarm
• Cloud infrastructure monitoring
on workload, CPU, memory
utilization.
• Customized report to read the...
Integration with Intel Validation Cloud
Leveraging existing system and provide user the consistent experience

Intel Infor...
Leveraging existing Portal

Simple
Smooth
Customer-Designed
Collaboration

Intel Information Technology
Intel Confidential...
Fit for Intel internal business needs
• Users doesn’t need to care the physical locations of VM
• More powerful permission...
Intel Confidential — Do Not Forward
Backup

Intel Information Technology
Intel Confidential – for internal use only

42
Configuration of Hadoop on Ceph
• Hadoop 1.1.x & Ceph 0.61.8
• Install Packages:libcephfs1 libcephfs-java
• Add Hadoop-cep...
Construct the Vector Space

Matrix
1 7-4
1 4-6
1 6-10
1 10-13
1 10-14
1 10-5
44 Intel Information Technology
Intel Confide...
Construct the Vector Space
Matrix 1
1 7-4
1 4-6
1 6-10
1 10-13
1 10-14
1 10-5

Matrix 2
1 7-4
1 4-6
1 6-10
1 10-13
1 10-14...
Snapshot Diagram
Nova

IMG0

copy

IMG1

Glance
IMG0
IMG1
IMG2

Linked
VM Instances
IMG0
VM1

VM2

VM3

VM2

VM3

IMG1
VM1...
Snapshot Diagram
Nova

Glance

IMG0

IMG0

IMG1

IMG3

IMG1
IMG2

VM Instances
IMG0
VM1

VM2

VM3

VM2

VM3

IMG1
VM1

•
•...
Snapshot Diagram
Nova

IMG0

Glance
IMG0

IMG3
copy

IMG1

IMG3

IMG1
IMG2

VM Instances
IMG0
VM1

VM2

VM3

VM2

VM3

IMG...
Snapshot Diagram
Nova

IMG0

Glance
IMG0

IMG3

IMG1

IMG3

IMG1
IMG2
Linked

VM Instances
IMG0
VM1

VM2

VM3

VM2

VM3

I...
Upcoming SlideShare
Loading in …5
×

Q con shanghai2013-[黄舒泉]-[intel it openstack practice]

755 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
755
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Q con shanghai2013-[黄舒泉]-[intel it openstack practice]

  1. 1. Intel IT OpenStack Practice Shuquan Huang, Intel IT Engineer Computing Intel Information Technology Intel Confidential — Do Not Forward
  2. 2. Agenda • About Me • Intel with OpenStack • The way we do Cloud • OpenStack Practice • Integration with Intel Validation Cloud • Q&A Intel Information Technology Intel Confidential – for internal use only 2
  3. 3. Huang, Shuquan • Join Intel since 2011 • Software Engineer, Certified Scrum Master • Leads Intel IT Engineer Computing OpenStack Team • Focus on Cloud Solution and Agile Coming from Intel IT Engineer Computing • Focus on Intel private cloud solution – Intel Validation Cloud. • Start OpenStack journey from 2011. Intel Information Technology Intel Confidential – for internal use only 3
  4. 4. Helping Fuel Innovation—and Opportunities 11.1% X.org 9.3% Eclipse GNU JQuery Webkit 4.9% Red Hat Intel SUSE 4.2% OpenStack Yocto Project Hadoop IBM #2 Linux Contributor improving performance, stability & efficiency Intel in Open Source 01.org Across the Stack contributions span every layer of the stack Project Contributor Code Contributions to Open Source Projects kernel.org SPCEvirt_sc2010* Performance 3,000 Clutter 2,500 Ofono QT 0% 50% 100% Intel is single largest contributor to these projects Proven Components Throughput 2,000 KVM 1,500 1,000 500 0 MC-DP WSM-EP SNB-EP WSM-EX KVM building blocks simplify development, reduce costs and speed time-to-market Intel Information Technology Intel Confidential – for internal use only 4
  5. 5. Intel® Enables OpenStack* Cloud Deployments Contributions • • • • Across OpenStack projects Open Source Tools Top 10 contributor to Grizzly and Havana releases1 Optimizations, validation, and patches Intel® IT Open Cloud • • • Intel IT Open Cloud with OpenStack Deliver Consumable Services Automated Management of Cloud • • • • Collection of best practices Intel IT Open Cloud Reference Arch Share best practices with IT and CSPs http://www.intel.com/cloudbuilders Intel® Cloud Builders 1Source: www.stackalytics.com Intel Information Technology Intel Confidential – for internal use only 5
  6. 6. Intel Contributions* to OpenStack User Interface (Horizon) Monitoring/Metering (Ceilometer) Expose Enhancements Metrics Compute (Nova) Object Store (Swift) Erasure Object Storage Code Policy Enhanced Platform Awareness Block Storage (Cinder) Filter Scheduler Trusted Compute Pools Network Services (Neutron) (Extended with Geo Tagging) Intel® DPDK vSwitch Intelligent Workload Scheduling VPN-as-a-Service (Accelerated with Intel® QuickAssist Technology) Image Store (Glance) OVF Meta-Data Import Key Encryption & Management Identity Services (Keystone) Compute • Enhanced Platform Awareness • CPU Feature Detection • PCIe SR-IOV Accelerators • OVF Meta-Data Import • • • Networking Trusted Compute Pools • With Geo Tagging Key Management Intelligent Workload Scheduling (Metrics) • Intel® DPDK vSwitch • VPN-as-aService with Intel® QuickAssist Acceleration Storage • Filter Scheduler • Erasure Code • Object Storage Policies *Note: A mixture of features that are completed, in development or in Planning (not PoR) Intel Information Technology Intel Confidential – for internal use only 6
  7. 7. OpenStack – Intel IT Convergence Platform Silicon Design Cloud Validation Cloud Hosting Cloud OpenStack Existing Infrastructure Intel Information Technology Intel Confidential – for internal use only New Infrastructure 7
  8. 8. The Way we do Cloud Our Developers Snapshot Improvement Local Git Bug Fixed & new Features Deployment Solution Cloud Data Analysis Build Pre-Release Tempest Launcher Intel Information Technology Intel Confidential – for internal use only Production One-Stop-Shop Admin Portal 8
  9. 9. Continuous Delivery Continuous Delivery (CD) is a pattern language used in software development to automate and improve the process of software delivery. - wiki Intel Information Technology Intel Confidential – for internal use only 9
  10. 10. Problem Statement • Configuration is easy to lost. • It takes long time to get the feedback after developers make some customized changed in OpenStack. • Track and taste the latest update from community. • Automated run tests after each code changed. Intel Information Technology Intel Confidential – for internal use only 10
  11. 11. Solution • Refer to http://ci.openstack.org/ and build our own system. • Configuration as Code. • Version Control: Code, Configuration, Environment and anything Deploy new cluster by using some else. version source code and cluster topology Internet Daily update OpenStack and Tempest Repository GitHub Bug fixed & new features Our Developers Intel Confidential – for internal use only After deployment, run Tempest to validate changes Polling changes Local Git Repo Define topologies of OpenStack Cluster Deployment Engineer Intel Information Technology Puppet Master OpenStack Cluster Jenkins Manual Trigger CI or Configure polling stratigies Admin 11
  12. 12. Challenges • Installation by source code. • Environment cleanup before each build. • Live upgrade • Rebuild totally • Configuration as Code Intel Information Technology Intel Confidential – for internal use only 12
  13. 13. Deployment Solution Rapid deployment different topologies OpenStack from bare metal Intel Information Technology Intel Confidential – for internal use only 13
  14. 14. Deployment Solution • Open Source Solution: • PXE boot up • puppet/dodai deployment • Fast Deployment • Deploy full stack of OpenStack on mult nodes bare-metal with several mins. Live USB: • Ubuntu 12.04 • Essex Customized Package • Dodai server • Auto deployment: • Only one 4G U-Disk is needed Intel Information Technology Intel Confidential – for internal use only 14
  15. 15. Tempest Launcher Have tempest more stable and visible to verify cloud functionalities Intel Information Technology Intel Confidential – for internal use only 15
  16. 16. Problem Statement & Objective • Problem Statement: • Result is not user unfriendly and exception may happen during running tempest scripts in batch directly. • Objective • Address the problems mentioned above. • Improve and enhance the capability of tempest automatic launching and handling test in batch execution. Intel Information Technology Intel Confidential – for internal use only
  17. 17. Problem 1 Dirty test environment when previous test teardown function is skipped for unexpected errors testFunc() setUp() tearDown() Test environment will be cleaned up in tearDown() function Normal test execution flow setUp() Error happens testFunc() tearDown() Exit Script Test execution flow with exception Intel Information Technology Intel Confidential – for internal use only Test environment cannot be cleaned up And the dirty test environment will impact following test .
  18. 18. Problem 2 Test script requires test pre-environment TEST PASS Test A 0 2 Available Networks Test B TEST FAIL Test A didn’t release the networks from the test specific tenant. Test B failed since no available network. Intel Information Technology Intel Confidential – for internal use only
  19. 19. Problem 3 Cannot provide fully customized or prioritized test execution plan …… when run test in bunch. It’s hard to run the tests across folders or scripts in one time. Run the tests from 1 script: root@hostname:~/tempest# nosetests -v tempest.tests.compute.test_images Run the tests from 1 folder: root@hostname:~/tempest# nosetests -v tempest.tests.compute Intel Information Technology Intel Confidential – for internal use only
  20. 20. Problem 4 Test result and error information is separated. XML report with ‘-with-xunit’ can’t import as test report in excel directly. An image for the provided server should be created ... ok An image should not be created if the server instance is removed ... ERROR An image should not be created with invalid server id ... ok Return error when creating an image of a server that is building ... ok Return an error when creating image of server that is terminating ... ERROR Disallow creating another image when first image is being saved ... ok ====================================================================== ERROR: An image should not be created if the server instance is removed ---------------------------------------------------------------------Traceback (most recent call last): File "/root/tempest/tempest/tests/compute/test_images.py", line 84, in test_create_image_from_deleted_server self.servers_client.wait_for_server_status(server['id'], 'ACTIVE') File "/root/tempest/tempest/services/nova/json/servers_client.py", line 145, in wait_for_server_status raise exceptions.BuildErrorException(server_id=server_id) BuildErrorException: Server 3d7a3d34-7715-40ef-bcc5-6246d0cc2cbf failed to build and is in ERROR status Intel Information Technology Intel Confidential – for internal use only
  21. 21. Solution Provide a launcher script to address these problems Start Read XML test plan file Clean test environment (Error handling) Create pre-test environment Y Load test case Execute test Test Failed? N N End of test plan? Y End Intel Information Technology Intel Confidential – for internal use only Write test log
  22. 22. Result • Customized test plan can be made on demand • Minimize the impact of test failure to follow up test cases • Excel report template for visual result Intel Information Technology Intel Confidential – for internal use only 22
  23. 23. Snapshot Improvement Reduce the time of original OpenStack Snapshot Intel Information Technology Intel Confidential – for internal use only 23
  24. 24. Problem Statement & Objective • Snapshot in OpenStack • Takes 20 mins for the snapshot of a 20GB VM • Fast Snapshot in OpenStack • Self-customized feature • Takes less than 1 min for the snapshot of a 20GB VM Intel Information Technology Intel Confidential – for internal use only 24
  25. 25. Solution • Analysis each step of original snapshot. • Reduce the time consuming stage. • Optimized the workflow to fit our own requirement. Intel Information Technology Intel Confidential – for internal use only 25
  26. 26. Fast Snapshot Diagram Nova IMG0 copy IMG1 Glance IMG0 IMG1 IMG2 IMG2 Linked VM Instances IMG0 VM1 VM2 VM3 VM2 VM3 IMG1 VM1 • • • This is the relation between VM Instances, Nova and Glance. When we launch an instance, the image from Glance, eg IMG0 will be copied into Nova. Then the nova will create a linked image as “disk” for VM Instances. Intel Information Technology Intel Confidential – for internal use only
  27. 27. Fast Snapshot Diagram Nova Glance IMG0 IMG0 IMG1 IMG1 IMG2 IMG2 IMG0 VM1 VM Instances IMG0 VM1 VM2 VM3 VM2 copy VM3 IMG1 VM1 • • When user run fast snapshot for VM1-IMG0, the instances “disk” will be copied and stored in Glance. The speed of this process is depend on the size of linked image. Size = VM1 which is smaller if we compare to the original snapshot, Size = VM1 + IMG0 Intel Information Technology Intel Confidential – for internal use only
  28. 28. Fast Snapshot Diagram Nova IMG0 IMG1 Glance IMG0 IMG0 copy VM1 IMG1 IMG0 VM1 IMG2 IMG2 VM Instances IMG0 VM1 VM2 VM3 VM2 VM3 IMG1 VM1 • • When the user launch the created snapshot, the image from Glance will be copied into Nova. The speed of the copy is depending on the size of the image. Intel Information Technology Intel Confidential – for internal use only
  29. 29. Fast Snapshot Diagram Nova IMG0 IMG1 Glance IMG0 IMG0 IMG1 VM1 IMG0 VM1 IMG2 IMG2 copy and resize VM Instances IMG0 VM1 VM2 VM3 VM2 VM3 VM4 IMG1 VM1 • • • The VM1 from the Nova will be copied into a newly created VM (VM4) instance folder as a “disk”. The resize of the instance also being done during this process if the user pick bigger “flavor” for their instance The speed in creating a new VM is depending on the size of IMG0-VM1 Intel Information Technology Intel Confidential – for internal use only
  30. 30. Cloud Data Analysis Operational Intelligence in OpenStack: Listen to your cloud data Intel Information Technology Intel Confidential – for internal use only 30
  31. 31. Problem Statement • Machine data provides insights into cloud status and plays a fundamental role in the cloud administration. • user behavior • machine behavior • security threats • system health • Data are Separated. • Data are too large to parse. • Hard to query data in various format Data Source Intel Information Technology Intel Confidential – for internal use only Data Aggregation Data Storage Data Processing 31
  32. 32. Solution Web Portal • Data aggregation Other Tools compatible with REST API API Services • CephFS Hadoop on OpenStack • Data storage Query/Search/Mining • CephFS Structured Data PreProcess:MapReduce Jobs • Data processing • MapReduce OpenStack Nodes Log data Intel Information Technology Intel Confidential – for internal use only Nagios/Ganglia Ceph Nodes Monitor data 32
  33. 33. Challenges • Enable MapReduce in CephFS • Hadoop 1.1.x & Ceph 0.61.8 • Add Hadoop-cephfs plugin to $HADOOP_HOME/lib • Create pool on Ceph MON Node for Hadoop use Table 1. Action Cluster • Log Mining instance.create instance.shutdown instance.delete instance.power_off • Training Clustering Give the first two example Intel Information Technology Intel Confidential – for internal use only Clustering Give more example instance.rebuild instance.power_on instance.resize. instance.snapshot prep instance.resize. instance.resize confirm instance.resize. instance.finish_resize revert network.delete instance.exists volume.create port.create port.update volume.exists • Clustering network.update instance.update volume.delete • Construct the Vector Space network.create port.delete subnet.create subnet.update subnet.delete 33
  34. 34. Result • As an operational intelligence helper in our day-to-day cloud operation work help administrator analyze unreadable machine data and reduce a lot effort on understanding machine data. • Because machine data aggregation, administrator can look into those data through one place and can see the connection easily. • It helps a lot in the OpenStack cloud troubleshooting and increase the cloud utilization especially at midnight. Intel Information Technology Intel Confidential – for internal use only 34
  35. 35. One-Stop-Shop Admin Portal Intel Information Technology Intel Confidential – for internal use only 35
  36. 36. One-Stop-Shop Admin Portal • VMs and Hosts status • Cluster Monitor & Alarm • CPU, Memory, Disk, Network • Log Browser • Cloud Test • Power Package Integration • Power Awareness Scheduler • Active Standby Management • Policy Based Migration Intel Information Technology Intel Confidential – for internal use only 36
  37. 37. Monitoring & Alarm • Cloud infrastructure monitoring on workload, CPU, memory utilization. • Customized report to read the power consumption and CPU temperature metrics. • Greatly help the operation team aware the healthy status of the cloud. Intel Information Technology Intel Confidential – for internal use only
  38. 38. Integration with Intel Validation Cloud Leveraging existing system and provide user the consistent experience Intel Information Technology Intel Confidential – for internal use only 38
  39. 39. Leveraging existing Portal Simple Smooth Customer-Designed Collaboration Intel Information Technology Intel Confidential – for internal use only
  40. 40. Fit for Intel internal business needs • Users doesn’t need to care the physical locations of VM • More powerful permission control • Physical machine & device management Intel Information Technology Intel Confidential – for internal use only 40
  41. 41. Intel Confidential — Do Not Forward
  42. 42. Backup Intel Information Technology Intel Confidential – for internal use only 42
  43. 43. Configuration of Hadoop on Ceph • Hadoop 1.1.x & Ceph 0.61.8 • Install Packages:libcephfs1 libcephfs-java • Add Hadoop-cephfs plugin to $HADOOP_HOME/lib • Create pool on Ceph MON Node for Hadoop use • Edit core-site.xml – fs.ceph.impl = org.apache.hadoop.fs.ceph.CephFileSystem – fs.default.name = Ceph MON Server(ceph://10.239.149.9:6789) – ceph.data.pools = name of the pool you create 43 Intel Information Technology Intel Confidential – for internal use only
  44. 44. Construct the Vector Space Matrix 1 7-4 1 4-6 1 6-10 1 10-13 1 10-14 1 10-5 44 Intel Information Technology Intel Confidential – for internal use only
  45. 45. Construct the Vector Space Matrix 1 1 7-4 1 4-6 1 6-10 1 10-13 1 10-14 1 10-5 Matrix 2 1 7-4 1 4-6 1 6-10 1 10-13 1 10-14 0 10-5 45 Intel Information Technology Intel Confidential – for internal use only
  46. 46. Snapshot Diagram Nova IMG0 copy IMG1 Glance IMG0 IMG1 IMG2 Linked VM Instances IMG0 VM1 VM2 VM3 VM2 VM3 IMG1 VM1 • • • This is the relation between VM Instances, Nova and Glance. When we launch an instance, the image from Glance, eg IMG0 will be copied into Nova. Then the nova will create a linked image as “disk” for VM Instances. Intel Information Technology Intel Confidential – for internal use only
  47. 47. Snapshot Diagram Nova Glance IMG0 IMG0 IMG1 IMG3 IMG1 IMG2 VM Instances IMG0 VM1 VM2 VM3 VM2 VM3 IMG1 VM1 • • Rebase image (combine linked image with its base) When we run the snapshot, the Nova will rebase the instance into full image, and upload it to glance. It is really time consuming process if the size of the rebased image is huge ( IMG0 + VM1) Intel Information Technology Intel Confidential – for internal use only
  48. 48. Snapshot Diagram Nova IMG0 Glance IMG0 IMG3 copy IMG1 IMG3 IMG1 IMG2 VM Instances IMG0 VM1 VM2 VM3 VM2 VM3 IMG1 VM1 • • Now the new snapshot image is in database, when we run a VM from that image, it will copy the new image from Glance to Nova. The time taken to copy is depend on the size of the image. Intel Information Technology Intel Confidential – for internal use only
  49. 49. Snapshot Diagram Nova IMG0 Glance IMG0 IMG3 IMG1 IMG3 IMG1 IMG2 Linked VM Instances IMG0 VM1 VM2 VM3 VM2 VM3 IMG1 VM1 IMG3 VM1 • • • A new VM instance launched based on the snapshot that have been created It will create a linked image based on IMG3 This is a really quick process since it is just create an empty linked clone image into VM Instances area. Intel Information Technology Intel Confidential – for internal use only

×