Presentation about OpenStack hacks by Rakuten and the community @ YLUG Kernel Code Reading Partyカーネル読書会 112th

1. OpenStack, Hacking, Community
Aug/01/2014 Kentaro Sasaki, Shuichiro Makigaki Global Operations Department, Rakuten, Inc.

2. 2
•OpenStack Overview
•Issue and Hack it
•Community
•Development
Agenda

3. 3
100% OSS cloud management software:
•OpenStack project began in 2010.
•NASA Nebula -> Nova.
•Rackspace Cloud Files -> Swift. All components are shared-nothing. 10 core components in Icehouse(2014.1)
OpenStack

4. 4
Shared-nothing component
Nova
(Compute)
Horizon
(Web UI)
Keystone
(Identity)
Glance
(Image)
Cinder
(Block Storage)
Neutron
(Networking)

5. 5
Nova architecture
REST API
SQL
RPC
Decide compute node
Consolidate DB access
Each modular components collaborate with REST/RPC message.
nova-scheduler
nova-conductor
MySQL
RabbitMQ
nova-compute
nova-api
Client

6. 6
Guiding Principles at Rakuten
Technology:
•Do not fork an OSS project w/o a good reason, and contribute to that rather than creating
•Industry Best Practice Functionality:
•Multi tenant portal for all services
•Auto Scaling to keep stability during peak traffic

7. 7
RIaaS – Rakuten Infrastructure as s Service:
•Original management tool (esp. SLB & DNS) > OpenStack
•Online migration of existing VMs on VMware
•Control VMware’s vNetwork Distributed Switch
Issue and Hack it

8. 8
e.g. Online migration of existing VMs
Control existing VMs on OpenStack
* “uuid” could not be updated because it has foreign key constraints.
Make VMs look like deploying by using OpenStack.
Run only
DB INSERT
at nova boot
UPDATE
nova’s DB
to be active
SELECT VM’s
instance_id
from vCenter

9. 9
def _create_instance(self, context, instance_type,
"""snip"""
instances = self._provision_instances(context, instance_type,
min_count, max_count, base_options, boot_meta, security_groups,
block_device_mapping)
"""comment out from here to the end of this function"""
# filter_properties = self._build_filter_properties(context,
# scheduler_hints, forced_host, forced_node, instance_type)
"""snip"""
return (instances, reservation_id)
Modify nova-api code #1
~/nova/compute/api.py: line 917
Reference:
http://docs.openstack.org/developer/nova/devref/vmstates.html

10. 10
def _populate_instance_for_create(self, instance, image,
index, security_groups, instance_type):
"""Build the beginning of a new instance."""
if not instance.obj_attr_is_set('uuid'):
# Generate the instance_uuid here so we can use it
# for additional setup before creating the DB entry.
instance['uuid'] = str(uuid.uuid4())
{"server": {
"min_count": 1, "flavorRef": "1", "name": "XXX",
"imageRef": "YYY", "max_count": 1, "specific_uuid": "ZZZ",}
}
Configure "specific_uuid" to instance['uuid']
~/nova/compute/api.py: line 1204
PUT method
Modify nova-api code #2

11. 11
UPDATE nova’s DB
vm_state -> active (=BUILD)
task_state -> NULL (=scheduling)
power_state -> 1
host -> nova-compute node
launched_on -> nova-compute node
node -> domain-cXX(VMware cluster_name)
updated_at -> now()
scheduled_at -> now()
launched_at -> now()
We can control all of existing VMs on VMware!
instances table in nova DB

12. 12
Roles
Technical committee
Program Leads (PTLs)
Active Program Contributors (APCs)
Active Technical Contributors (ATCs)
•Define OpenStack Programs
•Manage day-to-day operations
•Drive the program goals
•Resolve technical disputes
•Voters for a given program PTL
•Subset of the Foundation Individual Member
•Committed a change over the last two 6-month
•Candidacy for the corresponding program PTL election.
•TC seats are elected by ATC
•Same as APC
•Bug triagers, technical documentation writers, etc can exceptionally apply for ATC

13. 13
Communication
Mailing list: https://wiki.openstack.org/wiki/Mailing_Lists
•Question, Discussion, Announce, … IRC: https://wiki.openstack.org/wiki/IRC
•Question, Discussion, Code review request, Meeting (weekly), …
•IRC is important place for decision making (Midnight in Japan…) Others:
•Ticket comment (Launchpad)
•Code review comment (Gerrit)
•QA: https://ask.openstack.org/en/questions/

14. 14
Atmosphere
Be kind, honest, correct, detailed, organized and professional. Full support for new contributors: Upstream Training (Official contribution training) Improve your presence in the community:
•Mails, ticket comments, reviewing patches, joining meeting, …
•Your patch can be reviewed quickly.
•Communication on IRC is important. (Patch review request, etc)

15. 15
Source code
Official git repository: http://git.openstack.org/cgit
•Github mirror: https://github.com/openstack (pull requests and issues are ignored.) Python, bash, HTML, JavaScript:
•Clean (pep8 is needed.)
•But, lack of comments and code document

16. 16
Bug, new feature
Launchpad:
https://launchpad.net/openstack
Bug -> bugs
New feature -> blueprints
Question -> answers
Github issue is ignored.

17. 17
Code review
Gerrit Review system:
•Open source code review system
•Used for CI, implementation discussion Github pull request is ignored. Patch can be discussed in Launchpad and mailing list, but must be reviewed and pass tests to merge.

18. 18
E-mail, Launchpad or Gerrit?
Question -> Mailing Lists Bug discussion -> Launchpad Implementation discussion -> Gerrit Importance: pushing patch > launchpad discussion Actual codes are always justice :-)

19. 19
Test, CI
Jenkins:
•CI runs every time automatically when the patch is pushed.
•The result is commented the review request, and you can see test logs.
•Tests are also developed in the community. (means there may be bugs in tests!)

20. 20
Code review process
Patch must be reviewed in Gerrit.
Patch must passed all tests.
Patch must be reviewed by core reviewers.
Patch must be approved by core developers.
•test fail? fix your code, and push it again.
•test fail?? the test is "valid" test? Check test bug.
•test fail??? Cosmic ray or lack of your luck.

21. 21
Community review
Common review checklist: https://wiki.openstack.org/wiki/ReviewChecklist
Everyone can do it: +/- 1 (this can get attentions of core reviewers)
Core reviewer can do it: +/- 2 (two "+2" are required to merge)

22. 22
Activity visualization
Contribution visualization: Stackalytics (http://stackalytics.com/) Activity visualization: http://activity.openstack.org/

23. 23
Activity besides OpenStack itself
Code contribution:
•Common library: oslo (messaging, DB, config paser, i18n, …) https://wiki.openstack.org/wiki/Oslo
•Git subcommand for gerrit: git-review https://wiki.openstack.org/wiki/Gerrit_Workflow#Git_Review_Installation Summit: Meetup, Conference, Design discussion https://www.openstack.org/summit/

24. 24
Thank you