• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Diablo vs Havana: How OpenStack Has Matured
 

Diablo vs Havana: How OpenStack Has Matured

on

  • 771 views

Joe Gordon, HP Cloud presented at the OpenStack Summit Hong Kong 2013

Joe Gordon, HP Cloud presented at the OpenStack Summit Hong Kong 2013

Statistics

Views

Total Views
771
Views on SlideShare
771
Embed Views
0

Actions

Likes
2
Downloads
28
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Diablo vs Havana: How OpenStack Has Matured Diablo vs Havana: How OpenStack Has Matured Presentation Transcript

    • DIABLO VS HAVANA HOW OPENSTACK HAS MATURED Joe Gordon
    • ABOUT ME Developer @ Full time upstream OpenStack developer nova-core ATC since Essex Interested in making OpenStack scale and work contact information jog0 on freenode github.com/jogo launchpad.net/~jogo
    • OPENSTACK FROM DIABLO TO HAVANA A BRIEF HISTORY OF THE INTEGRATED RELEASE
    • DIABLO Swift Nova Nova-volume Nova-network Identity Glance Keystone*
    • ESSEX Swift Nova Nova-volume Nova-network Identity Glance Keystone Horizon
    • FOLSOM Swift Nova Nova-volume Nova-network Glance Keystone Horizon Quantum (Neutron) Cinder
    • GRIZZLY Swift Nova Nova-network Glance Keystone Horizon Quantum (Neutron) Cinder
    • HAVANA Swift (4) Nova (6) Nova-network Glance (2) Keystone (1) Horizon (1) Neutron (6) Cinder (4) Ceilometer (4) Heat (4) Total number of OpenStack services: 32
    • HAVANA FEATURES SUPPORTED BY DIABLO Swift Nova Nova-network Glance Keystone Horizon Neutron (Quantum) Cinder Ceilometer Heat
    • HAVANA FEATURES SUPPORTED BY DIABLO Swift Nova Nova-network Glance Keystone Cinder
    • OPENSTACK HAS GROWN
    • GROWN OUT NEW SERVICES Heat Ceilometer More in pipeline Icehouse Trove Incubation Ironic Marconi Savanna
    • GROWN UP Expand existing APIs Split out projects Extensions
    • DIABLO NOVA EXTENSIONS |Hss ot |Hs amnsrto ot diitain |Vlms oue |Vlmsspot oue upr |Qoa uts |Qoa mngmn spot uts aaeet upr |SmlTnnUae|Sml tnn uaeetnin ipeeatsg ipe eat sg xeso |FaoEtaaa |Poieadtoa dt frfaos lvrxrDt rvd diinl aa o lvr |ScrtGop euiyrus |Scrt gopspot euiy ru upr |FaoEtapc |Isac tp (lvr etases lvrxrSes ntne ye fao) xr pc |Cetsreet |Etne spott teCet Sre v. AI raeevrx xedd upr o h rae evr 11 P |Mlii utnc |Mlil ntokspot utpe ewr upr |VA Ss |VrulSoaeAry spot ita trg ras upr |VlmTps oueye |Vlm tpsspot oue ye upr |Kyar epis |KyarSpot epi upr |Rsu ece |Isac rsu md ntne ece oe |Vrulnefcs|Vrulitraespot itaItrae ita nefc upr |Fotn_p laigis |Fotn Isspot laig P upr
    • POST DIABLO NOVA EXTENSIONS Amncin diAtos Ageae grgts AtcItrae tahnefcs BrMtlxSau aeeaEttts BokeieapnVBo lcDvcMpig2ot Cls el Codie lupp CniDie ofgrv Cnoe osls DfreDlt eerdeee Eaut vcae EtneFotnIs xeddlaigp EtneIsa xeddpMc EtneSreAtiue xeddevrtrbts EtneSau xeddtts EtneVlms xeddoue FaoAcs lvrces FaoMng lvraae FaoSa lvrwp FotnIPos laigpol FotnIsuk laigpBl HdSreAdess ieevrdrse Iaeie mgSz Mgain irtos NtoksoitoSpot ewrAscainupr OIsacUaeuiLg SntnesgAdto Qoalse utCass ScrtGopealRls euiyruDfutue SrePswr evrasod SreUae evrsg Sev hle UeLmtFrdi sdiisoAmn UeQoa sruts Aet gns AsseVlmSasos sitdouenpht AalbltZn viaiiyoe BrMtloe aeeaNds ClCpcte elaaiis Criiae etfcts Codiepae luppUdt Cnoeupt oslOtu Cvrg oeae DsCni ikofg EtneAalbltZn xeddviaiiyoe EtneIs xeddp EtneQoa xedduts EtneSrie xeddevcs EtneVFe xeddINt FxdP ieIs FaoDsbe lvriald FaoRt lvrxx FotnIDs laigpn FotnIs laigp Fig pn Hprios yevsr IsacAtos ntnecin MlilCet utperae Ntok ewrs OTnnNtok Seatewrs Shdlrit ceueHns SreDansis evrigotc SreSattp evrtrSo Srie evcs UeLmt sdiis UeDt sraa VlmAtcmnUdt ouetahetpae
    • SO ONCE WE NORMALIZE FOR THE GROWTH IN OPENSTACK WHAT ELSE HAVE WE DONE IN THE PAST FOUR RELEASES?
    • CHANGES Drivers Testing and bugs Performance and scalability Forward facing internal changes Process
    • MORE DRIVERS Nova: baremetal, docker, hyperv, powervm Cinder: coraid, emc, glusterfs, gpfs, nexenta, rbd, sheepdog, soldfire, ...
    • BETTER TESTING, FEWER BUGS Unit tests Integration tests (Tempest) Lots of fixed bugs
    • UNIT TESTS PERCENT UNIT TESTS BY LOC Project Nova Glance Swift Keystone Diablo 43% 62% 65% 41% Havana 56% 62% 71% 45%
    • TEMPEST THE OPENSTACK INTEGRATION TEST SUITE
    • DIABLO TEMPEST Orignally named 'openstack-integration-tests' Just 107 tests* extensions flavors floating ips image metadata images keypairs servers volumes 2,655 lines of python code First release of Tempest
    • HAVANA TEMPEST 1275* tests 1150 api tests compute, image, identity, network, object_storage, orchestration, volume 102 test files 92 cli tests 11 scenario tests 18 thirdparty (boto) tests 4 other Parallel 20-30 minutes to run in gate 24,837 lines of python code
    • HAVANA API TESTS 785 compute 216 identity 24 image 17 network 50 object_storage 54 volume
    • HAVANA TEMPEST VS OPENSTACK DIABLO Same APIs as Diablo plus new ones Diablo has been 'end of life'ed Many tests cover new features. Ignore those Tempest not designed to be version independent Got 622 tests to run and 154 (27%) to pass.* 110 of which were in compute
    • HAVANA TEMPEST VS OPENSTACK DIABLO RESULTS Diablo has poor parameter validation Invalid key names Min/max count for multiple creates Changed return codes for negative cases Return empty list and 200 instead of 400 when invalid filter value in nova list servers We shouldn't have done this Poor pagination in Diablo JSON only tests in Diablo, Havana tests XML too Many false negatives in results due to slight changes in now APIs and metadata are used
    • CRITICAL BUGS Project Diablo Essex Nova 14 22 Glance 5 15 Keystone 31 Cinder Neutron Folsom 30 30 11 10 27 Grizzly 31 13 9 3 14 Havana 33 8 8 12 24 Categories Non-gated features Dependencies Race conditions Performance
    • PERFORMANCE AND SCALABILITY
    • PERFORMANCE AND SCALABILITY Nova-scheduler Active-active No RPC fanout DB Indices Smarter queries Fewer queries service group Heartbeat to DB every 10 seconds 1k DB calls a second if have 10K nodes. Adjustable interval Alternatives memcache and zookeeper
    • PERFORMANCE AND SCALABILITY More efficient image work flow Glance, Nova, Cinder Move large files around less PKI Tokens Fewer round trips to keystone
    • PERFORMANCE AND SCALABILITY CASE STUDY FROM HP PUBLIC CLOUD Diablo 40,836 LOC changed to 239 files in nova 2,265 LOC to manage DB +100s of hours of DB tuning to make it work with more then just a few nodes Havana DB just worked 1,167 LOC changed in 35 files Most of that is in patches in review today Thanks to Thom Leggett for these statistics
    • FORWARD FACING INTERNAL CHANGES Split out projects Oslo Road to live and continuous upgrades Iterate on APIs
    • PROCESS
    • DRIVING PRINCIPLES Never break trunk Master branch is always green Developers are never blocked Transparency Automate everything Egalitarian Be strict. Reduce burden on reviewers
    • DIABLO Automated Gate 87 Contributors in September 2011 (release of Diablo) No Gate No Git or Gerrit Bzr and Launchpad
    • HAVANA 346 Contributors in October 2013 (release of Havana) Scale development process Integrated Gate 9 Projects 30+ separate services 1 Merge pipeline New tools for an asynchronous world Tests not pass fail, rather classified with a race percentage Fix that race condition we hit 0.5% of the time Debugging is hard elastic-recheck
    • HOW WILL OPENSTACK KEEP MATURING WHAT CHALLENGES ARE WE FACING TODAY? More testing Live rolling upgrades Limiting factor to the whole system is reviewers (and not just core) Improving quality while growing the project as a whole Scalability
    • THANK YOU QUESTIONS? Powered by reveal.js