State of the Union: Open Source Netw
ork Function Virtualization

Mario Smarduch
Senior Virtualization Architect
Open Sour...
Talk Description

laitnedifnoC

One of the hottest developments today for Fixed and Mobile Networks is
'Network Function V...
laitnedifnoC

Agenda
General Public Clouds
NFV Introduction, Status
Cloud RAN NFV use case
KVM (ARM) – limitations/require...
Public Cloud Control

laitnedifnoC

Focus on IaaS – PaaS, SaaS build on top of each other
- NFV does have PaaS, SaaS – pow...
Public Cloud Network

laitnedifnoC

L3 & L2 in public cloud – Scaling the Cloud
- Public clouds 40,000 Physical machines p...
© 2013 SAMSUNG Electronics Co.

6

swolf lortnoc – ltcfo-svo 0rb ot pat dda ltcsv-svo – pufi.umeq >cftni-syhp< 0rb trop-dd...
Web front end, SQL data base backend
– eCommerce
Social Networking
SaaS apps like email, Content Backup
High Performance C...
laitnedifnoC

Introduction to NFV
Mobile Network – LTE EUTRAN/EPC

© 2013 SAMSUNG Electronics Co.

8
laitnedifnoC

Introduction to NFV
• EUTRAN – eNodeB, UE

- Radio – bearer, admission, mobility, scheduling dynamic radio r...
© 2013 SAMSUNG Electronics Co.

DIET 5S



TFT-LD ICQ

1L
2L
PI / P D U
U- PT G
PI

DIET 5S



01

DIET 1S

S5-U Bearer
...
laitnedifnoC

LTE EUTRAN/EPC Load Characteristics
Resources
- Radio BW, Network (CN), CPU, Memory, Storage (varies on NE l...
Current State of NVF

laitnedifnoC

NFV ETSI ISG

- Initial White Paper published Oct 2012
- Spans Mobile, and Fixed Netwo...
NFV Cloud-RAN Use Case

laitnedifnoC

Evolution of Radio Access Network
Single mode – 2G,3G – combined BBU & RRU
Scaled to...
laitnedifnoC

New Virtualization HYP Mode

© 2013 SAMSUNG Electronics Co.

41
laitnedifnoC

Virtualization MMU Extensions

© 2013 SAMSUNG Electronics Co.

51
laitnedifnoC

Interrupt Virtualization Extensions

© 2013 SAMSUNG Electronics Co.

61
Device Pass-through

laitnedifnoC

Architecture/cost of interrupts
BBU cloud has hundreds of devices passed through – smal...
Device Pass-through

laitnedifnoC

IRQ over head and optimizations

1. Guest executes – exit to hyp mode – save guest/rest...
laitnedifnoC

Dynamic Load Balancing
Cloud Ran – dynamic load balancing between VMs
- Cell sites exhibit various loads thr...
© 2013 SAMSUNG Electronics Co.

02

) e nil n e e r g( U M E Q h g u o r h t e ci v e d
d e t al u m e ai v l o c o t o r ...
RT-Scheduling

laitnedifnoC

Network stack time sensitive - requirements
- Highres timers a must
- Preemptibility – event ...
RT-Scheduling

laitnedifnoC

Possible Optimizations – area of research
Host PREEMPT_RT
Eliminate spinlocks, replace with m...
Thank you.
Mario Smarduch
Senior Virtualization Architect
Open Source Group
Samsung Research America (Silicon Valley)
m.sm...
Upcoming SlideShare
Loading in …5
×

State of the Union: Open Source Network Function Virtualization

1,626 views

Published on

Mario Smarduch, Senior Virtualization Engineer from the Samsung OSG, gives his perspective on Network Function Virtualization, including discussions of the Xen platform.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,626
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

State of the Union: Open Source Network Function Virtualization

  1. 1. State of the Union: Open Source Netw ork Function Virtualization Mario Smarduch Senior Virtualization Architect Open Source Group Samsung Research America (Silicon Valley) m.smarduch@samsung.com © 2013 SAMSUNG Electronics Co.
  2. 2. Talk Description laitnedifnoC One of the hottest developments today for Fixed and Mobile Networks is 'Network Function Virtualization', headed by ETSI (European Telecommunications Standard Institute) ISG which managed to become the largest ISG in matter of six months with close to 70 members and 90 participants. Goals of NFV are to eliminate proprietary hardware appliances, to reduce energy, space, and hardware turnover cost. Leverage IT virtualization benefits like consolidation, time to market, multi-tenancy of heterogeneous applications, scaling out and in, and encourage an open eco-system not tied to any specific hardware. However IT virtualization is currently not fit for some NFV scenarios, Network Elements, User Equipment. Proprietary vendors and chip manufacturers are rushing to close this gap. This presentation focuses on open source virtualization technology primarily KVM-ARM to contrast these Gaps and identify required low level enhancements in hypervisor, guest, and ongoing community development to address these gaps is presented. Real uses cases are presented to illustrate why IT virtualization is not always a fit for many NFV scenarios. A brief overview of ARM-KVM virtualization and hardware extensions are also 2 covered. © 2013 SAMSUNG Electronics Co.
  3. 3. laitnedifnoC Agenda General Public Clouds NFV Introduction, Status Cloud RAN NFV use case KVM (ARM) – limitations/required enhancements © 2013 SAMSUNG Electronics Co. 3
  4. 4. Public Cloud Control laitnedifnoC Focus on IaaS – PaaS, SaaS build on top of each other - NFV does have PaaS, SaaS – powerful use cases as well (see NFV use case document) - IaaS to grow – 2011 $4.2B $24B 2016 (Source: Gartner) IaaS owner issues new VM request via portal server with params # of cores, memory, storage, image to load/install Scheduler – view physical server/storage/network DB selects optimal server, loads image, creates raid creates VM in Compute cloud o May need to migrate load, create NAT entries o For KVM issue libvirt qemu, commands Update DB to maintain availability IaaS owner – unaware of physical topology, migration, i.e. other management – cloud infrastructure control plane OpenStack equivalent components – Dashboard, Network Compute, Image, Block Storage, … esabataD e g a mi 2 woc Q U ME Q DIAR kl b - oi t ri v t e n - oi t ri v s d a er h T OI / U P C v U ME Q VM WSv srenwO SaaI reludehcS tnegA duolC egamI & egarotoS VM )..,ERG ,SNALV osla( lortnoC NDS – wolFnepO latroP tnegA CNV ,HSS duolC etupmoC VM VM © 2013 SAMSUNG Electronics Co. 4
  5. 5. Public Cloud Network laitnedifnoC L3 & L2 in public cloud – Scaling the Cloud - Public clouds 40,000 Physical machines possibly up to 1,000,000 VMs ∙ 2011-Gartner 8VMs/Server, probable 30:1 ratio - IaaS – typically don’t require L2 broadcast domain, scale through multiple VMs - VMs place on unique subnets – isolated for security - Very few apps require – L2 in Cloud (broadcast, multicast – discovery of services) ∙ Large cloud providers – support L2 subnets ∙ Some client/server architectures – i.e. front end/backend processing - Large cloud Scaling achieved through L3 hierarchical aggregated routes © 2013 SAMSUNG Electronics Co. 5
  6. 6. © 2013 SAMSUNG Electronics Co. 6 swolf lortnoc – ltcfo-svo 0rb ot pat dda ltcsv-svo – pufi.umeq >cftni-syhp< 0rb trop-dda ltcsv-svo # 0rb rb-dda ltcsv-svo # WG 1 ,PI 1 hcaE 4 * stenbus 8 – etagergga xx.]03-82[ stib 03/13.452.02.01 – 03/0.452.02.01 MVK htiw hciwsvnepO - NDS x.x.861.291 WSv 23 * stenbus 8 – etagergga xxxx_x.]72-52[ 72/422.452.02.01 – 72/0.452.02.01 …… 42/0.0.02.01 42/0.552.02.01 sdleif yna no etuor/hctiws wolFnepO noitartsihcrO NDS 652 * stenbus 652 – etagerggA xxxx_xxxx.]42-71[ stib – 61/0.0.02.01 sPI elpitlum ot drocer A elpmaxe roF ecnalaB daoL SND - L2 overlay of L3, support isolated L2 Subnets For VMs IaaS Scaling via L3 Public Cloud Network laitnedifnoC
  7. 7. Web front end, SQL data base backend – eCommerce Social Networking SaaS apps like email, Content Backup High Performance Computing in the clo ud Characteristics • • • • • • Resources – traditional compute – cpu, ram, storage, network Response - Not Real-time – response driven by user perception (web interface) Scalability – out, in – add/remove VMs - Front frontend server, or load balancer distributes load, I/O – primarily virtualized – storage, network Overcommit – as much as current average 8:1, future 30:1 per Server [Source: cloudscaling] Orchestration – spans few VM types, small geographic area – same Pod 7 Workloads laitnedifnoC Public Cloud Characteristics (IaaS) © 2013 SAMSUNG Electronics Co.
  8. 8. laitnedifnoC Introduction to NFV Mobile Network – LTE EUTRAN/EPC © 2013 SAMSUNG Electronics Co. 8
  9. 9. laitnedifnoC Introduction to NFV • EUTRAN – eNodeB, UE - Radio – bearer, admission, mobility, scheduling dynamic radio resource allocation for uplink and downlink • EPC - PCRF = Policy Control and Charging Rules ∙ Determines QoS Class Identifier for data flow ∙ QoS – GBR/non-GBR, Priority, Delay, Pkt error loss rate – RT Gaming, Voice, Live - - 9 - Streaming most demanding HSS = Home Serving Server ∙ Subscriber profile – QoS, APN (PDN), current user MME P-GW = Packet Data Network Gateway ∙ UP IP alloc, enforce PCRF QCI map to DL bearers S-GW = Serving Gateway ∙ UE anchor for all IP traffic as UE roams through eNodeBs, retain bearer info for UE in idle MME = Mobility Management Engine ∙ Control node, UE attachment, bearer setup, UE context management from HSS, process Tracking Area Update, paging, UE-IDLE to CONNECT state © 2013 SAMSUNG Electronics Co.
  10. 10. © 2013 SAMSUNG Electronics Co. DIET 5S  TFT-LD ICQ 1L 2L PI / P D U U- PT G PI DIET 5S  01 DIET 1S S5-U Bearer WG-P DIET 1S DI-BR BedoNe WG-S g ni m a g T R s m 0 5 y al e D R B G 4I C Q o e di v s m 0 0 1 y al e D R B G 2I C Q e ci o v s m 0 0 1 y al e D R B G 1I C Q DI BR TFT-LU YHP CA M CLR PCDP PI EU egasseM EU skcabyggiP dna BedoNe @ reraeB etavitcA tsqR reraeB etaerC tsqR reraeB etaerC DIET U-PTG dna ICQ ot pam ,TFT etacollA  S1-U Bearer dmC ecruoseR reraeB reraeB rof secruoseR E2E enimreteD  Radio Bearer …….. Data Plane Three traffic pipes - call setup dmC ecruoseR reraeB notacolla CRR no EU llet BedoNe esnopseR reraeB yfidoM esnopseR reraeB yfidoM tseuqeR reraeB yfidoM tseuqeR reraeB yfidoM )erudecorp gnol( tseuqeR noitazirohtuA & noitacifitnedI qeR noitacollA ecruoseR reraeB )sGSM lareveS( hsilbatsE nnoC CRR SSH WGS WG-P EU BedoNe EMM Paging – MME 500-800 UE msgs/hr, heavy load – 1500msgs/hr - Example Call Setup – range from 2-3sec - - LTE supports Public Safety – call setup time < 300ms – support group calls - Other procedures –Sys Info Bcast, UE Rand Access Proc., UE Attach/Detach, TAU, Call Term., Control Plane – idea of messaging in Bearer setup – mobile initiated Establishing Bearers laitnedifnoC
  11. 11. laitnedifnoC LTE EUTRAN/EPC Load Characteristics Resources - Radio BW, Network (CN), CPU, Memory, Storage (varies on NE like HSS). Response - State Machine driven - Attachment, idle-connect, bearer setup – associated with timers/states Real-time sensitive – various parameters can be tuned – but User Experience Suffers User perception still all important – but hard deadlines exist Near native scheduling Scalability & Orchestration - Network tightly coupled – scaling out – ripples through NEs - Unlike Public Cloud just adding new VMs will not do it - Orchestration for scale out/in extremely complex I/O – Need near native - RAN – massive device pass-through BBU accelerators, EPC NIC device pass-through Overcommit - Delicate load calculation required for PLMN to scale on demand where needed - Can’t apply Cloud 8:1, 30:1 ratios © 2013 SAMSUNG Electronics Co. 11
  12. 12. Current State of NVF laitnedifnoC NFV ETSI ISG - Initial White Paper published Oct 2012 - Spans Mobile, and Fixed Networks - First serious attempt to virtualize Mobile/Fixed networks ∙ Members Service Providers and all eco-system players Proof of Concepts – Cloud Ran, Migration with Dev Pass-through, Cloud rGW - Network Function Virtualization as a Service (NVF IaaS) ∙ Target Big Telco/Small Telco – lease NFVI as IasS for VNF and Cloud - VNFaaS – move enterprise CPE into SP cloud, and later PE simplify Opex/Capex ∙ AR, NG-FW, QoS/DPI in owned/provisioned by SP - VNPasS – Platform as a Service for example DNS, DHCP, email, FW ∙ Bring closer to APN – no tunneling back central IT infrastructure – total control ∙ SP provides bare services and Enterprise with config tools to manage the service - VNF Forwarding Graphs ∙ Essential SDN – in multi-tenant environment OpenFlow capable config required to host f.e. small telco in VNFI ∙ Need SDN orchestration OpenStack enhancing Quantum for SDN – to span VNFs and Physical Network functions - Mobile Core Virtualization – Goes along with NVFIaaS (to some extent) ∙ Improves Self Optimizing Networks – deliver performance where needed - Cloud-RAN - key features for SON, on demand Radio BW, Opex/Capex savings - Virtualizing home – vSTB, vGW – Fixed Network video/internet delivery to home © 2013 SAMSUNG Electronics Co. 21
  13. 13. NFV Cloud-RAN Use Case laitnedifnoC Evolution of Radio Access Network Single mode – 2G,3G – combined BBU & RRU Scaled to maximum peak – waste of resources Base Band Processing co-located with Remote Radio Unit Remote Radio Units distributed via fiber links Base Band Processing support multiple technologies BBU can be housed in-door RRUs strategically distributed MME/SGW sURR Pooling of Radio Base Band Unit Processing Capacity dynamically adjusted – example sport event Resources maximized – delivered on demand Several Technologies supported SGSN s r o t a r el e e c A Y H P UBB 31 • • • • ETL UBBv • • • Hard access, power an issue in some locations ETL UBBv UBB o STMU UBBv • • • © 2013 SAMSUNG Electronics Co.
  14. 14. laitnedifnoC New Virtualization HYP Mode © 2013 SAMSUNG Electronics Co. 41
  15. 15. laitnedifnoC Virtualization MMU Extensions © 2013 SAMSUNG Electronics Co. 51
  16. 16. laitnedifnoC Interrupt Virtualization Extensions © 2013 SAMSUNG Electronics Co. 61
  17. 17. Device Pass-through laitnedifnoC Architecture/cost of interrupts BBU cloud has hundreds of devices passed through – small cells many RRUs and fiber links Libvirt, qemu not ready for such passive pasthrough, another issue handling faults RRU to/from BBU PHY OFDMA (channels framing,FEC) to MAC – L2 logical Channels L3 - RRC, NAS, IP • MMU Pass-through – to user o o Devices emulated – trap to QEMU – not this type GVA IPA HPA – Direct access to HW regs o No performance penalty for MMU pass-through PCI – looks up target BARs for HPA, QEMU selects IPA DT – Device node with HPA, QEMU selects IPA ) CI N / E U o t Y H P ( h g u o r h t s s a P e ci v e D OI M M tseuG res U • Cost of Exit/Enter – executed in HYP – optimized assembler o Similar to process switch, Guest switch very costly o o No concept of light-weight context switch like threads Goal avoid at all costs )… ,qri ,tbai ,tbad( sger deknab llA derotser/devas sretsiger metsys SO – os eroM tseuG etatS tseuG daoL s g er 5 1 P C D MI S / P F V sgeRP G s g er 0 8 ~ tsoH etatS tsoH evaS 51PC D MI S / P F V sgeRP G s g er 0 8 ~ tsoH etatS tsoH daoL 51PC D MI S / P F V sgeRP G s g er 0 8 ~ tseuG etatS tseuG evaS s g er 5 1 P C D MI S / P F V sgeRP G s g er 0 8 ~ 71 - © 2013 SAMSUNG Electronics Co. AVG l e nr e K ksaT ts o H oT nr ut e R UMEQ ksaT AVG AVG s r e vi r D A PI ti x E tseu G tseuG UMEQ s r e vi r D A PI yromeM e ci v e D E &T tseu G oT t c ej nI ti x E e t a d p u I O E D CI G v tsoH/MVK APH )2LP( edoM PYH )NC ot CIN ,EU ot YHP( erawdraH ts o H oT Q RI
  18. 18. Device Pass-through laitnedifnoC IRQ over head and optimizations 1. Guest executes – exit to hyp mode – save guest/restore host 2. Host enable Interrupts – deliver to host – 1 Complete IRQ OS PATH 3. Inject to Guest – save host, restore guest & 2 Complete IRQ OS PATH 4. Guest EOI – exit save guest/restore host 5. Update virtual distributor 6. Resume Guest – save host/restore guest Note: Applying most direct injection no – irqfd, and additional threads ) CI N / E U o t Y H P ( h g u o r h t s s a P e ci v e D OI M M ts dn tseuG res U AVG l e nr e K Testing by Virtual Open Systems reveals atleast 5x delay Optimization 1 • ARM supports piority drop/deactivation after ack IRQ priority drops and can deactivate from Guest during EOI w/no exit • ARM can inject hwirq • Eliminate 4-6 (experimenting) Optimization 2 • Process Interrupts directly from HYP mode • Build hwirq inject to Guest • Eliminate 2-6 HOWEVER requires C-code, more overhead in HYP mode l e v el w ol o t d e ti mil e d o m P Y H yl t n e r r u C o In Addition • IRQ CPU affinity must match vCPU affinity – either bind or follow vCPU otherwise you need IPIs – very slow • Prefereable vCPU in idle not exit, wait for event not exit • Future GIC versions per IRQ – direct delivery – still handle sleeping guests © 2013 SAMSUNG Electronics Co. 81 ksaT ts o H oT nr ut e R UMEQ ksaT AVG AVG s r e vi r D A PI ti x E tseu G tseuG UMEQ s r e vi r D A PI yromeM e ci v e D E &T tseu G oT t c ej nI ti x E e t a d p u I O E D CI G v tsoH/MVK APH )2LP( edoM PYH )NC ot CIN ,EU ot YHP( erawdraH ts o H oT Q RI
  19. 19. laitnedifnoC Dynamic Load Balancing Cloud Ran – dynamic load balancing between VMs - Cell sites exhibit various loads throughout the day - vCPU hotplug ∙ Unplug/plug vCPUs – dynamically scale to demand sUPCv gulp/gulpnu Multicore Platform Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core vBBU vBBU 91 vBBU © 2013 SAMSUNG Electronics Co. gnildI sUPCv tmgM rewoP
  20. 20. © 2013 SAMSUNG Electronics Co. 02 ) e nil n e e r g( U M E Q h g u o r h t e ci v e d d e t al u m e ai v l o c o t o r p r e v o c si d y r o m e m d e r a h s : n oi t ul o S y o r t s e d / e t a e r c d n a m e d n o s d u ol c U B B v y n a M g nil a n gi S t p urr et nI d e zi mi t p O o o Pull packet from Shared Memory Ring buffer Tx/Rx to Core Network SCTP or GTP-U o Signaling – signaling/interrupt path too long (red lines) o Discovery – to pair Guest must discover shared memory segments dynamically e c s el a o c , e d o m P Y H t p u r r e t ni : n oi t ul o S t s e u G o t t p u r r e t ni t c ej ni r e e p n o e ci v e D d e t al u m E U M E Q r e e p ’ t n e v e‘ U M E Q U M E Q o t OI M M , ti x e , g e r Q RI o t s e ti r w OI U ai v t s e u G )NC ot CIN ,EU ot YHP( erawdraH edo M PYH Issues: Core Network y r o m e m d e r a h s t s e u g - r e t ni o t y p o c o r e Z t ek c a p 3 L + C A M + e c a p s r e s u o t yl t c e ri d e ri w f o ll u P • yromeM derahS M V K/ T S O H e ci v e D E &T • o Dedicate CPUs – poll –or- optimized dev passthrough CIN VRD MHS e ci v e D E &T VED MHS YHP NAR VED MHS VRD MHS o PHY Device passthrough – . d o o g r e v e n si e t a ci d e D ksaT ksaT UMEQ tseuG tseuG h g u or ht-ss a P UMEQ vBBU Instance • • • • BBU needs fast switching – radio  core network Can’t have full stack with expensive IPC Want to separate Radio and Core functions Radio - ivshmem one example, add enhancements Zero copy message passing – Guest/Guest, Guest to Host Fast Path between Radio/CN laitnedifnoC
  21. 21. RT-Scheduling laitnedifnoC Network stack time sensitive - requirements - Highres timers a must - Preemptibility – event PREEMPT_RT a must – prevent interrupt inversion - Scheduling at several levels – host and guest threads Timers - Arch-timers improvement no exit on reg updates But still exit on timer fire – need injection Issue for high res timers in Guest Again near native IRQ pass-through important IRQs & page faults - Any host IRQ can prevent guest from running - PFRA as well (if so most likely not tuned for RT) PREEMPT_RT - Not really tested with virtualization 12 Guest © 2013 SAMSUNG Electronics Co. )TR_TPMEERP( MVK & t s o H x u ni L )TR_TPMEERP( WG-Sv xet EMMvum=kcoL nipS … ,MSF locotorP ,OIFV ycnetaL stnevE remiT revnI oirP/tnI on setucexE daerhT oirP rehgiH tpurretnI xetum=kcoL nipS ycnetaL .revnI oirP/tnI on setucexE daerhT oirP rehgiH tpurretnI stnevE remiT er a w dr a H s e cr u o S Q RI
  22. 22. RT-Scheduling laitnedifnoC Possible Optimizations – area of research Host PREEMPT_RT Eliminate spinlocks, replace with mutexes Prioritize interrupts - non-VM targeted IRQs vCPUs – prioritize at higher priority VM IRQs don’t run as threads – timers, dev-pa ssthrough IRQs - Use Priority Drop/Deactivation to schedule highest priority interrupts for VMs - Guests most likely PREEMPT only Challenges – - multiple VMs sharing CPU ∙ Priority between them ∙ Priority of their IRQs ∙ Context switching an issue, depends on load - OS periodic tick work - CONFIG_NO_HZ_FULL ∙ Promising, for dedicated vCPU to core reduces tick ov erhread ∙ Improves multiple vCPUs as well tick rate 22 Guest © 2013 SAMSUNG Electronics Co. )TPMEERP( MVK & t s o H x u ni L )TR_TPMEERP( WG-Sv xet EMMvum=kcoL nipS … ,MSF locotorP ,OIFV ycnetaL stnevE remiT revnI oirP/tnI on setucexE daerhT oirP rehgiH tpurretnI xetum=kcoL nipS ycnetaL .revnI oirP/tnI on setucexE daerhT oirP rehgiH tpurretnI stnevE remiT er a w dr a H s e cr u o S Q RI
  23. 23. Thank you. Mario Smarduch Senior Virtualization Architect Open Source Group Samsung Research America (Silicon Valley) m.smarduch@samsung.com © 2013 SAMSUNG Electronics Co.

×