Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
M a r c h 2 0 1 7
Building Super Fast Cloud-Native Platforms
Yaron Haviv, CTO, iguazio
@yaronhaviv
1
iguazio © 2017
2
Building A High-Performance Cloud-Native Data Platform
Fast Data and API Gateways REST API and Web UI
Dat...
iguazio © 2017
3
Redefining the Stack, Delivering Magnitudes-Faster Performance
8
iguazio © 2017
4
High-Performance Requires Careful Hardware Integration
V3IO
Cap’n Proto
+ Accelio*
(OS Bypass)
2 x 100GbE...
iguazio © 2017
5
Challenges with Containers and Kubernetes
• Kubernetes Limitations/ Challenges
– Only one IF per POD, usu...
iguazio © 2017
6
Container Networking 101
Container Networking Options
• SR-IOV allows native hardware access
• Multus ena...
iguazio © 2017
7
Stateless Apps with Fastest Unified Data Access
• Fastest messaging/ DB/obj access
– Like Go channels acr...
iguazio © 2017
8
Today: Server-Less is Cool, But Inefficient
• 1st generation is slow to init and a resource drain
– https...
iguazio © 2017
9
Building Server-Less on Steroids
API
Gateways
API GW
(ingress)
Controller
Function
instance
Function
Inst...
iguazio © 2017
10
Example: Simple HTTP Function
Built-in log stream
Data binding and credentials in
function context
(simp...
iguazio © 2017
11
High-Speed “Server-Less” Data Processing, Everything is a Stream
Select name, mtime, f1, f2, …
Where con...
iguazio © 2017
12
Example: Scanning for Sensitive Text Files on Upload/ Update
Init part, e.g. define RegEx filters
Data/m...
iguazio © 2017
13
Image Example, Leveraging a Unified Data Model
Access the objects via file
semantics, mount and share
ha...
iguazio © 2017
14
Connecting The Dots: Continuous Analytics Example
iguazio © 2017
15
Kubernetes Helps us Simplify & Accelerate Analytics at the Edge
Sources
Event Driven Code
Analytics Fram...
Backup
iguazio © 2017
17
Generic Data Services Binding API
Service Major APIs
Object
e.g. S3, Minio,
v3io
ListBucket(prefix strin...
Upcoming SlideShare
Loading in …5
×

Building Super Fast Cloud-Native Data Platforms - Yaron Haviv, KubeCon 2017 EU

1,754 views

Published on

Deep dive into iguazio high-performance data platform architecture, using Kubernetes and Cloud-Native for elasticity and CI/CD, along with with extreme performance tricks

YouTube link: https://youtu.be/ujuWt6mvIig

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building Super Fast Cloud-Native Data Platforms - Yaron Haviv, KubeCon 2017 EU

  1. 1. M a r c h 2 0 1 7 Building Super Fast Cloud-Native Platforms Yaron Haviv, CTO, iguazio @yaronhaviv 1
  2. 2. iguazio © 2017 2 Building A High-Performance Cloud-Native Data Platform Fast Data and API Gateways REST API and Web UI Data Processing and Caching Elastic Platform Services Redundant 100GbE Fabric Stateless Load-Balancing (for Incoming Requests) Multi Site External Notifications DBMQ Monitor… RegistryLog Identity LDAP/SSOObject Storage (Archive) Fast Native Access “Server-Less”
  3. 3. iguazio © 2017 3 Redefining the Stack, Delivering Magnitudes-Faster Performance 8
  4. 4. iguazio © 2017 4 High-Performance Requires Careful Hardware Integration V3IO Cap’n Proto + Accelio* (OS Bypass) 2 x 100GbE Fabric K/V Services VN Services 8-24 x NVMe DirectSSL & Overlay Offload Shared Mem Lock-Free Qs Web API Nodes Data Nodes Web APIs S3, Kinesis, DynamoDB... NV Mem Serve 800K** Web Req/Sec Serve 2M Ops/Sec and 100Gbps 0.1ms Latency @ 99% (percentile) 1 proc /core1 proc /tenant 1 vNIC/ tenant ** Tested with: https://github.com/v3io/http_blaster * Accelio: https://github.com/v3io/accelio
  5. 5. iguazio © 2017 5 Challenges with Containers and Kubernetes • Kubernetes Limitations/ Challenges – Only one IF per POD, usually going through a slow overlay layer – No native support for HW NICs (SR-IOV) – Hard to expose low level drivers/ libraries to container – Hard to use shared memory IPC/ files between PODs – Docker and Kubernetes are different (security, net, volume, shmem, ..) • Solution – Custom network (CNI) and volume drivers – Use granular privileges – Many trials and errors 
  6. 6. iguazio © 2017 6 Container Networking 101 Container Networking Options • SR-IOV allows native hardware access • Multus enable multiple IFs per POD: https://github.com/ Intel-Corp/multus-cni • More details on my blog https:// thenewstack.io/ hackers-guide-kubernetes-networking/
  7. 7. iguazio © 2017 7 Stateless Apps with Fastest Unified Data Access • Fastest messaging/ DB/obj access – Like Go channels across processes – Lock-free, async, parallel – Zero-copy end to end – + Native Spark DataFrame API • Fast container initialization – No TCP/ IP connections init – No memory alloc/register • Share TCP/ RDMA connections • NO Kernel drivers/changes ! V3IOd (DB, stream, ..) Fuse++ (file mounts) Fuse++ - Applications Services Other Apps Orchestrated via FlexVolume Driver Serverless SDK/DFSDK Fast Data IPC Cap’n Proto + Lock-free shmem queues (dev/shm) Modified fuse lib to run async, 15x faster (~100K IOPs/thread) See: https://github.com/v3io/libfuse 50/100 GbE Accelerate Performance Using Shared Memory 100% Stateless Apps
  8. 8. iguazio © 2017 8 Today: Server-Less is Cool, But Inefficient • 1st generation is slow to init and a resource drain – https://medium.com/ @ferdingler/aws-lambda-no-thank-you-9c586990e67d • Complex and unsecured data bindings (performed in the init part of the function) • TCP/ IP or DB connections may need to re-establish on every invocation • Slow, limited concurrency, runs one task at a time per container • Events structure has no common schema (see: http://docs.aws.amazon.com/ lambda/ latest/dg/eventsources.html) Source, my blog: https://medium.com/@yaronhaviv/serverless-background-challenges-and-future-d0928df71758
  9. 9. iguazio © 2017 9 Building Server-Less on Steroids API Gateways API GW (ingress) Controller Function instance Function Instance (N workers) invoker Async calls • Objects • Files • Tables • Streams 1 update for N invocations to save IO 1 hop, low latency msg Create Del Scale Data Changes, Streaming * Iguazio’s server-less framework will be open sourced later this year User defined micro-batch size and parallelism (N Go routines per function) Credit based rate limiter Workers State and Stats Control Data Distributed Log Stream Cut overhead, add parallelism and concurrency, without violating isolation F PODs NGINX PODs
  10. 10. iguazio © 2017 10 Example: Simple HTTP Function Built-in log stream Data binding and credentials in function context (simple, fast, reusable and secure) See Data class APIs in backup slide, allow integrating with various sources
  11. 11. iguazio © 2017 11 High-Speed “Server-Less” Data Processing, Everything is a Stream Select name, mtime, f1, f2, … Where condition1, 2, 3, .. StatefulSet* Transactional (exactly once) update of object metadata e.g. object last backup/scan time Control Data Task stats, state, and logs Scheduled or Continuous Metadata Queries Example Tasks: • incremental backup modifies objects/records to Amazon S3 • Scan/convert content • Data/Metadata stream is sharded consistently to N stateless workers • Using StatefulSets to maintain consistent shard index id (derived from POD name) • Tables • Streams • Objects • Files Stream/Push matching Metadata & Data
  12. 12. iguazio © 2017 12 Example: Scanning for Sensitive Text Files on Upload/ Update Init part, e.g. define RegEx filters Data/metadata PUSH to reduce latency 1-N records per call Async and micro-batch updates of object/record/file attributes Function can use distributed task counters Update data/attrs in one ATOMIC transaction
  13. 13. iguazio © 2017 13 Image Example, Leveraging a Unified Data Model Access the objects via file semantics, mount and share handled automatically Thumbnail stored as a small blub record on the same object for quick UI access Use standard file access
  14. 14. iguazio © 2017 14 Connecting The Dots: Continuous Analytics Example
  15. 15. iguazio © 2017 15 Kubernetes Helps us Simplify & Accelerate Analytics at the Edge Sources Event Driven Code Analytics FrameworksServer-Less Processing 18 Queries & Functions Security Data Lifecycle Unified Data File APIObject APIStream APINoSQL API iguazio Data Platform Cloud Aggregation Insights 100% Stateless Apps
  16. 16. Backup
  17. 17. iguazio © 2017 17 Generic Data Services Binding API Service Major APIs Object e.g. S3, Minio, v3io ListBucket(prefix string) (ListBucketResp, error) Get(path string, ranges ...Range) ([]byte, error) Put(path string, body []byte ...) ([]byte, error) Del(path string) (error) NoSQL e.g. DynamoDB, Cassandra, v3io GetItem(path, attrs string) (GetItemResp, error) GetItems(path, attrs, filter, marker string, ...) (GetItemsResp, error) PutItem(path string, list map[string]interface{}, condition string) ([]byte, error) UpdateItem(path string, updatestr string, condition string) ([]byte, error) DelItem(path string, condition string) (error) Stream e.g. Kinesis, Kafka, v3io GetRecords(path, offset string, maxrec int) (GetRecordsResp, error) PutRecords(path string, records []StreamRecord) ([]byte, error) Seek(path string, seek string) (string, error) File Open(name string, flag int, perm FileMode) (*File, error)

×