SlideShare a Scribd company logo
https://github.com/nuclio/nuclio
https://www.youtube.com/watch?v=xlOp9BR5xcs
Serverless for Real-Time Events and Data Processing
iguazio © 2016
2
Event
Listeners
Function Processors
Runtime
Function
Workers
Data
Bindings
Control, Logging, Monitoring, Security, ..
HTTP, stream
, msg Q, DB, ..
Pluggable
Data Services
Pluggable
Event Sources
Dealer BuilderController image
repo
Platform: Kubernetes, Cloud Provider, Device (IoT) .. Local or remote
nuclio - Comprehensive, Open, Portable and Super Fast “Serverless”
• Real-time processing, low CPU overhead and maximum parallelism
• Simple debugging, regression, and multi-versioned CI/CD pipeline
• Pluggable data/event sources with common APIs
• Portable across low-power devices, laptops, on-prem and public cloud
External
Monitoring
& Logging
Nuctl (CLI)
Playground
UI
https://github.com/nuclio/nuclio
iguazio © 2016
3
Function Processor
Event
Listeners
Fetch/Serve
events Language Runtime Engine
Function
Workers
Data
Bindings
Connect
& Cache
Control Framework: Portal, Logging, Monitoring, Security, …
Event Sources (Pluggable):
• Sync: HTTP
• Async: RabbitMQ, MQTT, NATS
• Stream: Kafka, Kinesis, v3io
• Polling: DB/file changes
Interface to platform resources through pluggable APIs Data Bindings (Pluggable):
• File & Obj: volumes, S3, v3io
• DB: DynamoDB, v3io
• Stream : Kafka, Kinesis, v3io
• Message: RabbitMQ
Develop, test, run ANYWHEREAny source and workload
Simple, fast, secure,
portable data integration
nuclio Processor – Fast, Modular and Extensible
400K events/sec per process (100x faster than leading implementations)
Super fast, Zero-copy access
to events and data
Multiple async workers for
maximum parallelism with
minimum CPU overhead
Events and data
abstractions enable
re-use and portability
iguazio © 2016
4
Perf Results, Single Process, Using Basic Functions
https://github.com/v3io/http_blaster
Tested using:
Native
Prometheus
Integration
iguazio © 2016
5
Nuclio Invocation Modes
Function
Instance
invoker
Message
Function
Instance
invoker
Function
Instance
invoker
Exchange
Message Queue
(e.g. RabbitMQ)
HTTP
API GW
Function
Instance
invoker
Function
Instance
invoker
Function
Instance
invoker
Req
Function
Instance
invoker
Function
Instance
invoker
Function
Instance
invokerPartition 1
Messages
Synchronous Req/Rep
Message Stream
Async Message Queue
Kafka,
Kinesis, …
Function
Instance
invoker
Function
Instance
invoker
Function
Instance
invoker
Job
Job (Master/Worker)
Priority
Queue
Master
(dealer)
Dealer
Partition 2
Partition 3
Partition 4
iguazio © 2016
6
Dealer
Processor
Function
Workers
Partitioned data or
Stream shards
Job Spec:
- functions (selector)
- Task num/list
- Max tasks per processor
- Min/Max processors
- ..
- Job Metadata
Function
Workers
Job X (w 5 tasks)Job Y (w 4 tasks)
POD Up/Down events
Deployment scale changes
Auto-scale based on
CPU load or Q delay
Allocate or Re-distribute tasks to
processors (1 task per worker)
Nuclio
controller
Nuclio Dealer
• Enable real-time stream processing, batch and
interactive jobs on auto-scaling Serverless functions
• By dynamically allocating tasks to workers and
handling task lifecycle, checkpoint and completion
Every job or stream is
partitioned to N smaller Tasks
Processor
iguazio © 2016
7
nuclio Features and Performance Make Serverless Broadly Applicable
• Enrich
• Aggregate
• Predict
INTERACTIVE UI &
REAL-TIME DASHBOARDS
ACTIONS
UNSTRUCTURED
EVENTS & DATA
EXTERNAL SOURCES
CHANGE
DATA CAPTURE (CDC)
OPERATIONAL DATA
CONTAINERIZED ML
& ANALYTICS TOOLS
Complex Event
Processing (CEP)
POLICY BASED
SYNC & BACKUP
DATA SERVICES
DATA INGESTION, PREP
& REAL-TIME DECISIONS
Higher-Productivity | Faster insights | No Infrastructure Hassle | Lower TCO
iguazio © 2016
8
Real Example: Event-Driven Analytics for Connected Cars
Geo Data
Weather/Road info
Vehicles Data
State
Changed?
Identify
Violation?
Drivers
Violations
Stream
State
Changes
Geo
Aggregate
Map
Process
Alerts
Process
Violations
External Sources
import
service
Enriched
Events
Parallel
Enrichment
ML Processing
Complex Events + Data processed in real-time without the infrastructure hassle
real-time, auto-Scaling
serverless functions
Model Update
Stats
Update
* See code in the
UI/Playground slide
iguazio © 2016
9
nuclio
Function Spec
Support Kubernetes CRD:
Functions can be created &
deleted using kubectl
tags/labels used for search and
event sources (Label Selectors)
Control Min/Max Replicas
for controlled auto-scale
Pass text or secret
environment variables
(k8s convention)
Flex resource allocation,
GPUs are coming
Pluggable Data Sources
Various src code options*: inline code, path
(local/http/git), or local/remote pre-built image
namespaced
*Advanced build instructions &
dependencies are in the build.yaml file
iguazio © 2016
10
Nuclio Common Event Model
Simplify and generalize
client implementation
Enable zero copy and zero
ser/des when possible
iguazio © 2016
11
Context.logger Interface
One log interface, multiple implementations (screen, file, stream, http, ..), extensible
Support both structured &
unstructured logging
Support nested/hierarchical logs
iguazio © 2016
12
Default Context.DataBinding API (sync & async ver), can be overwritten
Service Major APIs Main Request Params
Object
e.g. S3, Minio, v3io
ListObjects
GetObject
PutObject
DeleteObject
Bucket, Prefix, MaxKeys
Bucket, Key, Range
Bucket, Key ,Metadata, Body
Bucket, Key
NoSQL
e.g. DynamoDB,
Cassandra, v3io
GetItem
GetItems
PutItem
UpdateItem
DeleteItem
Table, Key ,Projection
Table, ConditionExpression, ProjectionExpression, Limit
Table, Key, ProjectionExpression, item
Table, Key, UpdateExpression, ConditionExpression
Table, Key, ConditionExpression
Stream
e.g. Kinesis, Kafka,
v3io
GetRecords
PutRecords
Seek
Stream, ShardId, Location, Limit
Stream, Records
Stream, ShardId, SeekType, SeekTime, StartingSequence, Timestamp
File
Open
Read
Write
Path, Mode, flags
Handle, offset, size
Handle, offset, size, data
iguazio © 2016
13
Nuclio Playground (run as isolated k8s deployment)
iguazio © 2016
14
CLI (run command example)
$ nuctl run --help
Build, deploy and run a function
Usage:
nuctl run function-name [flags]
Flags:
--data string Comma separated list of data bindings (in json)
--data-bindings string JSON encoded data bindings for the function
--desc string Function description
-d, --disabled Start function disabled (don't run yet)
-e, --env string Environment variables (name1=val1,name2=val2..)
--events string Comma separated list of event sources (in json)
-f, --file string Function Spec File
-h, --help help for run
-i, --image string Docker image name, will use function name if not specified
-l, --labels string Additional function labels (lbl1=val1,lbl2=val2..)
--max-replica int32 Maximum number of function replicas
--min-replica int32 Minimum number of function replicas
--no-pull Don't pull base images - use local versions
--nuclio-src-dir string Local directory with nuclio sources (avoid cloning)
--nuclio-src-url string nuclio sources url for git clone (default "https://github.com/nuclio/nuclio.git")
-o, --output string Build output type - docker|binary (default "docker")
-p, --path string Function source code path
--port int32 Public HTTP port (node port)
--publish Publish the function
-r, --registry string URL of container registry (env: NUCTL_REGISTRY)
--run-registry string The registry URL to pull the image from, if differs from -r (env: NUCTL_RUN_REGISTRY)
--runtime string Runtime – golang, python, ..
-s, --scale string Function scaling (auto|number) (default "1")
--version string Docker image version (default "latest")
Global Flags:
-k, --kubeconfig string Path to Kubernetes config (admin.conf) (default ~/.kube/config")
-n, --namespace string Kubernetes namespace (default "default")
-v, --verbose verbose output
See more in: https://github.com/nuclio/nuclio/blob/master/docs/nuctl/nuctl.md
iguazio © 2016
15
Data Bindings
$ nuctl run <name> <source> [options]
Enabling Simple and Continuous Dev and Ops (CI/CD)
One Click to test, deploy, upgrade or rollback code
Runs ANYWHERE, Self-healing and Auto-Scaling
LOCAL or CLOUD
iguazio © 2016
16
ORCHESTRATION SERVERLESS PROCESSING ML & AI FRAMEWORKS DATA SERVICES APIS
HYBRID DEPLOYMENT
NoSQL API Stream API Object API File API
Security Queries & Functions Unified Data Data Lifecycle
On-Premises Hosted Cloud Edge
Event Driven Code
• Used in iguazio’s platform
• Developed for the real world
• Now completely re-written to:
• Support the broader open
source & CNCF eco-system
• Incorporate learnings from G1
• Future proof architecture
• Address new use cases
 Low latency 100GbE TCP or RDMA Data Fabric (V3IO) 
UNIFIED &
AUTOMATED
MANAGEMENT
https://github.com/nuclio/nuclio

More Related Content

What's hot

Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
Johan Andersson
 
徹底解説 Unity Reflect【概要編 ver2.0】
徹底解説 Unity Reflect【概要編 ver2.0】徹底解説 Unity Reflect【概要編 ver2.0】
徹底解説 Unity Reflect【概要編 ver2.0】
Unity Technologies Japan K.K.
 
Game Engine Architecture
Game Engine ArchitectureGame Engine Architecture
Game Engine Architecture
Attila Jenei
 
「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出
「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出
「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出
小林 信行
 
CEDEC2017 アーティストのためのリアルタイムシェーダー学習法
CEDEC2017 アーティストのためのリアルタイムシェーダー学習法CEDEC2017 アーティストのためのリアルタイムシェーダー学習法
CEDEC2017 アーティストのためのリアルタイムシェーダー学習法
小林 信行
 
The Unique Lighting of Mirror's Edge
The Unique Lighting of Mirror's EdgeThe Unique Lighting of Mirror's Edge
The Unique Lighting of Mirror's Edge
Electronic Arts / DICE
 
HoloLensで外部定義ファイルを使う
HoloLensで外部定義ファイルを使うHoloLensで外部定義ファイルを使う
HoloLensで外部定義ファイルを使う
Takahiro Miyaura
 
データ収集の基本と「JapanTaxi」アプリにおける実践例
データ収集の基本と「JapanTaxi」アプリにおける実践例データ収集の基本と「JapanTaxi」アプリにおける実践例
データ収集の基本と「JapanTaxi」アプリにおける実践例
Tetsutaro Watanabe
 
Graphics Gems from CryENGINE 3 (Siggraph 2013)
Graphics Gems from CryENGINE 3 (Siggraph 2013)Graphics Gems from CryENGINE 3 (Siggraph 2013)
Graphics Gems from CryENGINE 3 (Siggraph 2013)
Tiago Sousa
 
Taking Killzone Shadow Fall Image Quality Into The Next Generation
Taking Killzone Shadow Fall Image Quality Into The Next GenerationTaking Killzone Shadow Fall Image Quality Into The Next Generation
Taking Killzone Shadow Fall Image Quality Into The Next Generation
Guerrilla
 
Rendering Tech of Space Marine
Rendering Tech of Space MarineRendering Tech of Space Marine
Rendering Tech of Space Marine
Pope Kim
 
Deferred shading
Deferred shadingDeferred shading
Deferred shading
Frank Chao
 
Siggraph2016 - The Devil is in the Details: idTech 666
Siggraph2016 - The Devil is in the Details: idTech 666Siggraph2016 - The Devil is in the Details: idTech 666
Siggraph2016 - The Devil is in the Details: idTech 666
Tiago Sousa
 
【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意
【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意
【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意
UnityTechnologiesJapan002
 
マシな画面を作る
マシな画面を作るマシな画面を作る
マシな画面を作る
okumasama
 
Unity In App Purchase (IAP)の使い方
Unity In App Purchase (IAP)の使い方Unity In App Purchase (IAP)の使い方
Unity In App Purchase (IAP)の使い方
Makoto Ito
 
The Real-time Volumetric Cloudscapes of Horizon Zero Dawn
The Real-time Volumetric Cloudscapes of Horizon Zero DawnThe Real-time Volumetric Cloudscapes of Horizon Zero Dawn
The Real-time Volumetric Cloudscapes of Horizon Zero Dawn
Guerrilla
 
Built for performance: the UIElements Renderer – Unite Copenhagen 2019
Built for performance: the UIElements Renderer – Unite Copenhagen 2019Built for performance: the UIElements Renderer – Unite Copenhagen 2019
Built for performance: the UIElements Renderer – Unite Copenhagen 2019
Unity Technologies
 
Frostbite on Mobile
Frostbite on MobileFrostbite on Mobile
Frostbite on Mobile
Electronic Arts / DICE
 
開発環境の認証を改善して Redmineを社内標準にした話
開発環境の認証を改善して Redmineを社内標準にした話開発環境の認証を改善して Redmineを社内標準にした話
開発環境の認証を改善して Redmineを社内標準にした話
Ryou Soda
 

What's hot (20)

Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
Terrain Rendering in Frostbite using Procedural Shader Splatting (Siggraph 2007)
 
徹底解説 Unity Reflect【概要編 ver2.0】
徹底解説 Unity Reflect【概要編 ver2.0】徹底解説 Unity Reflect【概要編 ver2.0】
徹底解説 Unity Reflect【概要編 ver2.0】
 
Game Engine Architecture
Game Engine ArchitectureGame Engine Architecture
Game Engine Architecture
 
「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出
「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出
「ユニティちゃんを踊らせよう!」モーションキャプチャーデータのアニメーション演出
 
CEDEC2017 アーティストのためのリアルタイムシェーダー学習法
CEDEC2017 アーティストのためのリアルタイムシェーダー学習法CEDEC2017 アーティストのためのリアルタイムシェーダー学習法
CEDEC2017 アーティストのためのリアルタイムシェーダー学習法
 
The Unique Lighting of Mirror's Edge
The Unique Lighting of Mirror's EdgeThe Unique Lighting of Mirror's Edge
The Unique Lighting of Mirror's Edge
 
HoloLensで外部定義ファイルを使う
HoloLensで外部定義ファイルを使うHoloLensで外部定義ファイルを使う
HoloLensで外部定義ファイルを使う
 
データ収集の基本と「JapanTaxi」アプリにおける実践例
データ収集の基本と「JapanTaxi」アプリにおける実践例データ収集の基本と「JapanTaxi」アプリにおける実践例
データ収集の基本と「JapanTaxi」アプリにおける実践例
 
Graphics Gems from CryENGINE 3 (Siggraph 2013)
Graphics Gems from CryENGINE 3 (Siggraph 2013)Graphics Gems from CryENGINE 3 (Siggraph 2013)
Graphics Gems from CryENGINE 3 (Siggraph 2013)
 
Taking Killzone Shadow Fall Image Quality Into The Next Generation
Taking Killzone Shadow Fall Image Quality Into The Next GenerationTaking Killzone Shadow Fall Image Quality Into The Next Generation
Taking Killzone Shadow Fall Image Quality Into The Next Generation
 
Rendering Tech of Space Marine
Rendering Tech of Space MarineRendering Tech of Space Marine
Rendering Tech of Space Marine
 
Deferred shading
Deferred shadingDeferred shading
Deferred shading
 
Siggraph2016 - The Devil is in the Details: idTech 666
Siggraph2016 - The Devil is in the Details: idTech 666Siggraph2016 - The Devil is in the Details: idTech 666
Siggraph2016 - The Devil is in the Details: idTech 666
 
【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意
【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意
【Unite Tokyo 2018】『崩壊3rd』開発者が語るアニメ風レンダリングの極意
 
マシな画面を作る
マシな画面を作るマシな画面を作る
マシな画面を作る
 
Unity In App Purchase (IAP)の使い方
Unity In App Purchase (IAP)の使い方Unity In App Purchase (IAP)の使い方
Unity In App Purchase (IAP)の使い方
 
The Real-time Volumetric Cloudscapes of Horizon Zero Dawn
The Real-time Volumetric Cloudscapes of Horizon Zero DawnThe Real-time Volumetric Cloudscapes of Horizon Zero Dawn
The Real-time Volumetric Cloudscapes of Horizon Zero Dawn
 
Built for performance: the UIElements Renderer – Unite Copenhagen 2019
Built for performance: the UIElements Renderer – Unite Copenhagen 2019Built for performance: the UIElements Renderer – Unite Copenhagen 2019
Built for performance: the UIElements Renderer – Unite Copenhagen 2019
 
Frostbite on Mobile
Frostbite on MobileFrostbite on Mobile
Frostbite on Mobile
 
開発環境の認証を改善して Redmineを社内標準にした話
開発環境の認証を改善して Redmineを社内標準にした話開発環境の認証を改善して Redmineを社内標準にした話
開発環境の認証を改善して Redmineを社内標準にした話
 

Similar to nuclio Overview October 2017

Running High-Speed Serverless with nuclio
Running High-Speed Serverless with nuclioRunning High-Speed Serverless with nuclio
Running High-Speed Serverless with nuclio
iguazio
 
iguazio - nuclio Meetup Nov 30th
iguazio - nuclio Meetup Nov 30thiguazio - nuclio Meetup Nov 30th
iguazio - nuclio Meetup Nov 30th
iguazio
 
Automated Application Management with SaltStack
Automated Application Management with SaltStackAutomated Application Management with SaltStack
Automated Application Management with SaltStack
inovex GmbH
 
Kubernetes for the PHP developer
Kubernetes for the PHP developerKubernetes for the PHP developer
Kubernetes for the PHP developer
Paul Czarkowski
 
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Thomas Weise
 
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Puppet
 
StrongLoop Overview
StrongLoop OverviewStrongLoop Overview
StrongLoop Overview
Shubhra Kar
 
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsSpark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka Streams
Guido Schmutz
 
FIWARE Wednesday Webinars - Short Term History within Smart Systems
FIWARE Wednesday Webinars - Short Term History within Smart SystemsFIWARE Wednesday Webinars - Short Term History within Smart Systems
FIWARE Wednesday Webinars - Short Term History within Smart Systems
FIWARE
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Guido Schmutz
 
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & PackerLAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
Jan-Christoph Küster
 
Saltstack - Orchestration & Application Deployment
Saltstack - Orchestration & Application DeploymentSaltstack - Orchestration & Application Deployment
Saltstack - Orchestration & Application Deployment
inovex GmbH
 
Apache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's NextApache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's Next
Prateek Maheshwari
 
Making cloud native platform by kubernetes
Making cloud native platform by kubernetesMaking cloud native platform by kubernetes
Making cloud native platform by kubernetes
어형 이
 
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Pulsar summit asia 2021   apache pulsar with mqtt for edge computingPulsar summit asia 2021   apache pulsar with mqtt for edge computing
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Timothy Spann
 
Vert.x devoxx london 2013
Vert.x devoxx london 2013Vert.x devoxx london 2013
Vert.x devoxx london 2013
Stuart (Pid) Williams
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to Kubernetes
Paul Czarkowski
 
Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4
Timothy Spann
 
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Zabbix
 
Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...
Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...
Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...
confluent
 

Similar to nuclio Overview October 2017 (20)

Running High-Speed Serverless with nuclio
Running High-Speed Serverless with nuclioRunning High-Speed Serverless with nuclio
Running High-Speed Serverless with nuclio
 
iguazio - nuclio Meetup Nov 30th
iguazio - nuclio Meetup Nov 30thiguazio - nuclio Meetup Nov 30th
iguazio - nuclio Meetup Nov 30th
 
Automated Application Management with SaltStack
Automated Application Management with SaltStackAutomated Application Management with SaltStack
Automated Application Management with SaltStack
 
Kubernetes for the PHP developer
Kubernetes for the PHP developerKubernetes for the PHP developer
Kubernetes for the PHP developer
 
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
 
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
 
StrongLoop Overview
StrongLoop OverviewStrongLoop Overview
StrongLoop Overview
 
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsSpark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka Streams
 
FIWARE Wednesday Webinars - Short Term History within Smart Systems
FIWARE Wednesday Webinars - Short Term History within Smart SystemsFIWARE Wednesday Webinars - Short Term History within Smart Systems
FIWARE Wednesday Webinars - Short Term History within Smart Systems
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
 
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & PackerLAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
LAMP Stack (Reloaded) - Infrastructure as Code with Terraform & Packer
 
Saltstack - Orchestration & Application Deployment
Saltstack - Orchestration & Application DeploymentSaltstack - Orchestration & Application Deployment
Saltstack - Orchestration & Application Deployment
 
Apache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's NextApache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's Next
 
Making cloud native platform by kubernetes
Making cloud native platform by kubernetesMaking cloud native platform by kubernetes
Making cloud native platform by kubernetes
 
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Pulsar summit asia 2021   apache pulsar with mqtt for edge computingPulsar summit asia 2021   apache pulsar with mqtt for edge computing
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
 
Vert.x devoxx london 2013
Vert.x devoxx london 2013Vert.x devoxx london 2013
Vert.x devoxx london 2013
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to Kubernetes
 
Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4
 
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016
 
Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...
Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...
Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...
 

More from iguazio

Accelerating Data Science With GPUs
Accelerating Data Science With GPUsAccelerating Data Science With GPUs
Accelerating Data Science With GPUs
iguazio
 
Challenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in ProductionChallenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in Production
iguazio
 
Webinar: Cutting Time, Complexity and Cost from Data Science to Production
Webinar: Cutting Time, Complexity and Cost from Data Science to ProductionWebinar: Cutting Time, Complexity and Cost from Data Science to Production
Webinar: Cutting Time, Complexity and Cost from Data Science to Production
iguazio
 
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
iguazio
 
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
iguazio
 
Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018
Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018
Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018
iguazio
 
Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...
Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...
Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...
iguazio
 
The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018
The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018
The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018
iguazio
 
Stac summit june 14th - goodbye datalakes
Stac summit june 14th - goodbye datalakesStac summit june 14th - goodbye datalakes
Stac summit june 14th - goodbye datalakes
iguazio
 

More from iguazio (9)

Accelerating Data Science With GPUs
Accelerating Data Science With GPUsAccelerating Data Science With GPUs
Accelerating Data Science With GPUs
 
Challenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in ProductionChallenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in Production
 
Webinar: Cutting Time, Complexity and Cost from Data Science to Production
Webinar: Cutting Time, Complexity and Cost from Data Science to ProductionWebinar: Cutting Time, Complexity and Cost from Data Science to Production
Webinar: Cutting Time, Complexity and Cost from Data Science to Production
 
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
 
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
 
Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018
Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018
Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018
 
Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...
Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...
Serverless real use cases and best practices: Asavari Tayal, Microsoft, Serve...
 
The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018
The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018
The Serverless Native Mindset: Ben Kehoe, iRobot, Serverless NYC 2018
 
Stac summit june 14th - goodbye datalakes
Stac summit june 14th - goodbye datalakesStac summit june 14th - goodbye datalakes
Stac summit june 14th - goodbye datalakes
 

Recently uploaded

"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
HackersList
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Kunal Gupta
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 

Recently uploaded (20)

"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 

nuclio Overview October 2017

  • 2. iguazio © 2016 2 Event Listeners Function Processors Runtime Function Workers Data Bindings Control, Logging, Monitoring, Security, .. HTTP, stream , msg Q, DB, .. Pluggable Data Services Pluggable Event Sources Dealer BuilderController image repo Platform: Kubernetes, Cloud Provider, Device (IoT) .. Local or remote nuclio - Comprehensive, Open, Portable and Super Fast “Serverless” • Real-time processing, low CPU overhead and maximum parallelism • Simple debugging, regression, and multi-versioned CI/CD pipeline • Pluggable data/event sources with common APIs • Portable across low-power devices, laptops, on-prem and public cloud External Monitoring & Logging Nuctl (CLI) Playground UI https://github.com/nuclio/nuclio
  • 3. iguazio © 2016 3 Function Processor Event Listeners Fetch/Serve events Language Runtime Engine Function Workers Data Bindings Connect & Cache Control Framework: Portal, Logging, Monitoring, Security, … Event Sources (Pluggable): • Sync: HTTP • Async: RabbitMQ, MQTT, NATS • Stream: Kafka, Kinesis, v3io • Polling: DB/file changes Interface to platform resources through pluggable APIs Data Bindings (Pluggable): • File & Obj: volumes, S3, v3io • DB: DynamoDB, v3io • Stream : Kafka, Kinesis, v3io • Message: RabbitMQ Develop, test, run ANYWHEREAny source and workload Simple, fast, secure, portable data integration nuclio Processor – Fast, Modular and Extensible 400K events/sec per process (100x faster than leading implementations) Super fast, Zero-copy access to events and data Multiple async workers for maximum parallelism with minimum CPU overhead Events and data abstractions enable re-use and portability
  • 4. iguazio © 2016 4 Perf Results, Single Process, Using Basic Functions https://github.com/v3io/http_blaster Tested using: Native Prometheus Integration
  • 5. iguazio © 2016 5 Nuclio Invocation Modes Function Instance invoker Message Function Instance invoker Function Instance invoker Exchange Message Queue (e.g. RabbitMQ) HTTP API GW Function Instance invoker Function Instance invoker Function Instance invoker Req Function Instance invoker Function Instance invoker Function Instance invokerPartition 1 Messages Synchronous Req/Rep Message Stream Async Message Queue Kafka, Kinesis, … Function Instance invoker Function Instance invoker Function Instance invoker Job Job (Master/Worker) Priority Queue Master (dealer) Dealer Partition 2 Partition 3 Partition 4
  • 6. iguazio © 2016 6 Dealer Processor Function Workers Partitioned data or Stream shards Job Spec: - functions (selector) - Task num/list - Max tasks per processor - Min/Max processors - .. - Job Metadata Function Workers Job X (w 5 tasks)Job Y (w 4 tasks) POD Up/Down events Deployment scale changes Auto-scale based on CPU load or Q delay Allocate or Re-distribute tasks to processors (1 task per worker) Nuclio controller Nuclio Dealer • Enable real-time stream processing, batch and interactive jobs on auto-scaling Serverless functions • By dynamically allocating tasks to workers and handling task lifecycle, checkpoint and completion Every job or stream is partitioned to N smaller Tasks Processor
  • 7. iguazio © 2016 7 nuclio Features and Performance Make Serverless Broadly Applicable • Enrich • Aggregate • Predict INTERACTIVE UI & REAL-TIME DASHBOARDS ACTIONS UNSTRUCTURED EVENTS & DATA EXTERNAL SOURCES CHANGE DATA CAPTURE (CDC) OPERATIONAL DATA CONTAINERIZED ML & ANALYTICS TOOLS Complex Event Processing (CEP) POLICY BASED SYNC & BACKUP DATA SERVICES DATA INGESTION, PREP & REAL-TIME DECISIONS Higher-Productivity | Faster insights | No Infrastructure Hassle | Lower TCO
  • 8. iguazio © 2016 8 Real Example: Event-Driven Analytics for Connected Cars Geo Data Weather/Road info Vehicles Data State Changed? Identify Violation? Drivers Violations Stream State Changes Geo Aggregate Map Process Alerts Process Violations External Sources import service Enriched Events Parallel Enrichment ML Processing Complex Events + Data processed in real-time without the infrastructure hassle real-time, auto-Scaling serverless functions Model Update Stats Update * See code in the UI/Playground slide
  • 9. iguazio © 2016 9 nuclio Function Spec Support Kubernetes CRD: Functions can be created & deleted using kubectl tags/labels used for search and event sources (Label Selectors) Control Min/Max Replicas for controlled auto-scale Pass text or secret environment variables (k8s convention) Flex resource allocation, GPUs are coming Pluggable Data Sources Various src code options*: inline code, path (local/http/git), or local/remote pre-built image namespaced *Advanced build instructions & dependencies are in the build.yaml file
  • 10. iguazio © 2016 10 Nuclio Common Event Model Simplify and generalize client implementation Enable zero copy and zero ser/des when possible
  • 11. iguazio © 2016 11 Context.logger Interface One log interface, multiple implementations (screen, file, stream, http, ..), extensible Support both structured & unstructured logging Support nested/hierarchical logs
  • 12. iguazio © 2016 12 Default Context.DataBinding API (sync & async ver), can be overwritten Service Major APIs Main Request Params Object e.g. S3, Minio, v3io ListObjects GetObject PutObject DeleteObject Bucket, Prefix, MaxKeys Bucket, Key, Range Bucket, Key ,Metadata, Body Bucket, Key NoSQL e.g. DynamoDB, Cassandra, v3io GetItem GetItems PutItem UpdateItem DeleteItem Table, Key ,Projection Table, ConditionExpression, ProjectionExpression, Limit Table, Key, ProjectionExpression, item Table, Key, UpdateExpression, ConditionExpression Table, Key, ConditionExpression Stream e.g. Kinesis, Kafka, v3io GetRecords PutRecords Seek Stream, ShardId, Location, Limit Stream, Records Stream, ShardId, SeekType, SeekTime, StartingSequence, Timestamp File Open Read Write Path, Mode, flags Handle, offset, size Handle, offset, size, data
  • 13. iguazio © 2016 13 Nuclio Playground (run as isolated k8s deployment)
  • 14. iguazio © 2016 14 CLI (run command example) $ nuctl run --help Build, deploy and run a function Usage: nuctl run function-name [flags] Flags: --data string Comma separated list of data bindings (in json) --data-bindings string JSON encoded data bindings for the function --desc string Function description -d, --disabled Start function disabled (don't run yet) -e, --env string Environment variables (name1=val1,name2=val2..) --events string Comma separated list of event sources (in json) -f, --file string Function Spec File -h, --help help for run -i, --image string Docker image name, will use function name if not specified -l, --labels string Additional function labels (lbl1=val1,lbl2=val2..) --max-replica int32 Maximum number of function replicas --min-replica int32 Minimum number of function replicas --no-pull Don't pull base images - use local versions --nuclio-src-dir string Local directory with nuclio sources (avoid cloning) --nuclio-src-url string nuclio sources url for git clone (default "https://github.com/nuclio/nuclio.git") -o, --output string Build output type - docker|binary (default "docker") -p, --path string Function source code path --port int32 Public HTTP port (node port) --publish Publish the function -r, --registry string URL of container registry (env: NUCTL_REGISTRY) --run-registry string The registry URL to pull the image from, if differs from -r (env: NUCTL_RUN_REGISTRY) --runtime string Runtime – golang, python, .. -s, --scale string Function scaling (auto|number) (default "1") --version string Docker image version (default "latest") Global Flags: -k, --kubeconfig string Path to Kubernetes config (admin.conf) (default ~/.kube/config") -n, --namespace string Kubernetes namespace (default "default") -v, --verbose verbose output See more in: https://github.com/nuclio/nuclio/blob/master/docs/nuctl/nuctl.md
  • 15. iguazio © 2016 15 Data Bindings $ nuctl run <name> <source> [options] Enabling Simple and Continuous Dev and Ops (CI/CD) One Click to test, deploy, upgrade or rollback code Runs ANYWHERE, Self-healing and Auto-Scaling LOCAL or CLOUD
  • 16. iguazio © 2016 16 ORCHESTRATION SERVERLESS PROCESSING ML & AI FRAMEWORKS DATA SERVICES APIS HYBRID DEPLOYMENT NoSQL API Stream API Object API File API Security Queries & Functions Unified Data Data Lifecycle On-Premises Hosted Cloud Edge Event Driven Code • Used in iguazio’s platform • Developed for the real world • Now completely re-written to: • Support the broader open source & CNCF eco-system • Incorporate learnings from G1 • Future proof architecture • Address new use cases  Low latency 100GbE TCP or RDMA Data Fabric (V3IO)  UNIFIED & AUTOMATED MANAGEMENT https://github.com/nuclio/nuclio