Future of Integration Microsoft. Focus sur Azure Event Hub et Stream Analytics

1. #JSS2014
Les journées
SQL Server 2014
Un événement organisé par GUSS

2. #JSS2014
Les journées
SQL Server 2014
Un événement organisé par GUSS
Session « Le futur de l’intégration »
Michel Hubert – MVP Azure
Radoine Douhou

3. #JSS2014
• Etat des lieux.
• Futur de l’intégration.
• Solutions d’intégration orientées Cloud.
• Solutions d’intégration orientées Events & IoT.
• Démo.
Agenda

4. #JSS2014
Evolution of Integration
Once upon a Time
Point to point
(“Spaghetti”)
Integration broker
(EAI/B2B)
Enterprise Service Bus
HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP

5. #JSS2014
BizTalk Server
Partners
Evolution of Integration
Microsoft BizTalk Server

6. #JSS2014
BizTalk Server ESB toolkit
e-Commerce
Suppliers
EDI
EDIFACT / AS2
WS
RESTFul
Market place
warehouse
Magasins
SI Back Office
Orchestration B2B
BI
SSIS
SSRS
SSAS
Microsoft BizTalk Server
Common scenario

7. #JSS2014
FUTUR DE L’INTEGRATION
Nouveaux usages

8. #JSS2014
Point to point
(“Spaghetti”)
Integration broker
(EAI/B2B)
Enterprise Service Bus What’s next?
?HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP
Future of Integration
What’s next ?

9. #JSS2014
Future of
Integration
Mobiles
Events
Messaging
EAI/ESB
Business Process
Automation
Telemetry
Analytics
BizTalk Server
BizTalk Services
Service Bus
Queuing
Events Hub
Stream Analytics
Hybrid Connection
B2B
Future of Integration
Integration = Cloud + Mobile + IoT
IoT
SaaS

10. #JSS2014
SOLUTIONS D’INTEGRATION ORIENTEES
CLOUD
BizTalk Server / BizTalk Services

11. #JSS2014
Upgrade to BizTalk Server 2013 R2
with confidence
Accelerate vertical solutions with
standards and platform support
Extend existing solutions to Azure
Service Bus Adapters
RESTFul/Json Adapter
BizTalk Server 2013 R2
Cloud Ready

12. #JSS2014
BizTalk Server
Microsoft Azure
Partners
BizTalk Services
Service Bus
Future of Integration
Federated Enterprise Service Bus

13. #JSS2014
Upgrade to BizTalk Server 2013 R2
with confidence
Accelerate vertical solutions with
standards and platform support
Extend existing solutions to Azure
Develop and test integration
solutions in the cloud without re-
writes
Clear up on-premises infrastructure
capacity and provision in minutes
instead of weeks
Service Bus Adapters
RESTFul/Json Adapter
BizTalk Server 2013
Virtual Machines
BizTalk Server 2013 R2
Cloud Ready

14. #JSS2014
App owner Datacenter
admin
Customer
data center
Microsoft Azure data center
LOB app Active Directory
BizTalk LOB app
SQL
Agility for app owners Control for IT pros
BizTalk Server 2013 R2
Develop in Azure

15. #JSS2014
Upgrade to BizTalk Server 2013 R2
with confidence
Accelerate vertical solutions with
standards and platform support
Extend existing solutions to Azure
Develop and test integration
solutions in the cloud without re-
writes
Clear up on-premises infrastructure
capacity and provision in minutes
instead of weeks
Enable hybrid extension of current
on-premises BizTalk Server
deployments with Azure BizTalk
Services
Power new cloud-hosted business-
to-business, Internet of Things, and
EDI capabilities
Service Bus Adapters
RESTFul/Json Adapter
BizTalk Server 2013
Virtual Machines
Microsoft Azure
BizTalk Services
BizTalk Server 2013 R2
Cloud Ready

16. #JSS2014
Microsoft Azure BizTalk Services (MABS)

17. #JSS2014
Equivalent to BizTalk ESB itinerary Transforms
Enrich
Validation XML and xsd schema
MABS
Bridge principle

18. #JSS2014
Hybrid Connections
Web Sites
Mobile Services
Corporate network
Microsoft SQL Server
Hybrid Connection
Other published
resourcesHybrid Connection
Manager

19. #JSS2014
Use Cases
Microsoft Azure
On-premises
AS2 SFTP HTTP
? !
Home office Branch office Regional
office

20. #JSS2014
SOLUTIONS D’INTEGRATION ORIENTEES
EVENTS / IOT
Service Bus, Event Hub, Stream Analytics

21. #JSS2014
Azure Service Bus
Relay
Queue
Topic
Notification Hub
Event Hub
NAT and Firewall Traversal Service
Request/Response Services
Unbuffered with TCP Throttling
Many publishers and many consumers to
communicate over a FIFO like channel.
(Competing consumers and Queue-based
Load leveling scenarios)
Pub / Sub communication channel. Each
Consumer subscribes to a copy of message
High-scale notification distribution
Most mobile push notification services
Millions of notification targets
Azure Service Bus
Cloud Messaging Broker

22. #JSS2014
Service Bus Events Hub
Event
Producers
Azure Event Hub
> 1M Producers
> 1GB/sec
Aggregate
Throughput
Up to 32 partitions via
portal, more on
request
Partitions
Direct
PartitionKey
Hash
Throughput Units:
• 1 ≤ TUs ≤ Partition Count
• TU: 1 MB/s writes, 2 MB/s reads
Consumer
Group(s)
Receivers
AMQP 1.0
Credit-based flow control
Client-side cursors
Offset by Id or Timestamp

23. #JSS2014
Event
Producers
> 1M Producers
> 1GB/sec
Aggregate
Throughput
Direct
PartitionKey
Hash
• Consistent security model with Service Bus,
extended by publisher policies
• Publisher protocols :
– HTTPS : Short lived, low throughput :
– AMQP : Long lived, High throughput :
• Publish modes :
– Directely to a PartitionId.
– Automatic hash-based distribution by
PartitionKey or Publisher Identity
• Stream or batch
Event Hub
Producers

24. #JSS2014
• 1 Partition : ordered sequence of events
that is held. A partition can be thought of
as a “commit log.”
• Segmentation of the event stream for
scale-out :
– Parallelism on both Publishing and Polling
sides.
• Default 16, minimum 8, self-service maximum
32 :
– 1024 partitions via Azure Support.
Azure Event Hub
Partitions
Event Hub
Partition principle

25. #JSS2014
• view (state, position, or offset) of an entire
Event Hub
• Publish/subscribe mechanism of Event
Hubs is enabled through consumer groups.
• Enable multiple consuming applications to
each have a separate view of the event
stream, and to read the stream
independently at its own pace and with its
own offsets
Consumer
Group(s)
Event Hub
Consumer Group

26. #JSS2014
• Receive from partitions via consumer
groups
• Client-side cursors allowing to freely
process and repeatedly reprocess the
available retained event stream based on
offsets or timestamps.
• Using .NET API or using generic AMQP 1.0
client (e.g. Apache Proton-C/J)
ID, Time, [Data]
ID, Time, [Data]
ID, Time, [Data]
Time
Id
Event Hub
Consumers

27. #JSS2014
Stream Analytics
Data Source
Collect Process
Consume
Deliver
Event Inputs
- Event Hub
- Azure Blob
Transform
- Temporal joins
- Filter
- Aggregates
- Projections
- Windows
- Etc.
Enrich
Correlate
Outputs
- SQL Azure
- Azure Blobs
- Event Hub
☁
BI
Dashboards
Predictive
Analytics
Azure
Storage
• Temporal Semantics
• Guaranteed delivery
• Guaranteed up time
Azure Stream Analytics
Reference Data
- Azure Blob

28. #JSS2014
• Easily filter, project,
aggregate, join streams,
add static data with
streaming data, detect
patterns or lack of
patterns with a few lines
of SQL
• Development and
debugging experience
through Azure Portal
• Built-in monitoring
trough Azure Portal
• No hardware acquisition
and maintenance
• Up and running in a few
clicks (and within
minutes) => Bypasses
deployment expertise
• Elasticity of Azure for
scale up or scale down
• Distributed, scale-out
architecture
• Integrated with highly-
scalable publish-
subscriber Events Hub.
• Transform, augment,
correlate, temporal
operations.
• Detect patterns
and anomalies in
streaming data
• Correlate streaming with
reference data
Stream Analytics
Value added

29. #JSS2014
CLICK STREAM ANALYSIS

30. #JSS2014
Demo architecture
e-Commerce
web site
Log file
Events Producer
ETW
Azure Event Hub
Stream Analytics
Tumbling window
Collect
Transform
SQL Azure
Deliver
Power View
Real Time
analysis
AMQP Client

31. #JSS2014
Merci à nos sponsors

32. #JSS2014#JSS2014
Les évaluations des sessions,
c’est important !!
…Et en plus on peut gagner
des cadeaux

33. #JSS2014
http://GUSS.Pro/jss

34. #JSS2014#JSS2014