© 2013 Acxiom Corporation. All Rights Reserved. © 2013 Acxiom Corporation. All Rights Reserved.
bb
1
Better connections. B...
© 2013 Acxiom Corporation. All Rights Reserved. © 2013 Acxiom Corporation. All Rights Reserved.
How to effectively process...
© 2013 Acxiom Corporation. All Rights Reserved.
The talk
3
The solution
Who we are
The problem
© 2013 Acxiom Corporation. All Rights Reserved.
Who we are ?
4
„One of the biggest companies
you've never heard of.”
sourc...
© 2013 Acxiom Corporation. All Rights Reserved.
Who we work for?
• 7 of the top 10 credit card issuers
• 7 of the top 10 r...
© 2013 Acxiom Corporation. All Rights Reserved.
Where we are?
Note: Acxiom also delivers solutions in many geographies whe...
© 2013 Acxiom Corporation. All Rights Reserved.
What we do?
7
We do data!
Marketing and information management services
Sa...
© 2013 Acxiom Corporation. All Rights Reserved.
We transform.
8
Formerly Microsoft,
aQuantive &
Razorfish
Formerly
MySpace...
© 2013 Acxiom Corporation. All Rights Reserved.
The Acxiom Audience operating
system
© 2013 Acxiom Corporation. All Rights Reserved.
Audience Propensities
10
© 2013 Acxiom Corporation. All Rights Reserved.
Regression Model
11
© 2013 Acxiom Corporation. All Rights Reserved.
The talk
12
The solution
Who we are
The problem
© 2013 Acxiom Corporation. All Rights Reserved.
The flow
13
Age
f(x,y,z,...)
f(x,y,z,...)
© 2013 Acxiom Corporation. All Rights Reserved.
The problem
14
• 21 dimentions (avg inputs)
For example
• 3823 regression ...
© 2013 Acxiom Corporation. All Rights Reserved.
The BIG problem
15
3823 x 21 x 31 x 242271350 = 602 958 394 553 550 B
376 ...
© 2013 Acxiom Corporation. All Rights Reserved.
The talk
16
The solution
Who we are
The problem
© 2013 Acxiom Corporation. All Rights Reserved.
3 Steps
17
µservices
µ
µ
µ
µ
µ
µ
µ
µ
µ
µ
µ
µ
virtual infrastructure
the sy...
© 2013 Acxiom Corporation. All Rights Reserved.
Can’t say much...
18
© 2013 Acxiom Corporation. All Rights Reserved.
The foundation
We needed a communication framework
• It’s needed to be Fas...
© 2013 Acxiom Corporation. All Rights Reserved.
The ØMQ
20
© 2013 Acxiom Corporation. All Rights Reserved.
It is not a Messaging Queue or ESB
21
© 2013 Acxiom Corporation. All Rights Reserved.
Confused?
22
© 2013 Acxiom Corporation. All Rights Reserved.
Authors
iMatix
23
Real time financial systems
© 2013 Acxiom Corporation. All Rights Reserved.
ØMQ
24
Source: http://zguide2.zeromq.org/
© 2013 Acxiom Corporation. All Rights Reserved.
Confused again?
25
© 2013 Acxiom Corporation. All Rights Reserved.
What it is ?
• Ø latency communication framework
• Queue based framework
•...
© 2013 Acxiom Corporation. All Rights Reserved.
Client
private static final ZMQ.Context zmqContext = ZMQ.context(zmqThread...
© 2013 Acxiom Corporation. All Rights Reserved.
Server
private static final ZMQ.Context zmqContext = ZMQ.context(zmqThread...
© 2013 Acxiom Corporation. All Rights Reserved.
ØMQ Message
• Atomic
• Can be Multipart
• Source/Dest
• Can be Routed/Prox...
© 2013 Acxiom Corporation. All Rights Reserved.
ØMQ Socket Types
30
• Unicast
• TCP („tcp://localhost:5555”)
• IPC („ipc:/...
© 2013 Acxiom Corporation. All Rights Reserved.
Basic Patterns
31
source: http://zguide.zeromq.org/
© 2013 Acxiom Corporation. All Rights Reserved.
Advanced Patterns
32
source: http://zguide.zeromq.org/
© 2013 Acxiom Corporation. All Rights Reserved.
Growing
zmq - 0MQ lightweight messaging kernel
zmq_bind - accept connectio...
© 2013 Acxiom Corporation. All Rights Reserved.
Cross platform
34
• NetMQ
https://github.com/zeromq/netmq
Haxe
C++
C# Cloj...
© 2013 Acxiom Corporation. All Rights Reserved.
Visit us on
35
https://developer.myacxiom.com/
http://acxiom.com/about-acx...
© 2013 Acxiom Corporation. All Rights Reserved. © 2013 Acxiom Corporation. All Rights Reserved.
QA: Mariusz.Rog@Acxiom.com...
Upcoming SlideShare
Loading in …5
×

infoShare 2014: Mariusz Róg, Big Data w praktyce -- jak efektywnie przetwarzać wielkie zbiory danych.

413 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
413
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

infoShare 2014: Mariusz Róg, Big Data w praktyce -- jak efektywnie przetwarzać wielkie zbiory danych.

  1. 1. © 2013 Acxiom Corporation. All Rights Reserved. © 2013 Acxiom Corporation. All Rights Reserved. bb 1 Better connections. Better results.
  2. 2. © 2013 Acxiom Corporation. All Rights Reserved. © 2013 Acxiom Corporation. All Rights Reserved. How to effectively process large data sets Big Data in practice Mariusz Róg, Team Leader of Engineering, Acxiom Global Service Center, Poland
  3. 3. © 2013 Acxiom Corporation. All Rights Reserved. The talk 3 The solution Who we are The problem
  4. 4. © 2013 Acxiom Corporation. All Rights Reserved. Who we are ? 4 „One of the biggest companies you've never heard of.” source: http://en.wikipedia.org/wiki/Acxiom
  5. 5. © 2013 Acxiom Corporation. All Rights Reserved. Who we work for? • 7 of the top 10 credit card issuers • 7 of the top 10 retail banks • 7 of the top 10 retailers • 6 of the top 10 telecom / media companies • 7 of the top 10 automotive manufacturers • 7 of the top 10 U.S. hotels • 5 of the top 10 technology companies • 3 of the top 5 brokerage firms • 3 of the top 5 pharmaceutical manufacturers • 8 of the top 10 insurance providers • 7 of the top 10 hotels • 3 of the top 5 domestic airlines • 4 of the top 5 gaming companies The trademarks and registered trademarks on this page are the property of their respective owners. Stats updated as of 7/8/13.
  6. 6. © 2013 Acxiom Corporation. All Rights Reserved. Where we are? Note: Acxiom also delivers solutions in many geographies where it does not have a physical presence. Argentina Australia Bahrain Bangladesh Brazil Canada Chile China (including Hong Kong and Taiwan) Colombia Egypt France Germany India Indonesia Israel Japan Jordan Korea Kuwait Lebanon Malaysia Mexico Mongolia New Zealand Oman Philippines Poland Qatar Russia Saudi Arabia Singapore South Africa Thailand United Arab Emirates United Kingdom United States Venezuela Vietnam Acxiom provides data, processing, consulting, SMS / digital and / or other services to more than 7,500 recurring clients around the globe in approximately 50 countries and 20 languages. Offices located in these markets Services available in these markets Sample of Countries
  7. 7. © 2013 Acxiom Corporation. All Rights Reserved. What we do? 7 We do data! Marketing and information management services SaaS development Analitics Customers developement services Technical and marketing consultingITO support and consulting Technology R&D Big Data managment PII and Web security PII and Healthcare world wide compilance Forrester Research named Acxiom one of the largest database marketing services and technology providers in the world
  8. 8. © 2013 Acxiom Corporation. All Rights Reserved. We transform. 8 Formerly Microsoft, aQuantive & Razorfish Formerly MySpace, MTV & AOL Formerly CFO Amazon, NBC & Electronic Arts Formerly Architect of Google Analytics Dennis D. Self CIO, SVP Formerly Electronic Arts and HP
  9. 9. © 2013 Acxiom Corporation. All Rights Reserved. The Acxiom Audience operating system
  10. 10. © 2013 Acxiom Corporation. All Rights Reserved. Audience Propensities 10
  11. 11. © 2013 Acxiom Corporation. All Rights Reserved. Regression Model 11
  12. 12. © 2013 Acxiom Corporation. All Rights Reserved. The talk 12 The solution Who we are The problem
  13. 13. © 2013 Acxiom Corporation. All Rights Reserved. The flow 13 Age f(x,y,z,...) f(x,y,z,...)
  14. 14. © 2013 Acxiom Corporation. All Rights Reserved. The problem 14 • 21 dimentions (avg inputs) For example • 3823 regression models • 31B (avg size) • 242271350 people (242M)
  15. 15. © 2013 Acxiom Corporation. All Rights Reserved. The BIG problem 15 3823 x 21 x 31 x 242271350 = 602 958 394 553 550 B 376 cores 47 nodes @ 8x3 Ghz and 32 GB ~548 TB More than a week!
  16. 16. © 2013 Acxiom Corporation. All Rights Reserved. The talk 16 The solution Who we are The problem
  17. 17. © 2013 Acxiom Corporation. All Rights Reserved. 3 Steps 17 µservices µ µ µ µ µ µ µ µ µ µ µ µ virtual infrastructure the system
  18. 18. © 2013 Acxiom Corporation. All Rights Reserved. Can’t say much... 18
  19. 19. © 2013 Acxiom Corporation. All Rights Reserved. The foundation We needed a communication framework • It’s needed to be Fast • It’s needed to be Stable • It’s needed to be Concurrent 19
  20. 20. © 2013 Acxiom Corporation. All Rights Reserved. The ØMQ 20
  21. 21. © 2013 Acxiom Corporation. All Rights Reserved. It is not a Messaging Queue or ESB 21
  22. 22. © 2013 Acxiom Corporation. All Rights Reserved. Confused? 22
  23. 23. © 2013 Acxiom Corporation. All Rights Reserved. Authors iMatix 23 Real time financial systems
  24. 24. © 2013 Acxiom Corporation. All Rights Reserved. ØMQ 24 Source: http://zguide2.zeromq.org/
  25. 25. © 2013 Acxiom Corporation. All Rights Reserved. Confused again? 25
  26. 26. © 2013 Acxiom Corporation. All Rights Reserved. What it is ? • Ø latency communication framework • Queue based framework • Concurrency framework • Easy and intuitive API • LGPL Example… 26
  27. 27. © 2013 Acxiom Corporation. All Rights Reserved. Client private static final ZMQ.Context zmqContext = ZMQ.context(zmqThreadCount); ZMQ.Socket socket = zmqContext.socket(ZMQ.REQ); socket.setSendTimeOut(sendTimeout); socket.connect("tcp://*:5555"); // ZMQ_MSG_FLAGS 0 = blocking socket type boolean messageSent = socket.send(msg, ZMQ_MSG_FLAGS); if(!messageSent){ LOG.error("Error receiving response for {}", this); } byte[] responseMsg = socket.recv(ZMQ_MSG_FLAGS); if(responseMsg == null){ LOG.error("Error receiving response for {}", this); } 27
  28. 28. © 2013 Acxiom Corporation. All Rights Reserved. Server private static final ZMQ.Context zmqContext = ZMQ.context(zmqThreadCount); ... ZMQ.Socket socket = zmqContext.socket(ZMQ.REQ); socket.setReceiveTimeOut(receiveTimeout); socket.bind("tcp://*:5555"); socket.bind("inproc://workers"); while (Thread.currentThread().isInterrupted() == false) { byte[] recivedBytes = socket.recv(0); if(recivedBytes == null){ LOG.error("Error receiving response for {}", this); } boolean messageSent = socket.send(msg, ZMQ_MSG_FLAGS); if(!messageSent){ LOG.error("Error receiving response for {}", this); } } 28
  29. 29. © 2013 Acxiom Corporation. All Rights Reserved. ØMQ Message • Atomic • Can be Multipart • Source/Dest • Can be Routed/Proxed/Analized • Data agnostic preffered Google Protobuf 29
  30. 30. © 2013 Acxiom Corporation. All Rights Reserved. ØMQ Socket Types 30 • Unicast • TCP („tcp://localhost:5555”) • IPC („ipc://storeandforward”) • INPROC („inproc://emailThread”) • Multicast • PGM/EPGM („epgm://192.168.1.1:5555”)
  31. 31. © 2013 Acxiom Corporation. All Rights Reserved. Basic Patterns 31 source: http://zguide.zeromq.org/
  32. 32. © 2013 Acxiom Corporation. All Rights Reserved. Advanced Patterns 32 source: http://zguide.zeromq.org/
  33. 33. © 2013 Acxiom Corporation. All Rights Reserved. Growing zmq - 0MQ lightweight messaging kernel zmq_bind - accept connections on a socket zmq_close - close 0MQ socket zmq_connect - connect a socket zmq_cpp - interface between 0MQ and C++ applications zmq_device - start built-in 0MQ device zmq_pgm - 0MQ reliable multicast transport using PGM zmq_errno - retrieve value of errno for the calling thread zmq_getsockopt - get 0MQ socket options zmq_init - initialise 0MQ context zmq_inproc - 0MQ local in-process (inter-thread) communication transport zmq_ipc - 0MQ local inter-process communication transport zmq_msg_close - release 0MQ message zmq_msg_copy - copy content of a message to another message zmq_msg_data - retrieve pointer to message content zmq_msg_init_data - initialise 0MQ message from a supplied buffer zmq_msg_init_size - initialise 0MQ message of a specified size zmq_msg_init - initialise empty 0MQ message zmq_msg_move - move content of a message to another message zmq_msg_size - retrieve message content size in bytes zmq_pgm - 0MQ reliable multicast transport using PGM zmq_poll - input/output multiplexing zmq_recv - receive a message from a socket zmq_send - send a message on a socket zmq_setsockopt - set 0MQ socket options zmq_socket - create 0MQ socket zmq_strerror - get 0MQ error message string zmq_tcp - 0MQ unicast transport using TCP zmq_term - terminate 0MQ context zmq_version - report 0MQ library version 33 zmq - 0MQ lightweight messaging kernel zmq_bind - accept incoming connections on a socket zmq_close - close 0MQ socket zmq_connect - create outgoing connection from socket zmq_ctx_destroy - terminate a 0MQ context zmq_ctx_get - get context options zmq_ctx_new - create new 0MQ context zmq_ctx_set - set context options zmq_ctx_shutdown - shutdown a 0MQ context zmq_ctx_term - destroy a 0MQ context zmq_curve_keypair - generate a new CURVE keypair zmq_curve - secure authentication and confidentiality zmq_disconnect - Disconnect a socket zmq_pgm - 0MQ reliable multicast transport using PGM zmq_errno - retrieve value of errno for the calling thread zmq_getsockopt - get 0MQ socket options zmq_init - initialise 0MQ context zmq_inproc - 0MQ local in-process (inter-thread) communication transport zmq_ipc - 0MQ local inter-process communication transport zmq_msg_close - release 0MQ message zmq_msg_copy - copy content of a message to another message zmq_msg_data - retrieve pointer to message content zmq_msg_get - get message property zmq_msg_init_data - initialise 0MQ message from a supplied buffer zmq_msg_init_size - initialise 0MQ message of a specified size zmq_msg_init - initialise empty 0MQ message zmq_msg_more - indicate if there are more message parts to receive zmq_msg_move - move content of a message to another message zmq_msg_recv - receive a message part from a socket zmq_msg_send - send a message part on a socket zmq_msg_set - set message property zmq_msg_size - retrieve message content size in bytes zmq_null - no security or confidentiality zmq_pgm - 0MQ reliable multicast transport using PGM zmq_plain - clear-text authentication zmq_poll - input/output multiplexing zmq_proxy_steerable - start built-in 0MQ proxy with PAUSE/RESUME/TERMINATE control flow zmq_proxy - start built-in 0MQ proxy zmq_recvmsg - receive a message part from a socket zmq_recv - receive a message part from a socket zmq_send_const - send a constant-memory message part on a socket zmq_sendmsg - send a message part on a socket zmq_send - send a message part on a socket zmq_setsockopt - set 0MQ socket options zmq_socket_monitor - register a monitoring callback zmq_socket - create 0MQ socket zmq_strerror - get 0MQ error message string zmq_tcp - 0MQ unicast transport using TCP zmq_term - terminate 0MQ context zmq_unbind - Stop accepting connections on a socket zmq_version - report 0MQ library version zmq_z85_decode - decode a binary key from Z85 printable text zmq_z85_encode - encode a binary key as Z85 printable text ØMQ v 2.2 ØMQ v 4.0
  34. 34. © 2013 Acxiom Corporation. All Rights Reserved. Cross platform 34 • NetMQ https://github.com/zeromq/netmq Haxe C++ C# Clojure CL Erlang F# Felix Go Haskell Java Lua Node.js Objective-CPerl PHP Python Racket ooc Basic Ada Tcl Scala Ruby Q • JeroMQ https://github.com/zeromq/jeromq
  35. 35. © 2013 Acxiom Corporation. All Rights Reserved. Visit us on 35 https://developer.myacxiom.com/ http://acxiom.com/about-acxiom/careers/
  36. 36. © 2013 Acxiom Corporation. All Rights Reserved. © 2013 Acxiom Corporation. All Rights Reserved. QA: Mariusz.Rog@Acxiom.com Thank You!

×