Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Koji Kawamura
Hortonworks Software Engineer
July 27, 2016
2 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
- Hortonworks DataFlow Team – Software Engineer
- ijokarumawak
- kkaw...
3 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.
4 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
DataFlows
5 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
6 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
The	Connected	Data	Platform
7 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
H O R T O N W O R K S D A T A F L O W
H O R T O N W O R K S D A T A F...
8 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
&
Powered by
Apache NiFi
9 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
8 NSA
NSA
-- NSA
Linda L. Burger
NiFi
10 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
HTTP
Syslog
HTML
HL7
FTP
UDP
XML
SFTP
11 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
12 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.
13 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
à Apache NiFi
à HDP
à Docker
à Apache NiFi
14 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
$ ./bin/nifi.sh start
$ ./bin/nifi.sh install
[1] https://nifi.apach...
15 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
HDF
[1] http://hortonworks.com/downloads/#dataflow
16 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
17 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.
18 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Step 1 :
19 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Ã
Ã
à Flow File
à Processor
Ã
à Processor
à Processor
à Relationship
20 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.
21 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Step 2 :
logAttributes
^([d]{4}-[d]{2}-[d]{2}
[d]{2}:[d]{2}:[d]{2},[...
22 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
à Data Flow
à Process Group
à Flow File
Ã
à TailFile
à Flow File
à D...
23 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
NiFi
NiFi
*_repository
logs, run, state, work
24 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.
25 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Step 3 :
26 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Ã
Ã
Ã
Ã
27 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
28 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Provenance
29 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
30 ©	Hortonworks	Inc.	2011	–2016.	All	Rights	Reserved
Thank	You
Upcoming SlideShare
Loading in …5
×

そのデータフロー NiFiで楽にしてあげましょう

4,497 views

Published on

Future of Data: Apache NiFi 勉強会 〜データフローの自動化〜
http://futureofdata.connpass.com/event/35428/

で発表したスライドです。皆さん是非ダウンロードしてNiFi, HDFを使ってみてください!

Published in: Technology
  • Dating direct: ♥♥♥ http://bit.ly/2Q98JRS ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ❤❤❤ http://bit.ly/2Q98JRS ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

そのデータフロー NiFiで楽にしてあげましょう

  1. 1. Koji Kawamura Hortonworks Software Engineer July 27, 2016
  2. 2. 2 © Hortonworks Inc. 2011 –2016. All Rights Reserved - Hortonworks DataFlow Team – Software Engineer - ijokarumawak - kkawamura hortonworks.com
  3. 3. 3 © Hortonworks Inc. 2011 –2016. All Rights Reserved Agenda HDF, Apache NiFi Step 1. Step 2. Step 3.
  4. 4. 4 © Hortonworks Inc. 2011 –2016. All Rights Reserved DataFlows
  5. 5. 5 © Hortonworks Inc. 2011 –2016. All Rights Reserved
  6. 6. 6 © Hortonworks Inc. 2011 –2016. All Rights Reserved The Connected Data Platform
  7. 7. 7 © Hortonworks Inc. 2011 –2016. All Rights Reserved H O R T O N W O R K S D A T A F L O W H O R T O N W O R K S D A T A F L O W HORTONWORKS DATAFLOW
  8. 8. 8 © Hortonworks Inc. 2011 –2016. All Rights Reserved & Powered by Apache NiFi
  9. 9. 9 © Hortonworks Inc. 2011 –2016. All Rights Reserved 8 NSA NSA -- NSA Linda L. Burger NiFi
  10. 10. 10 © Hortonworks Inc. 2011 –2016. All Rights Reserved HTTP Syslog HTML HL7 FTP UDP XML SFTP
  11. 11. 11 © Hortonworks Inc. 2011 –2016. All Rights Reserved
  12. 12. 12 © Hortonworks Inc. 2011 –2016. All Rights Reserved Agenda HDF, Apache NiFi Step 1. Step 2. Step 3.
  13. 13. 13 © Hortonworks Inc. 2011 –2016. All Rights Reserved à Apache NiFi à HDP à Docker à Apache NiFi
  14. 14. 14 © Hortonworks Inc. 2011 –2016. All Rights Reserved $ ./bin/nifi.sh start $ ./bin/nifi.sh install [1] https://nifi.apache.org/download.html [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
  15. 15. 15 © Hortonworks Inc. 2011 –2016. All Rights Reserved HDF [1] http://hortonworks.com/downloads/#dataflow
  16. 16. 16 © Hortonworks Inc. 2011 –2016. All Rights Reserved
  17. 17. 17 © Hortonworks Inc. 2011 –2016. All Rights Reserved Agenda HDF, Apache NiFi Step 1. Step 2. Step 3.
  18. 18. 18 © Hortonworks Inc. 2011 –2016. All Rights Reserved Step 1 :
  19. 19. 19 © Hortonworks Inc. 2011 –2016. All Rights Reserved à à à Flow File à Processor à à Processor à Processor à Relationship
  20. 20. 20 © Hortonworks Inc. 2011 –2016. All Rights Reserved Agenda HDF, Apache NiFi Step 1. Step 2. Step 3.
  21. 21. 21 © Hortonworks Inc. 2011 –2016. All Rights Reserved Step 2 : logAttributes ^([d]{4}-[d]{2}-[d]{2} [d]{2}:[d]{2}:[d]{2},[d]{3}) ([A-Z]+) (.+)$
  22. 22. 22 © Hortonworks Inc. 2011 –2016. All Rights Reserved à Data Flow à Process Group à Flow File à à TailFile à Flow File à Dynamic Property
  23. 23. 23 © Hortonworks Inc. 2011 –2016. All Rights Reserved NiFi NiFi *_repository logs, run, state, work
  24. 24. 24 © Hortonworks Inc. 2011 –2016. All Rights Reserved Agenda HDF, Apache NiFi Step 1. Step 2. Step 3.
  25. 25. 25 © Hortonworks Inc. 2011 –2016. All Rights Reserved Step 3 :
  26. 26. 26 © Hortonworks Inc. 2011 –2016. All Rights Reserved à à à Ã
  27. 27. 27 © Hortonworks Inc. 2011 –2016. All Rights Reserved
  28. 28. 28 © Hortonworks Inc. 2011 –2016. All Rights Reserved Provenance
  29. 29. 29 © Hortonworks Inc. 2011 –2016. All Rights Reserved
  30. 30. 30 © Hortonworks Inc. 2011 –2016. All Rights Reserved Thank You

×