Downloaded 109 times
Uploaded byEvans Ye
Trend Micro Big Data Platform and Apache Bigtop
The document presents insights from Evans Ye at the 2015 Big Data Conference, covering the Trend Micro Big Data Platform and Apache Bigtop. It discusses the advantages of using Bigtop for building custom big data stacks, its features for deployment and testing, and the integration of various technologies like Hadoop, Spark, and Kafka. Additionally, it outlines the evolving big data landscape and future trends, including in-memory computing solutions and Bigtop's roadmap for upcoming releases.
More Related Content
Similar to Trend Micro Big Data Platform and Apache Bigtop
![20260201 [FOSDEM] gomodjail - library sandboxing for Go modules.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/20260201fosdemgomodjail-librarysandboxingforgomodules-260201225659-76609ec4-thumbnail.jpg?width=640&height=640&fit=bounds)
Trend Micro Big Data Platform and Apache Bigtop
- 1. 葉祐欣 (Evans Ye) BigData Conference 2015 Trend Micro Big Data Platform and Apache Bigtop
- 2. Who am I •Apache Bigtop PMC member • Apache Big Data Europe 2015 Speaker • Software Engineer @ Trend Micro • Develop big data apps & infra • Has some experience in Hadoop, HBase, Pig, Spark, Kafka, Fluentd, Akka, and Docker
- 3. Outline • Quick Introto Bigtop • Trend Micro Big Data Platform • Mission-specific Platform • Big Data Landscape (3p) • Bigtop 1.1 Release (6p)
- 4. Quick Intro toBigtop
- 5.
- 6.
- 7. Hadoop Distributions We’re fullyopen sourced !
- 8. How do Iadd patches?
- 10. From source code topackages Bigtop Packaging
- 11. Bigtop feature set PackagingTesting Deployment Virtualization for you to easily build your own Big Data Stack
- 12.
- 13. • $ gitclone https://github.com/apache/bigtop.git • $ docker run --rm --volume `pwd`/bigtop:/bigtop --workdir /bigtop bigtop/slaves:trunk-centos-7 bash -l -c ‘./gradlew rpm’ One click to build packages
- 14. • $ ./gradlewtasks’
- 15. Easy to doCI ci.bigtop.apache.org
- 16.
- 17. One click Hadoopprovisioning ./docker-hadoop.sh -c 3
- 18. bigtop/deploy image onDocker hub ./docker-hadoop.sh -c 3 One click Hadoop provisioning
- 19. bigtop/deploy image onDocker hub ./docker-hadoop.sh -c 3 puppet apply puppet apply puppet apply One click Hadoop provisioning Just google bigtop provisioner
- 20. Should I useBigtop?
- 21. If you wantto build your own customised Big Data Stack
- 22.
- 23. Pros & cons •Bigtop • You need a talented Hadoop team • Self-service: troubleshoot, find solutions, develop patches • Add any patch at any time you want (additional efforts) • Choose any version of component you want (additional efforts) • Vendors (Hortonworks, Cloudera, etc) • Better support since they’re the guy who write the code ! • $
- 24. Trend Micro BigData Platform
- 25. • Use Bigtopas the basis for our internal custom distribution of Hadoop • Apply community, private patches to upstream projects for business and operational need • Newest TMH7 is based on Bigtop 1.0 SNAPSHOT Trend Micro Hadoop (TMH)
- 26. Working with community madeour life easier • Knowing community status made TMH7 release based on Bigtop 1.0 SNAPSHOT possible
- 27. Working with community madeour life easier • Contribute Bigtop Provisioner, packaging code, puppet recipes, bugfixes, CI infra, anything! • Knowing community status made TMH7 release based on Bigtop 1.0 SNAPSHOT possible
- 28. Working with community madeour life easier • Leverage Bigtop smoke tests and integration tests with Bigtop Provisioner to evaluate TMH7
- 29. Working with community madeour life easier • Contribute feedback, evaluation, use case through Production level adoption • Leverage Bigtop smoke tests and integration tests with Bigtop Provisioner to evaluate TMH7
- 30. Hadoop YARN Hadoop HDFS Mapreduce Ad-hocQuery UDFs Pig App A App C Oozie Resource Management Storage Processing Engine APIs and Interfases In-house Apps Trend Micro Big Data Stack Powered by Bigtop Kerberos App B App D HBase Wuji Solr Cloud Hadooppet (prod) Hadoocker (dev)Deployment
- 31. Hadooppet • Puppet recipesto deploy and manage TMH Big Data Platform • HDFS, YARN, HA auto-configured • Kerberos, LDAP auto-configured • Kerberos cross realm authentication auto-configured (For distcp to run across secured clusters)
- 33. • A Devopstoolkit for Hadoop app developer to develop and test its code on • Big Data Stack preload images —> dev & test env w/o deployment —> support end-to-end CI test • A Hadoop env for apps to test against new Hadoop distribution • https://github.com/evans-ye/hadoocker Hadoocker
- 34. internal Docker registry ./execute.sh Hadoopserver Hadoop client data Docker based dev & test env TMH7 Hadoop app Restful APIs sample data hadoop fs put
- 35. internal Docker registry ./execute.sh Hadoopserver Hadoop client data TMH7 Hadoop app Restful APIs sample data hadoop fs putSolr Oozie(Wuji) Dependency service Docker based dev & test env
- 36.
- 37. Use case • Real-timestreaming data flows in • Lookup external info when data flows in • Detect threat/malicious activities on streaming data • Correlate with other historical data (batch query) to gather more info • Can also run batch detections by specifying arbitrary start time and end time • Support Investigation down to raw log level
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44. • High-throughput, distributedpublish-subscribe messaging system • Supports multiple consumers attached to a topic • Configurable partition(shard), replication factor • Load-balance within same consumer group • Only consume message once a b c
- 45. • Distributed NoSQLkey-value storage, no SPOF • Super fast on write, suitable for data keeps coming in • Decent read performance, if design it right • Build data model around your queries • Spark Cassandra Connector • Configurable CA (CAP theorem) • Choose A over C for availability and vise-versa Dynamo: Amazon’s Highly Available Key-value Store
- 46. • Fast, distributed,in-memory processing engine • One system for streaming and batch workloads • Spark streaming
- 47. Akka • High performanceconcurrency framework for Java and Scala • Actor model for message-driven processing • Asynchronous by design to achieve high throughput • Each message is handled in a single threaded context (no lock, synchronous needed) • Let-it-crash model for fault tolerance and auto-healing system • Clustering mechanism to scale out The Road to Akka Cluster, and Beyond
- 48. Akka Streams • AkkaStreams is a DSL library for streaming computation on Akka • Materializer to transform each step into Actor • Back-pressure enabled by default Source Flow Sink The Reactive Manifesto
- 49. No back-pressure Source Fast!!!SinkSlow… (＞﹏＜)’v(￣︶￣)y
- 50. No back-pressure Source Fast!!!SinkSlow… (＞﹏＜)’’’’’v(￣︶￣)y
- 51. With back-pressure Source Fast!!!SinkSlow…
- 52. With back-pressure Source Fast!!!SinkSlow… request 3request 3
- 53. Data pipeline withAkka Streams • Scale up using balance and merge source: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-cookbook.html#working-with-flows worker worker worker balance merge
- 54. • Scale outusing docker Data pipeline with Akka Streams $ docker-compose scale pipeline=3
- 55. Reactive Kafka • AkkaStreams wrapper for Kafka • Commit processed offset back into Kafka • Provide at-least-once delivery guarantee https://github.com/softwaremill/reactive-kafka
- 56. Message delivery guarantee •Actor Model: at-most-once • Akka Persistence: at-least-once • Persist log to external storage (like WAL) • Reactive Kafka: at-least-once + back-pressure • Write offset back into Kafka • At-least-once + Idempotent writes = exactly-once
- 57. • Spark: bothstreaming and batch analytics • Docker: resource management (fine for one app) • Akka: fine-grained, elastic data pipelines • Cassandra: batch queries • Kafka: durable buffer, fan-out to multiple consumers Recap: SDACK Stack
- 58. Your mileage mayvary
- 59.
- 60.
- 61. The SMACK Stack Toolboxfor wide variety of data processing scenarios
- 62. SMACK Stack • Spark:fast and general engine for large-scale data processing • Mesos: cluster resource management system • Akka: toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications • Cassandra: distributed, highly available database designed to handle large amounts of data across datacenters • Kafka: high-throughput, low-latency distributed pub-sub messaging system for real-time data feeds Source: http://www.slideshare.net/akirillov/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka
- 63. Reference • Spark SummitEurope 2015 • Streaming Analytics with Spark, Kafka, Cassandra, and Akka (Helena Edelson) • Big Data AW Meetup • SMACK Architectures (Anton Kirillov)
- 64.
- 65. • Memory isfaster than SSD/disk, and is cheaper • In Memory Computing & Fast Data • Spark : In memory batch/streaming engine • Flink : In memory streaming/batch engine • Iginte : In memory data fabric • Geode (incubating) : In memory database Big Data moving trend
- 66. • Off-Heap storageis a JVM process memory outside of the heap, which is allocated and managed using native calls. • size not limited by JVM (it is limited by physical memory limits) • is not subject to GC which essentially removes long GC pauses • Project Tungsten, Flink, Iginte, Geode, HBase Off-Heap, Off-Heap, Off-Heap
- 67. Pig Hadoop YARN Hadoop HDFS Resource Management Storage Processing Engine (Some)Apache Big Data Components Slider Flink Spark Flink ML, Gelly Streaming, MLlib, GraphX Kafka HBase Mesos Tez Hive Phoenix Ignite APIs and Interfases Geode Trafodion Solr } messaging system in memory data grid search engine Bigtop Ambari Hadoop Distribution Hadoop Management Cassandra NoSQL
- 68. Bigtop 1.1 Release Jan,2016, I expect…
- 69. Bigtop 1.1 Release •Hadoop 2.7.1 • Spark 1.5.1 • Hive 1.2.1 • Pig 0.15.0 • Oozie 4.2.0 • Flume 1.6.0 • Zeppelin 0.5.5 • Ignite Hadoop 1.5.0 • Phoenix 4.6.0 • Hue 3.8.1 • Crunch 0.12 • …, 24 components included!
- 71. Hadoop 2.6 • HeterogeneousStorages • SSD + hard drive • Placement policy (all_ssd, hot, warm, cold) • Archival Storage (cost saving) • HDFS-7285 (Hadoop 3.0) • Erasure code to save storage from 3X to 1.5X http://www.slideshare.net/Hadoop_Summit/reduce-storage- costs-by-5x-using-the-new-hdfs-tiered-storage-feature
- 72. Hadoop 2.7 • Transparentencryption (encryption zone) • Available in 2.6 • Known issue: Encryption is sometimes done incorrectly (HADOOP-11343) • Fixed in 2.7 http://events.linuxfoundation.org/sites/events/files/slides/ HDFS2015_Past_present_future.pdf
- 73. Rising star: Flink •Streaming dataflow engine • Treat batch computing as fixed length streaming • Exactly-once by distributed snapshotting • Event time handling by watermarks
- 74. • Integrate andpackage Apache Flink • Re-implement Bigtop Provisioner using docker-machine, compose, swarm • Deploy containers on multiple hosts • Support any kind of base image for deployment Bigtop Roadmap
- 75.
- 76. • Hadoop Distribution •Choose Bigtop if you want more control • The SMACK Stack • Toolbox for variety data processing scenarios • Big Data Landscape • In-memory, off-heap solutions are hot Wrap up
- 77.