Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

End to End Processing of 3.7 Million Telemetry Events per Second using Lambda Architecture

1,860 views

Published on

End to End Processing of 3.7 Million Telemetry Events per Second using Lambda Architecture

Published in: Technology
  • Be the first to comment

End to End Processing of 3.7 Million Telemetry Events per Second using Lambda Architecture

  1. 1. End to End Processing of 3.7 Million Telemetry Events per Second Using Lambda Architecture Saurabh Mishra Raghavendra Nandagopal
  2. 2. Who are We? Saurabh Mishra Solution Architect, Hortonworks Professional Services @draftsperson smishra@hortonworks.com Raghavendra Nandagopal Cloud Data Services Architect, Symantec @speaktoraghav raghavendra_nandagop@symantec.com
  3. 3. Cloud Platform Engineering Symantec - Global Leader in Cyber Security - Symantec is the world leader in providing security software for both enterprises and end users - There are 1000’s of Enterprise Customers and more than 400 million devices (Pcs, Tablets and Phones) that rely on Symantec to help them secure their assets from attacks, including their data centers, emails and other sensitive data Cloud Platform Engineering (CPE) - Build consolidated cloud infrastructure and platform services for next generation data powered Symantec applications - A big data platform for batch and stream analytics integrated with both private and public cloud. - Open source components as building blocks - Hadoop and Openstack - Bridge feature gaps and contribute back
  4. 4. Agenda • Security Data Lake @ Global Scale • Infrastructure At Scale • Telemetry Data Processing Architecture • Tunable Targets • Performance Benchmarks • Service Monitoring
  5. 5. Security Data Lake @ Global Scale
  6. 6. Security Data Lake @ Global Scale Products HDFS Analytic Applications, Workload Management (YARN) Stream Processing (Storm) Real-time Results (HBase, ElasticSearch) Query (Hive, Spark SQL) Device Agents Telemetry Data Data Transfer Threat Protection Inbound Messaging (Data Import, Kafka) Physical Machine , Public Cloud, Private Cloud
  7. 7. Lambda Architecture 7 Speed Layer Batch Layer Serving Layer Complexity Isolation Once Data Makes to Serving Layer via Batch , Speed Layer can be Neglected Compensate for high latency of updates to serving layer Fast and incremental algorithms on real time data Batch layer eventually overrides speed layer Random access to batch views Updated by batch layer Stores master dataset Batch layer stores the master copy of the Serving layer Computes arbitrary Views
  8. 8. Infrastructure At Scale
  9. 9. Yarn Applications in Production - 669388 submitted - 646632 completed - 4640 killed - 401 failed Hive in Production - 25887 Tables - 306 Databases - 98493 Partitions Storm in Production - 210 Nodes 50+ Topologies Kafka in Production - 80 Nodes Hbase in Production - 135 Nodes ElasticSearch in Production - 62 Nodes Ambari Infrastructure At Scale Centralize d Logging and Metering Ironic Ansible Cloudbreak Hybrid Data Lake OpenStack (Dev) 350 Nodes Metal (Production) 600 Nodes Public Cloud (Production ) 200 Nodes
  10. 10. Telemetry Data Processing Architecture
  11. 11. Telemetry Data Processing Architecture Telemetry Data Collector Telemetry Gateway Raw Events Data Centers Avro Serialized Telemetry Avro Serialized Telemetry Opaque Trident Kafka Spout Deserialized Objects Transformations Functions Transformation Topology Trident Streams (Micro batch implementation, Exactly Once semantics) Persist Avro Objects Avro Serialized Transformed Objects ElasticSearc h Ingestion Topology Opaque Trident Kafka Spout Trident ElasticSearch Writer Bolt HBase Ingestion Topology Trident HBase Bolt Trident Hive Streamin g Opaque Trident Kafka Spout YARN Hive Ingestion Topology Identity Topology
  12. 12. Tunable Targets
  13. 13. Operating System Tuning Targets ● Operating System ● Disk ● Network Tunables ● Disable Transparent Huge Pages echo never > defrag and > enabled ● Disable Swap ● Configure VM Cache flushing ● Configure IO Scheduler as deadline ● Disk Jbod Ext4 Mount Options- inode_readahead_blks=128, data=writeback,noatime,nodiratime ● Network Dual Bonded 10gbps rx-checksumming: on, tx-checksumming: on, scatter-gather: on, tcp-segmentation-offload: on
  14. 14. Kafka Tuning Targets ● Broker ● Producer ● Consumer Tunables ● replica.fetch.max.bytes ● socket.send.buffer.bytes ● socket.receive.buffer.bytes ● replica.socket.receive.buffer.bytes ● num.network.threads ● num.io.threads ● zookeeper.*.timeout.ms Type Metal 2.6 GHz E5-2660 v3 12 * 4TB JBOD 128 GB DDR4 ECC Cloud AWS: D2*8xlarge v 0.8.2.1
  15. 15. Kafka Tuning Targets ● Broker ● Producer ● Consumer Tunables ● buffer.memory ● batch.size ● linger.ms ● compression.type ● socket.send.buffer.bytes Type Metal 2.6 GHz E5-2660 v3 12 * 4TB JBOD 128 GB DDR4 ECC Cloud AWS: D2*8xlarge v 0.8.2.1
  16. 16. Kafka Tuning Targets ● Broker ● Producer ● Consumer Tunables ● num.consumer.fetchers ● socket.receive.buffer.bytes ● fetch.message.max.bytes ● fetch.min.bytes Type Metal 2.6 GHz E5-2660 v3 12 * 4TB JBOD 128 GB DDR4 ECC Cloud AWS: D2*8xlarge v 0.8.2.1
  17. 17. Storm Tuning Targets ● Nimbus ● Supervisors ● Workers and Executors ● Topology Tunables ● Nimbus High Availability - 4 Nimbus Servers Avoid downtime and performance degradation. ● Reduce workload on Zookeeper and Decreased topology Submission time. storm.codedistributor.class = HDFSCodeDistributor ● topology.min.replication.count = 3 floor(number_of_nimbus_hosts/2 + 1) ● max.replication.wait.time.sec = -1 ● code.sync.freq.secs = 2 mins ● storm.messaging.netty.buffer_size = 10 mb ● nimbus.thrift.threads = 256 Type Metal 2.6 GHz E5-2660 v3 2 * 500GB SSD 256 GB DDR4 ECC Cloud AWS: r3*8xlarge v 0.10.0.2.4
  18. 18. Storm Tuning Targets ● Nimbus ● Supervisors ● Workers and Executors ● Topology Tunables ● Use supervisord to control Supervisors ● Supervisor.slots.ports = Min (No of HT Cores , TotalMem of Server/Worker heap size) ● Supervisor.childopts = -Xms4096m -Xmx4096m - verbose:gc- Xloggc:/var/log/storm/supervisor_%ID%_gc.log Type Metal 2.6 GHz E5-2660 v3 2 * 500GB SSD 256 GB DDR4 ECC Cloud AWS: r3*8xlarge v 0.10.0.2.4
  19. 19. Storm Tuning Targets ● Nimbus ● Supervisors ● Workers and Executors ● Topology Tunables Rule of Thumb! - Use Case of Storm – Telemetry Processing ● CPU bound tasks 1 Executor Per worker ● IO Bound tasks 8 Executors Per worker, ● Fixed the JVM Memory for each Worker Based on Fetch Size of Kafka Trident Spout and Split Size of Bolt -Xms8g -Xmx8g -XX:MaxDirectMemorySize=2048m -XX:NewSize=2g -XX:MaxNewSize=2g -XX:+UseParNewGC -XX:MaxTenuringThreshold=2 -XX:SurvivorRatio=8 - XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32768 - XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled - XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=80 - XX:+UseCMSInitiatingOccupancyOnly -XX:-CMSConcurrentMTEnabled - XX:+AlwaysPreTouch Type Metal 2.6 GHz E5-2660 v3 2 * 500GB SSD 256 GB DDR4 ECC Cloud AWS: r3*8xlarge v 0.10.0.2.4
  20. 20. Storm Tuning Targets ● Nimbus ● Supervisors ● Workers and Executors ● Topology Tunables ● Topology.optimize = true ● Topology.message.timeout.secs = 110 ● Topology.max.spout.pending = 3 ● Remove Topology.metrics.consumer.register- AMBARI-13237 Incoming queue and Outgoing queue ● Topology.transfer.buffer.size = 64 – Batch Size ● Topology.receiver.buffer.size = 16 – Queue Size ● Topology.executor.receive.buffer.size = 32768 ● Topology.executor.send.buffer.size = 32768 Type Metal 2.6 GHz E5-2660 v3 2 * 500GB SSD 256 GB DDR4 ECC Cloud AWS: r3*8xlarge v 0.10.0.2.4
  21. 21. Storm Tuning Targets ● Nimbus ● Supervisors ● Workers and Executors ● Topology Tunables ● topology.trident.parallelism.hint = (number of worker nodes in cluster * number cores per worker node) - (number of acker tasks) ● Kafka.consumer.fetch.size.byte = 209715200 ( 200MB - Yes! We process large batches) ● Kafka.consumer.buffer.size.byte = 209715200 ● Kafka.consumer.min.fetch.byte = 100428800 Type Metal 2.6 GHz E5-2660 v3 2 * 500GB SSD 256 GB DDR4 ECC Cloud AWS: r3*8xlarge v 0.10.0.2.4
  22. 22. ZooKeeper Tunables ● Keep data and log directories separately and on different mounts ● Separate Zookeeper quorum of 5 Servers each for Kafka, Storm, Hbase, HA quorum. ● Zookeeper GC Configurations -Xms4192m -Xmx4192m -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime - XX:+PrintGCApplicationConcurrentTime -XX:+PrintGC -XX:+PrintGCTimeStamps - XX:+PrintGCDetails -verbose:gc -Xloggc:/var/log/zookeeper/zookeeper_gc.log Type Metal 2.6 GHz E5-2660 v3 2*400 GB SSD 128 GB DDR4 ECC Cloud AWS: r3.2xlarge v 3.4.6 Tuning Targets ● Data and Log directory ● Garbage Collection
  23. 23. Elasticsearch Type Metal 2.6 GHz E5-2660 v3 14 * 400 GB SSD 256 GB DDR4 ECC Cloud AWS: i2.4xlarge Tunables ● bootstrap.mlockall: true ● indices.fielddata.cache.size: 25% ● threadpool.bulk.queue_size: 5000 ● index.refresh_interval: 30s ● Index.memory.index_buffer_size: 10% ● index.store.type: mmapfs ● GC settings: -verbose:gc -Xloggc:/var/log/elasticsearch/elasticsearch_gc.log - Xss256k -Djava.awt.headless=true -XX:+UseCompressedOops -XX:+UseG1GC - XX:MaxGCPauseMillis=200 -XX:+DisableExplicitGC -XX:+PrintGCDateStamps - XX:+PrintGCDetails -XX:+PrintGCTimeStamps - XX:ErrorFile=/var/log/elasticsearch_err.log -XX:ParallelGCThreads=8 ● Bulk api ● Client node v 1.7.5 Tuning Targets ● Index Parameters ● Garbage Collection
  24. 24. Hbase Tuning Targets ● Region server GC ● Hbase Configurations Tunables export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVE R_OPTS -XX:+UseConcMarkSweepGC -Xmn2500m - XX:SurvivorRatio=4 -XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly -Xmx{{regionserver_heapsize}} -Xms{{regionserver_heapsize}} -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps - XX:+PrintPromotionFailure -XX:+PrintTenuringDistribution - XX:+PrintGCApplicationStoppedTime -Xloggc:${HBASE_LOG_DIR}/hbase-gc-regionserver.log.`date +'%Y%m%d%H%M'` Type Metal 2.6 GHz E5-2660 v3 14 * 400 GB SSD 256 GB DDR4 ECC Cloud AWS: i2.8xlarge v 1.1.0
  25. 25. Hive Tuning Targets ● Table Structure ● Partition and Bucketing Scheme ● Orc Tuning Tunables ● Use strings instead of binaries. ● Use Integer fields. Type Metal 2.6 GHz E5-2660 v3 14 * 6TB HDD 256 GB DDR4 ECC Cloud AWS: d2*8xlarge v 1.2.1
  26. 26. Hive Tuning Targets ● Table Structure ● Partition and Bucketing Scheme ● Orc Tuning Tunables ● Partitioning by Date Timestamp. ● Additional partitioning - Resulted in explosion of number of partitions, small file size and inefficient ORC compression. ● Bucketing: If two tables bucket on the same column, they should use the same number of buckets to support joining ● Sorting : Each table should optimize its sorting. The bucket column typically should be the first sorted column. Type Metal 2.6 GHz E5-2660 v3 14 * 6TB HDD 256 GB DDR4 ECC Cloud AWS: d2*8xlarge v 1.2.1
  27. 27. Hive Tuning Targets ● Table Structure ● Partition and Bucketing Scheme ● Orc Tuning Tunables ● Table Structure , Bucketing and Partition and Sorting Impact ORC Performance. ● ORC Stripe Size default 128MB Balanced Insert and Query Optimized. ● ORC use ZLIB Compression. Smaller data size improves any query. ● Predicate Push Down. Type Metal 2.6 GHz E5-2660 v3 14 * 6TB HDD 256 GB DDR4 ECC Cloud AWS: d2*8xlarge v 1.2.1 No of Yarn Containers Per Query orc.compress ZLIB high level compression (one of NONE, ZLIB, SNAPPY) orc.compress.size 262144 number of bytes in each compression chunk orc.stripe.size 130023424 number of bytes in each stripe orc.row.index.stride 64,000 number of rows between index entries (must be >= 1000) orc.create.index true whether to create row indexes orc.bloom.filter.column s "file_sha2" comma separated list of column names for which bloom filter created orc.bloom.filter.fpp 0.05 false positive probability for bloom filter (must >0.0 and <1.0)
  28. 28. Hive Streaming Tuning Targets ● Hive Metastore Stability ● Evaluate BatchSize & TxnsPerBatch Tunables ● No Hive Shell Access only Hiveserver2. ● Multiple Hive Metastore Process ○ Compaction Metastore - 5 - 10 Compaction thread ○ Streaming Metastore - 5 - Connection pool ● 16 GB Heap Size. ● Metastore Mysql Database Scalability. ● Maximum EPS was achieved by Increasing Batch Size and keeping TxnPerBatch as Smaller. Type Metal 2.6 GHz E5-2660 v3 14 * 6TB HDD 256 GB DDR4 ECC Cloud AWS: d2*8xlarge v 1.2.1
  29. 29. Performance Benchmarks
  30. 30. Benchmarking Suite Kafka Producer Consumer Throughput Test Storm Core and Trident Topologies Standard Platform Test Suite Hive TPC
  31. 31. Kafka Producer and Consumer Tests The benchmark set contains Producer and Consumer test executing at various message size. Producer and Consumer Together ● 100 bytes ● 1000 bytes - Average Telemetry Event Size ● 10000 bytes ● 100000 bytes ● 500000 bytes ● 1000000 bytes Type of Tests ● Single thread no replication ● Single-thread, async 3x replication ● Single-thread, sync 3x replication ● Throughput Versus Stored Data Ingesting 10 Telemetry Sources in Parallel
  32. 32. Storm Topology The benchmark set custom topologies for processing telemetry data source transformation and ingestion which simulates end to end use cases for real time streaming of Telemetry. ● Storm Trident HDFS Telemetry Transformation and Ingestion ● Storm Trident Hive Telemetry Ingestion ● Storm Trident Hbase Telemetry Ingestion ● Storm Trident Elasticsearch Telemetry Ingestion Ingesting 10 Telemetry Sources in Parallel
  33. 33. Standard Platform Tests TeraSort benchmark suite ( 2TB , 5TB, 10TB) RandomWriter(Write and Sort) - 10GB of random data per node DFS-IO Write and Read - TestDFSIO NNBench (Write, Read, Rename and Delete) MRBench Data Load (Upload and Download)
  34. 34. TPC-DS 20TB TPC-DS: Decision Support Performance Benchmarks ● Classic EDW Dimensional model ● Large fact tables ● Complex queries Scale: 20TB TPC-DS Benchmarking
  35. 35. Service Monitoring
  36. 36. Service Monitoring Architecture
  37. 37. Kafka Monitoring
  38. 38. ElasticSearch
  39. 39. Storm Kafka Lag
  40. 40. Storm Kafka Logging Collection
  41. 41. Thank You Q&A

×