Download as PDF, PPTX
Uploaded byPaolo Nesi
Data Flow Management and Visual Analytic for Big Data Smart City/IOT
The document describes a data management and visualization platform for smart cities and IoT called Snap4City. It uses a lambda architecture to process streaming and batch data using tools like HDFS, Elasticsearch, Kibana and Node-RED. The platform was tested using SolrCloud and Elasticsearch to index 30 million documents totaling 7GB. Elasticsearch had better performance, scaling to 4500 parallel requests while SolrCloud broke down after 1500 requests. CPU usage increased linearly as requests grew.
More Related Content
Similar to Data Flow Management and Visual Analytic for Big Data Smart City/IOT
Data Flow Management and Visual Analytic for Big Data Smart City/IOT
- 1. Snap4City, IEEE SCALCOM, Leicester, August 2019 1 Title Paolo Nesi, paolo.nesi@unifi.it https://www.snap4city.org ‐https://www.disit.org Powered by Data Flow Management and Visual Analytic for Big Data Smart City/IOT P. Bellini, F. Bugli, P. Nesi, G. Pantaleo, M. Paolucci, I. Zaza DISIT Lab, Dept. of Information Engineering, University of Florence, Italy
- 2.
- 3. Snap4City, IEEE SCALCOM, Leicester, August 2019 3 Sentient Solutions Dashboards with IOT Applications for enforcingdata driven and intelligence Dashboards and AppsIOT and City data World IOT Applications My IOT Devices Big Data Analytics, Artificial Intelligence
- 4.
- 5. DISIT Lab, DistributedData Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Mobile and Web Apps Smart City Cloud Infrastructure Km4City Smart City API Data Management Tools Knowledge Base IoT/IoE Social Media IOT App; ETL Processes, Data Analytic, etc. Data Processing Tools Development Tools ETL processes Resource Manager Datagate/CKAN Tools for Final users Dashboards Open Data KM4City Ontology Phoenix HBase Big Data Storages Personal Data IoT/IoE Applications Industry 4.0 AMMA Linked Open Graph ServiceMap Dashboards Data Flow Analysis DevDash AMMA & DevDash Indexes and tools Elastic Management of Containers Smart City Back OfficeSnap4City, IEEE SCALCOM, Leicester, August 2019 5
- 6.
- 7.
- 8.
- 9. Problematics, REQs • Implement DataShadow – Collect data that are coming from PUSH and PULL to create historical data and cover asynchronous requests for IOT data as well –Open to new protocols • indexing – Providing historical and last data with high performance, continuous machine learning – Faceted browsing of data indexed – GeoIndexed, geofiltering, temporal indexing – Time filtering, on time Drill Down • Fast enough to manage high throughput of data entering into the platform • Scalable, reasonable performance even for small to large data storage • Robust with respect to VM/host problems • REST API to access • Open Source solutions Snap4City, IEEE SCALCOM, Leicester, August 2019 9
- 10. Data Indexing Id Numerical Id of the IoT device or ETL data source. serviceUri URI of the IoT device or city entity. srcData source kind (IoT or ETL). kind Kind of IoT (sensor or actuator) deviceName Name of the IoT device or ETL. healthiness_criteria It shows if the device is “healthy”, according to specific rules (based, for instance, on value refresh rate) checked by dedicated scripts in the back‐office. sensorID Unique identifier of the IoT device or ETL data source. date_time Date and time related to the instant in which the measure/data has been provided by the IoT device or ETL. latlon Pair of latitude and longitude coordinates. geolocation Special geolocation format for geographical faceting functionalities. data_type The type of data representing the measure provided by the IoT device or ETL (integer, float, string. etc.). value_refresh_rate Frequency to which data or measure is provided (used also for checking the above‐described healthiness_criteria) value_type Type of the measure provided by the IoT device or ETL (temperature, humidity, speed etc.) value Actual value of the provided measure. value_name Name of the provided measure (MyRoomTemperature, AirHumidity etc.). Organization The identification of the organization for which the data is collected Snap4City, IEEE SCALCOM, Leicester, August 2019 10
- 11. Two Solutions Considered • SOLR‐cloud Index with shards •Banana for Visual Interface • API, Faceted • 4 VM with 4 shards, so that 4x4 16 shards – Banana on Node4 – automated failover with Zookeeper on 3VM Snap4City, IEEE SCALCOM, Leicester, August 2019 11 • Elastic Search Index with replicas • Kibana for Visual Interface • API, Nested Aggregations • 1VM master and 3VM Nodes – 5 primary shards + 2 replicas, so that 3x5=15 shards, over 3VM Nodes
- 12. The Two Solutions • Each VM: – 16 CPU x Logic 4 cores, 2.3 GHz –24 Gbyte Ram – HDD 500 Gbyte, 15Krps • 30 million of documents • 7 Gbyte of data • Workload for testing – A set of queries – Parallel requests Snap4City, IEEE SCALCOM, Leicester, August 2019 12 Banana
- 13. Test queries • executing – Faceted Search and Facets Pivot on SolrCloud, and –Aggregations and Nested Aggregations on Elasticsearch; • requesting sorted results for one or more fields, of the data table; • perform textual searches on textual fields of the data table. (e.g., the equivalent of the LIKE operator used in the SQL language); • perform temporal filtering on data, so called drill down. • 15 queries executed several times in parallel, for minutes Snap4City, IEEE SCALCOM, Leicester, August 2019 13
- 14. Response Time tests Number of Parallel Requests ElasticsearchSolrCloud Average Response Time (s) Variance Average Response Time (s) Variance T1 150 23,216 0,848 48,669 0,532 T2 750 75,097 18,031 407,444 3768,326 T3 1500 147,463 153,106 318,046 2633,315 T4 2250 484,289 452,823 601,223 10434,492 T5 4500 1060,429 2363,393 981,419 10407,077 T6 7500 1775,962 281418,250 Cluster Breakdown Snap4City, IEEE SCALCOM, Leicester, August 2019 14 ‐ Linear trend up to 4000 requests ‐ Over 4500 SOLR crashed so that it was not possible to go ahead. 0,000 200,000 400,000 600,000 800,000 1000,000 1200,000 1400,000 1600,000 1800,000 2000,000 0 1000 2000 3000 4000 5000 6000 7000 80 Number of requests Avg time Elasticsearch Solr Point of cluster breakdown
- 15. CPU % usage typical trends (T1, T2, T3, T4, T5) Snap4City, IEEE SCALCOM, Leicester, August 201915 0 10 20 30 40 50 60 70 Thu Sep 6 16:51:20 GMT+0200 2018 Thu Sep 6 16:52:40 GMT+0200 2018 Thu Sep 6 16:54:00 GMT+0200 2018 Thu Sep 6 16:55:20 GMT+0200 2018 Thu Sep 6 16:56:40 GMT+0200 2018 Thu Sep 6 16:58:00 GMT+0200 2018 Thu Sep 6 16:59:20 GMT+0200 2018 Thu Sep 6 17:00:40 GMT+0200 2018 Thu Sep 6 17:02:00 GMT+0200 2018 Thu Sep 6 17:03:20 GMT+0200 2018 Thu Sep 6 17:04:40 GMT+0200 2018 Thu Sep 6 17:06:00 GMT+0200 2018 Thu Sep 6 17:07:20 GMT+0200 2018 Thu Sep 6 17:08:40 GMT+0200 2018 Thu Sep 6 17:10:00 GMT+0200 2018 Thu Sep 6 17:11:20 GMT+0200 2018 Thu Sep 6 17:12:40 GMT+0200 2018 Thu Sep 6 17:14:00 GMT+0200 2018 Thu Sep 6 17:15:20 GMT+0200 2018 Thu Sep 6 17:16:40 GMT+0200 2018 Thu Sep 6 17:18:00 GMT+0200 2018 Thu Sep 6 17:19:20 GMT+0200 2018 Thu Sep 6 17:20:40 GMT+0200 2018 Thu Sep 6 17:22:00 GMT+0200 2018 Thu Sep 6 17:23:20 GMT+0200 2018 Thu Sep 6 17:24:40 GMT+0200 2018 Thu Sep 6 17:26:00 GMT+0200 2018 Thu Sep 6 17:27:20 GMT+0200 2018 Thu Sep 6 17:28:40 GMT+0200 2018 Thu Sep 6 17:30:00 GMT+0200 2018 Thu Sep 6 17:31:20 GMT+0200 2018 Thu Sep 6 17:32:40 GMT+0200 2018 Thu Sep 6 17:34:00 GMT+0200 2018 Thu Sep 6 17:35:20 GMT+0200 2018 Thu Sep 6 17:36:40 GMT+0200 2018 Thu Sep 6 17:38:00 GMT+0200 2018 Thu Sep 6 17:39:20 GMT+0200 2018 Thu Sep 6 17:40:40 GMT+0200 2018 Thu Sep 6 17:42:00 GMT+0200 2018 Thu Sep 6 17:43:20 GMT+0200 2018 Thu Sep 6 17:44:40 GMT+0200 2018 Thu Sep 6 17:46:00 GMT+0200 2018 Shard 1 Shard 2 Shard 3 Shard 4 0 10 20 30 40 50 60 Thu Sep 6 09:44:20 GMT+0200 2018 Thu Sep 6 09:45:40 GMT+0200 2018 Thu Sep 6 09:47:00 GMT+0200 2018 Thu Sep 6 09:48:20 GMT+0200 2018 Thu Sep 6 09:49:40 GMT+0200 2018 Thu Sep 6 09:51:00 GMT+0200 2018 Thu Sep 6 09:52:20 GMT+0200 2018 Thu Sep 6 09:53:40 GMT+0200 2018 Thu Sep 6 09:55:00 GMT+0200 2018 Thu Sep 6 09:56:20 GMT+0200 2018 Thu Sep 6 09:57:40 GMT+0200 2018 Thu Sep 6 09:59:00 GMT+0200 2018 Thu Sep 6 10:00:20 GMT+0200 2018 Thu Sep 6 10:01:40 GMT+0200 2018 Thu Sep 6 10:03:00 GMT+0200 2018 Thu Sep 6 10:04:20 GMT+0200 2018 Thu Sep 6 10:05:40 GMT+0200 2018 Thu Sep 6 10:07:00 GMT+0200 2018 Thu Sep 6 10:08:20 GMT+0200 2018 Thu Sep 6 10:09:40 GMT+0200 2018 Thu Sep 6 10:11:00 GMT+0200 2018 Thu Sep 6 10:12:20 GMT+0200 2018 Thu Sep 6 10:13:40 GMT+0200 2018 Thu Sep 6 10:15:00 GMT+0200 2018 Thu Sep 6 10:16:20 GMT+0200 2018 Thu Sep 6 10:17:40 GMT+0200 2018 Thu Sep 6 10:19:00 GMT+0200 2018 Thu Sep 6 10:20:20 GMT+0200 2018 Thu Sep 6 10:21:40 GMT+0200 2018 Thu Sep 6 10:23:00 GMT+0200 2018 Thu Sep 6 10:24:20 GMT+0200 2018 Thu Sep 6 10:25:40 GMT+0200 2018 Thu Sep 6 10:27:00 GMT+0200 2018 Thu Sep 6 10:28:20 GMT+0200 2018 Thu Sep 6 10:29:40 GMT+0200 2018 Thu Sep 6 10:31:00 GMT+0200 2018 Thu Sep 6 10:32:20 GMT+0200 2018 Thu Sep 6 10:33:40 GMT+0200 2018 Thu Sep 6 10:35:00 GMT+0200 2018 Master Node Data Node 1 Data Node 2 Data Node 3 T1 T2 T3 T4 T5 T1 T2 T3 T4 T5 SOLR Cloud Elastic Search
- 16. 0 2.000.000 4.000.000 6.000.000 8.000.000 10.000.000 12.000.000 14.000.000 16.000.000 Thu Sep 6 16:53:40 GMT+0200 2018 Thu Sep 6 16:55:20 GMT+0200 2018 Thu Sep 6 16:57:00 GMT+0200 2018 Thu Sep 6 16:58:40 GMT+0200 2018 Thu Sep 6 17:00:20 GMT+0200 2018 Thu Sep 6 17:02:00 GMT+0200 2018 Thu Sep 6 17:03:40 GMT+0200 2018 Thu Sep 6 17:05:20 GMT+0200 2018 Thu Sep 6 17:07:00 GMT+0200 2018 Thu Sep 6 17:08:40 GMT+0200 2018 Thu Sep 6 17:10:20 GMT+0200 2018 Thu Sep 6 17:12:00 GMT+0200 2018 Thu Sep 6 17:13:40 GMT+0200 2018 Thu Sep 6 17:15:20 GMT+0200 2018 Thu Sep 6 17:17:00 GMT+0200 2018 Thu Sep 6 17:18:40 GMT+0200 2018 Thu Sep 6 17:20:20 GMT+0200 2018 Thu Sep 6 17:22:00 GMT+0200 2018 Thu Sep 6 17:23:40 GMT+0200 2018 Thu Sep 6 17:25:20 GMT+0200 2018 Thu Sep 6 17:27:00 GMT+0200 2018 Thu Sep 6 17:28:40 GMT+0200 2018 Thu Sep 6 17:30:20 GMT+0200 2018 Thu Sep 6 17:32:00 GMT+0200 2018 Thu Sep 6 17:33:40 GMT+0200 2018 Thu Sep 6 17:35:20 GMT+0200 2018 Thu Sep 6 17:37:00 GMT+0200 2018 Thu Sep 6 17:38:40 GMT+0200 2018 Thu Sep 6 17:40:20 GMT+0200 2018 Thu Sep 6 17:42:00 GMT+0200 2018 Thu Sep 6 17:43:40 GMT+0200 2018 Thu Sep 6 17:45:20 GMT+0200 2018 Shard 1 Shard2 Shard 3 Shard 4 Memory usage: typical trends Snap4City, IEEE SCALCOM, Leicester, August 2019 16 SOLR Cloud (T1, T5) Elastic Search (T1, T5) 0 5.000.000 10.000.000 15.000.000 20.000.000 25.000.000 Thu Sep 6 10:53:00 GMT+0200 2018 Thu Sep 6 10:54:20 GMT+0200 2018 Thu Sep 6 10:55:40 GMT+0200 2018 Thu Sep 6 10:57:00 GMT+0200 2018 Thu Sep 6 10:58:20 GMT+0200 2018 Thu Sep 6 10:59:40 GMT+0200 2018 Thu Sep 6 11:01:00 GMT+0200 2018 Thu Sep 6 11:02:20 GMT+0200 2018 Thu Sep 6 11:03:40 GMT+0200 2018 Thu Sep 6 11:05:00 GMT+0200 2018 Thu Sep 6 11:06:20 GMT+0200 2018 Thu Sep 6 11:07:40 GMT+0200 2018 Thu Sep 6 11:09:00 GMT+0200 2018 Thu Sep 6 11:10:20 GMT+0200 2018 Thu Sep 6 11:11:40 GMT+0200 2018 Thu Sep 6 11:13:00 GMT+0200 2018 Thu Sep 6 11:14:20 GMT+0200 2018 Thu Sep 6 11:15:40 GMT+0200 2018 Thu Sep 6 11:17:00 GMT+0200 2018 Thu Sep 6 11:18:20 GMT+0200 2018 Thu Sep 6 11:19:40 GMT+0200 2018 Thu Sep 6 11:21:00 GMT+0200 2018 Thu Sep 6 11:22:20 GMT+0200 2018 Thu Sep 6 11:23:40 GMT+0200 2018 Thu Sep 6 11:25:00 GMT+0200 2018 Thu Sep 6 11:26:20 GMT+0200 2018 Thu Sep 6 11:27:40 GMT+0200 2018 Thu Sep 6 11:29:00 GMT+0200 2018 Thu Sep 6 11:30:20 GMT+0200 2018 Thu Sep 6 11:31:40 GMT+0200 2018 Thu Sep 6 11:33:00 GMT+0200 2018 Thu Sep 6 11:34:20 GMT+0200 2018 Thu Sep 6 11:35:40 GMT+0200 2018 Thu Sep 6 11:37:00 GMT+0200 2018 Thu Sep 6 11:38:20 GMT+0200 2018 Thu Sep 6 11:39:40 GMT+0200 2018 Thu Sep 6 11:41:00 GMT+0200 2018 Thu Sep 6 11:42:20 GMT+0200 2018 Thu Sep 6 11:43:40 GMT+0200 2018 Thu Sep 6 11:45:00 GMT+0200 2018 Thu Sep 6 11:46:20 GMT+0200 2018 Thu Sep 6 11:47:40 GMT+0200 2018 Master Node Data Node 1 Data Node 2 Data Node 3
- 17. Network Usage (T1, T2, T3, T4, T5) Snap4City, IEEE SCALCOM, Leicester, August 2019 17 SOLR CloudElastic Search 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 Thu Sep 6 09:45:20… Thu Sep 6 09:47:00… Thu Sep 6 09:48:40… Thu Sep 6 09:50:20… Thu Sep 6 09:52:00… Thu Sep 6 09:53:40… Thu Sep 6 09:55:20… Thu Sep 6 09:57:00… Thu Sep 6 09:58:40… Thu Sep 6 10:00:20… Thu Sep 6 10:02:00… Thu Sep 6 10:03:40… Thu Sep 6 10:05:20… Thu Sep 6 10:07:00… Thu Sep 6 10:08:40… Thu Sep 6 10:10:20… Thu Sep 6 10:12:00… Thu Sep 6 10:13:40… Thu Sep 6 10:15:20… Thu Sep 6 10:17:00… Thu Sep 6 10:18:40… Thu Sep 6 10:20:20… Thu Sep 6 10:22:00… Thu Sep 6 10:23:40… Thu Sep 6 10:25:20… Thu Sep 6 10:27:00… Thu Sep 6 10:28:40… Thu Sep 6 10:30:20… Thu Sep 6 10:32:00… Thu Sep 6 10:33:40… Cluster Elasticsearch ‐ Network Usage (Kbit/s) Master Node Data Node 1 Data Node 2 Data Node 3 0 10000 20000 30000 40000 50000 60000 70000 80000 Thu Sep 6 16:52:00… Thu Sep 6 16:53:40… Thu Sep 6 16:55:20… Thu Sep 6 16:57:00… Thu Sep 6 16:58:40… Thu Sep 6 17:00:20… Thu Sep 6 17:02:00… Thu Sep 6 17:03:40… Thu Sep 6 17:05:20… Thu Sep 6 17:07:00… Thu Sep 6 17:08:40… Thu Sep 6 17:10:20… Thu Sep 6 17:12:00… Thu Sep 6 17:13:40… Thu Sep 6 17:15:20… Thu Sep 6 17:17:00… Thu Sep 6 17:18:40… Thu Sep 6 17:20:20… Thu Sep 6 17:22:00… Thu Sep 6 17:23:40… Thu Sep 6 17:25:20… Thu Sep 6 17:27:00… Thu Sep 6 17:28:40… Thu Sep 6 17:30:20… Thu Sep 6 17:32:00… Thu Sep 6 17:33:40… Thu Sep 6 17:35:20… Thu Sep 6 17:37:00… Thu Sep 6 17:38:40… Thu Sep 6 17:40:20… Thu Sep 6 17:42:00… Thu Sep 6 17:43:40… Thu Sep 6 17:45:20… Thu Sep 6 17:47:00… SolrCloud ‐ Network Usage (Kbit/s) Shard 1 Shard 2 Shard 3 Shard 4 T1 T2 T3 T4 T5 T1 T2 T3 T4 T5
- 18.
- 19. Querying on User Interface • Dashboard’s loading. First opening of the Visual Analytic tool on browser with the recent data view. •QueryA: a drill down operation is performed, which selects a time frame of interest from the histogram panel (time filtering). • QueryB: a temporal filtered search is performed on a histogram panel, and a Facet Fields filtering is applied on: device name (field "deviceName"), data type returned by the sensor (field "value_type") and unit of measure of the returned value (field "value_unit"). • QueryC: a specific sensor is searched by geo‐spatial filtering from the map panel and a temporal range of interest is selected. • QueryD: any value of a field from the search bar is searched, in this case the specific name of a device (field “deviceName”). Snap4City, IEEE SCALCOM, Leicester, August 2019 19
- 20. Kibana vs Banana Banana-SOLRCloud Number ofRequests Transferred Bytes (KB) AVG Response Time (s) Dash Loading 186 5900 28,55 QueryA 26 846 11,09 QueryB 76 1900 96 QueryC 175 3300 68,5 QueryD 28 776 14 Kibana-ElasticSercch Number of Requests Transferred Bytes (KB) AVG Response Time (s) Dash Loading 57 3500 6,93 QueryA 14 25,4 3,8 QueryB 10 52,2 48 QueryC 199 3300 35 QueryD 5 19,4 0,45 Snap4City, IEEE SCALCOM, Leicester, August 2019 20
- 21. Conclusions • SOLR‐Banana vs ElasticSearch‐Kibana comparison • Elastic Search demonstrated to –Provide better performance in • Loading • Querying with faceted/aggregation and drill down – Be more robust when massive search queries are performed • The tests have been continued on Elastic Search until billions of elements in the storage and thousands of queries at the same time. The solution has been scaled adding more Nodes but the solution still remain robust and effective. • Snap4City is from today an EOSC Smart City IOT as a Service Snap4City, IEEE SCALCOM, Leicester, August 2019 21