Big & Open Data: Challenges for Smartcity


Published on

This work is about how both private enterprise and government wish to improve their data value and how they deal with this issue. The talk summarizes the way of thinking about Big Data, Open Data and their use by organizations or individuals. Big Data is explained from collecting, storing, analyzing and put in value. This data is collected from numerous sources including sensor networks, government data holdings, company market databases, and public profiles on social networking sites. Organizations use many data analytical techniques to study both structured and unstructured data. Due to the volume, velocity and variety of data, some specific techniques have been developed. MapReduce, Hadoop and other related as RHadoop are trending topic nowadays.
Data which come from government must be open. Every day more and more cities and countries are opening their data. Open Data is then presented as a specific case of public data with a special role in Smartcity. The main goal of Big and Open Data in Smartcity is to develop systems which can be useful for citizens. In this sense RMap (Mapa de Recursos) is shown as an Open Data application, an open system for Madrid City Council, avalaible for smarthphones and totally developed by the researching group G-TeC (

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Big & Open Data: Challenges for Smartcity

  1. 1. Big and Open data. Challenges for Smartcity Victoria López Grupo G-TeC Universidad Complutense de Madrid ICIST 2014 Valencia 1
  2. 2. Index • Introduction • Fighting with Big Data: Genoma data • What is Big Data? • Technology transfer: Open Data opportunities • Developing projects for Smartcity. • Rmap, a real example in Madrid • Conclusions 2
  3. 3. Introduction – Mobile technologies – Intelligent agents – Optimization and forecasting – Bioinformatics, Biostatistics – … – 3
  4. 4. Fighting with the Big Data • Every day we need to deal with more and more data. • For many years, new computers with more memory and higher speed seem to be the solution for data growing. • Many researching areas which was fighting with the Big Data: Bioinformatics, Genoma data, DNA, RNA, proteins and, in general all biological data have been required by computing monitors and storing in large data bases in several laboratories and researching centers along the world. The future of genomics rests on the foundation of the Human Genome Project4
  5. 5. Fighting with the Big Data • Each time an organization or an individual is not able to deal with data, a big data problem is facing. • Same philosophy than modern Big Data: large data bases distributed along the world with parallel processing when available and suitable • (Sequence alignment and Dynamic Programming) • The amount of biological data is a big data base. 5
  6. 6. Big Data From Data Warehouse to Big Data 6 1970 relational model invented RDBMS declared mainstream till 90s One-size fits all, Elephant vendors- heavily encoded even indexing by B-trees.
  7. 7. Alex ' Sandy' Pentland, director of 'Media Lab' at Massachusetts Institute of Technology (MIT) 7 Nowadays bussiness needs a high avalailability of data, then new techniques must be developed: Complex analytics, Graph Databases
  8. 8. unstructured data 8 ¿Quién genera Big Data? Progress and innovation are no longer hampered by the ability to collect data, but the ability to manage, analyze, synthesize, visualize, and discover knowledge from data collected in a timely manner and in a scalable way
  9. 9. Big Data Big Data 3+1+1 V’s 9
  10. 10. Big Data 1. High Availability is now a requirement 2. Host and Cloudcomputing 3. Running in parallel 1. Data Aggregation process 2. Analytics on Data 3. GraphDBMSs similarities 4. Not only SQL: Cassandra* and MongoDB** 5. Moving toward ACID, people from Google admit ACID as a good idea for working with dababases. *The Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance. **Document oriented storage 10 MONGO
  11. 11. 11 • Main feature: scalability to many nodes – Scan of 100 TB in 1 node @ 50 MB/sec = 23 days – Scan in a cluster of 1000 nodes = 33 minutes MapReduce – Parallel programming model – Simple concept, smart, suitable for multiple applications – Big datasets  multi-node in multiprocessors – Sets of nodes: Clusters or Grids (distributed programming) • By Google (2004) – Able to process 20 PB per day – Based on Map & Reduce, classiclal methods in functional programming related to the classic divide & conquer – Come from numeric analysis (big matrix products). Big Data: Map Reduce MapReduce
  12. 12. • Friendly for non technical users Map Reduce 12 Big Data: Map Reduce
  13. 13. – UsedbyYahoo!,Facebook,Twitter Amazon,eBay… – Canbeusedindifferentarchitectures: bothclusters(in-house)andgrid (Cloudcomputing) Hadoop 13 Big Data: Hadoop
  14. 14. Big Data: Datamining & Scalability • Techniques of Datamining (Machine Learning, Data Clustering, Predictive Models, etc.) are compatible with big data by complex analytics • Modeling prices in electricity Spanish markets under uncertainty G. Miñana, H. Marrao, R. Caro, J. Gil, V. Lopez, B. González , F. Sun et al. (eds.), Knowledge Engineering and Management, Advances in Intelligent Systems and Computing 214,DOI: 10.1007/978-3-642-37832- 4_46, Springer-Verlag Berlin Heidelberg 2014 • To get a scalable system – Aggregation – Generalization – (Formal specification) • Not only many cores, many nodes and out of memory data - Host and Cloudcomputing - Not all problems can be solve with the same techniques, Hadoop is not enough 14
  15. 15. Technology transfer • A great oportunity for researchers working to transfer technology, who can increase their efforts in developing new techniques for – Monitoring data (Sensors, smartphones, …) – Storing data (Cloudcomputing, Amazon S3, EC2, Google BigQuery, Tableau …) – Cleaning, Integrating & Processing data – data (Data Curation at Scale: The Data Tamer System, M. Stonebraker et al., CIDR 2013) – Analysing data (R, SAS… but also Google, Amazon, eBay..) – Fully homomorphic encryption & searching on encrypted data 15
  16. 16. Open Data “Open data is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and sharealike.” - “Open data is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and share alike.” Availability and Access: the data must be available as a whole and at no more than a reasonable reproduction cost, preferably by downloading over the internet. The data must also be available in a convenient and modifiable form. Reuse and Redistribution: the data must be provided under terms that permit reuse and redistribution including the intermixing with other datasets. The data must be machine- readable. Universal Participation: everyone must be able to use, reuse and redistribute – there should be no discrimination against fields of endeavour or against persons or groups. For example, ‘non-commercial’ restrictions that would prevent ‘commercial’ use, or restrictions of use for certain purposes (e.g. only in education), are not allowed. 16
  17. 17. Open Data 17
  18. 18. Why Open Data by Open Knowledge Foundation 18
  19. 19. Open Data for Smartcity • What a citizen can expect when living in a city? • Internet of the things – Libraries – Public transportation, trafic monitoring – Pets, devices, cars, even people • Intelligent agents – Interacting without our control – Credit cards control (BBVA case of use) 19
  20. 20. Basic structure Patrón Cliente/Servidor PUBLIC DATA Web Service SERVER CLIENT WEB SERVER 20
  22. 22. 22
  23. 23. Data Analytics FROM (UNSTRUCTURED) DATA TO VALUE 23
  24. 24. Mariam Saucedo Pilar Torralbo Daniel Sanz Ana Alfaro Sergio Ballesteros Lidia Sesma Héctor Martos Álvaro Bustillo Arturo Callejo Belén Abellanas Jaime Ramos Ignacio P. de Ziriza Victor Torres Alberto Segovia Miguel Bueno Mar Octavio de Toledo Antonio Sanmartín Carlos Fernández MAPA DE RECURSOS RECYCLA.TE 24
  25. 25. • Parks and gardens • Parkings for • Cars • Motorbikes • Bikes • Recycing Points • Fixed • Mobile • Cloths • Stations • Bioetanol • Gas • Oil • Electric • Routes for bikes • Vías ciclistas • Calles seguras • Áreas de Prioridad Residencial Madrid – Smart City RMapRMap 25
  26. 26. 26
  27. 27. Big and Open data. Challenges for Smartcity Victoria López Grupo G-TeC Universidad Complutense de Madrid ICIST 2014 Valencia