Big Data Day LA 2015 - Using data visualization to find patterns in multidimensional data by Amelia McNamara of UCLA

•

3 likes•2,214 views

While machine learning methods have made great strides in predictive analytics, there are many components of data science that still require human intervention. In particular, people are great at finding visual patterns in data. John Tukey was talking about exploratory data analysis in the 1970s, but advances in computer graphics have given us additional powers. I'll demonstrate methods for finding patterns in high-dimensional data, including the generalized pairs plot, the Grand Tour, and the lineup protocol for graphical inference. Of course, we will be implementing these methods using R and Shiny.

Technology

library(ggplot2)
data(diamonds)
set.seed(123)
diamonds <- diamonds[sample(500),]
str(diamonds)
## 'data.frame': 500 obs. of 10 variables:
## $ carat : num 0.7 0.32 0.7 0.7 0.86 0.23 0.71 0.9 0.7 0.72 ...
## $ cut : Ord.factor w/ 5 levels "Fair"<"Good"<..: 3 4 2 5 4 3 4 1 3 5 ...
## $ color : Ord.factor w/ 7 levels "D"<"E"<"F"<"G"<..: 3 6 2 5 3 5 3 7 2 1 ...
## $ clarity: Ord.factor w/ 8 levels "I1"<"SI2"<"SI1"<..: 5 3 4 3 2 5 5 4 4 3 ...
## $ depth : num 60 61 64.1 61.4 59.3 61 61.8 65 62.9 60.8 ...
## $ table : num 57 59 59 56 62 57 59 56 57 57 ...
## $ price : int 2767 554 2777 2815 2818 353 2790 2815 2793 2782 ...
## $ x : num 5.8 4.39 5.64 5.72 6.36 3.94 5.73 6.08 5.66 5.76 ...
## $ y : num 5.87 4.36 5.59 5.77 6.22 3.96 5.69 6.04 5.69 5.75 ...
## $ z : num 3.5 2.67 3.6 3.53 3.73 2.41 3.53 3.94 3.57 3.5 ...

library(dplyr)
select(diamonds, carat, depth, table) %>% plot()

library(gpairs)
gpairs(diamonds, outer.rot=c(90, 0))

select(diamonds, carat, cut, color, depth) %>% gpairs(outer.rot=c(90, 0))

library(tourr)
select(diamonds, carat, depth, table, x, y, z) %>% animate(grand_tour(), display = display_x

library(nullabor)
library(ggplot2)
qplot(carat, depth, data=diamonds) %+%
lineup(null_permute('carat'), diamonds) +
facet_wrap(~.sample)
## decrypt("OlCE bQTQ Aw GWPATAWw qd")

decrypt("OlCE bQTQ Aw GWPATAWw qd")
## [1] "True data in position 17"

Big Data Day LA 2015 - Using data visualization to find patterns in multidimensional data by Amelia McNamara of UCLA

A prioritized user story backlog helps to understand what to do next, but is a difficult tool for understanding what your whole system is intended to do. A user story map arranges user stories into a useful model to help understand the functionality of the system, identify holes and omissions in your backlog, and effectively plan holistic releases that delivery value to users and business with each release.

DBI

Lambert Lum

Jasmine frontinrio

Andre Fonseca

PhoneGap: Local Storage

Ivano Malavolta

Annayya 01venkatesha9

Crash Course to SQL in PHPwebhostingguy

Informe Final Red de Area ComunalMaria Tobon

CGI::Prototype (NPW 2006)brian d foy

wget.plYasuhiro Onishi

Bouncingballs sh

Ben Pope

JQuery In RailsLouie Zhao

De CRUD à DDD pas à pas

Charles Desneuf

Comment migrer une application peu testée, développée avec des outils RAD, pour intégrer de nombreuses évolutions de règles métiers et par la même occasion la rendre plus compréhensible, en ayant une démarche orientée Domain Driven Design ? La réponse n’est pas si évidente. On peut bien sûr opter pour une réécriture complète du système mais ce n’est pas sans problème. Comment continuer à délivrer des fonctionnalités ? Comment s’assurer que le nouveau système aura bien le même comportement que l’ancien ? Et surtout, combien de temps cela va t’il prendre ? Une autre piste est de refactorer le code petit à petit, d’introduire les nouveaux concepts lorsqu’un besoin se fait sentir, sans avoir besoin de bloquer tout autre développement. C’est ce que nous avons choisi de faire pour migrer un backoffice construit autour du bundle Symfony EasyAdmin. Dans ce retour d’expérience nous verrons comment nous avons commencé à mieux connaître les règles métiers en lisant le code et en animant des Event Storming. Nous parlerons de la manière dont nous avons réussi à créer un filet de sécurité avec des tests en nous laissant guider par le code coverage, ou encore de comment nous avons introduit graduellement les nouveaux concepts métiers à l’aide des interfaces avant de modifier la base de données. Tout au long de la conférence nous présenterons les difficultés rencontrées et ce que nous avons mis en œuvre pour les déjouer.

Tajseed Arab Infographics

Tajseed | تجسيد

Daily notes

meghendra168

Do an say ca phe thung quayThịnh Đức

TerminalでTwitter

Yoshihiro Sugi

Dot pab forum september 2011The Social Executive

Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter

Data Con LA

Tajo is an advanced open source data warehouse system on Hadoop. Tajo has rapidly evolved over couple of years. In this talk, I will present how Tajo has been improved for years. In particular, this talk will introduce new features of the most recent major release Tajo 0.10: Hbase storage support, thin JDBC driver, direct JSON support, and better Amazon EMR support. Then, I will present the upcoming features that currently Tajo community is doing: Multi-tenant scheduler, allowing multiple users to submit multiple queries into one cluster; nested schema support, allowing users to directly handle complex data types without flattening; more advanced SQL features like WITH clause, window frame, and subqueries.

Big Data Day LA 2015 - The Big Data Journey: How Big Data Practices Evolve at...

Data Con LA

It isn't easy to drink from the technology firehose of today's Internet economy. At Connexity, we have gone from home-grown MapReduce frameworks and custom in-house search-engines to extensive use of Apache Hadoop, Hive, Pig, Cassandra, Solr and other technologies to power our business. This talk will explore some of the evolutionary steps that we've made and what lessons you might draw from our 15+ years of experience of swimming with the Internet sharks.

What's hot

04 2 오버플로 상수 매크로

Changwon National University

ZAPP_automotive_museumJakub Goralski

Who Needs Ruby When You've Got CodeIgniterciconf

User Story Mapping

Naresh Jain

DBI

Lambert Lum

Jasmine frontinrio

Andre Fonseca

PhoneGap: Local Storage

Ivano Malavolta

Annayya 01venkatesha9

Crash Course to SQL in PHPwebhostingguy

Informe Final Red de Area ComunalMaria Tobon

CGI::Prototype (NPW 2006)brian d foy

wget.plYasuhiro Onishi

Bouncingballs sh

Ben Pope

JQuery In RailsLouie Zhao

De CRUD à DDD pas à pas

Charles Desneuf

Tajseed Arab Infographics

Tajseed | تجسيد

Daily notes

meghendra168

Do an say ca phe thung quayThịnh Đức

TerminalでTwitter

Yoshihiro Sugi

What's hot (19)

04 2 오버플로 상수 매크로

ZAPP_automotive_museum

Who Needs Ruby When You've Got CodeIgniter

User Story Mapping

DBI

Jasmine frontinrio

PhoneGap: Local Storage

Annayya 01

Crash Course to SQL in PHP

Informe Final Red de Area Comunal

CGI::Prototype (NPW 2006)

wget.pl

Bouncingballs sh

JQuery In Rails

De CRUD à DDD pas à pas

Tajseed Arab Infographics

Daily notes

Do an say ca phe thung quay

TerminalでTwitter

Viewers also liked

Dot pab forum september 2011The Social Executive

Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter

Data Con LA

Big Data Day LA 2015 - The Big Data Journey: How Big Data Practices Evolve at...

Data Con LA

101129 tokyopref bochibochiredgang

Big Data Day LA 2015 - Transforming into a data driven enterprise using exist...

Data Con LA

Leading entrepreneurial outfits are disrupting traditional companies by rapidly building data-driven apps. They employ top software talent and effectively use storage, analytics and app-dev tools from various open source ecosystems. We show how companies of all sizes are now transforming into data-driven enterprises using their existing software skill sets by leveraging a single platform that combines flexible data storage systems, advanced analytics and agile app-dev PaaS frameworks, all available now in open source forums.

Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...

Data Con LA

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.

Data Con LA

Jay Kreps, Open Source Visionary and Co Founder of Confluent and several open source projects will be visiting LA. I have asked him to come present at our group. He will present his vision and will answer questions regarding Kafka and other projects Bio:- Jay is the co-founder and CEO at Confluent a company built around realtime data streams and the open source messaging system Apache Kafka. He is the original author of several of open source projects including Apache Kafka, Apache Samza, Voldemort, and Azkaban.

Big Data Day LA 2015 - Lessons learned from scaling Big Data in the Cloud by...

Data Con LA

Companies analyzing big data help achieve important business objectives such as customer retention, real-time in-context marketing, omni-channel marketing productivity, campaign productivity and operational efficiencies. Cloud-based big data architectures create lower risk, lower startup costs and faster time-to-market. This session will examine the key advantages from deploying big data in the cloud, such as the flexibility to auto scale and the ability to experiment with on-demand and hybrid nodes. We will also discuss lessons learned from big data in the cloud, such as how to avoid bottlenecks by building caches or how to design instances to leverage spotting.

Big Data Day LA 2016/ Data Science Track - Data Storytelling for Impact - Dav...

Data Con LA

How can our data make the biggest impact? How do we find the stories worth sharing buried in our analytics? How important are visuals, hooks, connections, content? As data science and journalism have co-evolved, the potential for effectively communicating with data has skyrocketed. We'll look at case studies of impactful data stories and share the process for developing data stories that drive action.

Do you know how the ultra affluent use social media? Find out.

The Social Executive

The social media real estate you put time into is as important as the suburb you invest in. The right place at the right price is what gives good returns. For time-poor professionals looking to start out in social media the sheer number of platforms to choose from can feel overwhelming – LinkedIn, Twitter, Facebook, YouTube, Pinterest, Google Plus? It’s a bit like selecting the ‘all suburbs’ search when you are trying to find somewhere to live. While I am loathe to suggest one platform to the exclusion of others (because together they create an amplification effect) if you’re a professional or need to reach high net worth individuals then research suggests that a great place to live is LinkedIn. This is why.

Spark after Dark by Chris Fregly of Databricks

Data Con LA

Spark After Dark is a mock dating site that uses the latest Spark libraries, AWS Kinesis, Lambda Architecture, and Probabilistic Data Structures to generate dating recommendations. There will be 5+ demos covering everything from basic data ETL to advanced data processing including Alternating Least Squares Machine Learning/Collaborative Filtering and PageRank Graph Processing. There is heavy emphasis on Spark Streaming and AWS Kinesis. Watch the video here https://www.youtube.com/watch?v=g0i_d8YT-Bs

Spark Streaming& Kafka-The Future of Stream Processing by Hari Shreedharan of...

Data Con LA

Abstract:- With its easy to use interfaces and native integration with some of the most popular ingest tools, such as Kafka, Flume, Kinesis etc, Spark Streaming has become go-to tool for stream processing. Code sharing with Spark also makes it attractive. In this talk, we will discuss the latest features in Spark Streaming and how it integrates with Kafka natively with no data loss, and even do exactly once processing! Bio:- Hari Shreedharan is a PMC member and committer on the Apache Flume Project. As a PMC member, he is involved in making decisions on the direction of the project. Author of the O’Reilly book Using Flume, Hari is also a software engineer at Cloudera, where he works on Apache Flume, Apache Spark, and Apache Sqoop. He also ensures that customers can successfully deploy and manage Flume, Spark, and Sqoop on their clusters, by helping them resolve any issues they are facing.

6 damaging myths about social media and the truths behind them

The Social Executive

Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...

Data Con LA

Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...

Data Con LA

Today’s Software Defined environments attempt to remove the weakness of computing hardware from the operational equation. There is no doubt that this is a natural progress away from overpriced, proprietary compute and storage layers. However, even at the heart of any Software Defined universe is an underlying hardware stack that must be robust, reliable and cost effective. Our 20+ years experience delivering over 2000 clusters and clouds has taught us how to properly design and engineer the right hardware solution for Big Data, Cluster and Cloud environments. This presentation will share this knowledge allowing user to make better design decisions for any deployment.

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

Data Con LA

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes per day of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. This session is especially recommended for data infrastructure engineers and architects planning, building, or maintaining similar systems.

Big Data Day LA 2016/ NoSQL track - Big Data and Real Estate, Jon Zifcak, CEO...

Data Con LA

Big Data Day LA 2016/ Data Science Track - Intuit's Payments Risk Platform, D...

Data Con LA

Big Data Day LA 2016/ Use Case Driven track - How to Use Design Thinking to J...

Data Con LA

Big Data Day LA 2015 - Event Driven Architecture for Web Analytics by Peyman ...

Data Con LA

As integrated web analytics evolves to both a service oriented and event based model, there will be higher emphasis on moving toward event based analytics. Business analytics is moving from purely counts of analytics to time-series, relationship and usage analytics. Examples of web analytics that can take advantage of this architecture are conversions analytics or cross channel marketing. The advantage of storing raw event data is that you have maximum flexibility for analysis. For example, you can trace the sequence of pages that one person visited over the course of their session. You can’t do that if you’ve squashed all the events into e.g. counters. That sort of analysis is really important for some offline processing tasks, such as training a recommender system (“people who bought X also bought Y”, that sort of thing). For such use cases, it’s best to simply keep all the raw events, so that you can later feed them all into your shiny new machine learning system. In this session we are going to elaborate on using Kafka, an Event Processing framework (e.g. Storm or Spark Streaming) and either Hadoop or EDW for building an Event Driven Architecture.

Viewers also liked (20)

Dot pab forum september 2011

Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter

Big Data Day LA 2015 - The Big Data Journey: How Big Data Practices Evolve at...

101129 tokyopref bochibochi