Taming the ever-evolving Compliance Beast : Lessons learnt at LinkedIn [Strat...Shirshanka Das
Just when you think you have your Kafka and Hadoop clusters set up and humming and you’re well on your path to democratizing data, you realize that you now have a very different set of challenges to solve. You want to provide unfettered access to data to your data scientists, but at the same time, you need to preserve the privacy of your members, who have entrusted you with their data.
Shirshanka Das and Tushar Shanbhag outline the path LinkedIn has taken to protect member privacy in its scalable distributed data ecosystem built around Kafka and Hadoop.
They also discuss three foundational building blocks for scalable data management that can meet data compliance regulations: a centralized metadata system, a standardized data lifecycle management platform, and a unified data access layer. Some of these systems are open source and can be of use to companies that are in a similar situation. Along the way, they also look to the future—specifically, to the General Data Protection Regulation, which comes into effect in 2018—and outline LinkedIn’s plans for addressing those requirements.
But technology is just part of the solution. Shirshanka and Tushar also share the culture and process change they’ve seen happen at the company and the lessons they’ve learned about sustainable process and governance.
Taming the ever-evolving Compliance Beast : Lessons learnt at LinkedIn [Strat...Shirshanka Das
Just when you think you have your Kafka and Hadoop clusters set up and humming and you’re well on your path to democratizing data, you realize that you now have a very different set of challenges to solve. You want to provide unfettered access to data to your data scientists, but at the same time, you need to preserve the privacy of your members, who have entrusted you with their data.
Shirshanka Das and Tushar Shanbhag outline the path LinkedIn has taken to protect member privacy in its scalable distributed data ecosystem built around Kafka and Hadoop.
They also discuss three foundational building blocks for scalable data management that can meet data compliance regulations: a centralized metadata system, a standardized data lifecycle management platform, and a unified data access layer. Some of these systems are open source and can be of use to companies that are in a similar situation. Along the way, they also look to the future—specifically, to the General Data Protection Regulation, which comes into effect in 2018—and outline LinkedIn’s plans for addressing those requirements.
But technology is just part of the solution. Shirshanka and Tushar also share the culture and process change they’ve seen happen at the company and the lessons they’ve learned about sustainable process and governance.
DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appl...Yuta Takahashi
This document describes DeepRemote, a smart remote controller that uses deep learning for intuitive home appliance selection and control. It consists of a control unit with a camera and buttons and a deep learning unit for appliance recognition. The system was tested for classification accuracy of over 80% on average, response time of under 2 seconds, and faster control times than traditional remotes in user tests. Overall, DeepRemote demonstrates an effective deep learning approach for selecting and controlling home appliances intuitively with a single remote controller.
An Identification Method of IR Signals to Collect Control Logs of Home Applia...Yuta Takahashi
This document proposes a method to identify infrared (IR) signals from home appliances in order to collect control logs. It involves preprocessing raw IR signals into pulse width sequences, comparing signals using mean absolute error and sum absolute error, and constructing statistical models to identify appliance type with 95.5% accuracy and command type with 92% accuracy based on a database of 1,400 signals from 14 appliances. A simple simulation shows identification stability is achieved when the database includes 6 or more signals per appliance. The method could help automatically understand user preferences from appliance usage logs.
DeepRemote: A Smart Remote Controller for Intuitive Control through Home Appl...Yuta Takahashi
This document describes DeepRemote, a smart remote controller that uses deep learning for intuitive home appliance selection and control. It consists of a control unit with a camera and buttons and a deep learning unit for appliance recognition. The system was tested for classification accuracy of over 80% on average, response time of under 2 seconds, and faster control times than traditional remotes in user tests. Overall, DeepRemote demonstrates an effective deep learning approach for selecting and controlling home appliances intuitively with a single remote controller.
An Identification Method of IR Signals to Collect Control Logs of Home Applia...Yuta Takahashi
This document proposes a method to identify infrared (IR) signals from home appliances in order to collect control logs. It involves preprocessing raw IR signals into pulse width sequences, comparing signals using mean absolute error and sum absolute error, and constructing statistical models to identify appliance type with 95.5% accuracy and command type with 92% accuracy based on a database of 1,400 signals from 14 appliances. A simple simulation shows identification stability is achieved when the database includes 6 or more signals per appliance. The method could help automatically understand user preferences from appliance usage logs.