Big data, data science, machine learning is coming to a lot of companies. Everyone is used to the creation of ordinary software, but BD/DS/ML requires special care. Managers and developers may get unfamiliar problems and I want to tell you about them and solutions - no money and nerves should be wasted. Everyone has heard of data science, machine learning and big data. Many companies are starting to build up teams and run projects. Everyone knows how to develop, deliver and deploy ordinary software, but data-driven software is a different animal. Scientists, developers and managers may not be familiar with the issues that may come up.

2. Me

5 years as a software developer

3 years in working with data

Worked in 3 companies (1 startup)

~ 15 projects

3. The context

Your company has valuable data
− Maybe lots of it

You want to create predictive
models/algorithms

You want to integrate it into your product

4. What would you want

“Let's start doing something!”

Create a process:
− Your raw data → … → … → … → Happy end-user

Robust

Monitorable

5. All starts with writing code

Model/algorithm creation needs writing code

Amazing data scientists come from academia
(physics, chemistry, math, CS)

Code in academia is a means to an end

Developers hate academics' code

6. How to test it?

Testability must be considered in advance!

Testing different parts of code with unit tests?

Testing functionality?

Code output: number. Or several.

7. How to test it?

Testability must be considered in advance!

Testing different parts of code with unit tests?

Testing functionality?

Code output: number. Or several.

8. How to test it?

Small test dataset != real dataset !=
representable dataset

data1.csv, data2.csv, data21.csv, data3.csv,
data3-old.csv
− please, keep better track of datasets used for
development :(

9. Let's deploy!

Danger: too full-stack data scientist

CI pipeline should contain:
− Back-testing
− Newest-data-testing

Human is too human (lazy)

Model quality will degrade over time

10. Let's deploy!

Danger: too full-stack data scientist

CI pipeline should contain:
− Back-testing
− Newest-data-testing

Human is too human (lazy)

Model quality will degrade over time

11. Tools

Some offers that automate parts of work
− Data preparation, model creation, deployment

They put you inside a specific box

They are not free

12. Tools

Microsoft Azure, Amazon ML – visual creation
− Tradeoff: development speed VS best fit for your
data
− Also privacy policies

13. Operations

Computations, data transformations
− Violent spikes in RAM demand

Distributed storage (esp. HBase)
− Violent spikes in bandwidth demand

14. Speaking of Hadoop

Do you really need it?

Hadoop = fancy hammer, all data = nails?

It's “unstable”

It's not very visual
− What is visual is often poor

15. Speaking of Hadoop

Data lineage is tricky

Security (Kerberos) is tricky

Data consistency when modifying ETL tasks
− Apache Oozie
− Apache NiFi

16. Speaking of Hadoop

Data lineage is tricky

Security (Kerberos) is tricky

Data consistency when modifying ETL tasks
− Apache Oozie
− Apache NiFi

17. If you choose Hadoop

Consider Hortonworks, Cloudera, MapR
− They are free of charge

Doing it yourself – if you are expert or paid to
tinker

18. If you choose Hadoop

Don't rely on more classic monitoring tools

Hortonworks, Cloudera, MapR provide
excellent monitoring

19. Hardware

Big corporate customers – fanciest hardware

I've operated an old PC cluster – totally OK

Amazon EMR + S3