Presented at PyData London, 6 Dec 2016

1. Rich Relational Data
from Thin-Air
How to Fake it – Pydata London
6 Dec 2016
John Stinson

2. Why Simulate Data?
• Early System Testing
• Cloud Migration
• Training Course

3. Device Type over Time
Source: https://www.nakono.com/tekcarta/company-profiles/microsoft-xbox-one/microsoft-xbox-360-and-
xbox-one-developer-programme-market-adoption-and-analysis/

4. Usage by Time of Day
Source: http://downloads.bbc.co.uk/mediacentre/iplayer/iplayer-performance-jan16.pdf

5. Tables
• users
• page
• device
• page_hit

6. Star Schema
page
device
page_hit
users
Fact Table
Dimension
Tables

7. Device Table
device
device_id (PK) 1, 2, 3
device_name Tablet, Mobile Device, Computer
devices = [
{'device_id': 1, 'device_name': 'Tablet'},
{'device_id': 2, 'device_name': 'Mobile Device'},
{'device_id': 3, 'device_name': 'Computer'}
]

8. Star Schema
page
device
page_hit
users
Fact Table
Dimension
Tables

9. Page Table
page
page_id (PK) 214
url
http://.../radio/... ,
http://.../iplayer/ …
platform radio, iplayer

10. Chrome Browser History
C:Users<user>AppDataLocalGoogleChromeUser DataDefaultHistory (Windows)
~Library/Application Support/Google/Chrome/Default/History (Mac)

11. Regular Expression
re = 'https?://bbc.co.uk[a-zA-Z0-9/.:-? _=,]+'
url_list = re.findall(re, file_contents)
http://bbc.co.uk/http://www.bbc.co.uk/iplayer

12. Closer Look at Browser History

14. Read in SQLite dump as dictionary
reader = csv.DictReader(my_file, delimiter='t')
print(reader.fieldnames)
urls = [(row['url'], row['title'], row['visit_count'])
for row in reader
if 'bbc.co.uk' in row['url']]
for url, title, visitcount in urls:
print(url, '|', title, '|', visitcount)
['id', 'url', 'title', 'visit_count', 'typed_count', 'last_visit_ti
http://www.bbc.co.uk/iplayer | BBC iPlayer | 53
http://www.bbc.co.uk/iplayer/guide | BBC iPlayer - TV Guide - Londo
http://www.bbc.co.uk/iplayer/live/bbcone | BBC iPlayer - Watch BBC
http://www.bbc.co.uk/iplayer/a-z | BBC iPlayer - A to Z - A | 4

15. Star Schema
page
device
page_hit
users
Fact Table
Dimension
Tables

16. Users table
users
user_id (PK) 12346
first_name Regen
last_name Schattschneid
date_of_birth 1963-11-22
region South East England
gender m
email_address regen.schattschneid@coelurosauria.com

17. Surname
import random
last_name = random.choice(all_last_names_list)

18. Distribute Users among Regions
region_distribution = [('NORTH EAST ENGLAND', 0.14),
('THE MIDLANDS', 0.13),
('NORTH WEST ENGLAND', 0.12),
('EAST ANGLIA', 0.11),
('SOUTH CENTRAL ENGLAND', 0.1),
('SOUTH EAST ENGLAND', 0.09),
('SCOTLAND', 0.08),
('LONDON', 0.08),
('SOUTH WEST ENGLAND', 0.08),
('WALES', 0.04),
('NORTHERN IRELAND', 0.03)]
my_region = get_random_key(region_distribution)

19. Find Random Key,
given List of Key, Probabilities
def get_random_key(list_of_distribution_tuples):
rnd = random.random()
cumulative_probability = 0.0
for (key, probability) in list_of_distribution_tuples:
cumulative_probability += probability
if rnd <= cumulative_probability:
return key
my_region = get_random_key(region_distribution)

20. Generate a User
def get_random_user():
gender = get_random_gender()
first_name = get_random_first_name(gender
last_name = random.choice(all_last_names_list)
email = get_random_email(first_name, last_name)
region = get_random_region()
return ({'gender': gender,
'first_name': first_name,
'last_name': last_name,
'email': email,
'region': region})

21. Star Schema
page
device
page_hit
users
Fact Table
Dimension
Tables

22. Fact Table
page_hit
page_id (FK) 214
user_id (FK) 12346
device_id (FK) 1
start_access_time 2016-09-20 00:48:00
access_duration_mins 57.91
PK

23. Hits per time of day

24. Normal Distribution
my_list = numpy.normal(mean, std_dev, samples)

25. numpy.normal(20, 0.5, 5000)
mean = 20 hours
std_dev = 0.5 hours
samples = 5000
see also random.normalvariate(mean, std_dev)

26. hits_per_hour_peak = [.35, .25, .1, .05, .04, .04,
.15, .2, .3, .33, .36, .38,
.4, .4, .4, .5, .6, .7,
.8, .9, .9, 1.0, .95, .55]

28. Page Hits per Hour, 2013-2015
0
1000
2000
3000
4000
5000
6000
7000
0…
6
12
18
0…
6
12
18
0…
6
12
18

29. Device, User, Page
• Use:
get_random_key_according_to_distribution()

30. Device Hits per Year
0%
20%
40%
60%
80%
100%
2013 2014 2015
Tablet
Mobile Device
Computer

31. Star Schema
page
device
page_hit
users
Fact Table
Dimension
Tables

32. • johnstinson99@gmail.com