Simeon Simeonov discusses the concept of goal-based data production within the context of a data-powered culture, emphasizing the need for agile data production methods utilizing machine learning and AI. He highlights the complexity and inflexibility of traditional SQL abstractions and the advantages of a generative approach that enhances metadata queries and allows for real-time context resolution. The document advocates for a self-service model for data scientists and business users, aiming to improve productivity, flexibility, and collaboration through open-source solutions.

1. Sim Simeonov, Founder/CTO, Swoop
Goal-Based Data Production
@simeons #EUent1

2. Hi, I’m @simeons.
I like software magic.

3. http://bit.ly/spark-records

4. a data-powered culture supported by
petabyte-scale data engineering & ML/AI
@simeons #EUent1 4

5. data-poweredculture
AHA ML AI
Ad-Hoc
Analytics

6. data-poweredculture
AHA ML AI
new data
new queries
biz users
data scientists

7. 82%

8. agile data production
data-poweredculture
AHA ML AI
new data
new queries
biz users
data scientists

9. 9@simeons #EUent1
Data-Powered
Culture
Spark
Goal-Based Data Production
SparkDBMS
BI Tools
Agile Data Production
SQL/DSL

10. agile data production request
top 10 campaigns by gross revenue
running on health sites,
weekly for the past 2 complete weeks,
with cost-per-click and
the click-through rate
@simeons #EUent1 10

11. @simeons #EUent1 11

12. with
origins as (select origin_id, site_vertical from dimension_origins where site_vertical = 'health'),
t1 as (
select campaign_id, origin_id, clicks, views, billing,
date(to_utc_timestamp(date_sub(to_utc_timestamp(from_unixtime(uts), 'America/New_York'),
cast(date_format(to_utc_timestamp(from_unixtime(uts), 'America/New_York'), 'u') as int) - 1), 'America/New_York')) as ts
from rep_campaigns_daily
where from_unixtime(uts) >= to_utc_timestamp(date_sub(to_utc_timestamp(current_date(), 'America/New_York'),
cast(date_format(to_utc_timestamp(current_date(), 'America/New_York'), 'u') as int) - 1) - INTERVAL 2 WEEK,
'America/New_York')
and from_unixtime(uts) < to_utc_timestamp(date_sub(to_utc_timestamp(current_date(), 'America/New_York'),
cast(date_format(to_utc_timestamp(current_date(), 'America/New_York'), 'u') as int) - 1), 'America/New_York')
),
t2 as (
select ts, campaign_id, sum(views) as views, sum(clicks) as clicks, sum(billing) / 1000000.0 as gross_revenue
from t1 lhs join origins rhs on lhs.origin_id = rhs.origin_id
group by campaign_id, ts
),
t3 as (select *, rank() over (partition by ts order by gross_revenue desc) as rank from t2),
t4 as (select * from t3 where rank <= 10)
select
ts, rank, campaign_short_name as campaign,
bround(gross_revenue) as gross_revenue,
format_number(if(clicks = 0, 0, gross_revenue / clicks), 3) as cpc,
format_number(if(views = 0, 0, 100 * clicks / views), 3) as pct_ctr
from t4 lhs join dimension_campaigns rhs on lhs.campaign_id = rhs.campaign_id
order by ts, rank
@simeons #EUent1 12

13. Spark DSL is equally complex
spark.table("rep_campaigns_daily")
.where("""
from_unixtime(uts) >= to_utc_timestamp(date_sub(to_utc_timestamp(current_date(), 'America/New_York'),
cast(date_format(to_utc_timestamp(current_date(), 'America/New_York'), 'u') as int) - 1) - INTERVAL 2 WEEK, 'America/New_York')
and from_unixtime(uts) < to_utc_timestamp(date_sub(to_utc_timestamp(current_date(), 'America/New_York'),
cast(date_format(to_utc_timestamp(current_date(), 'America/New_York'), 'u') as int) - 1), 'America/New_York')""")
.join(spark.table("dimension_origins").where('site_vertical === "health").select('origin_id), "origin_id")
.withColumn("ts", expr("date(to_utc_timestamp(date_sub(to_utc_timestamp(from_unixtime(uts), 'America/New_York'),
cast(date_format(to_utc_timestamp(from_unixtime(uts), 'America/New_York'), 'u') as int) - 1), 'America/New_York'))"))
.groupBy('campaign_id, 'ts)
.agg(
sum('views).as("views"),
sum('clicks).as("clicks"),
(sum('billing) / 1000000).as("gross_revenue")
)
.withColumn("rank", rank.over(Window.partitionBy('ts).orderBy('gross_revenue.desc)))
.where('rank <= 10)
.join(spark.table("dimension_campaigns").select('campaign_id, 'campaign_short_name.as("campaign")), "campaign_id")
.withColumn("gross_revenue", expr("bround(gross_revenue)"))
.withColumn("cpc", format_number(when('clicks === 0, 0).otherwise('gross_revenue / 'clicks), 3))
.withColumn("pct_ctr", format_number(when('views === 0, 0).otherwise(lit(100) * 'clicks / 'views), 3))
.select('ts, 'rank, 'campaign, 'gross_revenue, 'cpc, 'pct_ctr)
.orderBy('ts, 'rank)
@simeons #EUent1 13

14. Humans Spark

15. 40+ year old abstraction
• powerful… but
• complex
• fragile
• inflexible
agile data production
SQL

16. traditional abstractions fail
BI tools
slow to add new
schema/data
SQL views
build only upon
existing schema
UDFs
very limited

17. Humans Spark
what I want
(implicit context)
what you want
how to produce it
(explicit context)

18. the curse of generality: verbosity
what (5%) vs. how (95%)
@simeons #EUent1 18

19. the curse of generality:
repetition
-- calculate click-through rate, %
format_number(if(views = 0, 0, 100 * clicks / views), 3) as pct_ctr
-- join to get campaign names from campaign IDs
SELECT campaign_short_name AS campaign, ...
FROM ... lhs
JOIN dimension_campaigns rhs
ON lhs.campaign_id = rhs.campaign_id
@simeons #EUent1 19

20. the curse of generality: inflexibility
-- code depends on time column datatype, format & timezone
date(to_utc_timestamp(
date_sub(
to_utc_timestamp(from_unixtime(uts), 'America/New_York'),
cast(
date_format(
to_utc_timestamp(from_unixtime(uts), 'America/New_York'),
'u')
as int) - 1),
'America/New_York'))
@simeons #EUent1 20

21. over time, generality creates
massive technical debt

22. Humans Spark
what I want
data
just-in-time
generative
context

23. goal-based data production
DataProductionRequest()
.select('campaign.top(10).by('gross_revenue), 'cpc, 'ctr)
.where("health sites")
.weekly.time("past 2 complete weeks")
.humanize
@simeons #EUent1 23

24. the DSL is not as important as the
processing & configuration models
@simeons #EUent1 24

25. Data Production
Request
Data Production
Goal
Data Production
Rule
@simeons #EUent1 25

26. Request Goal
Rule
dpr.addGoal(
ResultColumn("cpc")
)
@simeons #EUent1 26

27. Request Goal
Rule
// .time("past 2 complete weeks")
dpr.addProductionRule(
TimeFilter(CompleteWeeks(ref = Now, beg = -2)),
at = SourceFiltering // production stage
)
@simeons #EUent1 27

28. Request Goal
Rule
// .time("past 2 complete weeks")
dpr.addProductionRule(
TimeFilter(CompleteWeeks(ref = Now, beg = -2)),
at = SourceFiltering // production stage
)
Requirement
SourceColumn(
role = TimeDimension()
)
@simeons #EUent1 28

29. Request Goal
Rule
// .time("past 2 complete weeks")
dpr.addProductionRule(
TimeFilter(CompleteWeeks(ref = Now, beg = -2)),
at = SourceFiltering // production stage
)
Requirement
Prop("time.week.start")
Prop("time.tz.source")
Prop("time.tz.reporting")
@simeons #EUent1 29

30. Rule
Requirement Context
Resolution
satisfying
partial resolution rewrites
@simeons #EUent1 30

31. Rule
Requirement Context
Resolution Dataset
satisfying
partial resolution rewrites
full resolution builds
transformation DAG
@simeons #EUent1 31

32. modern approach to metadata
• Minimal, just-in-time context
• Generative rules
@simeons #EUent1 32

33. context is a metadata query
smartDataWarehouse.context
.release("3.1.2")
.merge(_.tagged("sim.ml.experiments"))
@simeons #EUent1 33
mix production context with personal context

34. automatic joins by column name
{
"table": "dimension_campaigns",
"primary_key": ["campaign_id"]
"joins": [ {"fields": ["campaign_id"]} ]
}
@simeons #EUent1 34
join any table with a campaign_id column

35. automatic semantic joins
{
"table": "dimension_campaigns",
"primary_key": ["campaign_id"]
"joins": [ {"fields": ["ref:swoop.campaign.id"]} ]
}
join any table with a Swoop campaign ID
@simeons #EUent1 35

36. calculated columns
{
"field": "ctr",
"description": "click-through rate",
"expr": "if(nvl(views,0)=0,0,nvl(clicks,0)/nvl(views,0))"
...
}
automatically available to matching tables
@simeons #EUent1 36

37. humanization
{
"field": "ctr",
"humanize": {
"expr": "format_number(100 * value, 3) as pct_ctr"
}
}
change column name as units change
@simeons #EUent1 37

38. optimization hints
{
"field": "campaign",
"unique": true
...
}
allows join after groupBy as opposed to before
@simeons #EUent1 38

39. automatic source & join selection
• 14+ Swoop tables can satisfy the request
– 100+x faster if one of two good choices are picked
• Execution statistics => smarter heuristics
– Natural partner to Databricks Delta
@simeons #EUent1 39

40. in case of insufficient context,
add more in real-time

41. 10x easier agile data production
val df = DataProductionRequest()
.select('campaign.top(10).by('gross_revenue), 'cpc, 'ctr)
.where("health sites")
.weekly.time("past 2 complete weeks")
.humanize
.toDF // result is a normal dataframe/dataset
@simeons #EUent1 41

42. revolutionary benefits
• Self-service for data scientists & many biz users
• Increased productivity & flexibility
• Improved performance & lower costs
• Easy collaboration (even across companies!)
@simeons #EUent1 42

43. At Swoop, we are committed to making
goal-based data production
the best free & open-source
interface for Spark agile data production
Interested in contributing?
Email spark@swoop.com
@simeons #EUent1 43

44. Join our open-source work.
Email spark@swoop.com
More Spark magic: http://bit.ly/spark-records