25 листопада відбувся TechTalk на тему “Marketing Data Lake in Azure” від Олександра Федірка (CEE Head of BigData Practice at GlobalLogiс, тренер та ментор по BigData, драйвер внутрішньої BigData спільноти). Ця доповідь – про участь в проєкті, присвяченому маркетингу. Ми поговорили про сам маркетинг, розкрили технічні аспекти проєкту, інструменти та загальну архітектуру. Розказали про виклики, з якими стикалися, створюючи та удосконалюючи дата лейк в Azure. Також сфокусувались на наступних кроках та трішки – на частині, пов’язаній з data science. Цей техток – про перевірені підходи, досвід і рішення для подібних проєктів. Відео та деталі заходу: https://bit.ly/3J85NOP

1. 1
Confidential

2. 2
Confidential
Marketing Data Lake
in the Azure

3. 3
Confidential
Agenda
● Marketing things are boring, aren’t they ?
● Starting points
● Challenges on a project
● Next steps and evolution
● Data science things
● Conclusions

4. 4
Confidential
Marketing things are boring,
aren’t they ?

5. 5
Confidential
What is marketing about ?
● Research what we buy

6. 6
Confidential
What is marketing about ?
● Research what we buy
● Figure out purchase behavior

7. 7
Confidential
What is marketing about ?
● Research what we buy
● Figure out purchase behavior
● Target audience for AD better

8. 8
Confidential
What is marketing about ?
● Research what we buy
● Figure out purchase behavior
● Target audience for AD better
● Help adjust AD campaigns

9. 9
Confidential
How advertising business works
Sellers
Buyers
Ad Network Ad Network
Agency DSP Ad Exchange SSP Publisher
DMP/Data Supply
Brand Audience
RTB

10. 10
Confidential
How advertising business works
Sellers
Buyers
Ad Network Ad Network
Agency DSP Ad Exchange SSP Publisher
DMP/Data Supply
Brand Audience
RTB

11. 11
Confidential
Business scenario
Figure out how advertising (online and offline, impression) leads us to the
purchase

12. 12
Confidential
Business scenario
Figure out how advertising (online and offline, impression) leads us to the
purchase
I saw ad on my cell phone, then on my laptop, then on a printed coupons
and I bought promoted item.

13. 13
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA

14. 14
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA
Q: Do they want to steal my private data ?
A: No, they don’t want

15. 15
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA
Q: Do they want to steal my private data ?
A: No, they don’t want
Q: Do they have my bank accounts numbercredit cards numbers ?
A: No, they don’t have

16. 16
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA
Q: Do they want to steal my private data ?
A: No, they don’t want
Q: Do they have my bank accounts numbercredit cards numbers ?
A: No, they don’t have
Q: Do they buy information about me from other companies ?
A: Yes, they do

17. 17
Confidential
Starting points

18. 18
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup

19. 19
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari IaaS

20. 20
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari IaaS
- No unit-tests

21. 21
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari IaaS
- No unit-tests
- NiFi extensive use

22. 22
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari IaaS
- No unit-tests
- NiFi extensive use
- Scala based Spark jobs

23. 23
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari IaaS
- No unit-tests
- NiFi extensive use
- Scala based Spark jobs
- No CI/CD

24. 24
Confidential
Starting points
Sources by technology
- Oracle

25. 25
Confidential
Starting points
Sources by technology
- Oracle
- Netezza

26. 26
Confidential
Starting points
Sources by technology
- Oracle
- Netezza
- MongoDB

27. 27
Confidential
Starting points
Sources by technology
- Oracle
- Netezza
- MongoDB
- File on S3 bucket

28. 28
Confidential
Starting points
Sources by technology
- Oracle
- Netezza
- MongoDB
- File on S3 bucket
- Hive on 3rd party server

29. 29
Confidential
Starting points
Scheduling execution chain
Oozie

30. 30
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script

31. 31
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application

32. 32
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi

33. 33
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark job

34. 34
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark job
Sqoop job

35. 35
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark job
Sqoop job
distcp job

36. 36
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark job
Sqoop job
distcp job
Shell script

37. 37
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark job
Sqoop job
distcp job
Shell script
PostgreSQL

38. 38
Confidential
Starting points
Solutions map

39. Our data
platform

40. Our data
platform
Matched
impressions
Setup
campaign

41. Our data
platform
Store orders
and
impressions
Store orders
and
impressions
Matched
impressions
Old data
platform
AdExhange3
AdExhange1
Identities
AdExhange2
Kafka
Analytics
AdExhange4
MDM
Impressions
Identities
Setup
campaign

42. UI
DataLake
Refined Target Presentation Index
ing
Raw
Business scenario 1
Data
sources
Starting points
Ingestion
Business scenario 2
Orchestration
Transformation
Business scenario 3

43. 43
Confidential
Challenges on a project

44. 44
Confidential
Challenges on a project
No unit-tests

45. 45
Confidential
Challenges on a project
No unit-tests
Let’s make it!

46. 46
Confidential
Challenges on a project
No unit-tests
Let’s make it!
import com.holdenkarau.spark.testing.DataFrameSuiteBase
import org.apache.spark.SparkConf
import org.scalatest.{BeforeAndAfter, FunSuite, Matchers}

47. 47
Confidential
test("load id map table data") {
// given
val expectedData = List(
MyMap(id1 = "GUID1", id2 = "GUID2", id_type = "aaid").......
)
val expected = spark.createDataFrame(sc.parallelize(expectedData))
expected.write.mode(SaveMode.Append).format("hive").partitionBy("id_
type").saveAsTable(Db.MySchema.name + "." +
Db.MySchema.Table.MyTable)
Challenges on a project

48. 48
Confidential
// when
val actual = MainClass.functionToTest(spark, Db.MySchema.name,
Db.MySchema.Table.MyTable)
// then
val actualFieldsQueried = actual.schema.fields.map(f => f.name)
withClue("Actual fields queried:n" + actual.schema.treeString) {
actualFieldsQueried shouldEqual Array("id1", "id2", "id_type")
}
Challenges on a project

49. 49
Confidential
val actualData = actual.collect()
withClue(actualData.mkString("n", "n", "n")) {
actualData.length should equal(expectedData.size)
withClue("Actual id2 field differs from expected") {
actualData.map(r => r.getAs[String]("id2")) should contain
theSameElementsAs expected.map(id => id.id2)
}
}
}
Challenges on a project

50. 50
Confidential
Challenges on a project
Scheduling invocation chain is too long

51. 51
Confidential
Challenges on a project
Scheduling invocation chain is too long
- Get rid of NiFi

52. 52
Confidential
Challenges on a project
Scheduling invocation chain is too long
- Get rid of NiFi
- Get rid of shell scripts

53. 53
Confidential
Challenges on a project
Scheduling invocation chain is too long
- Get rid of NiFi
- Get rid of shell scripts
- Get rid of Oozie

54. 54
Confidential
Challenges on a project
One job runs for 4 hours and take all resources of the cluster

55. 55
Confidential
Challenges on a project
One job runs for 4 hours and take all resources of the cluster
Job has to analyze history for the last 52 weeks of orders history

56. 56
Confidential
Challenges on a project
One job runs for 4 hours and take all resources of the cluster
Job has to analyze history for the last 52 weeks of orders history
We can make it incremental!

57. 57
Confidential
Challenges on a project
No automation of rollout for the cluster

58. 58
Confidential
Challenges on a project
No automation of rollout for the cluster
- Time to setup new cluster is about 7-10 days

59. 59
Confidential
Challenges on a project
No automation of rollout for the cluster
- Time to setup new cluster is about 7-10 days
- Cloudbreak blueprints do not help too much

60. 60
Confidential
Challenges on a project
No CICD

61. 61
Confidential
Challenges on a project
No CICD
- Make at least build, unit-test and deployment of the jars automated

62. 62
Confidential
Challenges on a project
No CICD
- Make at least build, unit-test and deployment of the jars automated
- Partially covered CICD of the Oozie scripts

63. 63
Confidential
Challenges on a project
No CICD
- Make at least build, unit-test and deployment of the jars automated
- Partially covered CICD of the Oozie scripts
- CD for the shell scripts

64. 64
Confidential
Challenges on a project
Low level of security

65. 65
Confidential
Challenges on a project
Low level of security
- Every developer uses sa account to get to the edge node

66. 66
Confidential
Challenges on a project
Low level of security
- Every developer uses sa account to get to the edge node
- Single admin user for Ambari

67. 67
Confidential
Challenges on a project
Low level of security
- Every developer uses sa account to get to the edge node
- Single admin user for Ambari
- Developers has access to PROD

68. 68
Confidential
Challenges on a project
Stuck with Kafka 0.7
Customer has lock himself with this old version of Kafka
There only option to consume messages is to use old Java library

69. 69
Confidential
Next steps and evolution

70. 70
Confidential
Next steps and evolution

71. 71
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only

72. 72
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow

73. 73
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow
- Move to the Databricks

74. 74
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow
- Move to the Databricks
Strategic:
- Get rid of Kafka 0.7

75. 75
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow
- Move to the Databricks
Strategic:
- Get rid of Kafka 0.7
- Switch to the Next Generation platform

76. 76
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform

77. 77
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory

78. 78
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory
- Azure Databricks

79. 79
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory
- Azure Databricks
- Azure Data Lake

80. 80
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory
- Azure Databricks
- Azure Data Lake
- Azure EventHub

81. 81
Confidential
Data science things

82. 82
Confidential
Existing data science
Categorization
Audience segmentation. From very basics, e.g. sex, age group, car
ownership, houshold income group, luxury car ownership to advanced
categories like, “daily dairy consumers who buy their goods every Thursday
in the morning”
Fuzzy matching
Match of IDs from different providers
Maintain internal ID, so called householdID

83. 83
Confidential
Next gen of data science
Impression chain analysis
Markov chain, build chains like ad1->ad2->ad3->ad4->buy
or ad5->ad6->ad7->no buy
Country-wide approximation
We see buyers inside our network, but what’s going on outside of the
network? We can buy info from personal accountant services that parses
recipes from all over the country. Based on this information we can calculate
factor for different product categories.

84. 84
Confidential
Next gen of data science
Smart segments pick for campaign
Change the way how brands do setup of ad campaign. Instead of classic
filter based criteria AI can help decide on a segment based on: campaign
budget, campaign goal (increase penetration, increase revenue, more trials,
from trials to repeaters), info about products (which category, who are
competitors) etc
Campaign automatic on fly adjustment
Based on ongoing campaign and immediate feedback we can adjust
audience

85. 85
Confidential
Conclusions

86. 86
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends

87. 87
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS

88. 88
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last

89. 89
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last
- General programming practices for everyone, not just for Software
Engineers

90. 90
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last
- General programming practices for everyone, not just for Software
Engineers
- Automate everything that you can

91. 91
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last
- General programming practices for everyone, not just for Software
Engineers
- Automate everything that you can
- Documentation first

92. 92
Confidential
92
Q&A session

93. 93
Confidential
Thank you!