Webinar by Oleksandr Fedirko, CEE Head of BigData Practice at GlobalLogic Marketing things are boring, aren’t they? Starting points Challenges on a project Next steps and evolution Conclusions

1. 1
Confidential

2. 2
Confidential
Marketing Data Lake in the Cloud

3. 3
Confidential
Agenda
● Marketing things are boring, aren’t they ?
● Starting points
● Challenges on a project
● Next steps and evolution
● Conclusions

4. 4
Confidential
Marketing things are boring,
aren’t they ?

5. 5
Confidential
What is marketing about ?
● Research what we buy

6. 6
Confidential
What is marketing about ?
● Research what we buy
● Figure out purchase behavior

7. 7
Confidential
What is marketing about ?
● Research what we buy
● Figure out purchase behavior
● Target audience for AD better

8. 8
Confidential
What is marketing about ?
● Research what we buy
● Figure out purchase behavior
● Target audience for AD better
● Help adjust AD campaigns

9. 9
Confidential
How advertiser business works
SellersBuyers
Ad Network Ad Network
Agency DSP Ad Exchange SSP Publisher
DMP/Data Supply
Brand Audience
RTB

10. 10
Confidential
How advertiser business works
SellersBuyers
Ad Network Ad Network
Agency DSP Ad Exchange SSP Publisher
DMP/Data Supply
Brand Audience
RTB

11. 11
Confidential
Business scenario
Figure out how advertising (online and offline, impression) leads us to the
purchase

12. 12
Confidential
Business scenario
Figure out how advertising (online and offline, impression) leads us to the
purchase
I saw ad on my cell phone, then on my laptop, then on a printed coupons
and I bought promoted item.

13. 13
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA

14. 14
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA
Q: Do they want to steal my private data ?
A: No, they don’t want

15. 15
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA
Q: Do they want to steal my private data ?
A: No, they don’t want
Q: Do they have my bank accounts numbercredit cards numbers ?
A: No, they don’t have

16. 16
Confidential
Privacy concerns
Q: Do such companies spy on me?
A: In some way yes, but you agreed to share this info about yourself. Read
EULA
Q: Do they want to steal my private data ?
A: No, they don’t want
Q: Do they have my bank accounts numbercredit cards numbers ?
A: No, they don’t have
Q: Do they buy information about me from other companies ?
A: Yes, they do

17. 17
Confidential
Starting points

18. 18
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup

19. 19
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari managed services

20. 20
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari managed services
- No unit-tests

21. 21
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari managed services
- No unit-tests
- NiFi extensive use

22. 22
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari managed services
- No unit-tests
- NiFi extensive use
- Scala based Spark jobs

23. 23
Confidential
Starting points
BigData platform in Microsoft Azure IaaS
- Cloudbreak based setup
- HDP Ambari managed services
- No unit-tests
- NiFi extensive use
- Scala based Spark jobs
- No CI/CD

24. 24
Confidential
Starting points
Sources by technology
- Oracle

25. 25
Confidential
Starting points
Sources by technology
- Oracle
- Netezza

26. 26
Confidential
Starting points
Sources by technology
- Oracle
- Netezza
- MongoDB

27. 27
Confidential
Starting points
Sources by technology
- Oracle
- Netezza
- MongoDB
- File on S3 bucket

28. 28
Confidential
Starting points
Sources by technology
- Oracle
- Netezza
- MongoDB
- File on S3 bucket
- Hive on 3rd party server

29. 29
Confidential
Starting points
Scheduling execution chain
Oozie

30. 30
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script

31. 31
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application

32. 32
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi

33. 33
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark job

34. 34
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark jobSqoop job

35. 35
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark jobSqoop jobdistcp job

36. 36
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark jobSqoop jobdistcp job
Shell script

37. 37
Confidential
Starting points
Scheduling execution chain
Oozie
Shell script
Java application
NiFi
Spark jobSqoop jobdistcp job
Shell script
PostgreSQL

38. 38
Confidential
Starting points
Solutions map

39. Our data
platform

40. Our data
platform
Matched
impressions
Setup
campaign

41. Our data
platform
Store orders
and
impressions
Store orders
and
impressions
Matched
impressions
Old data
platform
AdExhange3
AdExhange1
Identities
AdExhange2
Kafka
Analytics
AdExhange4
MDM
Impressions
IdentitiesSetup
campaign

42. UI
DataLake
Refined Target Presentation Index
ing
Raw
Business scenario 1
Data
sources
Starting points
Ingestion
Business scenario 2
Orchestration
Transformation

43. 43
Confidential
Challenges on a project

44. 44
Confidential
Challenges on a project
No unit-tests

45. 45
Confidential
Challenges on a project
No unit-tests
Let’s make it!

46. 46
Confidential
Challenges on a project
No unit-tests
Let’s make it!
import com.holdenkarau.spark.testing.DataFrameSuiteBase
import org.apache.spark.SparkConf
import org.scalatest.{BeforeAndAfter, FunSuite, Matchers}

47. 47
Confidential
Challenges on a project
test("load id map table data") {
// given
val expectedData = List(
MyMap(id1 = "GUID1", id2 = "GUID2", id_type = "aaid").......
)
val expected = spark.createDataFrame(sc.parallelize(expectedData))
expected.write.mode(SaveMode.Append).format("hive").partitionBy("id_type").sav
eAsTable(Db.MySchema.name + "." + Db.MySchema.Table.MyTable)
// when
val actual = MainClass.functionToTest(spark, Db.MySchema.name,
Db.MySchema.Table.MyTable)

48. 48
Confidential
Challenges on a project
// then
val actualFieldsQueried = actual.schema.fields.map(f => f.name)
withClue("Actual fields queried:n" + actual.schema.treeString) {
actualFieldsQueried shouldEqual Array("id1", "id2", "id_type")
}
val actualData = actual.collect()
withClue(actualData.mkString("n", "n", "n")) {
actualData.length should equal(expectedData.size)
withClue("Actual id2 field differs from expected") {
actualData.map(r => r.getAs[String]("id2")) should contain
theSameElementsAs expected.map(id => id.id2)
}
}
}

49. 49
Confidential
Challenges on a project
Scheduling invocation chain is too long

50. 50
Confidential
Challenges on a project
Scheduling invocation chain is too long
- Get rid of NiFi

51. 51
Confidential
Challenges on a project
Scheduling invocation chain is too long
- Get rid of NiFi
- Get rid of shell scripts

52. 52
Confidential
Challenges on a project
Scheduling invocation chain is too long
- Get rid of NiFi
- Get rid of shell scripts
- Get rid of Oozie

53. 53
Confidential
Challenges on a project
One job runs for 4 hours and take all resources of the cluster

54. 54
Confidential
Challenges on a project
One job runs for 4 hours and take all resources of the cluster
Job has to analyze history for the last 52 weeks of orders history

55. 55
Confidential
Challenges on a project
One job runs for 4 hours and take all resources of the cluster
Job has to analyze history for the last 52 weeks of orders history
We can make it incremental!

56. 56
Confidential
Challenges on a project
No automation of rollout for the cluster

57. 57
Confidential
Challenges on a project
No automation of rollout for the cluster
- Time to setup new cluster is about 7-10 days

58. 58
Confidential
Challenges on a project
No automation of rollout for the cluster
- Time to setup new cluster is about 7-10 days
- Cloudbreak blueprints do not help too much

59. 59
Confidential
Challenges on a project
No CICD

60. 60
Confidential
Challenges on a project
No CICD
- Make at least build, unit-test and deployment of the jars automated

61. 61
Confidential
Challenges on a project
No CICD
- Make at least build, unit-test and deployment of the jars automated
- Partially covered CICD of the Oozie scripts

62. 62
Confidential
Challenges on a project
No CICD
- Make at least build, unit-test and deployment of the jars automated
- Partially covered CICD of the Oozie scripts
- CD for the shell scripts

63. 63
Confidential
Challenges on a project
Low level of security

64. 64
Confidential
Challenges on a project
Low level of security
- Every developer uses sa account to get to the edge node

65. 65
Confidential
Challenges on a project
Low level of security
- Every developer uses sa account to get to the edge node
- Single admin user for Ambari

66. 66
Confidential
Challenges on a project
Low level of security
- Every developer uses sa account to get to the edge node
- Single admin user for Ambari
- Developers has access to PROD

67. 67
Confidential
Challenges on a project
Stuck with Kafka 0.7
Customer has lock himself with this old version of Kafka
There only option to consume messages is to use old Java library

68. 68
Confidential
Next steps and evolution

69. 69
Confidential
Next steps and evolution

70. 70
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only

71. 71
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow

72. 72
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow
- Move to the Databricks

73. 73
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow
- Move to the Databricks
Strategic:
- Get rid of Kafka 0.7

74. 74
Confidential
Next steps and evolution
Tactical:
- NiFi for stream ingestion only
- Move to the Airflow
- Move to the Databricks
Strategic:
- Get rid of Kafka 0.7
- Switch to the Next Generation platform

75. 75
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform

76. 76
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory

77. 77
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory
- Azure Databricks

78. 78
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory
- Azure Databricks
- Azure Data Lake

79. 79
Confidential
Next steps and evolution
What is Next Generation platform
Fully managed by Azure PaaS based data platform
- Azure Data Factory
- Azure Databricks
- Azure Data Lake
- Azure EventHub

80. 80
Confidential
Conclusions

81. 81
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends

82. 82
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS

83. 83
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last

84. 84
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last
- General programming practices for everyone, not just for Software
Engineers

85. 85
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last
- General programming practices for everyone, not just for Software
Engineers
- Automate everything that you can

86. 86
Confidential
Conclusions
- DevOps, QA, L2 and InfoSec are your best friends
- There is a big demand from companies to move into PaaS
- Don’t leave security at last
- General programming practices for everyone, not just for Software
Engineers
- Automate everything that you can
- Documentation first

87. 87
Confidential
87
Q&A session

88. 88
Confidential