This document details the advancements in Walmart's search architecture and data processing using Spark Streaming and DataFrames, highlighting the exponential growth in product catalogs and the need for real-time analytics. The new architecture significantly improves user experience by facilitating real-time updates for price, inventory, and more, processing tens of thousands of events per second. Additionally, it discusses how Spark DataFrames simplify code for building scalable data pipelines, enhancing efficiency in data preparation and analysis.

1. LEARNINGS USING SPARK STREAMING & DATAFRAMES FOR
WALMART SEARCH
Yan Zheng Nirmal Sharma
Director, Search-Walmart Labs Principle Engineer, Search - Walmart Labs

2. Search growth in last couple of years
• Product Catalog is growing exponentially
• Product updates by merchants happen almost real time
• Price update happens almost real time
• Inventory update happens almost real time
• Data used for relevance signals increased 10 times
• Number of functionalities/business use cases to support increased a lot
• Need for real time analytics increased to analyze data quickly to make faster
business decisions

3. Product catalog growth…

4. Other data growth…
0
20
40
60
80
100
120
140
2014 2015 2016
Data(TB)

5. Old Search Architecture (simplified version )

6. • Issue with the old architecture was that the index
update was happeningonce a day……and that
was taking us back in terms of user experience
and business

7. New Search Architecture

8. How we did it
• Started capturing all the catalog updates in Kafka and then using spark streaming to
process these real time events and make them available to our indexes ( processing
upto 10,000 events per sec ).
• Micro services using spark streaming to directly update price, inventory etc. details to
indexes (close to 8000 events per sec ).
• Spark streaming and custom built elastic search data loader to load data directly to
ES for real time analytics ( processing 20,000-25,000 events per sec )
• Custom hadoop and spark jobs to process user data faster for all our data science
signals (50 TB - 100 TB data per batch ) and also to make data available faster for
analyst and business people.
• Started updated all our BI reports with couple of hours which used to take days ( or
sometimes days )

9. Further deep dive in technologies used……..
• Technology stack used for our data pipelines
– Spark – Streaming, Dataframes
– Hadoop – Hive, Map-Reduce
– Cassandra – for lookup ( fast read/write )
– Kafka – Event Processing
– Elastic Search – for logging and analytics
– Solr – Indexing walmart.com

10. How we used spark dataframe to build scalable and flexible
pipelines…..

11. • DATA PIPELINES:
1. Batch processing (Analytics, Data Science )
2. Real time processing ( Index update, micro services)

12. What is required to build data pipelines

13. • So the issue is that there no single uniform language to
build data pipeline
• No easier way to reuse the code or templating the code
for others to reuse for similar work

14. Dataframe has the potential to become the next unified language
for data engineering………

15. Here are some examples to explain….
• This is the current code for K-Means clustering using python, hive, java(for UDF )
• https://gecgithub01.walmart.com/LabsSearch/DOD-
BE/tree/master/src/main/scripts/query_categorization_daily
• The current python code is more than 1000 line which includes first data preparation,
then data transformation to calculate feature vector and then model training and
finally data post procession to validate and store data

16. This is the new code for the same K-Means clustering using Spark
Dataframes
• The new code is hardly contains 60-70 lines
• The whole code is just one single file
• https://gecgithub01.walmart.com/LabsSearch/polaris-data-
gen/blob/master/application/polaris-analytics/kmeans-query-
tier/src/main/scala/QueryClustering.scala
• And the additional advantage is that the whole code is parallelized and just take 1/5th
of the time taken by original code

17. Another example is our scalable Anomaly framework
• For any data pipeline data quality is the key
– All data is correlated in one way or another because almost all data feeds our search
– Upfront data checks at source is equally important as final data check at target
– Cause and effect analysis
– Easy to use
– Pluggable to all kind of data pipelines
– Easy to enhance
– Light weight and easy to install

18. This is how we can run ETL’s using templates and the process using Dataframes..
• {
"check_desc": "primary_Category_path_null_count",
"check_id" : 1,
"check_metric" : ["count"],
"backfill": [0],
"fact_info" : [{
"table_name" : "item_attribute_table",
"table_type" : [],
"key_names" : [],
"check_metric_field": "",
"database_name" : "polaris",
"filters" : ["primary_category_path is null"],
"run_datetime" : "LATEST_DATE_MINUS_10",
"datetime_partition_field" : "data_date",
"datetime_partition_index" : 0,
"other_partition_fields" : ""
}]

19. Code snippet…
• val aggsResult = metric match {
case s: Vector[String] if(s.isEmpty ) => resultSet //.map(row => (check_id.toString, row._1, row._2)) //
toDF("check_id","key", "aggValue")
case _ =>
val caseResult = {
metric(0) match {
case "average" => resultSet.mapValues(_.toDouble)
.aggregateByKey(init)((a, b) => (a._1 + b, a._2 + 1), (x, y) => (x._1 + y._1, x._2 + y._2))
.mapValues(x => x._1 / x._2)
case "sum" => resultSet.mapValues(_.toDouble)
.reduceByKey((a, b) => a + b)
case "count" | "dups" => resultSet
.aggregateByKey(0.0)((a, b) => a + 1.0, (a, b) => a + b) //this is for count
case "max" => resultSet.mapValues(_.toDouble)
.reduceByKey((a, b) => max(a.toDouble, b.toDouble))
case "min" => resultSet.mapValues(_.toDouble)
.reduceByKey((a, b) => min(a.toDouble, b.toDouble))
case "countdistinct" => resultSet
.aggregateByKey(0.0)((a, b) => a + 1.0, (a, b) => a + b)
}
}
caseResult.mapValues(_.toString) //.map(row => (check_id.toString, row._1, row._2) ) //.toDF("check_id","key",
"aggValue")
}

20. Reusable templates…….
• "alert_files": [
{
"check_desc": "Index file size for gm should be in range",
"check_id": 12,
"check_metric": ["size"],
"backfill": [0],
"file_info": [
{
"file_type": "hadoop",
"file_path" : "/tmp/polaris/nsharm9/alertN ame=misc_outliers/LATEST_PAR T_DA TE_MIN US_10"
}
]
}
],
"alert_cmds": [
{
"check_desc": "file counts",
"check_id": 14,
"backfill": [0],
"cmd_info": [
{
"cmd": "ls -ls | wc -l"
}
]
}
],
"alert_sqls": [
{
"check_desc": "some counts",
"check_id": 15,
"backfill": [0],
"query_info": [
{
"query": "select count(*) as check_agg from users where partition_epoch= LATEST_PART_EPOC H "
}
]
}
]

21. Code snippet….
• /*File processing*/
• val output =
obj.file_info.map(x => (x.file_type, inMetric) match {
case ("hadoop","size") =>
SearchFileSystem.getHDFSFileDetails(SearchFileSystem.evalFilePartition(x.file_path,fillDays)).getOrElse("length
", 0.toDouble)
case ("hadoop","filecounts") =>
SearchFileSystem.getHDFSFileDetails(SearchFileSystem.evalFilePartition(x.file_path,fillDays)).getOrElse("fileCo
unt", 0.toDouble)
case ("hadoop","dircounts") =>
SearchFileSystem.getHDFSFileDetails(SearchFileSystem.evalFilePartition(x.file_path,fillDays)).getOrElse("direct
oryCount", 0.toDouble)
case ("local","size") =>
SearchFileSystem.getLocalFileDetails(SearchFileSystem.evalFilePartition(x.file_path,fillDays)).getOrElse("directo
ryCount", 0.toDouble)
case ("local","filecounts") =>
SearchFileSystem.getLocalFileDetails(SearchFileSystem.evalFilePartition(x.file_path,fillDays)).getOrElse("directo
ryCount", 0.toDouble)
case ("local","dircounts") =>
SearchFileSystem.getLocalFileDetails(SearchFileSystem.evalFilePartition(x.file_path,fillDays)).getOrElse("directo
ryCount", 0.toDouble)
})map(x => ETLOutput(inCheckId, inCheckId, x.toString) )

22. Thank You.
Contact information or call to action goes here.