Data Con LA 2020 Description Join this session to learn how to build a modern cloud-scale data compute platform with code in just minutes! Using the industry's first IDE for building data applications, developers can now create data marts and data applications, while working interactively with large datasets. We will explore how easy it is to develop, test and operationalize powerful data compute applications over streaming data using SQL and Python and eager execution in Xcalar with the combination of declarative and visual imperative programming and eager execution You will see how you can reduce time to market for analyzing large volumes of data and building enterprise-level complex data compute applications. You will learn how to increase your developer productivity with SQL and Python, and put your complex business logic and ML models into production pipelines with the fastest time to value in industry. Speaker Nikita Ogievetsky, Xcalar, VP Product Engineering

1. Complex Data Transformations Made Easy
Building Data Apps for Large Scale transformations
Nikita Ogievetsky
Head Of Solutions Engineering
October 25 2020
LA Data Conference

2. Xcalar Application Platform
Xcalar Value Proposition
• 10x cost savings
• 20x improvement in time-to-value
• 100s of nodes, 1000s of users
• Petabyte scale
• Open standards: SQL + Python
• No vendor lock-in
• Billions of parallel ML ops
• Real-time data
Enterprise Grade Features
• Full memory hierarchy, in-memory
performance
• Crash consistent & recoverable
• Snapshots, checkpoints
• ACID transactions
• Point-in-Time rollbacks
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.2
“Accelerate development & operationalization of cloud-scale data applications with
unprecedented simplicity, speed and scale.”
Vikram Joshi - Founder’s vision

3. • What: Modern cloud-scale data compute platform
• Declarative + imperative programming paradigm, open standards, no vendor lock-in
• Industry’s first IDE for data apps
• CI/CD: Fastest Dev -> Test -> Operationalize cycle in industry
• Who: Goal oriented data engineers
• Good at SQL, some knowledge of Python or other scripting languages
• Working with large volumes of data inside or across large scale data systems
• Why:
• Reduce time to market for analyzing large volumes of data and building enterprise-
level complex data compute applications
• Minimize dependency on highly skilled backend system engineers
• Optimize performance SLA in prod and reduce infrastructure and software costs.
Xcalar Application Platform – Who/What/Why
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.3

4. Xcalar Architecture
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.4
Source Tables User-Generated
Data Marts
Connectors
& Targets
Editors/IDEs
Jupyter
Notebook
VS Code Zeplin PyCharm Xcalar Design
BI Tools
DWs
Oracle
Teradata
Spark Connector
Datasets for ML
Data-Driven
Data Lake
DWs
Oracle
Teradata
Spark Connector
Datasets for ML
Data Lake
Business Logic

5. • Xcalar has created the industry’s first IDE for data apps
• 5x dev productivity
• Iterative, imperative programming with eager execution
• Visual analysis, monitoring and debug for every operation
• High confidence production results as you develop
• Work with your entire datasets (Terabytes)
• SQL + Python programming paradigm (Also, Scala, Java, JavaScript, C++)
• Focus on Business Logic
• CI/CD
• Fast Dev -> Test -> Operationalize cycles
• Zero Changes for Production Deployment
Xcalar IDE - Better, Stronger, Faster
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.5

6. Case Study
Accelerating Complex, Real-time Insights at a
Top Global Investment Bank
6 Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.

7. Preexisting Technology Challenges with Spark
Problem Statement
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.7
• Very Slow
• Daily batch run: 4.5 hours (2 TB of data)
• Long time to value, long development and troubleshooting cycles
• Last revision: 6 person month
• Dependency on scarce expert programmers
• 25,000 lines of Java/Spark code
• Lack of dev IDE for distributed applications
• High TCO

8. Transparent Modular Xcalar IDE
25,000 lines of Java code -> 12 visual dataflows
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.8
Data Lake
Xcalar Application Platform
Kafka Streaming
Kafka Broker 1
TopicX ds1
Kafka Broker 2
TopicX ds1
Kafka Streaming
Kafka Broker 1
Topic 1
ds1
ds2
Topic 2
ds3
ds4
Kafka Broker 2
Topic 3
ds5
ds6
Topic 4
ds7
ds8

9. Xcalar Solution replaced Spark
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.9
Develop Business Logic
Troubleshoot & Maintain
Operationalize
Develop Distributed Compute
Develop Business Logic
Troubleshoot & Maintain
Develop Distributed Compute
20x improvement in time-to-value
10x TCO reduction & performance boost
5x Developer efficiency: Intuitive IDE, open standards, no niche skills
All code scales naturally, no need to use SDK/API for distributed compute
Zero changes for product deployment and operationalization
Easy troubleshooting, immediate feedback, simple administration
Better, Stronger, Faster

10. Case Study
Analyzing Xcalar with Xcalar
(at a top Wall Street bank)
10 Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.

11. Streaming Applications Platform
Complexity that needs to be managed
• Initial sources
• Kafka streams (topics, partitions, offsets, etc.)
• Applications configurations and metadata
• App regions, sources and dependencies
• Batches, batch versions
• Batch stages & microservices
• load, ingest, imd, execute, export, snapshot, recovery
• Table lifecycles (microbatch row counts, sizes, versions)
• Transactions and their metadata
• Errors and warning if any
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.11

12. Use Xcalar SaaS to analyze Xcalar logs
Powered by my.xcalar.cloud
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.12
1. Register and Login
2. Start my instance
3. Click on Xcalar Load or
start working on your
Notebook projects

13. Xcalar Load
Load log files as is, leverage power of SQL to parse, analyze and compute
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.13
1a. Navigate to my S3 bucket
1b. Upload my log file
and select it
1c. Select single column CSV
as input format
2. Review schema
3. Load table and
navigate to the notebook project

14. Parserless Parsing
I used SQL and Scalar Function to parse row-by-row (all CPUs work in parallel)
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.14
Load
1-column
temp table
Parse each
row in
parallel
Extract
syntax
Extract
semantics
Share
temp table
No predefined log parsers
Interactive development
Parallel parsing – all CPUs

15. Data Application Development Paradigm
I used Declarative + Imperative approach to build a Data Mart
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.15
Business
Logic
Transform
Build App
Data Mart
BI &
Visualizations
SQL
Scalar
Functions
Analyze
Profile
Test
SQL
Table
Functions
Modules
Data
Model
Profile
SQL
Declarative
Programming
Imperative
Programming
Load & Parse

16. Data Application
End-to-end
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.16
1. Log Parsing
2. Data Prep
& Enrich
3. Compute
& Normalize
Data
Mart
4. Visualization
Data Cube

17. Table Function Example
get_incremetns() Table Function
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.17
Explode list of
table-infos into
rows
Extract size,
row counts and
other features
Keep only info
about user
tables
Input
Interface
Ouput
Interface
For log entries containing arrays of
JSON dictionaries describing
streaming table increments:
• Explode each dictionary into a row
• Extract features
3. Compute
& Normalize

18. Operationalized Data Application
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.18
Schedule DW updates as a Microservice

19. BI Integration
Tableau, Qlik, Looker, … Jupyter
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.19
Xcalar Data Mart
SQL & BI Processing
Jupyter Notebook
Visualization
ODBC
REST

20. Review
20 Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.

21. Xcalar Programming Model
Xcalar Inc. Proprietary and Confidential. © 2020, Xcalar Inc. All rights reserved.21
• SQL SELECT statements + Python code to develop apps
• Lowers technical barrier to entry for developers
• Enables SQL + Python developers to build distributed apps
• No need to learn scale-out programming or machine architecture
• Xcalar Data Apps naturally scale
• No iterators to worry about
• Users write code to process row/column/matrix only
• Xcalar’s runtime does underlying distribution and scheduling

22. Cloud-Scale Platform for Data Apps
Learn More
xcalar.com
Free Developer Sign Up
my.xcalar.cloud