QBIC is an experimental project that aims to speed up queries to databases using machine learning. It uses techniques like predicting data dependencies and pre-generating optimized query results ("pregens") to return data without needing to send the full query to the database. QBIC represents queries and their relationships as a lattice structure and applies algorithms like knapsack problems to determine the optimal pregens to maintain based on factors like space usage and query popularity. It also aims to predict metrics and cardinalities for queries to further improve performance through techniques like "sharpening" that can return approximated results without querying the database.

1. QBIC
from Labs with ❤ by Misha Kozik

2. QBIC
• Cubic a.k.a Query Boost Intelligent Compiler
• Experimental project in Labs
• Recall Math
• Find Data Dependencies
• Use Machine Learning
• ...to speed up queries to db

3. Rationale
• Most BI-tools has nothing related with letter I
• UI -> buttons -> query -> response -> chart
• Query could run forever
• There are ways to speed up (partitions, indexes,
cache, cubes)
• ...
• Is there a way to get data without sending query?

4. Computers are
smarter than we
are think

5. A bit of
Math

6. Set
• Not related to java.util.Set
• Set of elements, for example {apple, tuesday,
42}
• No duplicates, {1, 1, 2} the same as {1, 2}
• No order, {1, 2} the same as {2, 1}
• Set could be element of another set {1, 2, {3,
4}}

7. Set of all subsets
• Powerset
• A is subset of B ( )
• Every element from A in B
• {1, 2} subset of {1, 2, 3}
• Powerset for {1, 2, 3}:

8. Ordered set
• Set, with defined order relation ( )
• is a binary operation (a b)
• Set {1, 9, 42} is ordered, with order relation
"less"
• Set {
!
,
"
,
#
} is ordered, with order "more
funny emoji"
• holds for every pair of elements!

10. Partially Ordered Set
• Something ordered, something not
• Everything what could be ordered, ordered.
• Some pair of elements could be incomparable
• {1, 4, 7, Tuesday, 42}

11. Lattice
• Partially ordered set, for every pair of
elements there is exact upper bound (supremum)
and exact lower bound (infimum)
• So, even elements A and B are not comparable,
there is an element C, which "greater" or
"less" than both A and B
• Hard?

12. Values of Senior Engineer
(sarcasm)
• House, car
• Build all possible combinations of those values
(powerset)
{{no house, no car},
{house, no car},
{no house, car},
{house, car}}

17. Every bottom-up
path in a lattice
is an ordered set

19. How that's related
to Databases?

20. Typical Query
select city, gender, count(*) -- groups and metrics
from table -- table
where income > 1000 and state = 'P' -- filters
group by city, gender -- groups again
order by count(*) desc -- sortings
limit 10 offset 0 -- limits
having count(*) > 5 -- more filters

21. Typical Query: simplify 1
select city, gender, count(*)
from table
group by city, gender
1
Leave attribute groups, only count(*) from metrics, no filters, no sorting, no limits. Later
we will try to add them back.

22. Build lattice by groups
• Take all attribute columns from table
city, gender, state
• Build all possible group combinations, powerset
{city, gender, state}
{city, gender}, {city, state}, {gender, state}
{city}, {gender}, {state}
{}

24. Pregen
optimization

25. Pregen
• Pregen is completed query from database with data
saved somewhere (aka cache or view)
• Pros:
(1) very fast getting data from pregen
(2) fast getting data from all pregens below
• Cons:
(1) need time for caculating pregen (and
supporting)
(2) need disk space for saving pregen

29. Light formalization
• There is a lattice
• If no pregens exist, user query time equals to
database query time ( ), no space for
pregens needed ( )
• If pregen exist, user query time will be very fast
( ), but pregen will occupy some space ( )
and database query time needed once to prepare a
pregen ( )
Note: you can calculate all pregens below

30. Pregen optimization flow
• Scan table
Read metadata
Detect what columns can be groups
• Build lattice by groups
• Fill every lattice node with props
- pregen size (rows or MB)
- time to retrieve data from db
• Apply optimization algorithm

32. What algorithm for
optimization
should we apply?

33. Knapsack Problem
• 0-1 Knapsack Problem
• Detect what items take into
knapsack, to maximize its
value and do not exceed
maximum available weight
• NP-hard
• Exact and Approximate
solutions APD1
APD1
Appendix 1 - Review of algorithms for Knapsack Problem

35. (Practical) Knaspsack #1
• Dynamic programming with rounded values
• Power10 (log) scale
• Examples:
0.2, could be rounded to 1 (10^0)
1.2, could be rounded to 1 (10^0)
8.6, could be rounded to 10 (10^1)
123.6, could be rounded to 100 (10^2)
..and so on

36. (Practical) Knaspsack #2
• Genetic with Greedy init population
• Genetic behaves not worse than Greedy
• You can control the running time of algorithm
• Tradeoff (time/accuracy)
• Open for modifications

37. Pregen
Optimization
Problem
• Detect what pregens should be
calculated in a lattice, to
maximize lattice value and do
not exceed maximum available
weight

38. Lattice: Weight
• Space occupied by pregen (rows, better
megabytes)
• Time to prepare pregen (time to query db )
• Time to support pregen (depends on update
frequency)

39. Lattice: Value
• If pregen is taken, the value 1, otherwise 0
• How many other pregens could be calculated
• Query popularity

40. Pregen Optimization
• At this point we can get a list of pregens
which are most optimal to maintain
• If new properties appear in the system
(popularity, data updates) we can recalculate
new optimal lattice
• Future: filters

41. Problem
• For large tables it's very expensive process to
label lattice nodes with actual values
• Basically, we need to calculate all possible
group bys
• Since its only optimization we don't need exact
values
• ...
• What if we can predict them?

42. Predict Lattice
Values: No Input
• Input: nothing
• We can't predict anything if
we have no input

43. Predict Lattice
Values: count(*)
• Input: select count(*)
• Again, we can't predict count
of groups

44. Predict Lattice
Values: Single
group by
• Input: all single group by
• Fill the first level of
lattice

45. Lattice Puzzle
• select city, count(*) -- 50
• select gender, count(*) -- 2
• ...
• How many rows will return
query?
(magic: without sending the
query)
select city, gender, count(*)

46. Lattice Puzzle: Linear
Scaling
• select city, count(*) -- 50
• select gender, count(*) -- 2
• select city, gender, count(*) -- ???
• Cartesian product (min:50, max:50*2)
• [50..100]

48. Lattice Puzzle: Linear
Scaling
• From sample you can get group counts
CITY(30), GENDER(2), CITY/GENDER(56)
• Build Interval as
[max(city,gender)..max(city,gender)*min(city,gender)]
Interval = [30..60], L=30, R=60, V=56
• Calculate
• Apply ratio to real data interval [50..100] to get estimate

49. Lattice Puzzle: Linear
Scaling
• Ok for small cardinality usecases
• Bad for high cardinality use case, because it get
diverged very soon
• Bad for small samples
• Bad for non-uniformly distributed samples
• Bad for non-linear functions
• Bad for other metrics

50. Curve Fitting
Approach

52. Linear Regression
•
• -> min
• One more optimization problemm
• Guess the coefficients a and b
• Apply function to find estimate

53. Actually,
not a good fit
* Anscombe Quartet

55. Anscombe Quartet
• Four different datasets
• All their statistical properties are equal
(mean, variance, correlation, etc.)
• Linear regression line is the same

56. you can always trick algorithm
if you know how it behaves
Be optimistic

57. Curve Fitting
• FitFunction.java
• If linear regression is bad, try other
functions
• LINEAR:
• POLY2:
• POWER:
• LOG:

58. Curve Fitting Flow
• Break sample into input points (x,y)
SAMPLE_SIZE, GROUP_COUNT
• Use the last point as evaluation of your model
• Apply different curves
• Model which is more closely to the evaluation
point will be your model
• Use this model to predict the lattice values

59. Cardinality Levels
• SMALL (0..10)
Gender, Income Group, Status
• MEDIUM (10..5000)
City, County, Category
• HIGH (5000..)
UserID, SKU

60. Cardinality Levels
• SMALL x SMALL (log works well)
• SMALL x MEDIUM (log works well)
• MEDIUM x MEDIUM (log works well)
• Anything x HIGH (hard to predict)
• ...
• Maybe drop high-cardinality use cases?

61. Predicted Lattice

63. Hypothesis: If we
can predict number
of rows, we can
predict data

64. Predict Data
GroupBy & Count(*)

68. Metric: COUNT(*)
• Additive
• Easily can be predicted by sample because looks
like LINEAR
• Could be improved by better sampling techniques
• Could be improved by microqueries
• ...
• What about other metrics?

69. Metric: SUM
• Additive
• Linear as well
• Can be predicted the same way as count(*)
by using linear scale
• Accuracy can be improved by using numeric field
ranges

70. Metric: AVG
• Not additive
• But if we assume avg=sum/count (not 100% true)
could be easily predicted as well by linear
scale

71. Metric: MIN/MAX
• Additive
• But not linear
• Do not apply linear scale
• Min/max from sample is not a worst idea, either
• If first level lattice calculated from db,
second could be predicted by few tricks

72. Produce TOP5 CITY,CATEGORY by MAX(TOTAL) without sending the query?

73. At least 50% can be
calculated

74. Rest data could be filled
through
• Extra query
• Additional carried data with the lattice
Along with max value, save the row
• Even without them, you have the structure and values
so, you can start drawing the results, while query
is running
• Another form of sharpening

75. Metric: DISTINCT_COUNT
• Not additive
• Not linear
• We can get a raw estimate by using min/max
scale
or by applying curve fitting on sampling
similar to what we did in predicting the
lattice

76. For other metrics,
or custom
calculations
we still can find
workarounds

77. If we can predict the data
we can use it in
Sharpening™

78. (Better) Sharpening
• Instant fast
• No queries to DB
• No requirement on time fields only
• No requirement on indexed/partitions support
• Still can be combined with microquerying!

79. Well, if you can return data
without sending the queries...

80. Offline Mode
• Explore the data without generating DB load
• Find correct questions before asking DB
• No need for the separate "RUN" button
• No need for cancellation support if you sent
wrong query
• You can (and should) always fallback to
database

81. Data Dependencies

82. Data Dependencies
• Data is not just rows and columns
• There are hidden connections between cells,
rows, columns
• We can identify them to make software easier
• They are not 100% accurate, since there are
always excepetions, but with clever approach...

83. DD1: High Cardinality
• High cardinality fields has a huge amount of distinct
values
• Define what high means for your application
• If distinct count on sample equal to sample size,
it's probably high cardinlaity field (some sort of id)
• High cardinality fields has lower importance in group
bys and defaults than regular fields, also you can
drop them from pregen and lattice since they hold too
much data.

84. DD2: One-to-one fields
• One-to-one fields are pairs, with related data
• STORE_ID and STORE_NAME
• Having one field, you can infer other and vice verca
• If two fields have the same distinct count on a sample,
they are probably one-to-one fields
• NAME has more importance than its ID brother for UI
• Also you can drop the ID part from lattice and maintain
only NAME with mapping.

85. DD3: Child-Parent fields
• Parent child fiels are similar to one-to-one fields,
but you can go only from child to parent not vice verca
• CITY and COUNTRY (child-parent)
• CITY is more important than COUNTRY (having CITY you
can guess the COUNTRY)
• To find child-parent fields explore the values in a
sample
• More optimization to the lattice and to UI
recomendations

86. DD4: Empty Columns
• Sounds easy, but currently empty columns
treated the same way as other
• We can easily drop them from the lattice to
reduce its size
• We can remove them from UI

87. DD?: Define your own Data
Dependency
• Explain dependency?
• Rule to identify data dependecy?
• Where the knowledge can be applied?

88. How the future of
Zoomdata may look
like?

89. Zoomdata Applications
• Pregen management
• Sharpening with no limits
• Offline Mode
• Data Dependencies
• UX Recomendations

91. Hope for the best,
prepare for the
worst
— Quan Luu

92. Appendix 1:
Review of
algorithms for
Knapsack Problem

93. Exact methods
• Bruteforce
• Bound & Branch
• Dynamic Programming

94. Bruteforce
• KSBruteforceSolver.java
• Label each object either 1 (take) or 0 (do not take)
• Build all possible configurations
• Select the best one
• Complexity , - number of columns, -
lattice size
• For example, for columns, there are
possible combinations

95. Bound & Branch
• KSBoundBranchSolver.java
• Explore search space in depth-first search
manner, providing optimistic estimate for each
node to be explored, this way you can prune
nodes which optimistic estimate worse than best
of found solutions
• In most cases faster than bruteforce, but worse
case compexity is still

96. Dynamic Programming
• KSDynamicProgrammingSolver.java
• Explore the classic DP idea, solve smaller
problem first, then reuse results in bigger
problem.
• For some cases extra fast (pseudopolynomial
complexity)
• A lot of constraints (weights are integers,
weight limits should be relatively small)

97. Approximate methods
• Greedy Algorithm
• Genetic Algorithm

98. Greedy Algorithm
• KSGreedySolver.java
• Order all items by "value" (do not confuse with
knapsack value) and take until you reach the weight
constraint
• Strategies: VALUABLE, LIGHTER,DENSITY` (value/weight)
• Complexity , could be even faster
• Practical approach when constraints are large values
(!)

99. Genetic Algorithm
• KSGeneticSolver.java
• Algorithm which could be used to solve any problem.
Init. generate random population (set of solutions)
Selection. Choose the best solutions
Crossover. Produce new solutions by breeding previous
best solutions.
Mutation (Optional). Randomly change new-born
solutions.
Generation Step. If new populations is better than
previous, replace it and move to selection again.

100. Thanks