"Venice is LinkedIn’s low latency derived database that powers most of the company’s AI use cases. Last year we open sourced Venice, which means that you (Yeah! You!!) can deploy and run Venice in your stack. Venice is designed from the ground up with asynchrony in mind, making it a great fit for ingesting the output of your stream processing jobs. It also supports Lambda and Kappa architectures out of the box. This talk presents Venice and deep dives into how we designed it to enable high data ingestion volumes via Kafka, merging it all coherently from many data sources and many geographically distributed regions. Venice’s active-active replication allows write operations to be produced in any region and be consumed in any order by all other regions, while still achieving eventual consistency. We’ll cover how Venice’s conflict resolution strategy can be a powerful abstraction when dealing with many writers."

2. •
•
•

3. •
•
•
•

5. •
•
•

7. •
•
•
•
•
•

8. •

9. •
•
•
•
•

10. Venice
Dataset v1
(backup)
Venice
Dataset v2
(current)
Venice
Client
Hadoop Grid Full Push (VPJ)

11. Venice
Dataset v1
(backup)
Venice
Dataset v2
(current)
Venice
Client
Hadoop Grid
Nearline
Write
Buffer
Incremental
Push
Incremental
Push
Incremental
Push

12. Incremental
Push
Input
Stream
Nearline
Samza
Job
Nearline
Write
Buffer
Venice
Dataset v2
(current)
Venice
Client
Nearline
Samza
Job
Hadoop Grid
Venice
Dataset v3
(future)
Reprocessing
Samza
Job
Brooklin
Bootstrap
Input

13. Venice
Dataset v2
(backup)
Venice
Dataset v3
(current)
Venice
Client
Nearline
Write
Buffer
Online Writer

14. •
•
•

15. •
•
•
•
•
•

16. •
•
•

25. {A=1, B="bar"} @ TS1
{A=10, B="pop"} @ TS2 1. {A=5, B=“foo”} @ TS {0}
1. {A=5, B=“foo”} @ TS {0}

26. 1. {A=5, B=“foo”} @ TS {0}
2. {A=1, B=“bar”} @ TS {1}
1. {A=5, B=“foo”} @ TS {0}
2. {A=10, B=“pop”} @ TS {2}
{A=1, B="bar"} @ TS1
{A=10, B="pop"} @ TS2

27. 1. {A=5, B=“foo”} @ TS {0}
2. {A=1, B=“bar”} @ TS {1}
3. {A=10, B="pop"} @ TS {2}
1. {A=5, B=“foo”} @ TS {0}
2. {A=10, B=“pop”} @ TS {2}
{A=1, B="bar"} @ TS1
{A=10, B="pop"} @ TS2

30. In the lost update problem, an update done to a data
item by a transaction is lost as it is overwritten by the
update done by another transaction.

31. {A=1, C=10} @ TS1

32. {A=1, C=10} @ TS1
{B=“bar”, C=15} @ TS2

33. ¡¿Que!?
{B=“bar”, C=15}

34. {A=1, C=10} @ TS1

35. {A=1, C=10} @ TS1
{B=“bar”, C=15} @ TS2
IF-Unmodified-Since: T0

36. {B=“bar”, C=15} @ TS2
IF-Unmodified-Since: T0

38. {A=1, C=10} @ TS1

39. {A=1, C=10} @ TS1
{B=“bar”, C=15} @ TS2
IF-Unmodified-Since: T0

41. {B=“bar”, C=15} {B=“bar”, C=15}

44. {B=“bar”, C=15}
{A=1, C=10}

45. {B=“bar”, C=15}
{A=1, C=10}
{A=1, C=10}

46. {B=“bar”, C=15}
{A=1, C=10}
{B=“bar”, C=15}

48. •
•
•
•

49. •
•
•

50. •
•
•

51. {A=5, B=“foo”}

52. {A=1}
{B=“bar”}
{A=5, B=“foo”}
{A=1, B=“bar”}

53. •
•

54. 1. {A=5, B=“foo”, C=0} @ TS {0, 0, 0}
1. {A=5, B=“foo”, C=0} @ TS {0, 0, 0}
{A=1, C=10} @ TS1
{B=“bar”, C=15} @ TS2

55. 1. {A=5, B=“foo”, C=0} @ TS {0, 0, 0}
2. {A=1, B=“foo”, C=10} @ TS {1, 0, 1}
1. {A=5, B=“foo”, C=0} @ TS {0, 0, 0}
2. {A=5, B=“bar”, C=15} @ TS {0, 2, 2}
{A=1, C=10} @ TS1
{B=“bar”, C=15} @ TS2

56. 1. {A=5, B=“foo”, C=0} @ TS {0, 0, 0}
2. {A=1, B=“foo”, C=10} @ TS {1, 0, 1}
3. {A=1, B=“bar”, C=15} @ TS {1, 2, 2}
1. {A=5, B=“foo”, C=0} @ TS {0, 0, 0}
2. {A=5, B=“bar”, C=15} @ TS {0, 2, 2}
3. {A=1, B=“bar”, C=15} @ TS {1, 2, 2}
{A=1, C=10} @ TS1
{B=“bar”, C=15} @ TS2

57. •

58. {B=“bar”, C=15}
{A=1, C=10}

59. {B=“bar”, C=15}t=1
{A=1, C=10} t=2
{A=1, C=10} w/
time=2

60. {B=“bar”, C=15}t=1
{A=1, C=10} t=2
{B=“bar”, C=15}w/ t=1
{A=1, B=“bar”, C=10}
@ TS {2, 1, 2}

61. •
•
•

62. •
•
•
•
•

63. •
•
•

64. •
•

66. •
•
•

67. •
•
•

68. •
•
•
•
•
•

78. •
•
•
•
•

80. •
•
•

81. •

82. Gaojie Liu Alex Dubrouski
Sr Staff Engineer
@LinkedIn
Sr Staff Engineer
@LinkedIn