This week, Liam Eagen (Blockstream Research) and Ariel Gabizon (Zeta Function Technologies) present cqlin - Efficient linear operations on KZG commitments with cached quotients.
Given two KZG-committed polynomials , a matrix , and subgroup of order , we present a protocol for checking that . After preprocessing, the prover makes field and group operations. This presents a significant improvement over the lincheck protocols in [CHMMVW, COS], where the prover's run-time (also after preprocessing) was quasilinear in the number of non-zeroes of M, which could be n^2.
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...Flink Forward
Getting data in and out of Flink is by far the most important aspect, and an everyday typical requirement of building Flink applications. Doing so in an end-to-end exactly-once manner, however, can be tricky. Being able to reliably consume data from the outside world without any duplicate processing and guaranteeing consistent distributed state, and at the same time provide computed results back to the outside world also without introducing duplicates, is crucial for the consistency and correctness of applications built upon stream processors. In this talk, we will talk about how end-to-end exactly-once guarantees can be achieved with Apache Flink. We will talk about Flink’s checkpointing mechanism, and how exactly to leverage it when consuming and producing data from your Flink streaming pipelines. In particular, we will be having a detailed review on how our supported connectors do so, with the aim to provide reference implementations for your own custom consumers and sinks.
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache FlinkVerverica
As Apache Flink continues to push the boundaries of stateful stream processing as an integral part of its past releases, increasing numbers of users are starting to realize the potential of stateful stream processing as a promising paradigm for robust and reactive data analytics as well as event-driven applications.
This talk aims at covering the general idea and motivations of stateful stream processing, and how Flink enables it with its powerful set of state management features and programming APIs. In addition to that, we will also take a look at the recent advancements related to Flink's state management and large state handling that were driven by our team at data Artisans team in the latest version 1.3 (expected release by end of May / early June).
How to add a yellow circle around mouse cursor? - ThiyaguThiyagu K
This presentation explains the procedure of how to get or add a yellow circle around our mouse cursor or pointer. PenAttention is a free Windows program that displays a highlight, pencil, or pointer at the location of the pen. It's intended for use in presentations on a pen-enabled laptop or PC so our audience can see what we are pointing at on the screen. It is a self explanatory tutorial..
Druid provides sub-second query latency and Flink provides SQL on streams allowing rich transformation/enrichment of events as it happens. In this talk we will learn how Lyft
uses flink sql and druid together to support real time analytics.
Meetup: https://www.meetup.com/druidio/events/252515792/
Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...Flink Forward
Getting data in and out of Flink is by far the most important aspect, and an everyday typical requirement of building Flink applications. Doing so in an end-to-end exactly-once manner, however, can be tricky. Being able to reliably consume data from the outside world without any duplicate processing and guaranteeing consistent distributed state, and at the same time provide computed results back to the outside world also without introducing duplicates, is crucial for the consistency and correctness of applications built upon stream processors. In this talk, we will talk about how end-to-end exactly-once guarantees can be achieved with Apache Flink. We will talk about Flink’s checkpointing mechanism, and how exactly to leverage it when consuming and producing data from your Flink streaming pipelines. In particular, we will be having a detailed review on how our supported connectors do so, with the aim to provide reference implementations for your own custom consumers and sinks.
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache FlinkVerverica
As Apache Flink continues to push the boundaries of stateful stream processing as an integral part of its past releases, increasing numbers of users are starting to realize the potential of stateful stream processing as a promising paradigm for robust and reactive data analytics as well as event-driven applications.
This talk aims at covering the general idea and motivations of stateful stream processing, and how Flink enables it with its powerful set of state management features and programming APIs. In addition to that, we will also take a look at the recent advancements related to Flink's state management and large state handling that were driven by our team at data Artisans team in the latest version 1.3 (expected release by end of May / early June).
How to add a yellow circle around mouse cursor? - ThiyaguThiyagu K
This presentation explains the procedure of how to get or add a yellow circle around our mouse cursor or pointer. PenAttention is a free Windows program that displays a highlight, pencil, or pointer at the location of the pen. It's intended for use in presentations on a pen-enabled laptop or PC so our audience can see what we are pointing at on the screen. It is a self explanatory tutorial..
Druid provides sub-second query latency and Flink provides SQL on streams allowing rich transformation/enrichment of events as it happens. In this talk we will learn how Lyft
uses flink sql and druid together to support real time analytics.
Meetup: https://www.meetup.com/druidio/events/252515792/
Donald Miner will do a quick introduction to Apache Hadoop, then discuss the different ways Python can be used to get the job done in Hadoop. This includes writing MapReduce jobs in Python in various different ways, interacting with HBase, writing custom behavior in Pig and Hive, interacting with the Hadoop Distributed File System, using Spark, and integration with other corners of the Hadoop ecosystem. The state of Python with Hadoop is far from stable, so we'll spend some honest time talking about the state of these open source projects and what's missing will also be discussed.
Intro to Complex Networks was a workshop for the master of students (M.Sc.) at the University of Zanjan. It was about Protein-Protein Interaction networks and some graph concepts.
Battle of the Stream Processing Titans – Flink versus RisingWaveYingjun Wu
The world of real-time data processing is constantly evolving, with new technologies and platforms emerging to meet the ever-increasing demands of modern data-driven businesses. Apache Flink and RisingWave are two powerful stream processing solutions that have gained significant traction in recent years. But which platform is right for your organization? Karin Wolok and Yingjun Wu go head-to-head to compare and contrast the strengths and limitations of Flink and RisingWave. They’ll also share real-world use cases, best practices for optimizing performance and efficiency, and key considerations for selecting the right solution for your specific business needs.
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfAltinity Ltd
Join the Altinity experts as we dig into ClickHouse sharding and replication, showing how they enable clusters that deliver fast queries over petabytes of data. We’ll start with basic definitions of each, then move to practical issues. This includes the setup of shards and replicas, defining schema, choosing sharding keys, loading data, and writing distributed queries. We’ll finish up with tips on performance optimization.
#ClickHouse #datasets #ClickHouseTutorial #opensource #ClickHouseCommunity #Altinity
-----------------
Join ClickHouse Meetups: https://www.meetup.com/San-Francisco-...
Check out more ClickHouse resources: https://altinity.com/resources/
Visit the Altinity Documentation site: https://docs.altinity.com/
Contribute to ClickHouse Knowledge Base: https://kb.altinity.com/
Join the ClickHouse Reddit community: https://www.reddit.com/r/Clickhouse/
----------------
Learn more about Altinity!
Site: https://www.altinity.com
LinkedIn: https://www.linkedin.com/company/alti...
Twitter: https://twitter.com/AltinityDB
Flink Forward Berlin 2017: Patrick Lucas - Flink in ContainerlandFlink Forward
Apache Flink, a powerful distributed stateful stream processing framework, is an especially good fit for deployment on a containerization platform: its storage requirement is primarily external (e.g. HDFS or S3), clusters often share the lifetime of the jobs that run on them, and the flexibility of allocating resources on such a platform allows for scaling jobs up and down as necessary. In this talk I will give a brief introduction to Apache Flink, then describe the journey to making it a first-class citizen of the container world. I will cover my experience preparing to publish the “official repository” of Flink images on Docker Hub, the challenges of fitting a Flink deployment in a Kubernetes-shaped box, and the rough edges of Flink itself that were exposed by this process.
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...Mihai Criveti
Automate your Data Science pipeline with Ansible, Python and Kubernetes - ODSC Talk
What is Data Science and the Data Science Landscape
Process and Flow
Understanding Data
The Data Science Toolkit
The Big Data Challenge
Cloud Computing Solutions
The rise of DevOps in Data Science
Automate your data pipeline with Ansible
Apache Hive is an Enterprise Data Warehouse build on top of Hadoop. Hive supports Insert/Update/Delete SQL statements with transactional semantics and read operations that run at Snapshot Isolation. This talk will describe the intended use cases, architecture of the implementation, new features such as SQL Merge statement and recent improvements. The talk will also cover Streaming Ingest API, which allows writing batches of events into a Hive table without using SQL. This API is used by Apache NiFi, Storm and Flume to stream data directly into Hive tables and make it visible to readers in near real time.
Apache Doris (incubating) is an MPP-based interactive SQL data warehousing for reporting and analysis. It is open-sourced by Baidu. Doris mainly integrates the technology of Google Mesa and Apache Impala. Unlike other popular SQL-on-Hadoop systems, Doris is designed to be a simple and single tightly coupled system, not depending on other systems. Doris not only provides high concurrent low latency point query performance, but also provides high throughput queries of ad-hoc analysis. Doris not only provides batch data loading, but also provides near real-time mini-batch data loading. Doris also provides high availability, reliability, fault tolerance, and scalability. The simplicity (of developing, deploying and using) and meeting many data serving requirements in single system are the main features of Doris.
Donald Miner will do a quick introduction to Apache Hadoop, then discuss the different ways Python can be used to get the job done in Hadoop. This includes writing MapReduce jobs in Python in various different ways, interacting with HBase, writing custom behavior in Pig and Hive, interacting with the Hadoop Distributed File System, using Spark, and integration with other corners of the Hadoop ecosystem. The state of Python with Hadoop is far from stable, so we'll spend some honest time talking about the state of these open source projects and what's missing will also be discussed.
Intro to Complex Networks was a workshop for the master of students (M.Sc.) at the University of Zanjan. It was about Protein-Protein Interaction networks and some graph concepts.
Battle of the Stream Processing Titans – Flink versus RisingWaveYingjun Wu
The world of real-time data processing is constantly evolving, with new technologies and platforms emerging to meet the ever-increasing demands of modern data-driven businesses. Apache Flink and RisingWave are two powerful stream processing solutions that have gained significant traction in recent years. But which platform is right for your organization? Karin Wolok and Yingjun Wu go head-to-head to compare and contrast the strengths and limitations of Flink and RisingWave. They’ll also share real-world use cases, best practices for optimizing performance and efficiency, and key considerations for selecting the right solution for your specific business needs.
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfAltinity Ltd
Join the Altinity experts as we dig into ClickHouse sharding and replication, showing how they enable clusters that deliver fast queries over petabytes of data. We’ll start with basic definitions of each, then move to practical issues. This includes the setup of shards and replicas, defining schema, choosing sharding keys, loading data, and writing distributed queries. We’ll finish up with tips on performance optimization.
#ClickHouse #datasets #ClickHouseTutorial #opensource #ClickHouseCommunity #Altinity
-----------------
Join ClickHouse Meetups: https://www.meetup.com/San-Francisco-...
Check out more ClickHouse resources: https://altinity.com/resources/
Visit the Altinity Documentation site: https://docs.altinity.com/
Contribute to ClickHouse Knowledge Base: https://kb.altinity.com/
Join the ClickHouse Reddit community: https://www.reddit.com/r/Clickhouse/
----------------
Learn more about Altinity!
Site: https://www.altinity.com
LinkedIn: https://www.linkedin.com/company/alti...
Twitter: https://twitter.com/AltinityDB
Flink Forward Berlin 2017: Patrick Lucas - Flink in ContainerlandFlink Forward
Apache Flink, a powerful distributed stateful stream processing framework, is an especially good fit for deployment on a containerization platform: its storage requirement is primarily external (e.g. HDFS or S3), clusters often share the lifetime of the jobs that run on them, and the flexibility of allocating resources on such a platform allows for scaling jobs up and down as necessary. In this talk I will give a brief introduction to Apache Flink, then describe the journey to making it a first-class citizen of the container world. I will cover my experience preparing to publish the “official repository” of Flink images on Docker Hub, the challenges of fitting a Flink deployment in a Kubernetes-shaped box, and the rough edges of Flink itself that were exposed by this process.
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...Mihai Criveti
Automate your Data Science pipeline with Ansible, Python and Kubernetes - ODSC Talk
What is Data Science and the Data Science Landscape
Process and Flow
Understanding Data
The Data Science Toolkit
The Big Data Challenge
Cloud Computing Solutions
The rise of DevOps in Data Science
Automate your data pipeline with Ansible
Apache Hive is an Enterprise Data Warehouse build on top of Hadoop. Hive supports Insert/Update/Delete SQL statements with transactional semantics and read operations that run at Snapshot Isolation. This talk will describe the intended use cases, architecture of the implementation, new features such as SQL Merge statement and recent improvements. The talk will also cover Streaming Ingest API, which allows writing batches of events into a Hive table without using SQL. This API is used by Apache NiFi, Storm and Flume to stream data directly into Hive tables and make it visible to readers in near real time.
Apache Doris (incubating) is an MPP-based interactive SQL data warehousing for reporting and analysis. It is open-sourced by Baidu. Doris mainly integrates the technology of Google Mesa and Apache Impala. Unlike other popular SQL-on-Hadoop systems, Doris is designed to be a simple and single tightly coupled system, not depending on other systems. Doris not only provides high concurrent low latency point query performance, but also provides high throughput queries of ad-hoc analysis. Doris not only provides batch data loading, but also provides near real-time mini-batch data loading. Doris also provides high availability, reliability, fault tolerance, and scalability. The simplicity (of developing, deploying and using) and meeting many data serving requirements in single system are the main features of Doris.
This paper presents an interesting idea how to compute a consensus of several k-partitions of a set by means of finding an antichain in the concept lattice of an appropriate formal context.
Detecting paraphrases using recursive autoencodersFeynman Liang
Presentation on deep learning applied to natural language processing, presented at University of Cambridge Machine Learning Group's Research and Communication Club 2-11-2015 meeting.
Bayesian Nonparametric Topic Modeling Hierarchical Dirichlet ProcessesJinYeong Bak
This is presentation slide files in machine learning summer school in Korea.
http://prml.yonsei.ac.kr/
I talked about dirichlet distribution, dirichlet process and HDP.
Perplexity of Index Models over Evolving Linked Data Thomas Gottron
ESWC presentation on the stability of 12 different index models for linked data. Provides a formalisation of the index models as well as stability evaluation based on data distributions and information theoretic metrics.
Sensors and Samples: A Homological ApproachDon Sheehy
In their seminal work on homological sensor networks, de Silva and Ghrist showed the surprising fact that its possible to certify the coverage of a coordinate free sensor network even with very minimal knowledge of the space to be covered. We give a new, simpler proof of the de Silva-Ghrist Topological Coverage Criterion that eliminates any assumptions about the smoothness of the boundary of the underlying space, allowing the results to be applied to much more general problems. The new proof factors the geometric, topological, and combinatorial aspects of this approach. This factoring reveals an interesting new connection between the topological coverage condition and the notion of weak feature size in geometric sampling theory. We then apply this connection to the problem of showing that for a given scale, if one knows the number of connected components and the distance to the boundary, one can also infer the higher betti numbers or provide strong evidence that more samples are needed. This is in contrast to previous work which merely assumed a good sample and gives no guarantees if the sampling condition is not met.
Regularity of Generalized Derivations in P Semi Simple BCIK AlgebrasYogeshIJTSRD
In this paper we study the regularity of inside or outside derivations in p semi simple BCIK – algebra X and prove that let d X X be an inside derivation of X. If there exists a X such that d x a = 0, then d is regular for all x X. It is also show that if X is a BCIK algebra, then every inside or outside derivation of X is regular. Furthermore the concepts of ideal, ideal and invariant inside or outside derivation of X are introduced and their related properties are investigated. Finally we obtain the following result If d X X is an outside derivation of X, then d is regular if and only if every ideal of X is d invariant. S Rethina Kumar "Regularity of Generalized Derivations in P-Semi Simple BCIK-Algebras" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-5 | Issue-3 , April 2021, URL: https://www.ijtsrd.com/papers/ijtsrd39949.pdf Paper URL: https://www.ijtsrd.com/mathemetics/algebra/39949/regularity-of-generalized-derivations-in-psemi-simple-bcikalgebras/s-rethina-kumar
Abstract: An enhanced hybrid approach to OWL query answering that combines an RDF triple-store with an OWL reasoner in order to provide scaleable pay-as-you-go performance. The enhancements presented here include an extension to deal with arbitary OWL ontologies and optimisations that significantly improve scalability. We have implemented these techniques in a prototype system, a preliminary evaluation of which has produced very encouraging results.
Similar to zkStudyClub - cqlin: Efficient linear operations on KZG commitments (20)
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
A decade of active research has led to practical constructions of zero-knowledge succinct non-interactive arguments of knowledge (zk-SNARKs) that are now being used in a wide variety of applications. Despite this astonishing progress, overheads in proof generation time remain significant.
In this work, we envision a world where consumers with low computational resources can outsource the task of proof generation to a group of untrusted servers in a privacy-preserving manner. The main requirement is that these servers should be able to collectively generate proofs at a faster speed (than the consumer). Towards this goal, we introduce a framework called zk-SNARKs-as-a-service () for faster computation of zk-SNARKs. Our framework allows for distributing proof computation across multiple servers such that each server is expected to run for a shorter duration than a single prover. Moreover, the privacy of the prover's witness is ensured against any minority of colluding servers.
We design custom protocols in this framework that can be used to obtain faster runtimes for widely used zk-SNARKs, such as Groth16 [EUROCRYPT 2016], Marlin [EUROCRYPT 2020], and Plonk [EPRINT 2019]. We implement proof of concept zkSaaS for the Groth16 and Plonk provers. In comparison to generating these proofs on commodity hardware, we show that not only can we generate proofs for a larger number of constraints (without memory exhaustion), but can also get speed-up when run with 128 parties for constraints with Groth16 and gates with Plonk.
https://eprint.iacr.org/2023/905
This week's session covers new work from Justin Thaler (GWU) et al on Lasso/Jolt.
Lasso is a new lookup argument (more on this below) with a dramatically faster prover. Our initial implementation provides roughly a 10x speedup over the lookup argument in the popular, well-engineered halo2 toolchain; we expect improvements of around 40x when optimizations are complete. To demonstrate, we’re releasing the open source implementation, written in Rust. We invite the community to help us make Lasso as fast and robust as possible.
The second, accompanying innovation to Lasso is Jolt, a new approach to zkVM (zero knowledge virtual machine) design that builds on Lasso. Jolt realizes the “lookup singularity” – a vision initially laid out by Barry Whitehat of the Ethereum Foundation for simpler tooling and lightweight, lookup-centric circuits (more on why this matters below). Relative to existing zkVMs, we expect Jolt to achieve similar or better performance – and importantly, a more streamlined and accessible developer experience. With Jolt, it will be easier for developers to write fast SNARKs in their high-level language of choice.
Lasso: https://people.cs.georgetown.edu/jthaler/Lasso-paper.pdf
Jolt: https://people.cs.georgetown.edu/jthaler/Jolt-paper.pdf
zkStudyClub - Improving performance of non-native arithmetic in SNARKs (Ivo K...Alex Pruden
In this zkStudyClub session, Ivo presents techniques for applying the log-derivative lookup tables in a circuit using LegoSNARK-style commitment. As an application, we show how this lookup table can be used to implement range checks, specifically applying it to the non-native arithmetic. Using these optimisations, we were able to reduce the proof time for BN254 pairing in Groth16 to approx 5s (MBP M1). The technique also works for PLONKish arithmetisation.
This week, Benedikt Bünz and Binyi Chen of Espresso Systems present ProtoStar:
Accumulation is a simple yet powerful primitive that enables incrementally verifi-able computation (IVC) without the need for recursive SNARKs. We provide a generic, efficient accumulation (or folding) scheme for any (2k − 1)-move special-sound protocol. The prover in each accumulation/IVC step is also only logarithmic in the number of supported circuits and independent of the table size in the lookup
https://eprint.iacr.org/2023/620
ZK Study Club: Supernova (Srinath Setty - MS Research)Alex Pruden
This week, Srinath Setty (MS Research) will present SuperNova, a new recursive proof system for incrementally producing succinct proofs of correct execution of programs on a stateful machine with a particular instruction set (e.g., EVM, RISC-V). A distinguishing aspect of SuperNova is that the cost of proving a step of a program is proportional only to the size of the circuit representing the instruction invoked by the program step. This is a stark departure from prior works that employ universal circuits where the cost of proving a program step is proportional at least to the sum of sizes of circuits representing each supported instruction—even though a particular program step invokes only one of the supported instructions. Naturally, SuperNova can support a rich instruction set without affecting the per-step proving costs. SuperNova achieves its cost profile by building on Nova, a prior high-speed recursive proof system, and leveraging its internal building block, folding schemes, in a new manner. We formalize SuperNova’s approach as a way to realize non-uniform IVC, a generalization of IVC. Furthermore, SuperNova’s prover costs and the recursion overhead are the same as Nova’s, and in fact, SuperNova is equivalent to Nova for machines that support a single instruction.
https://eprint.iacr.org/2022/1758
Eos - Efficient Private Delegation of zkSNARK proversAlex Pruden
Succinct zero knowledge proofs (i.e. zkSNARKs) are powerful cryptographic tools that enable a prover to convince a verifier that a given statement is true without revealing any additional information. Unfortunately, existing systems for generating zkSNARKs are expensive, which limits the applications in which these proofs can be used.
This new work (presented by co-author Pratyush Mishra) achieves security against malicious workers without relying on heavyweight cryptographic tools. We implement and evaluate our delegation protocols for a state-of-the-art zkSNARK in a variety of computational and bandwidth settings, and demonstrate that our protocols
are concretely efficient. When compared to local proving, using our protocols to delegate proof generation from a recent smartphone (a) reduces end-to-end latency by up to 26×, (b) lowers the delegator’s active computation time by up to 1447×, and (c) enables proving up to 256× larger instances
https://www.usenix.org/system/files/sec23fall-prepub-492-chiesa.pdf
Paper: https://eprint.iacr.org/2022/1355
Plonk is a widely used succinct non-interactive proof system that uses univariate polynomial commitments. Plonk is quite flexible: it supports circuits with low-degree ``custom'' gates as well as circuits with lookup gates (a lookup gate ensures that its input is contained in a predefined table). For large circuits, the bottleneck in generating a Plonk proof is the need for computing a large FFT.
In this work, the authors present HyperPlonk, an adaptation of Plonk to the boolean hypercube, using multilinear polynomial commitments. HyperPlonk retains the flexibility of Plonk but provides several additional benefits. First, it avoids the need for an FFT during proof generation. Second, and more importantly, it supports custom gates of much higher degree than Plonk without harming the running time of the prover. Both of these can dramatically speed up the prover's running time. Since HyperPlonk relies on multilinear polynomial commitments, the authors revisit two elegant constructions: one from Orion and one from Virgo. The authors also show how to reduce the Orion opening proof size to less than 10kb (an almost factor 1000 improvement) and show how to make the Virgo FRI-based opening proof simpler and shorter.
Caulk: zkStudyClub: Caulk - Lookup Arguments in Sublinear Time (A. Zapico)Alex Pruden
This week, Arantxa Zapico of the Ethereum Foundation presents new work (co-authored with Vitalik Buterin, Dmitry Khovratovich, Mary Maller, Anca Nitulescu, and Mark Simkin) called Caulk, which examines position-hiding linkability for vector commitment schemes. One can prove in zero knowledge that one or more values that comprise commitment cm all belong to the vector of size committed to in C. Caulk can be used for membership proofs and lookup arguments and outperforms all existing alternatives in prover time by orders of magnitude.
https://eprint.iacr.org/2022/621
zkStudyClub: Zero-Knowledge Proofs Security, in Practice [JP Aumasson, Taurus]Alex Pruden
Slides accompanying zkStudyClub talk: Zero-Knowledge Proofs Security, in Practice. JP Aumasson (co-creator of the BLAKE hash function family) will share his experience doing security auditing for various projects that use zero-knowledge proofs. He will describe his approach, the common pitfalls in the different components of a proof system, as well as a catalog of bugs that have been discovered in various projects
This week, Luke Pearson (Polychain Capital) and Joshua Fitzgerald (Anoma) present their work on Plonkup, a protocol that combines Plookup and PLONK into a single, efficient protocol. The protocol relies on a new hash function, called Reinforced Concrete, written by Dmitry Khovratovich. The three of them will present their work together at this week's edition of zkStudyClub!
Slides:
---
To Follow the Zero Knowledge Podcast us at https://www.zeroknowledge.fm
To the listeners of Zero Knowledge Podcast, if you like what we do:
- Follow us on Twitter - @zeroknowledgefm
- Join us on Telegram - https://t.me/joinchat/TORo7aknkYNLHmCM
- Support our Gitcoin Grant - https://gitcoin.co/grants/329/zero-knowledge-podcast-2
- Support us on Patreon - https://www.patreon.com/zeroknowledge
zkStudy Club: Subquadratic SNARGs in the Random Oracle ModelAlex Pruden
Slides for Eylon Yogev's (Bar-Ilan University) presentation at ZKStudyClub, covering his new work (co-authored w/ Alessandro Chiesa of UC Berkeley) about SNARGs in the random oracle model of sub- quadratic complexity.
Link to the original paper: https://eprint.iacr.org/2021/281.pdf
ZK Study Club: Sumcheck Arguments and Their ApplicationsAlex Pruden
Talk given at the ZK Study Club by Jonathan Bootle and Katerina Sotiraki about the universality of sumcheck arguments and their importance in zero-knowledge cryptography.
zkStudyClub: CirC and Compiling Programs to CircuitsAlex Pruden
The programming languages community, the cryptography community, and others rely on translating programs in high-level source languages (e.g., C) to logical constraint representations. Unfortunately, building compilers for this task is difficult and time consuming. In this work, Alex Ozdemir et al present CirC, an infrastructure for building compilers for SNARKs that build upon a common abstraction: stateless, non-deterministic computations called existentially quantified circuits, or EQCs.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
zkStudyClub - cqlin: Efficient linear operations on KZG commitments
1. cqlin
New Applications for Universal, Pairing Based SNARKs
with Preprocessing
Liam Eagen
Blockstream Research
June 2, 2023
Liam Eagen (Blockstream Research) cqlin June 2, 2023 1 / 17
3. Motivation
Lin-check problem is ubiquitous in SNARKs
Many approaches linear in sparsity (number of non-zero entries) of
matrix
Liam Eagen (Blockstream Research) cqlin June 2, 2023 2 / 17
4. Motivation
Lin-check problem is ubiquitous in SNARKs
Many approaches linear in sparsity (number of non-zero entries) of
matrix
This is O(n2) for dense matrices
Liam Eagen (Blockstream Research) cqlin June 2, 2023 2 / 17
5. Motivation
Lin-check problem is ubiquitous in SNARKs
Many approaches linear in sparsity (number of non-zero entries) of
matrix
This is O(n2) for dense matrices
Dense matrix multiplication useful in zkML
Liam Eagen (Blockstream Research) cqlin June 2, 2023 2 / 17
6. Motivation
Lin-check problem is ubiquitous in SNARKs
Many approaches linear in sparsity (number of non-zero entries) of
matrix
This is O(n2) for dense matrices
Dense matrix multiplication useful in zkML
Prior work [GKMMM] implies O(n) lin-check protocol
▶ Special SRS depends on the size of the matrix
▶ Has O(n3
) setup time
Liam Eagen (Blockstream Research) cqlin June 2, 2023 2 / 17
8. Why Pairings and Preprocessing?
Ariel’s SNARK trilogy (zkSummit 9/Zero Knowledge Podcast ep.
274)
1 Pairings get succinctness without PCP, need per-circuit trusted setup
Liam Eagen (Blockstream Research) cqlin June 2, 2023 3 / 17
9. Why Pairings and Preprocessing?
Ariel’s SNARK trilogy (zkSummit 9/Zero Knowledge Podcast ep.
274)
1 Pairings get succinctness without PCP, need per-circuit trusted setup
2 Polynomial commitment schemes abstract pairing, universal trusted
setup
Liam Eagen (Blockstream Research) cqlin June 2, 2023 3 / 17
10. Why Pairings and Preprocessing?
Ariel’s SNARK trilogy (zkSummit 9/Zero Knowledge Podcast ep.
274)
1 Pairings get succinctness without PCP, need per-circuit trusted setup
2 Polynomial commitment schemes abstract pairing, universal trusted
setup
3 Pairing with preprocessing breaks PCS abstraction for more power in
particular applications, e.g. lookup arguments
Liam Eagen (Blockstream Research) cqlin June 2, 2023 3 / 17
11. Why Pairings and Preprocessing?
Ariel’s SNARK trilogy (zkSummit 9/Zero Knowledge Podcast ep.
274)
1 Pairings get succinctness without PCP, need per-circuit trusted setup
2 Polynomial commitment schemes abstract pairing, universal trusted
setup
3 Pairing with preprocessing breaks PCS abstraction for more power in
particular applications, e.g. lookup arguments
What other applications benefit from pairings and preprocessing?
Liam Eagen (Blockstream Research) cqlin June 2, 2023 3 / 17
12. Why Pairings and Preprocessing?
Ariel’s SNARK trilogy (zkSummit 9/Zero Knowledge Podcast ep.
274)
1 Pairings get succinctness without PCP, need per-circuit trusted setup
2 Polynomial commitment schemes abstract pairing, universal trusted
setup
3 Pairing with preprocessing breaks PCS abstraction for more power in
particular applications, e.g. lookup arguments
What other applications benefit from pairings and preprocessing?
cqlin!
Liam Eagen (Blockstream Research) cqlin June 2, 2023 3 / 17
14. Preliminaries
Fix a pairing friendly curve
Commit to polynomials using KZG, write [F(x)] for commitment to
polynomial F(X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 4 / 17
15. Preliminaries
Fix a pairing friendly curve
Commit to polynomials using KZG, write [F(x)] for commitment to
polynomial F(X)
Prove F(a) = b by proving Q(X) = (F(X) − b)/(X − a) is a
polynomial
Liam Eagen (Blockstream Research) cqlin June 2, 2023 4 / 17
16. Preliminaries
Fix a pairing friendly curve
Commit to polynomials using KZG, write [F(x)] for commitment to
polynomial F(X)
Prove F(a) = b by proving Q(X) = (F(X) − b)/(X − a) is a
polynomial
More generally for Z(X) =
Q
i (X − ai ), F(ai ) = G(ai ) iff
F(X) = G(X) mod Z(X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 4 / 17
17. Preliminaries
Fix a pairing friendly curve
Commit to polynomials using KZG, write [F(x)] for commitment to
polynomial F(X)
Prove F(a) = b by proving Q(X) = (F(X) − b)/(X − a) is a
polynomial
More generally for Z(X) =
Q
i (X − ai ), F(ai ) = G(ai ) iff
F(X) = G(X) mod Z(X)
Equivalently, if there exists a polynomial
Q(X) = (F(X) − G(X))/Z(X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 4 / 17
18. Preliminaries
Let ω be a root of unity or order N
Liam Eagen (Blockstream Research) cqlin June 2, 2023 5 / 17
19. Preliminaries
Let ω be a root of unity or order N
Lagrange polynomials Li (ωj ) = δi,j
Liam Eagen (Blockstream Research) cqlin June 2, 2023 5 / 17
20. Preliminaries
Let ω be a root of unity or order N
Lagrange polynomials Li (ωj ) = δi,j
Encode vector ⃗
v ∈ FN as F(ωi ) = vi
Liam Eagen (Blockstream Research) cqlin June 2, 2023 5 / 17
21. Preliminaries
Let ω be a root of unity or order N
Lagrange polynomials Li (ωj ) = δi,j
Encode vector ⃗
v ∈ FN as F(ωi ) = vi
Equivalently F(X) =
P
i∈[n] vi Li (X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 5 / 17
22. Preliminaries
Let ω be a root of unity or order N
Lagrange polynomials Li (ωj ) = δi,j
Encode vector ⃗
v ∈ FN as F(ωi ) = vi
Equivalently F(X) =
P
i∈[n] vi Li (X)
If deg F(X) < N, then F(0) =
P
i∈[N] F(ωi ) [Aurora]
Liam Eagen (Blockstream Research) cqlin June 2, 2023 5 / 17
23. Sparse Commitments
What is a Sparse Commitment?
Liam Eagen (Blockstream Research) cqlin June 2, 2023 6 / 17
24. Sparse Commitments
What is a Sparse Commitment?
Let P(X) be a degree d polynomial
In general, committing to [P(x)] requires d scalar multiplications
Liam Eagen (Blockstream Research) cqlin June 2, 2023 6 / 17
25. Sparse Commitments
What is a Sparse Commitment?
Let P(X) be a degree d polynomial
In general, committing to [P(x)] requires d scalar multiplications
Sometimes, P(X) is sparse with respect to some basis Bi (X) of
n ≤ d polynomials
Liam Eagen (Blockstream Research) cqlin June 2, 2023 6 / 17
26. Sparse Commitments
What is a Sparse Commitment?
Let P(X) be a degree d polynomial
In general, committing to [P(x)] requires d scalar multiplications
Sometimes, P(X) is sparse with respect to some basis Bi (X) of
n ≤ d polynomials
That is P(X) =
P
i∈[n] fi Bi (X) where at most k values of fi are
non-zero and k << d
Liam Eagen (Blockstream Research) cqlin June 2, 2023 6 / 17
27. Sparse Commitments
What is a Sparse Commitment?
Let P(X) be a degree d polynomial
In general, committing to [P(x)] requires d scalar multiplications
Sometimes, P(X) is sparse with respect to some basis Bi (X) of
n ≤ d polynomials
That is P(X) =
P
i∈[n] fi Bi (X) where at most k values of fi are
non-zero and k << d
When P(X) is k, Bi (X) sparse, we can compute a commitment to
P(X) in O(k) time given precomputed commitments to Bi (X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 6 / 17
28. Sparse Commitments
What is a Sparse Commitment?
Let P(X) be a degree d polynomial
In general, committing to [P(x)] requires d scalar multiplications
Sometimes, P(X) is sparse with respect to some basis Bi (X) of
n ≤ d polynomials
That is P(X) =
P
i∈[n] fi Bi (X) where at most k values of fi are
non-zero and k << d
When P(X) is k, Bi (X) sparse, we can compute a commitment to
P(X) in O(k) time given precomputed commitments to Bi (X)
P(X) =
P
i∈[n] ki [Bi (x)]
Liam Eagen (Blockstream Research) cqlin June 2, 2023 6 / 17
29. Sparse Commitments
What is a Sparse Commitment?
Let P(X) be a degree d polynomial
In general, committing to [P(x)] requires d scalar multiplications
Sometimes, P(X) is sparse with respect to some basis Bi (X) of
n ≤ d polynomials
That is P(X) =
P
i∈[n] fi Bi (X) where at most k values of fi are
non-zero and k << d
When P(X) is k, Bi (X) sparse, we can compute a commitment to
P(X) in O(k) time given precomputed commitments to Bi (X)
P(X) =
P
i∈[n] ki [Bi (x)]
Sparsity is linear: if P(X) is Bi (X) sparse, then P(X)F(X) is
Bi (X)F(X) sparse
Liam Eagen (Blockstream Research) cqlin June 2, 2023 6 / 17
30. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
31. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
32. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
1 Suppose want to show deg P(X) = d, but SRS has degree N
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
33. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
1 Suppose want to show deg P(X) = d, but SRS has degree N
2 Precompute Di = [xN−d
Bi (x)]
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
34. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
1 Suppose want to show deg P(X) = d, but SRS has degree N
2 Precompute Di = [xN−d
Bi (x)]
3 Prover can compute D =
P
i∈[n] fi Di in O(k) time
4 Verifier checks e([P(x)], [xN−d
]) = e(D, [1])
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
35. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
1 Suppose want to show deg P(X) = d, but SRS has degree N
2 Precompute Di = [xN−d
Bi (x)]
3 Prover can compute D =
P
i∈[n] fi Di in O(k) time
4 Verifier checks e([P(x)], [xN−d
]) = e(D, [1])
2 Opening at fixed value
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
36. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
1 Suppose want to show deg P(X) = d, but SRS has degree N
2 Precompute Di = [xN−d
Bi (x)]
3 Prover can compute D =
P
i∈[n] fi Di in O(k) time
4 Verifier checks e([P(x)], [xN−d
]) = e(D, [1])
2 Opening at fixed value
1 Suppose prover wants to open P(0) = c, for example as part of
sum-check
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
37. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
1 Suppose want to show deg P(X) = d, but SRS has degree N
2 Precompute Di = [xN−d
Bi (x)]
3 Prover can compute D =
P
i∈[n] fi Di in O(k) time
4 Verifier checks e([P(x)], [xN−d
]) = e(D, [1])
2 Opening at fixed value
1 Suppose prover wants to open P(0) = c, for example as part of
sum-check
2 Precompute Zi = [Bi (x) − Bi (0)/x]
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
38. Sparse Commitments: Simple Examples
Assume P(X) is Bi (X) sparse
1 Degree Checks
1 Suppose want to show deg P(X) = d, but SRS has degree N
2 Precompute Di = [xN−d
Bi (x)]
3 Prover can compute D =
P
i∈[n] fi Di in O(k) time
4 Verifier checks e([P(x)], [xN−d
]) = e(D, [1])
2 Opening at fixed value
1 Suppose prover wants to open P(0) = c, for example as part of
sum-check
2 Precompute Zi = [Bi (x) − Bi (0)/x]
3 Prover can compute Z =
P
i∈[n] fi Zi in O(k) time
4 Verifier checks e([P(x)] − c[1], [1]) = e(Z, [x])
Liam Eagen (Blockstream Research) cqlin June 2, 2023 7 / 17
39. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
40. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Given T(X) encoding a table ⃗
t ∈ FN, want to prove T(ωi ) = v for
some i
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
41. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Given T(X) encoding a table ⃗
t ∈ FN, want to prove T(ωi ) = v for
some i
Let A(X) be n Lagrange sparse and B(X) and Q(X) satisfy
T(X)A(X) = B(X) + (XN − 1)Q(X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
42. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Given T(X) encoding a table ⃗
t ∈ FN, want to prove T(ωi ) = v for
some i
Let A(X) be n Lagrange sparse and B(X) and Q(X) satisfy
T(X)A(X) = B(X) + (XN − 1)Q(X)
T(ωi )A(ωi ) = B(ωi ), so B(X) is also n Lagrange sparse
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
43. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Given T(X) encoding a table ⃗
t ∈ FN, want to prove T(ωi ) = v for
some i
Let A(X) be n Lagrange sparse and B(X) and Q(X) satisfy
T(X)A(X) = B(X) + (XN − 1)Q(X)
T(ωi )A(ωi ) = B(ωi ), so B(X) is also n Lagrange sparse
Q(X) is n sparse wrt to the basis Qi (X) such that
T(X)Li (X) = ti Li (X) + (XN − 1)Qi (X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
44. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Given T(X) encoding a table ⃗
t ∈ FN, want to prove T(ωi ) = v for
some i
Let A(X) be n Lagrange sparse and B(X) and Q(X) satisfy
T(X)A(X) = B(X) + (XN − 1)Q(X)
T(ωi )A(ωi ) = B(ωi ), so B(X) is also n Lagrange sparse
Q(X) is n sparse wrt to the basis Qi (X) such that
T(X)Li (X) = ti Li (X) + (XN − 1)Qi (X)
Can compute all [Qi (x)] in O(N log N) time using FK technique
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
45. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Given T(X) encoding a table ⃗
t ∈ FN, want to prove T(ωi ) = v for
some i
Let A(X) be n Lagrange sparse and B(X) and Q(X) satisfy
T(X)A(X) = B(X) + (XN − 1)Q(X)
T(ωi )A(ωi ) = B(ωi ), so B(X) is also n Lagrange sparse
Q(X) is n sparse wrt to the basis Qi (X) such that
T(X)Li (X) = ti Li (X) + (XN − 1)Qi (X)
Can compute all [Qi (x)] in O(N log N) time using FK technique
Can commit to [A(x)], [B(x)], and [Q(x)] in O(n) scalar
multiplications
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
46. Sparse Commitments: cq and Lookup Arguments
Sparse commitments are a key part of lookup arguments [Caulk,etc.]
especially cq
Given T(X) encoding a table ⃗
t ∈ FN, want to prove T(ωi ) = v for
some i
Let A(X) be n Lagrange sparse and B(X) and Q(X) satisfy
T(X)A(X) = B(X) + (XN − 1)Q(X)
T(ωi )A(ωi ) = B(ωi ), so B(X) is also n Lagrange sparse
Q(X) is n sparse wrt to the basis Qi (X) such that
T(X)Li (X) = ti Li (X) + (XN − 1)Qi (X)
Can compute all [Qi (x)] in O(N log N) time using FK technique
Can commit to [A(x)], [B(x)], and [Q(x)] in O(n) scalar
multiplications
cq defines A(X) and B(X) to encode log derivative lookup [BP++,
MVLookup]
Liam Eagen (Blockstream Research) cqlin June 2, 2023 8 / 17
47. Lin-Check
Fix n × n matrix M
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
48. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
49. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
50. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
How?
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
51. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
How?
1 State M⃗
a = ⃗
b in terms of (bivariate) polynomials
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
52. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
How?
1 State M⃗
a = ⃗
b in terms of (bivariate) polynomials
2 Use a pairing to perform M⃗
a as polynomial multiplication
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
53. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
How?
1 State M⃗
a = ⃗
b in terms of (bivariate) polynomials
2 Use a pairing to perform M⃗
a as polynomial multiplication
3 Exploit the fact M is fixed and precompute quotients
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
54. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
How?
1 State M⃗
a = ⃗
b in terms of (bivariate) polynomials
2 Use a pairing to perform M⃗
a as polynomial multiplication
3 Exploit the fact M is fixed and precompute quotients
4 Find a sparse basis for all commitments
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
55. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
How?
1 State M⃗
a = ⃗
b in terms of (bivariate) polynomials
2 Use a pairing to perform M⃗
a as polynomial multiplication
3 Exploit the fact M is fixed and precompute quotients
4 Find a sparse basis for all commitments
5 Commit to everything in O(n) group scalar multiplications
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
56. Lin-Check
Fix n × n matrix M
Given commitments to vectors ⃗
a and ⃗
b show M⃗
a = ⃗
b
Goal: do this in O(n) group and scalar operations
How?
1 State M⃗
a = ⃗
b in terms of (bivariate) polynomials
2 Use a pairing to perform M⃗
a as polynomial multiplication
3 Exploit the fact M is fixed and precompute quotients
4 Find a sparse basis for all commitments
5 Commit to everything in O(n) group scalar multiplications
6 Reduce bivariate form to univariate form
Liam Eagen (Blockstream Research) cqlin June 2, 2023 9 / 17
57. Encode into Polynomials
Recall: M⃗
a =
P
i∈[n] ai ⃗
ci where ⃗
ci the columns of M
Liam Eagen (Blockstream Research) cqlin June 2, 2023 10 / 17
58. Encode into Polynomials
Recall: M⃗
a =
P
i∈[n] ai ⃗
ci where ⃗
ci the columns of M
Encode ⃗
b and ⃗
ci pointwise as polynomials B(X) and Ci (X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 10 / 17
59. Encode into Polynomials
Recall: M⃗
a =
P
i∈[n] ai ⃗
ci where ⃗
ci the columns of M
Encode ⃗
b and ⃗
ci pointwise as polynomials B(X) and Ci (X)
Now: M⃗
a = ⃗
b iff
P
i∈[n] ai Ci (X) = B(X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 10 / 17
60. Encode into Polynomials
Recall: M⃗
a =
P
i∈[n] ai ⃗
ci where ⃗
ci the columns of M
Encode ⃗
b and ⃗
ci pointwise as polynomials B(X) and Ci (X)
Now: M⃗
a = ⃗
b iff
P
i∈[n] ai Ci (X) = B(X)
M is a vector of vectors, so we can encode into bivariate M(X, Y )
such that M(X, ωi ) = Ci (X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 10 / 17
61. Encode into Polynomials
Recall: M⃗
a =
P
i∈[n] ai ⃗
ci where ⃗
ci the columns of M
Encode ⃗
b and ⃗
ci pointwise as polynomials B(X) and Ci (X)
Now: M⃗
a = ⃗
b iff
P
i∈[n] ai Ci (X) = B(X)
M is a vector of vectors, so we can encode into bivariate M(X, Y )
such that M(X, ωi ) = Ci (X)
Encode ⃗
a as polynomial A(Y )
Liam Eagen (Blockstream Research) cqlin June 2, 2023 10 / 17
62. Encode into Polynomials
Recall: M⃗
a =
P
i∈[n] ai ⃗
ci where ⃗
ci the columns of M
Encode ⃗
b and ⃗
ci pointwise as polynomials B(X) and Ci (X)
Now: M⃗
a = ⃗
b iff
P
i∈[n] ai Ci (X) = B(X)
M is a vector of vectors, so we can encode into bivariate M(X, Y )
such that M(X, ωi ) = Ci (X)
Encode ⃗
a as polynomial A(Y )
Now: M⃗
a = ⃗
b iff
P
i∈[n] A(ωi )M(X, ωi ) = B(X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 10 / 17
63. Encode into Polynomials (cont.)
Let R(X, ωi ) = M(X, ωi )A(ωi ) = ai Ci (X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 11 / 17
64. Encode into Polynomials (cont.)
Let R(X, ωi ) = M(X, ωi )A(ωi ) = ai Ci (X)
Which means R(X, Y ) = M(X, Y )A(Y ) mod Y n − 1
Liam Eagen (Blockstream Research) cqlin June 2, 2023 11 / 17
65. Encode into Polynomials (cont.)
Let R(X, ωi ) = M(X, ωi )A(ωi ) = ai Ci (X)
Which means R(X, Y ) = M(X, Y )A(Y ) mod Y n − 1
Which means R(X, Y ) = M(X, Y )A(Y ) + (Y n − 1)Q(X, Y )
Liam Eagen (Blockstream Research) cqlin June 2, 2023 11 / 17
66. Encode into Polynomials (cont.)
Let R(X, ωi ) = M(X, ωi )A(ωi ) = ai Ci (X)
Which means R(X, Y ) = M(X, Y )A(Y ) mod Y n − 1
Which means R(X, Y ) = M(X, Y )A(Y ) + (Y n − 1)Q(X, Y )
Now: M⃗
a = ⃗
b iff
1 R(X, Y ) = M(X, Y )A(Y ) + (Y n
− 1)Q(X, Y )
2
P
i∈[n] R(X, ωi
) = B(X)
3 degY R(X, Y ) < n, etc.
Liam Eagen (Blockstream Research) cqlin June 2, 2023 11 / 17
67. Encode into Polynomials (cont.)
Let R(X, ωi ) = M(X, ωi )A(ωi ) = ai Ci (X)
Which means R(X, Y ) = M(X, Y )A(Y ) mod Y n − 1
Which means R(X, Y ) = M(X, Y )A(Y ) + (Y n − 1)Q(X, Y )
Now: M⃗
a = ⃗
b iff
1 R(X, Y ) = M(X, Y )A(Y ) + (Y n
− 1)Q(X, Y )
2
P
i∈[n] R(X, ωi
) = B(X)
3 degY R(X, Y ) < n, etc.
Sum-check equivalent to R(X, 0) = B(X)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 11 / 17
68. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
69. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
70. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Commitments to A(Y ) and B(X) are easy given [Li (x)] and [Li (y)]
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
71. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Commitments to A(Y ) and B(X) are easy given [Li (x)] and [Li (y)]
Degree checks are also easy
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
72. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Commitments to A(Y ) and B(X) are easy given [Li (x)] and [Li (y)]
Degree checks are also easy
Can compute [R(x, y)] =
P
i∈[n] ai [Li (y)Ci (x)] in O(n) scalar
multiplications given precomputed [Li (y)Ci (x)]
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
73. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Commitments to A(Y ) and B(X) are easy given [Li (x)] and [Li (y)]
Degree checks are also easy
Can compute [R(x, y)] =
P
i∈[n] ai [Li (y)Ci (x)] in O(n) scalar
multiplications given precomputed [Li (y)Ci (x)]
The sum check can be computed in O(n) time using KZG openings
of Li (Y )Ci (X) at Y = 0
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
74. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Commitments to A(Y ) and B(X) are easy given [Li (x)] and [Li (y)]
Degree checks are also easy
Can compute [R(x, y)] =
P
i∈[n] ai [Li (y)Ci (x)] in O(n) scalar
multiplications given precomputed [Li (y)Ci (x)]
The sum check can be computed in O(n) time using KZG openings
of Li (Y )Ci (X) at Y = 0
How to compute Q(X, Y )? Use the cq technique
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
75. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Commitments to A(Y ) and B(X) are easy given [Li (x)] and [Li (y)]
Degree checks are also easy
Can compute [R(x, y)] =
P
i∈[n] ai [Li (y)Ci (x)] in O(n) scalar
multiplications given precomputed [Li (y)Ci (x)]
The sum check can be computed in O(n) time using KZG openings
of Li (Y )Ci (X) at Y = 0
How to compute Q(X, Y )? Use the cq technique
Define Qi (X, Y ) such that
Ci (X)Li (Y ) = M(X, Y )Li (Y ) + (Y n − 1)Qi (X, Y )
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
76. Proving in Linear Time
Want to construct [A(y)], [B(x)], [R(x, y)], and [Q(x, y)] in O(n)
time
Need to find basis for each that is n sparse
Commitments to A(Y ) and B(X) are easy given [Li (x)] and [Li (y)]
Degree checks are also easy
Can compute [R(x, y)] =
P
i∈[n] ai [Li (y)Ci (x)] in O(n) scalar
multiplications given precomputed [Li (y)Ci (x)]
The sum check can be computed in O(n) time using KZG openings
of Li (Y )Ci (X) at Y = 0
How to compute Q(X, Y )? Use the cq technique
Define Qi (X, Y ) such that
Ci (X)Li (Y ) = M(X, Y )Li (Y ) + (Y n − 1)Qi (X, Y )
Now [Q(X, Y )] =
P
i∈[n] ai [Qi (x, y)] can be computed in O(n) time
Liam Eagen (Blockstream Research) cqlin June 2, 2023 12 / 17
77. Final touches
Prefer to use a univariate SRS
Liam Eagen (Blockstream Research) cqlin June 2, 2023 13 / 17
78. Final touches
Prefer to use a univariate SRS
Can make the substitution Y = Xn
Liam Eagen (Blockstream Research) cqlin June 2, 2023 13 / 17
79. Final touches
Prefer to use a univariate SRS
Can make the substitution Y = Xn
Prefer to accept the input vector via F(X) that pointwise encodes ⃗
a
rather than A(Y )
Liam Eagen (Blockstream Research) cqlin June 2, 2023 13 / 17
80. Final touches
Prefer to use a univariate SRS
Can make the substitution Y = Xn
Prefer to accept the input vector via F(X) that pointwise encodes ⃗
a
rather than A(Y )
Can check A(α) = F(αn) at a random point in O(n) time
Liam Eagen (Blockstream Research) cqlin June 2, 2023 13 / 17
81. Final touches
Prefer to use a univariate SRS
Can make the substitution Y = Xn
Prefer to accept the input vector via F(X) that pointwise encodes ⃗
a
rather than A(Y )
Can check A(α) = F(αn) at a random point in O(n) time
That’s it! Given O(n) precomputed commitments can prove lin-check
in O(n) time
Liam Eagen (Blockstream Research) cqlin June 2, 2023 13 / 17
83. Future Work
Improvements to cqlin
▶ Generalize to other vanishing sets
Liam Eagen (Blockstream Research) cqlin June 2, 2023 14 / 17
84. Future Work
Improvements to cqlin
▶ Generalize to other vanishing sets
▶ Reduce prover work and/or proof size
Liam Eagen (Blockstream Research) cqlin June 2, 2023 14 / 17
85. Future Work
Improvements to cqlin
▶ Generalize to other vanishing sets
▶ Reduce prover work and/or proof size
▶ Improve setup time and SRS size for sparse matrices
Liam Eagen (Blockstream Research) cqlin June 2, 2023 14 / 17
86. Future Work
Improvements to cqlin
▶ Generalize to other vanishing sets
▶ Reduce prover work and/or proof size
▶ Improve setup time and SRS size for sparse matrices
Other applications of precomputation and pairings?
Liam Eagen (Blockstream Research) cqlin June 2, 2023 14 / 17
88. Precomputation
Most of the precomputation straightforward
1 Degree checks very simple
2 Openings at zero for Li (X) follow from symmetry
Complicated part: computing Qi (X, Y ) in O(n2) group operations
Solution: use the FK technique on the rows of M(X, Y ) and sum the
results
FK technique
1 Want to compute KZG openings at N roots of unity
2 Write vector of KZG openings as a 2N × 2N circulant matrix times
vector of [xi
]
3 Circulant matrix diagonalizes as FDF−1
where F is the DFT matrix
4 This product computable in
O(N log N) + O(N) + O(N log N) = O(N log N) time
Takes O(n log n) per row, for a total of O(n2 log n)
Liam Eagen (Blockstream Research) cqlin June 2, 2023 16 / 17
89. Precomputation (cont.)
Naive FK takes O(n2 log n) group and field operations
It is possible to exploit the structure of FK to avoid the extra log n
1 Precompute F−1
times vector of [xi
]
2 Compute D multiplication in O(n)
3 The final F multiplication is linear, so first add the results and then do
a single F multiplication
4 Takes O(n2
) + O(n log n) = O(n2
) time.
Liam Eagen (Blockstream Research) cqlin June 2, 2023 17 / 17