The document discusses PostgreSQL extensions for indexing and querying semi-structured data like JSON. It introduces hstore, an existing PostgreSQL extension for storing key-value pairs, and notes its limitations compared to JSON. It then summarizes upcoming talks on supporting JSON natively in PostgreSQL, including indexing JSON with GIN and GIST indexes, a JSON query language called Jsquery, and a new indexing access method called VODKA. Exercises are also planned for working with JSON GIN indexes and Jsquery.
2015-12-05 Александр Коротков, Иван Панченко - Слабо-структурированные данные...HappyDev
Появление большого количества NoSQL СУБД обусловлено требованиями современных информационных систем, которым большинство традиционных реляционных баз данных не удовлетворяет. Одним из таких требований является поддержка данных, структура которых заранее не определена. Однако при выборе NoSQL БД ради отсутствия схем данных можно потерять ряд преимуществ, которые дают зрелые SQL-решения, а именно: транзакции, скорость чтения строк из таблиц. PostgreSQL, являющаяся передовой реляционной СУБД, имела поддержку слабо-структурированных данных задолго до появления NoSQL, которая обрела новое дыхание в последнем релизе в виде типа данных jsonb, который не только поддерживает стандарт JSON, но и обладает производительностью, сравнимой или даже превосходящей наиболее популярные NoSQL СУБД.
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesJonathan Katz
All data is relational and can be represented through relational algebra, right? Perhaps, but there are other ways to represent data, and the PostgreSQL team continues to work on making it easier and more efficient to do so!
With the upcoming 9.4 release, PostgreSQL is introducing the "JSONB" data type which allows for fast, compressed, storage of JSON formatted data, and for quick retrieval. And JSONB comes with all the benefits of PostgreSQL, like its data durability, MVCC, and of course, access to all the other data types and features in PostgreSQL.
How fast is JSONB? How do we access data stored with this type? What can it do with the rest of PostgreSQL? What can't it do? How can we leverage this new data type and make PostgreSQL scale horizontally? Follow along with our presentation as we try to answer these questions.
What's the great thing about a database? Why, it stores data of course! However, one feature that makes a database useful is the different data types that can be stored in it, and the breadth and sophistication of the data types in PostgreSQL is second-to-none, including some novel data types that do not exist in any other database software!
This talk will take an in-depth look at the special data types built right into PostgreSQL version 9.4, including:
* INET types
* UUIDs
* Geometries
* Arrays
* Ranges
* Document-based Data Types:
* Key-value store (hstore)
* JSON (text [JSON] & binary [JSONB])
We will also have some cleverly concocted examples to show how all of these data types can work together harmoniously.
PG Day'14 Russia, Работа со слабо-структурированными данными в PostgreSQL, Ол...pgdayrussia
Доклад был представлен на официальной российской конференции PG Day'14 Russia, посвященной вопросам разработки и эксплуатации PostgreSQL.
<сарказм> MongoDB правит бал в мире слабо-структурированных данных. Привлеченные в MongoDB инвестиции часто затмевают разум (особенно начинающих и доверчивых) разработчиков, которые с радостью бросаются в океан возможностей, предоставляемых NoSQL (это же круто!). Энтузиазм затихает после осознания того факта, что бесплатно ничего не бывает и надо писать своими руками то, что десятилетиями хорошо работает в традиционных реляционных базах данных, которые прекрасно справляются с нагрузками и данными 99% проектов, и ваш проект не входит в оставшийся один процент. </сарказм>
Мир баз данных за последние годы существенно изменился. Всеместное проникновение Интернет-технологий привело к необходимости работы с большим количеством разнородных данных в реальном времени, к чему традиционные реляционные СУБД оказались не готовы. Принято считать, что слабая масштабируемость и излишняя “жесткость” модели данных реляционных СУБД и являются основными причинами появляния и роста популярности NoSQL баз данных (далее, NoSQL).
В докладе мы остановимся на концептуальных предпосылках появления NoSQL и их классификации. Одним из “жупелов” NoSQL является поддержка типа данных JSON, который реализует документо-ориентированную модель данных. Документо-ориентированная модель данных является более гибкой и позволяет менять схему данных “на лету”, что сделать очень трудно в реляционных СУБД, особенно в системах, работающих под большой нагрузкой. Несмотря на успех NoSQL (активно распиаренный использованием в некоторых популярных Интернет-проектах), многие пользователи не готовы приносить в жертву целостность данных в угоду масштабируемости, но хотят иметь гибкость схемы данных в проверенных и надежных реляционных СУБД.
Нами была предложена и реализована поддержка документо-ориентированной модели в PostgreSQL (версия 9.4). Уже более 10 лет в PostgreSQL существует возможность работать со schema-less данными, используя наш модуль расширения hstore. Hstore предлагает хранилище вида "ключ-значение" с сохранением всех реляционных возможностей, что сделало его самым используемым
Developing and Deploying Apps with the Postgres FDWJonathan Katz
I couldn't wait to use the Postgres Foreign Data Wrapper (postgres_fdw) in a project; imagine being able to read and write data to many databases all from a single database! I finally found a project where it made sense to use this amazing technology.
I mapped out my architecture and began to code, and realized there were some things that did not work as expected: I could not call remote functions or insert into a table with a serial primary key and have it autoupdate. I found workarounds (which I will share), so the project went on.
We tested the setup, everything seemed to work well, and then we went to deploy to production. And then the real fun began.
Despite the title, I still love the Postgres FDW but wanted to provide some cautionary tales from a hybrid developer/DBA perspective on how to properly use them in your working environment. This talk will cover:
* Basic Postgres FDW setup in a development environment vs. production environment
* Handling some common FDW uses case that you think are trivial but are not
* Working with advanced Postgres constructs such as schemas and sequences with FDWs
* Putting it all together to make sure your production application is safe with your FDWs
* ...and when you really, really need to make a remote call and it is not supported by a FDW, how to do that too!
2015-12-05 Александр Коротков, Иван Панченко - Слабо-структурированные данные...HappyDev
Появление большого количества NoSQL СУБД обусловлено требованиями современных информационных систем, которым большинство традиционных реляционных баз данных не удовлетворяет. Одним из таких требований является поддержка данных, структура которых заранее не определена. Однако при выборе NoSQL БД ради отсутствия схем данных можно потерять ряд преимуществ, которые дают зрелые SQL-решения, а именно: транзакции, скорость чтения строк из таблиц. PostgreSQL, являющаяся передовой реляционной СУБД, имела поддержку слабо-структурированных данных задолго до появления NoSQL, которая обрела новое дыхание в последнем релизе в виде типа данных jsonb, который не только поддерживает стандарт JSON, но и обладает производительностью, сравнимой или даже превосходящей наиболее популярные NoSQL СУБД.
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesJonathan Katz
All data is relational and can be represented through relational algebra, right? Perhaps, but there are other ways to represent data, and the PostgreSQL team continues to work on making it easier and more efficient to do so!
With the upcoming 9.4 release, PostgreSQL is introducing the "JSONB" data type which allows for fast, compressed, storage of JSON formatted data, and for quick retrieval. And JSONB comes with all the benefits of PostgreSQL, like its data durability, MVCC, and of course, access to all the other data types and features in PostgreSQL.
How fast is JSONB? How do we access data stored with this type? What can it do with the rest of PostgreSQL? What can't it do? How can we leverage this new data type and make PostgreSQL scale horizontally? Follow along with our presentation as we try to answer these questions.
What's the great thing about a database? Why, it stores data of course! However, one feature that makes a database useful is the different data types that can be stored in it, and the breadth and sophistication of the data types in PostgreSQL is second-to-none, including some novel data types that do not exist in any other database software!
This talk will take an in-depth look at the special data types built right into PostgreSQL version 9.4, including:
* INET types
* UUIDs
* Geometries
* Arrays
* Ranges
* Document-based Data Types:
* Key-value store (hstore)
* JSON (text [JSON] & binary [JSONB])
We will also have some cleverly concocted examples to show how all of these data types can work together harmoniously.
PG Day'14 Russia, Работа со слабо-структурированными данными в PostgreSQL, Ол...pgdayrussia
Доклад был представлен на официальной российской конференции PG Day'14 Russia, посвященной вопросам разработки и эксплуатации PostgreSQL.
<сарказм> MongoDB правит бал в мире слабо-структурированных данных. Привлеченные в MongoDB инвестиции часто затмевают разум (особенно начинающих и доверчивых) разработчиков, которые с радостью бросаются в океан возможностей, предоставляемых NoSQL (это же круто!). Энтузиазм затихает после осознания того факта, что бесплатно ничего не бывает и надо писать своими руками то, что десятилетиями хорошо работает в традиционных реляционных базах данных, которые прекрасно справляются с нагрузками и данными 99% проектов, и ваш проект не входит в оставшийся один процент. </сарказм>
Мир баз данных за последние годы существенно изменился. Всеместное проникновение Интернет-технологий привело к необходимости работы с большим количеством разнородных данных в реальном времени, к чему традиционные реляционные СУБД оказались не готовы. Принято считать, что слабая масштабируемость и излишняя “жесткость” модели данных реляционных СУБД и являются основными причинами появляния и роста популярности NoSQL баз данных (далее, NoSQL).
В докладе мы остановимся на концептуальных предпосылках появления NoSQL и их классификации. Одним из “жупелов” NoSQL является поддержка типа данных JSON, который реализует документо-ориентированную модель данных. Документо-ориентированная модель данных является более гибкой и позволяет менять схему данных “на лету”, что сделать очень трудно в реляционных СУБД, особенно в системах, работающих под большой нагрузкой. Несмотря на успех NoSQL (активно распиаренный использованием в некоторых популярных Интернет-проектах), многие пользователи не готовы приносить в жертву целостность данных в угоду масштабируемости, но хотят иметь гибкость схемы данных в проверенных и надежных реляционных СУБД.
Нами была предложена и реализована поддержка документо-ориентированной модели в PostgreSQL (версия 9.4). Уже более 10 лет в PostgreSQL существует возможность работать со schema-less данными, используя наш модуль расширения hstore. Hstore предлагает хранилище вида "ключ-значение" с сохранением всех реляционных возможностей, что сделало его самым используемым
Developing and Deploying Apps with the Postgres FDWJonathan Katz
I couldn't wait to use the Postgres Foreign Data Wrapper (postgres_fdw) in a project; imagine being able to read and write data to many databases all from a single database! I finally found a project where it made sense to use this amazing technology.
I mapped out my architecture and began to code, and realized there were some things that did not work as expected: I could not call remote functions or insert into a table with a serial primary key and have it autoupdate. I found workarounds (which I will share), so the project went on.
We tested the setup, everything seemed to work well, and then we went to deploy to production. And then the real fun began.
Despite the title, I still love the Postgres FDW but wanted to provide some cautionary tales from a hybrid developer/DBA perspective on how to properly use them in your working environment. This talk will cover:
* Basic Postgres FDW setup in a development environment vs. production environment
* Handling some common FDW uses case that you think are trivial but are not
* Working with advanced Postgres constructs such as schemas and sequences with FDWs
* Putting it all together to make sure your production application is safe with your FDWs
* ...and when you really, really need to make a remote call and it is not supported by a FDW, how to do that too!
The presentation from PiterPy Meetup #10 Hardcore about the data structures used in databases for storing and retrieving data.
Two approaches to data processing are considered: OLTP and OLAP.
SQL, NoSQL and New SQL databases are discussed.
The tradeoffs that the developers face when creating storage systems are shown.
Also the methods of data storage and interaction with the database provides CPython are considered.
The presentation and the list of references and books helps more easily navigate the data storage engines and understand which tool is better suited for a particular task.
Bucket your partitions wisely - Cassandra summit 2016Markus Höfer
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
There are number of players that provide full text search feature, starting from embedded search to dedicated search servers [solr, sphinx, elasticsearch etc], but setting up and configuring them is a time consuming process and requires considerable knowledge of the tools.
What if we could get comparable search results using full text search capabilities of Postgres. Developers already have the working knowledge of the database, so this should come natural. In addition to that, it will be one less tool to manage.
Code: https://github.com/Syerram/postgres_search
NoSQL Analytics: JSON Data Analysis and Acceleration in MongoDB WorldAjay Gupte
In analytics world, when you need to process many millions or billions of documents to generate a single report. Novel techniques have been developed for exploiting modern processor architecture (larger on-chip cache, SIMD processing, compression, vector processing, columnar approach). Now, this technology is available to process your large JSON data. This talk will discuss analysis of JSON data using advanced data warehousing techniques and make it simple and seamless for the application/tool developer.
In these slide following projects are presented:
* redis_wrap: A Pythonic wrapper that makes it nicer to work with builtin Redis datatypes
* redis_graph: A sample graph database
* redis_simple_queue: A simple queue implemented on Redis list structure
* bitmapist: a powerful analytics library using Redis bitmaps, great for retention and cohort tracking
* fixedlist: a fixed list structure that can optimize timelines (and other things)
* how to use Lua scripting for more advanced data structures and better performance
These are Python projects, but some of them (like bitmapist) have been ported to other languages.
This talk was given to PyCon Belarus on 31 Jan. 2015
This presentation covers some common terminology used to describe NoSQL databases, goes into depth on some popular scalable database architectures, and includes an overview of Hypertable
PostgreSQL has kept up the momentum around JSON with version 9.4 featuring JSONB as demand for working with unstructured data continues to grow. In this talk delivered during Postgres Open 2014, Vibhor Kumar, principal systems engineer at EnterpriseDB, offered some scenarios for working with JSON in PostgreSQL and demonstrated performance metrics. This session also gave some instruction on how to use different operations and explored comparisons to BSON.
Володимир Гоцик. Getting maximum of python, django with postgres 9.4. PyCon B...Alina Dolgikh
Postgres предоставляет много встроенных возможностей для создания эфективных приложений, использующих базы данных. А в версии 9.4 появляется еще и полноценное JSON поле, при правильном использовании которого, отпадает необходимость использвания NoSQL баз данных. В докладе мы рассмотрим, как использовать этот потенциал по максимуму в своих Python/Django приложениях.
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
About the Speaker
Markus Hofer IT Consultant, codecentric AG
Markus Hofer works as an IT Consultant for codecentric AG in Minster, Germany. He works on microservice architectures backed by DSE and/or Apache Cassandra. Markus supports and trains customers building cassandra based applications.
PostgreSQL comes built-in with a variety of indexes, some of which are further extensible to build powerful new indexing schemes. But what are all these index types? What are some of the special features of these indexes? What are the size & performance tradeoffs? How do I know which ones are appropriate for my application?
Fortunately, this talk aims to answer all of these questions as we explore the whole family of PostgreSQL indexes: B-tree, expression, GiST (of all flavors), GIN and how they are used in theory and practice.
The presentation from PiterPy Meetup #10 Hardcore about the data structures used in databases for storing and retrieving data.
Two approaches to data processing are considered: OLTP and OLAP.
SQL, NoSQL and New SQL databases are discussed.
The tradeoffs that the developers face when creating storage systems are shown.
Also the methods of data storage and interaction with the database provides CPython are considered.
The presentation and the list of references and books helps more easily navigate the data storage engines and understand which tool is better suited for a particular task.
Bucket your partitions wisely - Cassandra summit 2016Markus Höfer
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
There are number of players that provide full text search feature, starting from embedded search to dedicated search servers [solr, sphinx, elasticsearch etc], but setting up and configuring them is a time consuming process and requires considerable knowledge of the tools.
What if we could get comparable search results using full text search capabilities of Postgres. Developers already have the working knowledge of the database, so this should come natural. In addition to that, it will be one less tool to manage.
Code: https://github.com/Syerram/postgres_search
NoSQL Analytics: JSON Data Analysis and Acceleration in MongoDB WorldAjay Gupte
In analytics world, when you need to process many millions or billions of documents to generate a single report. Novel techniques have been developed for exploiting modern processor architecture (larger on-chip cache, SIMD processing, compression, vector processing, columnar approach). Now, this technology is available to process your large JSON data. This talk will discuss analysis of JSON data using advanced data warehousing techniques and make it simple and seamless for the application/tool developer.
In these slide following projects are presented:
* redis_wrap: A Pythonic wrapper that makes it nicer to work with builtin Redis datatypes
* redis_graph: A sample graph database
* redis_simple_queue: A simple queue implemented on Redis list structure
* bitmapist: a powerful analytics library using Redis bitmaps, great for retention and cohort tracking
* fixedlist: a fixed list structure that can optimize timelines (and other things)
* how to use Lua scripting for more advanced data structures and better performance
These are Python projects, but some of them (like bitmapist) have been ported to other languages.
This talk was given to PyCon Belarus on 31 Jan. 2015
This presentation covers some common terminology used to describe NoSQL databases, goes into depth on some popular scalable database architectures, and includes an overview of Hypertable
PostgreSQL has kept up the momentum around JSON with version 9.4 featuring JSONB as demand for working with unstructured data continues to grow. In this talk delivered during Postgres Open 2014, Vibhor Kumar, principal systems engineer at EnterpriseDB, offered some scenarios for working with JSON in PostgreSQL and demonstrated performance metrics. This session also gave some instruction on how to use different operations and explored comparisons to BSON.
Володимир Гоцик. Getting maximum of python, django with postgres 9.4. PyCon B...Alina Dolgikh
Postgres предоставляет много встроенных возможностей для создания эфективных приложений, использующих базы данных. А в версии 9.4 появляется еще и полноценное JSON поле, при правильном использовании которого, отпадает необходимость использвания NoSQL баз данных. В докладе мы рассмотрим, как использовать этот потенциал по максимуму в своих Python/Django приложениях.
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
About the Speaker
Markus Hofer IT Consultant, codecentric AG
Markus Hofer works as an IT Consultant for codecentric AG in Minster, Germany. He works on microservice architectures backed by DSE and/or Apache Cassandra. Markus supports and trains customers building cassandra based applications.
PostgreSQL comes built-in with a variety of indexes, some of which are further extensible to build powerful new indexing schemes. But what are all these index types? What are some of the special features of these indexes? What are the size & performance tradeoffs? How do I know which ones are appropriate for my application?
Fortunately, this talk aims to answer all of these questions as we explore the whole family of PostgreSQL indexes: B-tree, expression, GiST (of all flavors), GIN and how they are used in theory and practice.
MariaDB Server 10.3 is a culmination of features from MariaDB Server 10.2+10.1+10.0+5.5+5.3+5.2+5.1 as well as a base branch from MySQL 5.5 and backports from MySQL 5.6/5.7. It has many new features, like a GA-ready sharding engine (SPIDER), MyRocks, as well as some Oracle compatibility, system versioned tables and a whole lot more.
Postgres vs Mongo / Олег Бартунов (Postgres Professional)Ontico
РИТ++ 2017, Backend Conf
Зал Конгресс-холл, 6 июня, 17:00
Тезисы:
http://backendconf.ru/2017/abstracts/2781.html
Я хочу немного порушить стереотипы, что Postgres - это чисто реляционная СУБД из прошлого века, плохо приспособленная под реалии современных проектов. Недавно мы прогнали YCSB для последних версий Postgres и Mongodb и увидели их плюсы и минусы на разных типах нагрузки, о которых я буду рассказывать. ...
The latest version of my PostgreSQL introduction for IL-TechTalks, a free service to introduce the Israeli hi-tech community to new and interesting technologies. In this talk, I describe the history and licensing of PostgreSQL, its built-in capabilities, and some of the new things that were added in the 9.1 and 9.2 releases which make it an attractive option for many applications.
This is a talk that I gave on July 20, 2012 at the Southern California Python Interest Group meetup at Cross Campus, with food and drinks provided by Graph Effect.
Beyond Postgres: Interesting Projects, Tools and forksSameer Kumar
This ppt was used at Aug Meetup of Postgres User Group Singapore. We talked about
• Cool tools and extensions like PostGIS
• Great projects like pgpool and pgbouncer
• Interesting forks of PostgreSQL like EnterpriseDB, GreenPlum etc
Interesting take-away from the session was -
• Ease Oracle Migration
• Load Balancing in PostgreSQL
• Spatial data in PostgreSQL
• Connection pooling and resource management
• Your next Data Warehouse project
Meetup page- http://www.meetup.com/PUGS-Postgres-Users-Group-Singapore/
Hadoop Summit 2014: Query Optimization and JIT-based Vectorized Execution in ...Gruter
Apache Tajo is an open source big data warehouse system on Hadoop. This slide shows two high-tech efforts for performance improvement in Tajo project. First one is query optimization including cost-based join order and progressive optimization. The second effort is JIT-based vectorized processing.
Probabilistic Data Structures (Edmonton Data Science Meetup, March 2018)Kyle Davis
Let's explore how Redis (and Redis Enterprise) can be used to store data in not only deterministic structures but also probabilistic structures like Bloom filters, HyperLogLog, Count Min Sketch and Cuckoo filters. We examine both usage and briefly summarize the algorithms that back these structures. Also we review the use-cases and applications for probabilistic structures.
Modern design is crucial in today's digital environment, and this is especially true for SharePoint intranets. The design of these digital hubs is critical to user engagement and productivity enhancement. They are the cornerstone of internal collaboration and interaction within enterprises.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Why React Native as a Strategic Advantage for Startup Innovation.pdfayushiqss
Do you know that React Native is being increasingly adopted by startups as well as big companies in the mobile app development industry? Big names like Facebook, Instagram, and Pinterest have already integrated this robust open-source framework.
In fact, according to a report by Statista, the number of React Native developers has been steadily increasing over the years, reaching an estimated 1.9 million by the end of 2024. This means that the demand for this framework in the job market has been growing making it a valuable skill.
But what makes React Native so popular for mobile application development? It offers excellent cross-platform capabilities among other benefits. This way, with React Native, developers can write code once and run it on both iOS and Android devices thus saving time and resources leading to shorter development cycles hence faster time-to-market for your app.
Let’s take the example of a startup, which wanted to release their app on both iOS and Android at once. Through the use of React Native they managed to create an app and bring it into the market within a very short period. This helped them gain an advantage over their competitors because they had access to a large user base who were able to generate revenue quickly for them.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
A Comprehensive Look at Generative AI in Retail App Testing.pdfkalichargn70th171
Traditional software testing methods are being challenged in retail, where customer expectations and technological advancements continually shape the landscape. Enter generative AI—a transformative subset of artificial intelligence technologies poised to revolutionize software testing.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
5. Agenda
• The problem
• Hstore
• Introduction to jsonb indexing
• Jsquery - Jsonb Query Language
• Exercises on jsonb GIN opclasses with Jsquery support
• VODKA access method
6. The problem
• The world of data and applications is changing
• BIG DATA (Volume of data,Velocity of data in-out, Variety of data)
• Web applications are service-oriented
• Service itself can aggregate data, check consistency of data
• High concurrency, simple queries
• Simple database (key-value) is ok
• Eventual consistency is ok, no ACID overhead
• Application needs faster releases
• NoSQL databases match all of these — scalable, efficient, fault-tolerant,
no rigid schema, ready to accept any data.
7. NoSQL (концептуальные предпосылки)
• Реляционные СУБД — интеграционные
• Все приложения общаются через СУБД
• SQL — универсальный язык работы с данными
• Все изменения в СУБД доступны всем
• Изменения схемы очень затратны, медл. релизы
• Рассчитаны на интерактивную работу
• Интересны агрегаты, а не сами данные, нужен SQL
• SQL отслеживает транзакционность, ограничения целостности... вместо человека
8. NoSQL (концептуальные предпосылки)
• Сервисная архитектура изменила подход к СУБД
• Приложение состоит из сервисов, SQL->HTTP
• Сервисам не нужна одна монолитная СУБД
• Часто достаточно простых key-value СУБД
• Схема меняется «на ходу», быстрые релизы
• ACID → BASE
• Сервисы — это программы, которые могут сами заниматься агрегированием
• Сервисы могут сами следить за целостностью данных
• Много данных, аналитика, большое кол-во одновременных запросов
• Распределенность - кластеры дешевых shared-nothing машин
• NoSQL —горизонтальная масштабируемость и производительность
9.
10. NoSQL
• Key-value databases
• Ordered k-v for ranges support
• Column family (column-oriented) stores
• Big Table — value has structure:
• column families, columns, and timestamped versions (maps-of maps-of maps)
• Document databases
• Value has arbitrary structure
• Graph databases — evolution od ordered-kv
11. NoSQL databases (wikipedia) …+++
Document store
* Lotus Notes
* CouchDB
* MongoDB
* Apache Jackrabbit
* Colayer
* XML databases
o MarkLogic Server
o eXist
Graph
* Neo4j
* AllegroGraph
Tabular
* BigTable
* Mnesia
* Hbase
* Hypertable
Key/value store on disk
* Tuple space
* Memcachedb
* Redis
* SimpleDB
* flare
* Tokyo Cabinet
* BigTable
Key/value cache in RAM
* memcached
* Velocity
* Redis
Eventually-consistent key-value store
* Dynamo
* Cassandra
* Project Voldemort
Ordered key-value store
* NMDB
* Luxio
* Memcachedb
* Berkeley DB
Object database
* Db4o
* InterSystems Caché
* Objectivity/DB
* ZODB
14. The problem
• What if application needs ACID and flexibility of NoSQL ?
• Relational databases work with data with schema known in advance
• One of the major compaints to relational databases is rigid schema.
It's not easy to change schema online (ALTER TABLE … ADD COLUMN...)
• Application should wait for schema changing, infrequent releases
• NoSQL uses json format, why not have it in relational database ?
JSON in PostgreSQL
This is the challenge !
15. Challenge to PostgreSQL !
• Full support of semi-stuctured data in PostgreSQL
• Storage
• Operators and functions
• Efficiency (fast access to storage, indexes)
• Integration with CORE (planner, optimiser)
• Actually, PostgreSQL is schema-less database since 2003 — hstore, one
of the most popular extension !
17. Introduction to Hstore
id col1 col2 col3 col4 col5 A lot of columns
key1, …. keyN
●
The problem:
●
Total number of columns may be very large
●
Only several fields are searchable ( used in WHERE)
●
Other columns are used only to output
●
These columns may not known in advance
●
Solution
●
New data type (hstore), which consists of (key,value) pairs (a'la perl hash)
18. Introduction to Hstore
id col1 col2 col3 col4 col5 Hstore
key1=>val1, key2=>val2,.....
●
Easy to add key=>value pair
●
No need change schema, just change hstore.
●
Schema-less PostgreSQL in 2003 !
19. Introduction to hstore
• Hstore — key/value binary storage (inspired by perl hash)
'a=>1, b=>2'::hstore
• Key, value — strings
• Get value for a key: hstore -> text
• Operators with indexing support (GiST, GIN)
Check for key: hstore ? text
Contains: hstore @> hstore
• check documentations for more
• Functions for hstore manipulations (akeys, avals, skeys, svals, each,......)
• Hstore provides PostgreSQL schema-less feature !
• Faster releases, no problem with schema upgrade
20. Hstore binary storage
Npairs:31
Varlena
header
New version flag:1
Key endpos:
31
Val endpos:
31
ISNULL:1
key... val ...
Start End
First key 0 HEntry[0]
i-th key HEntry[i*2 - 1] HEntry[i*2]
i-th value HEntry[i*2] HEntry[i*2 + 1]
String arrayHEntry array
Pairs are lexicographically ordered by key
21. Hstore limitations
• Levels: unlimited
• Number of elements in array: 2^31
• Number of pairs in hash: 2^31
• Length of string: 2^31 bytes
2^31 bytes = 2 GB
22. History of hstore development
• May 16, 2003 — first version of hstore
23. History of hstore development
• May 16, 2003 - first (unpublished) version of hstore for PostgreSQL
7.3
• Dec, 05, 2006 - hstore is a part of PostgreSQL 8.2
(thanks, Hubert Depesz Lubaczewski!)
• May 23, 2007 - GIN index for hstore, PostgreSQL 8.3
• Sep, 20, 2010 - Andrew Gierth improved hstore, PostgreSQL 9.0
26. GIN improvements
• GIN in 9.4 is greatly improved
• Posting lists compression (varbyte encoding) — smaller indexes
• 9.3: always 6 bytes (4 bytes blockNumber , 2 bytes offset): 90 bytes
(0,8) (0,14) (0,17) (0,22) (0,26) (0,33) (0,34) (0,35) (0,45) (0,47) (0,48) (1,3) (1,4)
(1,6) (1,8)
• 9.4: 1-6 bytes per each item, deltas from previous item: 21 bytes
(0,8) +6 +3 +5 +4 +7 +1 +1 +10 +2 +1 +2051 +1+2 +2
SELECT g % 10 FROM generate_series(1,10000000) g; 11Mb vs 58Mb
• Fast scan of posting lists - «rare & frequent» queries much faster
• 9.3: read posting lists for «rare» and «frequent» and join them
Time(frequent & rare) ~ Time(frequent)
• 9.4: start from posting list for «rare» and skip «frequent» list if no match
Time(frequent & rare) ~ Time(rare)
27. Hstore is DEAD ? No !
• How hstore benefits by GIN improvement in 9.4 ?
GIN stands for Generalized Inverted Index, so virtually all data types, which use
GIN, get benefit !
• Default hstore GIN opclass considers keys and values separately
• Keys are «frequent», value are «rare»
• Contains query: hstore @> 'key=>value' improved a lot for «rare» values
• Index size is smaller, less io
28. Hstore 9.3 vs 9.4
Total: 7240858 geo records:
"fcode"=>"RFSU",
"point"=>"(8.85,112.53333)",
"fclass"=>"U",
"asciiname"=>"London Reefs",
"elevation"=>NULL,
"geonameid"=>"1879967",
"population"=>"0"
Query:
SELECT count(*) FROM geo
WHERE geo @> 'fcode=>STM';
gin_hstore_ops: index keys and values
gin_hstore_bytea_ops = gin_hstore_ops, no collation comparison
gin_hstore_hash_ops: index hash(key.value)
29. Hstore 9.3 vs 9.4
|-------------------------+-------+----------+-------+---------|
| Name | Type | Owner | Table | Size |
|-------------------------+-------+----------+-------+---------|
| geo | table | postgres | | 1352 MB |
| geo_hstore_bytea_ops | index | postgres | geo | 1680 MB |
| geo_hstore_hash_ops_idx | index | postgres | geo | 1073 MB |
|-------------------------+-------+----------+-------+---------|
|-------------------------+-------+----------+-------+---------|
| Name | Type | Owner | Table | Size |
|-------------------------+-------+----------+-------+---------|
| geo | table | postgres | | 1352 MB |
| geo_hstore_bytea_ops | index | postgres | geo | 1296 MB |
| geo_hstore_hash_ops_idx | index | postgres | geo | 925 MB |
|-------------------------+-------+----------+-------+---------|
9.4
9.3
CREATE OPERATOR CLASS gin_hstore_bytea_ops FOR TYPE hstore
….....................................................................................
FUNCTION 1 byteacmp(bytea,bytea),
….....................................................................................
STORAGE bytea;
CREATE INDEX: 239 s Much faster comparison (no collation)
CREATE OPERATOR CLASS gin_hstore_ops FOR TYPE hstore
….....................................................................................
FUNCTION 1 bttextcmp(text,text),,
….....................................................................................
STORAGE text;
CREATE INDEX: 2870 s
30. Hstore 9.3 vs 9.4
SUMMARY:
●
9.4 GIN posting list compression:
indexes are smaller
●
9.4 GIN is smart regarding 'freq & rare' queries:
time (freq & rare) ~ time (rare) instead of
time (freq & rare) ~ time (freq)
●
gin_hstore_hash_ops is good on 9.3 & 9.4 and
faster default gin opclass
●
Use gin_hstore_bytea_ops instead of default
gin_hstore_ops — much faster create index
Get hstore_ops from:
from https://github.com/akorotkov/hstore_ops
31. Introduction to hstore
• Hstore benefits
• In provides a flexible model for storing a semi-structured data in relational
database
• hstore has binary storage and rich set of operators and functions, indexes
• Hstore drawbacks
• Too simple model !
Hstore key-value model doesn't supports tree-like structures as json
(introduced in 2006, 3 years after hstore)
• Json — popular and standartized (ECMA-404 The JSON Data
Interchange Standard, JSON RFC-7159)
• Json — PostgreSQL 9.2, textual storage
32. Hstore vs Json
SELECT sum((v->'a')::text::int) FROM json_test;
851.012 ms
SELECT sum((v->'a')::int) FROM hstore_test;
330.027 ms
• hstore is faster than json even on simple data
CREATE TABLE hstore_test AS (SELECT
'a=>1, b=>2, c=>3, d=>4, e=>5'::hstore AS v
FROM generate_series(1,1000000));
CREATE TABLE json_test AS (SELECT
'{"a":1, "b":2, "c":3, "d":4, "e":5}'::json AS v
FROM generate_series(1,1000000));
33. Hstore vs Json
• PostgreSQL already has json since 9.2, which supports document-
based model, but
• It's slow, since it has no binary representation and needs to be parsed every
time
• Hstore is fast, thanks to binary representation and index support
• It's possible to convert hstore to json and vice versa, but current hstore is
limited to key-value
• Need hstore with document-based model. Share it's
binary representation with json !
35. Nested hstore & jsonb
• Nested hstore at PGCon-2013, Ottawa, Canada ( May 24) — thanks
Engine Yard for support !
One step forward true json data type.Nested hstore with arrays support
• Binary storage for nested data at PGCon Europe — 2013, Dublin, Ireland
(Oct 29)
Binary storage for nested data structuresand application to hstore data type
• November, 2013 — binary storage was reworked, nested hstore and
jsonb share the same storage. Andrew Dunstan joined the project.
• January, 2014 - binary storage moved to core
36. Nested hstore & jsonb
• Feb-Mar, 2014 - Peter Geoghegan joined the project, nested hstore was
cancelled in favour to jsonb (Nested hstore patch for 9.3).
• Mar 23, 2014 Andrew Dunstan committed jsonb to 9.4 branch !
pgsql: Introduce jsonb, a structured format for storing json.
Introduce jsonb, a structured format for storing json.
The new format accepts exactly the same data as the json type. However, it is
stored in a format that does not require reparsing the orgiginal text in order
to process it, making it much more suitable for indexing and other operations.
Insignificant whitespace is discarded, and the order of object keys is not
preserved. Neither are duplicate object keys kept - the later value for a given
key is the only one stored.
37. Jsonb vs Json
SELECT '{"c":0, "a":2,"a":1}'::json, '{"c":0, "a":2,"a":1}'::jsonb;
json | jsonb
-----------------------+------------------
{"c":0, "a":2,"a":1} | {"a": 1, "c": 0}
(1 row)
• json: textual storage «as is»
• jsonb: no whitespaces
• jsonb: no duplicate keys, last key win
• jsonb: keys are sorted
38. Jsonb vs Json
• Data
• 1,252,973 Delicious bookmarks
• Server
• MBA, 8 GB RAM, 256 GB SSD
• Test
• Input performance - copy data to table
• Access performance - get value by key
• Search performance contains @> operator
39. Jsonb vs Json
• Data
• 1,252,973 bookmarks from Delicious in json format (js)
• The same bookmarks in jsonb format (jb)
• The same bookmarks as text (tx)
=# dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------+-------+----------+---------+-------------
public | jb | table | postgres | 1374 MB | overhead is < 4%
public | js | table | postgres | 1322 MB |
public | tx | table | postgres | 1322 MB |
40. Jsonb vs Json
• Input performance (parser)
Copy data (1,252,973 rows) as text, json,jsonb
copy tt from '/path/to/test.dump'
Text: 34 s - as is
Json: 37 s - json validation
Jsonb: 43 s - json validation, binary storage
41. Jsonb vs Json (binary storage)
• Access performance — get value by key
• Base: SELECT js FROM js;
• Jsonb: SELECT j->>'updated' FROM jb;
• Json: SELECT j->>'updated' FROM js;
Base: 0.6 s
Jsonb: 1 s 0.4
Json: 9.6 s 9
Jsonb ~ 20X faster Json
42. Jsonb vs Json
EXPLAIN ANALYZE SELECt count(*) FROM js WHERE js #>>'{tags,0,term}' = 'NYC';
QUERY PLAN
----------------------------------------------------------------------------
Aggregate (cost=187812.38..187812.39 rows=1 width=0)
(actual time=10054.602..10054.602 rows=1 loops=1)
-> Seq Scan on js (cost=0.00..187796.88 rows=6201 width=0)
(actual time=0.030..10054.426 rows=123 loops=1)
Filter: ((js #>> '{tags,0,term}'::text[]) = 'NYC'::text)
Rows Removed by Filter: 1252850
Planning time: 0.078 ms
Execution runtime: 10054.635 ms
(6 rows)
Json: no contains @> operator,
search first array element
43. Jsonb vs Json (binary storage)
EXPLAIN ANALYZE SELECT count(*) FROM jb WHERE jb @> '{"tags":[{"term":"NYC"}]}'::jsonb;
QUERY PLAN
---------------------------------------------------------------------------------------
Aggregate (cost=191521.30..191521.31 rows=1 width=0)
(actual time=1263.201..1263.201 rows=1 loops=1)
-> Seq Scan on jb (cost=0.00..191518.16 rows=1253 width=0)
(actual time=0.007..1263.065 rows=285 loops=1)
Filter: (jb @> '{"tags": [{"term": "NYC"}]}'::jsonb)
Rows Removed by Filter: 1252688
Planning time: 0.065 ms
Execution runtime: 1263.225 ms Execution runtime: 10054.635 ms
(6 rows)
Jsonb ~ 10X faster Json
44. Jsonb vs Json (GIN: key && value)
EXPLAIN ANALYZE SELECT count(*) FROM jb WHERE jb @> '{"tags":[{"term":"NYC"}]}'::jsonb;
QUERY PLAN
---------------------------------------------------------------------------------------
Aggregate (cost=4772.72..4772.73 rows=1 width=0)
(actual time=8.486..8.486 rows=1 loops=1)
-> Bitmap Heap Scan on jb (cost=73.71..4769.59 rows=1253 width=0)
(actual time=8.049..8.462 rows=285 loops=1)
Recheck Cond: (jb @> '{"tags": [{"term": "NYC"}]}'::jsonb)
Heap Blocks: exact=285
-> Bitmap Index Scan on gin_jb_idx (cost=0.00..73.40 rows=1253 width=0)
(actual time=8.014..8.014 rows=285 loops=1)
Index Cond: (jb @> '{"tags": [{"term": "NYC"}]}'::jsonb)
Planning time: 0.115 ms
Execution runtime: 8.515 ms Execution runtime: 10054.635 ms
(8 rows)
CREATE INDEX gin_jb_idx ON jb USING gin(jb);
Jsonb ~ 150X faster Json
45. Jsonb vs Json (GIN: hash path.value)
EXPLAIN ANALYZE SELECT count(*) FROM jb WHERE jb @> '{"tags":[{"term":"NYC"}]}'::jsonb;
QUERY PLAN
---------------------------------------------------------------------------------------
Aggregate (cost=4732.72..4732.73 rows=1 width=0)
(actual time=0.644..0.644 rows=1 loops=1)
-> Bitmap Heap Scan on jb (cost=33.71..4729.59 rows=1253 width=0)
(actual time=0.102..0.620 rows=285 loops=1)
Recheck Cond: (jb @> '{"tags": [{"term": "NYC"}]}'::jsonb)
Heap Blocks: exact=285
-> Bitmap Index Scan on gin_jb_path_idx
(cost=0.00..33.40 rows=1253 width=0) (actual time=0.062..0.062 rows=285 loops=1)
Index Cond: (jb @> '{"tags": [{"term": "NYC"}]}'::jsonb)
Planning time: 0.056 ms
Execution runtime: 0.668 ms Execution runtime: 10054.635 ms
(8 rows)
CREATE INDEX gin_jb_path_idx ON jb USING gin(jb jsonb_path_ops);
Jsonb ~ 1800X faster Json
46. PostgreSQL 9.4 vs Mongo 2.6.0
• Operator contains @>
• json : 10 s seqscan
• jsonb : 8.5 ms GIN jsonb_ops
• jsonb : 0.7 ms GIN jsonb_path_ops
• mongo : 1.0 ms btree index
• Index size
• jsonb_ops - 636 Mb (no compression, 815Mb)
jsonb_path_ops - 295 Mb
• jsonb_path_ops (tags) - 44 Mb USING gin((jb->'tags') jsonb_path_ops
• mongo (tags) - 387 Mb
mongo (tags.term) - 100 Mb
•Table size
•postgres : 1.3Gb
•mongo : 1.8Gb
•Input performance:
• Text : 34 s
• Json : 37 s
• Jsonb : 43 s
• mongo : 13 m
47. Что сейчас может Jsonb ?
• Contains operators - jsonb @> jsonb, jsonb <@ jsonb (GIN indexes)
jb @> '{"tags":[{"term":"NYC"}]}'::jsonb
Keys should be specified from root
●
Equivalence operator — jsonb = jsonb (GIN indexes)
• Exists operators — jsonb ? text, jsonb ?! text[], jsonb ?& text[] (GIN indexes)
jb WHERE jb ?| '{tags,links}'
Only root keys supported
• Operators on jsonb parts (functional indexes)
SELECT ('{"a": {"b":5}}'::jsonb -> 'a'->>'b')::int > 2;
CREATE INDEX ….USING BTREE ( (jb->'a'->>'b')::int);
Very cumbersome, too many functional indexes
49. Найти что-нибудь красное
• WITH RECURSIVE t(id, value) AS ( SELECT * FROM
js_test
UNION ALL
(
SELECT
t.id,
COALESCE(kv.value, e.value) AS value
FROM
t
LEFT JOIN LATERAL
jsonb_each(
CASE WHEN jsonb_typeof(t.value) =
'object' THEN t.value
ELSE NULL END) kv ON true
LEFT JOIN LATERAL
jsonb_array_elements(
CASE WHEN
jsonb_typeof(t.value) = 'array' THEN t.value
ELSE NULL END) e ON true
WHERE
kv.value IS NOT NULL OR e.value IS
NOT NULL
)
)
SELECT
js_test.*
FROM
(SELECT id FROM t WHERE value @> '{"color":
"red"}' GROUP BY id) x
JOIN js_test ON js_test.id = x.id;
• Весьма непростое решение !
50. Что хочется ?
• Need Jsonb query language
• Simple and effective way to search in arrays (and other iterative
searches)
• More comparison operators (сейчас только =)
• Types support
• Schema support (constraints on keys, values)
• Indexes support
51. Что хочется ?
• Need Jsonb query language
• Simple and effective way to search in arrays (and other iterative
searches)
• More comparison operators (сейчас только =)
• Types support
• Schema support (constraints on keys, values)
• Indexes support
• Introduce Jsquery - textual data type and @@ match operator
jsonb @@ jsquery
56. Jsonb query language (Jsquery)
• Type checking
select '{"x": true}' @@ 'x IS boolean'::jsquery,
'{"x": 0.1}' @@ 'x IS numeric'::jsquery;
?column? | ?column?
----------+----------
t | t
IS BOOLEAN
IS NUMERIC
IS ARRAY
IS OBJECT
IS STRINGselect '{"a":{"a":1}}' @@ 'a IS object'::jsquery;
?column?
----------
t
select '{"a":["xxx"]}' @@ 'a IS array'::jsquery, '["xxx"]' @@ '$ IS array'::jsquery;
?column? | ?column?
----------+----------
t | t
57. Jsonb query language (Jsquery)
• How many products are similar to "B000089778" and have
product_sales_rank in range between 10000-20000 ?
• SQL
SELECT count(*) FROM jr WHERE (jr->>'product_sales_rank')::int > 10000
and (jr->> 'product_sales_rank')::int < 20000 and
….boring stuff
• Jsquery
SELECT count(*) FROM jr WHERE jr @@ ' similar_product_ids &&
["B000089778"] AND product_sales_rank( $ > 10000 AND $ < 20000)'
• Mongodb
db.reviews.find( { $and :[ {similar_product_ids: { $in ["B000089778"]}},
{product_sales_rank:{$gt:10000, $lt:20000}}] } ).count()
58. Найти что-нибудь красное
• WITH RECURSIVE t(id, value) AS ( SELECT * FROM
js_test
UNION ALL
(
SELECT
t.id,
COALESCE(kv.value, e.value) AS value
FROM
t
LEFT JOIN LATERAL
jsonb_each(
CASE WHEN jsonb_typeof(t.value) =
'object' THEN t.value
ELSE NULL END) kv ON true
LEFT JOIN LATERAL
jsonb_array_elements(
CASE WHEN
jsonb_typeof(t.value) = 'array' THEN t.value
ELSE NULL END) e ON true
WHERE
kv.value IS NOT NULL OR e.value IS
NOT NULL
)
)
SELECT
js_test.*
FROM
(SELECT id FROM t WHERE value @> '{"color":
"red"}' GROUP BY id) x
JOIN js_test ON js_test.id = x.id;
• Jsquery
SELECT * FROM js_test
WHERE
value @@ '*.color = "red"';
59. Еще пример
• SQL
SELECT * FROM js_test2 js
WHERE NOT EXISTS (
SELECT 1
FROM
jsonb_array_elements(js.value) el
WHERE EXISTS (
SELECT 1
FROM jsonb_each(el.value) kv
WHERE NOT
kv.value::text::numeric BETWEEN
0.0 AND 1.0));
• Jsquery
SELECT * FROM js_test2 js
WHERE '#:.%:($ >= 0 AND $ <= 1)';
60. Jsonb query language (Jsquery)
explain( analyze, buffers) select count(*) from jb where jb @> '{"tags":[{"term":"NYC"}]}'::jsonb;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------
Aggregate (cost=191517.30..191517.31 rows=1 width=0) (actual time=1039.422..1039.423 rows=1 loops=1)
Buffers: shared hit=97841 read=78011
-> Seq Scan on jb (cost=0.00..191514.16 rows=1253 width=0) (actual time=0.006..1039.310 rows=285 loops=1)
Filter: (jb @> '{"tags": [{"term": "NYC"}]}'::jsonb)
Rows Removed by Filter: 1252688
Buffers: shared hit=97841 read=78011
Planning time: 0.074 ms
Execution time: 1039.444 ms
explain( analyze,costs off) select count(*) from jb where jb @@ 'tags.#.term = "NYC"';
QUERY PLAN
--------------------------------------------------------------------
Aggregate (actual time=891.707..891.707 rows=1 loops=1)
-> Seq Scan on jb (actual time=0.010..891.553 rows=285 loops=1)
Filter: (jb @@ '"tags".#."term" = "NYC"'::jsquery)
Rows Removed by Filter: 1252688
Execution time: 891.745 ms
61. Jsquery (indexes)
• GIN opclasses with jsquery support
• jsonb_value_path_ops — use Bloom filtering for key matching
{"a":{"b":{"c":10}}} → 10.( bloom(a) or bloom(b) or bloom(c) )
• Good for key matching (wildcard support) , not good for range query
• jsonb_path_value_ops — hash path (like jsonb_path_ops)
{"a":{"b":{"c":10}}} → hash(a.b.c).10
• No wildcard support, no problem with ranges
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+-------------------------+-------+----------+--------------+---------+-------------
public | jb | table | postgres | | 1374 MB |
public | jb_value_path_idx | index | postgres | jb | 306 MB |
public | jb_gin_idx | index | postgres | jb | 544 MB |
public | jb_path_value_idx | index | postgres | jb | 306 MB |
public | jb_path_idx | index | postgres | jb | 251 MB |
62. Jsquery (indexes)
explain( analyze,costs off) select count(*) from jb where jb @@ 'tags.#.term = "NYC"';
QUERY PLAN
-------------------------------------------------------------------------------------------------
Aggregate (actual time=0.609..0.609 rows=1 loops=1)
-> Bitmap Heap Scan on jb (actual time=0.115..0.580 rows=285 loops=1)
Recheck Cond: (jb @@ '"tags".#."term" = "NYC"'::jsquery)
Heap Blocks: exact=285
-> Bitmap Index Scan on jb_value_path_idx (actual time=0.073..0.073 rows=285 loops=1)
Index Cond: (jb @@ '"tags".#."term" = "NYC"'::jsquery)
Execution time: 0.634 ms
(7 rows)
63. Jsquery (indexes)
explain( analyze,costs off) select count(*) from jb where jb @@ '*.term = "NYC"';
QUERY PLAN
-------------------------------------------------------------------------------------------------
Aggregate (actual time=0.688..0.688 rows=1 loops=1)
-> Bitmap Heap Scan on jb (actual time=0.145..0.660 rows=285 loops=1)
Recheck Cond: (jb @@ '*."term" = "NYC"'::jsquery)
Heap Blocks: exact=285
-> Bitmap Index Scan on jb_value_path_idx (actual time=0.113..0.113 rows=285 loops=1)
Index Cond: (jb @@ '*."term" = "NYC"'::jsquery)
Execution time: 0.716 ms
(7 rows)
64. Citus dataset{
"customer_id": "AE22YDHSBFYIP",
"product_category": "Business & Investing",
"product_group": "Book",
"product_id": "1551803542",
"product_sales_rank": 11611,
"product_subcategory": "General",
"product_title": "Start and Run a Coffee Bar (Start & Run a)",
"review_date": {
"$date": 31363200000
},
"review_helpful_votes": 0,
"review_rating": 5,
"review_votes": 10,
"similar_product_ids": [
"0471136174",
"0910627312",
"047112138X",
"0786883561",
"0201570483"
]
}
• 3023162 reviews from Citus
1998-2000 years
• 1573 MB
65. Jsquery (indexes)
explain (analyze, costs off) select count(*) from jr where
jr @@ ' similar_product_ids && ["B000089778"]';
QUERY PLAN
------------------------------------------------------------------------------------------------
Aggregate (actual time=0.359..0.359 rows=1 loops=1)
-> Bitmap Heap Scan on jr (actual time=0.084..0.337 rows=185 loops=1)
Recheck Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
Heap Blocks: exact=107
-> Bitmap Index Scan on jr_path_value_idx (actual time=0.057..0.057 rows=185 loops=1)
Index Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
Execution time: 0.394 ms
(7 rows)
66. Jsquery (indexes)
explain (analyze, costs off) select count(*) from jr where
jr @@ ' similar_product_ids && ["B000089778"]
AND product_sales_rank( $ > 10000 AND $ < 20000)';
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------
Aggregate (actual time=126.149..126.149 rows=1 loops=1)
-> Bitmap Heap Scan on jr (actual time=126.057..126.143 rows=45 loops=1)
Recheck Cond: (jr @@ '("similar_product_ids" && ["B000089778"] &
"product_sales_rank"($ > 10000 & $ < 20000))'::jsquery)
Heap Blocks: exact=45
-> Bitmap Index Scan on jr_path_value_idx (actual time=126.029..126.029 rows=45 loops=1)
Index Cond: (jr @@ '("similar_product_ids" && ["B000089778"] &
"product_sales_rank"($ > 10000 & $ < 20000))'::jsquery)
Execution time: 129.309 ms !!! No statistics
(7 rows)
• No statistics, no planning :(
Not selective, better not use index!
68. Jsquery (indexes)
explain (analyze,costs off) select count(*) from jr where
jr @@ ' similar_product_ids && ["B000089778"]'
and (jr->>'product_sales_rank')::int>10000 and (jr->>'product_sales_rank')::int<20000;
-----------------------------------------------------------------------------------------------------------------------------------------
Aggregate (actual time=0.479..0.479 rows=1 loops=1)
-> Bitmap Heap Scan on jr (actual time=0.079..0.472 rows=45 loops=1)
Recheck Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
Filter: ((((jr ->> 'product_sales_rank'::text))::integer > 10000) AND
(((jr ->> 'product_sales_rank'::text))::integer < 20000))
Rows Removed by Filter: 140
Heap Blocks: exact=107
-> Bitmap Index Scan on jr_path_value_idx (actual time=0.041..0.041 rows=185 loops=1)
Index Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
Execution time: 0.506 ms Potentially, query could be faster Mongo !
(9 rows)
• If we rewrite query and use planner
69. Jsquery (optimiser) — NEW !
• Jsquery now has built-in optimiser for simple queries.
explain (analyze, costs off) select count(*) from jr where
jr @@ 'similar_product_ids && ["B000089778"]
AND product_sales_rank( $ > 10000 AND $ < 20000)'
------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (actual time=0.422..0.422 rows=1 loops=1)
-> Bitmap Heap Scan on jr (actual time=0.099..0.416 rows=45 loops=1)
Recheck Cond: (jr @@ '("similar_product_ids" && ["B000089778"] AND
"product_sales_rank"($ > 10000 AND $ < 20000))'::jsquery)
Rows Removed by Index Recheck: 140
Heap Blocks: exact=107
-> Bitmap Index Scan on jr_path_value_idx (actual time=0.060..0.060 rows=185 loops=1)
Index Cond: (jr @@ '("similar_product_ids" && ["B000089778"] AND
"product_sales_rank"($ > 10000 AND $ < 20000))'::jsquery)
Execution time: 0.480 ms vs 7 ms MongoDB !
70. Jsquery (optimiser) — NEW !
• Jsquery now has built-in optimiser for simple queries.
Analyze query tree and push non-selective parts to recheck (like filter)
Selectivity classes:
1) Equality (x = c)
2) Range (c1 < x < c2)
3) Inequality (c > c1)
4) Is (x is type)
5) Any (x = *)
SELECT gin_debug_query_path_value('similar_product_ids && ["B000089778"]
AND product_sales_rank( $ > 10000 AND $ < 20000)');
gin_debug_query_path_value
-------------------------------------------------
similar_product_ids.# = "B000089778" , entry 0 +
71. Jsquery (optimiser) — NEW !
• Jsquery optimiser pushes non-selective operators to recheck
explain (analyze, costs off) select count(*) from jr where
jr @@ 'similar_product_ids && ["B000089778"]
AND product_sales_rank( $ > 10000 AND $ < 20000)'
------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (actual time=0.422..0.422 rows=1 loops=1)
-> Bitmap Heap Scan on jr (actual time=0.099..0.416 rows=45 loops=1)
Recheck Cond: (jr @@ '("similar_product_ids" && ["B000089778"] AND
"product_sales_rank"($ > 10000 AND $ < 20000))'::jsquery)
Rows Removed by Index Recheck: 140
Heap Blocks: exact=107
-> Bitmap Index Scan on jr_path_value_idx (actual time=0.060..0.060 rows=185 loops=1)
Index Cond: (jr @@ '("similar_product_ids" && ["B000089778"] AND
"product_sales_rank"($ > 10000 AND $ < 20000))'::jsquery)
Execution time: 0.480 ms
72. Jsquery (HINTING) — NEW !
• Jsquery now has HINTING ( if you don't like optimiser)!
explain (analyze, costs off) select count(*) from jr where jr @@ 'product_sales_rank > 10000'
----------------------------------------------------------------------------------------------------------
Aggregate (actual time=2507.410..2507.410 rows=1 loops=1)
-> Bitmap Heap Scan on jr (actual time=1118.814..2352.286 rows=2373140 loops=1)
Recheck Cond: (jr @@ '"product_sales_rank" > 10000'::jsquery)
Heap Blocks: exact=201209
-> Bitmap Index Scan on jr_path_value_idx (actual time=1052.483..1052.48
rows=2373140 loops=1)
Index Cond: (jr @@ '"product_sales_rank" > 10000'::jsquery)
Execution time: 2524.951 ms
• Better not to use index — HINT /* --noindex */
explain (analyze, costs off) select count(*) from jr where jr @@ 'product_sales_rank /*-- noindex */ > 10000';
----------------------------------------------------------------------------------
Aggregate (actual time=1376.262..1376.262 rows=1 loops=1)
-> Seq Scan on jr (actual time=0.013..1222.123 rows=2373140 loops=1)
Filter: (jr @@ '"product_sales_rank" /*-- noindex */ > 10000'::jsquery)
Rows Removed by Filter: 650022
Execution time: 1376.284 ms
73. Contrib/jsquery
• Jsquery index support is quite efficient ( 0.5 ms vs Mongo 7 ms ! )
• Future direction
• Make jsquery planner friendly
• Need statistics for jsonb
• Availability
• Jsquery + opclasses are available as extensions
• Grab it from https://github.com/akorotkov/jsquery (branch master) ,
we need your feedback !
76. Что дальше ?
• SQL-level jsquery (расширяемость, статистика)
• VODKA access method ! VODKA Optimized Dendriform Keys Array
• Комбинация произвольных методов доступа
77. Better indexing ...
• GIN is a proven and effective index access method
• Need indexing for jsonb with operations on paths (no hash!) and values
• B-tree in entry tree is not good - length limit, no prefix compression
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+-----------------------------+-------+----------+---------------+---------+-------------
public | jb | table | postgres | | 1374 MB |
public | jb_uniq_paths | table | postgres | | 912 MB |
public | jb_uniq_paths_btree_idx | index | postgres | jb_uniq_paths | 885 MB |text_pattern_ops
public | jb_uniq_paths_spgist_idx | index | postgres | jb_uniq_paths | 598 MB |now much less !
78. Better indexing ...
• Provide interface to change hardcoded B-tree in Entry tree
• Use spgist opclass for storing paths and values as is (strings hashed in values)
• We may go further - provide interface to change hardcoded B-tree in
posting tree
• GIS aware full text search !
• New index access method
CREATE INDEX … USING VODKA
79. GIN History
• Introduced at PostgreSQL Anniversary Meeting in Toronto, Jul 7-8, 2006
by Oleg Bartunov and Teodor Sigaev
80. GIN History
• Introduced at PostgreSQL Anniversary Meeting in Toronto, Jul 7-8, 2006
by Oleg Bartunov and Teodor Sigaev
• Supported by JFG Networks (France)
• «Gin stands for Generalized Inverted iNdex and should be considered as
a genie, not a drink.»
• Alexander Korotkov, Heikki Linnakangas have joined GIN++ development
in 2013
81. GIN History
TODO
----
Nearest future:
* Opclasses for all types (no programming, just many catalog changes).
Distant future:
* Replace B-tree of entries to something like GiST (VODKA ! 2014)
* Add multicolumn support
* Optimize insert operations (background index insertion)
• From GIN Readme, posted in -hackers, 2006-04-26
82. GIN index structure for jsonb
{
"product_group": "Book",
"product_sales_rank": 15000
},
{
"product_group": "Music",
"product_sales_rank": 25000
}
83. Vodka index structure for jsonb
{
"product_group": "Book",
"product_sales_rank": 15000
},
{
"product_group": "Music",
"product_sales_rank": 25000
}
84. CREATE INDEX … USING VODKA
set maintenance_work_mem = '1GB';
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+--------------------+-------+----------+-------+---------+-------------
public | jb | table | postgres | | 1374 MB | 1252973 rows
public | jb_value_path_idx | index | postgres | jb | 306 MB | 98769.096
public | jb_gin_idx | index | postgres | jb | 544 MB | 129860.859
public | jb_path_value_idx | index | postgres | jb | 306 MB | 100560.313
public | jb_path_idx | index | postgres | jb | 251 MB | 68880.320
public | jb_vodka_idx | index | postgres | jb | 409 MB | 185362.865
public | jb_vodka_idx5 | index | postgres | jb | 325 MB | 174627.234 new spgist
(6 rows)
• Delicious bookmarks, mostly text data
85. CREATE INDEX … USING VODKA
select count(*) from jb where jb @@ 'tags.#.term = "NYC"';
-------------------------------------------------------------------------------------------
Aggregate (actual time=0.423..0.423 rows=1 loops=1)
-> Bitmap Heap Scan on jb (actual time=0.146..0.404 rows=285 loops=1)
Recheck Cond: (jb @@ '"tags".#."term" = "NYC"'::jsquery)
Heap Blocks: exact=285
-> Bitmap Index Scan on jb_vodka_idx (actual time=0.108..0.108 rows=285 loops=1)
Index Cond: (jb @@ '"tags".#."term" = "NYC"'::jsquery)
Execution time: 0.456 ms (0.634 ms, GIN jsonb_value_path_ops)
select count(*) from jb where jb @@ '*.term = "NYC"';
-------------------------------------------------------------------------------------------
Aggregate (actual time=0.495..0.495 rows=1 loops=1)
-> Bitmap Heap Scan on jb (actual time=0.245..0.474 rows=285 loops=1)
Recheck Cond: (jb @@ '*."term" = "NYC"'::jsquery)
Heap Blocks: exact=285
-> Bitmap Index Scan on jb_vodka_idx (actual time=0.214..0.214 rows=285 loops=1)
Index Cond: (jb @@ '*."term" = "NYC"'::jsquery)
Execution time: 0.526 ms (0.716 ms, GIN jsonb_path_value_ops)
86. CREATE INDEX … USING VODKA
set maintenance_work_mem = '1GB';
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+--------------------+-------+----------+-------+---------+-------------
public | jr | table | postgres | | 1573 MB | 3023162 rows
public | jr_value_path_idx | index | postgres | jr | 196 MB | 79180.120
public | jr_gin_idx | index | postgres | jr | 235 MB | 111814.929
public | jr_path_value_idx | index | postgres | jr | 196 MB | 73369.713
public | jr_path_idx | index | postgres | jr | 180 MB | 48981.307
public | jr_vodka_idx3 | index | postgres | jr | 240 MB | 155714.777
public | jr_vodka_idx4 | index | postgres | jr | 211 MB | 169440.130 new spgist
(6 rows)
• CITUS data, text and numeric
87. CREATE INDEX … USING VODKA
explain (analyze, costs off) select count(*) from jr where jr @@ ' similar_product_ids && ["B000089778"]';
QUERY PLAN
-------------------------------------------------------------------------------------------
Aggregate (actual time=0.200..0.200 rows=1 loops=1)
-> Bitmap Heap Scan on jr (actual time=0.090..0.183 rows=185 loops=1)
Recheck Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
Heap Blocks: exact=107
-> Bitmap Index Scan on jr_vodka_idx (actual time=0.077..0.077 rows=185 loops=1)
Index Cond: (jr @@ '"similar_product_ids" && ["B000089778"]'::jsquery)
Execution time: 0.237 ms (0.394 ms, GIN jsonb_path_value_idx)
(7 rows)
95. Summary
• contrib/jsquery for 9.4
• Jsquery - Jsonb Query Language
• Two GIN opclasses with jsquery support
• Grab it from https://github.com/akorotkov/jsquery (branch master)
• Prototype of VODKA access method
• Plans for improving indexing infrastructure
• This work was supported by
96. Another view on VODKA
• VODKA CONNECTING INDEXES
• composite index, which combines different access methods
• Nested search trees
97. postgrespro.ru
• Ищем инженеров:
• 24x7 поддержка
• Консалтинг & аудит
• Разработка админских
приложений
• Пакеты
• Ищем си-шников для работы
над постгресом:
• Неубиваемый и масштабируемый
кластер
• Хранилища (in-memory, column-
storage...)