This document discusses PostgreSQL's support for JSON, including:
- An overview of new features in versions 9.2 and 9.3 for JSON production, processing, and extraction.
- Explanations of JSON data types, operators, and functions added in 9.3 for converting between JSON and other PostgreSQL types.
- Examples of using the new JSON operators and functions to extract, transform, and produce JSON.
- Details on leveraging the new JSON C API to build custom JSON functions and transformations.
- Discussions of future work, like a possible binary JSON representation, to further improve PostgreSQL's JSON support.
This presentation will demonstrate how you can use the aggregation pipeline with MongoDB similar to how you would use GROUP BY in SQL and the new stage operators coming 3.4. MongoDB’s Aggregation Framework has many operators that give you the ability to get more value out of your data, discover usage patterns within your data, or use the Aggregation Framework to power your application. Considerations regarding version, indexing, operators, and saving the output will be reviewed.
Wprowadzenie do technologii Big Data / Intro to Big Data EcosystemSages
Introduction to Hadoop Map Reduce, Pig, Hive and Ambari technologies.
Workshop deck prepared and presented on September 5th 2015 by Radosław Stankiewicz.
During that the day participants had also the possibility to go through prepared tutorials and test their analysis on real cluster.
How can you use PosgreSQL as a schemaless (NoSQL) database? Here we cover our use case and highlight upcoming features in postgres 9.4 and its integration with Django 1.7
This presentation will demonstrate how you can use the aggregation pipeline with MongoDB similar to how you would use GROUP BY in SQL and the new stage operators coming 3.4. MongoDB’s Aggregation Framework has many operators that give you the ability to get more value out of your data, discover usage patterns within your data, or use the Aggregation Framework to power your application. Considerations regarding version, indexing, operators, and saving the output will be reviewed.
Wprowadzenie do technologii Big Data / Intro to Big Data EcosystemSages
Introduction to Hadoop Map Reduce, Pig, Hive and Ambari technologies.
Workshop deck prepared and presented on September 5th 2015 by Radosław Stankiewicz.
During that the day participants had also the possibility to go through prepared tutorials and test their analysis on real cluster.
How can you use PosgreSQL as a schemaless (NoSQL) database? Here we cover our use case and highlight upcoming features in postgres 9.4 and its integration with Django 1.7
JavaScript client API for Google Apps Script API primerBruce McPherson
An API for JavaScript/jQuery client webapps providing CRUD access to Google Apps Script scriptDB - a free noSQL databases. This adds to a VBA API for Excel already published. Now Excel, Google Apps Script and JavaScript clients can share the same noSQL database and data.
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...MongoDB
MongoDB Stitch is a serverless platform designed to help you easily and securely build an application on top of MongoDB Atlas. It lets developers focus on building applications rather than on managing data manipulation code, service integration, or backend infrastructure. MongoDB Stitch also makes it simple to respond to backend changes immediately, allowing you to simplify client side code and build complex flows more easily. This talk will cover ways that MongoDB Stitch helps you respond to changes in your database and take your applications to the next level.
JavaScript client API for Google Apps Script API primerBruce McPherson
An API for JavaScript/jQuery client webapps providing CRUD access to Google Apps Script scriptDB - a free noSQL databases. This adds to a VBA API for Excel already published. Now Excel, Google Apps Script and JavaScript clients can share the same noSQL database and data.
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...MongoDB
MongoDB Stitch is a serverless platform designed to help you easily and securely build an application on top of MongoDB Atlas. It lets developers focus on building applications rather than on managing data manipulation code, service integration, or backend infrastructure. MongoDB Stitch also makes it simple to respond to backend changes immediately, allowing you to simplify client side code and build complex flows more easily. This talk will cover ways that MongoDB Stitch helps you respond to changes in your database and take your applications to the next level.
Present and future of Jsonb in PostgreSQL
Json - is an ubiquitous data format, which supported in almost any popular databases. PostgreSQL was the first relational database, which received support of native textual json and very efficient binary jsonb data types. Recently published SQL 2016 standard describes the JSON data type and specifies the functions and operators to work with json in SQL, which greatly simplifies the further development of json support in PostgreSQL. We compare existing features of json/jsonb data types with proposed SQL standard and discuss the ways how we could improve json/jsonb support in PostgreSQL.
PostgreSQL offers to application developers a rich support of json data type, providing known advantages of the json data model with traditional benefits of relational databases, such as declarative query language, rich query processing, transaction management providing ACID safety guarantees. However, current support of json is far from ideal, for example, json is still "foreign" data type to SQL - existed jsquery extension tries to implement their own query language, which is being powerfull, is opaque to Postgres planner and optimizer and not extendable. Extending SQL to support json, without commonly accepted standard, is difficult and perspectiveless task. Recently published SQL 2016 standard describes the JSON data type and specifies the functions and operators to work with json in SQL, which makes clear the direction of future development of json support in PostgreSQL. We present our ideas and prototype of future json data type in PostgreSQL with some further non-standard extensions and improvements in storage requirement and index support.
Postgres-XC as a Key Value Store Compared To MongoDBMason Sharp
This presentation discusses how Postgres-XC can be used as a PostgreSQL-based key-value store using features like hstore and JSON. It also compares performance to MongoDB for a read workload
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesJonathan Katz
All data is relational and can be represented through relational algebra, right? Perhaps, but there are other ways to represent data, and the PostgreSQL team continues to work on making it easier and more efficient to do so!
With the upcoming 9.4 release, PostgreSQL is introducing the "JSONB" data type which allows for fast, compressed, storage of JSON formatted data, and for quick retrieval. And JSONB comes with all the benefits of PostgreSQL, like its data durability, MVCC, and of course, access to all the other data types and features in PostgreSQL.
How fast is JSONB? How do we access data stored with this type? What can it do with the rest of PostgreSQL? What can't it do? How can we leverage this new data type and make PostgreSQL scale horizontally? Follow along with our presentation as we try to answer these questions.
NYC* 2013 - "Advanced Data Processing: Beyond Queries and Slices"DataStax Academy
The ColumnFamily data model and wide-row support provides the ability to store and access data efficiently in a de-normalized state. Recent enhancements for CQL's spare tables and built-in indexing provide the capability to store data in a manner similar to that of relational databases. For many use cases hybrid approaches are needed, because complete de-normalization is appropriate for some access patterns whereas more structured data is appropriate for others. At times a single logical event becomes multiple insertions across multiple column families. Likewise a user request might require a several reads across different column families. This talk describes some of these scenarios and demonstrates how advanced operations such multiple step procedures, filtering, intersection, and paging can be implemented client side or server side with the help of the IntraVert plugin.
Node.js allows you to run JavaScript code outside of a browser window, offering powerful tools to interact with a computer filesystem, run web servers, and create terminal applications. Node handles these tasks by running asynchronously, which means that reading user input from a terminal isn’t as simple as calling a getInput() function. In this article, we’ll show you an easy way around that (and some tips and tricks for handling user input) by using a helpful Node module.
readline-sync: This is the third party module that is used for taking the input from the user synchronously. So, the execution of the programs is line by line.
Installing module:
npm install readline-sync
Node handles these tasks by running asynchronously, which means that reading user input from a terminal isn't as simple as calling a getInput() function.
Groovy speech I held last year for introducing a new JVM language as substitute of Java. Easy and intuitive, it offers new features unknow to its parent yet.
Razvan Rotari shows an experiment to see how far you can go with binding in C++; Cristian Neamtu follows with an insight on how to achieve this in Rust using Serde.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
1. PostgreSQL 9.3 and JSON
Andrew Dunstan
andrew@dunslane.net
andrew.dunstan@pgexperts.com
2. Overview
● What is JSON?
● Why use JSON?
● Quick review of 9.2 features
● 9.3 new features
● Building on 9.3 features
● Some useful extensions
● Future work
3. What is JSON?
● Data serialization format
● Lightweight
● Human readable
● Becoming ubiquitous
● Simpler and more compact than XML
4. What it looks like
{
"books" : [
{ "title": "Catch 22”, "author": "Joseph Heller"},
{ "title": "Catcher in the Rye", "author": "J. D. Salinger"}
],
"publishers": [
{ "name": "Random House" },
{ "name": "Penguin" }
],
"active": true,
"version": 35,
"date": "2003-09-13",
“reference”: null
}
Scalars:
● quoted strings
● numbers
● true, false, null
No extensions
No date/time types
5. Why use it?
● Everyone is moving that way
● Understood everywhere there is a JavaScript
interpreter
– Especially browsers
● ... and in a large number of other languages
– e.g. Perl, Python
● node.js is becoming very widely used
● More compact than XML
● Most applications don't need the richer
structure of XML
6. Why not use it?
● Overly verbose
● Field names are repeated
● Arguably less readable than, say, YAML
● Not suitable for huge objects
7. Review – pre 9.2 facilities
Nothing – store JSON as text
● No validation
● No JSON production
● No JSON extraction
8. Review – 9.2 data type
New JSON type
● Stored as text
● Reasonably performant state-based validating
parser
● Kudos: Robert Haas
9. Review – 9.2 production functions
● Turn non-JSON data into JSON
● row_to_json(anyrecord)
● array_to_json(anyarray)
● Optional second param for pretty printing
● My humble contribution ☺
10. What's missing?
● JSON production features are incomplete
● JSON processing is totally absent
● Have to use PLV8, PLPerl or some such
11. 9.3 Features – JSON production
● to_json(any)
● Can be used on any datum, not just
arrays and records
● json_agg(record)
● Much faster than
array_to_json(array_agg(record))
12. 9.3 and casts to JSON
● Production functions honor casts to JSON
for non-builtin types
● Not needed for builtins, as we know how
to convert them
● Saves syscache lookups where we know
it's not necessary
13. 9.3 hstore and JSON
● hstore_to_json(hstore)
● Also used as a cast function
● hstore_to_json_loose(hstore)
● Uses heuristics about whether or not
certain possibly numeric and boolean
values need to be quoted.
14. 9.3 JSON parser rewrite
● New parser uses recursive descent
pattern
● Caller can supply event handlers for
certain events
● c.f. XML SAX parsers
● Validator uses NULL handlers for all
events
● Tokenizing routines of previous parser
largely kept
15. 9.3 JSON processsing functions
● All leverage new parser API
● Operators give a more natural style to
extraction operations
● Many have two forms, producing either
JSON output, which can be further
processed, or text output, which cannot.
● Text output is de-escaped and dequoted
16. 9.3 extraction operators (1)
● -> fetch an array element or object
member as json
● '[4,5,6]'::json->2 ⟹ 6
– json arrays are 0 based, unlike SQL arrays
● '{"a":1,"b":2}'::json->'b' ⟹ 2
17. 9.3 extraction operators (2)
● ->> fetch an array element orobject
member as text
● '["a","b","c"]'::json->2 ⟹ c
– Instead of "c"
18. 9.3 extraction operators (3)
● #> and #>> fetch data pointed at by a
path
● Path is an array of text elements
● Treats arrays correctly by some trying to
treat path element as an integer of
necessary
● '{"a":[6,7,8]}'::json#>'{a,1}' ⟹ 7
●
19. 9.3 extraction functions
● json_extract_path(json, VARIADIC
path_elems text[]);
● json_extract_path_text(json, VARIADIC
path_elems text[]);
● Same as #> and #>> operators, but
you can pass the path as a variadic
array
● json_extract_path('{"a":[6,7,8]}','a','1')
⟹ 7
20. 9.3 turn JSON into records
● CREATE TYPE x AS (a int, b int);
● SELECT * FROM
json_populate_record(null::x,
'{"a":1,"b":2}', false);
● SELECT * FROM
json_populate_recordset(null::x,'[{"a":1,"
b":2},{"a":3,"b":4}]', false);
21. 9.3 turn JSON into key/value pairs
● SELECT * FROM
json_each('{"a":1,"b":"foo"}')
● SELECT * FROM
json_each_text('{"a":1,"b":"foo"}')
● Deliver columns named “key” and
“value”
22. 9.3 get keys from JSON object
● SELECT * FROM
json_object_keys('{"a":1,"b":"foo"}')
24. 9.3 the JSON C API
● Need
● A state object
● Some handlers
● Stash these in a JsonSemAction object
● call pg_json_parse()
25. 9.3 API example
● Code can be cloned from
https://bitbucket.org/adunstan/json_typeof
● CREATE FUNCTION json_typeof(json)
RETURNS text
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT IMMUTABLE;
26. 9.3 API example state object
● #include <utils/jsonapi.h>
● typedef struct typeof_state {
JsonLexContext *lex;
JsonTokenType result;
} TypeofState;
27. 9.3 API example event handlers
● Need one for each of:
● Array start
● Object start
● Scalar
30. 9.3 example function code
Datum json_typeof(PG_FUNCTION_ARGS)
{
text *json = PG_GETARG_TEXT_P(0);
TypeofState *state;
JsonLexContext *lex = makeJsonLexContext(json, false);
JsonSemAction *sem;
state = palloc0(sizeof(TypeofState));
sem = palloc0(sizeof(JsonSemAction));
state->lex = lex;
sem->semstate = (void *) state;
sem->object_start = json_typeof_ostart;
sem->array_start = json_typeof_astart;
sem->scalar = json_typeof_scalar;
pg_parse_json(lex, sem);
PG_RETURN_TEXT_P(token_to_type(state->result));
}
31. 9.3 CAPI – what else is it good for?
● Transformations of all kinds
● XML
● YAML
● ....
● An XML transformation was the original
working model for creating the API
32. Other useful modules
● Developed for use by my clients
● json_build lets you compose json of arbitrary
complexity
● json_object creates a json object from a 1- or
2-dimensional array of text, similar to hstore
facility.
33. json_build
● Two functions:
● build_json_object(VARIADIC “any”)
● build_json_array(VARIADIC “any”)
● Together these let you compose deeply nested
tree structures safely.
34. json_build example
SELECT build_json_object(
'a', build_json_object('b',false,'c',99),
'd', build_json_object('e',array[9,8,7]::int[],
'f', (select row_to_json(r) from
( select relkind, oid::regclass as name
from pg_class where relname = 'pg_class') r)));
{"a" : {"b" : false, "c" : 99}, "d" : {"e" : [9,8,7],
"f" : {"relkind":"r","name":"pg_class"}}}
35. json_object examples
SELECT json_object('{a,1,b,2,3,NULL,"d e f","a b c"}');
json_object
-------------------------------------------------------
{"a" : "1", "b" : "2", "3" : null, "d e f" : "a b c"}
-- same but with two dimensions
SELECT json_object('{{a,1},{b,2},{3,NULL},{"d e f","a b c"}}');
json_object
-------------------------------------------------------
{"a" : "1", "b" : "2", "3" : null, "d e f" : "a b c"}
36. json_object examples
SELECT json_object('{a,1,b,2,3,NULL,"d e f","a b c"}');
json_object
-------------------------------------------------------
{"a" : "1", "b" : "2", "3" : null, "d e f" : "a b c"}
-- same but with two dimensions
SELECT json_object('{{a,1},{b,2},{3,NULL},{"d e f","a b c"}}');
json_object
-------------------------------------------------------
{"a" : "1", "b" : "2", "3" : null, "d e f" : "a b c"}
38. Future of JSON in PostgreSQL
● Possible binary representation
● Avoid reparsing
● Could be vastly faster for some operations
● Should we use a dictionary and store structure
separately?
● General document store
● Need to get around the “rewrite a whole
datum” issue
39. Credit
● 9.3 core development work sponsored by
Heroku
● Other work sponsored by PGExperts clients
and IVC