This document summarizes a meeting about ResourceSync, which is a project to design an approach for synchronizing resources between systems as they change over time. The meeting agenda included an overview of the problem from Herbert Van de Sompel, technical directions from Robert Sanderson, and joining the effort from Nettie Lagace. Key topics discussed were the needs for initial baseline synchronization, incremental synchronization of changes, auditing for consistency, and the technical challenges involved in change notification and content transfer between systems.
This presentation provides a problem perspective from the recently launched NISO/OAI ResourceSync effort that aims at devisions a framework for synchronizing web resources. The slides were used during a WebEx conference on March 6 2012.
The slides were used to accompany an overview of the outcomes of the ResourceSync project at the 2014 Spring Membership Meeting of the Coalition for Networked Information (CNI).
The launch of ResourceSync, a joint project of the National Information Standards Organization (NISO) and the Open Archives Initiative (OAI) funded by the Alfred P. Sloan Foundation, was motivated by the ubiquitous need to synchronize resources for applications in the realm of cultural heritage and research communication. After an initial problem definition and scoping phase, the project has designed, specified, and tested a framework for web-based synchronization that is based on SiteMaps, a protocol widely used by web servers to advertise the resources they make available to search engines for indexing. This choice allows repositories to address both search engine optimization and resource synchronization needs using the same technology.
The ResourceSync framework specifies various modular capabilities that a repository can support in order to allow third party systems to remain synchronized with its evolving resources. For example, a Resource List provides an inventory of resources whereas a Change List details resources that were created, deleted or updated during a given temporal interval. Support for capabilities can be combined in order to meet local or community requirements. The framework specifies capabilities that require a third party to recurrently poll for up-to-date information about a repositories’ resources but also publish/subscribe capabilities that keep third parties informed about changes through notifications, thereby significantly reducing synchronization latency.
ResourceSync core team members Bernhard Haslhofer and Simeon Warner will present on the ResourceSync specification and provide practical examples and scenarios for its application.
Pesented at SWIB13 in Hamburg, 2013-11-27. ResourceSync slides excerpted from the full tutorial at http://www.slideshare.net/OpenArchivesInitiative/resourcesync-tutorial
This presentation introduces ResourceSync, a specification aimed to enable web-based synchronization of resources. The specification is the result of a collaboration between NISO and the Open Archives Initiative funded by the Sloan Foundation and JISC. The proposed resource synchronization approach is based on several existing specifications (e.g. Sitemaps, PubSubHubbub, well-known URI) and is aligned with common architectural principles (e.g. REST, follow your nose).
A 15 minute video version of these slides is available at https://www.youtube.com/watch?v=ASQ4jMYytsA
This presentation provides a problem perspective from the recently launched NISO/OAI ResourceSync effort that aims at devisions a framework for synchronizing web resources. The slides were used during a WebEx conference on March 6 2012.
The slides were used to accompany an overview of the outcomes of the ResourceSync project at the 2014 Spring Membership Meeting of the Coalition for Networked Information (CNI).
The launch of ResourceSync, a joint project of the National Information Standards Organization (NISO) and the Open Archives Initiative (OAI) funded by the Alfred P. Sloan Foundation, was motivated by the ubiquitous need to synchronize resources for applications in the realm of cultural heritage and research communication. After an initial problem definition and scoping phase, the project has designed, specified, and tested a framework for web-based synchronization that is based on SiteMaps, a protocol widely used by web servers to advertise the resources they make available to search engines for indexing. This choice allows repositories to address both search engine optimization and resource synchronization needs using the same technology.
The ResourceSync framework specifies various modular capabilities that a repository can support in order to allow third party systems to remain synchronized with its evolving resources. For example, a Resource List provides an inventory of resources whereas a Change List details resources that were created, deleted or updated during a given temporal interval. Support for capabilities can be combined in order to meet local or community requirements. The framework specifies capabilities that require a third party to recurrently poll for up-to-date information about a repositories’ resources but also publish/subscribe capabilities that keep third parties informed about changes through notifications, thereby significantly reducing synchronization latency.
ResourceSync core team members Bernhard Haslhofer and Simeon Warner will present on the ResourceSync specification and provide practical examples and scenarios for its application.
Pesented at SWIB13 in Hamburg, 2013-11-27. ResourceSync slides excerpted from the full tutorial at http://www.slideshare.net/OpenArchivesInitiative/resourcesync-tutorial
This presentation introduces ResourceSync, a specification aimed to enable web-based synchronization of resources. The specification is the result of a collaboration between NISO and the Open Archives Initiative funded by the Sloan Foundation and JISC. The proposed resource synchronization approach is based on several existing specifications (e.g. Sitemaps, PubSubHubbub, well-known URI) and is aligned with common architectural principles (e.g. REST, follow your nose).
A 15 minute video version of these slides is available at https://www.youtube.com/watch?v=ASQ4jMYytsA
ResourceSync: Web-Based Resource Synchronization. Also for Data.
Herbert Van de Sompel, Digital Library Researcher, Los Alamos National Laboratory, and Co-chair of NISO’s ResourceSync Working Group
Web applications frequently leverage resources made available by remote Web servers. As resources are created, updated, or deleted these applications face challenges to remain in lockstep with the server’s change dynamics. Several approaches exist to help meet this challenge for use cases where “good enough” synchronization is acceptable. But when strict resource coverage or low synchronization latency is required, commonly accepted Web-based solutions remain elusive. Motivated by the need to synchronize resources for applications in the realm of cultural heritage and research communication, the National Information Standards Organization (NISO) and the Open Archives Initiative (OAI) have launched the ResourceSync project that aims at designing an approach for resource synchronization that is aligned with the web architecture and that has a fair chance of adoption by different communities. The presentation will discuss some motivating use cases and will provide a perspective on the resource synchronization problem that results from ResourceSync project discussions. It will provide an overview of the ongoing thinking regarding an approach to address the challenges and will pay special attention to aspects that are relevant for the synchronization of data.
ResourceSync Tutorial from Open Repositories 2013Simeon Warner
Slides form the tutorial on the ResourceSync Framework presented at Open Repositories 2013 in Charlottetown, PEI on 8 July 2013. The latest set of ResourceSync tutorial slides are available at http://www.slideshare.net/OpenArchivesInitiative/resourcesync-tutorial
Persistent Identifiers and the Web: The Need for an Unambiguous MappingHerbert Van de Sompel
Presentation given at the International Digital Curation Conference in San Francisco, February 26 2014. Highlights the lack of machine-actionability of persistent identifiers assigned to scholarly communication assets. Proposes an approach to address the issue that meets requirements that take into account the changing nature of web based research communication. A draft paper provides more details: http://public.lanl.gov/herbertv/papers/Papers/2014/IDCC2014_vandesompel.pdf
From a talk at the SF CTO Summit 2017 (https://www.ctoconnection.com/summits/sf2017), these slides cover the speaker's experience at Stitch Fix with managing data in a microservices environment. Areas include:
* Breaking up a monolithic database into services
* Using events as a first-class part of your architecture
* Sharing data among microservices
* Handling "joins" among microservices
* Simulating "transactions" among microservices using the Saga pattern
LavaCon 2017 - Connecting Silos With Content PipelinesJack Molisani
The source for published content is often far outside the domain of subject matter experts, making it difficult to keep the experts engaged for timely content updates. This presentation is a case study in moving the authoritative source for API documentation out of a DITA repository and pushing it upstream through the development organization’s design/specification environment and into the code base itself. The result is increased ownership by subject matter experts, better collaboration, and more up-to-date and accurate published content.
Slidedeck on performing large scale migrations for SharePoint presented during the SharePoint Connect Conference 2014 in Amsterdam on Tuesday 18 November.
This webinar will provide an overview of the current work undertaken to re-write the techniques for electronic resource management with the incorporation of open access workflow management. This overview will provide insight into the key areas under exploration and outline the feedback compiled from the two interactive sessions held at the UKSG Annual Conference. We will also talk about the next steps we undertake to share the development of this project.
The experience of the Private Academic Library Network of Indiana implementing cloud based next gen ILS system WMS, including the selection process and change management work.
*Updated and reorganised following feedback in the breakouts*
While many librarians have developed mechanisms and
structures for managing local scholarship separate from
their standard resource management practices, the
intersection of the two content streams is occurring at
many institutions. During the past decade the presenters
have dedicated themselves to capturing best practices
of electronic resource management and mapping out
paths for creating open access workflows. Join them for a
lively discussion and interactive session where they outline
ways to bring these two initiatives together and identify the
teams needed.
Graham Stone
Jisc Collections
Peter McCracken
Cornell University
Jill Emery
Portland State University Library
Navigating SAP’s Integration Options (Mastering SAP Technologies 2013)Sascha Wenninger
Provides an overview of popular integration approaches, maps them to SAP's integration tools and concludes with some lessons learnt in their application.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the closing segment of the NISO training series "AI & Prompt Design." Session Eight: Limitations and Potential Solutions, was held on May 23, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the seventh segment of the NISO training series "AI & Prompt Design." Session 7: Open Source Language Models, was held on May 16, 2024.
ResourceSync: Web-Based Resource Synchronization. Also for Data.
Herbert Van de Sompel, Digital Library Researcher, Los Alamos National Laboratory, and Co-chair of NISO’s ResourceSync Working Group
Web applications frequently leverage resources made available by remote Web servers. As resources are created, updated, or deleted these applications face challenges to remain in lockstep with the server’s change dynamics. Several approaches exist to help meet this challenge for use cases where “good enough” synchronization is acceptable. But when strict resource coverage or low synchronization latency is required, commonly accepted Web-based solutions remain elusive. Motivated by the need to synchronize resources for applications in the realm of cultural heritage and research communication, the National Information Standards Organization (NISO) and the Open Archives Initiative (OAI) have launched the ResourceSync project that aims at designing an approach for resource synchronization that is aligned with the web architecture and that has a fair chance of adoption by different communities. The presentation will discuss some motivating use cases and will provide a perspective on the resource synchronization problem that results from ResourceSync project discussions. It will provide an overview of the ongoing thinking regarding an approach to address the challenges and will pay special attention to aspects that are relevant for the synchronization of data.
ResourceSync Tutorial from Open Repositories 2013Simeon Warner
Slides form the tutorial on the ResourceSync Framework presented at Open Repositories 2013 in Charlottetown, PEI on 8 July 2013. The latest set of ResourceSync tutorial slides are available at http://www.slideshare.net/OpenArchivesInitiative/resourcesync-tutorial
Persistent Identifiers and the Web: The Need for an Unambiguous MappingHerbert Van de Sompel
Presentation given at the International Digital Curation Conference in San Francisco, February 26 2014. Highlights the lack of machine-actionability of persistent identifiers assigned to scholarly communication assets. Proposes an approach to address the issue that meets requirements that take into account the changing nature of web based research communication. A draft paper provides more details: http://public.lanl.gov/herbertv/papers/Papers/2014/IDCC2014_vandesompel.pdf
From a talk at the SF CTO Summit 2017 (https://www.ctoconnection.com/summits/sf2017), these slides cover the speaker's experience at Stitch Fix with managing data in a microservices environment. Areas include:
* Breaking up a monolithic database into services
* Using events as a first-class part of your architecture
* Sharing data among microservices
* Handling "joins" among microservices
* Simulating "transactions" among microservices using the Saga pattern
LavaCon 2017 - Connecting Silos With Content PipelinesJack Molisani
The source for published content is often far outside the domain of subject matter experts, making it difficult to keep the experts engaged for timely content updates. This presentation is a case study in moving the authoritative source for API documentation out of a DITA repository and pushing it upstream through the development organization’s design/specification environment and into the code base itself. The result is increased ownership by subject matter experts, better collaboration, and more up-to-date and accurate published content.
Slidedeck on performing large scale migrations for SharePoint presented during the SharePoint Connect Conference 2014 in Amsterdam on Tuesday 18 November.
This webinar will provide an overview of the current work undertaken to re-write the techniques for electronic resource management with the incorporation of open access workflow management. This overview will provide insight into the key areas under exploration and outline the feedback compiled from the two interactive sessions held at the UKSG Annual Conference. We will also talk about the next steps we undertake to share the development of this project.
The experience of the Private Academic Library Network of Indiana implementing cloud based next gen ILS system WMS, including the selection process and change management work.
*Updated and reorganised following feedback in the breakouts*
While many librarians have developed mechanisms and
structures for managing local scholarship separate from
their standard resource management practices, the
intersection of the two content streams is occurring at
many institutions. During the past decade the presenters
have dedicated themselves to capturing best practices
of electronic resource management and mapping out
paths for creating open access workflows. Join them for a
lively discussion and interactive session where they outline
ways to bring these two initiatives together and identify the
teams needed.
Graham Stone
Jisc Collections
Peter McCracken
Cornell University
Jill Emery
Portland State University Library
Navigating SAP’s Integration Options (Mastering SAP Technologies 2013)Sascha Wenninger
Provides an overview of popular integration approaches, maps them to SAP's integration tools and concludes with some lessons learnt in their application.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the closing segment of the NISO training series "AI & Prompt Design." Session Eight: Limitations and Potential Solutions, was held on May 23, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the seventh segment of the NISO training series "AI & Prompt Design." Session 7: Open Source Language Models, was held on May 16, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the sixth segment of the NISO training series "AI & Prompt Design." Session Six: Text Classification with LLMs, was held on May 9, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the fifth segment of the NISO training series "AI & Prompt Design." Session Five: Named Entity Recognition with LLMs, was held on May 2, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the fourth segment of the NISO training series "AI & Prompt Design." Session Four: Structured Data and Assistants, was held on April 25, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the third segment of the NISO training series "AI & Prompt Design." Session Three: Beginning Conversations, was held on April 18, 2024.
This presentation was provided by Kaveh Bazargan of River Valley Technologies, during the NISO webinar "Sustainability in Publishing." The event was held April 17, 2024.
This presentation was provided by Dana Compton of the American Society of Civil Engineers (ASCE), during the NISO webinar "Sustainability in Publishing." The event was held April 17, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, during the second segment of the NISO training series "AI & Prompt Design." Session Two: Large Language Models, was held on April 11, 2024.
This presentation was provided by Teresa Hazen of the University of Arizona, Geoff Morse of Northwestern University. and Ken Varnum of the University of Michigan, during the Spring ODI Conformance Statement Workshop for Libraries. This event was held on April 9, 2024
This presentation was provided by William Mattingly of the Smithsonian Institution, during the opening segment of the NISO training series "AI & Prompt Design." Session One: Introduction to Machine Learning, was held on April 4, 2024.
This presentation was provided by William Mattingly of the Smithsonian Institution, for the eight and final session of NISO's 2023 Training Series on Text and Data Mining. Session eight, "Building Data Driven Applications" was held on Thursday, December 7, 2023.
This presentation was provided by William Mattingly of the Smithsonian Institution, for the seventh session of NISO's 2023 Training Series on Text and Data Mining. Session seven, "Vector Databases and Semantic Searching" was held on Thursday, November 30, 2023.
This presentation was provided by William Mattingly of the Smithsonian Institution, for the sixth session of NISO's 2023 Training Series on Text and Data Mining. Session six, "Text Mining Techniques" was held on Thursday, November 16, 2023.
This presentation was provided by William Mattingly of the Smithsonian Institution, for the fifth session of NISO's 2023 Training Series on Text and Data Mining. Session five, "Text Processing for Library Data" was held on Thursday, November 9, 2023.
This presentation was provided by Todd Carpenter, Executive Director, during the NISO webinar on "Strategic Planning." The event was held virtually on November 8, 2023.
This presentation was provided by Rhonda Ross of CAS, a division of the American Chemical Society, and Jonathan Clark of the International DOI Foundation, during the NISO webinar on "Strategic Planning." The event was held virtually on November 8, 2023.
This presentation was provided by William Mattingly of the Smithsonian Institution, for the fourth session of NISO's 2023 Training Series on Text and Data Mining. Session four, "Data Mining Techniques" was held on Thursday, November 2, 2023.
This presentation was provided by Tiffany Straza of UNESCO, during the two-day "NISO Tech Summit: Reflections Upon The Year of Open Science." Day two was held on October 26, 2023.
More from National Information Standards Organization (NISO) (20)
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
2. ResourceSync – WebEx Agenda
Problem Perspective – Herbert Van de Sompel (LANL)
Technical Directions – Robert Sanderson (LANL)
Joining the Effort – Nettie Lagace (NISO)
Q&A
ResourceSync – March 6 2012
3. ResourceSync – Problem Perspective
Presenter – Herbert Van de Sompel
ResourceSync – March 6 2012
4. ResourceSync Core Team
Cornell University & OAI:
BerhardHaslhofer, Carl Lagoze, Simeon Warner
Old Dominion University & OAI:
Michael L. Nelson
Los Alamos National Laboratory & OAI:
Martin Klein, Robert Sanderson, Herbert Van de Sompel
NISO:
Todd Carpenter, Nettie Lagace, Peter Murray
ResourceSync – March 6 2012
5. ResourceSync Problem
• Consideration:
• Source (server) A has resources that change over time: they
get created, modified, deleted, moved, …
• Destination (servers) X, Y, and Z leverage (some) resources
of Source A.
• Problem:
• Destinations want to keep in step with the resource changes
at Source A: resource synchronization.
• Task of ResourceSync effort:
• Design an approach for resource synchronization aligned
with the Web Architecture that has a fair chance of adoption
by different communities.
• The approach must scale better than recurrent HTTP
HEAD/GET on resources.
ResourceSync – March 6 2012
6. ResourceSync Use Cases
• arXiv mirroring.
• Metadata and content transfer from Europeana partners to
central Europeana hub.
• Synchronization of Linked Data content.
• Recurrently collecting Memento metadata from IIPC web
archives to central aggregator.
• Keeping up-to-data with resources that reside on a Web server.
• Use cases have different requirements regarding synchronization
accuracy:
• Synchronization coverage: perfect …… good enough
• Synchronization speed: fast …….…. fast enough
ResourceSync – March 6 2012
7. Web Context
• The context in which resource synchronization will take place is
the Web.
• Resources considered for synchronization:
• Are identified by dereference-able URIs.
• Are cache-able.
=> Caveat: It is understood that resources that require client-side
processing (e.g. hashbangURIs) cause problems.
ResourceSync – March 6 2012
8. 3 Synchronization Needs
• Overall, there are 3 distinct needs regarding resource
synchronization. All 3 are considered in scope for the effort:
1. Baseline matching: An approach to allow a Destination that
wants to start synchronizing with a Source to perform an
initial catch up – Dump.
2. Incremental resource synchronization: An approach to allow
a Destination to remain up-to-date regarding changes at the
Source.
3. Audit: An approach to allow checking whether a Destination
is in sync with a Source – Inventory.
ResourceSync – March 6 2012
9. Change Notification – Content Transfer
• Distinguish between two aspects related to incremental resource
synchronization:
2.a. Change notification: An approach to allow a Destination to
understand that a Source’s resource has changed; and what
the nature of the change is.
2.b. Content transfer: An approach to allow a Destination to
update its holdings to reflect the change the resource
underwent at the Source.
=> For both, PUSH or PULL approaches are conceivable.
ResourceSync – March 6 2012
10. Selective Synchronization
• There is a need to be able to synchronize with a limited set of a
Source’s resources: selective synchronization.
• This leads to the notion of channels for resource
synchronization.
• The need for channels is most apparent for resource
synchronization but may have to be carried through in
baseline matching and audit.
=> This raises an issue:
• The definition of channels by the Source is straightforward,
yet may not meet the Destination’s needs.
• The definition of channels by a Destination is less
straightforward.
ResourceSync – March 6 2012
11. Memory
• For robustness, both change notification and content transfer
may need a memory function to allow for synchronization after a
Destination has missed updates:
• Catching up on change notifications - Digests.
• Catching up on resource versions - Mementos.
However, it would be beneficial if resource synchronization could
operate in a memory-less mode in cases where less than perfect
sync is acceptable.
An important architectural consideration is where that memory
should reside: Source, intermediary.
ResourceSync – March 6 2012
12. Changes Only – Entire Resource
• Two approaches for content transfer:
2.b.1. Synchronization is achieved by transfer of the entire
resource.
2.b.2. Synchronization is achieved by transfer of resource
changes only.
Focus is on (2.b.1) because of complexities re content-type-
specific resource segment description and re-assembly
instructions involved in (2.b.2).
Focus re (2.b.1) is on content transfer using a PULL, not PUSH,
approach.
ResourceSync – March 6 2012
13. Event Types
• Event types to be considered for change notifications:
• Tier 1:
• Regarding resources: create ; delete ; update ; new
dump ; new inventory
• Regarding channels: removed from channel ; added to
channel
• Tier 2:
• Regarding resources: move ; copy
Focus is on Tier 1 and on events pertaining to single resources.
Tier 2 and consideration of groups of resources (e.g. URI
templates, URI regex) is postponed.
ResourceSync – March 6 2012
14. Resource Representations
• Resource representations are subject to synchronization:
• But limited to those that can be requested using protocol
parameters for the resource’s URI.
=> Caveats:
• It is understood this causes problems in light of resource
representation personalization, e.g. geo-dependent
representations.
• If the change notification function is provided by 3rd party
instead of Source, change notifications may be subjective to
the 3rd parties perspective.
ResourceSync – March 6 2012
15. Discovery
• Several Discovery needs arise:
• Regarding change notification channels:
• How to find channels that provide change notifications
for a given Source Server?
• How to find information about the nature of the changes
that are communicated on such channels?
• Regarding dumps and inventories:
• Where to obtain the most recent dump?
• Where to obtain the most recent inventory?
• Regarding memory:
• Where to obtain digests?
• How to access Mementos?
ResourceSync – March 6 2012
16. And Also …
• Authorization: For baseline matching, resource synchronization
(change notification, content transfer), and audit a distinction
may be required between Destinations that are authorized or
unauthorized to perform these functions.
• Embargo: How soon after a resource change are parties allowed
to know about it, to act upon it?
• Trust in a change notification channel:
• Can a change notification channel that provides information
about a given Source Server be trusted?
ResourceSync – March 6 2012
18. ResourceSync Architectural Challenges
Major Research Areas:
1. Initial Baseline Synchronization
• Retrieve dump of data
2. Incremental Synchronization
• Main focus of research
3. Audit to ensure Consistency
• Not yet significantly investigated
Incremental Synchronization split into two separate aspects:
1. Change Notification (CN)
• Alert that something happened (create,update,delete)
2. Content Transfer (CT)
• Transfer of just the change or the full resource
ResourceSync – March 6 2012
19. Incremental Synchronization - Architectures
Trivial Approach:
• Retrieve every resource and compare to current copy
• Not scalable: too many wasted, large transactions
• No way to discover newly created resources
ResourceSync – March 6 2012
20. Incremental Synchronization - Architectures
Theoretically Optimal Solution:
• Whenever a resource changes, push only the change to only the
Destinations monitoring the resource
• No wasted transactions, only as much data transferred as needed
• Newly created resources are discovered
• But overly burdens the source! Not economically viable
• Q: Sweet point between Trivial and Optimal?
ResourceSync – March 6 2012
21. Incremental Synchronization - Architectures
Trivial Approach plus Conditional GET:
• Retrieve every resource if it has changed
• Essentially this is a Change Notification Pull
• Still not scalable: too many wasted transactions
• Still no way to discover newly created resources
ResourceSync – March 6 2012
22. Incremental Synchronization - Architectures
Simplest Workable Model:
• Introduce a Feed of change notifications for all resources
• Atom, RSS, OAI-PMH, SiteMaps, etc.
• Significantly reduces wasted transactions
• Newly created resources are discovered
• But still not very efficient as don’t know when to pull
ResourceSync – March 6 2012
23. Incremental Synchronization - Architectures
Feed Extension Solution:
• Continue the Feed paradigm, but introduce aggregating service
and ping notification to re-pull (simulated push)
• No wasted transactions, but pull will get already-seen notifications
• Only advantageous if Source already supports a Feed
ResourceSync – March 6 2012
24. Incremental Synchronization - Architectures
Push Solution:
• Instead of ping+pull, Source can push the notification
• No wasted transactions, no wasted data transfer
• Service maintains subscriber list, not Source
• Change Notification from Source is easier than a Ping+Feed if no
feed already, and minimally harder than just a Ping
ResourceSync – March 6 2012
25. Change Notification - Protocols
• Atom PubSubHubbub (PuSH)
• XMPP
• PubSub extension
• BoSH (XMPP over HTTP)
• Comet / HTTP Streaming
• Open an HTTP connection and keep reading from it
• Bayeux Protocol
• Long Polling
• Keep HTTP connection open until a message, then reopen
• BoSH, Bayeux option
• WebSockets
• NullMQ / ZeroMQ
• XMPP over WebSockets?
ResourceSync – March 6 2012
26. Change Notification - Messages
• Experiments with “bleep” change notifications, inspired by tweets
• Text that is writeable & readable by machine and human
• Simple “resync” language:
• URIcreated at=“time-of-create”#hashtag$resync
• URIupdated at=“time-of-update”#hashtag$resync
• URIdeleted at=“time-of-delete”#hashtag$resync
• Approach is extensible via creation of new languages for bleeps
http://megalodon.lanl.gov/dbpedia/data/Paris updated at=“2012-03-
05T19:54:39Z”
#dbpedia $resync
ResourceSync – March 6 2012
27. Ongoing Investigations
• Change Notification - XMPP & XMPP PubSub& bleeps
• LANL
• Ongoing Experiment with Live DBPedia
• Change Notification - Comet / HTTP Streaming & bleeps
• ODU
• Bayeux Protocol via Faye Implementation
• Change Notification - Change Simulator
• Cornell U
• Generate configurable change notifications
• Use as standardized input to different systems for testing
• Baseline Matching & Audit
• Cornell U
ResourceSync – March 6 2012