sos4R @ OGC TC
•Download as PPTX, PDF•
0 likes•393 views
Presentation of the sos4R SOS client for R at the Open Geospatial Consortium Technical Committee meeting in Bonn.
Report
Share
Report
Share
Recommended
Las Vegas Commerce Office Market Review
Please enjoy the 1st quarter 2010 Commerce Real Estate Solutions Office Market Report
sos4R - 52° North Innovation Price Presentation
Final presentation of the winning project of the 52° North Innovation Price sos4R - Accessing a Sensor Observation Service from R.
Mining Newsletterpage Sept09
Commerce CRG / Cushman and Wakefield Las Vegas - Monthly Report and Listings
WPS Application Patterns
Presentation at the workshop "Models for scientific exploitation of EO Data" organised by the HMA Architecture Working Group
Connecting R to the Sensor Web
Interoperable data exchange and reproducibility are increasingly important for modern scientific research. This paper shows how three open source projects work together to realize this: (i) the R project, providing the lingua franca for statistical analysis, (ii) the Open Geospatial Consortium's Sensor Observation Service (SOS), a standardized data warehouse service for storing and retrieving sensor measurements, and (iii) sos4R, a new project that connects the former two. We show how sos4R can bridge the gap be-tween two communities in science: spatial statistical analysis and visuali-zation on one side, and the Sensor Web community on the other. sos4R enables R users to integrate (near real-time) sensor observations directly into R. Finally, we evaluate the functionality of sos4R. The software en-capsulates the service's complexity with typical R function calls in a com-mon analysis workflow, but still gives users full flexibility to handle in-teroperability issues. We conclude that it is able to close the gap between R and the sensor web.
Recommended
Las Vegas Commerce Office Market Review
Please enjoy the 1st quarter 2010 Commerce Real Estate Solutions Office Market Report
sos4R - 52° North Innovation Price Presentation
Final presentation of the winning project of the 52° North Innovation Price sos4R - Accessing a Sensor Observation Service from R.
Mining Newsletterpage Sept09
Commerce CRG / Cushman and Wakefield Las Vegas - Monthly Report and Listings
WPS Application Patterns
Presentation at the workshop "Models for scientific exploitation of EO Data" organised by the HMA Architecture Working Group
Connecting R to the Sensor Web
Interoperable data exchange and reproducibility are increasingly important for modern scientific research. This paper shows how three open source projects work together to realize this: (i) the R project, providing the lingua franca for statistical analysis, (ii) the Open Geospatial Consortium's Sensor Observation Service (SOS), a standardized data warehouse service for storing and retrieving sensor measurements, and (iii) sos4R, a new project that connects the former two. We show how sos4R can bridge the gap be-tween two communities in science: spatial statistical analysis and visuali-zation on one side, and the Sensor Web community on the other. sos4R enables R users to integrate (near real-time) sensor observations directly into R. Finally, we evaluate the functionality of sos4R. The software en-capsulates the service's complexity with typical R function calls in a com-mon analysis workflow, but still gives users full flexibility to handle in-teroperability issues. We conclude that it is able to close the gap between R and the sensor web.
[DRAFT] Workshop - Technical Introduction to joola.io
This workshop focuses on hands-on introduction to joola.io
For a complete breakdown of the Workshop itself, refer to the project's wiki @ http://github.com/joola/joola.io/wiki/workshops
Ops Jumpstart: MongoDB Administration 101
New to MongoDB? We'll provide an overview of installation, high availability through replication, scale out through sharding, and options for monitoring and backup. No prior knowledge of MongoDB is assumed. This session will jumpstart your knowledge of MongoDB operations, providing you with context for the rest of the day's content.
BOTTARI: Location based Social Media Analysis with Semantic Web
Bottari is a LarKC application http://www.larkc.eu/. It offers a real-time personalized recommendation service for restaurants in Insa-dong(Seoul) listening to the reputation of the restaurants on social media. Social media anlytics is powered by LarKC inductive and deductive stream reasoning solution. Learn more at http://larkc.cefriel.it/lbsma/bottari/ .
Multiplatform Spark solution for Graph datasources by Javier Dominguez
https://www.bigdataspain.org/2016/program/thu-multiplatform-spark-solution-graph-datasources.html
https://www.youtube.com/watch?v=_86udgjeK8w&index=6&t=16s&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik
Design of Experiments on Federator Polystore Architecture
This presentation talks about the Design of Experiments on Federator Polystore Architecture and the current (bad) results.
Frameworks provide structure. The core objective of the Big Data Framework is...
Frameworks provide structure. The core objective of the Big Data Framework is to provide a structure for enterprise organisations that aim to benefit from the potential of Big Data
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Jennie Wang, Software Engineer (Intel)
Tsai Louie, Software Engineer (Intel)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Elastic container platforms (like Kubernetes, Docker Swarm, Apache Mesos) fit very well with existing cloud-native application architecture approaches. So it is more than astonishing, that these already existing and open source available elastic platforms are not considered more consequently for multi-cloud approaches. Elastic container platforms provide inherent multi-cloud support that can be easily accessed. We present a solution proposal of a control process which is able to scale (and migrate as a side effect) elastic container platforms across different public and private cloud-service providers. This control loop can be used in an execution phase of self-adaptive auto-scaling MAPE loops (monitoring, analysis, planning, execution). Additionally, we present several lessons learned from our prototype implementation which might be of general interest for researchers and practitioners. For instance, to describe only the intended state of an elastic platform and let a single control process take care to reach this intended state is far less complex than to define plenty of specific and necessary multi-cloud aware workflows to deploy, migrate, terminate, scale up and scale down elastic platforms or applications.
ICWE2017 BigDataEurope
A paper about the BigDataEurope Project and Platform has been accepted for publication as a full-paper at ICWE2017 in Technical/Vision track.
Scaling AI in production using PyTorch
Slides from my talk at MLOps World' 21
Deploying AI models in production and scaling the ML services is still a big challenge. In this talk we will cover details of how to deploy your AI models, best practices for the deployment scenarios, and techniques for performance optimization and scaling the ML services. Come join us to learn how you can jumpstart the journey of taking your PyTorch models from Research to production.
PointNet
Journal club done with Vid Stojevic for PointNet:
https://arxiv.org/abs/1612.00593
https://github.com/charlesq34/pointnet
http://stanford.edu/~rqi/pointnet/
Deep learning for Indoor Point Cloud processing. PointNet, provides a unified architecture operating directly on unordered point clouds without voxelisation for applications ranging from object classification, part segmentation, to scene semantic parsing.
Alternative download link:
https://www.dropbox.com/s/ziyhgi627vg9lyi/3D_v2017_initReport.pdf?dl=0
Vector databases and neural search
Dmitry Kan, Principal AI Scientist at Silo AI and host of the Vector Podcast [1], will give an overview of the landscape of vector search databases and their role in NLP, along with the latest news and his view on the future of vector search. Further, he will share how he and his team participated in the Billion-Scale Approximate Nearest Neighbor Challenge and improved recall by 12% over a baseline FAISS.
Presented at https://www.meetup.com/open-nlp-meetup/events/282678520/
YouTube: https://www.youtube.com/watch?v=RM0uuMiqO8s&t=179s
Follow Vector Podcast to stay up to date on this topic: https://www.youtube.com/@VectorPodcast
Bne impact iif
Presentation introducing IMPACT Evaluation framework presented by Clemens Neudecker during demo session held at the BNE 5th of October 2011.
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
A long time ago in a database far, far away...
SQL was the only option to save vast amounts of application data for a long period of time. There were always some rebellion activities, to overcome the SQL Empire, which brought a new hope, but all other ways of storing data were never more than a phantom menace.
Now Cosmos DB awakens and is ready for the revenge of the NoSQL.
During this talk, we will have a look at what Azure Cosmos DB is, what you can achieve with its possibilities and how to use it in a galactic environment of data and applications.
Join me and find your way to the right solution for your application.
May the data be with you!
Big data serving: Processing and inference at scale in real time
Jon Bratseth (VP Architect) @ Verizon Media:
The big data world has mature technologies for offline analysis and learning from data, but have lacked options for making data-driven decisions in real time.
When it is sufficient to consider a single data point model servers such as TensorFlow serving can be used but in many cases you want to consider many data points to make decisions.
This is a difficult engineering problem combining state, distributed algorithms and low latency, but solving it often makes it possible to create far superior solutions when applying machine learning.
This talk will explain why this is a hard problem, show the advantages of solving it, and introduce the open source Vespa.ai platform which is used to implement such solutions in some of the largest scale problems in the world including the world's third largest ad serving system.
zenoh: The Edge Data Fabric
Zenoh is rapidly growing Eclipse project that unifies data in motion, data at rest and computations. It elegantly blends traditional pub/sub with geo distributed storage, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks. This presentation will provide an introduction to Eclipse Zenoh along with a crisp explanation of the challenges that motivated the creation of this project. We will go through a series of real-world use cases that demonstrate the advantages brought by Zenoh in enabling and optimising typical edge scenarios and in simplifying the development of any scale distributed applications.
Integrating Apache Phoenix with Distributed Query Engines
This talk will describe the work being done to create connectors for Presto and Apache Spark to read and write data in Phoenix tables. We will describe the new phoenix connector that implements Spark’s DataSource v2 API which will enable customizing and optimizing reads and writes to Phoenix tables.
We will also demo the Presto-phoenix connector, showing how it can be used to federate multiple Phoenix clusters and join Phoenix data with different types of data sources.
We will also describe some in progress work to more tightly integrate with the query optimizers of these frameworks in order to provide table statistics and push down filters, limits and aggregates into Phoenix whenever possible in order to speed up query execution.
Another area being worked on is to provide a way to support bulk loading using HFiles.
Deploy and Destroy: Testing Environments - Michael Arenzon - DevOpsDays Tel A...
One of the critical factors for development velocity is software correctness. Our ability to develop and ship new features fast is bounded by our ability to validate several aspects of the change: * Does the feature meet the requirements? * How does the feature affect existing code, and how can it affect the production environment? With continues codebase growth and new features being added, naturally our productivity decreases, and our need to improve the guarantees for quality and correctness increase.
In this talk, I’ll focus on testing environments: why developers need a self-serve platform to create a full functioning environment on-demand, how such environments should be managed, and how can one restore part of the lost velocity. I’ll cover an internal system we use at AppsFlyer called ‘Namespaces’ that addresses the issue with the help of Mesos / Marathon, Docker, Traefik, and Consul.
RR & Docker @ MuensteR Meetup (Sep 2017)
A short introduction to reproducible research, reproducibility with R, Docker, and all together for reproducible research using R and Docker containers. Includes demos of Rocker and containerit.
containerit at useR!2017 conference, Brussels
**Webpage**
https://github.com/o2r-project/containerit/
**Abstract**
Reproducibility of computations is crucial in an era where data is born digital and analysed algorithmically. Most studies however only publish the results, often with figures as important interpreted outputs. But where do these figures come from? Scholarly articles must provide not only a description of the work but be accompanied by data and software. R offers excellent tools to create reproducible works, i.e. Sweave and RMarkdown. Several approaches to capture the workspace environment in R have been made, working around CRAN’s deliberate choice not to provide explicit versioning of packages and their dependencies. They preserve a collection of packages locally (packrat, pkgsnap, switchr/GRANBase) or remotely (MRAN timemachine/checkpoint), or install specific versions from CRAN or source (requireGitHub, devtools). Installers for old versions of R are archived on CRAN. A user can manually re-create a specific environment, but this is a cumbersome task.
We introduce a new possibility to preserve a runtime environment including both, packages and R, by adding an abstraction layer in the form of a container, which can execute a script or run an interactive session. The package containeRit automatically creates such containers based on Docker. Docker is a solution for packaging an application and its dependencies, but shows to be useful in the context of reproducible research (Boettiger 2015). The package creates a container manifest, the Dockerfile, which is usually written by hand, from sessionInfo(), R scripts, or RMarkdown documents. The Dockerfiles use the Rocker community images as base images. Docker can build an executable image from a Dockerfile. The image is executable anywhere a Docker runtime is present. containeRit uses harbor for building images and running containers, and sysreqs for installing system dependencies of R packages. Before the planned CRAN release we want to share our work, discuss open challenges such as handling linked libraries (see discussion on geospatial libraries in Rocker), and welcome community feedback.
containeRit is developed within the DFG-funded project Opening Reproducible Research to support the creation of Executable Research Compendia (ERC) (Nüst et al. 2017).
**References**
Boettiger, Carl. 2015. “An Introduction to Docker for Reproducible Research, with Examples from the R Environment.” ACM SIGOPS Operating Systems Review 49 (January): 71–79. doi:10.1145/2723872.2723882.
Nüst, Daniel, Markus Konkol, Edzer Pebesma, Christian Kray, Marc Schutzeichel, Holger Przibytzin, and Jörg Lorenz. 2017. “Opening the Publication Process with Executable Research Compendia.” D-Lib Magazine 23 (January). doi:10.1045/january2017-nuest.
More Related Content
Similar to sos4R @ OGC TC
[DRAFT] Workshop - Technical Introduction to joola.io
This workshop focuses on hands-on introduction to joola.io
For a complete breakdown of the Workshop itself, refer to the project's wiki @ http://github.com/joola/joola.io/wiki/workshops
Ops Jumpstart: MongoDB Administration 101
New to MongoDB? We'll provide an overview of installation, high availability through replication, scale out through sharding, and options for monitoring and backup. No prior knowledge of MongoDB is assumed. This session will jumpstart your knowledge of MongoDB operations, providing you with context for the rest of the day's content.
BOTTARI: Location based Social Media Analysis with Semantic Web
Bottari is a LarKC application http://www.larkc.eu/. It offers a real-time personalized recommendation service for restaurants in Insa-dong(Seoul) listening to the reputation of the restaurants on social media. Social media anlytics is powered by LarKC inductive and deductive stream reasoning solution. Learn more at http://larkc.cefriel.it/lbsma/bottari/ .
Multiplatform Spark solution for Graph datasources by Javier Dominguez
https://www.bigdataspain.org/2016/program/thu-multiplatform-spark-solution-graph-datasources.html
https://www.youtube.com/watch?v=_86udgjeK8w&index=6&t=16s&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik
Design of Experiments on Federator Polystore Architecture
This presentation talks about the Design of Experiments on Federator Polystore Architecture and the current (bad) results.
Frameworks provide structure. The core objective of the Big Data Framework is...
Frameworks provide structure. The core objective of the Big Data Framework is to provide a structure for enterprise organisations that aim to benefit from the potential of Big Data
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Jennie Wang, Software Engineer (Intel)
Tsai Louie, Software Engineer (Intel)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Elastic container platforms (like Kubernetes, Docker Swarm, Apache Mesos) fit very well with existing cloud-native application architecture approaches. So it is more than astonishing, that these already existing and open source available elastic platforms are not considered more consequently for multi-cloud approaches. Elastic container platforms provide inherent multi-cloud support that can be easily accessed. We present a solution proposal of a control process which is able to scale (and migrate as a side effect) elastic container platforms across different public and private cloud-service providers. This control loop can be used in an execution phase of self-adaptive auto-scaling MAPE loops (monitoring, analysis, planning, execution). Additionally, we present several lessons learned from our prototype implementation which might be of general interest for researchers and practitioners. For instance, to describe only the intended state of an elastic platform and let a single control process take care to reach this intended state is far less complex than to define plenty of specific and necessary multi-cloud aware workflows to deploy, migrate, terminate, scale up and scale down elastic platforms or applications.
ICWE2017 BigDataEurope
A paper about the BigDataEurope Project and Platform has been accepted for publication as a full-paper at ICWE2017 in Technical/Vision track.
Scaling AI in production using PyTorch
Slides from my talk at MLOps World' 21
Deploying AI models in production and scaling the ML services is still a big challenge. In this talk we will cover details of how to deploy your AI models, best practices for the deployment scenarios, and techniques for performance optimization and scaling the ML services. Come join us to learn how you can jumpstart the journey of taking your PyTorch models from Research to production.
PointNet
Journal club done with Vid Stojevic for PointNet:
https://arxiv.org/abs/1612.00593
https://github.com/charlesq34/pointnet
http://stanford.edu/~rqi/pointnet/
Deep learning for Indoor Point Cloud processing. PointNet, provides a unified architecture operating directly on unordered point clouds without voxelisation for applications ranging from object classification, part segmentation, to scene semantic parsing.
Alternative download link:
https://www.dropbox.com/s/ziyhgi627vg9lyi/3D_v2017_initReport.pdf?dl=0
Vector databases and neural search
Dmitry Kan, Principal AI Scientist at Silo AI and host of the Vector Podcast [1], will give an overview of the landscape of vector search databases and their role in NLP, along with the latest news and his view on the future of vector search. Further, he will share how he and his team participated in the Billion-Scale Approximate Nearest Neighbor Challenge and improved recall by 12% over a baseline FAISS.
Presented at https://www.meetup.com/open-nlp-meetup/events/282678520/
YouTube: https://www.youtube.com/watch?v=RM0uuMiqO8s&t=179s
Follow Vector Podcast to stay up to date on this topic: https://www.youtube.com/@VectorPodcast
Bne impact iif
Presentation introducing IMPACT Evaluation framework presented by Clemens Neudecker during demo session held at the BNE 5th of October 2011.
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
A long time ago in a database far, far away...
SQL was the only option to save vast amounts of application data for a long period of time. There were always some rebellion activities, to overcome the SQL Empire, which brought a new hope, but all other ways of storing data were never more than a phantom menace.
Now Cosmos DB awakens and is ready for the revenge of the NoSQL.
During this talk, we will have a look at what Azure Cosmos DB is, what you can achieve with its possibilities and how to use it in a galactic environment of data and applications.
Join me and find your way to the right solution for your application.
May the data be with you!
Big data serving: Processing and inference at scale in real time
Jon Bratseth (VP Architect) @ Verizon Media:
The big data world has mature technologies for offline analysis and learning from data, but have lacked options for making data-driven decisions in real time.
When it is sufficient to consider a single data point model servers such as TensorFlow serving can be used but in many cases you want to consider many data points to make decisions.
This is a difficult engineering problem combining state, distributed algorithms and low latency, but solving it often makes it possible to create far superior solutions when applying machine learning.
This talk will explain why this is a hard problem, show the advantages of solving it, and introduce the open source Vespa.ai platform which is used to implement such solutions in some of the largest scale problems in the world including the world's third largest ad serving system.
zenoh: The Edge Data Fabric
Zenoh is rapidly growing Eclipse project that unifies data in motion, data at rest and computations. It elegantly blends traditional pub/sub with geo distributed storage, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks. This presentation will provide an introduction to Eclipse Zenoh along with a crisp explanation of the challenges that motivated the creation of this project. We will go through a series of real-world use cases that demonstrate the advantages brought by Zenoh in enabling and optimising typical edge scenarios and in simplifying the development of any scale distributed applications.
Integrating Apache Phoenix with Distributed Query Engines
This talk will describe the work being done to create connectors for Presto and Apache Spark to read and write data in Phoenix tables. We will describe the new phoenix connector that implements Spark’s DataSource v2 API which will enable customizing and optimizing reads and writes to Phoenix tables.
We will also demo the Presto-phoenix connector, showing how it can be used to federate multiple Phoenix clusters and join Phoenix data with different types of data sources.
We will also describe some in progress work to more tightly integrate with the query optimizers of these frameworks in order to provide table statistics and push down filters, limits and aggregates into Phoenix whenever possible in order to speed up query execution.
Another area being worked on is to provide a way to support bulk loading using HFiles.
Deploy and Destroy: Testing Environments - Michael Arenzon - DevOpsDays Tel A...
One of the critical factors for development velocity is software correctness. Our ability to develop and ship new features fast is bounded by our ability to validate several aspects of the change: * Does the feature meet the requirements? * How does the feature affect existing code, and how can it affect the production environment? With continues codebase growth and new features being added, naturally our productivity decreases, and our need to improve the guarantees for quality and correctness increase.
In this talk, I’ll focus on testing environments: why developers need a self-serve platform to create a full functioning environment on-demand, how such environments should be managed, and how can one restore part of the lost velocity. I’ll cover an internal system we use at AppsFlyer called ‘Namespaces’ that addresses the issue with the help of Mesos / Marathon, Docker, Traefik, and Consul.
Similar to sos4R @ OGC TC (20)
[DRAFT] Workshop - Technical Introduction to joola.io
[DRAFT] Workshop - Technical Introduction to joola.io
BOTTARI: Location based Social Media Analysis with Semantic Web
BOTTARI: Location based Social Media Analysis with Semantic Web
Multiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier Dominguez
Design of Experiments on Federator Polystore Architecture
Design of Experiments on Federator Polystore Architecture
Frameworks provide structure. The core objective of the Big Data Framework is...
Frameworks provide structure. The core objective of the Big Data Framework is...
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Ultra Fast Deep Learning in Hybrid Cloud using Intel Analytics Zoo & Alluxio
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Big data serving: Processing and inference at scale in real time
Big data serving: Processing and inference at scale in real time
Integrating Apache Phoenix with Distributed Query Engines
Integrating Apache Phoenix with Distributed Query Engines
Deploy and Destroy: Testing Environments - Michael Arenzon - DevOpsDays Tel A...
Deploy and Destroy: Testing Environments - Michael Arenzon - DevOpsDays Tel A...
More from Daniel Nüst
RR & Docker @ MuensteR Meetup (Sep 2017)
A short introduction to reproducible research, reproducibility with R, Docker, and all together for reproducible research using R and Docker containers. Includes demos of Rocker and containerit.
containerit at useR!2017 conference, Brussels
**Webpage**
https://github.com/o2r-project/containerit/
**Abstract**
Reproducibility of computations is crucial in an era where data is born digital and analysed algorithmically. Most studies however only publish the results, often with figures as important interpreted outputs. But where do these figures come from? Scholarly articles must provide not only a description of the work but be accompanied by data and software. R offers excellent tools to create reproducible works, i.e. Sweave and RMarkdown. Several approaches to capture the workspace environment in R have been made, working around CRAN’s deliberate choice not to provide explicit versioning of packages and their dependencies. They preserve a collection of packages locally (packrat, pkgsnap, switchr/GRANBase) or remotely (MRAN timemachine/checkpoint), or install specific versions from CRAN or source (requireGitHub, devtools). Installers for old versions of R are archived on CRAN. A user can manually re-create a specific environment, but this is a cumbersome task.
We introduce a new possibility to preserve a runtime environment including both, packages and R, by adding an abstraction layer in the form of a container, which can execute a script or run an interactive session. The package containeRit automatically creates such containers based on Docker. Docker is a solution for packaging an application and its dependencies, but shows to be useful in the context of reproducible research (Boettiger 2015). The package creates a container manifest, the Dockerfile, which is usually written by hand, from sessionInfo(), R scripts, or RMarkdown documents. The Dockerfiles use the Rocker community images as base images. Docker can build an executable image from a Dockerfile. The image is executable anywhere a Docker runtime is present. containeRit uses harbor for building images and running containers, and sysreqs for installing system dependencies of R packages. Before the planned CRAN release we want to share our work, discuss open challenges such as handling linked libraries (see discussion on geospatial libraries in Rocker), and welcome community feedback.
containeRit is developed within the DFG-funded project Opening Reproducible Research to support the creation of Executable Research Compendia (ERC) (Nüst et al. 2017).
**References**
Boettiger, Carl. 2015. “An Introduction to Docker for Reproducible Research, with Examples from the R Environment.” ACM SIGOPS Operating Systems Review 49 (January): 71–79. doi:10.1145/2723872.2723882.
Nüst, Daniel, Markus Konkol, Edzer Pebesma, Christian Kray, Marc Schutzeichel, Holger Przibytzin, and Jörg Lorenz. 2017. “Opening the Publication Process with Executable Research Compendia.” D-Lib Magazine 23 (January). doi:10.1045/january2017-nuest.
Docker @ Data Science Meetup
Docker is a very useful tool in every data scientists toolbox. In this talk I present motivations to use Docker and made some live demos of typical tools used in data science, such as RStudio, Jupyter Notebook, or Elasticsearch.
Containers for sensor web services, applications and research @ Sensor Web Co...
Using Docker for research and production in the Sensor Web / OGC SWE domain with examples from 52°North Sensor Observation Service and Helgoland.
Docker @ FOSS4G 2016, Bonn
http://2016.foss4g.org/talks.html#146
Docker is a growing open-source platform for building and shipping applications as cloud services in so called containers. But containers can be more than that! Following the idea of DevOps, Dockerfiles are a complete scripted definition of an application with all it's dependencies, which can be build and published as ready to use images. As each container is only running "one thing" (e.g. one application, one database, a worker instance), multiple containers can be configured with the help of docker-compose.
More and more geospatial open source projects or third parties provide Dockerfiles. In this talk, we try to give an overview of the existing Docker images and docker-compose configurations for FOSS4G projects. We report on test runs that we conducted with them, informing about the evaluation results, target purposes, licenses, commonly used base images, and more. We will also give a short introduction into Docker and present the purposes that Docker images can be used for, such as easy evaluation for new users, education, testing, or common development environments.
This talk integrates and summarizes information from previous talks at FOSS4G and FOSSGIS conferences, so I'd like to thank Sophia Parafina, Jonathan Meyer, and Björn Schilberg for their contributions.
Atlas Zukünfte
Kurzpräsentation beim Werkstattgespräch "Atlas Zukünfte" des Leipniz-Institut für Länderkunder, Leipzig.
Wie sehen Atlanten der Zukunft aus?
Atlanten der Zukunft sind der Einstieg zu digitalen Informationen. Sie ermöglichen kritische Interaktionen mit Informationen, weil sie alle Bausteine (Daten, Quellcode, Analysecode, Software, Interaktionsschnittstellen) enthalten um sie zu durchdringen.
Sie ermöglichen dies weil sie änderbar (technisch, Lizenzen) und archivierbar sind.
Atlanten der Zukunft sind ausführbare Forschungskompendien (http://o2r.info/2016/04/08/o2r-at-EGU).
Frameworks for geoprocessing on the web with R
AGILE 2015 workshop presentation
Workshop website: http://tu-dresden.de/die_tu_dresden/fakultaeten/fakultaet_forst_geo_und_hydrowissenschaften/fachrichtung_geowissenschaften/gis/agile_gp_2015
Visualising Interpolations of Mobile Sensor Observations
Presentation at the GeoViz conference, Hamburg, 2013.
Abstract (excerpt): An integrated visualisation of observations’ locations and the interpolation of a dynamic phenomenon increases a user’s understanding of the processes underlying the measured data. The main contributions of this work are visualisation techniques, an implementation in a live 3D visualisation environment, and a subsequent user study.
The techniques are tailored to the challenge of mobile sensor data interpolations and focus on interactive exploration instead of extending interpolation methods as a first step.
JavaScript Client Libraries for the (Former) Long Tail of OGC Standards
Presented at FOSS4-G Europe 2014, Bremen
Authors:
Daniel Nüst (d.nuest@52north.org, 52°North GmbH)
Matthes Rieke (m.rieke@52north.org, 52°North GmbH)
Paul Breen (pbree@bas.ac.uk, British Antarctic Survey)
More and more information technology is moving into a cloud-based infrastructures for both data storage as well as user interfaces and leverages browser technologies, i.e. Javascript and HTML5, also for mobile devices. Users always use the latest version and the environment is well controlled: an internet browser. General purpose libraries (e.g. jQuery) and web-application frameworks (e.g. AngularJS) facilitate the development of complex applications. In the geospatial domain such frameworks and libraries are combined with mapping libraries, such as OpenLayers (OL) or Leaflet, and visualisation libraries to build complex applications. These applications display geospatial data coming from standardized view and feature services, most importantly the Open Geospatial Consortium’s (OGC) Web Map Service (WMS) and Web Features Service (WFS). Both server and client libraries are mature and have reached a very stable level and wide distribution.
What is missing today are generic libraries that operate at the same level of performance and quality to (i) access observation and time series data coming from OGC Sensor Observation Services (SOS), and (ii) control online geoprocesses published as an OGC Web Processing Service (WPS). These standards are less widespread than W(M,F)S but gain momentum as data volumes increase, for example with a myriad of smart sensors in the internet of things or new EO satellite missions, and subsequent requirements for sophisticated architectures for processing and management of time series data.
Observing these developments lead to the birth of two new open source Javascript library projects that are presented in this talk. SOS.js (https://github.com/52North/sos-js) can access SOS data and be used for sophisticated lightweight browser applications for discovering and displaying time series data as plots, tables, and maps. wps-js (https://github.com/52North/wps-js/) is a client library for the WPS generating forms based on the standardized metadata from the service and interactively creating and submitting processing tasks.
During the talk we demonstrate applications build with the libraries and share experiences from development. A goal for both libraries is to become independent of OL for request and response encoding and provide service access with a minimal footprint. We see an advantage of developing such small and focussed libraries maintained by field experts in these non-mainstream domains. We’ll happily discuss if this is the best approach and pose the following question: Is there a (technical, organisational) way to build a compatible Javascript client frameworks across all geo-service standards?
Open Source and GitHub for Teaching with Software Development Projects
Experiences in using GitHub for collaborative software development in project seminars using and creating open source software.
Authors:
Daniel Nüst (d.nuest@52north.org, 52°North Initiative for Geospatial Open Source Software GmbH)
Thomas Bartoschek (bartoschek@uni-muenster.de, Institute for Geoinformatics Münster)
Open source software is particularly suitable for teaching and organisations like Teaching Open Source (http://teachingopensource.org) present actively advertise this. In this talk we want to present some practical benefits that open source programming and publishing software on an open online platform has for teaching project-oriented software engineering seminars at university level. In these courses students together develop a new system for a specific task in form of a group project. For project groups, we suggest to use an adjusted variant of Scrum for project management (http://en.wikipedia.org/wiki/Scrum_%28software_development%29), git as source code management system (http://git-scm.com/), and GitHub as a collaboration platform (http://github.com/, https://education.github.com/). Thanks to GitHub’s collaboration models such as “fork & pull”, each student’s work, may they be in lines of code or contributions to a discussion, can be tracked. Students fulfil different tasks in a project setting: some develop, some spend their time issuing bugs or improving documentation. But for all of them GitHub allows to quantify contributions and set concrete goals, e.g. two pull requests created, one merged, and five issues written. GitHub also offers graphical overviews of project activities. The goal is of course not to expose the student but to create a transparent environment for evaluation and grading. Teachers can even weigh in on discussions and make suggestions on the same platform as the students.
In our experience, students estimate very well their performance in comparison with their colleagues. However, using Scrum as a development model is challenging for them. We adopted the classic Scrum schedule and defined two week long sprints. Students sometimes quarrel with the role of supervising other students and delegating tasks among their peers. But in the end, the clear schedule and the focus on the iterative and communicative aspects of project management are a key to ensure success. Teachers should be ready to step in a Scrum masters and to support the product owners and must be open to adjust plans and expectations in the same way that the students have to.
We think this approach can considerably increase quality of a course from both a teaching and a learning perspective.
5 Star Open Geoprocessing
A classification scheme for open web processes for geospatial data, inspired by the 5 star rating scheme of Linked Open Data.
The 52°North Web Processing Service
Feature description and demonstration of the 52°North implementation of the OGC Web Processing Service interface 1.0.0 along with plans for future development.
Prese
OGC SOS for Your Data
How can you publish your own datasets using the Open Geospatial Consortium's Sensor Observation Service Standard? We present straightforward solutions for the 52°North open source SOS implementation for both the stable and current development version.
sos4R - Accessing SensorWeb Data from R
Presentation of the package sos4R - a generic client to the OGC Sensor Observation Service for the R-project. It connects the Sensor Web with the most powederful statistical analysis and visualisation environment of today.
Visualizing the Availability of Temporally Structured Sensor Data
A crucial task in sensor web based analysis of spatio-temporal data is to get an overview on the spatial and temporal extent for which data is available. This work presents an approach for accessing the necessary information about the availability of temporally structured sensor data from sensor web
services. We show different kinds of data availability visualization. Based on the required values we specify a new generic sensor web service interface operation that constitutes the foundation for realizing the presented visualization methods.
More from Daniel Nüst (17)
Containers for sensor web services, applications and research @ Sensor Web Co...
Containers for sensor web services, applications and research @ Sensor Web Co...
Visualising Interpolations of Mobile Sensor Observations
Visualising Interpolations of Mobile Sensor Observations
JavaScript Client Libraries for the (Former) Long Tail of OGC Standards
JavaScript Client Libraries for the (Former) Long Tail of OGC Standards
Open Source and GitHub for Teaching with Software Development Projects
Open Source and GitHub for Teaching with Software Development Projects
Visualizing the Availability of Temporally Structured Sensor Data
Visualizing the Availability of Temporally Structured Sensor Data
Recently uploaded
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
GridMate - End to end testing is a critical piece to ensure quality and avoid...
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
UiPath Test Automation using UiPath Test Suite series, part 4
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
The Art of the Pitch: WordPress Relationships and Sales
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Video Streaming: Then, Now, and in the Future
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Recently uploaded (20)
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Free Complete Python - A step towards Data Science
Free Complete Python - A step towards Data Science
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
sos4R @ OGC TC
- 1. Daniel Nüst 52°North Initiative forGeospatial Open Source Software GmbH
- 2. motivation
- 3. open dataandreproducibleresearch needfor SWE analysisclients spread SOS power of R
- 4. standard client server implementation
- 8. features client classesandmethods exchangeability filtering tests
- 9. issues
- 10. live demo
- 17. http://52north.org/geostatistics Workshop Handling and analyzing spatio-temporal data in R 21-22 March 2011, Münster http://opengeostatistics.org
- 18. tasksandroadmap
- 19. thankyou! contact: d.nuest@52north.org blog: www.nordholmen.net/sos4r community: http://52north.org/geostatistics/
- 22. exchangeability myParseSensorML <- function(obj) { root <- xmlRoot(obj) return(xmlName(root)) } mysos = SOS( url = "http://www.sos.de/sos", parsers = SosParsingFunctions( "DescribeSensor" = myParseSensorML) )