OGR2OSM is a tool that converts geospatial data into OpenStreetMap (.osm) format. It can read data from any format supported by OGR, reproject it to EPSG:4326 if needed, and use user-written Python functions to convert source fields to OSM tags. This allows complex logic for accurate tagging. The tool processes layers sequentially, applying functions like filtering, reprojecting, converting geometries to OSM formats, and filtering tags. It outputs OSM XML that can be opened in JOSM.
The document introduces a plan to cover Java and Android basics, Android UI topics, and Android in action over three parts. Part I provides an introduction to Java concepts like the JVM, garbage collection, threads, and an overview of the Android architecture and building blocks. It also demonstrates an NDK example. The document outlines the topics that will be covered in each part of the training.
The National Congress of Argentina houses the Delegates' Chamber and Senators' Chamber and exercises legislative power. The Pink House is the executive branch's seat and the office of the President of Argentina, painted pink under Sarmiento's presidency. The Colon Theatre is one of the most important opera houses in the world for its size and spectacular acoustics. Recoleta is a central suburb known for its historical architecture, expensive real estate, and the Recoleta Cemetery. Plaza de Mayo is the foundational site of Buenos Aires located in the downtown area opposite the Metropolitan Cathedral, Buenos Aires' main Catholic church.
This document provides an overview of installing and deploying Apache Spark, including:
1. Spark can be installed via prebuilt packages or by building from source.
2. Spark runs in local, standalone, YARN, or Mesos cluster modes and the SparkContext is used to connect to the cluster.
3. Jobs are deployed to the cluster using the spark-submit script which handles building jars and dependencies.
Unity - Internals: memory and performanceCodemotion
by Marco Trivellato - In this presentation we will provide in-depth knowledge about the Unity runtime. The first part will focus on memory and how to deal with fragmentation and garbage collection. The second part will cover implementation details and their memory vs cycles tradeoffs in both Unity4 and the upcoming Unity5.
The document summarizes the evolution of Java from versions 8 to the upcoming version 19. It outlines the major new features introduced in each version, including module system in Java 9, local variable type inference in Java 10, and records in Java 16. It also discusses some ongoing projects that aim to further enhance Java, such as Project Loom, Project Panama, Project Valhalla and Project Amber. Key language concepts like encapsulation, private methods in interfaces, pattern matching and sealed classes are briefly explained.
The document discusses Emergent Game Technologies' Floodgate cross-platform stream processing library. It describes Floodgate as a foundation for easing multi-core development across platforms like PC, Xbox, PS3 and Wii. It outlines how Floodgate uses a stream processing model to partition work into tasks that can run concurrently, improving performance by taking advantage of multiple cores. Examples are given showing how tasks like skinning and morphing benefit from being offloaded to Floodgate.
Decima Engine: Visibility in Horizon Zero DawnGuerrilla
Download the full presentation here: http://www.guerrilla-games.com/read/decima-engine-visibility-in-horizon-zero-dawn
Abstract: Horizon Zero Dawn presented the Decima engine with new challenges in rendering large and dense environments. In particular, we needed to be able to quickly query a very large set of potential objects to find which should be visible. This talk looks at the problems we faced moving from more constrained Killzone levels to Horizon's open world, and our approach to fast visibility queries using the PS4's asynchronous compute hardware. It also covers our recent work on efficiently collecting batches of object instances during the query to reduce load on the entire rendering pipeline. A basic familiarity with GPU compute will be helpful to get the most out of this talk.
The document introduces a plan to cover Java and Android basics, Android UI topics, and Android in action over three parts. Part I provides an introduction to Java concepts like the JVM, garbage collection, threads, and an overview of the Android architecture and building blocks. It also demonstrates an NDK example. The document outlines the topics that will be covered in each part of the training.
The National Congress of Argentina houses the Delegates' Chamber and Senators' Chamber and exercises legislative power. The Pink House is the executive branch's seat and the office of the President of Argentina, painted pink under Sarmiento's presidency. The Colon Theatre is one of the most important opera houses in the world for its size and spectacular acoustics. Recoleta is a central suburb known for its historical architecture, expensive real estate, and the Recoleta Cemetery. Plaza de Mayo is the foundational site of Buenos Aires located in the downtown area opposite the Metropolitan Cathedral, Buenos Aires' main Catholic church.
This document provides an overview of installing and deploying Apache Spark, including:
1. Spark can be installed via prebuilt packages or by building from source.
2. Spark runs in local, standalone, YARN, or Mesos cluster modes and the SparkContext is used to connect to the cluster.
3. Jobs are deployed to the cluster using the spark-submit script which handles building jars and dependencies.
Unity - Internals: memory and performanceCodemotion
by Marco Trivellato - In this presentation we will provide in-depth knowledge about the Unity runtime. The first part will focus on memory and how to deal with fragmentation and garbage collection. The second part will cover implementation details and their memory vs cycles tradeoffs in both Unity4 and the upcoming Unity5.
The document summarizes the evolution of Java from versions 8 to the upcoming version 19. It outlines the major new features introduced in each version, including module system in Java 9, local variable type inference in Java 10, and records in Java 16. It also discusses some ongoing projects that aim to further enhance Java, such as Project Loom, Project Panama, Project Valhalla and Project Amber. Key language concepts like encapsulation, private methods in interfaces, pattern matching and sealed classes are briefly explained.
The document discusses Emergent Game Technologies' Floodgate cross-platform stream processing library. It describes Floodgate as a foundation for easing multi-core development across platforms like PC, Xbox, PS3 and Wii. It outlines how Floodgate uses a stream processing model to partition work into tasks that can run concurrently, improving performance by taking advantage of multiple cores. Examples are given showing how tasks like skinning and morphing benefit from being offloaded to Floodgate.
Decima Engine: Visibility in Horizon Zero DawnGuerrilla
Download the full presentation here: http://www.guerrilla-games.com/read/decima-engine-visibility-in-horizon-zero-dawn
Abstract: Horizon Zero Dawn presented the Decima engine with new challenges in rendering large and dense environments. In particular, we needed to be able to quickly query a very large set of potential objects to find which should be visible. This talk looks at the problems we faced moving from more constrained Killzone levels to Horizon's open world, and our approach to fast visibility queries using the PS4's asynchronous compute hardware. It also covers our recent work on efficiently collecting batches of object instances during the query to reduce load on the entire rendering pipeline. A basic familiarity with GPU compute will be helpful to get the most out of this talk.
The document summarizes Linux memory management architecture. It has two parts - an architecture independent memory model and implementation for a specific architecture. The memory model divides virtual address space into pages and uses page tables to map virtual to physical addresses. It also describes data structures like page table entries and region descriptors that track memory mappings.
This document provides an overview of OSPF and EIGRP routing protocols including how they work, configure, and troubleshoot. It describes key concepts such as how OSPF uses the Dijkstra algorithm to calculate the shortest path and elect designated routers, and how EIGRP uses the DUAL algorithm and has characteristics of both distance vector and link state protocols. It also provides configuration examples and show commands for setting up and monitoring OSPF and EIGRP routing.
This document describes how to set up a single-node Hadoop installation to perform MapReduce operations. It discusses supported platforms, required software including Java and SSH, and preparing the Hadoop cluster in either local, pseudo-distributed, or fully-distributed mode. The main components of the MapReduce execution pipeline are explained, including the driver, mapper, reducer, and input/output formats. Finally, a simple word count example MapReduce job is described to demonstrate how it works.
Newer version about JDK10 and 11 is here
https://www.slideshare.net/nowokay/summary-of-jdk10-and-what-will-come-into-jdk11-99363835
The material for the presentation of the JJUG CCC 2018 Spring
This document summarizes the key features and enhancements in Apache YARN 3.1 and beyond. It discusses improvements to scheduling such as global scheduling for better placement decisions and multi-node scheduling for faster allocation. New features like node attributes, absolute resource configuration, auto queue creation, and timeline service V2 are covered. Support for GPUs, FPGAs, and TensorFlow on YARN is also summarized. The presentation provides an overview of community updates and highlights upcoming features in YARN 3.2.
Introduction to Docker (and a bit more) at LSPE meetup SunnyvaleJérôme Petazzoni
What's Docker, why does it matter, how does it use Linux Containers, why should you use it, and how? You'll find answers to those questions (and a bit more) in this presentation, given February 20th 2014 at the Large Scale Production Engineering Meet-Up at Yahoo, in Sunnyvale.
This document summarizes an introduction to Java 8 streams presentation given on September 2, 2015 in Louvain-La-Neuve, Belgium by Marc Tritchler. The presentation covered the history and features of Java 8 streams including lambda expressions, default methods, and functional interfaces. It provided examples of creating streams from collections, files, and primitive types as well as common stream operations like filter, map, and forEach. The ordering and laziness of intermediate operations was discussed.
This document summarizes an introduction to Java 8 streams presentation given on September 2, 2015 in Louvain-La-Neuve, Belgium by Marc Tritchler. The presentation covered the history and features of Java 8 streams including lambda expressions, default methods, and functional interfaces. It provided examples of creating streams from collections, files, and primitive types as well as intermediate operations like filter, map, and sorted and terminal operations like forEach, reduce, and collect. It discussed lazy evaluation, processing order, and stateful vs stateless operations.
Spark's distributed programming model uses resilient distributed datasets (RDDs) and a directed acyclic graph (DAG) approach. RDDs support transformations like map, filter, and actions like collect. Transformations are lazy and form the DAG, while actions execute the DAG. RDDs support caching, partitioning, and sharing state through broadcasts and accumulators. The programming model aims to optimize the DAG through operations like predicate pushdown and partition coalescing.
Blocks is a cool concept and is very much needed for performance improvements and responsiveness. GCD helps run blocks effortlessly by scheduling on a desired queue, priority and lots more.
This document provides an overview of the erLocator application, which allows users to efficiently find other nearby users on a map without performing direct distance calculations. It stores user locations grouped by region using a geohashing algorithm. The no-SQL Redis backend stores these regions and user data. A simple Erlang backend interfaces with Redis via Redo and exposes an API through the Mochiweb web server. Documentation is generated with eDoc. Builds are managed by Rebar. The open source code is hosted on GitHub under the MIT license.
This document provides an introduction to Docker and Openshift including discussions around infrastructure, storage, monitoring, metrics, logs, backup, and security considerations. It describes the recommended infrastructure for a 3 node Openshift cluster including masters, etcd, and nodes. It also discusses strategies for storage, monitoring both internal pod status and external infrastructure metrics, collecting and managing logs, backups, and security features within Openshift like limiting resource usage and isolating projects.
This document discusses parallel computing with GPUs. It introduces parallel computing, GPUs, and CUDA. It describes how GPUs are well-suited for data-parallel applications due to their large number of cores and throughput-oriented design. The CUDA programming model is also summarized, including how kernels are launched on the GPU from the CPU. Examples are provided of simple CUDA programs to perform operations like squaring elements in parallel on the GPU.
The document provides an overview of writing a Linux USB device driver. It discusses key concepts such as loadable kernel modules (LKMs), registering and unregistering a device driver, common device driver types, and important kernel functions for device drivers. It also outlines the general steps to implement a device driver, including understanding the device, mapping operations to file operations, creating a special device file, and loading the driver module.
Bsdtw17: george neville neil: realities of dtrace on free-bsdScott Tsai
This document summarizes a talk on the history and current state of DTrace, a dynamic tracing framework originally developed for Solaris and later ported to FreeBSD and MacOS. It discusses how DTrace has been used for performance analysis, distributed systems tracing, and teaching operating systems. Recent improvements include machine-readable output, new providers, and performance tuning. Future work includes the OpenDTrace cross-platform project and improving the D programming language used to write probes.
Finding Xori: Malware Analysis Triage with Automated DisassemblyPriyanka Aash
"In a world of high volume malware and limited researchers we need a dramatic improvement in our ability to process and analyze new and old malware at scale. Unfortunately what is currently available to the community is incredibly cost prohibitive or does not rise to the challenge. As malware authors and distributors share code and prepackaged tool kits, the corporate sponsored research community is dominated by solutions aimed at profit as opposed to augmenting capabilities available to the broader community. With that in mind, we are introducing our library for malware disassembly called Xori as an open source project. Xori is focused on helping reverse engineers analyze binaries, optimizing for time and effort spent per sample.
Xori is an automation-ready disassembly and static analysis library that consumes shellcode or PE binaries and provides triage analysis data. This Rust library emulates the stack, register states, and reference tables to identify suspicious functionality for manual analysis. Xori extracts structured data from binaries to use in machine learning and data science pipelines.
We will go over the pain-points of conventional open source disassemblers that Xori solves, examples of identifying suspicious functionality, and some of the interesting things we've done with the library. We invite everyone in the community to use it, help contribute and make it an increasingly valuable tool for researchers alike."
DConf2015 - Using D for Development of Large Scale Primary StorageLiran Zvibel
The talk will discuss using D for a large scale distributed project implementing a primary storage system with strict performance and resources requirements. Will go over the pros and cons of using D and compare our experience to previous similar projects implemented in C and C++ with Python.
We are an experienced group of system programmers, implementing a large software based storage system. We have leveraged D specific features to make sure we have a very sound infrastructure to use, some of the things we did were previously either impossible or impractical using only C or C++ forcing us to use Python for code generation, on the other hand—some D aspects make it more difficult to handle than the other options. I will present what we really like, and what we’ve learned to live with.
This document provides an overview of embedded Linux for an embedded systems design course. It discusses various commercial and open source embedded Linux distributions and their characteristics. It also covers important topics for embedded Linux including tool chains, the Linux kernel, debugging, driver development, memory management, and synchronization techniques. Example code snippets are provided for common Linux system programming tasks like file I/O, processes, threads, IPC, signals, and sockets. Cross-compiling for embedded targets is also briefly explained.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
The document summarizes Linux memory management architecture. It has two parts - an architecture independent memory model and implementation for a specific architecture. The memory model divides virtual address space into pages and uses page tables to map virtual to physical addresses. It also describes data structures like page table entries and region descriptors that track memory mappings.
This document provides an overview of OSPF and EIGRP routing protocols including how they work, configure, and troubleshoot. It describes key concepts such as how OSPF uses the Dijkstra algorithm to calculate the shortest path and elect designated routers, and how EIGRP uses the DUAL algorithm and has characteristics of both distance vector and link state protocols. It also provides configuration examples and show commands for setting up and monitoring OSPF and EIGRP routing.
This document describes how to set up a single-node Hadoop installation to perform MapReduce operations. It discusses supported platforms, required software including Java and SSH, and preparing the Hadoop cluster in either local, pseudo-distributed, or fully-distributed mode. The main components of the MapReduce execution pipeline are explained, including the driver, mapper, reducer, and input/output formats. Finally, a simple word count example MapReduce job is described to demonstrate how it works.
Newer version about JDK10 and 11 is here
https://www.slideshare.net/nowokay/summary-of-jdk10-and-what-will-come-into-jdk11-99363835
The material for the presentation of the JJUG CCC 2018 Spring
This document summarizes the key features and enhancements in Apache YARN 3.1 and beyond. It discusses improvements to scheduling such as global scheduling for better placement decisions and multi-node scheduling for faster allocation. New features like node attributes, absolute resource configuration, auto queue creation, and timeline service V2 are covered. Support for GPUs, FPGAs, and TensorFlow on YARN is also summarized. The presentation provides an overview of community updates and highlights upcoming features in YARN 3.2.
Introduction to Docker (and a bit more) at LSPE meetup SunnyvaleJérôme Petazzoni
What's Docker, why does it matter, how does it use Linux Containers, why should you use it, and how? You'll find answers to those questions (and a bit more) in this presentation, given February 20th 2014 at the Large Scale Production Engineering Meet-Up at Yahoo, in Sunnyvale.
This document summarizes an introduction to Java 8 streams presentation given on September 2, 2015 in Louvain-La-Neuve, Belgium by Marc Tritchler. The presentation covered the history and features of Java 8 streams including lambda expressions, default methods, and functional interfaces. It provided examples of creating streams from collections, files, and primitive types as well as common stream operations like filter, map, and forEach. The ordering and laziness of intermediate operations was discussed.
This document summarizes an introduction to Java 8 streams presentation given on September 2, 2015 in Louvain-La-Neuve, Belgium by Marc Tritchler. The presentation covered the history and features of Java 8 streams including lambda expressions, default methods, and functional interfaces. It provided examples of creating streams from collections, files, and primitive types as well as intermediate operations like filter, map, and sorted and terminal operations like forEach, reduce, and collect. It discussed lazy evaluation, processing order, and stateful vs stateless operations.
Spark's distributed programming model uses resilient distributed datasets (RDDs) and a directed acyclic graph (DAG) approach. RDDs support transformations like map, filter, and actions like collect. Transformations are lazy and form the DAG, while actions execute the DAG. RDDs support caching, partitioning, and sharing state through broadcasts and accumulators. The programming model aims to optimize the DAG through operations like predicate pushdown and partition coalescing.
Blocks is a cool concept and is very much needed for performance improvements and responsiveness. GCD helps run blocks effortlessly by scheduling on a desired queue, priority and lots more.
This document provides an overview of the erLocator application, which allows users to efficiently find other nearby users on a map without performing direct distance calculations. It stores user locations grouped by region using a geohashing algorithm. The no-SQL Redis backend stores these regions and user data. A simple Erlang backend interfaces with Redis via Redo and exposes an API through the Mochiweb web server. Documentation is generated with eDoc. Builds are managed by Rebar. The open source code is hosted on GitHub under the MIT license.
This document provides an introduction to Docker and Openshift including discussions around infrastructure, storage, monitoring, metrics, logs, backup, and security considerations. It describes the recommended infrastructure for a 3 node Openshift cluster including masters, etcd, and nodes. It also discusses strategies for storage, monitoring both internal pod status and external infrastructure metrics, collecting and managing logs, backups, and security features within Openshift like limiting resource usage and isolating projects.
This document discusses parallel computing with GPUs. It introduces parallel computing, GPUs, and CUDA. It describes how GPUs are well-suited for data-parallel applications due to their large number of cores and throughput-oriented design. The CUDA programming model is also summarized, including how kernels are launched on the GPU from the CPU. Examples are provided of simple CUDA programs to perform operations like squaring elements in parallel on the GPU.
The document provides an overview of writing a Linux USB device driver. It discusses key concepts such as loadable kernel modules (LKMs), registering and unregistering a device driver, common device driver types, and important kernel functions for device drivers. It also outlines the general steps to implement a device driver, including understanding the device, mapping operations to file operations, creating a special device file, and loading the driver module.
Bsdtw17: george neville neil: realities of dtrace on free-bsdScott Tsai
This document summarizes a talk on the history and current state of DTrace, a dynamic tracing framework originally developed for Solaris and later ported to FreeBSD and MacOS. It discusses how DTrace has been used for performance analysis, distributed systems tracing, and teaching operating systems. Recent improvements include machine-readable output, new providers, and performance tuning. Future work includes the OpenDTrace cross-platform project and improving the D programming language used to write probes.
Finding Xori: Malware Analysis Triage with Automated DisassemblyPriyanka Aash
"In a world of high volume malware and limited researchers we need a dramatic improvement in our ability to process and analyze new and old malware at scale. Unfortunately what is currently available to the community is incredibly cost prohibitive or does not rise to the challenge. As malware authors and distributors share code and prepackaged tool kits, the corporate sponsored research community is dominated by solutions aimed at profit as opposed to augmenting capabilities available to the broader community. With that in mind, we are introducing our library for malware disassembly called Xori as an open source project. Xori is focused on helping reverse engineers analyze binaries, optimizing for time and effort spent per sample.
Xori is an automation-ready disassembly and static analysis library that consumes shellcode or PE binaries and provides triage analysis data. This Rust library emulates the stack, register states, and reference tables to identify suspicious functionality for manual analysis. Xori extracts structured data from binaries to use in machine learning and data science pipelines.
We will go over the pain-points of conventional open source disassemblers that Xori solves, examples of identifying suspicious functionality, and some of the interesting things we've done with the library. We invite everyone in the community to use it, help contribute and make it an increasingly valuable tool for researchers alike."
DConf2015 - Using D for Development of Large Scale Primary StorageLiran Zvibel
The talk will discuss using D for a large scale distributed project implementing a primary storage system with strict performance and resources requirements. Will go over the pros and cons of using D and compare our experience to previous similar projects implemented in C and C++ with Python.
We are an experienced group of system programmers, implementing a large software based storage system. We have leveraged D specific features to make sure we have a very sound infrastructure to use, some of the things we did were previously either impossible or impractical using only C or C++ forcing us to use Python for code generation, on the other hand—some D aspects make it more difficult to handle than the other options. I will present what we really like, and what we’ve learned to live with.
This document provides an overview of embedded Linux for an embedded systems design course. It discusses various commercial and open source embedded Linux distributions and their characteristics. It also covers important topics for embedded Linux including tool chains, the Linux kernel, debugging, driver development, memory management, and synchronization techniques. Example code snippets are provided for common Linux system programming tasks like file I/O, processes, threads, IPC, signals, and sockets. Cross-compiling for embedded targets is also briefly explained.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
"Choosing proper type of scaling", Olena SyrotaFwdays
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
Discover top-tier mobile app development services, offering innovative solutions for iOS and Android. Enhance your business with custom, user-friendly mobile applications.
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...Alex Pruden
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Essentials of Automations: Exploring Attributes & Automation ParametersSafe Software
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
5th LF Energy Power Grid Model Meet-up SlidesDanBrown980551
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyScyllaDB
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
AppSec PNW: Android and iOS Application Security with MobSFAjin Abraham
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...Jason Yip
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
4. History
4
Written in 2009 by Iván Sánchez Ortega
Rewritten 2012 by Andrew Guertin for UVM
buildings
I now maintain it
5. Features
5
Can read any ogr supported data source
.shp, .mdb, .gdb, sqlite, etc
Reprojects if necessary – eliminates a step with many
sources
Works with multiple layer sources or shapefile
directories
Uses python translation functions that you write to
convert source field values to OSM tags
This allows you to use complicated logic to get the tagging right
Documentation
6. Installing
6
Requires gdal with python bindings
Simply sudo apt-get install python-gdal git on
Ubuntu
May require compiling gdal from source and third-
party SDKs for some formats (.mdb, .gdb)
Run git clone --recursive
https://github.com/pnorman/ogr2osm to install
Full instructions at
https://github.com/pnorman/ogr2osm
7. Code flow
7
Read in data source Process each layer Merge nodes
• Uses python ogr bindings • Converts from ogr to osm • Merges duplicate nodes
to read the files tagging and objects • Adjustable threshold for
distance
preOutputTransform() Output XML
• A user-defined filtering • Write to a .osm file that
step, not commonly used can be opened in JOSM
8. Code flow
8
Read in data source Process each layer Merge nodes
• Uses python ogr bindings • Converts from ogr to osm • Merges duplicate nodes
to read the files tagging and objects • Adjustable threshold for
distance
preOutputTransform() Output XML
• A user-defined filtering • Write to a .osm file that
step, not commonly used can be opened in JOSM
9. Code flow
9
Read in data source Process each layer Merge nodes
• Uses python ogr bindings • Converts from ogr to osm • Merges duplicate nodes
to read the files tagging and objects • Adjustable threshold for
distance
preOutputTransform() Output XML
• A user-defined filtering • Write to a .osm file that
step, not commonly used can be opened in JOSM
10. Layer processing
10
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering step,
EPSG:4326 • Creates nodes and ways not commonly used
• Only creates multipolygons if
necessary
11. Layer processing
11
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering
EPSG:4326 • Creates nodes and ways step, not commonly used
• Only creates multipolygons if
necessary
12. Layer processing
12
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering step,
EPSG:4326 • Creates nodes and ways not commonly used
• Only creates multipolygons if
necessary
13. Layer processing
13
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering step,
EPSG:4326 • Creates nodes and ways not commonly used
• Only creates multipolygons if
necessary
14. Layer processing
14
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering step,
EPSG:4326 • Creates nodes and ways not commonly used
• Only creates multipolygons if
necessary
15. Layer processing
15
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering
EPSG:4326 • Creates nodes and ways step, not commonly used
• Only creates multipolygons if
necessary
16. Layer processing
16
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering
EPSG:4326 • Creates nodes and ways step, not commonly used
• Only creates multipolygons if
necessary
17. Layer processing
17
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering step,
EPSG:4326 • Creates nodes and ways not commonly used
• Only creates multipolygons if
necessary
18. Layer processing
18
filterLayer() Reproject filterFeature()
• Allows layers to be dropped • Projects the layer into • Allows features to be
• Allows for the creation of new EPSG:4326 removed
fields
• e.g. a field that indicates the
layer of a feature for later
Reproject Convert to OSM filterTags() filterFeaturePost()
• Projects the feature into geometries • Where all the magic occurs • A user-defined filtering step,
EPSG:4326 • Creates nodes and ways not commonly used
• Only creates multipolygons if
necessary
19. Code flow
19
Read in data source Process each layer Merge nodes
• Uses python ogr bindings • Converts from ogr to osm • Merges duplicate nodes
to read the files tagging and objects • Adjustable threshold for
distance
preOutputTransform() Output XML
• A user-defined filtering • Write to a .osm file that
step, not commonly used can be opened in JOSM
20. Code flow
20
Read in data source Process each layer Merge nodes
• Uses python ogr bindings • Converts from ogr to osm • Merges duplicate nodes
to read the files tagging and objects • Adjustable threshold for
distance
preOutputTransform() Output XML
• A user-defined filtering • Write to a .osm file that
step, not commonly used can be opened in JOSM
21. Code flow
21
Read in data source Process each layer Merge nodes
• Uses python ogr bindings • Converts from ogr to osm • Merges duplicate nodes
to read the files tagging and objects • Adjustable threshold for
distance
preOutputTransform() Output XML
• A user-defined filtering • Write to a .osm file that
step, not commonly used can be opened in JOSM
22. Surrey case study
22
Shapefile fields similar to other government GIS
sources
Fields or values periodically change with no notice
58 layers in 7 zip files
Not counting orthos and LIDAR-derived contours
153 MB compressed, 1.7 GB uncompressed
Covers 187 km2
Too much data to write conversions for without a
method
25. Reduce the amount of data
25
ogr2osm will happily turn out a gigabyte .osm but
good luck opening it
Use ogr2ogr -spat to trim the input files down
Converting from some formats to shapefiles will
truncate field names
Can use .gdb when coming from a format with long field
names and layers
-spat wants coordinates in layer coordinate system
Use gdaltransform to turn latitude/longitude into desired
coordinates
26. Drop layers
26
def filterLayer(layer):
Use the layer layername = layer.GetName()
if layername in ('WBD_HU2', 'WBD_HU4', 'WBD_HU6'):
translation (-t layer) return
and see what layers if layername not in ('NHDArea', 'NHDAreaEventFC'):
print 'Unknown layer ' + layer.GetName()
should be dropped field = ogr.FieldDefn('__LAYER', ogr.OFTString)
field.SetWidth(len(layername))
Most multi-layer layer.CreateField(field)
sources have layers for j in range(layer.GetFeatureCount()):
ogrfeature = layer.GetNextFeature()
that should not be ogrfeature.SetField('__LAYER', layername)
layer.SetFeature(ogrfeature)
imported layer.ResetReading()
return layer
In the case of the
Surrey data filtering is
done in the script that
downloads the data
27. Writing a good filterTags(attrs)
27
When testing you def filterTags(attrs):
if not attrs: return
tags = {}
want unknown fields if '__LAYER' in attrs and attrs['__LAYER'] ==
to be kept 'wtrHydrantsSHP':
# Delete the warranty date
if 'WARR_DATE' in attrs: del attrs['WARR_DATE']
Delete items from if 'HYDRANT_NO' in attrs:
tags['ref'] = attrs['HYDRANT_NO'].strip()
attrs as you convert del attrs['HYDRANT_NO']
elif '__LAYER' in attrs and attrs['__LAYER'] ==
them to OSM tags 'trnRoadCentrelinesSHP':
# ... More logic ...
Delete fields which for k,v in attrs.iteritems():
if v.strip() != '' and not k in tags:
shouldn’t be tags[k]=v
return tags
converted to an OSM
tag
28. What not to include
28
Duplications of geodata
SHAPE_AREA, SHAPE_LENGTH, latitude and
longitude
Unnecessary meta-data
e.g. username of the last person in the GIS
department to edit the object
A single object ID can be useful but generally isn’t
A good translation will often drop more than it
includes
29. Identify the main field
29
COMMENTS LC_COST RIGHTTO
Convert to .osm with no CONDDATE LEFTFROM ROADCODE
CONDTN LEFTTO ROAD_NAME
translation DATECLOSED LEGACYID ROW_WIDTH
DATECONST LOCATION SNW_RTEZON
View statistics about DESIGNTN
DISR_ROUTE
MATERIAL
MRN
SPEED
STATUS
tags FAC_ID
GCNAME
NO_LANE
OWNER
STR_ROUTE
TRK_ROUTE
Easiest way is to open GCPREDIR
GCROADS
PAV_DATE
PROJ_NO
WAR_DATE
WTR_PRIOR
in JOSM, select GCSUFDIR RC_TYPE WTR_VEHCL
GCTYPE RC_TYPE2 YR
‐untagged, select the GIS_ES RD_CLASS YTD_COST
GREENWAY RIGHTFROM
tags, paste into a text
editor NOT INCLUDED IN ROADS TRANSLATION
NOT INCLUDED IN ANY TRANSLATION
MAIN FIELD
Need to look at a large
area for this
30. The main field
30
A numeric field and a text RC_TYPE RC_TYPE2 Count Tagging
field in this case 0 Road 11375 highway=?
Don’t trust field 1 Frontage
Road
38 highway=residential
descriptions when writing 2 Highway highway=motorway_link
54
OSM tagging Interchange
3 Street Lane 20 highway=service
Always verify!
4 Access Lane highway=?
Access Lane would be 1442
highway=service from the 5 Railway 28 railway=rail
description but this would
be wrong
Use imagery, surveys or
other sources
31. Looking at a value in more detail
31
Should be carried out RD_CLAS highway= Coun
for each value, even if S t
you think you’re sure Local residential 8284
on the tagging Major tertiary 1350
Collector
Look at all tags for primary
just those matching Arterial 1583
secondary
the field value tertiary
Provincial motorway 156
In this case search in primary
JOSM for Highway
RC_TYPE2="Road" Translink unclassified 1
32. Even more detail
32
Gets very close to MRN highway= Count
OSM tagging practice Yes secondary 504
locally No tertiary 1079
Loss of information
with Arterial MRN=No
and Major Collector
both mapping to
tertiary
Does this matter in
this case? No, road
classifications require
some judgment
33. Dropping objects
33
def filterFeature(ogrfeature, fieldNames, reproject):
You may come across if not ogrfeature: return
objects that you index = ogrfeature.GetFieldIndex('STATUS')
if index >= 0 and ogrfeature.GetField(index) in
('History', 'For Construction', 'Proposed'):
shouldn’t add to OSM return None
return ogrfeature
In this case there are
“paper roads” in the
data
Use filterFeature() to
remove these
34. Putting it all together
34
def filterLayer(layer):
layername = layer.GetName()
field = ogr.FieldDefn('__LAYER', ogr.OFTString)
Code presented is a
field.SetWidth(len(layername))
layer.CreateField(field)
for j in range(layer.GetFeatureCount()):
simplification and
ogrfeature = layer.GetNextFeature()
ogrfeature.SetField('__LAYER', layername)
layer.SetFeature(ogrfeature)
does not deal with
layer.ResetReading()
return layer all fields
def filterFeature(ogrfeature, fieldNames, reproject):
if not ogrfeature: return
Filter features and
index = ogrfeature.GetFieldIndex('STATUS')
if index >= 0 and ogrfeature.GetField(index) in
('History', 'For Construction', 'Proposed'):
return None
layers
return ogrfeature
35. Putting it all together
35
def filterTags(attrs):
if not attrs: return elif 'RD_CLASS' in attrs and attrs['RD_CLASS'] == 'Provincial Highway':
tags = {} # Special-case motorways
if 'ROAD_NAME' in attrs and attrs['ROAD_NAME'] in
if '__LAYER' in attrs and attrs['__LAYER'] == ('No 1 Hwy', 'No 99 Hwy'):
'trnRoadCentrelinesSHP': tags['highway'] = 'motorway'
if 'COMMENTS' in attrs: del attrs['COMMENTS'] else:
if 'DATECLOSED' in attrs: del attrs['DATECLOSED'] tags['highway'] = 'primary'
# Lots more to delete del attrs['RD_CLASS']
elif 'RD_CLASS' in attrs and attrs['RD_CLASS'] == 'Translink':
if 'NO_LANE' in attrs: tags['highway'] = 'unclassified'
tags['lanes'] = attrs['NO_LANE'].strip() del attrs['RD_CLASS']
del attrs['NO_LANE'] else:
l.error('trnRoadCentrelinesSHP RC_TYPE=0 logic fell through')
if 'RC_TYPE' in attrs and attrs['RC_TYPE'].strip() == '0': # Normal roads tags['fixme'] = 'yes'
del attrs['RC_TYPE'] tags['highway'] = 'road'
if 'RC_TYPE2' in attrs: del attrs['RC_TYPE2'] elif 'RC_TYPE' in attrs and attrs['RC_TYPE'].strip() == '1':
if 'RD_CLASS' in attrs and attrs['RD_CLASS'] == 'Local': # More logic
tags['highway'] = 'residential'
del attrs['RD_CLASS'] elif '__LAYER' in attrs and attrs['__LAYER'] == 'trnTrafficSignalsSHP':
elif 'RD_CLASS' in attrs and attrs['RD_CLASS'] == 'Major Collector': # More logic
tags['highway'] = 'tertiary'
del attrs['RD_CLASS'] for k,v in attrs.iteritems():
elif 'RD_CLASS' in attrs and attrs['RD_CLASS'] == 'Arterial': if v.strip() != '' and not k in tags:
if 'ROAD_NAME' in attrs and attrs['ROAD_NAME'] in tags[k]=v
('King George Blvd', 'Fraser Hwy'):
tags['highway'] = 'primary' return tags
else:
if 'MRN' in attrs and attrs['MRN'] == 'Yes':
tags['highway'] = 'secondary'
else:
tags['highway'] = 'tertiary'
del attrs['RD_CLASS']
I’ll talk aboutwhat ogr2osm can do for youHow it worksPresent a case studyBut first, why care?
This is why you should careI wish I could say problems like this are uncommon but it didn’t take long for my database to find ways with over 300 tags from bad importsImports like this are useless with no osm tags and even if they include some osm tags the number of confusing tags puts new mappers off
I did not start ogr2osm but I have taken over maintenance of it
Ogr2osm has some features that make it superior to other general-purposeshapefile and geodata convertersBecause it uses ogr it can read virtually any file format. About the only format it doesn’t support is a postgisdb but that can be easily added if someone wants itIt reprojects to WGS84, saving a step with ogr2ogr with most sourcesIt will work with multi-layer sources, correctly combining shared nodes across layers with an error toleranceAnd most importantly, you can use python functions to convert datasource tags to OSM tagsLastly, it comes with some documentation
Installing ogr2osm is fairly easy. The only dependencies are python and gdal with python bindings. Git is required to install it and the translationsShould mention Translations are their own git sub-module. This makes it possible to use your own repo for the translations without touching ogr2osm itself
The easiest way for me to explain how ogr2osm works is to briefly run through the code execution
First the file is read inBy default it reads with OGR it and it makes an in-memory copy of it
Then it works on each layerOften there is just one layer but it will handle multiple ones
A sequence of steps happens for each layer
The first step for processing the layer is passing the layer through the filterLayer translation functionThis is a function that you can writeThere are two common usesThe first is to drop layers that you don’t want to convert to osm for importingThe second is to add new fields to the layer and to add fields to objects indicating what layer they’re from for later on
The next step is to reproject the layer to EPSG:4326 that OSM usesOgr does all the work here so not much to say ---Next is filterFeature()Another user defined functionMost common use is to drop features that you aren’t interested---Each feature is turned from an ogr object into osm nodes ways and MPsThis is where the bulk of the code isOgr2osm will only create multipolygons if absolutely necessary. The old SVN version did so whenever there were overlapping ways but the new one does not do so.Intend to add this as an optionIt will handle stuff like multilinestrings without problemsIf you’re only using and not developing the details don’t matter too much---Next is the filterTags() functionThis is the most important user defined function. It turns source attributions into OSM tags and it’s up to you to write it. The use of a function allows complicated logic and stuff like name expansion---filterFeaturePost is another not commonly used function. UVM uses these for buildings in a fairly complicated way.--- You then repeat for each
Next is filterFeature()Another user defined functionMost common use is to drop features that you aren’t interested
In addition to each layer the features also need to be turned into EPSG:4326. not much to say here either
Each feature is turned from an ogr object into osm nodes ways and MPsThis is where the bulk of the code isOgr2osm will only create multipolygons if absolutely necessary. The old SVN version did so whenever there were overlapping ways but the new one does not do so.Intend to add this as an optionIt will handle stuff like multilinestrings without problemsIf you’re only using and not developing the details don’t matter too much
Next is the filterTags() functionThis is the most important user defined function. It turns source attributions into OSM tags and it’s up to you to write it. The use of a function allows complicated logic and stuff like name expansionAs it’s one argument it is passed a dictionary which is the fields on the feature and it returns another dictionary which is the OSM tags
filterFeaturePost is another not commonly used function. UVM uses these for buildings in a fairly complicated way.--- You then repeat for each
You then repeat for each
As an example of writing translations, I have a case study of data from the City of Surrey that I’ve been working onThis is a large and extremely detailed data setI should mention that converting the data is only half the problem. Once it’s converted you have to figure out how to best use the data, but that’s another talk in itselfAlthough I am looking at one specific dataset most government datasets are fairly similar in what they tag and how their taggings don’t exactly correspond to OSM taggingsWhat do I mean by a lot of data?
This is about 10x5 city blocks, about 40 hectares. As you can see, there’s a lot of data
This is one corner showing how much data there is to sort through, but I won’t go over all of it here
The first step is generally to clip the data to a smaller areaogr2osm will deal with gigabyte .osm files but normally you want to load these up into JOSM which doesn’t work so wellOgr2ogr can trim the files down to an area, but you need to be aware of a couple gotchasIf your source is .mdb or something that supports long field names converting to a shapefile will truncate themYou also have to deal with coordinate systems. I have a one line command in my speaker’s notes that will be up later that does thisI have a one-line ogr2ogr command in my notes that handlesogr2ogr -progress -spat `echo "-122.89 49.18" | gdaltransform -s_srs EPSG:4326 -t_srs EPSG:26910 | cut -d \\ -f 1,2` `echo "-122.85 49.22" | gdaltransform -s_srs EPSG:4326 -t_srs EPSG:26910 | cut -d \\ -f 1,2` SurreyData_clipSurreyData
The first step is to see what layers you can dropThe easiest way to do this is use the layer translation and open the resulting .osm fileI’m have scripting for the surrey example before it makes it to ogr2osm that removes the undesired layers so this example is actually from a NHD translationIt removes some layers which indicate NHD subbasin divisions which don’t correspond to anything physicalIt also adds a __LAYER field which is needed to identify which layer a feature comes from
With a complex shapefile there might be 50 fields to use when converting to OSM tags, so it really helps to have a system for writing your filterTags functionThis is the system I use, with an exampleThis function first exits if it’s an untagged object like a nodeIt then deletes some items from attrs and converts one to a ref tagIt then adds any unknown fields as tagsFor an automatically run script you’d want to make it error if it came across unknown fields since this would indicate the fields had changed
The first stop to writing a good filtertags is to get rid of most of the fieldsDepending on how the data was generated you’ll have a lot of useless fields. The surrey data is basically a dump out of their database so it includes every possible field
The first step is to identify the main field for the layer. Most layers will have some kind of main field.In this case there are a total of 44 fields. 17 can be dropped right away and it’s helpful to do this just to clear up what you’re looking at
In this case the main field is RC_TYPE and RC_TYPE2Generally the main field will give you the main tagIn this case it tells you that some are highways and others are railways but doesn’t always tell you what type of highwayThe important point is, and I cannot emph. this enough, is to always verify the fields. You cannot assume that the descriptions mean what you think they doFor a big dataset like NHD you can’t even assume that it’s consistent across the countryBecause road is ambiguous we need to look at it in more detail
Once you’ve identified a value, in this case RC_TYPE2=Road you want to then look at that in detail.As you can see, we still don’t have enough detail to reliably tag the class of road for RD_CLASS arterial, so we need to look for more detail