The port of a city is a very dynamic environment that houses
lots of companies. Ships come and go, and goods are always on the move. Information integration in geographic information systems is of great importance for tracking purposes, and for proactive and reactive incident handling by the port operators. Linked data and semantic web technologies can be of benefit for the integration of both streaming data such as location data about ships, trains and containers, with static data about companies, their activities and storage sites. With current RDF stream processors it is possible to execute SPARQL queries over RDF data streams taking into account static background data. However, none of them is capable of handling GeoSPARQL queries. GeoSPARQL is an extension to the SPARQL query language for processing geospatial data. To address this challenge we extended the RSP engine C-SPARQL with GeoSPARQL support, making it possible to query geospatial data streams.
Goman, Khramtsovsky, Kolesnikov (2006) - Computational Framework for Analysis...Project KRIT
М.Г.Гоман. А.В.Храмцовский, Е.Н.Колесников "Методика численного анализа нелинейной динамики самолета и синтеза системы управления на основе качественных методов анализа", встреча участников проекта GARTEUR в Стокгольме, 21-22 марта 2006 года.
M.Goman, A.Khramtsovsky, E.Kolesnikov "Computational Framework for Analysis of Aircraft Nonlinear Dynamics & Control Design Based on Qualitative Methods", GARTEUR meeting in Stockholm 21-22 March 2006.
The port of a city is a very dynamic environment that houses
lots of companies. Ships come and go, and goods are always on the move. Information integration in geographic information systems is of great importance for tracking purposes, and for proactive and reactive incident handling by the port operators. Linked data and semantic web technologies can be of benefit for the integration of both streaming data such as location data about ships, trains and containers, with static data about companies, their activities and storage sites. With current RDF stream processors it is possible to execute SPARQL queries over RDF data streams taking into account static background data. However, none of them is capable of handling GeoSPARQL queries. GeoSPARQL is an extension to the SPARQL query language for processing geospatial data. To address this challenge we extended the RSP engine C-SPARQL with GeoSPARQL support, making it possible to query geospatial data streams.
Goman, Khramtsovsky, Kolesnikov (2006) - Computational Framework for Analysis...Project KRIT
М.Г.Гоман. А.В.Храмцовский, Е.Н.Колесников "Методика численного анализа нелинейной динамики самолета и синтеза системы управления на основе качественных методов анализа", встреча участников проекта GARTEUR в Стокгольме, 21-22 марта 2006 года.
M.Goman, A.Khramtsovsky, E.Kolesnikov "Computational Framework for Analysis of Aircraft Nonlinear Dynamics & Control Design Based on Qualitative Methods", GARTEUR meeting in Stockholm 21-22 March 2006.
Process RTK Data Based On Kalman Filtering AlgorithmIJRES Journal
With the development of satellite positioning technology, there is a strong need for high accuracy
position information. Currently the most widely used high-precision positioning technology is RTK(Real-Time
Kinematic).RTK technology is the key to using carrier phase measurements. It takes advantage of the base
stations and monitor stations observed error of spatial correlation, except monitor stations observed by means of
differential most of the errors in the data, in order to achieve high accuracy positioning.[3]Based on Kalman
filtering algorithm to handle the noise of RTK data and selecting appropriate models to further improve the
accuracy of the data. This paper will explore the use of Kalman filtering method of RTK data processing, which
reduces random noise interference, thus improving the accuracy of GNSS deformation monitoring data.[1]
PostgreSQL Procedural Languages: Tips, Tricks and GotchasJim Mlodgenski
One of the most powerful features of PostgreSQL is its diversity of procedural languages, but with that diversity comes a lot of options.
Did you ever wonder:
- What all of those options are on the CREATE FUNCTION statement?
- How do they affect my application?
- Does my choice of procedural language affect the performance of my statements?
- Should I create a single trigger with IF statements or several simple triggers?
- How do I debug my code?
- Can I tell which line in my function is taking all of the time?
Goal of this project is to study what factors and how they would impact the landing distance of a commercial flight. For this I worked with dataset containing details of BOEING and AIRBUS flights.
These are slides from the presentation "Aerospace Applications of Perl" given by Ian Kluft at Silicon Valley Perl on March 3, 2016 in Santa Clara, California. It explores aerospace topics and related CPAN modules, with some Perl coding examples.
MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Agg...MongoDB
The United States will be deploying 16,000 traffic speed monitoring sensors - 1 on every mile of US interstate in urban centers. These sensors update the speed, weather, and pavement conditions once per minute. MongoDB will collect and aggregate live sensor data feeds from roadways around the country, support real-time queries from cars on traffic conditions on their route as well as be the platform for real-time dashboards displaying traffic conditions and more complex analytical queries used to identify traffic trends. In this session, we’ll implement a few different data aggregation techniques to query and dashboard the metrics gathered from the US interstate.
We show GEO map search request with The Nominatim Search Service and one GPS Example. It has a Map Quest Search with a simple and efficient interface and powerful capabilities, and relies solely on data contributed to OpenStreetMap.
On the menu View/GEO Map View you can use the function without scripting.
Scalable and Adaptive Graph Querying with MapReduceKyong-Ha Lee
We address the problem of processing multiple graph queries over a massive set of data graphs in this letter. As the number of data graphs is growing rapidly, it is often hard to process graph queries with serial algorithms in a timely manner. We propose a distributed graph querying algorithm, which employs feature-based comparison and a filterand-verify scheme working on the MapReduce framework. Moreover, we devise an ecient scheme that adaptively tunes a proper feature size at runtime by sampling data graphs. With various experiments, we show that the proposed method outperforms conventional algorithms in terms of both scalability and efficiency.
Predicting likelihood of flight cancellations using machine learning:
Commerical flights get delayed quite often but canceled much less frequently. Even though the flight cancellation rate is not high, that one rare event causes a lot of troubles to passengers in terms of rescheduling their travel plans. It would be helpful to passengers if they knew the chance that their flights will be canceled or not. Travel planner and booking companies such as booking.com, expedia.com, kayak.com, priceline.com, etc.. can use a model capable of predicting the likelihood of the cancellation of a flight. They can then inform their customers well in advance, even before the airlines' management inform the passengers, about the probability of the cancellation of their upcoming flight. In this data science project, I develop such a predictive model for only U.S. domestic flights operating at selected airports.
The flight data is acquired from the Bureau of Transportation Statistics. This data contains information about each flight and their on-time performance, including delay, cancellation and diversion details. For this study, the data is acquired only for the years 2015 and 2016. Often the flight cancellations are caused due to bad weather conditions. Therefore, it is also important to have the hourly weather data at selected airports for 2015 and 2016. We use wunderground.com API to collect all the weather data.
The link to the GitHub repository: https://github.com/aajains/springboard-datascience-intensive/tree/master/capstone_project
Process RTK Data Based On Kalman Filtering AlgorithmIJRES Journal
With the development of satellite positioning technology, there is a strong need for high accuracy
position information. Currently the most widely used high-precision positioning technology is RTK(Real-Time
Kinematic).RTK technology is the key to using carrier phase measurements. It takes advantage of the base
stations and monitor stations observed error of spatial correlation, except monitor stations observed by means of
differential most of the errors in the data, in order to achieve high accuracy positioning.[3]Based on Kalman
filtering algorithm to handle the noise of RTK data and selecting appropriate models to further improve the
accuracy of the data. This paper will explore the use of Kalman filtering method of RTK data processing, which
reduces random noise interference, thus improving the accuracy of GNSS deformation monitoring data.[1]
PostgreSQL Procedural Languages: Tips, Tricks and GotchasJim Mlodgenski
One of the most powerful features of PostgreSQL is its diversity of procedural languages, but with that diversity comes a lot of options.
Did you ever wonder:
- What all of those options are on the CREATE FUNCTION statement?
- How do they affect my application?
- Does my choice of procedural language affect the performance of my statements?
- Should I create a single trigger with IF statements or several simple triggers?
- How do I debug my code?
- Can I tell which line in my function is taking all of the time?
Goal of this project is to study what factors and how they would impact the landing distance of a commercial flight. For this I worked with dataset containing details of BOEING and AIRBUS flights.
These are slides from the presentation "Aerospace Applications of Perl" given by Ian Kluft at Silicon Valley Perl on March 3, 2016 in Santa Clara, California. It explores aerospace topics and related CPAN modules, with some Perl coding examples.
MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Agg...MongoDB
The United States will be deploying 16,000 traffic speed monitoring sensors - 1 on every mile of US interstate in urban centers. These sensors update the speed, weather, and pavement conditions once per minute. MongoDB will collect and aggregate live sensor data feeds from roadways around the country, support real-time queries from cars on traffic conditions on their route as well as be the platform for real-time dashboards displaying traffic conditions and more complex analytical queries used to identify traffic trends. In this session, we’ll implement a few different data aggregation techniques to query and dashboard the metrics gathered from the US interstate.
We show GEO map search request with The Nominatim Search Service and one GPS Example. It has a Map Quest Search with a simple and efficient interface and powerful capabilities, and relies solely on data contributed to OpenStreetMap.
On the menu View/GEO Map View you can use the function without scripting.
Scalable and Adaptive Graph Querying with MapReduceKyong-Ha Lee
We address the problem of processing multiple graph queries over a massive set of data graphs in this letter. As the number of data graphs is growing rapidly, it is often hard to process graph queries with serial algorithms in a timely manner. We propose a distributed graph querying algorithm, which employs feature-based comparison and a filterand-verify scheme working on the MapReduce framework. Moreover, we devise an ecient scheme that adaptively tunes a proper feature size at runtime by sampling data graphs. With various experiments, we show that the proposed method outperforms conventional algorithms in terms of both scalability and efficiency.
Predicting likelihood of flight cancellations using machine learning:
Commerical flights get delayed quite often but canceled much less frequently. Even though the flight cancellation rate is not high, that one rare event causes a lot of troubles to passengers in terms of rescheduling their travel plans. It would be helpful to passengers if they knew the chance that their flights will be canceled or not. Travel planner and booking companies such as booking.com, expedia.com, kayak.com, priceline.com, etc.. can use a model capable of predicting the likelihood of the cancellation of a flight. They can then inform their customers well in advance, even before the airlines' management inform the passengers, about the probability of the cancellation of their upcoming flight. In this data science project, I develop such a predictive model for only U.S. domestic flights operating at selected airports.
The flight data is acquired from the Bureau of Transportation Statistics. This data contains information about each flight and their on-time performance, including delay, cancellation and diversion details. For this study, the data is acquired only for the years 2015 and 2016. Often the flight cancellations are caused due to bad weather conditions. Therefore, it is also important to have the hourly weather data at selected airports for 2015 and 2016. We use wunderground.com API to collect all the weather data.
The link to the GitHub repository: https://github.com/aajains/springboard-datascience-intensive/tree/master/capstone_project
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...Chester Chen
Uber developed an new Spark ingestion system, Marmaray, for data ingestion from various sources. It’s designed to ingest billions of Kafka messages every 30 minutes. The amount of data handled by the pipeline is of the order hundreds of TBs. Omar details how to tackle such scale and insights into the optimizations techniques. Some key highlights are how to understand bottlenecks in Spark applications, to cache or not to cache your Spark DAG to avoid rereading your input data, how to effectively use accumulators to avoid unnecessary Spark actions, how to inspect your heap and nonheap memory usage across hundreds of executors, how you can change the layout of data to save long-term storage cost, how to effectively use serializers and compression to save network and disk traffic, and how to reduce amortize the cost of your application by multiplexing your jobs, different techniques for reducing memory footprint, runtime, and on-disk usage. CGI was able to significantly (~10%–40%) reduce memory footprint, runtime, and disk usage.
Speaker: Omkar Joshi (Uber)
Omkar Joshi is a senior software engineer on Uber’s Hadoop platform team, where he’s architecting Marmaray. Previously, he led object store and NFS solutions at Hedvig and was an initial contributor to Hadoop’s YARN scheduler.
The objective of the project is to predict whether a flight will be delayed or not by studying the various features of the flight and calculating the probability for delay using Bayesian Classification.
The objective of this analysis is to quantify the factors that impact the landing distance of a commercial flight and built a linear regression model to predict the risk of landing overrun.
Slides from my Introduction to PostGIS workshop at the FOSS4G conference in 2009. The material is available at http://revenant.ca/www/postgis/workshop/
Similar to Using PostgreSQL for Flight Planning (20)
A 30 minute presentation given to the Toronto Web Performance Meetup about Google's PageSpeed and the SPDY Protocol.
This presentation covers the following topics.
- What is PageSpeed, how to use and install.
- Should you use SPDY? Or wait for HTTP 2?
- Shortcomings of HTTP 1.1
How Do you Scale for both Predictable and Unpredictable Events on such a Larg...Blake Crosby
The Canadian Broadcasting Corporation is Canada's national public broadcaster. Our website, www.cbc.ca, is one of the largest and most visited in the country, delivering 700 million hits per day on an origin infrastructure composed of only six web servers.
With the right combination of publishing methods, content delivery networks and fine-tuned caching rules, the CBC’s infrastructure has enough headroom to handle spikes of 40x normal traffic during major news events.
How do you scale to almost infinite capacity when you can't predict the world’s events? It's impossible to prepare for that influx of visitors when a celebrity dies, a natural disaster occurs or for other breaking news. Scaling for predictable events is easier, but although we know when the next Federal Election, Olympics Games or FIFA Cup is scheduled, these events present different challenges. Balancing the architecture for both scenarios is important.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
2. Agenda Introduction Tools Math Class Everything you need to know about the Earth Ground School Challenges Displaying the data Q&A Session 11-05-18 Using PostgreSQL for Flight Planning 2
3. Some Introductions… Senior System Administrator Keep cbc.ca up and running Manage Apache+Tomcat & Postgres servers. 11-05-18 Using PostgreSQL for Flight Planning 3 PHP Programmer Pilot (~250 hours) Co-Founder of World Flight Planner fly.blakecrosby.com facebook.com/blakecrosby @blakecrosby me@blakecrosby
4. World Flight Planner 11-05-18 Using PostgreSQL for Flight Planning 4 www.worldflightplanner.com Launched in 2009 Gives pilots the tools to plan safe and efficient flights: Weather (windspeed, temperature, etc…) Notices to Airmen (NOTAMs) Airspace information Front end is written in Ruby on Rails Back end uses PHP, Perl, Postgres+PostGIS.
8. The Data 11-05-18 Using PostgreSQL for Flight Planning 8 Over 150,000 unique points, which include: 23,800 Airports 7,300 Heliports 10,500 Traditional radio navigation aids 104,700 Waypoints for GPS/Computer navigation Over 2,900 unique polygons, which include: Over 560 restricted airspace definitions Over 2,400 terminal, control, and airport airspace definitions Over 800 unique line strings. Over 1,600,000 points of wind data covering 72 hours and 15 different altitudes.
9. Tools We Used: 11-05-18 Using PostgreSQL for Flight Planning 9 Geographic Data (Source data) Storing data: PostgreSQL 9.0 Storing spatial data: PostGIS 1.5.1 To Visualize Data: uDig (Desktop GIS) Mapnik + Open Layers (in browser) Map Server Google Earth
10. PostgreSQL + PostGIS: 11-05-18 Using PostgreSQL for Flight Planning 10 Allows us to use SQL to answer questions like: How long is my route? Will it pass through any restricted airspace? Will I have a head or tail wind during the flight? What are the nearest airports along my route? Is there any severe weather along the way? Postgres’ indexes are fast. Gives the tools to make pilots safer by: Only showing them data relevant to their flight Visualizing complex data on a map The key to a successful flight is detailed and accurate planning.
11. Geographic Data (Vector) 11-05-18 Using PostgreSQL for Flight Planning 11 A Point (-75.6692 45.3225) A Line String (-75.99 36.89,-74.16 38.78,-73.36 39.97,-72.86 40.10) A Polygon (-75.99 36.89,-74.16 38.78,-73.36 39.97,-72.86 40.10,-75.99 36.89)
12. Geographic Data (Raster) 11-05-18 Using PostgreSQL for Flight Planning 12 Multi dimensional data per point in space. Known as “bands” Multiple formats for storing data GeoTIFF Gridded Binary (GRIB) Top View Side View
13. Cartesian Math 11-05-18 Using PostgreSQL for Flight Planning 13 What is the shortest distance between two points? 𝑑= 𝑥2−𝑥12+𝑦2 −𝑦12 𝑑= 3−22+1−32 𝑑=5 A Straight Line
14. Spherical Math 11-05-18 Using PostgreSQL for Flight Planning 14 What is the shortest distance between two points on a sphere? ∆𝜎=arctan𝑐𝑜𝑠∅𝑓𝑠𝑖𝑛∆𝜆2+𝑐𝑜𝑠∅𝑠𝑠𝑖𝑛∅𝑓−𝑠𝑖𝑛∅𝑠𝑐𝑜𝑠∅𝑓𝑐𝑜𝑠∆𝜆2𝑠𝑖𝑛∅𝑠𝑐𝑜𝑠∅𝑓+𝑐𝑜𝑠∅𝑠𝑠𝑖𝑛∅𝑓𝑐𝑜𝑠∆𝜆 𝑑=𝑟∆𝜎 Thankfully, PostGIS can handle this! A Geodesic (Great Circle)
15. The Earth 11-05-18 Using PostgreSQL for Flight Planning 15 Is not flat Is not a sphere Is actually an ellipsoid It’s important to know this distinction. Sphere Spheroid(Ellipsoid)
16. A flight from Ottawa to Amsterdam 11-05-18 Using PostgreSQL for Flight Planning 16 If the Earth was a perfect sphere 5,631.441 KM Using the WGS84 Geoid 5,648.054 KM A Geoid is a definition for the shape of the Earth. The longer the distance, the greater the error. There is still a problem, however.
17. A flight from Ottawa to Amsterdam 11-05-18 Using PostgreSQL for Flight Planning 17 If the Earth is round? How do we display it on a flat surface (ie: your computer monitor)? Answer: Projections Mercator Gnomonic
18. A flight from Ottawa to Amsterdam 11-05-18 Using PostgreSQL for Flight Planning 18
19. Ground School 11-05-18 Using PostgreSQL for Flight Planning 19 Aircraft want to get from A B as quickly as possible. Items that help you: Direct routes (no roads to follow) Mother nature (tail winds) Items that hinder you: Mother Nature (head winds, thunderstorms) Airspace (going around restricted airspace) Tools we use to get where we’re going: GPS Compass & Charts Air Traffic Control Radio Navigation Aids (NDBs and VORs)
20. Ground School 11-05-18 Using PostgreSQL for Flight Planning 20 In order to calculate the best route we need to have: Locations of all the airports (departure and destination points) Locations of all the navigation aids (waypoints) Airspace definitions Wind data for any point on the planet (and altitude)
21. Challenges: Navaids 11-05-18 Using PostgreSQL for Flight Planning 21 Every airport has an uniqueidentifier: Ottawa, CA: CYOW London, UK: EGLL Sydney, AU: YSSY Every navaid has an identifier: Ottawa, CA NDB: OW Owensboro, US NDB: OW Norwood, US, NDB: OW Idents are not unique, not even in the same country!
23. Navaid Schema 11-05-18 Using PostgreSQL for Flight Planning 23 flightplan=> navaid Table "public.navaid" Column | Type | Modifiers -----------+----------+----------------------------------------------------- id | integer | not null default nextval('navaid_id_seq'::regclass) ident | text | not null name | text | not null country | text | not null city | text | state | text | location | geometry | not null elevation | integer | type | integer | not null Indexes: "navaid_pkey" PRIMARY KEY, btree (country, type, ident) Check constraints: "enforce_dims_location" CHECK (ndims(location) = 2) "enforce_geotype_location" CHECK (geometrytype(location) = 'POINT'::text) "enforce_srid_location" CHECK (srid(location) = 4326)
24. Navaid Schema 11-05-18 Using PostgreSQL for Flight Planning 24 SELECT ident,location,elevation FROM navaid WHERE ident = 'CYOW' or ident = 'EHAM'; ident | location | elevation -------+----------------------------------------------------+----------- CYOW | 0101000020E6100000093A6DA0D3EA52C0E17A14AE47A94640 | 374 EHAM | 0101000020E6100000521BE8B4810E134033C4B12E6E274A40 | -11 (2 rows) Using PostGIS’ st_astext() function: SELECT ident,st_astext(location),elevation FROM navaid WHERE ident = 'CYOW’ … ident | st_astext | elevation -------+----------------------------------+----------- CYOW | POINT(-75.6691666666667 45.3225) | 374 EHAM | POINT(4.76416666666667 52.30805) | -11 (2 rows)
25. Challenges: Airspace & Arcs 11-05-18 Using PostgreSQL for Flight Planning 25 Creating simple polygons is easy. Squares, triangles, octagons, etc… However airspace is a complex polygon: The airspace to 2000´ (1400´ AAE) within the area bounded by a line beginning at: N43°43'30.37" W079°24'07.83" to N43°47'10.29" W079°25'01.74" thence clockwise along the arc of a circle of 5 miles radius centred on N43°51'44.00" W079°22'12.00" (Buttonville Muni, ON - AD) to N43°55'57.12" W079°18'29.25" thence counter-clockwise along the arc of a circle of 2 miles radius centred on N43°56'09.00" W079°15'44.00" (Markham, ON - AD) to N43°54'09.53" W079°15'59.81" to N43°48'57.53" W079°14'39.57" to N43°44'38.30" W079°18'37.97" to N43°43'30.37" W079°24'07.83" point of beginning
27. Challenges: Airspace & Arcs 11-05-18 Using PostgreSQL for Flight Planning 27 There are two ways to define arcs: Using a lot of individual lines (st_curvetoline()) Using the circular line string geometry type. We created a custom function called createarc(): Starting Point Ending Point Centre Direction (clockwise or anti-clockwise) OR Size (major or minor arc) Returns a line string with 128 points per circle. http://trac.osgeo.org/postgis/wiki/UsersWikiplpgsqlfunctions
29. Challenges: Airspace & Arcs 11-05-18 Using PostgreSQL for Flight Planning 29 If we could do it all over again… Stick with the circular curve geometry types: CIRCULARSTRING COMPOUNDCURVE CURVEPOLYGON
30. Challenges: Storing Raster Data 11-05-18 Using PostgreSQL for Flight Planning 30 PostGIS doesn’t support raster data (in version 1.5) Weather data from NOAA is stored in GRIB format Problem: How to store raster data in PostgreSQL?
31. Challenges: Storing Raster Data 11-05-18 Using PostgreSQL for Flight Planning 31 We use the array datatype. flightplan=> grib Table "public.grib" Column | Type | Modifiers ----------+--------------------------+-------------------------------------------------------- id | integer | not null default nextval('gribtemp_id_seq1'::regclass) data | double precision[] | not null pressure | double precision | not null valid | timestamp with time zone | not null location | geometry |
32. Challenges: Storing Raster Data 11-05-18 Using PostgreSQL for Flight Planning 32 Convert from GRIB to CSV to Array (using Perl) Each element in the array is a different altitude flightplan=> SELECT data,st_astext(location),valid FROM grib LIMIT 1; data | st_astext | valid ------------------------------------+-----------------+------------------------ {{-42.4,30,11.4},{-33.6,13,1.4},…} | POINT(-179 -90) | 2011-05-16 12:00:00+00 (1 row) Fast querying, but long load time (~40 min) Version 2.0 of PostGIS will have full support for rasters Hurry Up!
34. How far is it? 11-05-18 Using PostgreSQL for Flight Planning 34 st_distance() function: SELECT st_distance( st_geomfromtext('POINT(4.76416666666667 52.30805)',4326), st_geomfromtext('POINT(-75.6691666666667 45.3225)',4326) ); st_distance ------------------ 80.7361072873446 (1 row) st_distance() returns the Euclidian distance. Not Good! WGS84 units are in “degrees”. So what are our options?
35. How far is it? 11-05-18 Using PostgreSQL for Flight Planning 35 st_distance_sphere() function: SELECT st_distance_sphere( st_geomfromtext('POINT(4.76416666666667 52.30805)',4326), st_geomfromtext('POINT(-75.6691666666667 45.3225)',4326) ); st_distance_sphere -------------------- 5631441.37550182 (1 row) st_distance_sphere() returns the geodesic distance. Good! Also returns distance in meters. But we still have a problem: the Earth is not a sphere, remember?
36. How far is it? 11-05-18 Using PostgreSQL for Flight Planning 36 st_distance_spheroid() function: SELECT st_distance_spheroid( st_geomfromtext('POINT(4.76416666666667 52.30805)',4326), st_geomfromtext('POINT(-75.6691666666667 45.3225)',4326), 'SPHEROID["WGS 84",6378137,298.257223563]' ); st_distance_spheroid -------------------- 5648054.61931799 (1 row) st_distance_spheroid() returns the geodesic distance. Good! Also returns distance in meters. Remember: If you want accurate results on the Earth, you must use spheroid functions!
37. Closest Airports Along My Route 11-05-18 Using PostgreSQL for Flight Planning 37 st_distance_sphere() st_dwithin() Nearest neighbour function (not ellipsoid “safe”) Boston GIS website
38. Closest Airports Along My Route 11-05-18 Using PostgreSQL for Flight Planning 38 flightplan=> SELECT ident,name FROM navaid WHERE type = '1' AND st_distance_sphere(geomfromtext('LINESTRING(-79.3687166666667 43.86085,-75.6691666666667 45.3225)',4326),location) <= '20000'; ident | name -------+------------------------------- CNP8 | GREENBANK CNA9 | PLEVNA/TOMVALE CYKZ | TORONTO/BUTTONVILLE MUNICIPAL CYOO | OSHAWA CYRP | CARP CYPQ | PETERBOROUGH CYOW | OTTAWA/MACDONALD-CARTIER INTL CNU8 | TORONTO/MARKHAM CYRO | ROCKCLIFFE CYZD | TORONTO/DOWNSVIEW CPS2 | KEENE/ELMHIRST'S RESORT CNR6 | CARLETON PLACE (12 rows)
39. Will I Fly Through Any Airspace? 11-05-18 Using PostgreSQL for Flight Planning 39 st_intersects() flightplan=> SELECT name,class,notes FROM airspace LEFT OUTER JOIN airspaceinfo ON airspace.id = airspaceinfo.airspaceid WHERE st_intersects(geomfromtext('LINESTRING(-79.3687166666667 43.86085,-75.6691666666667 45.3225)',4326),geometry); name | class | notes -----------------------------------+-------+------- Toronto TCA 3 | TCA | Peterborough Control Zone | E | Ottawa International Control Zone | C | Toronto TCA 4 | TCA | Buttonville Control Zone | D | Buttonville Control Zone 2 | D | Ottawa TCA | TCA | Ottawa TCA | TCA | Ottawa TCA | TCA | (9 rows)
40. Displaying Data To Users 11-05-18 Using PostgreSQL for Flight Planning 40 How do you display your route and data to end users? Rendering: Mapnik Tile Cache Map Server Navigating the data: Google Maps API KML Open Layers API WKT GML
41. Pilot Reports st_translate() RotateAtPoint() http://trac.osgeo.org/postgis/wiki/UsersWikiplpgsqlfunctions 11-05-18 Using PostgreSQL for Flight Planning 41 Given to Air Traffic Control to relay weather information to other pilots. UACN10 CYXU 181236 YZ UA /OV CYXZ 125050 /TM 1231 /FLUNKN /TP J3 /RM LOW IFR CEILING 50 MILES SE CYXZ ENROUTE TO CYEL
44. Severe Weather SIGMET (Signifigant Meterological Report) Can be quite complex and hard to visualize. 11-05-18 Using PostgreSQL for Flight Planning 44 WSCN33 CWTO171805 SIGMET A5 VALID171805/172205 CWTO WTN 30 NM OF LN /4622N 07925W/NORTH BAY–/4458N07918W/MUSKOKA– /4302N08109W/ LONDON.TS MAX TOPS 300 OBSD ON RADAR. LN MOVG EWD AT 20 KT.LT LCHG IN INTSTY. Create a line string st_buffer() to create 30nm wide polygon
46. The “geography” type PostGIS 1.5 introduced the geography datatype Great because it assumes you’re on an ellipsoid and not a plane Bad because it doesn’t support all of the functions that we need to use Slowly moving our queries & data over to the geography type as function support increases. 11-05-18 Using PostgreSQL for Flight Planning 46
47. In Conclusion… Allows us to export the data in multiple formats: KML Well Known Text (WKT) GML Plain old ints (using st_x() and st_y() functions) Allows us to JOIN spatial and non spatial data easily. Allows us to perform complex math on geometries: Joining and splitting polygons Clipping shapes to fit inside a specific country Allows us to use any programming language without having to have GIS support. (just SQL support) 11-05-18 Using PostgreSQL for Flight Planning 47
Introduction: Who Is Blake?Tools: What is needed to store geographical data in PostgresMath Class: Go over some geometry, what is a point? Or linestring? Raster vs. Vector dataThe Earth: The earth is not flat. It’s important that we know that we’re doing everything on an ellipsoid (not a sphere!)Ground School: We’ll go over some aeronautical termsDisplaying the Data: Tools you can use to display the data to usersQ&A Session: I’ll leave some time at the end for Questions.
Talk about self:Live in TorontoBeen employed by the CBC for the past 9 years.Went to seneca college to become a pilotContact information
Mostly focus on backend, specifically postgres set up
Weather radar and airspace around Ottawa
A typical flight from Toronto to Ottawa, showing airports and navigational aids.
Flight Plan. Indicating speeds, weather, and waypoints.
A sample set of data. Moderate size database
Specific versions of the tools we useExplain that mapnik is a rendering engine that generate tiles for display on maps.Why we picked Postgres/PostGIS: - Oracle is too expensive (and not open source) - MySQL doesn’t have mature GIS support
These types of shapes are known as vector data.Points are used for items like: airports, navigational aids, waypointsLine Strings are used for “airways”, highways in the sky. Pre-defined routes from two airportsPolygons are used to define areas, such as airspace. There are other types of shapes, but we only use these types.
Each band: - can be a specific altitude OR - multiple data per point (wind, humidity, cloud cover)PostGIS currently cant store raster data natively. However, version 2.0 will!
There are over 3000 different definitions for the shape of the earth.The WSG84 Geoid is the most common reference system used. It’s the geoid approved by ICAO for aviation use.
Projecting the data on-to a map is not the responsibility of PostGIS or Postgres.Gnomonic projection displays great circles as straight lines
This is the most direct flight from Ottawa to Amsterdam
Explain the schema: Why: - The primary key is set the way it is - We need the elevation
The code is known as Well Known Binary (displayed in hex)