Apply Hammer Directly to Thumb; Avoiding Apache Spark and Cassandra AntiPatt...Databricks
Learn from someone who has made just about every basic Apache Spark mistake possible so you don’t have to! We’ll go over some of the most common things that users do that end up doing that cause unnecessary pain and actually explain how to avoid them.
Confused about serialization? Not sure what is meant by use a singleton to share connections? Together we will walk through concrete examples of how to handle these situation. Learn how to: do all your work remotely, not break your catalyst optimizations, use all your resources, and much more! Together lets learn how to make our Spark Applications better!
Sydney GeoRabble Presentation - GovHack 2012 - Happiness MattersKelvin Nicholson
This was a snippet of my presentation at Sydney GeoRabble explaining what my team did at the 2012 GovHack event. It is only two slides, but covers some of the GIS trickery we needed to do as part of the analysis for the event.
Slides are with limited context, unfortunately, as I presented most of the information.
Analyzing the Performance Effects of Meltdown + Spectre on Apache Spark Workl...Databricks
Meltdown and Spectre are two security vulnerabilities disclosed in early 2018 that expose systems to cross-VM and cross-process attacks. They were the first of their kind and opened up a new class of exploits that allow one program to scan another program’s memory. The kernel and VM patches released to address these vulnerabilities have shown to degrade the performance of Apache Spark workloads in the cloud by 2-5%.
This talk will dive deep into the exploits and their patches in order to help explain the origin of this decline in performance.
Exploring Color Spaces with Gesture Tracking and Smart Bulbs (Distill 2014)Daniel Luxemburg
RGB, CMYK, HSV, HSL… We have a lot of ways to write code about colors. One thing they all have in common is that they define a space with more than two dimensions. When visualizing or interacting with these spaces we are forced to flatten them to fit on our two-dimensional screens and to interact with them using our puny, two-dimensional pointing devices. The results fail to convey the reasoning behind different modes of color definition. As a result, it can be difficult to develop an intuitive sense of what the numbers that go with these acronyms represent. In turn, communicating about color with others can be a challenge.
Around the world with extensions | PostgreSQL Conference Europe 2018 | Craig ...Citus Data
Postgres continues to get more and more feature rich. But equally as impressive is the network of extensions that are growing around Postgres. With the rich extension APIs you can now add advanced functionality to Postgres without having to fork the codebase or wait for the main PostgreSQL release cycle. In this talk we'll cover some of the basics of what an extension is and then take a tour through a variety of Postgres extensions including:
pg_stat_statments
PostGIS
HyperLogLog and TopN
Timescale
pg_partman
Citus
Foreign data wrappers which are their own whole class
Coordination of Distributed Software with RedisKonrad Bucheli
This is a practice report showing how Redis, an in-memory data structure store, can be used in distributed environments to communicate, coordinate and to share state.
The talk contains a small introduction to Redis and shows how it can be used in Perl by highlighting key implementation details of a productively used job scheduling system.
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2a2Djbp.
Gerard Sans explains RxJS' data architecture based on reactive programming, exploring Observables API using RxJS koans and unit tests. RxJS 5 focuses on performance and usability. Filmed at qconlondon.com.
Gerard Sans is a multi-talented Computer Science Engineer specialised in Web. He has lived and worked for all sorts of companies in Germany, Brazil, UK and Spain. He enjoys running AngularJS Labs London, mentoring AngularJS students, participating in the community, giving talks and writing technical articles at Medium.
Apply Hammer Directly to Thumb; Avoiding Apache Spark and Cassandra AntiPatt...Databricks
Learn from someone who has made just about every basic Apache Spark mistake possible so you don’t have to! We’ll go over some of the most common things that users do that end up doing that cause unnecessary pain and actually explain how to avoid them.
Confused about serialization? Not sure what is meant by use a singleton to share connections? Together we will walk through concrete examples of how to handle these situation. Learn how to: do all your work remotely, not break your catalyst optimizations, use all your resources, and much more! Together lets learn how to make our Spark Applications better!
Sydney GeoRabble Presentation - GovHack 2012 - Happiness MattersKelvin Nicholson
This was a snippet of my presentation at Sydney GeoRabble explaining what my team did at the 2012 GovHack event. It is only two slides, but covers some of the GIS trickery we needed to do as part of the analysis for the event.
Slides are with limited context, unfortunately, as I presented most of the information.
Analyzing the Performance Effects of Meltdown + Spectre on Apache Spark Workl...Databricks
Meltdown and Spectre are two security vulnerabilities disclosed in early 2018 that expose systems to cross-VM and cross-process attacks. They were the first of their kind and opened up a new class of exploits that allow one program to scan another program’s memory. The kernel and VM patches released to address these vulnerabilities have shown to degrade the performance of Apache Spark workloads in the cloud by 2-5%.
This talk will dive deep into the exploits and their patches in order to help explain the origin of this decline in performance.
Exploring Color Spaces with Gesture Tracking and Smart Bulbs (Distill 2014)Daniel Luxemburg
RGB, CMYK, HSV, HSL… We have a lot of ways to write code about colors. One thing they all have in common is that they define a space with more than two dimensions. When visualizing or interacting with these spaces we are forced to flatten them to fit on our two-dimensional screens and to interact with them using our puny, two-dimensional pointing devices. The results fail to convey the reasoning behind different modes of color definition. As a result, it can be difficult to develop an intuitive sense of what the numbers that go with these acronyms represent. In turn, communicating about color with others can be a challenge.
Around the world with extensions | PostgreSQL Conference Europe 2018 | Craig ...Citus Data
Postgres continues to get more and more feature rich. But equally as impressive is the network of extensions that are growing around Postgres. With the rich extension APIs you can now add advanced functionality to Postgres without having to fork the codebase or wait for the main PostgreSQL release cycle. In this talk we'll cover some of the basics of what an extension is and then take a tour through a variety of Postgres extensions including:
pg_stat_statments
PostGIS
HyperLogLog and TopN
Timescale
pg_partman
Citus
Foreign data wrappers which are their own whole class
Coordination of Distributed Software with RedisKonrad Bucheli
This is a practice report showing how Redis, an in-memory data structure store, can be used in distributed environments to communicate, coordinate and to share state.
The talk contains a small introduction to Redis and shows how it can be used in Perl by highlighting key implementation details of a productively used job scheduling system.
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2a2Djbp.
Gerard Sans explains RxJS' data architecture based on reactive programming, exploring Observables API using RxJS koans and unit tests. RxJS 5 focuses on performance and usability. Filmed at qconlondon.com.
Gerard Sans is a multi-talented Computer Science Engineer specialised in Web. He has lived and worked for all sorts of companies in Germany, Brazil, UK and Spain. He enjoys running AngularJS Labs London, mentoring AngularJS students, participating in the community, giving talks and writing technical articles at Medium.
Hadoop isn't limited to running Java code, you can write your jobs in a variety of dynamic languages.
This talk is about Hadoop's Streaming API, and the best way we found to run Perl jobs on Amazon's Elastic MapReduce platform.
Server side geo_tools_in_drupal_pnw_2012Mack Hardy
Mack Hardy @mackaffinity from Affinity Bridge @affinitybridge discusses server side mapping tools for drupal, using PostGIS as a spatial backend, generating tiles and managing large sets of geodata and displaying it in Drupal CMS
Postgres Vision 2018: PostGIS and Spatial ExtensionsEDB
The extensibility of PostgreSQL has enabled the combination of PostgreSQL and the geospatial extension PostGIS, creating what many say is the most powerful SQL/MM compliant database for location-based applications. This presentation, delivered at Postgres Vision 2018 by Regina Obe, Co-founder of Paragon Corporation and PostGIS Project Steering Committee member, and Leo Hsu, Co-founder of Paragon Corporation, examines PostgreSQL spatial extensions that work with PostGIS.
2017 02-07 - elastic & spark. building a search geo locatorAlberto Paro
Presentazione dell'evento EsInRome del 7 Febbraio 2017 - Integrazione Elasticsearch in architettura BigData e facilità di integrazione con Apache Spark.
2017 02-07 - elastic & spark. building a search geo locatorAlberto Paro
Using Elasticsearch in a BigData environment is very simple. In this talk, we analyse what's Big Data and we show how it is easy integrating ElasticSearch with Apache Spark
As part of the GSP’s capacity development and improvement programme, FAO/GSP have organised a one week training in Izmir, Turkey. The main goal of the training was to increase the capacity of Turkey on digital soil mapping, new approaches on data collection, data processing and modelling of soil organic carbon. This 5 day training is titled ‘’Training on Digital Soil Organic Carbon Mapping’’ was held in IARTC - International Agricultural Research and Education Center in Menemen, Izmir on 20-25 August, 2017.
With the open source Geo2tag platform, developers can use JSON or XML to manage location references in apps for Nokia X and Nokia Asha phones. In this webinar, we’ll show how to use the Geo2tag API and how to manage a local database of georeferences. We’ll begin the training by introducing the fundamentals of Location Based Services and the REST API of Geo2Tag LBS Platform (www.geo2tag.org). We’ll focus on networking, JSON and web services. Then we will demonstrate several applications developed on top of Geo2Tagand share the newest enhancements to the platform. We’ll end the training with a discussion of integrating Geo2Tag and third-party map widgets.
Recent developments in Hadoop version 2 are pushing the system from the traditional, batch oriented, computational model based on MapRecuce towards becoming a multi paradigm, general purpose, platform. In the first part of this talk we will review and contrast three popular processing frameworks. In the second part we will look at how the ecosystem (eg. Hive, Mahout, Spark) is making use of these new advancements. Finally, we will illustrate "use cases" of batch, interactive and streaming architectures to power traditional and "advanced" analytics applications.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
2. Timothy Boronczyk
Senior Developer at Optanix, Inc.
Former Managing Editor of phpmaster.com
Author
CentOS 7 Server Deployment Cookbook
Jump Start MySQL
Beginning PHP6, Apache, MySQL Web Development
PHP and MySQL: Create, Modify, Reuse
zaemis.blogspot.com @zaemis
Coordinates are a set of values used to identify the location of an object. They allow us to answer the question “Where am I?”
Coordinate systems provide context for working with locations. There are different coordinate systems but they all include: origin, distance, and direction.
Example: an object is 5 steps (distance) in front (direction) of me (origin).
Coordinates also allow us to answer the question “Where will I be?”
Examples:
I move 4 steps forward and 3 steps to my right
I turn 30 degrees to my right and walk for 8 steps
A whole branch of mathematics is based on angle/distance (trigonometry).
Drawing a line from the point to the origin forms an angle with the X axis.
The point’s Y coordinate is the angle’s sine.
The point’s X coordinate is the angle’s cosine.
Coordinate systems that identify locations on Earth are known as geodetic systems. Geodetic systems use latitude and longitude to specify coordinates.
Latitude runs east-west as parallel rings.
0° latitude is the Equator
+90° is the North Pole; -90° is the South Pole
Longitude lines circle the earth at angles around the poles.
0° longitude is the Prime Meridian
Positive values are east; negative values are west
The 180th meridian is the Antimeridian
The distance between two longitude lines near the equator is not the same as near the poles
Different geodetic systems use slightly different positions for 0° latitude and 0° longitude, as well different reference values like the Earth’s diameter, for more accurate results in different parts of the world.
Examples:
CH1903–Swiss Grid
ED 50–European Datum
ETRS89–European Terrestrial Reference System
GCJ-02–Chinese Encrypted Geodetic Datum
GDA94– Geocentric Datum of Australia
JGD2011– Japanese Geodetic Datum (adjusted for changes caused by 2011 earthquake and tsunami)
NAD83–North American Datum
OSGB36–Ordnance Survey of Great Britain
RT 90–Swedish Grid (Rikets Nät)
SAD69–South American Datum
WGS84–World Geodetic System
Additional geodetic systems can be found online at epsg.io.
Different systems are needed because the Earth is not perfectly round—the distance from its center to sea level is 13 miles greater at the equator than at the poles.
Adoption of WGS84 is increasing because the benefits of a global system outweigh greater accuracy.
Server-side, we don’t know the exact location of a user unless they tell us. However, we may be able to get a rough idea from their IP address.
Location databases relate IP address to a general area (i.e., city or state) and other related information (area code, latitude and longitude of the city’s center, etc.)
Use cases for IP-based geolocation:
Digital rights management (boo!)
Provide relative content (news stories, all search is local)
Fraud detection/prevention
Marketing/demographics
Databases can be self-hosted or accessed via a web service.
Databases:
IP2Location
MaxMind GeoIP2
MaxMind GeoLite2 – free City and Country databases
Web Services:
MaxMind GeoIP2 Precision Services
Neustar IP Intelligence
ipinfo.io – free, 1000 requests a day
geoPlugin.com – free, 10 requests per minute
In general, a paid database/service will have more information and more frequent updates. Free offerings are more limited, may have usage restrictions, and are unsupported.
Example of IP-based geolocation: Sexy Singles
Prepare a base image
Look up the user’s IP address in the database and retrieve the name of their city
Draw the name on the image
Instantiate the Reader object with the path to the database file and then provide the IP address to the appropriate method (city() is using the City database, country() if using the Country database, etc.)
The Reader class works with both GeoLite2 City and Country databases and GeoIP2 databases.
To use the web service instead, instantiate GeoIP2\WebService\Client with a user ID and license key and then provide the address to the appropriate method.
The GeoLite2 City database used in the demo contains the following information:
city->geoname
city->name
continent->code
continent->geoname
continent->name
country->geonameid
country->isoCode
country->name
location->accuracyRadius
location->latitude
location->longitude
location->metroCode
location->timeZone
postal->code
subdivisions[]->geonameId
subdivisions[]->isoCode
subdivisions[]->name
mostSpecificSubdivision->*
The paid GeoIP2 City database also includes:
location->averageIncome
location->populationDensity
The website for the geoPlugin web service hosts a class file for working with the service, but its just as easy to issue a curl call or use file_get_contents() to retrieve data.
$record = json_decode(file_get_contents(
'http://www.geoplugin.net/json.gp?ip=' . $_SERVER['REMOTE_ADDR']));
• json.gp returns a JSON string
• php.gp returns a serialized PHP array
• xml.gp returns an XML document
API key can be purchased for SSL for 12€ (~$13)
The geoPlugin service returns the following information:
• geoplugin_city
• geoplugin_region
• geoplugin_areaCode
• geoplugin_dmaCode
• geoplugin_countryCode
• geoplugin_countryName
• geoplugin_continentCode
• geoplugin_latitude
• geoplugin_longitude
• geoplugin_regionCode
• geoplugin_regionName
• geoplugin_currencyCode
• geoplugin_currencySymbol
IP-based geolocation is inherently imprecise.
Things to be aware of:
An address can only be mapped to a general area.
The returned latitude and longitude is the center of the area
The database can become outdated (example: Verizon re-allocates on of its address blocks to a different part of their network)
The address will not be the user’s real IP address if they are behind a proxy
The JavaScript geolocation API gives developers access to more accurate location information.
Supported by:
• Firefox 3.5+
• Chrome 5.0+
• Safari 5.0+
• Opera 10.60+
• IE 9+
• Android 2.0+
• iPhone 3.0+
The platform can use different methods to determine a user’s location:
• Google Location Service database (uses visible Wi-Fi networks, signal strength, MAC addresses, etc.)
• GPS receiver
• Cell towers (triangulation)
Example of location detection using JavaScript API
The browser prompts the user before sharing their location for privacy reasons.
Graceful degradation/progressive enhancement — consider using IP-based location first and use the JavaScript geolocation API for greater resolution.
The Geolocation Privacy and Surveillance Act (GPS Act) was introduced to Congress in June 2011.
The act is “a legal framework designed to give government agencies, commercial entities and private citizens clear guidelines for when and how geolocation information can be accessed and used.”
The Obama Administration opposes the act because GPS tracking is “no more invasive than visual surveillance.”
Once we have the user’s coordinates, we can calculate the distance between them and another object’s position.
The pythagorean theorem can be used for small areas where the curvature of the earth is not a factor (indoor).
Other formulas for calculating distances on a sphere:
• Spherical Law of Cosines
• Haversine Formula
• Vincenty’s Formulas
Choosing a formula to use depends on the required level of accuracy and computation speed.
Suggested Resources:
• www.movable-type.co.uk/scripts/latlong.html - Calculate distance, bearing and more between Latitude/Longitude points
• gis.stackexchange.com/q/4906 - Why is law of cosines more preferable than haversine when calculating distance between two latitude-longitude points?
The Spherical Law of Cosines states the distance between two points ((AB)) on a sphere is the same as the measure of the angle they form (∠ACB) to the center of the sphere (C).
d = R ⋅ arccos( sin φ1 ⋅ sin φ2 + cos φ1 ⋅ cos φ2 ⋅ cos Δλ )
d Distance
R Radius (3,963.19 miles per WGS84)
φ (phi) Latitude
λ (lambda) Longitude
Note: Latitude and longitude are expressed in degrees but the trigonometric functions expect values in radians; use deg2rad() to convert them.
Assume we have a database of records for objects with their locations and we want to retrieve records for those objects within a given distance—calculating the distance for every location to filter the results is inefficient. Use a minimum bounding rectangle to narrow search results.
Strategy:
• Calculate a minimum bounding rectangle
• Select records that fall within the rectangle
• Use the distance formula to narrow down the results
We cannot simply say 1 mile = 0.001 degree because the distance between two lines of longitude vary depending on latitude. A bounding rectangle requires calculating coordinates at a certain distance in the different direction.
I assume one can rewrite the distance equation to solve for different variables but my trigonometry skills are rusty. Wolfram-Alpha results were scary.
An internet search returned these formulas (www.movable-type.co.uk/ scripts/latlong.html):
φ2 = asin(sin φ1 ⋅ cos δ + cos φ1 ⋅ sin δ ⋅ cos θ)
λ2 = λ1 + atan2(sin θ ⋅ sin δ ⋅ cos φ1, cos δ - sin φ1 ⋅ sin φ2)
δ (delta) is d/R (angular radius)
Θ (theta) is angle of direction (0° = east, 90° = north, 180° = west, 270° = south)
atan2 is arctan that accepts two arguments (i.e. coordinates X, Y)
• signs the arctan result depending on where the point lies in the coordinate system, positive to the top/right and negative to the bottom/left.
• Wolfram’s results were scary because it was performing the underlying mathematical acrobatics of atan2.
Example of searching for nearby libraries
The Open Geospatial Consortium defined an extension to SQL to support spatial data. MySQL implements some of the definition.
MySQL 5.7 or newer is required.
• It was not possible to create spatial indexes prior to 5.7.
• 5.7 implemented st_distance_sphere() which calculates the distance between two points on a sphere; other distance functions use planer geometry
Recommended reading: www.percona.com/blog/2016/02/03/new-gis-features-in-mysql-5-7