This Python script takes GPS data with date/time fields and extracts the hour, minute, and second values into separate fields. It then calculates a "TimeSec" field that represents the time in seconds. The points are converted to lines sorted by the "TimeSec" field and split at points within a 2 meter radius. Spatial joins are used to add min and max "TimeSec" fields for each segment. Finally, the data is converted to 3D using the min and max time fields for Z-values.
SSN-TC workshop talk at ISWC 2015 on EmroozMarkus Stocker
Slides for the talk describing the paper on Emrooz, a scalable database for sensor observations with semantics according to the Semantic Sensor Network ontology.
SSN-TC workshop talk at ISWC 2015 on EmroozMarkus Stocker
Slides for the talk describing the paper on Emrooz, a scalable database for sensor observations with semantics according to the Semantic Sensor Network ontology.
For a class project we developed a whole set of Pig scripts for TPC-H. Our goals are:
1) identifying the bottlenecks of Pig's performance especially of its relational operators,
2) studying how to write efficient scripts by making full use of Pig Latin's features,
3) comparing with Hive's TPC-H results for verifying both 1) and 2)
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...Rob Emanuele
Slides from the 2017 FOSS4G Workshop "Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS"
See the repository at https://github.com/lossyrob/foss4g-2017-geopyspark-workshop
Network simulator 2 :
Object-oriented, discrete event driven network simulator
It was normally used in wired & wireless protocol
Written in C++ and OTcl
A short introduction for new programmers to the concepts behind a general, workhorse machine learning algorithm, k-Nearest Neighbor.
Talk given at Pair Columbus, 23 May 2015
For a class project we developed a whole set of Pig scripts for TPC-H. Our goals are:
1) identifying the bottlenecks of Pig's performance especially of its relational operators,
2) studying how to write efficient scripts by making full use of Pig Latin's features,
3) comparing with Hive's TPC-H results for verifying both 1) and 2)
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...Rob Emanuele
Slides from the 2017 FOSS4G Workshop "Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS"
See the repository at https://github.com/lossyrob/foss4g-2017-geopyspark-workshop
Network simulator 2 :
Object-oriented, discrete event driven network simulator
It was normally used in wired & wireless protocol
Written in C++ and OTcl
A short introduction for new programmers to the concepts behind a general, workhorse machine learning algorithm, k-Nearest Neighbor.
Talk given at Pair Columbus, 23 May 2015
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...Chester Chen
Uber developed an new Spark ingestion system, Marmaray, for data ingestion from various sources. It’s designed to ingest billions of Kafka messages every 30 minutes. The amount of data handled by the pipeline is of the order hundreds of TBs. Omar details how to tackle such scale and insights into the optimizations techniques. Some key highlights are how to understand bottlenecks in Spark applications, to cache or not to cache your Spark DAG to avoid rereading your input data, how to effectively use accumulators to avoid unnecessary Spark actions, how to inspect your heap and nonheap memory usage across hundreds of executors, how you can change the layout of data to save long-term storage cost, how to effectively use serializers and compression to save network and disk traffic, and how to reduce amortize the cost of your application by multiplexing your jobs, different techniques for reducing memory footprint, runtime, and on-disk usage. CGI was able to significantly (~10%–40%) reduce memory footprint, runtime, and disk usage.
Speaker: Omkar Joshi (Uber)
Omkar Joshi is a senior software engineer on Uber’s Hadoop platform team, where he’s architecting Marmaray. Previously, he led object store and NFS solutions at Hedvig and was an initial contributor to Hadoop’s YARN scheduler.
Fullstack Conference - Proxies before proxies: The hidden gems of Javascript...Tim Chaplin
Tired of console.logging your way through applications? Want a way to slice through your application without adding complexity? AOP has been the answer to these questions for object oriented languages, such as Java and C#, but is not available in Javascript. ScarletJS(https://github.com/scarletjs/scarlet) is a project that tackles AOP using a clean, fluent, performant interface.
The ScarletJS project provides Javascript developers a different way of thinking about traditional javascript problems. The project is still growing and looking into the future of what ES6 proxies will open up to the Javascript community.
The talk will highlight the problems that javascript developers face with logging application behavior, security, and more. It will discuss the benefits of identifying a cross cutting concern, and programming using aspects. The talk will highlight how thinking about a project and cross cutting concerns can lead to cleaner more SOLID code. It will also discuss the future of ES6 proxies and the benefits that they will bring.
Sperasoft talks about several important aspects of ECMAScript6 - language widely used for client-side scripting on the web, in the form of several well-known implementations such as JavaScript, JScript and ActionScript.
Object-Oriented JavaScript presentation given at the 2010 ESRI Developer Summit. Code and slides are also available at http://github.com/kvangork/OOJS-Presentation
Find me on twitter @kvangork
or my blog http://prng.vangorkom.org
Object-Oriented JavaScript presentation given at ESRI's 2010 Developer Summit. Slides and code available at http://github.com/kvangork/OOJS-Presentation
Find me on twitter @kvangork
or at my blog: http://prng.vangorkom.org
Video games are written as a main loop: process player input, update the state of the game, render a new frame to the screen, repeat. They do this 60 times a second, with millisecond timing. Most monitoring tools are also written as loops: send a probe, wait for the response, update a data store, sleep. Often this is done pretty slowly, maybe once a second! In video games if you can’t update fast enough, you skip the rendering step and the frame rate drops. With monitoring tools if your loop takes to long you also stop logging data as often, and instead of choppy gameplay you get gaps in your graphs, often when you need that data the most!
Let’s use ping as an example and see how we can rewrite its main loop to function more like a video game, keeping a high frame rate.
In this InfluxDays NYC 2019 talk, InfluxData Founder & CTO Paul Dix will outline his vision around the platform and its new data scripting and query language Flux, and he will give the latest updates on InfluxDB time series database. This talk will walk through the vision and architecture with demonstrations of working prototypes of the projects.
2017 02-07 - elastic & spark. building a search geo locatorAlberto Paro
Presentazione dell'evento EsInRome del 7 Febbraio 2017 - Integrazione Elasticsearch in architettura BigData e facilità di integrazione con Apache Spark.
2017 02-07 - elastic & spark. building a search geo locatorAlberto Paro
Using Elasticsearch in a BigData environment is very simple. In this talk, we analyse what's Big Data and we show how it is easy integrating ElasticSearch with Apache Spark
Emerging Languages: A Tour of the HorizonAlex Payne
A tour of a number of new programming languages, organized by the job they're best suited for. Presented at Philadelphia Emerging Technology for the Enterprise 2012.
1. Space Time Visualization Using Python
I- 190
State Hwy 31
Delaware St
Python Script:
# arcpy
import arcpy
# environment and parameters for the tool
arcpy.env.workspace = arcpy.GetParameterAsText(0)
GPSinFeature = arcpy.GetParameterAsText(1)
DateTimeS = arcpy.GetParameterAsText(2)
OutOut4 = arcpy.GetParameterAsText(3)
#set local variable
FieldName1 = "hour"
eldPrecision = 9
FieldName2 = minute
eldPrecision = 9
FieldName3 = second
eldPrecision = 9
FieldName4 = TimeXAll
FieldName5 = TimeXMin
FieldName7 = TimeSec
# add feild 3 times for three new elds (in gps
shapele)
arcpy.AddField_management(GPSinFeature,
FieldName1, LONG, eldPrecision)
arcpy.AddField_management(GPSinFeature,
FieldName2, LONG, eldPrecision)
arcpy.AddField_management(GPSinFeature,
FieldName3, LONG, eldPrecision)
# calculate time
# creat a refrence eld which includes just
time not date
arcpy.AddField_management(GPSinFeature
, FieldName4, TEXT, eldPrecision)
# New: Convert any type of date eld to one
standard date eld
arcpy.ConvertTimeField_management
(GPSinFeature,DateTimeS,yyyy-MM-dd
HH:mm:ss;1033;;,HHMMSS,TEXT,
yyyyMMddHHmmss)
# New: copy the hh:mm:ss part of the
Standard DateTimeS (just created)
to the eld just created (TimeXAll)
arcpy.CalculateField_management
(GPSinFeature,TimeXAll,Mid( [HHMMSS],
9, 14),VB,#)
# add a eld to be able to copy the source
for minute
arcpy.AddField_management(GPSinFeature,
FieldName5, TEXT, eldPrecision)
# New: calculate the source eld for minute and
sec
arcpy.CalculateField_management(GPSin
Feature,TimeXMin,Right( [TimeXAll], 4)
,VB,#)#calculate hour
arcpy.CalculateField_management
(GPSinFeature,hour,Left( [TimeXAll], 2)
,VB,#)#calculate min
arcpy.CalculateField_management
(GPSinFeature,
minute,Left( [TimeXMin], 2),VB,#)
#calculate sec
arcpy.CalculateField_management
(GPSinFeature,second,Right( [TimeXMin]
, 2),VB,#)
# create a eld for time based on seconds
arcpy.AddField_management
(GPSinFeature, FieldName7, LONG,
eldPrecision)
#add hour + minute+ sec and calculate
the time based on sec
arcpy.CalculateField_management
(GPSinFeature,TimeSec,[hour] *3600
+ [minute]*60 + [second],VB,#)
# points to lines
#parameters
outFeature = Line_ready.shp
outFeature2 = splited
sortField = TimeSec
radious = 2 Meter
# create line based on sort led
arcpy.PointsToLine_management
(GPSinFeature, outFeature, , sortField)
#spilit line based on point with .09 radious
arcpy.SplitLineAtPoint_management
(outFeature,GPSinFeature, outFeature2
, radious)
# extract two numbers based on time and
make two elds
#in order to do that we need to copy
sample.shp and use it in spatial join script
# Set local variables
outFeatureClass = ForSpatial
# Execute FeatureClassToFeatureClass
arcpy.FeatureClassToFeatureClass_
conversion(GPSinFeature, arcpy.env.
workspace, outFeatureClass)
#parameters for spatial join
OutPut3 = MinMaxSec
inFeature8 = splited.shp
# Replace a layer/table view name with a
path to a dataset (which can be a layer le)
or create the layer/table view within the script
# The following inputs are layers or table views
: splited, 5
arcpy.SpatialJoin_analysis(inFeature8,
GPSinFeature,OutPut3,JOIN_ONE_TO_ONE,
KEEP_ALL,max max true false
false 50 Double 0 0 ,Max,#,ForSpatial.shp,
TimeSec,-1,-1;min min true false false 50
Double 0 0 ,Min,#,ForSpatial.shp,TimeSec
,-1,-1,INTERSECT,#,#)
Sheridan Dr State Hwy 324
Main St
I- 90
Shawnee Rd
River Rd
Saunders Settlement
State Hwy 265
State Hwy 266
State Hwy 182
Harlem Rd
Erie Ave
Townline Rd
Main St
State Hwy 198
Main St
Main St
# feature to 3d
arcpy.CheckOutExtension('3D')
# parameters
arcpy.FeatureTo3DByAttribute_3d
(MinMaxSec.shp,OutOut4,min,max)
PrimaryRoads
Destinations
GPS Track
Water
Erie County Block Groups
Miles
0.25 0.5 1 1.5 2