Python crash course for geologists in the mining industry
•
0 likes•108 views
This two-day Python crash course for geologists in the mining industry will teach participants how to use Python for common tasks like manipulating geological data. The course costs 450 euro and will be held in Perth, Australia on September 5-6, 2018. It is aimed at geologists, mining engineers, and managers who want to learn Python scripting. Over the two days, participants will learn how to set up a Python environment, do basic coding, import and explore geochemical and grid data, automate tasks with procedures, and understand where to go next for additional learning. The agenda covers topics like introduction to Python coding, setting up environments, programming fundamentals, loading and transforming data, exploring drillhole files, and applying skills
Report
Share
Report
Share
Download to read offline
![Agenda
Day 1 - Introduction to Python
● 9am - 9.30am: Introduction to Python coding
○ How open source is going to change the mining industry
○ Why use Python when you are a geologist
○ Python vs R
● 9.30am - 10.30am: Before you start coding
○ Overview of useful libraries
○ How to set up your computer environment to start coding
● 10.45m - 12pm: Programming in Python
○ Loop, if else
○ Dictionary
○ Pandas dataframe
● 1pm - 2pm: Getting data in
○ Load a file and read it
○ Example on XYZ csv file
○ Use basic plots using Pandas to QC
● 2pm - 3pm: Your first program - Transforming data
○ Basic data manipulation
○ Merging files
○ Data cleaning
Day 2 - Data exploration and plotting using Python
● 9am - 10.30am: Explore a Drillhole file
○ Clean and select the relevant information
○ Merge data
● 10.30am - 12.30pm: Interrogate your data
○ Understand data through general statistics
○ Going further with graphics using Seaborn
● 1.30pm - 3pm: Applications to Model Validation
○ Writing procedure for automatic reporting
○ understanding['more'] = understanding['bits'] + practice['more']](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Debugging and Profiling Rails Application
The document discusses debugging and profiling a Ruby on Rails application that is using excessive memory. It explains how the Ruby garbage collector works and that allocating too many objects can lead to longer garbage collection times and eventually running out of memory. Various tools are presented for debugging memory issues, including pry-debugger, ObjectSpace.count_objects, enabling heap dumps, and gdb.rb. Profiling tools discussed include Benchmark, ruby-prof, and perftools.rb for measuring time, allocations, memory usage, and garbage collection metrics. The key message is that allocating too many objects leads to slowdowns from longer garbage collection, so examining the heap and using debugging and profiling tools is important.
Scaling metrics
This document discusses scaling Graphite, an open source monitoring tool. It describes how to feed data into Graphite using tools like StatsD and Collectd. It then discusses how to scale these data collection tools to handle high volumes of metrics. The document reviews several open source projects for scaling Graphite like Carbonapi, Carbonzipper, and go-carbon. It provides an overview of how these projects improve Graphite's flexibility, redundancy, and performance when dealing with large numbers of metrics.
Behind the Scenes at Coolblue - Feb 2017
This document discusses various tools in the Elastic Stack including Kibana, Elasticsearch, Beats, and Logstash. It provides brief descriptions of each tool and why they are used. Additional logging and monitoring tools are also mentioned, along with links to documentation, code samples, and other resources from the discussion.
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS. Euromicro 2001
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS
The BSP model can be extended with a zero cost synchronization mechanism, which can be used when the number of messages due to receives is known. This mechanism, usually known as "oblivious synchronization" implies that different processors can be in different supersteps at the same time. An unwanted consequence of these software improvements is a loss of accuracy in prediction. This paper proposes an extension of the BSP complexity model to deal with oblivious barriers and shows its accuracy.
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS. Euromicro 2001
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS
The BSP model can be extended with a zero cost synchronization mechanism, which can be used when the number of messages due to receives is known. This mechanism, usually known as "oblivious synchronization" implies that different processors can be in different supersteps at the same time. An unwanted consequence of these software improvements is a loss of accuracy in prediction. This paper proposes an extension of the BSP complexity model to deal with oblivious barriers and shows its accuracy.
Gdal introduction
The slide gives an introduction to GDAL:
geospatial data, abstraction, library or framework, getting started and more...
Migration of audio files using Hadoop, SCAPE Information Day, 25 June 2014
Hadoop has been used at the State and University Library, Denmark, in connection with an experiment on the migration of a large collection of audio files from mp3 to wav. This experiment was presented by Bolette Ammitzbøll Jurik at ‘SCAPE Information Day at the State and University Library, Denmark’, on 25 June 2014. The information day introduced the EU-funded project SCAPE (Scalable Preservation Environments) and its tools and services to the participants.
The experiment used Hadoop and Taverna but also xcorrSound waveform-compare which is a small tool developed within SCAPE to compare the content of audio files.
Read more about the event in this blog post, http://bit.ly/SCAPE_SB_Demo.
FleetDB
FleetDB is a NoSQL document database that uses a schema-free and document-based data model. It stores all data in memory for high performance and persistence. FleetDB supports rich data structures, single document access across multiple tables, clear path to horizontal scalability without migrations, multi-record transactions, and excellent concurrency. Client libraries are available for Java, Python, and Clojure.
Recommended
Debugging and Profiling Rails Application
The document discusses debugging and profiling a Ruby on Rails application that is using excessive memory. It explains how the Ruby garbage collector works and that allocating too many objects can lead to longer garbage collection times and eventually running out of memory. Various tools are presented for debugging memory issues, including pry-debugger, ObjectSpace.count_objects, enabling heap dumps, and gdb.rb. Profiling tools discussed include Benchmark, ruby-prof, and perftools.rb for measuring time, allocations, memory usage, and garbage collection metrics. The key message is that allocating too many objects leads to slowdowns from longer garbage collection, so examining the heap and using debugging and profiling tools is important.
Scaling metrics
This document discusses scaling Graphite, an open source monitoring tool. It describes how to feed data into Graphite using tools like StatsD and Collectd. It then discusses how to scale these data collection tools to handle high volumes of metrics. The document reviews several open source projects for scaling Graphite like Carbonapi, Carbonzipper, and go-carbon. It provides an overview of how these projects improve Graphite's flexibility, redundancy, and performance when dealing with large numbers of metrics.
Behind the Scenes at Coolblue - Feb 2017
This document discusses various tools in the Elastic Stack including Kibana, Elasticsearch, Beats, and Logstash. It provides brief descriptions of each tool and why they are used. Additional logging and monitoring tools are also mentioned, along with links to documentation, code samples, and other resources from the discussion.
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS. Euromicro 2001
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS
The BSP model can be extended with a zero cost synchronization mechanism, which can be used when the number of messages due to receives is known. This mechanism, usually known as "oblivious synchronization" implies that different processors can be in different supersteps at the same time. An unwanted consequence of these software improvements is a loss of accuracy in prediction. This paper proposes an extension of the BSP complexity model to deal with oblivious barriers and shows its accuracy.
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS. Euromicro 2001
PREDICTING THE TIME OF OBLIVIOUS PROGRAMS
The BSP model can be extended with a zero cost synchronization mechanism, which can be used when the number of messages due to receives is known. This mechanism, usually known as "oblivious synchronization" implies that different processors can be in different supersteps at the same time. An unwanted consequence of these software improvements is a loss of accuracy in prediction. This paper proposes an extension of the BSP complexity model to deal with oblivious barriers and shows its accuracy.
Gdal introduction
The slide gives an introduction to GDAL:
geospatial data, abstraction, library or framework, getting started and more...
Migration of audio files using Hadoop, SCAPE Information Day, 25 June 2014
Hadoop has been used at the State and University Library, Denmark, in connection with an experiment on the migration of a large collection of audio files from mp3 to wav. This experiment was presented by Bolette Ammitzbøll Jurik at ‘SCAPE Information Day at the State and University Library, Denmark’, on 25 June 2014. The information day introduced the EU-funded project SCAPE (Scalable Preservation Environments) and its tools and services to the participants.
The experiment used Hadoop and Taverna but also xcorrSound waveform-compare which is a small tool developed within SCAPE to compare the content of audio files.
Read more about the event in this blog post, http://bit.ly/SCAPE_SB_Demo.
FleetDB
FleetDB is a NoSQL document database that uses a schema-free and document-based data model. It stores all data in memory for high performance and persistence. FleetDB supports rich data structures, single document access across multiple tables, clear path to horizontal scalability without migrations, multi-record transactions, and excellent concurrency. Client libraries are available for Java, Python, and Clojure.
ESCAPE Kick-off meeting - LSST (Feb 2019)
1) The document discusses plans for an upcoming LSST kick-off meeting on February 7-8, 2019.
2) It provides technical specifications for the LSST telescope including an 8.4m primary mirror, 3.2 billion pixel camera, and plans to collect 20TB of data per day.
3) It outlines the challenges of storing, processing and providing access to the enormous amount of data to be collected by LSST, including 500PB of raw data and catalogs containing tens of billions of objects.
R user-group-2011-09
Talk given on September 21 to the Bay Area R User Group. The talk walks a stochastic project SVD algrorithm through the steps from initial implementation in R to a proposed implementation using map-reduce that integrates cleanly with R via NFS export of the distributed file system. Not surprisingly, this algorithm is essentially the same as the one used by Mahout.
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...
This document outlines a presentation on analyzing large raster data in a Jupyter notebook with GeoPySpark on AWS. The presentation covers introductory material, exercises on working with land cover and Landsat imagery data, combining data layers to detect crop cycles, and combining different data types to create maps. It discusses where the notebooks are running, data sources, and GeoPySpark capabilities like working with space-time raster data. Attendees are encouraged to tweet maps created during the exercises.
Geo Package and OWS Context at FOSS4G PDX
Presentation made at FOSS4G Portland, September 10, 2014 by Luis Bermudez about New OGC Standards: GeoPackage and OWS Context
Data Lessons Learned at Scale
Charlie Reverte, VP of Engineering at AddThis, discusses lessons learned from processing large-scale web data. AddThis processes data from 14 million domains, including 100 billion monthly page views and 50,000 events per second. Reverte outlines challenges around distributed ID generation, counting unique values, joining distributed data, sampling large datasets, and deploying systems that invalidate over 1.4 billion browser caches. He advocates for loose coupling between systems using approaches like Kafka for asynchronous event logging. Reverte also discusses techniques for columnar compression, tunable quality of service, and open sourcing Hydra, AddThis' custom processing system optimized for real-time data.
Sprint 6
The document outlines work done for a class project on predictive analytics use cases in Python and R. It summarizes the outcomes of a client meeting and updates made to a scorecard for Use Case 3, which involved completing an object detection use case using Python and R tools. It also provides documentation of the packages and steps used to complete Use Case 3 in both languages. Finally, it outlines next steps to continue working on a final presentation for the project.
Triple store
The document discusses storing a geopolitical ontology containing about 21,000 triples. It currently stores the ontology in a flat file and uses the JENA API. The document proposes alternative architectures for storage, including using a triple store or relational database, and compares loading times for different options based on benchmark results. It suggests hosting a SPARQL endpoint for querying if using a relational database-based storage.
Redis Day TLV 2018 - RediSearch Aggregations
This document discusses RediSearch aggregations, which allow processing search results to produce statistical insights. Aggregations take a search query, group and reduce the results, apply transformations, and sort. Key steps include filtering results, grouping and reducing with functions like count and average, applying expressions, and sorting. Examples show finding top GitHub committers and visits by hour. Scaling aggregations to multiple nodes requires pushing processing stages to nodes and merging results, such as summing counts or taking list intersections.
Redis Day TLV 2018 - Redis as a Time-Series DB
Redis can be used as a time-series database by using the redis-timeseries module. The module provides a custom data structure and commands for storing and querying time-series data in Redis. Data can be added with a timestamp and value and queried within a time range. Downsampling aggregates and stores data at regular intervals to reduce the size of long time-series data. Global configuration allows defining downsampling rules and retention policies for all keys.
GIS on Rails by Oleksandr Kychun
This document discusses integrating geospatial data and functionality into a Ruby on Rails application using the OpenGeo Suite. It provides an overview of key concepts like raster vs vector data, and describes how to store geospatial data from shapefiles in PostGIS, serve it with GeoServer, and display it in maps using OpenLayers. It also outlines the typical steps to publish a geospatial layer and introduces various APIs for GeoServer configuration. Finally, it lists requirements and describes a process for importing shapefiles into PostGIS through a background job to create layers in GeoServer.
Hadoop導入事例 in クックパッド
1. Hadoop is a framework for distributed processing of large datasets across clusters of computers.
2. Hadoop can be used to perform tasks like large-scale sorting and data analysis faster than with traditional databases like MySQL.
3. Example applications of Hadoop include processing web server logs, managing user profiles for a large website, and performing machine learning on massive datasets.
2021 Dask Summit - Using STAC to catalog SpatioTemporal datasets
The document discusses using the SpatioTemporal Asset Catalog (STAC) to catalog geospatial datasets. STAC defines JSON schemas to encode metadata about spatiotemporal data like remote sensing imagery. This allows datasets like the European Space Agency's Sentinel-2 satellite data, containing petabytes of images, to be more easily searched. The STAC API also defines standards for searching and discovering STAC metadata. Tools like PySTAC and pystac-client make it easier to work with STAC catalogs and APIs in Python. Open questions remain around best representing multi-dimensional datasets like Zarr in STAC.
C* Summit 2013: Time-Series Metrics with Cassandra by Mike Heffner
This document discusses using Cassandra to store time-series metrics data. It describes how the schema was matched to storage by using a measurement column family with rows organized by metric ID and time. It also covers optimizing data expiration through techniques like TTL expiration, synchronized compactions, and leveraging immutable sstable modification times. Effective monitoring is emphasized as well, including dashboards to track the ring and using Cassandra log volumes to identify issues.
PTD and beyond
Hadoop is used extensively at TD, with 1000 daily active users running 65K Hive jobs, 180K Yarn apps, and scanning 20 trillion records from 500 petabytes stored in S3. The team runs 5 Hadoop clusters across 3 regions using a patched version of Hadoop 2.7.3 called PTD. They have improved the clusters to boot faster, be more ephemeral by storing data directly in S3, and made changes to reduce failures by enabling circuit breakers and disk quotas. The team is working on migrating to the latest Hive and simplifying configurations, as well as moving to auto-scaling and code deploy for faster operations.
How does one go from binary data to HDF files efficiently?
This slide presents a performance problem with converting the 180 zonal binary files to the 30/31 day HDF files and discusses what CERES team can do to improve this performance.
Python crash course for geologists in the mining industry
The two-day workshop will provide an introduction to using Python for geological data analysis. Over the course of the workshop, participants will learn how to set up a Python environment, perform basic coding, import and manipulate geological data like drillholes, block models and surfaces, generate custom graphics, and write procedures to automate tasks like reporting. The workshop is designed for geologists, mining engineers, and managers who want to leverage open source tools to more easily manipulate and understand geological data. No prior coding experience is required.
Python crash course for geologists in the mining industry
The two-day workshop will provide an introduction to using Python for geological data analysis. Over the course of the workshop, participants will learn how to set up a Python environment, perform basic coding, import and manipulate geological data like drillholes, block models and surfaces, generate custom graphics, and write procedures to automate reporting tasks. The workshop is designed for geologists, mining engineers, and managers who want to leverage open source tools like Python. No coding experience is required, but participants should have experience in geology.
Data Con LA 2022 - Intro to Data Science
Zia Khan, Computer Systems Analyst and Data Scientist at LearningFuze
Data Science tutorial is designed for people who are new to Data Science. This is a beginner level session so no prior coding or technical knowledge is required. Just bring your laptop with WiFi capability. The session starts with a review of what is data science, the amount of data we generate and how companies are using that data to get insight. We will pick a business use case, define the data science process, followed by hands-on lab using python and Jupyter notebook. During the hands-on portion we will work with pandas, numpy, matplotlib and sklearn modules and use a machine learning algorithm to approach the business use case.
Stanford/SLAC Cryo-EM Computing and Storage, Yee-Ting Li
The PRP is a partnership of more than 50 institutions, led by researchers at UC San Diego and UC Berkeley and includes the National Science Foundation, Department of Energy, and multiple research universities in the US and around the world. The PRP builds on the optical backbone of Pacific Wave, a joint project of CENIC and the Pacific Northwest GigaPOP (PNWGP) to create a seamless research platform that encourages collaboration on a broad range of data-intensive fields and projects.
Clearing Airflow Obstructions
Presentation given on the 15th July 2021 at the Airflow Summit 2021
Conference website: https://airflowsummit.org/sessions/2021/clearing-airflow-obstructions/
Recording: https://www.crowdcast.io/e/airflowsummit2021/40
Introduction to Data Engineer and Data Pipeline at Credit OK
The document discusses the role of data engineers and data pipelines. It begins with an introduction to big data and why data volumes are increasing. It then covers what data engineers do, including building data architectures, working with cloud infrastructure, and programming for data ingestion, transformation, and loading. The document also explains data pipelines, describing extract, transform, load (ETL) processes and batch versus streaming data. It provides an example of Credit OK's data pipeline architecture on Google Cloud Platform that extracts raw data from various sources, cleanses and loads it into BigQuery, then distributes processed data to various applications. It emphasizes the importance of data engineers in processing and managing large, complex data sets.
008 GNNs at Scale With Graph Data Science Sampling and Python Client Integrat...
Graph neural networks (GNN) is a tool that brings great predictive power to graph machine learning tasks such as link prediction and node classification. However, GNN architectures are typically very compute heavy and as such are not feasible to run at massive scale. In this talk, we will leverage the graph sampling features of the Neo4j Graph Data Science (GDS) library as well as the inductive power of GNNs to bring GNNs to scale. We will also show how the GDS Python Client can, with great performance, be used to integrate the GDS workflow with other GNN Python libraries.
More Related Content
What's hot
ESCAPE Kick-off meeting - LSST (Feb 2019)
1) The document discusses plans for an upcoming LSST kick-off meeting on February 7-8, 2019.
2) It provides technical specifications for the LSST telescope including an 8.4m primary mirror, 3.2 billion pixel camera, and plans to collect 20TB of data per day.
3) It outlines the challenges of storing, processing and providing access to the enormous amount of data to be collected by LSST, including 500PB of raw data and catalogs containing tens of billions of objects.
R user-group-2011-09
Talk given on September 21 to the Bay Area R User Group. The talk walks a stochastic project SVD algrorithm through the steps from initial implementation in R to a proposed implementation using map-reduce that integrates cleanly with R via NFS export of the distributed file system. Not surprisingly, this algorithm is essentially the same as the one used by Mahout.
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...
This document outlines a presentation on analyzing large raster data in a Jupyter notebook with GeoPySpark on AWS. The presentation covers introductory material, exercises on working with land cover and Landsat imagery data, combining data layers to detect crop cycles, and combining different data types to create maps. It discusses where the notebooks are running, data sources, and GeoPySpark capabilities like working with space-time raster data. Attendees are encouraged to tweet maps created during the exercises.
Geo Package and OWS Context at FOSS4G PDX
Presentation made at FOSS4G Portland, September 10, 2014 by Luis Bermudez about New OGC Standards: GeoPackage and OWS Context
Data Lessons Learned at Scale
Charlie Reverte, VP of Engineering at AddThis, discusses lessons learned from processing large-scale web data. AddThis processes data from 14 million domains, including 100 billion monthly page views and 50,000 events per second. Reverte outlines challenges around distributed ID generation, counting unique values, joining distributed data, sampling large datasets, and deploying systems that invalidate over 1.4 billion browser caches. He advocates for loose coupling between systems using approaches like Kafka for asynchronous event logging. Reverte also discusses techniques for columnar compression, tunable quality of service, and open sourcing Hydra, AddThis' custom processing system optimized for real-time data.
Sprint 6
The document outlines work done for a class project on predictive analytics use cases in Python and R. It summarizes the outcomes of a client meeting and updates made to a scorecard for Use Case 3, which involved completing an object detection use case using Python and R tools. It also provides documentation of the packages and steps used to complete Use Case 3 in both languages. Finally, it outlines next steps to continue working on a final presentation for the project.
Triple store
The document discusses storing a geopolitical ontology containing about 21,000 triples. It currently stores the ontology in a flat file and uses the JENA API. The document proposes alternative architectures for storage, including using a triple store or relational database, and compares loading times for different options based on benchmark results. It suggests hosting a SPARQL endpoint for querying if using a relational database-based storage.
Redis Day TLV 2018 - RediSearch Aggregations
This document discusses RediSearch aggregations, which allow processing search results to produce statistical insights. Aggregations take a search query, group and reduce the results, apply transformations, and sort. Key steps include filtering results, grouping and reducing with functions like count and average, applying expressions, and sorting. Examples show finding top GitHub committers and visits by hour. Scaling aggregations to multiple nodes requires pushing processing stages to nodes and merging results, such as summing counts or taking list intersections.
Redis Day TLV 2018 - Redis as a Time-Series DB
Redis can be used as a time-series database by using the redis-timeseries module. The module provides a custom data structure and commands for storing and querying time-series data in Redis. Data can be added with a timestamp and value and queried within a time range. Downsampling aggregates and stores data at regular intervals to reduce the size of long time-series data. Global configuration allows defining downsampling rules and retention policies for all keys.
GIS on Rails by Oleksandr Kychun
This document discusses integrating geospatial data and functionality into a Ruby on Rails application using the OpenGeo Suite. It provides an overview of key concepts like raster vs vector data, and describes how to store geospatial data from shapefiles in PostGIS, serve it with GeoServer, and display it in maps using OpenLayers. It also outlines the typical steps to publish a geospatial layer and introduces various APIs for GeoServer configuration. Finally, it lists requirements and describes a process for importing shapefiles into PostGIS through a background job to create layers in GeoServer.
Hadoop導入事例 in クックパッド
1. Hadoop is a framework for distributed processing of large datasets across clusters of computers.
2. Hadoop can be used to perform tasks like large-scale sorting and data analysis faster than with traditional databases like MySQL.
3. Example applications of Hadoop include processing web server logs, managing user profiles for a large website, and performing machine learning on massive datasets.
2021 Dask Summit - Using STAC to catalog SpatioTemporal datasets
The document discusses using the SpatioTemporal Asset Catalog (STAC) to catalog geospatial datasets. STAC defines JSON schemas to encode metadata about spatiotemporal data like remote sensing imagery. This allows datasets like the European Space Agency's Sentinel-2 satellite data, containing petabytes of images, to be more easily searched. The STAC API also defines standards for searching and discovering STAC metadata. Tools like PySTAC and pystac-client make it easier to work with STAC catalogs and APIs in Python. Open questions remain around best representing multi-dimensional datasets like Zarr in STAC.
C* Summit 2013: Time-Series Metrics with Cassandra by Mike Heffner
This document discusses using Cassandra to store time-series metrics data. It describes how the schema was matched to storage by using a measurement column family with rows organized by metric ID and time. It also covers optimizing data expiration through techniques like TTL expiration, synchronized compactions, and leveraging immutable sstable modification times. Effective monitoring is emphasized as well, including dashboards to track the ring and using Cassandra log volumes to identify issues.
PTD and beyond
Hadoop is used extensively at TD, with 1000 daily active users running 65K Hive jobs, 180K Yarn apps, and scanning 20 trillion records from 500 petabytes stored in S3. The team runs 5 Hadoop clusters across 3 regions using a patched version of Hadoop 2.7.3 called PTD. They have improved the clusters to boot faster, be more ephemeral by storing data directly in S3, and made changes to reduce failures by enabling circuit breakers and disk quotas. The team is working on migrating to the latest Hive and simplifying configurations, as well as moving to auto-scaling and code deploy for faster operations.
How does one go from binary data to HDF files efficiently?
This slide presents a performance problem with converting the 180 zonal binary files to the 30/31 day HDF files and discusses what CERES team can do to improve this performance.
What's hot (15)
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...
2021 Dask Summit - Using STAC to catalog SpatioTemporal datasets
2021 Dask Summit - Using STAC to catalog SpatioTemporal datasets
C* Summit 2013: Time-Series Metrics with Cassandra by Mike Heffner
C* Summit 2013: Time-Series Metrics with Cassandra by Mike Heffner
How does one go from binary data to HDF files efficiently?
How does one go from binary data to HDF files efficiently?
Similar to Python crash course for geologists in the mining industry
Python crash course for geologists in the mining industry
The two-day workshop will provide an introduction to using Python for geological data analysis. Over the course of the workshop, participants will learn how to set up a Python environment, perform basic coding, import and manipulate geological data like drillholes, block models and surfaces, generate custom graphics, and write procedures to automate tasks like reporting. The workshop is designed for geologists, mining engineers, and managers who want to leverage open source tools to more easily manipulate and understand geological data. No prior coding experience is required.
Python crash course for geologists in the mining industry
The two-day workshop will provide an introduction to using Python for geological data analysis. Over the course of the workshop, participants will learn how to set up a Python environment, perform basic coding, import and manipulate geological data like drillholes, block models and surfaces, generate custom graphics, and write procedures to automate reporting tasks. The workshop is designed for geologists, mining engineers, and managers who want to leverage open source tools like Python. No coding experience is required, but participants should have experience in geology.
Data Con LA 2022 - Intro to Data Science
Zia Khan, Computer Systems Analyst and Data Scientist at LearningFuze
Data Science tutorial is designed for people who are new to Data Science. This is a beginner level session so no prior coding or technical knowledge is required. Just bring your laptop with WiFi capability. The session starts with a review of what is data science, the amount of data we generate and how companies are using that data to get insight. We will pick a business use case, define the data science process, followed by hands-on lab using python and Jupyter notebook. During the hands-on portion we will work with pandas, numpy, matplotlib and sklearn modules and use a machine learning algorithm to approach the business use case.
Stanford/SLAC Cryo-EM Computing and Storage, Yee-Ting Li
The PRP is a partnership of more than 50 institutions, led by researchers at UC San Diego and UC Berkeley and includes the National Science Foundation, Department of Energy, and multiple research universities in the US and around the world. The PRP builds on the optical backbone of Pacific Wave, a joint project of CENIC and the Pacific Northwest GigaPOP (PNWGP) to create a seamless research platform that encourages collaboration on a broad range of data-intensive fields and projects.
Clearing Airflow Obstructions
Presentation given on the 15th July 2021 at the Airflow Summit 2021
Conference website: https://airflowsummit.org/sessions/2021/clearing-airflow-obstructions/
Recording: https://www.crowdcast.io/e/airflowsummit2021/40
Introduction to Data Engineer and Data Pipeline at Credit OK
The document discusses the role of data engineers and data pipelines. It begins with an introduction to big data and why data volumes are increasing. It then covers what data engineers do, including building data architectures, working with cloud infrastructure, and programming for data ingestion, transformation, and loading. The document also explains data pipelines, describing extract, transform, load (ETL) processes and batch versus streaming data. It provides an example of Credit OK's data pipeline architecture on Google Cloud Platform that extracts raw data from various sources, cleanses and loads it into BigQuery, then distributes processed data to various applications. It emphasizes the importance of data engineers in processing and managing large, complex data sets.
008 GNNs at Scale With Graph Data Science Sampling and Python Client Integrat...
Graph neural networks (GNN) is a tool that brings great predictive power to graph machine learning tasks such as link prediction and node classification. However, GNN architectures are typically very compute heavy and as such are not feasible to run at massive scale. In this talk, we will leverage the graph sampling features of the Neo4j Graph Data Science (GDS) library as well as the inductive power of GNNs to bring GNNs to scale. We will also show how the GDS Python Client can, with great performance, be used to integrate the GDS workflow with other GNN Python libraries.
Big data internship plan at Contemi Vietnam
It's about the passion in Big Data and we are here at Contemi Vietnam to prepare ourselves for the hits.
Train the Trainers: Cloud Computing & Big Data Workshop
This document announces a 5-day training workshop on cloud computing and big data from June 9-13, 2014. The workshop aims to help university lecturers and PhD students understand emerging technologies and revise curricula. Attendees will gain hands-on experience using Amazon Web Services, Salesforce, and Hadoop through workshops. The agenda covers technology trends, cloud concepts, virtualization, NoSQL databases, and MapReduce on Amazon EMR. The workshop is limited to 20 people and will be held at The Connection in Bangkok for a fee of 4,900 Baht.
Python for Earth
- Python has become a robust platform for scientific and engineering work, from data analysis to modeling and visualization. It has clear syntax, many open source libraries, and can be used across operating systems.
- This document discusses the history and advantages of using Python for earth sciences, including modeling hydrodynamics and earthquakes. It also provides examples of using Python with libraries like NumPy, SciPy, and matplotlib for tasks like data analysis, visualization, and GIS processing.
- Python is now widely used for tasks that previously required other languages or programs, offering an integrated environment while maintaining high performance via compiled extensions.
IPv6 on Mikrotik
This document contains an agenda and presentation slides for a webinar hosted by GLC Networks on IPv6 with Mikrotik RouterOS. The webinar covers an introduction to IPv6, how to enable and configure IPv6 on RouterOS, and a demonstration. It provides information on GLC Networks as an Indonesian training and consulting company, the trainer's background, and how to assign IPv6 addresses on RouterOS using SLAAC and DHCPv6 prefix delegation. Slides include comparisons of IPv4 and IPv6, IPv6 addressing rules, and assigning IPv6 addresses to clients.
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
This document summarizes a panel discussion on building the National Research Platform ecosystem with regional networks. The panelists discussed how their regional networks are connecting to and using the Nautilus nodes of the NRP. Examples included using NRP for deep learning and computer vision research at the University of Missouri, challenges of adoption in Nevada and potential solutions, and Georgia Tech's new involvement through the Southern Crossroads regional network. The regional networks see opportunities to expand NRP access and training to enable more researchers in their regions to take advantage of the platform.
Data engineering zoomcamp introduction
The document outlines the plan and syllabus for a Data Engineering Zoomcamp hosted by DataTalks.Club. It introduces the four instructors for the course - Ankush Khanna, Sejal Vaidya, Victoria Perez Mola, and Alexey Grigorev. The 10-week course will cover topics like data ingestion, data warehousing with BigQuery, analytics engineering with dbt, batch processing with Spark, streaming with Kafka, and a culminating 3-week student project. Pre-requisites include experience with Python, SQL, and the command line. Course materials will be pre-recorded videos and there will be weekly live office hours for support. Students can earn a certificate and compete on a
Machine Learning with Python
This document contains the slides for a webinar on machine learning with Python presented by Achmad Mardiansyah from GLC Networks, Indonesia. The webinar agenda includes an introduction to machine learning and Python, a review of prerequisite Python knowledge, a discussion of machine learning concepts, and a live coding practice session. The slides provide background on the presenter and GLC Networks, an overview of machine learning methods, and instructions for participating in the live coding demo.
Open Chemistry, JupyterLab and data: Reproducible quantum chemistry
The Open Chemistry project is developing an ambitious platform to facilitate reproducible quantum chemistry workflows by integrating the best of breed open source projects currently available in a cohesive platform with extensions specific to the needs of quantum chemistry. The core of the project is a Python-based data server capable of storing metadata, executing quantum chemistry calculations, and processing the output. The platform exposes RESTful endpoints using programming language agnostic web endpoints, and uses Linux container technology to package quantum codes that are often difficult to build.
The Jupyter project has been leveraged as a web-based frontend offering reproducibility as a core principle. This has been coupled with the data server to initiate quantum chemistry calculations, cache results, make them searchable, and even visualize the results within a modern browser environment. The Avogadro libraries have been reused for visualization workflows, coupled with Open Babel for file translation, and examples of the use of NWChem and Psi4 will be demonstrated.
The core of the platform is developed upon JSON data standards, and encouraging the wider adoption of JSON/HDF5 as the principle storage mediums. A single page web application using React at its core will be shown for sharing simple views of data output, and linking to the Jupyter notebooks that documents how they were made. Command line tools and links to the Avogadro graphical interface will be shown demonstrating capabilities from web through to desktop.
Data science workflows: from notebooks to production
Slides from the 2019 Big Mountain Data conference, covering best practices and python tools to help organize and accelerate data science development
Spark Meetup at Uber
1) Uber uses Spark and Hadoop to process large amounts of transportation data in real-time and batch. This includes building pipelines to ingest trip data from databases into a data warehouse within 1-2 hours.
2) Paricon is Uber's first Spark application which infers schemas from raw JSON data, converts it to Parquet format for faster querying, and validates the results. It processes over 15TB of data daily.
3) Future work includes building a SQL-based ETL platform on Spark, open sourcing SQL-on-Hadoop, and creating a machine learning platform with Spark and a real-time analytics system called Apollo using Spark Streaming.
Scalable Clusters On Demand
At Opendoor, we do a lot of big data processing, and use Spark and Dask clusters for the computations. Our machine learning platform is written in Dask and we are actively moving data ingestion pipelines and geo computations to PySpark. The biggest challenge is that jobs vary in memory, cpu needs, and the load in not evenly distributed over time, which causes our workers and clusters to be over-provisioned. In addition to this, we need to enable data scientists and engineers run their code without having to upgrade the cluster for every request and deal with the dependency hell.
To solve all of these problems, we introduce a lightweight integration across some popular tools like Kubernetes, Docker, Airflow and Spark. Using a combination of these tools, we are able to spin up on-demand Spark and Dask clusters for our computing jobs, bring down the cost using autoscaling and spot pricing, unify DAGs across many teams with different stacks on the single Airflow instance, and all of it at minimal cost.
Digital Preservation in Production (DPN and DuraCloud Vault)
Describes the first year of production operation for DPN (the Digital Preservation Network) and DuraCloud Vault.
Industrialiser spark
(1) The document discusses industrializing Apache Spark jobs by applying best practices from software engineering including setting up a development environment, testing, continuous integration/delivery (CI/CD), monitoring, and data governance.
(2) It provides recommendations for developing Spark jobs like unit testing functions, integration testing on sample data, validating job efficiency, and benchmarking for performance testing.
(3) Applying practices like CI/CD, logging to tools like ELK/Splunk, monitoring resources and optimizing data storage formats can help improve the reliability, performance and maintainability of Spark jobs.
Similar to Python crash course for geologists in the mining industry (20)
Python crash course for geologists in the mining industry
Python crash course for geologists in the mining industry
Python crash course for geologists in the mining industry
Python crash course for geologists in the mining industry
Stanford/SLAC Cryo-EM Computing and Storage, Yee-Ting Li
Stanford/SLAC Cryo-EM Computing and Storage, Yee-Ting Li
Introduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OK
008 GNNs at Scale With Graph Data Science Sampling and Python Client Integrat...
008 GNNs at Scale With Graph Data Science Sampling and Python Client Integrat...
Train the Trainers: Cloud Computing & Big Data Workshop
Train the Trainers: Cloud Computing & Big Data Workshop
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
Open Chemistry, JupyterLab and data: Reproducible quantum chemistry
Open Chemistry, JupyterLab and data: Reproducible quantum chemistry
Data science workflows: from notebooks to production
Data science workflows: from notebooks to production
Digital Preservation in Production (DPN and DuraCloud Vault)
Digital Preservation in Production (DPN and DuraCloud Vault)
Recently uploaded
11.1 Role of physical biological in deterioration of grains.pdf
Storagedeteriorationisanyformoflossinquantityandqualityofbio-materials.
Themajorcausesofdeteriorationinstorage
•Physical
•Biological
•Mechanical
•Chemical
Storageonlypreservesquality.Itneverimprovesquality.
Itisadvisabletostartstoragewithqualityfoodproduct.Productwithinitialpoorqualityquicklydepreciates
ESA/ACT Science Coffee: Diego Blas - Gravitational wave detection with orbita...
ESA/ACT Science Coffee: Diego Blas - Gravitational wave detection with orbita...Advanced-Concepts-Team
Presentation in the Science Coffee of the Advanced Concepts Team of the European Space Agency on the 07.06.2024.
Speaker: Diego Blas (IFAE/ICREA)
Title: Gravitational wave detection with orbital motion of Moon and artificial
Abstract:
In this talk I will describe some recent ideas to find gravitational waves from supermassive black holes or of primordial origin by studying their secular effect on the orbital motion of the Moon or satellites that are laser ranged.快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
学校原件一模一样【微信：741003700 】《(UAM毕业证书)马德里自治大学毕业证学位证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
8.Isolation of pure cultures and preservation of cultures.pdf
Isolation of pure culture, its various method.
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
A proprietary approach developed by bringing together the best of learning theories from Psychology, design principles from the world of visualization, and pedagogical methods from over a decade of training experience, that enables you to: Learn better, faster!
Gadgets for management of stored product pests_Dr.UPR.pdf
Insectsplayamajorroleinthedeteriorationoffoodgrainscausingbothquantitativeandqualitativelosses
Wellprovedthatnogranariescanbefilledwithgrainswithoutinsectsastheharvestedproducecontainegg(or)larvae(or)pupae(or)adultinsectinthembecauseoffieldcarryoverinfestationwhichcannotbeavoidedindevelopingcountrieslikeIndia
Simpletechnologiesfortimelydetectionofinsectsinthestoredproduceandtherebyplantimelycontrolmeasures
Applied Science: Thermodynamics, Laws & Methodology.pdf
When I was asked to give a companion lecture in support of ‘The Philosophy of Science’ (https://shorturl.at/4pUXz) I decided not to walk through the detail of the many methodologies in order of use. Instead, I chose to employ a long standing, and ongoing, scientific development as an exemplar. And so, I chose the ever evolving story of Thermodynamics as a scientific investigation at its best.
Conducted over a period of >200 years, Thermodynamics R&D, and application, benefitted from the highest levels of professionalism, collaboration, and technical thoroughness. New layers of application, methodology, and practice were made possible by the progressive advance of technology. In turn, this has seen measurement and modelling accuracy continually improved at a micro and macro level.
Perhaps most importantly, Thermodynamics rapidly became a primary tool in the advance of applied science/engineering/technology, spanning micro-tech, to aerospace and cosmology. I can think of no better a story to illustrate the breadth of scientific methodologies and applications at their best.
ESR spectroscopy in liquid food and beverages.pptx
With increasing population, people need to rely on packaged food stuffs. Packaging of food materials requires the preservation of food. There are various methods for the treatment of food to preserve them and irradiation treatment of food is one of them. It is the most common and the most harmless method for the food preservation as it does not alter the necessary micronutrients of food materials. Although irradiated food doesn’t cause any harm to the human health but still the quality assessment of food is required to provide consumers with necessary information about the food. ESR spectroscopy is the most sophisticated way to investigate the quality of the food and the free radicals induced during the processing of the food. ESR spin trapping technique is useful for the detection of highly unstable radicals in the food. The antioxidant capability of liquid food and beverages in mainly performed by spin trapping technique.
Immersive Learning That Works: Research Grounding and Paths Forward
We will metaverse into the essence of immersive learning, into its three dimensions and conceptual models. This approach encompasses elements from teaching methodologies to social involvement, through organizational concerns and technologies. Challenging the perception of learning as knowledge transfer, we introduce a 'Uses, Practices & Strategies' model operationalized by the 'Immersive Learning Brain' and ‘Immersion Cube’ frameworks. This approach offers a comprehensive guide through the intricacies of immersive educational experiences and spotlighting research frontiers, along the immersion dimensions of system, narrative, and agency. Our discourse extends to stakeholders beyond the academic sphere, addressing the interests of technologists, instructional designers, and policymakers. We span various contexts, from formal education to organizational transformation to the new horizon of an AI-pervasive society. This keynote aims to unite the iLRN community in a collaborative journey towards a future where immersive learning research and practice coalesce, paving the way for innovative educational research and practice landscapes.
waterlessdyeingtechnolgyusing carbon dioxide chemicalspdf
The technology uses reclaimed CO₂ as the dyeing medium in a closed loop process. When pressurized, CO₂ becomes supercritical (SC-CO₂). In this state CO₂ has a very high solvent power, allowing the dye to dissolve easily.
Micronuclei test.M.sc.zoology.fisheries.
Current Ms word generated power point presentation covers major details about the micronuclei test. It's significance and assays to conduct it. It is used to detect the micronuclei formation inside the cells of nearly every multicellular organism. It's formation takes place during chromosomal sepration at metaphase.
Farming systems analysis: what have we learnt?.pptx
Presentation given at the official farewell of Prof Ken Gillet at Wageningen on 13 June 2024
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Current descriptions of immersive learning cases are often difficult or impossible to compare. This is due to a myriad of different options on what details to include, which aspects are relevant, and on the descriptive approaches employed. Also, these aspects often combine very specific details with more general guidelines or indicate intents and rationales without clarifying their implementation. In this paper we provide a method to describe immersive learning cases that is structured to enable comparisons, yet flexible enough to allow researchers and practitioners to decide which aspects to include. This method leverages a taxonomy that classifies educational aspects at three levels (uses, practices, and strategies) and then utilizes two frameworks, the Immersive Learning Brain and the Immersion Cube, to enable a structured description and interpretation of immersive learning cases. The method is then demonstrated on a published immersive learning case on training for wind turbine maintenance using virtual reality. Applying the method results in a structured artifact, the Immersive Learning Case Sheet, that tags the case with its proximal uses, practices, and strategies, and refines the free text case description to ensure that matching details are included. This contribution is thus a case description method in support of future comparative research of immersive learning cases. We then discuss how the resulting description and interpretation can be leveraged to change immersion learning cases, by enriching them (considering low-effort changes or additions) or innovating (exploring more challenging avenues of transformation). The method holds significant promise to support better-grounded research in immersive learning.
Recently uploaded (20)
aziz sancar nobel prize winner: from mardin to nobel
aziz sancar nobel prize winner: from mardin to nobel
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
ESA/ACT Science Coffee: Diego Blas - Gravitational wave detection with orbita...
ESA/ACT Science Coffee: Diego Blas - Gravitational wave detection with orbita...
8.Isolation of pure cultures and preservation of cultures.pdf
8.Isolation of pure cultures and preservation of cultures.pdf
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Gadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdf
Basics of crystallography, crystal systems, classes and different forms
Basics of crystallography, crystal systems, classes and different forms
ESR spectroscopy in liquid food and beverages.pptx
ESR spectroscopy in liquid food and beverages.pptx
Immersive Learning That Works: Research Grounding and Paths Forward
Immersive Learning That Works: Research Grounding and Paths Forward
GBSN - Biochemistry (Unit 6) Chemistry of Proteins
GBSN - Biochemistry (Unit 6) Chemistry of Proteins
waterlessdyeingtechnolgyusing carbon dioxide chemicalspdf
waterlessdyeingtechnolgyusing carbon dioxide chemicalspdf
Farming systems analysis: what have we learnt?.pptx
Farming systems analysis: what have we learnt?.pptx
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...
Python crash course for geologists in the mining industry
- 1. Python crash course for geologists in the mining industry Sponsored by Dates: 5-6 September 2018 Place: Core, 191 St Georges Tce, Perth Lecturers: Laurent Wagner & Johann Dangin Max participants: 14 Price: 450euro (about 700AUD) for the 2 days Pre-requisites: no background in coding is required. You do need to have some sense of geology to understand the examples and applications given in this course. Whom for: geologists, mining engineers or managers who want to understand how open source and scripting can easily be used for day to day tasks in manipulating geological data. Objectives: At the end of the 2 days sessions you will be able to : 1. Setup a python environment from scratch 2. Understand basic coding in Python 3. Import geochem and grid data 4. Put your data into custom graphics using Pandas and Seaborn 5. Write procedures to automate tasks such as reporting 6. Understand where to look next
- 2. Agenda Day 1 - Introduction to Python ● 9am - 9.30am: Introduction to Python coding ○ How open source is going to change the mining industry ○ Why use Python when you are a geologist ○ Python vs R ● 9.30am - 10.30am: Before you start coding ○ Overview of useful libraries ○ How to set up your computer environment to start coding ● 10.45m - 12pm: Programming in Python ○ Loop, if else ○ Dictionary ○ Pandas dataframe ● 1pm - 2pm: Getting data in ○ Load a file and read it ○ Example on XYZ csv file ○ Use basic plots using Pandas to QC ● 2pm - 3pm: Your first program - Transforming data ○ Basic data manipulation ○ Merging files ○ Data cleaning Day 2 - Data exploration and plotting using Python ● 9am - 10.30am: Explore a Drillhole file ○ Clean and select the relevant information ○ Merge data ● 10.30am - 12.30pm: Interrogate your data ○ Understand data through general statistics ○ Going further with graphics using Seaborn ● 1.30pm - 3pm: Applications to Model Validation ○ Writing procedure for automatic reporting ○ understanding['more'] = understanding['bits'] + practice['more']