xray at SciPy 2015
•
1 like•2,021 views
N-D labeled arrays and datasets in Python Watch the talk on Youtube: https://www.youtube.com/watch?v=X0pAhJgySxk For more info: http://xray.readthedocs.org http://github.com/xray/xray
Report
Share
Report
Share
Download to read offline
Recommended
Htdp27.key
This document discusses algorithms analysis and complexity including:
1) Common sorting algorithms have quadratic time complexity O(N^2) while searching algorithms can have exponential complexity like O(2^N).
2) It provides an example of determining the complexity of two functions, showing one is O(N) while the other is O(N^N).
3) Vectors in Lisp can be used to initialize a data structure of a given length by applying a function to each index, and operations on vectors have constant time complexity.
CLIM Undergraduate Workshop: Tutorial on R Software - Huang Huang, Oct 23, 2017
CLIM Undergraduate Workshop: Tutorial on R Software - Huang Huang, Oct 23, 2017The Statistical and Applied Mathematical Sciences Institute
This document provides an R tutorial for an undergraduate climate workshop. It introduces key concepts in R including data types, arrays, matrices, data frames, packages, and basic plotting. It demonstrates how to perform calculations, subset data, install and load packages, create different plot types like histograms and maps, and use functions like quantile and quilt.plot. Exercises include drawing a histogram of ozone values and calculating quantiles.Mate tarea - 2º
The document contains examples of algebraic expressions that can be factorized. Specifically, it lists 14 different expressions involving variables like x, y, z, a, b, c, m, n, and coefficients. The expressions include differences of terms, sums of terms, products of terms with common factors that can be pulled out, and expressions within parentheses that can be distributed and combined.
Fs for creditors aging report
The document outlines the functional specifications for a creditor aging report. It takes in input parameters like vendor number, date range, and generates a report with 10 columns showing amounts owed in different aging periods: current, 1-30 days, 31-60 days, 61-90 days, 91-180 days, 181-365 days, 1-2 years, 2-3 years, 3-5 years, and over 5 years. The logic places the amount owed in the appropriate column based on the number of days between the invoice date and current date.
Ch14 34
This document discusses calculating the number of digits needed to write out large numbers. It finds that 11 digits are needed for 43,000,000,000. It also calculates that 95 digits are needed for 999, 369,693,100 digits for 9(9)9, 78 digits for (99)9, and 18 digits for 999. It concludes by showing the relative sizes of these numbers.
Nora
A player is running from first to second base at 25 ft/s when they are 10 ft from second base. The distance from home plate to each base is 60 ft. To find the rate of change of the player's distance from home plate, the Pythagorean theorem is used to relate the distances. Plugging in the known values of 50 ft from first base and 10 ft from second base yields that the rate of change of the player's distance from home plate is 125 ft/s.
R
R is a programming language and software environment for statistical analysis and graphics. It allows users to analyze data, create visualizations, and perform statistical tests. Common R commands include functions to get and set the working directory, list objects in the workspace, remove objects, view and set options, save and load the command history, and save and load the entire workspace. R supports various data structures like vectors, arrays, matrices, data frames, and lists to store and manipulate different types of data. Data can be input into R from files, databases, and Excel spreadsheets. Graphs and visualizations created in R can be exported to file formats like PNG, JPEG, PDF and others.
7th PreAlg - L74--May4
1) The document outlines assignments due on Tuesday May 11th including an odds math worksheet and turning in math CDs. It also provides a warm up with probability and geometry problems.
2) The lesson discusses simplifying square roots using the product property of square roots. It gives examples of simplifying square root expressions including 20, 24, 27, 125, 48, 216, 210, and 1000.
3) Additional context is provided about converting between square feet and square inches using multiplication.
Recommended
Htdp27.key
This document discusses algorithms analysis and complexity including:
1) Common sorting algorithms have quadratic time complexity O(N^2) while searching algorithms can have exponential complexity like O(2^N).
2) It provides an example of determining the complexity of two functions, showing one is O(N) while the other is O(N^N).
3) Vectors in Lisp can be used to initialize a data structure of a given length by applying a function to each index, and operations on vectors have constant time complexity.
CLIM Undergraduate Workshop: Tutorial on R Software - Huang Huang, Oct 23, 2017
CLIM Undergraduate Workshop: Tutorial on R Software - Huang Huang, Oct 23, 2017The Statistical and Applied Mathematical Sciences Institute
This document provides an R tutorial for an undergraduate climate workshop. It introduces key concepts in R including data types, arrays, matrices, data frames, packages, and basic plotting. It demonstrates how to perform calculations, subset data, install and load packages, create different plot types like histograms and maps, and use functions like quantile and quilt.plot. Exercises include drawing a histogram of ozone values and calculating quantiles.Mate tarea - 2º
The document contains examples of algebraic expressions that can be factorized. Specifically, it lists 14 different expressions involving variables like x, y, z, a, b, c, m, n, and coefficients. The expressions include differences of terms, sums of terms, products of terms with common factors that can be pulled out, and expressions within parentheses that can be distributed and combined.
Fs for creditors aging report
The document outlines the functional specifications for a creditor aging report. It takes in input parameters like vendor number, date range, and generates a report with 10 columns showing amounts owed in different aging periods: current, 1-30 days, 31-60 days, 61-90 days, 91-180 days, 181-365 days, 1-2 years, 2-3 years, 3-5 years, and over 5 years. The logic places the amount owed in the appropriate column based on the number of days between the invoice date and current date.
Ch14 34
This document discusses calculating the number of digits needed to write out large numbers. It finds that 11 digits are needed for 43,000,000,000. It also calculates that 95 digits are needed for 999, 369,693,100 digits for 9(9)9, 78 digits for (99)9, and 18 digits for 999. It concludes by showing the relative sizes of these numbers.
Nora
A player is running from first to second base at 25 ft/s when they are 10 ft from second base. The distance from home plate to each base is 60 ft. To find the rate of change of the player's distance from home plate, the Pythagorean theorem is used to relate the distances. Plugging in the known values of 50 ft from first base and 10 ft from second base yields that the rate of change of the player's distance from home plate is 125 ft/s.
R
R is a programming language and software environment for statistical analysis and graphics. It allows users to analyze data, create visualizations, and perform statistical tests. Common R commands include functions to get and set the working directory, list objects in the workspace, remove objects, view and set options, save and load the command history, and save and load the entire workspace. R supports various data structures like vectors, arrays, matrices, data frames, and lists to store and manipulate different types of data. Data can be input into R from files, databases, and Excel spreadsheets. Graphs and visualizations created in R can be exported to file formats like PNG, JPEG, PDF and others.
7th PreAlg - L74--May4
1) The document outlines assignments due on Tuesday May 11th including an odds math worksheet and turning in math CDs. It also provides a warm up with probability and geometry problems.
2) The lesson discusses simplifying square roots using the product property of square roots. It gives examples of simplifying square root expressions including 20, 24, 27, 125, 48, 216, 210, and 1000.
3) Additional context is provided about converting between square feet and square inches using multiplication.
Quantum espresso G Vector distributon
This document describes the process of distributing reciprocal space grid points (G-vectors) across multiple processors for parallel computation in density functional theory (DFT) calculations. It involves 4 steps:
1) Initializing FFT descriptors and allocating data across processors
2) Mapping G-vectors to processors by iterating through grid points and checking if the vector satisfies cutoff criteria
3) Counting the number of G-vectors assigned to each processor
4) Sorting and distributing the G-vectors to optimize load balancing across processors
5.1 vertex y int zeros d r
This document contains 3 sections describing quadratic functions f(x) with their vertex, y-intercept, zeros, domain, and range. The first function is f(x) = 10 - 3x - x^2, the second is f(x) = 2x^2 - 12x, and the third is f(x) = (2 - x)(5 + x). For each function, the document lists the relevant properties to be determined but does not show the calculations or results.
8.7 numerical integration
The trapezoidal rule is used to approximate the area under a curve by dividing it into trapezoids. It takes the average of the function values at the beginning and end of each sub-interval. The area is calculated as the sum of the areas of each trapezoid multiplied by the width of the sub-interval. An example calculates the area under y=1+x^3 from 0 to 1 using n=4 sub-intervals, giving an approximate result of 1.26953125. The document also provides an example of using the trapezoidal rule with n=8 sub-intervals to estimate the area under the curve of the function y=x from 0 to 3.
8th PreAlg - L74--April12
The document contains an assignment list, reminders for SAT testing, and math problems and examples for Lesson 74. It provides instructions to complete Set 74 of even problems by Monday and mentions the possibility of extra credit on Test #10. Students are reminded to bring a calculator and SSR book for upcoming SAT testing. The math problems cover probability, geometry, simplifying fractions and percentages. Examples show how to simplify square roots using properties.
Ch17 25
This document discusses calculating the limit of the composition of two functions f and g as x approaches a value. For part (a), as x approaches 0 from the positive side, g(x) approaches 2 from the positive side and f(x) approaches negative infinity as x approaches 2 from the positive side. For part (b), as x approaches 5 from the negative side, g(x) approaches positive infinity and as x approaches positive infinity, f(x) approaches 2.
Ejercicio1
The document discusses two calculations of the volume of a solid:
1) The volume of a disc bounded by the parabola x + y = 3 and the x-axis is 9π.
2) The volume of the shell bounded by the same parabola and the planes y = 0 and y = 3 is also 9π.
Basic Scientific notation problems
Scientific notation is used to write very large or small numbers in a normalized way as the product of a number between 1 and 10 and a power of 10. It has useful properties for calculators, scientists, mathematicians and engineers. Some examples of numbers in scientific notation and their standard decimal form are provided. Conversions between scientific notation and standard decimal notation are demonstrated.
Python Lecture 12
This lecture discusses Python's datetime and OS modules. It shows how to use datetime to work with dates, times, and datetimes, including creating, formatting, and manipulating them. It also demonstrates various OS module functions for working with directories and files, such as getting/changing the current directory, listing contents, creating/removing directories, renaming files, checking file attributes, and more.
Halus hukumlinear
This document contains a log-log plot with x and y values listed. It also contains instructions to (a) plot the points on log-log axes with a uniform scale, draw a line of best fit, (b) use the y-intercept to determine p, use the gradient to determine q, and use the line equation to determine y.
Practica productos notables
This document contains several algebra problems involving factoring and expanding expressions. It provides the steps to solve equations like (x+3)(x+7)=x^2 +10x+21 and determines coefficients of expressions like (x+3)(x-5)=x^2 -2x-15. It also covers topics like simplifying radicals, finding coefficients from expanded forms, and using formulas for factoring cubic expressions.
Capinha de matemática
This document contains 8 math problems of varying difficulty including multiplication, addition, subtraction and division. The problems range from single digit operations like 6x9+8 to longer multi-digit problems such as 967-878 and 9876+1899. Solving all the problems completely would require showing the step-by-step work and calculations.
MNISTデータセットを用いたCNNの実装
This document discusses convolutional neural networks (CNNs) for classifying images from the MNIST dataset. It begins with an introduction to MNIST, then describes the basic architecture of a CNN including convolution, ReLU, pooling, affine transformations, and softmax layers. It also discusses implementing CNNs in Python using frameworks like TensorFlow and Keras. The goal is to classify MNIST images with a CNN running on a GPU to take advantage of deep learning.
Ejercicios de ecuaciones diferenciales
1. This document lists 20 differential equations problems assigned as homework for a class.
2. The problems cover a range of differential equation types including separable, exact, linear, and higher order equations.
3. The document provides the list of problems but no further context or explanations for solving the equations.
New Capabilities in the PyData Ecosystem
This document summarizes new capabilities in the PyData ecosystem of tools for scientific computing and data science in Python. It focuses on Bokeh and Dask, which enable interactive visualization and out-of-core parallel computing respectively. Bokeh allows creating interactive web-based visualizations without writing JavaScript, while Dask enables parallel computing on large datasets that exceed memory using task scheduling. The document also briefly mentions related tools like Blaze, NumPy, Pandas, Jupyter notebooks, and conda for package and environment management.
Numba Overview
Numba is a just-in-time compiler for Python that can optimize numerical code to achieve speeds comparable to C/C++ without requiring the user to write C/C++ code. It works by compiling Python functions to optimized machine code using type information. Numba supports NumPy arrays and common mathematical functions. It can automatically optimize loops and compile functions for CPU or GPU execution. Numba allows users to write high-performance numerical code in Python without sacrificing readability or development speed.
Numba
Slides from a SciPy 2012 Talk on the motivation and progress of Numba: a Python dynamic compiler using LLVM.
Fast and Scalable Python
This document discusses tools for making NumPy and Pandas code faster and able to run in parallel. It introduces the Dask library, which allows users to work with large datasets in a familiar Pandas/NumPy style through parallel computing. Dask implements parallel DataFrames, Arrays, and other collections that mimic their Pandas/NumPy counterparts. It can scale computations across multiple cores on a single machine or across many machines in a cluster. The document provides examples of using Dask to analyze large CSV and text data in parallel through DataFrames and Bags. It also discusses scaling computations from a single laptop to large clusters.
Numba: Flexible analytics written in Python with machine-code speeds and avo...
Numba provides a way to write high performance Python code using NumPy-like syntax. It works by compiling Python code with NumPy arrays and loops into fast machine code using the LLVM compiler. This allows code written in Python to achieve performance comparable to C/C++ with little or no code changes required. Numba supports CPU and GPU execution via backends like CUDA. It can improve performance of numerical Python code with features like releasing the global interpreter lock during compilation.
IPython & Jupyter
Este documento describe la evolución de IPython y Jupyter, desde sus inicios como un shell interactivo de Python hasta convertirse en una plataforma multi-lenguaje para computación interactiva y publicación de documentos. Se explica cómo el protocolo REPL genérico de Jupyter permite ejecutar código en múltiples lenguajes y cómo herramientas como JupyterHub, nbviewer y notebooks han impulsado su adopción en educación, investigación y comunicación científica.
LS1.pptx
I am lokesh kanna from Anna University Regional Campus Tirunelveli make use of the resource you got
Machine learning is the ability of machine to understand languages to machine it is a low level language that is used by humans to give command
A variety of algorithms may be applied depending on the nature of the earth science exploration. Some algorithms may perform significantly better than others for particular objectives. For example, convolutional neural networks (CNN) are good at interpreting images, artificial neural networks (ANN) perform well in soil classification[4] but more computationally expensive to train than support-vector machine (SVM) learning. The application of machine learning has been popular in recent decades, as the development of other technologies such as unmanned aerial vehicles (UAVs),[5] ultra-high resolution remote sensing technology and high-performance computing units[6] lead to the availability of large high-quality datasets and more advanced algorithms.
Données hospitalières relatives à l'épidémie de COVID-19 FRANCE
Exploitation rapide des données hospitalières relatives à l'épidémie de COVID-19 - Source Santé Publique France
Apache Cassandra for Timeseries- and Graph-Data
Apache Cassandra has proven to be one of the best solutions for storing and retrieving data at high velocity and high volume.
In the first part of the talk we will discuss how the storage model of Cassandra is ideal for time series use cases, which are often of high velocity and high volume. Time series data is everywhere today: Internet of Things, sensor data, transactional data, social media streams. We go over examples of how to best build data models.
We will also cover pairing Apache Spark with Apache Cassandra to create a real time data analytics platform.
The second part of the talk will present Titan:db, an open source distributed graph database build on top of Cassandra that can power real-time applications with thousands of concurrent users over graphs with billions of edges. It exposes a property graph data model directly atop Cassandra which makes storing and querying relationship data fast, easy, and scalable to huge graphs. This talk demonstrates how Titan's features enable complex, multi-relational databases in Cassandra and discusses how Titan:db has been used in a customer case to store social network data.
More Related Content
What's hot
Quantum espresso G Vector distributon
This document describes the process of distributing reciprocal space grid points (G-vectors) across multiple processors for parallel computation in density functional theory (DFT) calculations. It involves 4 steps:
1) Initializing FFT descriptors and allocating data across processors
2) Mapping G-vectors to processors by iterating through grid points and checking if the vector satisfies cutoff criteria
3) Counting the number of G-vectors assigned to each processor
4) Sorting and distributing the G-vectors to optimize load balancing across processors
5.1 vertex y int zeros d r
This document contains 3 sections describing quadratic functions f(x) with their vertex, y-intercept, zeros, domain, and range. The first function is f(x) = 10 - 3x - x^2, the second is f(x) = 2x^2 - 12x, and the third is f(x) = (2 - x)(5 + x). For each function, the document lists the relevant properties to be determined but does not show the calculations or results.
8.7 numerical integration
The trapezoidal rule is used to approximate the area under a curve by dividing it into trapezoids. It takes the average of the function values at the beginning and end of each sub-interval. The area is calculated as the sum of the areas of each trapezoid multiplied by the width of the sub-interval. An example calculates the area under y=1+x^3 from 0 to 1 using n=4 sub-intervals, giving an approximate result of 1.26953125. The document also provides an example of using the trapezoidal rule with n=8 sub-intervals to estimate the area under the curve of the function y=x from 0 to 3.
8th PreAlg - L74--April12
The document contains an assignment list, reminders for SAT testing, and math problems and examples for Lesson 74. It provides instructions to complete Set 74 of even problems by Monday and mentions the possibility of extra credit on Test #10. Students are reminded to bring a calculator and SSR book for upcoming SAT testing. The math problems cover probability, geometry, simplifying fractions and percentages. Examples show how to simplify square roots using properties.
Ch17 25
This document discusses calculating the limit of the composition of two functions f and g as x approaches a value. For part (a), as x approaches 0 from the positive side, g(x) approaches 2 from the positive side and f(x) approaches negative infinity as x approaches 2 from the positive side. For part (b), as x approaches 5 from the negative side, g(x) approaches positive infinity and as x approaches positive infinity, f(x) approaches 2.
Ejercicio1
The document discusses two calculations of the volume of a solid:
1) The volume of a disc bounded by the parabola x + y = 3 and the x-axis is 9π.
2) The volume of the shell bounded by the same parabola and the planes y = 0 and y = 3 is also 9π.
Basic Scientific notation problems
Scientific notation is used to write very large or small numbers in a normalized way as the product of a number between 1 and 10 and a power of 10. It has useful properties for calculators, scientists, mathematicians and engineers. Some examples of numbers in scientific notation and their standard decimal form are provided. Conversions between scientific notation and standard decimal notation are demonstrated.
Python Lecture 12
This lecture discusses Python's datetime and OS modules. It shows how to use datetime to work with dates, times, and datetimes, including creating, formatting, and manipulating them. It also demonstrates various OS module functions for working with directories and files, such as getting/changing the current directory, listing contents, creating/removing directories, renaming files, checking file attributes, and more.
Halus hukumlinear
This document contains a log-log plot with x and y values listed. It also contains instructions to (a) plot the points on log-log axes with a uniform scale, draw a line of best fit, (b) use the y-intercept to determine p, use the gradient to determine q, and use the line equation to determine y.
Practica productos notables
This document contains several algebra problems involving factoring and expanding expressions. It provides the steps to solve equations like (x+3)(x+7)=x^2 +10x+21 and determines coefficients of expressions like (x+3)(x-5)=x^2 -2x-15. It also covers topics like simplifying radicals, finding coefficients from expanded forms, and using formulas for factoring cubic expressions.
Capinha de matemática
This document contains 8 math problems of varying difficulty including multiplication, addition, subtraction and division. The problems range from single digit operations like 6x9+8 to longer multi-digit problems such as 967-878 and 9876+1899. Solving all the problems completely would require showing the step-by-step work and calculations.
MNISTデータセットを用いたCNNの実装
This document discusses convolutional neural networks (CNNs) for classifying images from the MNIST dataset. It begins with an introduction to MNIST, then describes the basic architecture of a CNN including convolution, ReLU, pooling, affine transformations, and softmax layers. It also discusses implementing CNNs in Python using frameworks like TensorFlow and Keras. The goal is to classify MNIST images with a CNN running on a GPU to take advantage of deep learning.
Ejercicios de ecuaciones diferenciales
1. This document lists 20 differential equations problems assigned as homework for a class.
2. The problems cover a range of differential equation types including separable, exact, linear, and higher order equations.
3. The document provides the list of problems but no further context or explanations for solving the equations.
What's hot (13)
Viewers also liked
New Capabilities in the PyData Ecosystem
This document summarizes new capabilities in the PyData ecosystem of tools for scientific computing and data science in Python. It focuses on Bokeh and Dask, which enable interactive visualization and out-of-core parallel computing respectively. Bokeh allows creating interactive web-based visualizations without writing JavaScript, while Dask enables parallel computing on large datasets that exceed memory using task scheduling. The document also briefly mentions related tools like Blaze, NumPy, Pandas, Jupyter notebooks, and conda for package and environment management.
Numba Overview
Numba is a just-in-time compiler for Python that can optimize numerical code to achieve speeds comparable to C/C++ without requiring the user to write C/C++ code. It works by compiling Python functions to optimized machine code using type information. Numba supports NumPy arrays and common mathematical functions. It can automatically optimize loops and compile functions for CPU or GPU execution. Numba allows users to write high-performance numerical code in Python without sacrificing readability or development speed.
Numba
Slides from a SciPy 2012 Talk on the motivation and progress of Numba: a Python dynamic compiler using LLVM.
Fast and Scalable Python
This document discusses tools for making NumPy and Pandas code faster and able to run in parallel. It introduces the Dask library, which allows users to work with large datasets in a familiar Pandas/NumPy style through parallel computing. Dask implements parallel DataFrames, Arrays, and other collections that mimic their Pandas/NumPy counterparts. It can scale computations across multiple cores on a single machine or across many machines in a cluster. The document provides examples of using Dask to analyze large CSV and text data in parallel through DataFrames and Bags. It also discusses scaling computations from a single laptop to large clusters.
Numba: Flexible analytics written in Python with machine-code speeds and avo...
Numba provides a way to write high performance Python code using NumPy-like syntax. It works by compiling Python code with NumPy arrays and loops into fast machine code using the LLVM compiler. This allows code written in Python to achieve performance comparable to C/C++ with little or no code changes required. Numba supports CPU and GPU execution via backends like CUDA. It can improve performance of numerical Python code with features like releasing the global interpreter lock during compilation.
IPython & Jupyter
Este documento describe la evolución de IPython y Jupyter, desde sus inicios como un shell interactivo de Python hasta convertirse en una plataforma multi-lenguaje para computación interactiva y publicación de documentos. Se explica cómo el protocolo REPL genérico de Jupyter permite ejecutar código en múltiples lenguajes y cómo herramientas como JupyterHub, nbviewer y notebooks han impulsado su adopción en educación, investigación y comunicación científica.
Viewers also liked (6)
Numba: Flexible analytics written in Python with machine-code speeds and avo...
Numba: Flexible analytics written in Python with machine-code speeds and avo...
Similar to xray at SciPy 2015
LS1.pptx
I am lokesh kanna from Anna University Regional Campus Tirunelveli make use of the resource you got
Machine learning is the ability of machine to understand languages to machine it is a low level language that is used by humans to give command
A variety of algorithms may be applied depending on the nature of the earth science exploration. Some algorithms may perform significantly better than others for particular objectives. For example, convolutional neural networks (CNN) are good at interpreting images, artificial neural networks (ANN) perform well in soil classification[4] but more computationally expensive to train than support-vector machine (SVM) learning. The application of machine learning has been popular in recent decades, as the development of other technologies such as unmanned aerial vehicles (UAVs),[5] ultra-high resolution remote sensing technology and high-performance computing units[6] lead to the availability of large high-quality datasets and more advanced algorithms.
Données hospitalières relatives à l'épidémie de COVID-19 FRANCE
Exploitation rapide des données hospitalières relatives à l'épidémie de COVID-19 - Source Santé Publique France
Apache Cassandra for Timeseries- and Graph-Data
Apache Cassandra has proven to be one of the best solutions for storing and retrieving data at high velocity and high volume.
In the first part of the talk we will discuss how the storage model of Cassandra is ideal for time series use cases, which are often of high velocity and high volume. Time series data is everywhere today: Internet of Things, sensor data, transactional data, social media streams. We go over examples of how to best build data models.
We will also cover pairing Apache Spark with Apache Cassandra to create a real time data analytics platform.
The second part of the talk will present Titan:db, an open source distributed graph database build on top of Cassandra that can power real-time applications with thousands of concurrent users over graphs with billions of edges. It exposes a property graph data model directly atop Cassandra which makes storing and querying relationship data fast, easy, and scalable to huge graphs. This talk demonstrates how Titan's features enable complex, multi-relational databases in Cassandra and discusses how Titan:db has been used in a customer case to store social network data.
Project presentation(View calender)
The document summarizes a student's course project presentation on creating a calendar program. The program displays a calendar for any given year, showing month names, days of the week, and dates. It allows the user to input the year to display. The presentation describes the motivation, implementation requirements, algorithms, demonstration of code, inputs, outputs, future improvements, references, and concludes with a thank you.
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022
The document summarizes the evolution of InfluxDB from its initial version 1.0 in 2013 to the current version 2.0 called IOx. It started as a time series database that stored time series data and associated metadata. Over time it incorporated features like tags, line protocol, TSM storage engine, and an inverted index to improve querying capabilities. Version 2.0 refocused it as an all-in-one platform with a new query language called Flux, and aims to be cloud-first. The latest version IOx leverages a columnar database and federated architecture to solve challenges of scale, providing SQL support and the ability to deploy on cloud or edge environments.
Time Series.pptx
Time series analysis in R allows one to analyze how a variable changes over time. The ts() function is used to create time series objects by specifying the data vector, start and end dates, and frequency. Common applications include sales analysis, inventory analysis, and analyzing trends in variables like COVID-19 cases over time. Multivariate time series can also be created to analyze multiple related time series in a single plot.
Brace yourselves, leap second is coming
As the leap second approaches, there is no better time to reflect on our misconceptions about time and numerals, past catastrophes and possible mitigation techniques.
Computer Science Presentation for various MATLAB toolboxes
The document describes modeling a satellite constellation using ephemeris data. It allows defining a constellation mission with start/end dates and duration. Ephemeris data for satellite position and velocity over time is loaded from a file. A satellite scenario is created using the ephemeris data to model the satellite positions. Ground stations in the US, Germany and India are defined. Access coverage between the satellites and ground stations over the scenario duration can then be analyzed.
Coding matlab
The document describes a scenario to analyze access between a constellation of 40 low-Earth orbit satellites and a ground station located at MathWorks Natick. A satellite scenario is created in MATLAB and the constellation satellites are added along with their orbital parameters. Each satellite is equipped with a conical sensor camera with a 90-degree field of view. The ground station representing MathWorks Natick is also added with a minimum elevation angle of 30 degrees. Access analysis is performed between each camera and the ground station to determine the times each camera can photograph the site. The results show the start and end times of access intervals for each camera over the 6-hour period from 1:00 PM to 7:00 PM UTC on May 12,
Coding matlab
The document describes a scenario to analyze access between a constellation of 40 low-Earth orbit satellites and a ground station located at MathWorks Natick. A satellite scenario is created in MATLAB and the constellation satellites are added along with their orbital parameters. Each satellite is equipped with a conical sensor camera with a 90-degree field of view. The ground station representing MathWorks Natick is also added with a minimum elevation angle of 30 degrees. Access analysis is performed between each camera and the ground station to determine the times each camera can photograph the site. The results show the start and end times of each access interval over the 6-hour period.
論文紹介：Is Space-Time Attention All You Need for Video Understanding?
Gedas Bertasius, Heng Wang, Lorenzo Torresani, "Is Space-Time Attention All You Need for Video Understanding?" ICML2021
https://proceedings.mlr.press/v139/bertasius21a.html
Introduction to R
R is an open source statistical computing platform that is rapidly growing in popularity within academia. It allows for statistical analysis and data visualization. The document provides an introduction to basic R functions and syntax for assigning values, working with data frames, filtering data, plotting, and connecting to databases. More advanced techniques demonstrated include decision trees, random forests, and other data mining algorithms.
D3.js - A picture is worth a thousand words
This document provides an overview of D3.js, a JavaScript library for data visualization. It discusses why data visualization is useful, some key concepts in D3 like selections, entering and updating data, and creating reusable components. It also covers transitions, scales, axes, SVG, and common layouts. The document encourages exploring more examples on the bl.ocks website and concludes by thanking the audience.
Fun with D3.js: Data Visualization Eye Candy with Streaming JSON
The document discusses creating dynamic bubble charts using D3.js and streaming JSON data from PubNub. It explains how to (1) create a static bubble chart with D3, (2) make the chart dynamic by subscribing to a PubNub data stream and updating the bubbles on new data, and (3) add smooth transitions as bubbles enter, update, and exit using D3's data binding and transition methods. The full article provides more details on implementing this dynamic bubble chart with animated transitions between data updates.
Datastax day 2016 : Cassandra data modeling basics
This document discusses data modeling with Apache Cassandra. It covers:
1. The objectives of data modeling like reducing query latency and avoiding disasters
2. Choosing the right partition key which is the main entry point for queries and helps distribute data
3. Using clustering columns to simulate one-to-many relationships and enable sorting and range queries
4. Other critical details like avoiding huge partitions, sub-partitioning techniques, and how deletes create tombstones
Cloudera - A Taste of random decision forests
This document provides an overview of random decision forests for classification using Apache Spark's MLlib. It begins with an introduction to decision trees using sample datasets. It then demonstrates how to build and evaluate a decision tree model on the Covertype dataset using MLlib. The document discusses techniques for improving decision trees such as hyperparameter tuning. It introduces the idea of random decision forests as an ensemble method that leverages multiple decision trees to improve accuracy. It explains how random decision forests create diversity among trees and shows how to train a random forest classifier on the Covertype data, achieving around 96% accuracy.
A Century Of Weather Data - Midwest.io
This document summarizes the key considerations and performance tests for storing and querying a large weather dataset containing over 2.5 billion data points. It describes the schema design using MongoDB to embed data and index on location. Bulk loading of data was 10 hours on a single server but only 3 hours on a sharded cluster. Queries for a single data point were fastest on the cluster at under 1ms while worldwide queries were faster at 310/second. Analytics like maximum temperature took 2.5 hours on a single server but only 2 minutes on the cluster. The cluster provided much higher throughput and better performance for complex queries while being more expensive.
data_selectionOctober 19, 2022[1] # Data Selection.docx
data_selection
October 19, 2022
[1]: # Data Selection
[2]: import numpy as np
[3]: # This is weather data recorded in Memphis during summer (June to September).
# Column 0: month
# Column 1: temperature in Farenheit
# Column 2: precipitation in inches
data = np.array([
[6, 70, 3],
[7, 75, 3],
[6, 85, 4],
[7, 90, 4],
[7, 91, 5],
[8, 85, 2],
[8, 87, 4],
[6, 83, 5],
[8, 77, 3],
[6, 69, 6],
[9, 68, 1],
[6, 80, 6],
[9, 65, 3],
[9, 75, 4],
[9, 80, 5]])
[4]: data.shape
[4]: (15, 3)
[5]: # Select the data for the row 0:
data[0, :]
# row_selection: 0
# column_selection: all
[5]: array([ 6, 70, 3])
1
[6]: # Select the data of column 2:
data[:, 2]
# row_selection: all
# column_selection: 2
[6]: array([3, 3, 4, 4, 5, 2, 4, 5, 3, 6, 1, 6, 3, 4, 5])
[7]: # Get the data for the first five rows.
data[0:5, :]
[7]: array([[ 6, 70, 3],
[ 7, 75, 3],
[ 6, 85, 4],
[ 7, 90, 4],
[ 7, 91, 5]])
[8]: # Get the data for the first five rows,
# and the first two columns.
data[0:5, 0:2]
[8]: array([[ 6, 70],
[ 7, 75],
[ 6, 85],
[ 7, 90],
[ 7, 91]])
[9]: # Get the data for the last two columns,
# and the first five rows.
data[0:5, 1:3]
[9]: array([[70, 3],
[75, 3],
[85, 4],
[90, 4],
[91, 5]])
[10]: # or can be written as
data[:5, 1:]
[10]: array([[70, 3],
[75, 3],
[85, 4],
[90, 4],
[91, 5]])
[11]: # or can be written as
data[:5, -2:]
2
[11]: array([[70, 3],
[75, 3],
[85, 4],
[90, 4],
[91, 5]])
[12]: # Get the last 4 rows
data[-4:, :]
[12]: array([[ 6, 80, 6],
[ 9, 65, 3],
[ 9, 75, 4],
[ 9, 80, 5]])
[13]: # Find the temperature values, and store them in a variable
temp = data[:, 1]
[14]: temp
[14]: array([70, 75, 85, 90, 91, 85, 87, 83, 77, 69, 68, 80, 65, 75, 80])
[15]: # Find the month values, and store them in a variable
month = data[:, 0]
[16]: month
[16]: array([6, 7, 6, 7, 7, 8, 8, 6, 8, 6, 9, 6, 9, 9, 9])
[17]: # Find the maximum temperature
np.max(temp)
[17]: 91
[18]: # Find the index (or position) of the maximum temperature
np.argmax(temp)
[18]: 4
[19]: # Find the month that corresponds to the maximum temperature
data[np.argmax(temp), 0]
[19]: 7
[20]: m = np.argmax(temp)
data[m, 0]
[20]: 7
3
[21]: # boolean selection
[22]: # Find all the temperatures below 70 degrees
data[temp < 70, 1]
[22]: array([69, 68, 65])
[23]: # Find the months with temperatures below 70 degrees
data[temp < 70, 0]
[23]: array([6, 9, 9])
[24]: np.unique(data[temp < 70, 0])
[24]: array([6, 9])
[25]: # Find all the temperatures for the month of August
data[month == 8, 1]
[25]: array([85, 87, 77])
[26]: # Find the average temperature for August
np.average(data[month == 8, 1])
[26]: 83.0
[27]: # Find the temperatures above 80 for June
data[(month == 6) & (temp > 80), 1]
# & means and
[27]: array([85, 83])
[28]: # Find the temperatures for the months of June, July, and August
data[month != 9, 1]
[28]: array([70, 75, 85, 90, 91, 85, 87, 83, 77, 69, 80])
[29]: data[(month == 6) | (month == 7) | (month == 8), 1]
[29]: array([70, 75, 85, 90.
Using the python_data_toolkit_timbers_slides
This document discusses using the Pandas Python data analysis library. It describes how Pandas makes it easy to load, manipulate, and visualize complex tabular data. Specific features highlighted include loading data from files, creating and manipulating dataframe data structures, performing element-wise math operations on columns, working with time series data through indexing and resampling, and quick visualization of data.
Jsr310 - Java 8 Date and Time API
This document provides an overview of the Java 8 Date and Time API as defined by JSR 310. It describes the key classes like Instant, LocalDate, LocalDateTime, and ZonedDateTime and how they can be used to work with dates, times, durations, and timezones. Examples are given showing how to get the current date and time, parse and format dates, add/subtract periods, and convert between different date and time representations.
Similar to xray at SciPy 2015 (20)
Données hospitalières relatives à l'épidémie de COVID-19 FRANCE
Données hospitalières relatives à l'épidémie de COVID-19 FRANCE
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022
Computer Science Presentation for various MATLAB toolboxes
Computer Science Presentation for various MATLAB toolboxes
論文紹介：Is Space-Time Attention All You Need for Video Understanding?
論文紹介：Is Space-Time Attention All You Need for Video Understanding?
Fun with D3.js: Data Visualization Eye Candy with Streaming JSON
Fun with D3.js: Data Visualization Eye Candy with Streaming JSON
Datastax day 2016 : Cassandra data modeling basics
Datastax day 2016 : Cassandra data modeling basics
data_selectionOctober 19, 2022[1] # Data Selection.docx
data_selectionOctober 19, 2022[1] # Data Selection.docx
Recently uploaded
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Recently uploaded (20)
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
xray at SciPy 2015
- 5. ● ● ●
- 8. >>> ds <xray.Dataset> Dimensions: (time: 10, latitude: 8, longitude: 8) Coordinates: * time (time) datetime64 2015-01-01 2015-01-02 2015-01-03 2015-01-04 ... * latitude (latitude) float64 50.0 47.5 45.0 42.5 40.0 37.5 35.0 32.5 * longitude (longitude) float64 -105.0 -102.5 -100.0 -97.5 -95.0 -92.5 ... elevation (longitude, latitude) int64 201 231 582 239 1848 1004 1004 ... land_cover (longitude, latitude) object 'forest' 'urban' 'farmland'... Data variables: temperature (time, longitude, latitude) float64 13.7 8.031 18.36 24.95 ... pressure (time, longitude, latitude) float64 1.374 1.142 1.388 0.9992 ...
- 9. # numpy style ds.temperature[0, :, :] # pandas style ds.temperature.loc[:, -90, 50] # with dimension names ds.sel(time='2015-01-01') ds.sel(longitude=-90, latitude=50, method='nearest')
- 10. # math (10 + ds) ** 0.5 ds.temperature + ds.pressure np.sin(ds.temperature) # aggregation ds.mean(dim='time') ds.max(dim=['latitude', 'longitude'])
- 11. time time space space + = Result has the union of all dimension names
- 12. year + = Result has the intersection of coordinate labels 2000 2001 2002 2003 2004 2005 2006 2007 2008 X X X X X
- 13. # average by calendar month ds.groupby('time.month').mean('time') # resample to every 10 days ds.resample('10D', dim='time', how='max')
- 14. # xray -> numpy ds.temperature.values # xray -> pandas ds.to_dataframe() # pandas -> xray xray.Dataset.from_dataframe(df)
- 15. >>> ds = xray.open_mfdataset('/Users/shoyer/data/era-interim/*.nc') >>> ds <xray.Dataset> Dimensions: (latitude: 256, longitude: 512, time: 52596) Coordinates: * latitude (latitude) float32 89.4628 88.767 88.067 87.3661 86.6648 ... * longitude (longitude) float32 0.0 0.703125 1.40625 2.10938 2.8125 ... * time (time) datetime64[ns] 1979-01-01 1979-01-01T06:00:00 ... Data variables: t2m (time, latitude, longitude) float64 240.6 240.6 240.6 ... >>> ds.nbytes * (2 ** -30) 51.363675981760025
- 16. ds_by_season = ds.groupby('time.season').mean('time') t2m_range = abs(ds_by_season.sel(season='JJA') - ds_by_season.sel(season='DJF')).t2m %time result = t2m_range.load() CPU times: user 2min 1s, sys: 49.5 s, total: 2min 51s Wall time: 38.6 s
- 19. pandas: indexing, factorize NumPy: arrays netCDF4, h5py, SciPy: IO dask.array: out of core arrays