This presentation has slides from a talk that I gave at the annual Experimental Biology meeting, 2015, on our curriculum for Big Data Analytics in the Inland Empire.
An introductory course on building ML applications with primary focus on supervised learning. Covers the typical ML application cycle - Problem formulation, data definitions, offline modeling, platform design. Also, includes key tenets for building applications.
Note: This is an old slide deck. The content on building internal ML platforms is a bit outdated and slides on the model choices do not include deep learning models.
This talk is a primer to Machine Learning. I will provide a brief introduction what is ML and how it works. I will walk you down the Machine Learning pipeline from data gathering, data normalizing and feature engineering, common supervised and unsupervised algorithms, training models, and delivering results to production. I will also provide recommendations to tools that help you provide the best ML experience, include programming languages and libraries.
If there is time at the end of the talk, I will walk through two coding examples, using the HMS Titanic Passenger List, present with Python scikit-learn using algorithm random-trees to check if ML can correctly predict passenger survival and with R programming for feature engineering of the same dataset
Note to data-scientists and programmers: If you sign up to attend, plan to visit my Github repository! I have many Machine Learning coding examples in Python scikit-learn, GNU Octave, and R Programming.
https://github.com/jefftune/gitw-2017-ml
This is an introductory workshop for machine learning. Introduced machine learning tasks such as supervised learning, unsupervised learning and reinforcement learning.
Big Data Analytics (ML, DL, AI) hands-onDony Riyanto
Ini adalah slide tambahan dari materi pengenalan Big Data Analytics (di file berikutnya), yang mengajak kita mulai hands-on dengan beberapa hal terkait Machine/Deep Learning, Big Data (batch/streaming), dan AI menggunakan Tensor Flow
Using only simple rules for local interactions, groups of agents can form self-organizing super-organisms or “flocks” that show global emergent behavior. When agents are also extended with memory and goals the resulting flock not only demonstrates emergent behavior, but also collective intelligence: the ability for the group to solve problems that might be beyond the ability of the individual alone. Until now, research has focused on the improvement of particle design for global behavior; however, techniques for human-designed particles are task-specific. In this paper we will demonstrate that evolutionary computing techniques can be applied to design particles, not only to optimize the parameters for movement but also the structure of controlling finite state machines that enable collective intelligence. The evolved design not only exhibits emergent, self-organizing behavior but also significantly outperforms a human design in a specific problem domain. The strategy of the evolved design may be very different from what is intuitive to humans and perhaps reflects more accurately how nature designs systems for problem solving. Furthermore, evolutionary design of particles for collective intelligence is more flexible and able to target a wider array of problems either individually or as a whole.
An introductory course on building ML applications with primary focus on supervised learning. Covers the typical ML application cycle - Problem formulation, data definitions, offline modeling, platform design. Also, includes key tenets for building applications.
Note: This is an old slide deck. The content on building internal ML platforms is a bit outdated and slides on the model choices do not include deep learning models.
This talk is a primer to Machine Learning. I will provide a brief introduction what is ML and how it works. I will walk you down the Machine Learning pipeline from data gathering, data normalizing and feature engineering, common supervised and unsupervised algorithms, training models, and delivering results to production. I will also provide recommendations to tools that help you provide the best ML experience, include programming languages and libraries.
If there is time at the end of the talk, I will walk through two coding examples, using the HMS Titanic Passenger List, present with Python scikit-learn using algorithm random-trees to check if ML can correctly predict passenger survival and with R programming for feature engineering of the same dataset
Note to data-scientists and programmers: If you sign up to attend, plan to visit my Github repository! I have many Machine Learning coding examples in Python scikit-learn, GNU Octave, and R Programming.
https://github.com/jefftune/gitw-2017-ml
This is an introductory workshop for machine learning. Introduced machine learning tasks such as supervised learning, unsupervised learning and reinforcement learning.
Big Data Analytics (ML, DL, AI) hands-onDony Riyanto
Ini adalah slide tambahan dari materi pengenalan Big Data Analytics (di file berikutnya), yang mengajak kita mulai hands-on dengan beberapa hal terkait Machine/Deep Learning, Big Data (batch/streaming), dan AI menggunakan Tensor Flow
Using only simple rules for local interactions, groups of agents can form self-organizing super-organisms or “flocks” that show global emergent behavior. When agents are also extended with memory and goals the resulting flock not only demonstrates emergent behavior, but also collective intelligence: the ability for the group to solve problems that might be beyond the ability of the individual alone. Until now, research has focused on the improvement of particle design for global behavior; however, techniques for human-designed particles are task-specific. In this paper we will demonstrate that evolutionary computing techniques can be applied to design particles, not only to optimize the parameters for movement but also the structure of controlling finite state machines that enable collective intelligence. The evolved design not only exhibits emergent, self-organizing behavior but also significantly outperforms a human design in a specific problem domain. The strategy of the evolved design may be very different from what is intuitive to humans and perhaps reflects more accurately how nature designs systems for problem solving. Furthermore, evolutionary design of particles for collective intelligence is more flexible and able to target a wider array of problems either individually or as a whole.
機器學習速遊 (Quick Tour of Machine Learning)
機器學習旨在讓電腦能由資料中累積的經驗來自我進步,近年來已廣泛應用於資料探勘、計算機視覺、自然語言處理、生物特徵識別、搜尋引擎、醫學診斷、檢測信用卡欺詐、證券市場分析、DNA序列測序、語音和手寫識別、戰略遊戲和機器人等領域。它已成為資料科學的基礎學科之一,為任何資料科學家必備的工具。
這門課程將由台大資訊工程系林軒田教授利用短短的六個小時,快速地帶大家探索機器學習的基石、介紹核心的模型及一些熱門的技法,希望幫助大家有效率而紮實地了解這個領域,以妥善地使用各式機器學習的工具。此課程適合所有希望開始運用資料的資料分析者,推薦給所有有志於資料分析領域的資料科學愛好者。
Mengenal Machine/Deep Learning, Artificial Intelligence dan mengenal apa bedanya dengan Business Intelligence, apa hubungannya dengan Big Data dan Data Science/Analytics.
Machine learning with Big Data power point presentationDavid Raj Kanthi
This is an article made form the articles of IEEE published in the year 2017
The following presentation has the slides for the Title called the
Machine Learning with Big data. that following presentation which has the challenges and approaches of machine learning with big data.
The integration of the Big Data with Machine Learning has so many challenges that Big data has and what is the approach made by the machine learning mechanism for those challenges.
Shou-de Lin is currently a full professor in the CSIE department of National Taiwan University. He holds a BS in EE department from National Taiwan University, an MS-EE from the University of Michigan, and an MS in Computational Linguistics and PhD in Computer Science both from the University of Southern California. He leads the Machine Discovery and Social Network Mining Lab in NTU. Before joining NTU, he was a post-doctoral research fellow at the Los Alamos National Lab. Prof. Lin's research includes the areas of machine learning and data mining, social network analysis, and natural language processing. His international recognition includes the best paper award in IEEE Web Intelligent conference 2003, Google Research Award in 2007, Microsoft research award in 2008, merit paper award in TAAI 2010, best paper award in ASONAM 2011, US Aerospace AFOSR/AOARD research award winner for 5 years. He is the all-time winners in ACM KDD Cup, leading or co-leading the NTU team to win 5 championships. He also leads a team to win WSDM Cup 2016 Champion. He has served as the senior PC for SIGKDD and area chair for ACL. He is currently the associate editor for International Journal on Social Network Mining, Journal of Information Science and Engineering, and International Journal of Computational Linguistics and Chinese Language Processing. He receives the Young Scholars' Creativity Award from Foundation for the Advancement of Outstanding Scholarship and Ta-You Wu Memorial Award.
Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. Machine learning is so pervasive today that you probably use it dozens of times a day without knowing it.
We will review some modern machine learning applications, understand variety of machine learning problem definitions, go through particular approaches of solving machine learning tasks.
This year 2015 Amazon and Microsoft introduced services to perform machine learning tasks in cloud. Microsoft Azure Machine Learning offers a streamlined experience for all data scientist skill levels, from setting up with only a web browser, to using drag and drop gestures and simple data flow graphs to set up experiments.
We will briefly review Azure ML Studio features and run machine learning experiment.
The AML group carries out both theoretical and experimental work on developing and applying new machine learning techniques for solving various application problems.
Artificial Intelligence for Automating Data AnalysisManuel Martín
The requirements for analysing big volumes of data have increased over the last few decades. The process of selecting, cleaning, modelling and interpreting data is called the KDD process. The decision of how to approach each step in this process has often been made manually by experts. However, experts cannot be aware of all methods, nor is it feasible to try all of them. Researchers have proposed different approaches for automating, or at least advising, the stages of the KDD process. This talk will outline the different types of Intelligent Discovery Assistants as described in the work of Serban et al. “A survey of intelligent assistants for data analysis” and point out some future directions.
Data Science, Machine Learning and Neural NetworksBICA Labs
Lecture briefly overviewing state of the art of Data Science, Machine Learning and Neural Networks. Covers main Artificial Intelligence technologies, Data Science algorithms, Neural network architectures and cloud computing facilities enabling the whole stack.
In this Lunch & Learn session, Chirag Jain gives us a friendly & gentle introduction to Machine Learning & walks through High-Level Learning frameworks using Linear Classifiers.
機器學習速遊 (Quick Tour of Machine Learning)
機器學習旨在讓電腦能由資料中累積的經驗來自我進步,近年來已廣泛應用於資料探勘、計算機視覺、自然語言處理、生物特徵識別、搜尋引擎、醫學診斷、檢測信用卡欺詐、證券市場分析、DNA序列測序、語音和手寫識別、戰略遊戲和機器人等領域。它已成為資料科學的基礎學科之一,為任何資料科學家必備的工具。
這門課程將由台大資訊工程系林軒田教授利用短短的六個小時,快速地帶大家探索機器學習的基石、介紹核心的模型及一些熱門的技法,希望幫助大家有效率而紮實地了解這個領域,以妥善地使用各式機器學習的工具。此課程適合所有希望開始運用資料的資料分析者,推薦給所有有志於資料分析領域的資料科學愛好者。
Mengenal Machine/Deep Learning, Artificial Intelligence dan mengenal apa bedanya dengan Business Intelligence, apa hubungannya dengan Big Data dan Data Science/Analytics.
Machine learning with Big Data power point presentationDavid Raj Kanthi
This is an article made form the articles of IEEE published in the year 2017
The following presentation has the slides for the Title called the
Machine Learning with Big data. that following presentation which has the challenges and approaches of machine learning with big data.
The integration of the Big Data with Machine Learning has so many challenges that Big data has and what is the approach made by the machine learning mechanism for those challenges.
Shou-de Lin is currently a full professor in the CSIE department of National Taiwan University. He holds a BS in EE department from National Taiwan University, an MS-EE from the University of Michigan, and an MS in Computational Linguistics and PhD in Computer Science both from the University of Southern California. He leads the Machine Discovery and Social Network Mining Lab in NTU. Before joining NTU, he was a post-doctoral research fellow at the Los Alamos National Lab. Prof. Lin's research includes the areas of machine learning and data mining, social network analysis, and natural language processing. His international recognition includes the best paper award in IEEE Web Intelligent conference 2003, Google Research Award in 2007, Microsoft research award in 2008, merit paper award in TAAI 2010, best paper award in ASONAM 2011, US Aerospace AFOSR/AOARD research award winner for 5 years. He is the all-time winners in ACM KDD Cup, leading or co-leading the NTU team to win 5 championships. He also leads a team to win WSDM Cup 2016 Champion. He has served as the senior PC for SIGKDD and area chair for ACL. He is currently the associate editor for International Journal on Social Network Mining, Journal of Information Science and Engineering, and International Journal of Computational Linguistics and Chinese Language Processing. He receives the Young Scholars' Creativity Award from Foundation for the Advancement of Outstanding Scholarship and Ta-You Wu Memorial Award.
Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. Machine learning is so pervasive today that you probably use it dozens of times a day without knowing it.
We will review some modern machine learning applications, understand variety of machine learning problem definitions, go through particular approaches of solving machine learning tasks.
This year 2015 Amazon and Microsoft introduced services to perform machine learning tasks in cloud. Microsoft Azure Machine Learning offers a streamlined experience for all data scientist skill levels, from setting up with only a web browser, to using drag and drop gestures and simple data flow graphs to set up experiments.
We will briefly review Azure ML Studio features and run machine learning experiment.
The AML group carries out both theoretical and experimental work on developing and applying new machine learning techniques for solving various application problems.
Artificial Intelligence for Automating Data AnalysisManuel Martín
The requirements for analysing big volumes of data have increased over the last few decades. The process of selecting, cleaning, modelling and interpreting data is called the KDD process. The decision of how to approach each step in this process has often been made manually by experts. However, experts cannot be aware of all methods, nor is it feasible to try all of them. Researchers have proposed different approaches for automating, or at least advising, the stages of the KDD process. This talk will outline the different types of Intelligent Discovery Assistants as described in the work of Serban et al. “A survey of intelligent assistants for data analysis” and point out some future directions.
Data Science, Machine Learning and Neural NetworksBICA Labs
Lecture briefly overviewing state of the art of Data Science, Machine Learning and Neural Networks. Covers main Artificial Intelligence technologies, Data Science algorithms, Neural network architectures and cloud computing facilities enabling the whole stack.
In this Lunch & Learn session, Chirag Jain gives us a friendly & gentle introduction to Machine Learning & walks through High-Level Learning frameworks using Linear Classifiers.
I am shubham sharma graduated from Acropolis Institute of technology in Computer Science and Engineering. I have spent around 2 years in field of Machine learning. I am currently working as Data Scientist in Reliance industries private limited Mumbai. Mainly focused on problems related to data handing, data analysis, modeling, forecasting, statistics and machine learning, Deep learning, Computer Vision, Natural language processing etc. Area of interests are Data Analytics, Machine Learning, Machine learning, Time Series Forecasting, web information retrieval, algorithms, Data structures, design patterns, OOAD.
Python for Data Science: A Comprehensive Guidepriyanka rajput
Python’s popularity in data science is undeniable, to sum up. It is the best option for data analysts and scientists because of its simplicity, extensive library environment, and community support. The essential Python tools and best practices have been highlighted in this thorough book, enabling data aficionados to succeed in this fast-paced industry.
To succeed as a data scientist, you should follow a structured path known as the “Data Science Roadmap.” This path outlines foundational knowledge in math and programming. Data manipulation and visualization, exploratory data analysis. Machine learning, deep learning, and advanced topics such as natural language processing and time series analysis. Following this roadmap can help you acquire the skills and knowledge needed to excel in this rapidly growing field.
Becoming a successful data scientist requires a unique combination of technical skills, business acumen, and critical thinking ability. To achieve your career goals in this field, you need a structured plan or a data science roadmap that outlines the skills, tools, and knowledge required to succeed. In this blog, we’ll take a closer look at what a data science roadmap is, why it’s important, and how to create one that works for you.
At its core, It is a structured plan that outlines the skills, tools, and knowledge required to become a successful data scientist. It serves as a guidepost to help individuals navigate the complex landscape of data science and provides a clear path towards achieving their career objectives.
Python is the choice llanguage for data analysis,
The aim of this slide is to provide a comprehensive learning path to people new to python for data analysis. This path provides a comprehensive overview of the steps you need to learn to use Python for data analysis.
A Maturing Role of Workflows in the Presence of Heterogenous Computing Archit...Ilkay Altintas, Ph.D.
cientific workflows are used by many scientific communities to capture, automate and standardize computational and data practices in science. Workflow-based automation is often achieved through a craft that combines people, process, computational and Big Data platforms, application-specific purpose and programmability, leading to provenance-aware archival and publications of the results. This talk summarizes varying and changing requirements for distributed workflows influenced by Big Data and heterogeneous computing architectures and present a methodology for workflow-driven science based on these maturing requirements.
Automatic Classification of Springer Nature Proceedings with Smart Topic MinerFrancesco Osborne
The process of classifying scholarly outputs is crucial to ensure timely access to knowledge. However, this process is typically carried out manually by expert editors, leading to high costs and slow throughput. In this paper we present Smart Topic Miner (STM), a novel solution which uses semantic web technologies to classify scholarly publications on the basis of a very large automatically generated ontology of research areas. STM was developed to support the Springer Nature Computer Science editorial team in classifying proceedings in the LNCS family. It analyses in real time a set of publications provided by an editor and produces a structured set of topics and a number of Springer Nature classification tags, which best characterise the given input. In this paper we present the architecture of the system and report on an evaluation study conducted with a team of Springer Nature editors. The results of the evaluation, which showed that STM classifies publications with a high degree of accuracy, are very encouraging and as a result we are currently discussing the required next steps to ensure large-scale deployment within the company.
The Evolution of Science Education PraxiLabs’ Vision- Presentation (2).pdfmediapraxi
The rise of virtual labs has been a key tool in universities and schools, enhancing active learning and student engagement.
💥 Let’s dive into the future of science and shed light on PraxiLabs’ crucial role in transforming this field!
Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...University of Maribor
Slides from talk:
Aleš Zamuda: Remote Sensing and Computational, Evolutionary, Supercomputing, and Intelligent Systems.
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Inter-Society Networking Panel GRSS/MTT-S/CIS Panel Session: Promoting Connection and Cooperation
https://www.etran.rs/2024/en/home-english/
Nucleophilic Addition of carbonyl compounds.pptxSSR02
Nucleophilic addition is the most important reaction of carbonyls. Not just aldehydes and ketones, but also carboxylic acid derivatives in general.
Carbonyls undergo addition reactions with a large range of nucleophiles.
Comparing the relative basicity of the nucleophile and the product is extremely helpful in determining how reversible the addition reaction is. Reactions with Grignards and hydrides are irreversible. Reactions with weak bases like halides and carboxylates generally don’t happen.
Electronic effects (inductive effects, electron donation) have a large impact on reactivity.
Large groups adjacent to the carbonyl will slow the rate of reaction.
Neutral nucleophiles can also add to carbonyls, although their additions are generally slower and more reversible. Acid catalysis is sometimes employed to increase the rate of addition.
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Ana Luísa Pinho
Functional Magnetic Resonance Imaging (fMRI) provides means to characterize brain activations in response to behavior. However, cognitive neuroscience has been limited to group-level effects referring to the performance of specific tasks. To obtain the functional profile of elementary cognitive mechanisms, the combination of brain responses to many tasks is required. Yet, to date, both structural atlases and parcellation-based activations do not fully account for cognitive function and still present several limitations. Further, they do not adapt overall to individual characteristics. In this talk, I will give an account of deep-behavioral phenotyping strategies, namely data-driven methods in large task-fMRI datasets, to optimize functional brain-data collection and improve inference of effects-of-interest related to mental processes. Key to this approach is the employment of fast multi-functional paradigms rich on features that can be well parametrized and, consequently, facilitate the creation of psycho-physiological constructs to be modelled with imaging data. Particular emphasis will be given to music stimuli when studying high-order cognitive mechanisms, due to their ecological nature and quality to enable complex behavior compounded by discrete entities. I will also discuss how deep-behavioral phenotyping and individualized models applied to neuroimaging data can better account for the subject-specific organization of domain-general cognitive systems in the human brain. Finally, the accumulation of functional brain signatures brings the possibility to clarify relationships among tasks and create a univocal link between brain systems and mental functions through: (1) the development of ontologies proposing an organization of cognitive processes; and (2) brain-network taxonomies describing functional specialization. To this end, tools to improve commensurability in cognitive science are necessary, such as public repositories, ontology-based platforms and automated meta-analysis tools. I will thus discuss some brain-atlasing resources currently under development, and their applicability in cognitive as well as clinical neuroscience.
ANAMOLOUS SECONDARY GROWTH IN DICOT ROOTS.pptxRASHMI M G
Abnormal or anomalous secondary growth in plants. It defines secondary growth as an increase in plant girth due to vascular cambium or cork cambium. Anomalous secondary growth does not follow the normal pattern of a single vascular cambium producing xylem internally and phloem externally.
DERIVATION OF MODIFIED BERNOULLI EQUATION WITH VISCOUS EFFECTS AND TERMINAL V...Wasswaderrick3
In this book, we use conservation of energy techniques on a fluid element to derive the Modified Bernoulli equation of flow with viscous or friction effects. We derive the general equation of flow/ velocity and then from this we derive the Pouiselle flow equation, the transition flow equation and the turbulent flow equation. In the situations where there are no viscous effects , the equation reduces to the Bernoulli equation. From experimental results, we are able to include other terms in the Bernoulli equation. We also look at cases where pressure gradients exist. We use the Modified Bernoulli equation to derive equations of flow rate for pipes of different cross sectional areas connected together. We also extend our techniques of energy conservation to a sphere falling in a viscous medium under the effect of gravity. We demonstrate Stokes equation of terminal velocity and turbulent flow equation. We look at a way of calculating the time taken for a body to fall in a viscous medium. We also look at the general equation of terminal velocity.
Toxic effects of heavy metals : Lead and Arsenicsanjana502982
Heavy metals are naturally occuring metallic chemical elements that have relatively high density, and are toxic at even low concentrations. All toxic metals are termed as heavy metals irrespective of their atomic mass and density, eg. arsenic, lead, mercury, cadmium, thallium, chromium, etc.
Professional air quality monitoring systems provide immediate, on-site data for analysis, compliance, and decision-making.
Monitor common gases, weather parameters, particulates.
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...Sérgio Sacani
Since volcanic activity was first discovered on Io from Voyager images in 1979, changes
on Io’s surface have been monitored from both spacecraft and ground-based telescopes.
Here, we present the highest spatial resolution images of Io ever obtained from a groundbased telescope. These images, acquired by the SHARK-VIS instrument on the Large
Binocular Telescope, show evidence of a major resurfacing event on Io’s trailing hemisphere. When compared to the most recent spacecraft images, the SHARK-VIS images
show that a plume deposit from a powerful eruption at Pillan Patera has covered part
of the long-lived Pele plume deposit. Although this type of resurfacing event may be common on Io, few have been detected due to the rarity of spacecraft visits and the previously low spatial resolution available from Earth-based telescopes. The SHARK-VIS instrument ushers in a new era of high resolution imaging of Io’s surface using adaptive
optics at visible wavelengths.
hematic appreciation test is a psychological assessment tool used to measure an individual's appreciation and understanding of specific themes or topics. This test helps to evaluate an individual's ability to connect different ideas and concepts within a given theme, as well as their overall comprehension and interpretation skills. The results of the test can provide valuable insights into an individual's cognitive abilities, creativity, and critical thinking skills
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
The ability to recreate computational results with minimal effort and actionable metrics provides a solid foundation for scientific research and software development. When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress. However, we have yet to fully achieve this; there are still many sociotechnical frictions.
Inspired by David Donoho's vision, this talk aims to revisit the three crucial pillars of frictionless reproducibility (data sharing, code sharing, and competitive challenges) with the perspective of deep software variability.
Our observation is that multiple layers — hardware, operating systems, third-party libraries, software versions, input data, compile-time options, and parameters — are subject to variability that exacerbates frictions but is also essential for achieving robust, generalizable results and fostering innovation. I will first review the literature, providing evidence of how the complex variability interactions across these layers affect qualitative and quantitative software properties, thereby complicating the reproduction and replication of scientific studies in various fields.
I will then present some software engineering and AI techniques that can support the strategic exploration of variability spaces. These include the use of abstractions and models (e.g., feature models), sampling strategies (e.g., uniform, random), cost-effective measurements (e.g., incremental build of software configurations), and dimensionality reduction methods (e.g., transfer learning, feature selection, software debloating).
I will finally argue that deep variability is both the problem and solution of frictionless reproducibility, calling the software science community to develop new methods and tools to manage variability and foster reproducibility in software systems.
Exposé invité Journées Nationales du GDR GPL 2024
Deep Software Variability and Frictionless Reproducibility
2015 03-28-eb-final
1. How to train the next generation for Big Data
Projects: building a curriculum
Christopher G. Wilson, Ph.D.
Associate Professor Physiology and Pediatrics
Center for Perinatal Biology
Experimental Biology, Mar 28th, 2015
2. Outline
• Assessing the need for a “Big Data” Analytics course
• Structure and grading of the course
• Overview of the curriculum
• Advantages to Python/IPython
• Examples/use cases
• Coalition institutions and participating faculty
3. Is a Big Data analytics course necessary?
• “Back in the day, when *I* was a graduate student…”
• First year Physics lab as a training ground…
• Contemporary students live in a digital world…
• Office suites are NOT suited to large-scale data analytics!
7. Why use Free/Open-Source Software?
• In this era of shrinking science funding, free software makes
more economic sense.
• Bugs/security issues are fixed FASTER than proprietary
software.
• With access to the source code, we can customize the
software to fit OUR needs.
• Reproducibility of analyses and algorithms is easier when all
code is free, can be shared, and examined/dissected.
• Free/Open-source software tends to be more reliable and
stable.
• See Eric Raymond’s The Cathedral and the Bazaar for a
more comprehensive explanation.
8. Using a “flipped” classroom
• On-line material or reading is provided to the student either
before or during the class meeting time
• The instructor provides a short summary/overview lecture
(~20 min)
• The remaining class time is spent working on the subject
matter as individuals and groups—with the instructor and TA
present
• More effective for learning “hands on” skills like
programming, bioinformatics, web design, etc.
9. Why use a flipped classroom
model instead of lecturing for 50
minutes and assigning
homework?
10. The data analytics team
• Project manager—responsible for setting clear project objectives and deliverables.
The project manager should be someone with more experience in data analysis
and a more comprehensive background than the other team members.
• Statistician—should have a strong mathematics/statistics background and will be
responsible for reporting and developing the statistics workflow for the project.
• Visualization specialist—responsible for the design/development of data
visualization (figures/animation) for the project.
• Database specialist—develops ontology/meta-tags to represent the data and
incorporate this information in the team's chosen database schema.
• Content Expert—has the strongest background in the focus area of the project
(Physiologist, systems biologist, molecular biologist, biochemist, clinician, etc.) and
is responsible for providing background material relevant to the project's focus.
• Web developer/integrator—responsible for web-content related to the project,
including the final report formatting (for web/hardcopy display).
• Data analyst—the most junior member of the team will take on general
responsibilities to assist the other team members. This is a learning opportunity for
a team member who is new to data analysis and needs time to develop the skills
necessary to fully participate in the workflow.
13. Grading
• Pass/No Pass
• Weekly quizzes (concepts from short lectures, on-line resources, simple
code fragments/pseudo-code, etc.)
• Projects
• One individual project (basics of using IPython, simple statistics
computed via interaction with R—or using Pandas—and simple
visualization of a dataset).
• Two short projects (small group, designed to develop team-based
distribution of workload, team roles assigned by instructor).
• Larger scale project using a Big Data dataset (students will “self-
organize” their team roles). This project is envisioned as the final
exam for the class and each team will present their results and
project summary to the class.
• Final projects will be posted on the class website along with IPython
notebooks and supporting materials used for the project.
14. Syllabus Overview (10 week course)
Foundations 1: Using text editors, using the IPython notebook for data exploration, using
version control software (git), using the class wiki.
Foundations 2: Using IPython/NumPy/SciPy, importing and manipulating data with Pandas,
data visualization in IPython.
Analysis Methods: Basic signal theory overview, time-series data, plotting (lines, histograms,
bars, etc.) dynamical systems analyses of data variability, information theory measures
(entropy) of complexity, frequency domain/spectral measures (FFT, time-varying spectrum),
wavelets.
Handling Sequence data: Using R/Bioconductor, differences between mRNA-Seq, gene-
array, proteomics, and deep-sequencing data, visualizing data from gene/RNA arrays.
Data set storage and retrieval: Basics of relational databases, SQL vs. NOSQL, cloud
storage/NAS/computing clusters, interfacing with Hadoop/MapReduce, metadata and ontology
for biomedical/patient data (XML), using secure databases (REDCap).
Data integrity and security: The Health Insurance Portability and Accountability Act (HIPAA)
and what it means for data management, de-identifying patient data (handling PHI), data
security best practices, making data available to the public—implications for data transparency
and large-scale data mining.
15. Why Python?
• Python is an easy-to-learn, complete programming language
that has rapidly become an important scientific programming
and data analysis environment with usage across multiple
disciplines.
• Python was originally developed with a philosophy of “easy to
read” code incorporating object-oriented, imperative, and
functional programming styles.
• Python allows the incorporation of specialized modules based
upon low-level code (C/C++) so it can run very fast.
• Python has modules developed specifically for scientific
computing and signal processing (NumPy/SciPy).
• Python has well-documented import/export hooks into
databases (both SQL and NOSQL) that are key to working with
Big Data.
16. Why IPython?
• IPython is an interactive data exploration and visualization shell
that supports the inclusion of code, inline text, mathematical
expressions, 2D/3D plotting, multimedia, and dynamic widgets.
• IPython is a suite of tools designed to cover scientific workflow
from interactive data transformation and analysis to publication.
• The IPython notebook uses a web browser as its display “front
end” and provides a rich interactive environment similar that
seen in Mathematica.
• IPython notebooks makes it possible to save analysis
procedures and output—providing reproducible, curatable data
analysis, and an easy way to share algorithms/methods.
• IPython supports parallel coding and distributed data analysis to
take advantage of cloud/high-performance clusters.
19. Line plots with error bars
import numpy as np
import matplotlib.pyplot as plt
# example data
x = np.arange(0.1, 4, 0.5)
y = np.exp(-x)
plt.errorbar(x, y, xerr=0.2, yerr=0.4)
plt.show()
20. Heatmaps
import numpy as np
import numpy.random
import matplotlib.pyplot as plt
# Generate some test data
x = np.random.randn(8873)
y = np.random.randn(8873)
heatmap, xedges, yedges = np.histogram2d(x, y, bins=50)
extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]
plt.clf()
plt.imshow(heatmap, extent=extent)
plt.show()
21. Scatterplots
import numpy as np
import matplotlib.pyplot as plt
N = 50
x = np.random.rand(N)
y = np.random.rand(N)
colors = np.random.rand(N)
area = np.pi * (15 * np.random.rand(N))**2 #
0 to 15 point radiuses
plt.scatter(x, y, s=area, c=colors,
alpha=0.5)
plt.show()
22. 3D contour map
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm
fig = plt.figure()
ax = fig.gca(projection='3d')
X, Y, Z = axes3d.get_test_data(0.05)
ax.plot_surface(X, Y, Z, rstride=8,
cstride=8, alpha=0.3)
cset = ax.contour(X, Y, Z, zdir='z',
offset=-100, cmap=cm.coolwarm)
cset = ax.contour(X, Y, Z, zdir='x',
offset=-40, cmap=cm.coolwarm)
cset = ax.contour(X, Y, Z, zdir='y',
offset=40, cmap=cm.coolwarm)
ax.set_xlabel('X')
ax.set_xlim(-40, 40)
ax.set_ylabel('Y')
ax.set_ylim(-40, 40)
ax.set_zlabel('Z')
ax.set_zlim(-100, 100)
plt.show()
25. Summary
• Free/Libre Open-Source software provides a viable “tool
stack” for Big Data analytics.
• Python provides a robust, easy-to-use foundation for data
analytics.
• IPython provides an easy to use interactive front-end for data
transformation, analysis, visualization, presentation, and
distribution.
• Team-based science depends upon developing a wide range
of data analytics skills.
• We have developed a coalition of institutions to serve
students who wish to be become data scientists.
27. The coding Queen and her Court…
Abby Dobyns
Princesses of Python
Rhaya Johnson
Regie Felix and Adaeze Anyanwu
And a Princeling….
Jamie Tillett
28. Acknowledgements
Loma Linda
• Traci Marin
• Charles Wang
• Wilson Aruni
• Valery Filippov
UC Riverside
• Thomas Girke
(Bioinformatics)
My laboratory’s git repository:
La Sierra University
• Marvin Payne
CSU San Bernardino
• Art Concepcion
(Bioinformatics)
UC Irvine
• Alex Nicolau
(Comp Sci/Bioinf)
https://github.com/drcgw/bass
29. Further reading
• Doing Data Science by Cathy O’Neil and Rachel Schutt
• Data Analysis with Open-Source Tools by Philipp Janert
• The Art of R Programming by Norman Matloff
• R for Everyone by Jared P. Lander
• Python for Data Analysis by Wes McKinney
• Think Python by Allen B. Downey
• Think Stats by Allen B. Downey
• Think Complexity by Allen B. Downey
• Every one of Edward Tufte’s books (The Visual Display
of Quantitative Information, Visual Explanations,
Envisioning Information, Beautiful Evidence)