Python for Earth


Published on

Slides from Python Indonesia meetup

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Python for Earth

  1. 1. © 2014 Mohammad Fadli Python for Earth Mohammad Fadli
  2. 2. Days before python (back in college era) • Using fortran for numerical programming • Using matlab for matrix operational or data plotting • Using excel for data processing • Using surfer or matlab for plotting • Using arcmap or or for spatial data processing • Using delphi for UI • No database (all textfile matrix)
  3. 3. Create matrix do j = 1, 3 do i = 1, 3 A(i,j) = real(i)/real(j) blah.. Blah .. read text file .. write text file .. enddo enddo
  4. 4. Matlab • Matrix Operation • a = [ 1 2 3 4 5; 2 3 4 5 6; 3 4 5 6 7; 4 5 6 7 8] • a(2,5) Courtesy of Mathworks Inc.
  5. 5. Plot stuff Courtesy of Golden Software Inc.
  6. 6. Courtesy of Embarcadero Technologies, Inc.
  7. 7. Courtesy of ESRI, Inc.
  8. 8. Hydrodynamics model Courtesy of TideTech, Inc.
  9. 9. Courtesy of Kementerian Kelautan dan Perikanan
  10. 10. Courtesy of Kementerian Kelautan dan Perikanan
  11. 11. Courtesy of Kementerian Kelautan dan Perikanan
  12. 12. Courtesy of Kementerian Kelautan dan Perikanan
  13. 13. Courtesy van oord nv • Build model for pollutant trajectory
  14. 14. Grid Analysis and Display System (GrADS) Courtesy of Weather & Climate Prediction Lab. ITB
  15. 15. Do something different! • Install freeBSD and runs model on freeBSD • Install mapserver on freeBSD • Using grads for plotting • Using fortran for data processing • Using php for web (still) • Parallel computing (not yet)
  16. 16. First time using python ArcGIS ArcPy Courtesy of ESRI Inc.
  17. 17. Heboh Python (Python Fuss) :p • A clear and readable syntax. • Modules can be easily written in C, C++. • It can be used on all major platforms (Windows, Linux/Unix, Mac etc.) • Easy to learn. • And it is free. • Python this, python that
  18. 18. Nowadays • ANUGA for Hydrodynamics model • EQRM for Earthquakes model • Python for Climate Data Analysis Tools (CDAT) • GDAL library for spatial data processing • Django, Grok, Pylons, web2py (and other tons of web library) for web • Psycopg for postgre database connection • PyMPI for Clusters computing and parallelization
  19. 19. Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
  20. 20. Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
  21. 21. SciPy: a collection of useful modules Courtesy of Andrew Walker
  22. 22. • Python enables users to do more and better science. • Python is a modern, interpreted, object-oriented, open-source language used in all kinds of software engineering. • Python is now a robust integration platform for all kinds of atmospheric sciences work, from data analysis to distributed computing, and graphical user interfaces to geographical information systems. • Python has a concise but natural syntax for both arrays and nonarrays, making programs exceedingly clear and easy to read; as the saying goes, “Python is executable pseudocode.” • Include visualization, numerical libraries, interconnection with compiled and other languages, memory caching, Web services, mobile and desktop graphical user interface programming, and others.
  23. 23. Another advantages • Several packages exist that handle file formats used by atmospheric and oceanic sciences users, including netCDF, HDF4, HDF-EOS 2, and GRIB 1 and 2 (e.g., via the netCDF4- Python, UV-CDAT, and PyNIO packages). • Python can be used as a scripting language for applications such as Unidata’s Integrated Data Viewer, the NCAR Command Language (via PyNGL), GrADS (via OpenGrADS), and Esri’s ArcGIS.
  24. 24. Disadvantages • To be fair, Python has real disadvantages, including the fact that pure Python code runs much slower than compiled code, there are fewer scientific libraries compared to Fortran, and documentation and support for new science users is relatively sparse. • There are tools to overcome the speed penalty, the collection of scientific libraries is growing, and science support resources are becoming more robust; nevertheless, these are real issues. • For most Earth science applications, the strengths of Python outweigh the weaknesses.
  25. 25. What we are doing with python now? • Try to runs (and optimize) anuga on 128 core HPC • Optimize and automate earthquake model • Create our own spatial analysis framework using GDAL lib • Prove what John-Wei said
  26. 26. Flood Research 2013: Domain Pasar Minggu to Manggarai (Segment 1) • Criteria: – Major factor only from discharge and local rainfall – Homogenous Hydrologic characteristic 27 Manggarai Detail Domain Katulampa Courtesy of PT Reasuransi Maipark Indonesia 2013
  27. 27. Model Hidrodinamika 2D AnuGA ANUGA adalah model hidrodinamika Free and Open Source (FOSS) yang dikembangkan oleh Australian National University dan Geoscience Australian; metode pemecahan persamaan menggunakan finite volume; dapat diunduh dari; Salah satu kelebihan ANUGA : dapat meng-input-kan hujan pada domain Kenapa Anuga: 1. Public Domain 2. Open Source 3. Parallel 4. Finite Volume  Shock Capturing Capability  Hydraulic Jump
  28. 28. Anuga for Jakarta Flood 2013 2013 flood simulation Bukit Duri dan Kampung Melayu Kalibata Courtesy of PT Reasuransi Maipark Indonesia 2014
  29. 29. Hasil Simulasi Banjir Jakarta Simulasi Banjir Jakarta 2013 Hadi, dkk., 2013 Input Curah Hujan Spasial, Robert, 2013 Anuga dapat mengakomodasi simulasi banjir dengan input curah hujan spasial, simulasi paralel Dan input debit serta TMA Courtesy of PT Reasuransi Maipark Indonesia 2014
  30. 30. Courtesy of LAPAN
  31. 31. Courtesy of LAPAN
  32. 32. Rendaman Max. vs Data UAV LAPAN Courtesy of LAPAN, PT Reasuransi Maipark Indonesia
  33. 33. Courtesy of PT Reasuransi Maipark Indonesia
  34. 34. K.H. Abdullah Syafei Foto: Kantor Berita ANTARA (19/01/2014) Courtesy of PT Reasuransi Maipark Indonesia
  35. 35. Depo, Bukit Duri (19/01/2014) Foto: Kantor Berita ANTARA Bukit Duri
  36. 36. Bukit Duri (Depo & SMA 8) Depo SMA 8 Depo Depo Depo SMA 8 SMA 8 (16/01/2013) Foto: Kantor Berita ANTARA (19/01/2014) Foto diambil dari Heli PMI Courtesy of PT Reasuransi Maipark Indonesia
  37. 37. K.H. Abdullah Syafei, (19/01/2014), Foto: ANTARA Courtesy of PT Reasuransi Maipark Indonesia
  38. 38. Jembatan Layang Kalibata, 19/1/2014, Foto: ANTARA Courtesy of PT Reasuransi Maipark Indonesia
  39. 39. Jl. Jatinegara barat raya Depo Depo Jatinegara Bukit Duri Courtesy of PT Reasuransi Maipark Indonesia
  40. 40. Rawajati Kampung Melayu Diambil dari helikopter PMI, 16/1/2013
  41. 41. Thunder System Summary Specification Nodes 4 nodes Cores 16 cores (4 cores/node) Processors (per nodes) AMD FX (tm) – 8350, 4 Ghz Memory (per nodes) 8 GB DDR 3, 800 Mhz Scenario: Spin up scenario for 16.67 hours Domain : Full Jakarta consist of Ciliwung River (Bogor-Pluit) Even though faster simulation could be performed, we have to notify that low resolution mesh is used in this case Scenario: 15-18 January 2013 (3 days) Domain : Ciliwung River (Pasar Minggu-Manggarai) “The computation speed may increase as the water volume inside of the domain is increasing” Courtesy of WCPL-ITB
  42. 42. National Computational Infrastructure (NCI), Australia Summary Specification Nodes 3592 node Cores 57,472 cores (16 cores/node) Model Fujitsu Server Primergy Cx 250 S2 Processors 2 x intel Sandy E5-2600 (8-core, 2.6 GHz) Memory 8 GB DDR 3, 1600 MHz 2395 nodes ->32 GB 1125 nodes -> 64 GB 72 nodes -> 128 GB TOTAL : 158 TB Harddisk 500 GB 7200 rpm SATA, Cluster Raijin, NCI Domain 2 :Jakarta Approximately 10 rivers will be included in our scenario Approximately total mesh element number: 3.300.940 triangles Problem in computation time while using large number of mesh elements could be tackled by employing large core number of machine used for simulation NCI Building Australia National University Contoh :
  43. 43. Reference • • visualization-in-python/ • • • • • • pdf • •
  44. 44. grads • The Grid Analysis and Display System (GrADS) is an interactive desktop tool that is used for easy access, manipulation, and visualization of earth science data. • Support binary (stream or sequential), GRIB (version 1 and 2), NetCDF, HDF (version 4 and 5), and BUFR (for station data). • Windows & Unix support