Successfully reported this slideshow.
Your SlideShare is downloading. ×

Scaling up globally: 30 years of FOSS4G development. Keynote at FOSS4G-CEE 2013, Romania


Check these out next

1 of 33 Ad

More Related Content

Slideshows for you (13)

Similar to Scaling up globally: 30 years of FOSS4G development. Keynote at FOSS4G-CEE 2013, Romania (20)


Recently uploaded (20)


Scaling up globally: 30 years of FOSS4G development. Keynote at FOSS4G-CEE 2013, Romania

  1. 1. ©MarkusNeteler2013,CC-BY-SA Scaling up globally: 30 years of FOSS4G development Keynote Markus Neteler Fondazione Edmund Mach Research and Innovation Centre Department of Biodiversity and Molecular Ecology GIS and Remote Sensing Unit FOSS4G Central and Eastern Europe 2013 16th - 20th June, National Library of Romania Bucharest, Romania PostGISomics
  2. 2. ©MarkusNeteler2013,CC-BY-SA Img. courtesy: J Westervelt 2006: Early GRASS Community Views on FOSS THE 80's … the beginning
  3. 3. ©MarkusNeteler2013,CC-BY-SA THE 80's ... 1980 – LAGRID – J. Westervelt master's thesis: GIS software, developed on a mainframe computer, then ported to Cromemco Z-80. FORTRAN 1982 – FHIS (Fort Hood Information System) Vax 11/780 minicomputer, UNIX, C language. Programmer: L. Van Warren 1983 – "GIS Version 1 Reference Manual" by J. Westervelt and M. O'Shea, 29 July 1983 Included GIS programs: 1983 – PROJ4 library development started by Gerald I. Evenden arctogrip area_stats cell_stats cellmod (grid editor) coin (r.coin) combine (boolean combination) distance (r.buffer) dotmap (graphics on a dot-matrix printer) erase (d.erase) griptocell layer_info ( list (g.list) over (d.rast, but for b/w monitor) reclass (r.reclass) sho_over (display images created by over) table (stats associated with over) whats_here ( with a mask) window (g.region) Vax 11/780 minicomputer
  4. 4. ©MarkusNeteler2013,CC-BY-SA Boolean combination - gridcell and polygon Weighted overlay - gridcell and polygon (reworked later into r.mapcalc) Distance-from (now r.buffer) Isoline generation (grid to poly - now r.contour) Coincident tabulation (now r.coin) Mathematical combination (reworked later into r.mapcalc) THE 80's ... August, 1983 – U. S. Army Corps of Engineers' Construction Engineering Research Laboratory (USA/CERL) in Champaign, Illinois Purchase of first Sun-150 computers (2) – specs: ● $8.900 16 bit, 10Mhz, 256Kb memory, 10Mbps Ethernet, 17" b/w monitor (100*800) ● $3.400 1Mbyte extra memory ● $2.000 3/4M Fast Sun Memory ● $1.590 Barko color monitor ● $6.540 80Mbyte hard disk ● $1.900 Disk controller board ● $5.500 Dot Matrix printer ● $1.500 Vanilla UNIX software -or- ● $2.000 4.2BSD plus library of graphics software 1983/1984 New analysis capabilities added in GRASS GIS: Source: … 33k USD
  5. 5. ©MarkusNeteler2013,CC-BY-SA THE 80's ... 1984 – GRASS running on SUN-1 and Masscomp 1985 – GRASS 1.0, GRASSnet established (pre-mailing list) 15 March 1985: first commercial Internet domain name registered 1985 – Richard Stallman publishes the GNU Manifesto 1987 – GRASS 2.0; first issue of GRASSClippings Newsletter GRASS video narrated by William Shatner (Captain Kirk of Star Trek) 1988 – GRASS 3.0; Army R&D Achievement Award (Webster, Goran, Shapiro, Westervelt)
  6. 6. ©MarkusNeteler2013,CC-BY-SA THE 80's ... 1989 – GRASS 3.1 First release available on Internet ( But how was software developed? Locally! While Revision Control System (RCS) was available a sometimes used, yet no server based system like “CVS“ (CVS 1.0 in 1990) (today: SVN, git, …) TCP/IP goes global (1989–2010) However: By 1992, still less than 15,000 .com domains registered... See also: ● ● ●
  7. 7. ©MarkusNeteler2013,CC-BY-SA THE 90's … visualization and analysis Source:
  8. 8. ©MarkusNeteler2013,CC-BY-SA THE 90's ... 1990 – GRASS 4.0 Letter-dot format adopted for commands (e.g. d.rast and g.region) 1991 – First mailing lists
  9. 9. ©MarkusNeteler2013,CC-BY-SA THE 90's ... 1992 – New graphics technology became available: openGL Software distribution: “Moon“ FTP server of CERL (note: still 2 years to wait for WWW!) SG3D SGI Indigo
  10. 10. ©MarkusNeteler2013,CC-BY-SA THE 90's ... ORGANIZATIONS 1992 GRASS Inter-Agency Coordinating Committee → Open GRASS Foundation (OGF) → OpenGIS Consortium (OGC) 1994 – Tim Berners-Lee decided to constitute the World Wide Web Consortium (W3C) 1994 – Open Geospatial Consortium (OGC) founded Open GIS Open Geospatial Consortium (OGC) Consortium (OGC) GRASS Interagency Steering Commitee 1990 1992 2006 Open GRASS Foundation (OGF) 1994 2004 OSGeo
  11. 11. ©MarkusNeteler2013,CC-BY-SA THE 90's ...
  12. 12. ©MarkusNeteler2013,CC-BY-SA Source: THE 90's … Internet growth as a crucial precursor to FOSS4G development
  13. 13. ©MarkusNeteler2013,CC-BY-SA Do you remember: 30 slackware disks downloaded, but unfortunately in FTP ASCII mode (7 bit, not 8)?? THE 90's ... First Web Applications come up, finally! 1993 – Xerox PARC Map Viewer 1994 – GNU/Linux 1.0 released 1995 – First GRASS 4.1.5 port to Linux (Andreas Holz, Greifswald, Germany) 1995 – UMN MapServer project started 1995 – Well, and also the first spam email in the GRASS mailing list
  14. 14. ©MarkusNeteler2013,CC-BY-SA THE 90's ... An interactive GIS via WWW, perhaps the first “Web Processing“ service... 1994-1998 – Susan M. Huse's GRASSLinks (PhD thesis at Berkeley)
  15. 15. ©MarkusNeteler2013,CC-BY-SA THE 90's ... GRASSLinks
  16. 16. ©MarkusNeteler2013,CC-BY-SA THE 90's ... Internet, its tools and geeks spreading! 1996 – GeoTools project started 1998 – deegree (originally JaGo) development started with an OGC Simple Features implementation 1998 – GDAL/OGR development started 1998 – First European GRASS GIS server at ILN, Uni Hannover, Germany 1999 – GRASS GIS source code moved from manual management to CVS, precisely on 29 Dec. 1999 :-)
  17. 17. ©MarkusNeteler2013,CC-BY-SA Since 2000... May the FOSS be with you
  18. 18. ©MarkusNeteler2013,CC-BY-SA 2001 – OSSIM initial revision in CVS PostGIS started GeoNetwork opensource started GeoServer started 2002 – Quantum GIS initial revision in CVS GEOS initial revision in CVS 2003 – Community MapBuilder started. End of life in 2008 Release of Mapbender under the GNU GPL license gvSIG was started 2004 – uDig was started 2005 – MapGuide Open Source 2006 – Mapbender gets first bits in CVS OpenLayers Started 2007 – GeoMoose was open sourced (started 2005) 2009 – rasdaman was open sourced (started 1995) Since 2000... growing communities Thanks to CVS, SVN and git 2006: OSGeo!
  19. 19. ©MarkusNeteler2013,CC-BY-SA Since 2000...growing communities Chulalongkorn University, Bangkok 2004: FOSS4G is born! Jim Westervelt Special thanks to Venkatesh Raghavan, Osaca City University
  20. 20. ©MarkusNeteler2013,CC-BY-SA Since 2000...growing communities QGIS Hackfest Pisa 2010 Mapbender code sprint 2007 FOSS4G Conferences: Lausanne, Denver, Victoria, Cape Town, Sydney, Barcelona, Denver, ...
  21. 21. ©MarkusNeteler2013,CC-BY-SA Community sprints, even more...
  22. 22. ©MarkusNeteler2013,CC-BY-SA Communication Flow of bug reporting and solution: 2. Developer detects bug 60% 20% 20% (Percentages are estimated) Other developers New release 3. New feature SVN code Repository Release cycle 1. User sends bug report Bugtracker Mailing list20% 80%
  23. 23. ©MarkusNeteler2013,CC-BY-SA Organization of distributed source code management: “Code habitats” Two main types of developers may be identified: ● generalist ● specialist (the majority) It appears that many developer assign themselves to “code habitats”, i.e. their area of expertise (e.g., in GRASS GIS a selection of libraries or topics which they maintain) These “code habitats” remain often stable over years There are also partially abandoned code areas (~ 10% of the code?) which are functional but aren't really getting improved A few “garbage collectors” (generalists) fix lots of odds 'n ends
  24. 24. ©MarkusNeteler2013,CC-BY-SA Organization of distributed source code management: “Code habitats” GRASS GIS 6.4 development visualization from 1999 to 2011 with
  25. 25. ©MarkusNeteler2013,CC-BY-SA Code vetting Legal aspects License compliance (e.g., GPL) No code copying from books like “Numerical Receipes in C” Ensure that 3rd party contributions are clean Employers must agree that work time is spent Full transparency and peer review help to minimize the risk. Apache or OSGeo Foundation Incubation phase Graduation
  26. 26. ©MarkusNeteler2013,CC-BY-SA Since 2010... Source: Blog of Arnulf Christl
  27. 27. ©MarkusNeteler2013,CC-BY-SA Emergency support Haiti, January 2010: 7.0 magnitude earthquake Available Geodata: almost absent See also: Haiti Dec 2009 Open Data:
  28. 28. ©MarkusNeteler2013,CC-BY-SA A few days later ..: international crowd mapping See also: Haiti Jan 2010 Open Data:
  29. 29. ©MarkusNeteler2013,CC-BY-SA New: Crowdfunding of development
  30. 30. ©MarkusNeteler2013,CC-BY-SA New cool stuff: plotting GIS data From Radar to voxels to 3D plots
  31. 31. ©MarkusNeteler2013,CC-BY-SA New cool stuff: massive data processing ● Since 2005 GRASS GIS is running natively on 64bit CPUs ● GRASS GIS 7 also offers Large File Support on 32bit Windows ● Installed on Grids and TOP500 supercomputers (AKKA Umeå, ENEA Frascati, Aurel Bratislava, …) ● Runs on Linux, AIX, Solaris, freeBSD, netBSD, ... ● Various ways of parallelization
  32. 32. ©MarkusNeteler2013,CC-BY-SA Massive data processing: also for you “Since the home computer only has 8 GB of RAM...“ “8.5 billion points...“
  33. 33. ©MarkusNeteler2013,CC-BY-SA Markus Neteler Fondazione E. Mach (FEM) Centro Ricerca e Innovazione GIS and Remote Sensing Unit 38010 S. Michele all'Adige (Trento), Italy Concluding remark: THANKS TO ALL CONTRIBUTORS! And more to come... PostGISomics

Editor's Notes

  • Good afternoon and welcome to my talk with the bold title „ Scaling up globally: 30 years of FOSS4G development“ I am very happy to be here, and would like to thank the organizers for the kind invitation. In my presentation I'll briefly review 3 decades of Open Source GIS development, compressed to 30 minutes! Hence, omissions are unavoidable. BTW: The term FOSS4G was coined only in 2004 but I use it in the title as an hommage to the community and its ideals. So, let's now look back 30 years – back to the 80 th ...
  • Well... what's this? Do you recognise it?
  • Hence we start in 1980: A student named Jim Westervelt was completing his Master thesis. He wrote the LAGRID software which became the core of the later GRASS GIS software. In the early 1980s the U.S. Army Corps of Engineers' Construction Engineering Research Laboratory (USA/CERL) in Champaign, Illinois, began to explore the possibilities of using GIS to conduct environmental research, assessments, monitoring and management of lands under the stewardship of the U.S. Department of Defense. Bill Goran of CERL conducted a survey and discovered that no existing GIS satisfied their needs. Hence CERL hired several programmers, and began to develop a hybrid raster-vector GIS for the VAX UNIX minicomputers (photo) . Lynn Van Warren was the founding software architect of GRASS. Very important: PROJ (today PROJ.4) – the Cartographic Projections Library essential for many projects, was started in 1983 by Gerald Evenden
  • CERL then upgraded their machines to the latest technology, see what you could get for 33,000 USD.... e.g. including a 80MB disks for 6500 USD. Around 1984 new analytical functionality was available, some command names at least the GRASS users in this room will be familiar with :-) Please note that it took still another almost 10 years to have the World Wide Web being invented...
  • In the corner photo you see what data exchange meant in those days. Landsat 5 was launched in 1984, GRASS 1.0 was released a year later. Of huge importance for the Free Software development was the publication of the GNU manifesto by Richard Stallman (the “four freedoms” were defined). In 1987 a GRASS video was produced with William Shatner, known as Captain Kirk of Star Trek, being the speaker. And 5.25 inch floppies were becoming fashion in these days for tiny data sets!
  • And then finally an initial internet release of Open Source GIS software! Still there was almost nobody online in those days, at least not able to download 100 MB of source code... The big spread of internet happened due to the acceptance of TCP/IP as protocol. Information exchange happened via FTP. Today all geeks are sharing code via SVN or git or other online code sharing repositories. But back in the 80th there was no online system like this. Either manual management or at most RCS which is file based.
  • So, the first decade of Open Source GIS concluded with the online publication of software (yet with a tiny user group being also online). Consider to think back when you went online the first time! Anyway, let us hop into the next decade: the 90th! A lot happened here, new hardware, new software, and way more data! And especially the advent of the internet society was a social revolution.
  • In 1990 GRASS 4.0 came out, a key release which was the foundation for the GRASS' architecture. On the “social” side, two mailing lists were started, the grass-user and the grass-dev lists. Those are probably the longest, active mailing lists of the internet with more than 2 decades of archive! Still no WWW, but information exchange happened FTP and Gopher. The photo shows core developers: - Michael Shapiro - Jim Westervelt - Bill Goran (the coordinator)
  • With the availability of new graphics CPUs a completely new way of rendering became possible. OpenGL was designed with hardware acceleration. Like this, fly-thoughs as seen in the left figure as well as 3D and even 4D animations came to life (here, water contamination modeled as voxels which are changing in time). I remember from working as a student at the Hannover CeBIT computer fair, to enjoy the demos of the SGI machines while carrying around newspapers across the fair area.
  • At the level of organizations many changes occurred: the GIS world became more professional and organized. In 1992 the GRASS Interagency Coordinating Commitee was founded, it was turned into the Open GRASS Foundation which then became the OpenGIS Consortium. The timeline shows the evolution. While Tim Berners-Lee constituted the World Wide Web Consortium, the today's OGC was established.
  • The rationale behind the establishing of the OGC you find described in an article written by Kenn Gardels. It got published in the “GRASSCLIPPINGS” issue of Fall 1993. Here the fact that interoperability is a core issue is pointed out. BTW: If you are interested, the scan is online, all links in my presentation which I'll upload later, are pointing to the respective documents.
  • This map shows the internet accessibility in the year 1998 – people being online in percent. As you see, in those days internet was yet restricted to a few countries in terms of accessibility. And, obviously: without internet no distributed source code development nor easy geodata exchange.
  • In the year 1993 the first Web application came up, the Xerox PARC map viewer [refer to the previous talk of Maria Brovelli] In 1994 the 1.0 version of Linux was released and shortly after GRASS GIS ported to it. This was still a tough job since the compiler tools where not as mature as nowadays. The year 1995 was a key year for the WebGIS community with the start of the Mapserver project. And... wow, a first spam email reached the ML
  • What happened in the WebGIS sector: Probably less known is the this first “Web Processing“ style service which was published back in 1994. It was an interactive GIS controllable through the WWW, developed by Susan Huse based on her PhD thesis. This system was able to ...
  • … perform true analysis in a user controllable way using CGI scripts. Of course it is not comparable with anything you can do today but remember the internet accessibility map of 1998 which I have shown earlier. We are still in the “prototype” years and draft WPS specifications were published only almost a decade later.
  • With the advent of the collaborative internet based development tools , a series of new projects like GeoTools, deegree, GDAL/OGR were started. In 1998 I started the European GRASS GIS server, completely non-authorized of course (except for the ILN landscape architecture institute at the university)... it turned out to be a good idea to take control. BTW: Anyone here remembering the year 2000 bug? It stimulated us to move the GRASS code from manual code management via email to something more sophisticated called CVS. This was a server based code repository and checkins became independent of a single person who had to read emails and act upon.
  • Well, we happily survived the Year 2000 bug and reached the next decade... Now Communities growing together! This drawing has been done by the older daugther of Venka (Venkatesh Raghavan in Osaka) for the initial 2004 FOSS4G conference in Bangkok which I'll mention shortly
  • In just a few years a lot of new software projects were started, I don't have the time to illustrate them in details. You see that all adopted the collaborative tools for development. From the demand to get coordination among these projects the idea arose to create an umbrella foundation , OSGeo, as illustrated by Jeff McKenna earlier today.
  • For the term “FOSS4G” we have to thank Venka once more. He proposed the conference title in 2004 for the conference at Chulalongkorn University. Also Jim Westervelt came to give us an authentic lecture about the early days at this meeting.
  • The next slides will be much more familiar for you... code sprint photos Here the mapbender team in 2007 and the QGIS hackfest in Pisa in 2010.
  • For those not familiar with it: - it is a gathering of likeminded people - no need to be a developer - newcomers are there up to core developers Instead of writing 10 emails to discuss an issue, we just discuss it directly, rarely long with some beer And: The outcome of such a week is often impressive. Next chance for you: on Thursday this week!
  • After these years of community based software development, the emerge of Wikipedia also crowd mapping came up. No need to explain OpenStreetMap here but for sure the Haiti earthquake example is to be mentioned. It is illustrating an outstanding effort of people around the globe gathering virtually together and helping in an emergency situation. The map shows the geodata situation in January 2010 when a magnitude 7 earthquake occurred – almost no geodata available which were urgently needed.
  • This is the situation a few days later. Based on high resolution satellite data which were made available by the big data providers, the OSM community was able to generate in an extremely short time a detailed coverage of digital data. A similar effort was done after the Great East Japan Earthquake 2011, you will remember it.
  • The increasing demand for geospatial data in the Web browser and increasing interactivity naturally requires the adoption of new emerging technologies. An interesting new method of fundraining, instead of direct sponsoring by a few is after crowd knowledge (Wikipedia) and crowd mapping (OSM) now crowd funding where till a deadline a certain funding goal should be achieved. Example: OpenLayers 3 funding, aiming at more than 300,000 USD – and they are almost there!
  • There are also new possibilities for plotting maps: from 2D to 3D using rapid prototyping technologies which have been directly connected to Open Source GIS. Here an example of the polar ice cap of Mars (so, no need to restrict yourself to mother Earth): The processing chain was: from Radar remote sensing data to voxels to 3D plots
  • Since we like to think big, why to getting our tools on Supercomputers running... Indeed, we are already there: PROJ4, GDAL and GRASS GIS are available even on TOP500 systems. The harder part was to get it compiled on Non-Linux Supercomputers :-) For massive data processing currently job managers are supported. Work is underway for GPU based clusters which require a major source code restructuring.
  • Since not everybody has a supercomputer reachable, a lot of efforts have been done on processing massive data on even consumer hardware. Here an example from a power user who managed to process 8.5 billion lidar points on his home computer. With this touching base I would like to conclude my quick ride through 30 years of Open Source GIS development and conclude with...
  • Happy birthday, GRASS GIS And Thanks to all FOSS4G contributors all over the world. Thank you for listening