Playing on the edge


Published on

Playing in the huge arena that is 'Java' requires a pragmatic approach and willingness to work at the edges:
* Languages - Java is the core, but there is value in moving out when appropriate (JRuby, SQL, JSON, etc.)
* Frameworks - Why code one, when you can find one: Eclipse RCP
* Developers - local and remote, loose and Agile, and open source

We find efficiency gains from the use of open source, distributed development, agile techniques and language integration.

Published in: Technology
1 Comment
  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Two and a half years ago I was on safari in South Africa and I took this photograph. This is my nephew standing on the edge of a 100m high cliff Why is he doing that? The view is great! You can see families of giraffe in the valley. It is exciting, kids are attracted to excitement. My talk is about finding excitement in development by playing on the edge. Taking risks and thereby finding a more efficient route. Unless you fall off the edge, that is! (Two types of edge – cliffs and interfaces)
  • I have been doing some really fun stuff over the last two years And so I applied to talk at Öredev because I was burning to share some of the cooler things we've done But what track to talk in, there are many elements of interest … do transitions I thought I could focus on agile development, distributed project management, or alternative languages So I asked the 'boffins' at Öredev, and those geniuses pointed out the obvious … it is all about Java ! Java is so omnipresent we easily forget that it is there.
  • I would like to make a personal observation For ten years I developed proprietary software, and each new project was more complex and as a result more difficult to meet deadlines.
  • But seriously, how do we enable users to do whatever they want every day? The best place to start is by looking at what they already use: Excel MapInfo Well, who in their right minds will consider competing with Excel? Certainly not me. I'm not mad, or am I? My vision started with three ideas: Eclipse RCP Open Source GIS Ruby scripting
  • What you see on the screen never existed. It was created by me using the gimp. This is the vision I had two years ago for what I planned to create. Not a single one of the projects I presented on the previous slide contained any of the visual features of this new vision. So, I was taking a major risk. A new vision, new technologies and only a piece of visual art to show for it. What is it really: Load, visualize and analyse data. Emphasis on visualize, because that is what was weakest before. But the second most critical difference was allowing our users to program the system.
  • So, in 2008 we took the first steps, and integrated JRuby scripting console into the GIS. This gives a domain specific language for talking to map data, using GIS terminology like map, layer and feature. The cool thing about this is the same API will work on data in local databases, as well as served across the internet from WFS and WMS web services. We even wrote our own JSON based language for communicating via REST to our own Ruby on Rails web application. But there are some problems with this: - Too slow to perform analyses over large volumns of data remotely - Too much abstraction, making it too complex - No support for non-GIS data
  • We made a key, revolutionary change. Look, the map is still there and the scripting console is still there, but what is that ferris wheel to the top right? Ported the data structures onto the graph database, Neo4j, made by Neo Technologies right here in Malmö. This is one of the very first screenshots of the prototype results of getting the network data onto neo4j.
  • The addition of neo4j lead to a revolution in thinking. Instead of data and analysis and reporting being different levels or different structures, they are all just graph structure in the database. This means you can analyse the results of previous analyses. Just keep adding structure to the database. All the views are direct maps onto the database: GIS traverses gis-next edges Tree traverses child edges Charts traverse aggregation-child edges Property tables display node properties Neo4j has added a whole new angle to the use of the word 'EDGE' :-)
  • Spatial data can extend to a much wider area than the current map view. So asking the database for all data when only viewing a small area is a big problem. Even though the map does filter the data out before the final draw, we are putting a high load on the database and on the map engine. We need a spatial index. I decided to make my own. It seemed simple, but after a day of bad java code, I stepped back and simplified the problem: 1- Do a paper analysis 2- Verify by writing a Ruby script (test the maths) 3- Go back and re-write the Java The result – a solid index with 25 times times map re-draw performance on on test case.
  • Remember that I said that writing a spreadsheet was too hard? That is true, unless you get a really lucky break and find a shortcut! I found two. The first was the realization that I did not need to write a formula parser, Ruby was already that. The second was that I did not need to write an efficient data stucture, neo4j was already that.
  • Remember that I said that writing a spreadsheet was too hard? That is true, unless you get a really lucky break and find a shortcut! I found two. The first was the realization that I did not need to write a formula parser, Ruby was already that. The second was that I did not need to write an efficient data stucture, neo4j was already that.
  • In early 2009 we decided to get started again, and this time to bet on AWE. We built a remote development team composed of outsourced developers. How can you get high quality developers using outsourcing? It is challenging, but not impossible. I decided to do it a new way: - no CV's - no interviews - wrote 10 mini-project specs on the website - evaluate people based on the code they submit - got 12 submissions, contracted 4 for further trials - took 2 months to get efficient - now 8 months later we have a highly efficient team of excellent developers
  • Playing on the edge

    1. 1. Playing on the Edge
    2. 2. Personal Story <ul><li>Edges in life </li><ul><li>South Africa to Sweden
    3. 3. Science to Software
    4. 4. Open to Closed
    5. 5. Closed to Open
    6. 6. Employee to Entrepreneur </li></ul></ul>“Success is not based on the quality of the original idea, but on the ability to learn and change and ' pivot '” - Eric Ries, 'The Lean Startup', 2 nd Nov 2009 “Freedom to Act” - Marc Lesser, Öredev 2009
    7. 7. So many ideas! Open Source Ruby DSLs Agile / Distr. Cool Applications Domain Model Rich Client GIS / Viz. Lean Startup Java
    8. 8. Efficiency Simple Complex Not Invented Here Standing on the Shoulders of Giants Compromise 1 2 3 S1 A1 4 &quot;That's part of the real power of free software; you can use it in ways that go beyond the imagination of the people who actually created the pieces.&quot; - Mark Shuttleworth, 'Ubuntu User 02' &quot;A good compromise satisfies no-one.&quot; - Jonas Hammarberg, 2005 Simple
    9. 9. AWE 2% market penetration 2% customers pay But cheaper and faster to produce! We cannot predict what users will need to do each day. We need to give them the power to do what they want. Let's open source! What! Are you mad? How would we achieve vendor lock-in?
    10. 10. AWE
    11. 11. AWE-2008
    12. 12. AWE-2009 Neo4j – graph database <ul><li>extremely high performance on deep traversals and mining of complex data,
    13. 13. rapid schema evolution for changing business requirements, and
    14. 14. simplified development through perfect match between domain model </li></ul>and database schema.
    15. 15. AWE-2009 AWE Distribution Analysis Germany AWE Distribution Analysis Germany
    16. 16. Neo4j Spatial Index
    17. 17. Neo4j Spatial Index
    18. 18. Splash =MAX(A1:A10) =max(a1..a10) =max(a1..a10) =max([a1,a2,a3,a4]) =max(a1,a2,a3,a4) Valid Ruby Range Array Varargs
    19. 19. Splash =count(a1..a10) =sum(a1..a10)
    20. 20. Splash = a1 + ' ' + a2 = “My name is #{a1} and your name is #{a2}” My name is <%= a1%> and your name is <%= a2 %> <% (0..6).each {|x| eval “b#{x+1} = '= a#{x+1}'”} %> <% Charts.create(:type => :bar, :categories => a1..a5, :values => b1..b5) %>
    21. 21. Splash-Neo4j = a1 + ' ' + a2 = sum(a1..a3)
    22. 22. AWE-2009 a new start
    23. 23. Reporting DSL
    24. 24. Our Favourite Edge
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.