How a Clojure pet project turned into a full-blown cloud-computing web-app

‘How a Clojure pet project turned into a full-blown cloud computing web app. Or: What are the differences between a Clojure web app and one written in Java or Common Lisp?’ Berlin, February 2, 2010 Presentation, Release 0.1 Hamburg Lispers @ Berlin Lispers Meetup

Thanks to Franz Lisp and the newthinking store here in Berlin for making this meetup possible! Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 2

Hi, I am Stefan Richter, CTO and founder of freiheit.com technologies. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 3

Why am I here? - We are working on a Clojure web-app - This app runs on Google App Engine - We use this app in our own organization - We want to provide the app as a software service - Why did we choose Clojure for such a mission critical project? - We want to share our experiences with you: - How Clojure made us more productive in developing web applications (and how much fun it is!) - What are the differences to Java and Common Lisp when we talk about web development? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 4

But first things first: What kind of app is this? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 5

I have to tell a little story first ... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 6

My company freiheit.com has been developing large internet systems for more than 10 years. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 7

We are about 70 people. Only 7 are non-hackers. So 90% are hackers. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 8

We have no project managers. The programmers ‘manage’ their projects themselves. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 9

We are very efficient. We are very well organized. How does this work? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 10

We have strong principles on how to plan and deliver software. We do this in a way which is natural to hackers. All principles are supported by internal software systems. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 11

Note: The way we work is ‘agile.’ But I would not call it that because I hate all this ‘agile’ hype. We have been working like this since 1999. We know what we are doing. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 12

So our principles have been forged not only into our culture, but also into our internal ‘project management’ systems. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 13

Our main system is ‘Captain Feature,’ which provides task-management and measures for self- and team management. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 14

The ‘Captain,’ as we fondly call him, is an intranet app, written in Java, over 8 years old. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 15

In 2008 I started a pet project to replace the old ‘Captain’ with new software written in Common Lisp. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 16

I was the only programmer on this project. But I got very far. I built a nice web-app using sbcl, hunchentoot, cl-who, cl-sql, JavaScript and CSS. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 17

Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 18

We thought: Our tools are so useful to us, why don’t we share them with other programmers? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 19

This would mean: Provide our internal systems as external services. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 20

We don‘t want to own hardware. And we don‘t want to operate our own server farm. Not even a virtual one. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 21

So we checked Google App Engine because it promised zero maintenance and a price model based on the resources you actually use. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 22

Problem: App Engine only runs Python and Java. But we wanted Lisp. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 23

Clojure is a Lisp and it runs on the JVM. If it runs on the JVM, it runs on App Engine, too! Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 24

First, we started to design and implement a ground- breaking to-do manager. The plan was to later build the new ‘Captain’on top of this to do manager infrastructure. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 25

We call this to-do manager ‘TheDeadline’. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 26

‘TheDeadline’ is an intelligent to-do manager that behaves like a real, human personal assistant. It supports you. I doesn’t annoy you. It helps you achieve your goals. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 27

I showed a screenshot of an early version of ‘TheDeadline’ in a lightning talk at the European Common Lisp Meeting 2009: Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 28

And this is how it looks today. (We release a new version every week. It is changing fast! Maybe I can give you a demo later): Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 30

Our vision? The dream of every hacker on this planet! Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 32

‘We want to eliminate all human project management work on this planet.’ Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 33

Don’t get me wrong: We want to eliminate the ‘project management work’ not the ‘project managers.’ ;) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 34

‘Instead we provide AI- based systems that help people (especially hackers) to become more productive and help them to manage themselves!’ Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 35

We use AI for Intelligence Amplification and not to replace humans. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 36

Sounds crazy?! Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 37

To say it in the words of Mike Muir of Suicidal Tendencies: ‘I’m not crazy. You’re the one that’s crazy.’ Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 38

With so many projects still failing today, we urgently need intelligent software for project management ... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 39

With an AI system doing 80% of the work that a human assistant would do for you, you can focus on the things that AI can’t do (today). And: ‘TheDeadline’ never sleeps and doesn’t need feeding. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 40

Here is why I don’t like the popular to do manager on the web: They are just list management apps with no intelligence at all. They don’t help you. They just store your to-dos. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 41

‘TheDeadline’ is currently in Closed Beta. Public Beta starts soon. (Sign-up link at the end of the presentation.) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 42

Now that you understand why we did what we did, we can start with the technical part ... :) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 43

Most people praise Clojure because of the concurrency features. Lots of stuff has been written about it. So, I won’t talk about this. (You can’t start a thread in App Engine anyway ...) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 44

So how does Clojure improve your life as a programmer, especially in web development?! Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 45

Three things that make your life much easier: 1. Structs vs. objects 2. Lisp syntax (data = code) 3. Macros (extending the language) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 46

Of course first-class/higher- order functions, everything being a sequence, immutability, the java virtual machine, etc., are all important factors. But let us take a closer look at the data structures and code generation features ... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 47

1. Structs vs. objects. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 48

I used CLOS - which is cool, especially compared to Java’s OO system. But sooner or later you get the feeling that you don’t need much OO in a functional language. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 49

You don’t want that much state anyway. You want to fold, transform and destructure your data in your app all the time. And: Not everything is an object. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 50

BTW: Putting objects into a relational database was called ‘impedance mismatch’ in the 1990’s. Today it is called ‘persistence mapping.’ But this doesn’t fix the problems. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 51

Did you ever ask yourself why not all data structures are represented in the same style? Especially when you use maps in your code, send JSON to your client and store data in a key-value store? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 52

This is what Clojure offers: Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 53

The following examples are taken from „Programming Clojure,“ Stuart Halloway, Pragmatic Programmers Series. Cool book. Buy it! :) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 54

You define a struct. You instantiate structs. Same structs share their keys. They behave like hash tables (maps). They are immutable. They are human- readable. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 55

The Clojure structs combined with multimethods are a really smart idea. It’s like in CLOS: Classes and methods are separated. But I think structs are a more readable and more flexible concept. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 56

You can even specialize a method on more than one class/type. (Which is unknown in the Java world). Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 58

Important: You don’t have to use multimethods. But sometimes they help to reduce the number of different function names in your namespace. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 59

2. Lisp syntax (data = code) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 60

Most people say they are afraid of the parenthesis. Maybe that is why 31% of the JAX mag readers think that Scala has the most potential of the JVM languages. Only 15% bet on Clojure ... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 61

Lisp syntax is easy to learn because there are almost no syntax rules. Your programs are essentially expression trees built from function calls. Code generation is an integral part of the language. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 62

This is a function call: (+ 3 2) This is data: ‘(+ 3 2) You can translate data into a function call at runtime and execute it! Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 63

So, functions generate functions. You generalize functions and parameterize them by passing other functions to them. There is no boilerplate code. Just small functional building blocks put together. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 64

3. Macros (extending the language) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 65

Macros are Lisp programs that generate other Lisp programs. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 66

Note: This is not the same as self-modifying assembler code! ;) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 67

The Lisp syntax is the precondition to make language extension very easy! (Yes, this parenthesis.) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 68

BTW: Scala has ‘no parenthesis’, which is why it is so bloated with syntactic sugar ... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 69

Why would I want to extend the language? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 70

In other languages, you define an API to implement a subsystem for a specific domain, like database access or html templating. This API is then encapsulated in a library. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 71

To use such an API, you normally build an integration layer between your application code and the API. You need this abstraction, but it also requires extra work. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 72

Instead, by extending the language with some kind of domain-specific language constructs, you don’t need an integration layer anymore. This layer still exists, but it is generated by the macro system. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 73

Most of the time, you can change the code generation inside the macros without having to change the code that is using the macros. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 74

First example: html templating. The current approach in other languages is broken. Putting your application code into the html code is wrong. And mixing html code with your app code is wrong. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 75

This is why there are so many complicated templating frameworks for Java: You have to mix two different languages - Java and html. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 76

But what if you could just extend your language to ‘know’ how to generate html? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 77

This example is taken from the TheDeadline’s printer- friendly tabular to do list. You build the html by using language constructs that ‘know’ how to generate html. You put function calls into the right spots to fill in the blanks. And you just call this as a function if you want to display a to do. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 79

Second example: Accessing the Google App Engine datastore. You can use a high-level Java interface providing Java objects or you can use a low-level interface to work directly on the key-value structures. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 80

The key-value structure of the Google datastore entities is almost an exact fit to our Clojure structs! So we decided to use the low- level API to the datastore. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 81

In addition to that, we wanted to have an abstraction layer to make the application itself independent from the underlying Google platform. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 82

This is how you would define a book entity in Python ... The following examples are taken from „Programming Google App Engine,“ Dan Sanderson, O‘Reilly Cool book. Buy it! :) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 83

This is how it is done in Java... Not enough space to show all the code that is needed for this example ... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 84

And this is how we do it in Clojure: This automatically generates the code to generate, store and retrieve entities of the type book. It converts the types to and from Google App Engine types and you can also add your own functions to validate and to convert every single property to and from a custom type (not shown in this example). We will write a blog post about it and post the code: http://www.hackers-with-attitude.com) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 85

Our Clojure data structure looks almost like the low- level key-value structure from Google’s datastore. We can work directly with it in our app. But the user of defentity doesn’t have to know how to use the datastore API at all. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 86

The difference to Python: We don’t declare types, but you can define functions for :pre-save and :after- read execution for every property and for the whole entity. We think, this is much more concise and much more powerful. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 87

One last example: We are using several AI techniques. We implemented a rule- based expert system that provides the functionality of systems like CLIPS, LISA or OPS5. © freiheit.com Berlin Lispers Meetup Presentation, Rel. 0.1 88

We extended the language to be able to do expert system tasks. So, we are not just creating APIs. We create powerful abstractions that generate their own integration layers on the fly. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 90

So, what is our conclusion? What made Clojure more fun and more productive for us in comparison with Java (or Scala)? Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 91

This is where you spend lots of your time in Java web development. It’s all about moving data around: User Interface Generating HTML, JSON from application objects Folding/ Translating HTTP-Requests to application objects Destructuring your data in your app Application code Translating application objects to SQL statements Translating SQL queries to application objects Database/Persistence Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 92

So you access a database, generate objects, iterate over objects, copy objects, convert objects to other objects, convert objects to JSON, to XML, etc., etc., etc. That is what you do in Java all the time... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 93

There are lots of Java frameworks that help you to generate the needed code because it is almost impossible to do this by hand. (That is also why you need Eclipse at all ...) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 94

But these frameworks add a lot of complexity. You need XML configuration and annotations. Java programming becomes more and more a configuration task. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 95

The edit-build-deploy-run cycle is very slow. All the code that is generated by the frameworks, the dependency injections, the start of the app server ... You go through this for every change you make. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 96

Our experience is that it is much better to have a simple unified data structure (struct) that you can pass around in your system and build powerful abstractions/mini-languages (with macros) to work on these data structures ... Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 97

... and to use a real dynamic and interactive programming environment that lets you change and test your code in a running system. Interactive programming even works with the Google App Engine dev server. We posted an article in our blog explaning how to setup your environment. After that, you need ‘ant’ only for the deployment on the App Engine production servers. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 98

One more thing: You don’t need Eclipse anymore. So you don’t have to reserve 1.0 Gigs on your machine for your editor/compiler anymore. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 99

You can do the same in Common Lisp! You could implement the same features that Clojure has in Common Lisp and vice versa. That’s the power of the Lisp programming languages! Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 101

But nobody is doing it. The Common Lisp Standard is hardly changing anymore. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 102

Cool people are writing world-class libraries for Common Lisp. But the platform itself is not changing anymore. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 103

Common Lisp is still cool, but Clojure is improving very fast. It’s a modern Lisp. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 104

For us, Clojure is the next evolutionary step of Common Lisp. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 105

That’s why we helped fund Rich Hickey’s development efforts on Clojure with a donation. Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 106

So if you want to learn Clojure, you can learn lots of advanced stuff from Common Lisp books. Especially from Paul Graham’s ‘On Lisp.’ Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 107

And it is much easier to sneak some Clojure code into the enterprise Java project your 40-person team is currently working on than to sneak in some Common Lisp code ... ;) Berlin Lispers Meetup © freiheit.com Presentation, Rel. 0.1 108

Thank you. TheDeadline Pre-Registration: https://the-deadline.appspot.com Our corporate Blog: http://www.hackers-with-attitude.com Looking for a job? http://www.freiheit.com Self: http://www.smartrevolution.de http://www.twitter.com/smartrevolution

http://www.hackers-with-attitude.com	7939
http://javierneirasanchez.blogspot.com	928
http://javierneirasanchez.blogspot.com.es	277
http://www.slideshare.net	261
http://paper.li	94
http://2804048809956568254_05b712714f55bcd8d268dddcd918aaa8fc4a5924.blogspot.de	91
http://javierneirasanchez.blogspot.mx	69
http://javierneirasanchez.blogspot.com.ar	49
http://www.clojure.cz	37
http://irr.posterous.com	36
http://www.dancheah.com	33
http://blog.simplefluous.com	12
http://jasoncwarner.posterous.com	12
http://javierneirasanchez.blogspot.ru	11
http://sigsegv.typepad.com	10
http://abtasty.com	10
http://localhost	8
http://daniel42.posterous.com	8
http://static.slidesharecdn.com	8
http://2804048809956568254_05b712714f55bcd8d268dddcd918aaa8fc4a5924.blogspot.com	7
http://translate.googleusercontent.com	7
http://sometimes-clojure.blogspot.com	7
http://javierneirasanchez.blogspot.de	5
http://javierneirasanchez.blogspot.pt	4
http://javierneirasanchez.blogspot.com.au	3
http://javierneirasanchez.blogspot.ch	3
http://webcache.googleusercontent.com	3
http://javierneirasanchez.blogspot.co.uk	3
http://a0.twimg.com	3
http://www.newsblur.com	2
http://javierneirasanchez.blogspot.in	2
http://javierneirasanchez.blogspot.com.br	1
http://www.iweb34.com	1
http://www.techgig.com	1
http://64.233.163.132	1
http://javierneirasanchez.blogspot.cz	1
http://javierneirasanchez.blogspot.jp	1
http://javierneirasanchez.blogspot.co.il	1
http://javierneirasanchez.blogspot.gr	1
http://javierneirasanchez.blogspot.fr	1

How a Clojure pet project turned into a full-blown cloud-computing web-app

by Stefan Richter

Accessibility

Categories

Upload Details

Usage Rights

40 Embeds 9,951

Statistics

How a Clojure pet project turned into a full-blown cloud-computing web-app Presentation Transcript