The document discusses the challenges and experiences of deploying 34,000 lines of Haskell code in a production web application, highlighting both the advantages and frustrations faced. Key issues include a broken library ecosystem, lack of established best practices, and difficulties with Haskell's tooling and error handling. Additionally, the text emphasizes the need for better ORM solutions and contributions to improve Haskell's development environment.

1. Haskell
Joys & frustrations of putting
34,000 lines of Haskell into production

2. Source of all this gyaan
34,000 LoC of Haskell in production
Touched (almost) all aspects of a typical web-app stack
Except Haskell (or Purescript) in the frontend.
Using Angular2 + TypeScript instead.
We benchmarked Miso, Reflex (GHCJS) and Pux, Thermite, Halogen (Purescript)
and wrote about it.

4. Why Haskell?Why Haskell?
आ बैल मुझे मारआ बैल मुझे मार

5. Vacation Labs
Travel-Commerce Platform
Enabling travel companies with e-commerce technology.
Responsive website, booking engine, payment integrations, backoffice system, distribution systems,
mobile app, analytics, etc.
Built mostly with Rails & Angular-v1 (Javascript)

6. Remember those numbers
“It's always good to remember where you come from and celebrate it. To
remember where you come from is part of where you're going.”
- Anthony Burgess

7. Problems with Rails/JS
Code
Unable to refactor core data-structures confidently
Unable to remove dead-code without fear of breaking
Unable to review code effectively

11. Building webapps in Haskell
is harder than it ought to be
What I felt a year ago, just starting out...
http://tinyurl.com/haskell-is-needlessly-hard

12. Haskell.
First, the GOOD parts.

13. factoringre
actoringref
ctoringrefa
toringrefac
refactoring
toringrefact
oringrefacto
ingrefactor
ngrefactori

14. factoringre
actoringref
ctoringrefa
toringrefac
refactoring
toringrefact
oringrefacto
ingrefactor
ngrefactori

15. { “Type-safe” : “JSON” }
Aeson is brilliant. Auto-derives a lot of boilerplate.
However, please be careful while using the GHC.Generics based derivation mechanism. It seems to be
slower during compilation AND runtime. Prefer TH, which is faster.
OTOH writing JSON codecs in TypeScript by hand.

16. HTTP / Web / App Server
WAI/Warp has pretty good performance.
One lightweight thread per-incoming request.
Solid concurrency without an event-driven / callback model
(unlike node).

17. Now, for the
BAD parts.

18. Big problems faced first-hand
Library eco-system is effectively broken
Lack of established best-practices
Editor tooling
Records
Lack of ORM

19. Library ecosystem is
effectively broken
Lack of libraries it NOT a problem
Fragmentation, non-standardization is the bigger problem
Heaps of cruft on Hackage
Broken library ecosystem

20. I can easily compare Haskell libraries
to select the best one
Strongly
disagree
Disagree Neutral Agree Strongly
agree
100
200
300
400
500

21. I can easily compare Haskell libraries
to select the best one
Strongly
disagree
Disagree Neutral Agree Strongly
agree
100
200
300
400
500
44% respondents
replied negatively

22. Emails
SES, Postmark, Mandrill, raw SMTP, etc. They are all there. But, first we
needed regular SMTP.
Fiasco with `smtp-mail` library.
We started out with the most obviously named `smtp-mail` library on Hackage. After deploying to
production, we realised that it doesn’t support SSL/TLS. Switched to HaskellNet, and it turns out even
that doesn’t support SSL/TLS. Instead there’s HaskellNet-SSL! We finally used HaskellNet, but had to
write extra code to call different functions for TLS and non-TLS SMTP configurations. What’s the
purpose of splitting common use-cases across libraries?
Broken library ecosystem

23. Sendgrid library
`haskell-sendgrid` is incomplete. We wrote our own.
Ping us on Twitter if you want to help open-source our code. (Open bounty)
Will include code clean-up, documentation, and tests.
http://www.vacationlabs.com/haskell-bounty-program/
Call for contribution

24. Testing
287 packages related to testing on Hackage
None of them runs tests in isolated DB txns out-of-the-box, which is a very
common use-case for webapps.
We don’t claim to have used all of them, but spent quite some time trying-out
various libraries and ultimately ran out of R&D time.
https://github.com/feuerbach/tasty/issues/171
Broken library ecosystem

25. DB libraries
persistent, groundhog, hasql, pg-simple, opaleye, HaskellDB, HRR, and more.
Which to use when? Why so many?
Had unsatisfactory experience with Persistent initially. Started looking for
alternatives, and settled with Opaleye. Still not completely happy.
Issues with Persistent: models file is a compile bottleneck contributing to slow reloads, no way to
have DEFAULT values come from DB (eg. createdAt, updatedAt), unable to figure out how to
compose SELECT/WHERE clauses, esqueleto was a dead-project when we looked at persistent.
Broken library ecosystem

26. Lack of established
best-practices
Community keeps arguing about core stuff and all this chatter confuses
new-comers. Personally wasted a lot of time chasing research-y ideas.
stack vs cabal
mtl vs free-monads
error handling
Is TH good or bad?
No best practices

27. Installing Haskell
FOUR different ways to install Haskell - seriously?!
OS’ native package manager, Haskell platform, Stack, Nix
PS: Just use “stack” - tried & tested. LTS snapshots are awesome!
Upgraded LTS twice during our development. Resulted in a few trivial compiler errors. Never had
such painless upgrades in Rails or Node.
No best practices

28. Our take on this divisive
issue...
https://www.reddit.com/r/haskell/comments/4zzmoa/haskellorg_and_the_evi
l_cabal/
https://github.com/haskell-infra/hl/issues/176
No best practices

30. Error handling
EIGHT different ways. Each library uses something else.
http://www.randomhacks.net/2007/03/10/haskell-8-ways-to-report-errors/
`ExceptT` vs `Conrol.Monad.Catch` fiasco
Discovered ExceptT via Servant, which uses it as its core monad. Started using it throughout our own
domain-specific code without realising what we were doing. Turns out ExceptT “exceptions are
completely different from what `error`, `throwIO` and `throwM` have. Had to refactor our entire
project to fix this. Haskell refactoring, ftw, btw!
No best practices

31. do
x <- runExceptT (liftIO $ throwIO DivideByZero)
case x of
Left e -> pure “exception caught”
Right r -> pure “should never happen!”

32. do
x <- runExceptT (liftIO $ throwIO DivideByZero)
case x of
Left e -> pure “exception caught”
Right r -> pure “should never happen!”
-- OUTPUT
*** Exception: divide by zero
-- Reaction: Wtf ?!

33. do
x <- Control.Monad.Catch.try (liftIO $ throwIO DivideByZero)
case x of
Left (e :: SomeException) -> pure “exception caught”
Right r -> pure “should never happen!”

34. do
x <- Control.Monad.Catch.try (liftIO $ throwIO DivideByZero)
case x of
Left (e :: SomeException) -> pure “exception caught”
Right r -> pure “should never happen!”
-- OUTPUT
“exception caught”

35. Error reporting
Haskell code throws runtime errors, mostly at system/IO boundaries.
Libraries should try to report them properly.
Hype - “if it compiles, it runs”. Reality - “after a refactor, if it compiles, then
there is a high chance that you haven’t broken anything”
No best practices

37. Which file?
Turns out “fastLogger” was trying to open a log-file in a non-existent
directory. (On LTS-9.0)

38. *** Exception: UnexpectedNull
{ errSQLType = “varchar”
, errSQLTableOid = Just (Oid 603560)
, errSQLField = “result22_2”
, errHaskellType = “Text”
, errMessage = “”
}

39. *** Exception: UnexpectedNull
{ errSQLType = “varchar”
, errSQLTableOid = Just (Oid 603560)
, errSQLField = “result22_2”
, errHaskellType = “Text”
, errMessage = “”
}
Which row? What SQL was being run?
Any other info that can help me debug this? Stacktrace?

41. Occurred during stress testing of JSON API. Wasted a lot of time!
Which file descriptor? Which library was accessing it? Nothing!
Turned out that DB pool size was larger than what DB could handle.

42. Absence of CallStacks can
probably be fixed
Call for contribution
Imperceptible perf-penalty with HasCallStack in IO code
We benchmarked with tight pure-compuations - 1.2x
Not being used by IO-heavy libraries. Raise some PRs!
GHC discussion to infer HasCallStack automatically
https://ghc.haskell.org/trac/ghc/ticket/13360

43. Testing
We drank the QuickCheck kool-aid. Heard John Hughes’ talk at FnConf 2016
and got super-excited. Wasted 2-3 months trying to figure out how to use
QuickCheck for typical webapps. Ran out of R&D time and had to launch
without tests. Hindsight - Regular HSpec would’ve worked out just fine!
QuickCheck only for round-tripping of codecs.
Discussed extensively at https://github.com/nick8325/quickcheck/issues/139
which resulted in `quickcheck-state-machine`, which, unfortunately, is too
complicated to understand and/or use.
No best practices

44. Logging & Instrumentation
“Deep instrumentation” in Haskell seems to be non-existent
Eg. kind of stuff provided by Rails out of box, or Skylight, or New Relic
Very hard to build drop-in instrumentation libraries for Haskell.
Probably because of the type-system itself.
http://tinyurl.com/haskell-instrumentation
Built our own ad-hoc logging & instrumentation layer by wrapping important
libs

45. Editor tooling
SAD TRUTH - Nothing works as well as it should.
Tried a lot of things (except VIM based stuff)
Emacs + haskell-mode, Spacemacs + Intero, SublimeHaskell, Haskero, Haskelly, HIE, ghcid, etc.
Underlying GHCi hogs/leaks memory.
https://ghc.haskell.org/trac/ghc/ticket/14336
Editor tooling

47. Important caveat about
memory usage
Haskell DOES NOT leak memory in production.
At least we haven’t encountered any mem-leaks in prod yet.
Only present in GHC-i -- used in development.
Production Rails-app: 360 MB of RSS (x 3 processes)
Production Haskell-app: 78 MB of RSS (single process).
Note: don’t believe the numbers that `top` reports. 1TB for our Haskell app. Use `ps -o rss <pid>`
instead

48. Recommendation:
Editor tooling
Use spacemacs + intero OR VSCode + HIE (Haskell IDE Engine)
Keep ghci open in a terminal window and make friends with
:set -fobject-code & :load & :reload
`ghcid` and `stack --file-watch -fast` are tools around this dev workflow.

49. Shout-out to
HIE (Haskell IDE Engine)
Call for contribution
Please use it, if it works for you. Make it work for you!
It implements the Language Server Protocol which IMO is the correct way
forward.
Contribute if you can.
https://github.com/haskell/haskell-ide-engine

50. Broken record
Records. Records. Records.
Records in Haskell are beyond horrible.
Some say, Haskell doesn’t even have records.
http://www.parsonsmatt.org/overcoming-records/

51. -- Your `users` table would probably map to the following…
data User = User
{ userId :: UserId
, userCreatedAt :: UTCTime
, userUpdatedAt :: UTCTime,
, userStatus :: UserStatus,
, userName :: Maybe Text
, userEmail :: Text
, userAge :: Maybe Int
}
-- But, you can’t use this in your `createUser` endpoint.
-- UI shouldn’t set `id, createdAt, updatedAt, status`
-- No easy way to create a “sub-record” from a master record.

52. -- Created our own library `record-splicer` to deal with this...
-- https://github.com/vacationlabs/record-splicer
-- Similar approach via Metamorphosis library
createRecordSplice SpliceArgs
{
sourcePrefix = "user"
, source = ''User
, requiredFields =
[
'userName
, 'userEmail
, 'userAge
]
, targetPrefix = "newuser"
, targetName = "NewUser"
, deriveClasses = [''Eq, ''Show, ''Generic]
}
makeLensesWith abbreviatedFields ''NewUser
-- Also generates utility functions to go from User -> NewUser and go
from NewUser -> NewUserDelta -> User

53. // Same thing in TypeScript
type interface User {
readonly id: number
, readonly createdAt: DateTime
, readonly updatedAt: DateTime
, readonly status: ‘unconfirmed’ | ‘confirmed’ | ‘blocked’
, readonly name: string | null
, readonly email: string
, readonly age: string | null
}
// Making a typesafe sub-record
type NewUser = Pick <User, ‘name’ | ‘email’ | ‘age’>;
// Making a typesafe super-record
type Admin extends User {
readonly permissions: Array<Permission>
}

54. DB Library.
Dare I say ORM?
Either you use an ORM or you write an ad-hoc ORM.
ORM seems to be a taboo in the FP/Haskell world. Call it by a different name, say FRM, if it makes
things more palatable. But, writing raw SQL queries for every small thing is definitely not the answer.
There are a lot of common patterns that an ORM/FRM solves. You either use them, or end-up writing
them in an ad-hoc manner.
Tried to understand how to build apps without an ORM. Not convinced. At all.
[1] [2] [3] [4]
Lack of ORM

55. Associations?
No DB library deals with associations.
Except two new discoveries - beam & postgresql-orm
`beam` is a new kid of the block and just overhearing the chatter on their git-repo gives an impression
that it is progressing rapidly. BEAUTIFUL docs, btw.
`postgresql-orm` seems like what we needed. Feel like kicking ourselves for discovering this so late!
Lack of ORM

56. Validations?
No DB library deals with validations.
Rails’ (also Django?) convention of putting strict invariants/validations in the
DB layer works ridiculously well.
Some validations are better placed at the DB layer, and some validations at the endpoint/handler
layer. It’s not an either/or decisions. It’s “and”. We need validations at both layers.
Lack of ORM

57. Validations in DB-layer are an
anti-pattern?
Let’s look state-of-the-art: “digestive-functors”
Lack of ORM

58. -- (a) Writing parsers by hand, thus negating the effect of Aeson’s
auto-derivation
-- (b) Positional constructors: extremely easy to break across
refactors

59. DB Library problem as well.
Incomplete / half-baked DB libraries
mysql-simple - (had?) concurrency issues
postgresql-simple doesn’t use PREPARE/EXECUTE
Opaleye generates slow queries
https://github.com/tomjaguarpaw/haskell-opaleye/issues/340 &
https://github.com/tomjaguarpaw/haskell-opaleye/pull/347 Working with the author to fix this.
Lack of ORM

60. Applicative parsers?
Or “Hodor” parsers.
Special hate for applicative parsers!
Negate all benefits of static typing.
Especially true for postgresql-simple.
Lack of ORM

61. query_ = ([qc|
SELECT
c.id as client_id
, coalesce(c.custom_domain, c.domain) as domain
, c.home_page as home_page
, c.support_phone as support_phone
, c.support_email as support_email
, c.logo_photo_id as loto_photo_id
, logo_photo.image_file_name as logo_file_name
, logo_photo.image_fingerprint as logo_fingerprint
, c.favicon_image_id as favicon_image_id
, favicon.image_file_name as favicon_file_name
…

62. query_ = ([qc|
SELECT
c.id as client_id
, coalesce(c.custom_domain, c.domain) as domain
, c.home_page as home_page
, c.support_phone as support_phone
, c.support_email as support_email
, c.logo_photo_id as loto_photo_id
, logo_photo.image_file_name as logo_file_name
, logo_photo.image_fingerprint as logo_fingerprint
, c.favicon_image_id as favicon_image_id
, favicon.image_file_name as favicon_file_name
…
rowParser assetHost_ = Storefront
<$> field <*> field <*> field <*> field <*> field
<*> (photoEssentialsParser assetHost_ Photo.WebsiteLogo)
<*> (photoEssentialsParser assetHost_ Photo.Favicon)
<*> field <*> field <*> field <*> field <*> field <*> field
<*> field <*> field <*> field <*> field <*> field <*> field
<*> field <*> field <*> hstoreBoolParser <*> field
<*> field <*> field <*> field <*> field <*> field <*> field

63. query_ = ([qc|
SELECT
c.id as hodor
, coalesce(c.custom_domain, c.domain) as hodor
, c.home_page as hodor
, c.support_phone as hodor
, c.support_email as hodor
, c.logo_photo_id as hodor
, logo_photo.image_file_name as hodor
, logo_photo.image_fingerprint as hodor
, c.favicon_image_id as hodor
, favicon.image_file_name as hodor
…
rowParser assetHost_ = Storefront
<$> hodor <*> hodor <*> hodor <*> hodor <*> hodor
<*> (photoEssentialsParser assetHost_ Photo.WebsiteLogo)
<*> (photoEssentialsParser assetHost_ Photo.Favicon)
<*> hodor <*> hodor <*> hodor <*> hodor <*> hodor <*> hodor
<*> hodor <*> hodor <*> hodor <*> hodor <*> hodor <*> hodor
<*> hodor <*> hodor <*> hstoreBoolParser <*> hodor
<*> hodor <*> hodor <*> hodor <*> hodor <*> hodor <*> hodor

64. PG-Simple’s “hodor” parser
can be fixed
There is an open bounty.
https://github.com/lpsmith/postgresql-simple/issues/43
http://www.vacationlabs.com/haskell-bounty-program
Call for contribution

65. What did we really build in
34,000 lines of Haskell?
Flexi-payment plugin
Allows travel operators to send a flexible payment link to customers that allows each individual in a group to
pay for him/herself and allows payments to be split in multiple parts.
DelayedJob clone (but using LISTEN/NOTIFY) along with UI
Sendgrid library
Working clone of Inky to send bulletproof HTML emails. In process of open-sourcing.
Opaleye boilerplate code-generator
Deep instrumentation wrappers over core libs

67. Job Queue
Rails has DelayedJob (with an RDBMS backend). Works really well for moderate
workloads. ACID compliant and doesn’t add more moving parts to production.
Couldn’t find anything similar in Haskell world. Built our own. Will open-source.
Except `yesod-job-queue` but we aren’t using Yesod.
Redis & SQS job-queues available, but we didn’t use first-hand. Can’t comment.

68. JobAdmin UI built by intern in 4 months (included learning Haskell).
Built using Servant, Lucid, Aeson, Opaleye.

69. Initial (excruciating) pain for long-term gain
We have already gone through the pain
Now, we are reaping the gains
Haskell sucks.
Why continue with it?

70. What’s next?
Raising awareness about the issues we faced.
The core language is not as confusing
as the ecosystem is.
Contributing in any way to fix the ecosystem.

71. Thank you
@saurabhnanda on twitter
@vacationlabs on instagram
Haskell Bounty Program
Haskell Internship Program
We’re hiring Haskellers (and Haskell enthusiasts)