The team at Picnic Software giving a detailed walkthrough of their application architecture and development processes for a large Angular and .NET Event Sourcing application.

1. Developing a flexible and
scalable application

3. Presenting Today
Andrew Browne Dave Churchill Basarat Syed
Nick Josevski Matt Walkenhorst

4. Disclaimer
The views expressed here are solely those of the
authors in their private capacity and do not in any
way represent the views of the Picnic Software Pty
Ltd, or any other associated entity or shareholder.
Picnic Software Pty Ltd has not approved, endorsed,
embraced, friended, liked, tweeted, google-plused,
pinterested, dugg, reddited, hacker-newsed,
sanctioned or authorized this presentation.

6. Agenda
• App & Tech
• Infrastructure & Data Flow
• Deployment & Scalability
• Permissions with Neo4j
• Client Side Technologies
• Development & Testing Workflow

7. What we do
• We are an ISV (Independent Software Vendor)
– Building and running a workflow/collaboration
application
• Partnerships with large businesses in the
Advertising/Marketing sector
• Our customers are primarily large retailers

8. Our App
• Media Library
– High resolution files; PSDs, Video
• Collaborative workflows
– Coordinating inputs
• Photography, Illustrations, Graphic Design
– Producing advertising outputs
• Over multiple media channels
– Catalogues, Billboards/Print, TV, Radio, Web

9. Our Tech Stack
• F#, C#, ASP.NET MVC, ServiceStack
• EventStore, Eventful, RavenDB, Neo4j
• Angular, TypeScript, Mocha, Node, Sass
• SignalR, AutoMapper, RabbitMQ, LINQ,
NSubstitute, Nunit, FSUnit, FParsec, FsCheck
• AWS, Docker, Riemann, Logstash, PowerShell,
PSake, NodaTime
• GitHub, TeamCity, Octopus Deploy, Slack,
YouTrack

10. Flexibility
• Architecture choices to support
– Changes in requirements
– Future customers working in same domain
• When we started building we had
– Known customer workflows
– Known unknown customer workflows
– Unknown unknown customers workflows

11. Questions

12. Our Infrastructure
Andrew

13. Sydney region
CloudFront
S3Core, Neo4j,
RavenDb
Rabbit,
EventStore
Media
Processing
Web
Elastic Load
Balancing
Availability Zone x 2
MySQL
HA Proxy

14. S3

15. CloudFront

16. Elastic Load Balancer

17. EC2

20. Event Store

23. Writes

24. Reads

25. S3
EventStore
Web
Availability Zone x 2
Core Neo4j RavenDb Media
Processing
Rabbit
Disposable

26. Questions

27. Deployment & Scalability
Matt

28. Deployment
• Server Infrastructure – AWS
– Starting new instances largely a manual affair at
this point
• Configuration Management – Chef
• Application deployment – Team City, Octopus
Deploy

29. Chef
“Give me six hours to chop down a tree and I will
spend the first four sharpening the axe.”

30. Chef
• Windows / Linux deployment nodes
• Each node is in an Environment
• Each node performs one or more Roles
• Each Role requires the running of one or more
Recipes
• Recipes are stored in cookbooks
• Keep configuration in Git
• Keep CHEF Server configuration in Git

32. Octopus Deploy
• Windows code deployments only – Linux coming
soon.
• Environments, Roles, Apps, Releases.
• Deployment Process – steps executed on Roles to
“Tentacles”
– Nuget packages retrieved from Team City
– Store configuration as variables
– Variable snapshot + Nuget packages = release
– IIS, Windows Service, PowerShell steps available
– Partial and Rolling deploys
– Easy to roll back – just re-deploy last working release.

33. SOON: Blue / Green Deployments
• Asgard brings up a NEW (GREEN) copy of
production AWS infrastructure.
• Automatically Bootstrap instances against Chef
and Octopus.
• Smoke test GREEN environment
• Add GREEN web servers to load balancer
• Remove OLD (BLUE) web servers from load
balancer
• Asgard tears down BLUE production AWS
infrastructure.

34. Message delivery between processes
• Requirements
– Reliable.
– Easy to manage.
– Easy to use.
– Low latency.

35. Things we looked at (2+ years ago)
• NServiceBus
– Tied to MSMQ at the time.
• MassTransit
– Lacked documentation.
– Not ready for prime-time at that point.
• RabbitMQ + EasyNetQ
– Simple, best fit for us.
– Wrote our own client – bad idea.

37. RabbitMQ
• Written in Erlang, maintained by Pivotal.
• Linux / Windows.
• Easy administration (Web, command line, JSON).
• Supports
– Clustering and failover.
– Durable and HA queues.
– ‘At least once’ delivery guaranteed.
– Direct, Fan-out and Topic exchanges.
– Partitioning (vhosts), Federation & Shovelling.

38. How Picnic use RabbitMQ
• Setup
– Cluster of RabbitMQ servers behind ELB in
multiple AZs.
• You can use HAProxy instead of ELB.
– EasyNetQ library by Mike Hadlow.
• Handles subscription, publish and reconnection logic.
– So solid now we hardly think about it.

39. Use case: Scaling of long-running CPU
and IO intensive operations
• File format conversion, Zip bundling, PDF &
InDesign creation etc.
• Uses Topic exchange. Currently just one topic!
• Subscribers are round-robined by the broker.
• Subscribers are isolated – no clustering.
• Scaling - just launch new instances.
• Redundancy – launch in multiple Azs
• This has worked really well for us.

40. Use case: Distribution to SignalR
clients
• In-app notifications, long running task
progress etc. to the browser.
• Each web server receives all messages (Fan-
out exchange)
• Messages delivered to users / groups via
SignalR

41. Questions

42. Scalable Permissions
Dave

43. Permissions Model
<approval-1-guid> Owner
<footwear-folder-guid> Write
• All entities in the system are identified by GUIDs
• Each permission applies to a specific entity

44. Permissions Model
<approval-1-guid> Owner
<footwear-folder-guid> Write
• Permissions also have a role
<footwear-folder-guid> Read

45. Permissions Model
<approval-1-guid> Owner
<footwear-folder-guid> Write
• Each user has a corresponding "Me" permission
<footwear-folder-guid> Read
<user-dave-guid> Me

46. Permissions Model
<approval-1-guid> Owner
<footwear-folder-guid> Write
• As events arrive, relationships are built up between permissions
• e.g. a JobCreated event might give Owner permission to the creator
<footwear-folder-guid> Read
<user-dave-guid> Me

47. Permissions Model
<approval-1-guid> Owner
<footwear-folder-guid> Write
• Relationships don't have to stem from the Me permissions
• e.g. having Write permission on a folder could mean you can also Read
<footwear-folder-guid> Read
<user-dave-guid> Me

48. Permissions Model
<approval-1-guid> Owner
<footwear-folder-guid> Write
• A user has a permission if there's a path from the user's Me node
• This user doesn't have Read on the folder
<footwear-folder-guid> Read
<user-dave-guid> Me

49. Permissions Model
<approval-1-guid> Owner
<footwear-folder-guid> Write
• A user has a permission if there's a path from the user's Me node
• Giving Write on the footwear folder also gives the user Read
<footwear-folder-guid> Read
<user-dave-guid> Me

50. Original Implementation
Write
Read
Me
ApprovalOwner
• RavenDB document for each permission
• Records which permissions directly inherit this permission
Read <footwear-folder-guid>:
[Write <footwear-folder-guid>]
Write <footwear-folder-guid>:
[Me <user-dave-guid>]
Owner <approval-1-guid>:
[Me <user-dave-guid>]
Me <user-dave-guid>:
[]

51. Original Implementation
Write
Read
Me
ApprovalOwner
• Worker task builds a second state document for each
permission
• Records all permissions which inherit this permission
Read <footwear-folder-guid>:
[Write <footwear-folder-guid>,
Me <user-dave-guid>]
Write <footwear-folder-guid>:
[Me <user-dave-guid>]
Owner <approval-1-guid>:
[Me <user-dave-guid>]
Me <user-dave-guid>:
[]

52. Original Implementation
Write
Read
Me
ApprovalOwner
• A user has a permission if their Me appears in the
permission's state document
Read <footwear-folder-guid>:
[Write <footwear-folder-guid>,
Me <user-dave-guid>]

53. Original Implementation - Issues
• State documents can get large
• Introduce intermediate groups
• Takes time for state documents to be updated
• Cache and update permission graph in process
• Other processes can still sometimes see out-of-date permissions
• Use a graph database!

55. New Implementation
• In process of switching to Neo4j
• Transactional updates
• No need to calculate intermediate state
• Faster
• Simpler
• Still need to send some state data across to RavenDB for permissions when
searching

57. Questions

58. Client Side Architecture
Bas

59. http://slides.com/basarat/picnic-frontend

60. Questions

61. Development Workflow
Nick

62. Development Workflow
• GitHub
– Feature branches
– Pull Requests

63. Pull Requests
• Just over 2 months now using the PR based
workflow
• Approx 120 closed pull requests so far
• How
– Features branches, GitHub Tagging, TeamCity build
process comments against PR
– Asynchronous task for a team member
• “Here’s a PR please review when you can”

64. Team City & Tagging

65. Development Workflow
• Why
– It was recommended to us.
– Supports consistent and frequent code reviews.
• Improves code quality
• Shares knowledge amongst the team
– Lets us catch some bugs much earlier.

67. Development Workflow
• The wins
– Build server is more often in a green state.
• Can push to your PR branch to rely on CI to give feedback
– Knowledge sharing
• “Oh, that’s how you solved that”
• Reducing silo effects
– Offer constructive feedback to others
• “This could be made better by…”
– Bugs / issues caught
• Typos, debug code left in, incomplete/missed features

68. Development Workflow
• Testing
– Each PR builds as if it was already merged
– Unit/Integration tests run against PR in TeamCity
– YouTrack bugs marked with build numbers to track
deployment

69. Development Workflow

70. Development Workflow
• Agility
– Tracking feature changes as they evolve alongside
code
• As with all documentation - trying our best to keep up
to date
• PRs feedback can “code change not reflected in docs”
– Testing team
• Can review these changes and be more up to date with
• Review PRs for an idea of scope of changes and where
to look for issues

71. Development Workflow

72. Questions

73. Thanks

74. Other ALT.NET Presentations by us
Event Sourcing with F# - Andrew Browne
Thinking in a document centric
world with RavenDB - Nick Josevski