Ian Robinson, Engineer at Neo4j, talks about how you productionize your Neo4j-based application. In this talk from GraphConnect San Francisco 2015, he looks at some of the most important considerations around designing, building and operating a Neo4j app.
Topics include:
* Where Neo4j fits in your application architecture, in both its embedded and server modes
* How to configure its clustering for high availability and high read throughput
* Backup strategies
* The new monitoring capabilities in Neo4j 2.3
11. Cluster Configura3on
Joining Cluster
• ha.initial_hosts (neo4j.proper)es)
• List of servers to contact when joining cluster
• All hosts must be available when star,ng instance
• For large clusters, supply only a small number of hosts, e.g. 3
Pull and Push Transac3ons
• ha.pull_interval=10s (off by default)
• ha.tx_push_factor=1 (default, but best efforts only)
Tuning
• ha.heartbeat_timeout=11s (default)
• Heartbeats sent, by default, every 5s
• Increase ,meouts if pauses cause heartbeats to be delayed
• Warning: it will take longer to discover an instance has failed
• ha.state_switch_timeout=120s (default)
• Increase if new instances ,meout while catching up with master on startup
12. HA Endpoints – Useful for Load Balancing
Endpoint State Status Code Body
/db/manage/server/ha/master
Master 200 OK true
Slave 404 Not Found false
Unknown 404 Not Found UNKNOWN
/db/manage/server/ha/slave
Master 404 Not Found false
Slave 200 OK true
Unknown 404 Not Found UNKNOWN
/db/manage/server/ha/available
Master 200 OK master
Slave 200 OK slave
Unknown 404 Not Found UNKNOWN
From 2.3 onwards
dbms.security.ha_status_auth_enabled=false
neo4j.proper)es
13. HA JMX Endpoint
JSON Response
• Alive?
• Role
• Last commiYed transac,on ID
• Instances in cluster
• Role
• Instance ID
• Available?
• URI
Iden,fy slaves
falling behind
Does everyone agree
on composi,on of
cluster?
/db/manage/server/jmx/domain/org.neo4j/instance%3Dkernel%230%2Cname%3DHigh%20Availability
22. Cache Sharding Using Consistent Rou3ng
Applica,on
1 2 3
Load Balancer
MATCH (c:Country{name:'Australia'})...MATCH (c:Country{name:'Zambia'})...MATCH (c:Country{name:'Norway'})...
A-I 1
J-R 2
S-Z 3
MATCH (c:Country{name:'Zambia'})...MATCH (c:Country{name:'Norway'})...MATCH (c:Country{name:'Australia'})...
23. Configure HAProxy for Cache Sharding
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend neo4j-slaves
backend neo4j-slaves
balance url_param country_code
server s1 10.0.1.10:7474 maxconn 32
server s2 10.0.1.11:7474 maxconn 32
server s3 10.0.1.12:7474 maxconn 32
listen admin
bind *:8080
stats enable
24. Configure HAProxy for Cache Sharding
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend neo4j-slaves
backend neo4j-slaves
balance url_param country_code
server s1 10.0.1.10:7474 maxconn 32
server s2 10.0.1.11:7474 maxconn 32
server s3 10.0.1.12:7474 maxconn 32
listen admin
bind *:8080
stats enable
25. Backups
Modes
• Full
• Incremental
• On top of a previous backup
• Uses logical logs to apply changes, so logs must be kept at least 2 x backup interval
Consistency Check
• Backup and standalone tool
• Evaluate store health
• Part of backup and standalone tool
• -verify false to disable in backup
keep_logical_logs=7 days
neo4j.proper)es