This document discusses Apache Hop, an open source data orchestration platform. It provides an overview of Apache Hop's capabilities for managing data pipelines and workflows. Key features highlighted include its modular architecture, support for technologies like Apache Spark and Neo4j, and focus on ease of use, testing, and community development. The roadmap outlines plans to graduate to a top-level Apache project and improve cloud and mobile support.

1. Neo4j and Apache Hop
Future-proof data orchestration for Neo4j

3. What is a graph database?
● A graph consists of nodes connected by relationships.
● Graph databases store data as graph structures (nodes and
relationships)

4. Property Graph Model Components
● Nodes
○ Represent the objects in the graph
○ Can be labeled
CAR
PERSON PERSON

5. Property Graph Model Components
● Nodes
○ Represent the objects in the graph
○ Can be labeled
● Relationships
○ Relate nodes by type and direction
CAR
DRIVES
LOVES
LOVES
LIVES WITH
O
W
N
S
PERSON PERSON

6. Property Graph Model Components
● Nodes
○ Represent the objects in the graph
○ Can be labeled
● Relationships
○ Relate nodes by type and direction
● Properties
○ Name-value pairs that can go on
nodes and relationships
CAR
DRIVES
name: “Dan”
born: May 29, 1970
twitter: “@dan”
name: “Ann”
born: Dec 5, 1975
since:
Jan 10, 2011
brand: “Volvo”
model: “V70”
LOVES
LOVES
LIVES WITH
O
W
N
S
PERSON PERSON

7. Cypher: Graph Query Language
MATCH (:Person { name:"Dan"} ) -[:LOVES]-> (:Person { name:"Ann"} )
LOVES
Dan Ann
LABEL PROPERTY
NODE NODE
LABEL PROPERTY

8. Neo4j Use Cases
● Social network analysis
● Recommendation engine
● Search & Discovery
● Network & Data Center
● Master Data Management
● Identity & Access Management
● GEO
● Fraud Detection
● …
Blog series: Top 10 Neo4j use cases

9. Neo4j
● The world’s leading graph database
● June 2021: raised $325M!
● On-premise, in the cloud or hybrid (Aura)
● Graph ML algorithms
● High performance
● Up to trillions of nodes and relationships

11. Common Approaches
● Neo4j tools:
○ Load CSV
○ APOC
○ Neo4j ETL (RDBMS to graph)
○ Code (Python, Java, Go, …)

12. Why Data Projects Fail...

14. What is required?
● Good scope and project management
● Easy to develop, run, debug, troubleshoot (enable citizen developers)
● Error handling, logging, monitoring, lineage
● Testing (unit, integration, regression)
● Repeatable, reproducible
● Life Cycle Management
● Scale with data requirements, volume and project
● Integrate perfectly with DevOps
● Integrate perfectly with Neo4j

16. Apache Hop
● Recursive acronym: Hop Orchestration Platform
● Orchestration:
○ Data: pipelines and workflows
○ Metadata: editing, handling, management,...
○ Insights: data/execution lineage, logging, …
○ Configurations: handling ecosystem complexity

17. Apache Hop : background
● Community lead initiative
●
● New scalable GUI
● New metadata back-end
● Simplified toolset
● Code refactored, renamed, trimmed down, ...
● Extra plugins: Projects, Testing, Apache Beam, Debugging, ...
● …

18. Apache Hop Incubator
● Forked PDI/Kettle 8.2 + WebSpoon + patches + plugins + …
● → Represents 20 years of development!
● Part of the ASF Incubator, close to TLP
● 1.0 early October!
● built continuously
● Active, quickly growing community
https://hop.apache.org

19. Why Apache Hop?
● A quickly diversifying technological landscape
→ Makes it hard to manage complexity
→ Drives the need for rapid innovation
● Development done independent from a single large corporation
● By and for data orchestration professionals

20. Guiding principles
Apache Hop aims to make data orchestration better:
● Easy: setup, build, maintenance, deployment, …
● Fast: startup time, supporting Spark, Flink & DataFlow, ...
● Transparent: before, during and after execution
● Predictable: unit and integration testing
● Innovative: need for the latest tech (digital transformation)
● Best practices: support version control, testing, CI/CD, project,
lifecycle management, ...

21. Apache Hop : key architecture features
● Metadata driven: no code generation
● Modular pluggable architecture: scale back to <30MB
● Fast startup, minimal overhead
● Apache Beam with support for Apache Spark, Apache Flink and GCP
DataFlow runners
● Version controlled documentation
● Ease of use: transparent naming and easy to use tools
● Integration test: critical components are tested daily with integration tests
→ runtime compatibility, stability, ...

22. Apache Hop : key GUI features
● Pluggable GUI features
● Scalable interface for high DPI displays or visually impaired
● Perspectives for easy fast context switching
● Designed for web browsers and mobile users
→ Single click mode for faster navigation
● Full support for 4 platforms: Windows, OSX, Linux & Web
● Support for “dark mode” themes on Linux and OSX

23. Apache Hop : key configuration features
● All GUI configuration options have a command line variant
● Single central system configuration JSON file
● Easy project and lifecycle environment configuration
● Configuration and metadata inheritance from other projects
● Standard docker container
● Stateless server supporting multi-tenancy
● Version control friendly setup

24. Apache Hop : Q4 & 2022 roadmap
● Graduation to Apache Top Level Project (TLP)
● Pluggable field expressions
● Java 11 (mirror Apache Beam)
● new execute/preview/debug GUI
● Improved cloud support
● Airflow runtime support
● Marketplace for 3rd party plugins
● ...

25. Apache Hop : Community!
● Accepted in the ASF Incubator in Sept ‘20, ready to graduate to TLP
● Apache is a community building organisation
● Great communities deliver great software
● During incubation we are asked to
→ Grow the community
→ Release software the Apache way

26. Apache Hop : Community
● No single company drives the software forward
● The Hop community is growing fast across all social media channels, chat
server, …
● Anyone is welcome with ideas, code, bug fixes, suggestions, documentation,
translations, …
● No bug is too small or too big to fix.
● No improvement suggestion is too small or too big to consider

27. Apache Hop and Neo4j
● Best Neo4j (incl Aura!) support of any platform!!
● Functionality:
○ Neo4j logging perspective
○ Neo4j connection type, graph data type
○ 20+ action and transform plugins to write data to Neo4j, run
Cypher, split graph in nodes and relationships etc

28. Apache Hop : Enterprise support
● Lean With Data wants to help you!
● See: www.leanwithdata.com
→ Support
→ Training
→ Certification
→ Custom development
→ Data orchestration tool migration
Lean Orchestration: professionally supported Apache Hop.
Get the best of both worlds.
Know.bi is a Lean With Data partner

29. Demo

30. Thank you for your interest and time!
@ApacheHop, @know_bi
https://www.linkedin.com/company/apachehop
https://www.linkedin.com/company/knowbi
https://chat.project-hop.org