An overview of how Opencast works, how it is built, and how it communicates within itself, and between nodes

1. Opencast Architecture
Greg Logan
gregorydlogan@gmail.com
February 15, 2018
Greg Logan February 15, 2018 1 / 22

2. Housekeeping
This is going to be a deeply technical talk
Greg Logan February 15, 2018 2 / 22

3. Housekeeping
This is going to be a deeply technical talk
If reality seems to be imploding...
Feel free to zone out for a bit
Ask questions
Greg Logan February 15, 2018 2 / 22

4. Housekeeping
This is going to be a deeply technical talk
If reality seems to be imploding...
Feel free to zone out for a bit
Ask questions
This is being recorded
Greg Logan February 15, 2018 2 / 22

5. Housekeeping
This is going to be a deeply technical talk
If reality seems to be imploding...
Feel free to zone out for a bit
Ask questions
This is being recorded
Shout questions as you think of them
Greg Logan February 15, 2018 2 / 22

6. Opencast Overview
What is Opencast, really?
Like an onion: It makes you cry
Greg Logan February 15, 2018 3 / 22

7. Opencast Overview
What is Opencast, really?
Like an onion: It makes you cry
A program, which runs inside of a container
Greg Logan February 15, 2018 3 / 22

8. Opencast Overview
What is Opencast, really?
Like an onion: It makes you cry
A program, which runs inside of a container
So it is a program, which runs on Linux, right?
Greg Logan February 15, 2018 3 / 22

9. Opencast Overview
What is Opencast, really?
Like an onion: It makes you cry
A program, which runs inside of a container
So it is a program, which runs on Linux, right?
Close
Greg Logan February 15, 2018 3 / 22

10. Containers
Who here has heard of Apache Tomcat?
Similar idea to what we are using, but not quite the same
Missing a few bits that we will need later: OSGi Wiring
Greg Logan February 15, 2018 4 / 22

11. Containers
Who here has heard of Apache Tomcat?
Similar idea to what we are using, but not quite the same
Missing a few bits that we will need later: OSGi Wiring
We use Apache Karaf
Like Tomcat, on steroids
Is an OSGi container, among a bunch of other things
Greg Logan February 15, 2018 4 / 22

12. Containers
Who here has heard of Apache Tomcat?
Similar idea to what we are using, but not quite the same
Missing a few bits that we will need later: OSGi Wiring
We use Apache Karaf
Like Tomcat, on steroids
Is an OSGi container, among a bunch of other things
Karaf, like Tomcat, runs on Java, which runs on your OS
Greg Logan February 15, 2018 4 / 22

13. OSGi
Wait, what is OSGi?
Greg Logan February 15, 2018 5 / 22

14. OSGi
Wait, what is OSGi?
The OSGi technology is a set of specifications that define a dynamic
component system for Java. These specifications enable a development
model where applications are (dynamically) composed of many
different (reusable) components.
Greg Logan February 15, 2018 5 / 22

15. OSGi
Wait, what is OSGi?
The OSGi technology is a set of specifications that define a dynamic
component system for Java. These specifications enable a development
model where applications are (dynamically) composed of many
different (reusable) components.
How does this play into Opencast?
Greg Logan February 15, 2018 5 / 22

16. OSGi
Wait, what is OSGi?
The OSGi technology is a set of specifications that define a dynamic
component system for Java. These specifications enable a development
model where applications are (dynamically) composed of many
different (reusable) components.
How does this play into Opencast?
Because Opencast is not one program
Greg Logan February 15, 2018 5 / 22

17. OSGi
Wait, what is OSGi?
The OSGi technology is a set of specifications that define a dynamic
component system for Java. These specifications enable a development
model where applications are (dynamically) composed of many
different (reusable) components.
How does this play into Opencast?
Because Opencast is not one program
Opencast does not really exist!
Greg Logan February 15, 2018 5 / 22

18. OSGi Architecture: Bundles
OSGI lets a bunch of modules work together
Greg Logan February 15, 2018 6 / 22

19. OSGi Architecture: Bundles
OSGI lets a bunch of modules work together
Inter-module communication is done via Java Interfaces
Defines a set of functions available outside the module
Greg Logan February 15, 2018 6 / 22

20. OSGi Architecture: Bundles
OSGI lets a bunch of modules work together
Inter-module communication is done via Java Interfaces
Defines a set of functions available outside the module
A module can can export 0 to N Java packages
A module can define a number of service components
A module is actually called a bundle
Greg Logan February 15, 2018 6 / 22

21. OSGi Architecture: Service Components
Inside a bundle are zero to N service components
Greg Logan February 15, 2018 7 / 22

22. OSGi Architecture: Service Components
Inside a bundle are zero to N service components
These map to your Opencast services
Composer service, Workflow service, etc
Greg Logan February 15, 2018 7 / 22

23. OSGi Architecture: Service Components
Inside a bundle are zero to N service components
These map to your Opencast services
Composer service, Workflow service, etc
These are what do the work
These services can depend on zero to N other services
Greg Logan February 15, 2018 7 / 22

24. OSGi Architecture: Service Components
Inside a bundle are zero to N service components
These map to your Opencast services
Composer service, Workflow service, etc
These are what do the work
These services can depend on zero to N other services
Services and Components are not the same thing, but for our
purposes they might as well be
Greg Logan February 15, 2018 7 / 22

25. OSGi Architecture: Services Components
Services...
depend on other services
import other Java classes
Greg Logan February 15, 2018 8 / 22

26. OSGi Architecture: Services Components
Services...
depend on other services
import other Java classes
Versioning
both dependencies and packages have versions attached to them
these can be strict, or loose
Greg Logan February 15, 2018 8 / 22

27. OSGi Architecture: Services Components
Services...
depend on other services
import other Java classes
Versioning
both dependencies and packages have versions attached to them
these can be strict, or loose
Imports must be present for the service to load
Dependencies can be hot swapped
Dependencies can also be one to many
Greg Logan February 15, 2018 8 / 22

28. OSGi and Maven
How do we go from code, to bundles?
Greg Logan February 15, 2018 9 / 22

29. OSGi and Maven
How do we go from code, to bundles?
Maven!
Like Ant, Gradle, or NPM
Greg Logan February 15, 2018 9 / 22

30. OSGi and Maven
How do we go from code, to bundles?
Maven!
Like Ant, Gradle, or NPM
Maven uses the pom.xml
Import-Package
Export-Package
Service-Component
Greg Logan February 15, 2018 9 / 22

31. OSGi Architecture: Resolution
How does a given service find its dependencies?
Greg Logan February 15, 2018 10 / 22

32. OSGi Architecture: Resolution
How does a given service find its dependencies?
Magic!
Greg Logan February 15, 2018 10 / 22

33. OSGi Architecture: Resolution
How does a given service find its dependencies?
Magic!
Not really, but it feels like it sometimes
OSGi has a service registry
OSGi handles version resolution
OSGi handles class resolution
OSGi handles wiring together dependencies
Greg Logan February 15, 2018 10 / 22

34. Onions
Remember how I said Opencast was like an onion?
Greg Logan February 15, 2018 11 / 22

35. Onions
Remember how I said Opencast was like an onion?
Who here has heard of the OSI model?
Greg Logan February 15, 2018 11 / 22

36. OSI Onions
Remember how I said Opencast was like an onion?
Who here has heard of the OSI model?
Layer OSI Opencast
7 Application Opencast
6 Presentation Components
5 Session Bundles
4 Transport OSGi bits
3 Network Karaf
2 Data Link Java
1 Physical Link The OS
Opencast : Karaf like $app : Tomcat
Greg Logan February 15, 2018 12 / 22

37. Opencast Modules
Opencast five types modules
Greg Logan February 15, 2018 13 / 22

38. Opencast Modules
Opencast five types modules
Libraries
APIs
Workflow Operations
Implementations
Remotes
Greg Logan February 15, 2018 13 / 22

39. Opencast Modules: Libraries
Opencast Libraries
Code which is used by other code, but does not stand on its own
Greg Logan February 15, 2018 14 / 22

40. Opencast Modules: Libraries
Opencast Libraries
Code which is used by other code, but does not stand on its own
The common, and kernel bundles come to mind
Provide utilities and bits used all over the codebase
Do not provide any useful work on their own
Example: Security modules, and the REST documentation generators
Greg Logan February 15, 2018 14 / 22

41. Opencast Modules: APIs
Opencast APIs
Define APIs used across bundles
Greg Logan February 15, 2018 15 / 22

42. Opencast Modules: APIs
Opencast APIs
Define APIs used across bundles
Sometimes include utility code specific to that API to make life easier
Greg Logan February 15, 2018 15 / 22

43. Opencast Modules: APIs
Opencast APIs
Define APIs used across bundles
Sometimes include utility code specific to that API to make life easier
These modules typically end in -api. Eg: workflow-service-api
Greg Logan February 15, 2018 15 / 22

44. Opencast Modules: Workflow Operations
Opencast Workflow Operations
Define the various Workflow Operation Handlers
Greg Logan February 15, 2018 16 / 22

45. Opencast Modules: Workflow Operations
Opencast Workflow Operations
Define the various Workflow Operation Handlers
These handle a given workflow operation in your workflow
Greg Logan February 15, 2018 16 / 22

46. Opencast Modules: Workflow Operations
Opencast Workflow Operations
Define the various Workflow Operation Handlers
These handle a given workflow operation in your workflow
These are service references, with a tiny bit of glue
Take everything with with a source flavour
Give them to the silence detection service
if results longer than 0, attach to workflow, done
else, done
Greg Logan February 15, 2018 16 / 22

47. Opencast Modules: Implementations
Opencast Implementations
Greg Logan February 15, 2018 17 / 22

48. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
Greg Logan February 15, 2018 17 / 22

49. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Greg Logan February 15, 2018 17 / 22

50. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Typically named -impl. Eg: workflow-service-impl
Greg Logan February 15, 2018 17 / 22

51. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Typically named -impl. Eg: workflow-service-impl
... but not always: serviceregistry
Greg Logan February 15, 2018 17 / 22

52. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Typically named -impl. Eg: workflow-service-impl
... but not always: serviceregistry
Sometimes there are even multiple implementations!
Greg Logan February 15, 2018 17 / 22

53. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Typically named -impl. Eg: workflow-service-impl
... but not always: serviceregistry
Sometimes there are even multiple implementations!
distribution-service-download vs distribution-service-aws-s3
Greg Logan February 15, 2018 17 / 22

54. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Typically named -impl. Eg: workflow-service-impl
... but not always: serviceregistry
Sometimes there are even multiple implementations!
distribution-service-download vs distribution-service-aws-s3
Coexistence requirements: no conflicting REST endpoints
Greg Logan February 15, 2018 17 / 22

55. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Typically named -impl. Eg: workflow-service-impl
... but not always: serviceregistry
Sometimes there are even multiple implementations!
distribution-service-download vs distribution-service-aws-s3
Coexistence requirements: no conflicting REST endpoints
Fun times: they are all allowed to export the same packages and
services
Greg Logan February 15, 2018 17 / 22

56. Opencast Modules: Implementations
Opencast Implementations
Provide implementations of the above APIs
These are what does the work in Opencast
Typically named -impl. Eg: workflow-service-impl
... but not always: serviceregistry
Sometimes there are even multiple implementations!
distribution-service-download vs distribution-service-aws-s3
Coexistence requirements: no conflicting REST endpoints
Fun times: they are all allowed to export the same packages and
services
Determining the correct version to import involves OSGi filters
Greg Logan February 15, 2018 17 / 22

57. Bags of onions: Opencast in a cluster
If each Opencast server is an onion, how do they talk to eachother?
Greg Logan February 15, 2018 18 / 22

58. Bags of onions: Opencast in a cluster
If each Opencast server is an onion, how do they talk to eachother?
Your admin server does not have the code to run encodes(*)
Greg Logan February 15, 2018 18 / 22

59. Bags of onions: Opencast in a cluster
If each Opencast server is an onion, how do they talk to eachother?
Your admin server does not have the code to run encodes(*)
Wait, but OSGi wires the service dependencies together...
Greg Logan February 15, 2018 18 / 22

60. Bags of onions: Opencast in a cluster
If each Opencast server is an onion, how do they talk to eachother?
Your admin server does not have the code to run encodes(*)
Wait, but OSGi wires the service dependencies together...
How does the admin encode?
Greg Logan February 15, 2018 18 / 22

61. Bags of onions: Opencast in a cluster
If each Opencast server is an onion, how do they talk to eachother?
Your admin server does not have the code to run encodes(*)
Wait, but OSGi wires the service dependencies together...
How does the admin encode?
It must use a shared registry across the nodes, right?
OSGi can do this natively, but we do not use that
Greg Logan February 15, 2018 18 / 22

62. Bags of onions: Opencast in a cluster
If each Opencast server is an onion, how do they talk to eachother?
Your admin server does not have the code to run encodes(*)
Wait, but OSGi wires the service dependencies together...
How does the admin encode?
It must use a shared registry across the nodes, right?
OSGi can do this natively, but we do not use that
Opencast uses remote proxies
Greg Logan February 15, 2018 18 / 22

63. Opencast Modules: Remotes
Opencast Remotes
Register themselves as the real thing
Instead of doing the work themselves they send the data to a real impl
This whole process is hidden from the caller
Greg Logan February 15, 2018 19 / 22

64. Opencast Modules: Remotes
Opencast Remotes
Register themselves as the real thing
Instead of doing the work themselves they send the data to a real impl
This whole process is hidden from the caller
How does the remote know who to talk to?!
It uses the Opencast service registry!
When a service starts up it registers with the service registry
This is why the first support question is ’what are you db settings’
Greg Logan February 15, 2018 19 / 22

65. Opencast: Service Registry
The service registers itself with the service registry
But the service registry lives in one place, right?
Greg Logan February 15, 2018 20 / 22

66. Opencast: Service Registry
The service registers itself with the service registry
But the service registry lives in one place, right?
Wrong
Greg Logan February 15, 2018 20 / 22

67. Opencast: Service Registry
The service registers itself with the service registry
But the service registry lives in one place, right?
Wrong
Each node has its own service registry
Shared database allows information about services to pass through
the system
Job dispatching is only done from one node
For the sake of sanity, this is the admin node
Greg Logan February 15, 2018 20 / 22

68. Opencast: Service Registry
The service registers itself with the service registry
But the service registry lives in one place, right?
Wrong
Each node has its own service registry
Shared database allows information about services to pass through
the system
Job dispatching is only done from one node
For the sake of sanity, this is the admin node
It does not have to be...
Please do not do this
Greg Logan February 15, 2018 20 / 22

69. Opencast: Profiles
Given what we know, how do we assemble Opencast?
Greg Logan February 15, 2018 21 / 22

70. Opencast: Profiles
Given what we know, how do we assemble Opencast?
Opencast profiles live in the assemblies
Specifically, feature.xml in the the karaf-features assembly
Greg Logan February 15, 2018 21 / 22

71. Opencast: Profiles
Given what we know, how do we assemble Opencast?
Opencast profiles live in the assemblies
Specifically, feature.xml in the the karaf-features assembly
Each profile is a list of bundles to include
These lists are absolute
Not guaranteed to work, but Karaf will do its best
Easy(*) to modify
Greg Logan February 15, 2018 21 / 22

72. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Greg Logan February 15, 2018 22 / 22

73. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Maven builds the sevices and bundles
Greg Logan February 15, 2018 22 / 22

74. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Maven builds the sevices and bundles
Karaf wires things together so they work
Greg Logan February 15, 2018 22 / 22

75. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Maven builds the sevices and bundles
Karaf wires things together so they work
Opencast handles the inter-node communication
Greg Logan February 15, 2018 22 / 22

76. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Maven builds the sevices and bundles
Karaf wires things together so they work
Opencast handles the inter-node communication
There are a lot of moving parts, do not try to understand it all
at once
Greg Logan February 15, 2018 22 / 22

77. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Maven builds the sevices and bundles
Karaf wires things together so they work
Opencast handles the inter-node communication
There are a lot of moving parts, do not try to understand it all
at once
Ask questions if you are lost, confused, or having a tough time
Greg Logan February 15, 2018 22 / 22

78. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Maven builds the sevices and bundles
Karaf wires things together so they work
Opencast handles the inter-node communication
There are a lot of moving parts, do not try to understand it all
at once
Ask questions if you are lost, confused, or having a tough time
I would bet money that a lot of our developers do not really understand
all of the OSGi magic
Greg Logan February 15, 2018 22 / 22

79. Summary
Opencast is a bunch services inside of bundles inside of Karaf
Maven builds the sevices and bundles
Karaf wires things together so they work
Opencast handles the inter-node communication
There are a lot of moving parts, do not try to understand it all
at once
Ask questions if you are lost, confused, or having a tough time
I would bet money that a lot of our developers do not really understand
all of the OSGi magic
We do not generally need to right now
Greg Logan February 15, 2018 22 / 22