Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Opencast Architecture

150 views

Published on

An overview of how Opencast works, how it is built, and how it communicates within itself, and between nodes

Published in: Software
  • Here's How YOU Can Stake Out Your Personal Claim In Our EIGHT MILLION DOLLAR GOLDMINE... ■■■ https://bit.ly/2Ruzr8s
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Opencast Architecture

  1. 1. Opencast Architecture Greg Logan gregorydlogan@gmail.com February 15, 2018 Greg Logan February 15, 2018 1 / 22
  2. 2. Housekeeping This is going to be a deeply technical talk Greg Logan February 15, 2018 2 / 22
  3. 3. Housekeeping This is going to be a deeply technical talk If reality seems to be imploding... Feel free to zone out for a bit Ask questions Greg Logan February 15, 2018 2 / 22
  4. 4. Housekeeping This is going to be a deeply technical talk If reality seems to be imploding... Feel free to zone out for a bit Ask questions This is being recorded Greg Logan February 15, 2018 2 / 22
  5. 5. Housekeeping This is going to be a deeply technical talk If reality seems to be imploding... Feel free to zone out for a bit Ask questions This is being recorded Shout questions as you think of them Greg Logan February 15, 2018 2 / 22
  6. 6. Opencast Overview What is Opencast, really? Like an onion: It makes you cry Greg Logan February 15, 2018 3 / 22
  7. 7. Opencast Overview What is Opencast, really? Like an onion: It makes you cry A program, which runs inside of a container Greg Logan February 15, 2018 3 / 22
  8. 8. Opencast Overview What is Opencast, really? Like an onion: It makes you cry A program, which runs inside of a container So it is a program, which runs on Linux, right? Greg Logan February 15, 2018 3 / 22
  9. 9. Opencast Overview What is Opencast, really? Like an onion: It makes you cry A program, which runs inside of a container So it is a program, which runs on Linux, right? Close Greg Logan February 15, 2018 3 / 22
  10. 10. Containers Who here has heard of Apache Tomcat? Similar idea to what we are using, but not quite the same Missing a few bits that we will need later: OSGi Wiring Greg Logan February 15, 2018 4 / 22
  11. 11. Containers Who here has heard of Apache Tomcat? Similar idea to what we are using, but not quite the same Missing a few bits that we will need later: OSGi Wiring We use Apache Karaf Like Tomcat, on steroids Is an OSGi container, among a bunch of other things Greg Logan February 15, 2018 4 / 22
  12. 12. Containers Who here has heard of Apache Tomcat? Similar idea to what we are using, but not quite the same Missing a few bits that we will need later: OSGi Wiring We use Apache Karaf Like Tomcat, on steroids Is an OSGi container, among a bunch of other things Karaf, like Tomcat, runs on Java, which runs on your OS Greg Logan February 15, 2018 4 / 22
  13. 13. OSGi Wait, what is OSGi? Greg Logan February 15, 2018 5 / 22
  14. 14. OSGi Wait, what is OSGi? The OSGi technology is a set of specifications that define a dynamic component system for Java. These specifications enable a development model where applications are (dynamically) composed of many different (reusable) components. Greg Logan February 15, 2018 5 / 22
  15. 15. OSGi Wait, what is OSGi? The OSGi technology is a set of specifications that define a dynamic component system for Java. These specifications enable a development model where applications are (dynamically) composed of many different (reusable) components. How does this play into Opencast? Greg Logan February 15, 2018 5 / 22
  16. 16. OSGi Wait, what is OSGi? The OSGi technology is a set of specifications that define a dynamic component system for Java. These specifications enable a development model where applications are (dynamically) composed of many different (reusable) components. How does this play into Opencast? Because Opencast is not one program Greg Logan February 15, 2018 5 / 22
  17. 17. OSGi Wait, what is OSGi? The OSGi technology is a set of specifications that define a dynamic component system for Java. These specifications enable a development model where applications are (dynamically) composed of many different (reusable) components. How does this play into Opencast? Because Opencast is not one program Opencast does not really exist! Greg Logan February 15, 2018 5 / 22
  18. 18. OSGi Architecture: Bundles OSGI lets a bunch of modules work together Greg Logan February 15, 2018 6 / 22
  19. 19. OSGi Architecture: Bundles OSGI lets a bunch of modules work together Inter-module communication is done via Java Interfaces Defines a set of functions available outside the module Greg Logan February 15, 2018 6 / 22
  20. 20. OSGi Architecture: Bundles OSGI lets a bunch of modules work together Inter-module communication is done via Java Interfaces Defines a set of functions available outside the module A module can can export 0 to N Java packages A module can define a number of service components A module is actually called a bundle Greg Logan February 15, 2018 6 / 22
  21. 21. OSGi Architecture: Service Components Inside a bundle are zero to N service components Greg Logan February 15, 2018 7 / 22
  22. 22. OSGi Architecture: Service Components Inside a bundle are zero to N service components These map to your Opencast services Composer service, Workflow service, etc Greg Logan February 15, 2018 7 / 22
  23. 23. OSGi Architecture: Service Components Inside a bundle are zero to N service components These map to your Opencast services Composer service, Workflow service, etc These are what do the work These services can depend on zero to N other services Greg Logan February 15, 2018 7 / 22
  24. 24. OSGi Architecture: Service Components Inside a bundle are zero to N service components These map to your Opencast services Composer service, Workflow service, etc These are what do the work These services can depend on zero to N other services Services and Components are not the same thing, but for our purposes they might as well be Greg Logan February 15, 2018 7 / 22
  25. 25. OSGi Architecture: Services Components Services... depend on other services import other Java classes Greg Logan February 15, 2018 8 / 22
  26. 26. OSGi Architecture: Services Components Services... depend on other services import other Java classes Versioning both dependencies and packages have versions attached to them these can be strict, or loose Greg Logan February 15, 2018 8 / 22
  27. 27. OSGi Architecture: Services Components Services... depend on other services import other Java classes Versioning both dependencies and packages have versions attached to them these can be strict, or loose Imports must be present for the service to load Dependencies can be hot swapped Dependencies can also be one to many Greg Logan February 15, 2018 8 / 22
  28. 28. OSGi and Maven How do we go from code, to bundles? Greg Logan February 15, 2018 9 / 22
  29. 29. OSGi and Maven How do we go from code, to bundles? Maven! Like Ant, Gradle, or NPM Greg Logan February 15, 2018 9 / 22
  30. 30. OSGi and Maven How do we go from code, to bundles? Maven! Like Ant, Gradle, or NPM Maven uses the pom.xml Import-Package Export-Package Service-Component Greg Logan February 15, 2018 9 / 22
  31. 31. OSGi Architecture: Resolution How does a given service find its dependencies? Greg Logan February 15, 2018 10 / 22
  32. 32. OSGi Architecture: Resolution How does a given service find its dependencies? Magic! Greg Logan February 15, 2018 10 / 22
  33. 33. OSGi Architecture: Resolution How does a given service find its dependencies? Magic! Not really, but it feels like it sometimes OSGi has a service registry OSGi handles version resolution OSGi handles class resolution OSGi handles wiring together dependencies Greg Logan February 15, 2018 10 / 22
  34. 34. Onions Remember how I said Opencast was like an onion? Greg Logan February 15, 2018 11 / 22
  35. 35. Onions Remember how I said Opencast was like an onion? Who here has heard of the OSI model? Greg Logan February 15, 2018 11 / 22
  36. 36. OSI Onions Remember how I said Opencast was like an onion? Who here has heard of the OSI model? Layer OSI Opencast 7 Application Opencast 6 Presentation Components 5 Session Bundles 4 Transport OSGi bits 3 Network Karaf 2 Data Link Java 1 Physical Link The OS Opencast : Karaf like $app : Tomcat Greg Logan February 15, 2018 12 / 22
  37. 37. Opencast Modules Opencast five types modules Greg Logan February 15, 2018 13 / 22
  38. 38. Opencast Modules Opencast five types modules Libraries APIs Workflow Operations Implementations Remotes Greg Logan February 15, 2018 13 / 22
  39. 39. Opencast Modules: Libraries Opencast Libraries Code which is used by other code, but does not stand on its own Greg Logan February 15, 2018 14 / 22
  40. 40. Opencast Modules: Libraries Opencast Libraries Code which is used by other code, but does not stand on its own The common, and kernel bundles come to mind Provide utilities and bits used all over the codebase Do not provide any useful work on their own Example: Security modules, and the REST documentation generators Greg Logan February 15, 2018 14 / 22
  41. 41. Opencast Modules: APIs Opencast APIs Define APIs used across bundles Greg Logan February 15, 2018 15 / 22
  42. 42. Opencast Modules: APIs Opencast APIs Define APIs used across bundles Sometimes include utility code specific to that API to make life easier Greg Logan February 15, 2018 15 / 22
  43. 43. Opencast Modules: APIs Opencast APIs Define APIs used across bundles Sometimes include utility code specific to that API to make life easier These modules typically end in -api. Eg: workflow-service-api Greg Logan February 15, 2018 15 / 22
  44. 44. Opencast Modules: Workflow Operations Opencast Workflow Operations Define the various Workflow Operation Handlers Greg Logan February 15, 2018 16 / 22
  45. 45. Opencast Modules: Workflow Operations Opencast Workflow Operations Define the various Workflow Operation Handlers These handle a given workflow operation in your workflow Greg Logan February 15, 2018 16 / 22
  46. 46. Opencast Modules: Workflow Operations Opencast Workflow Operations Define the various Workflow Operation Handlers These handle a given workflow operation in your workflow These are service references, with a tiny bit of glue Take everything with with a source flavour Give them to the silence detection service if results longer than 0, attach to workflow, done else, done Greg Logan February 15, 2018 16 / 22
  47. 47. Opencast Modules: Implementations Opencast Implementations Greg Logan February 15, 2018 17 / 22
  48. 48. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs Greg Logan February 15, 2018 17 / 22
  49. 49. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Greg Logan February 15, 2018 17 / 22
  50. 50. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Typically named -impl. Eg: workflow-service-impl Greg Logan February 15, 2018 17 / 22
  51. 51. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Typically named -impl. Eg: workflow-service-impl ... but not always: serviceregistry Greg Logan February 15, 2018 17 / 22
  52. 52. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Typically named -impl. Eg: workflow-service-impl ... but not always: serviceregistry Sometimes there are even multiple implementations! Greg Logan February 15, 2018 17 / 22
  53. 53. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Typically named -impl. Eg: workflow-service-impl ... but not always: serviceregistry Sometimes there are even multiple implementations! distribution-service-download vs distribution-service-aws-s3 Greg Logan February 15, 2018 17 / 22
  54. 54. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Typically named -impl. Eg: workflow-service-impl ... but not always: serviceregistry Sometimes there are even multiple implementations! distribution-service-download vs distribution-service-aws-s3 Coexistence requirements: no conflicting REST endpoints Greg Logan February 15, 2018 17 / 22
  55. 55. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Typically named -impl. Eg: workflow-service-impl ... but not always: serviceregistry Sometimes there are even multiple implementations! distribution-service-download vs distribution-service-aws-s3 Coexistence requirements: no conflicting REST endpoints Fun times: they are all allowed to export the same packages and services Greg Logan February 15, 2018 17 / 22
  56. 56. Opencast Modules: Implementations Opencast Implementations Provide implementations of the above APIs These are what does the work in Opencast Typically named -impl. Eg: workflow-service-impl ... but not always: serviceregistry Sometimes there are even multiple implementations! distribution-service-download vs distribution-service-aws-s3 Coexistence requirements: no conflicting REST endpoints Fun times: they are all allowed to export the same packages and services Determining the correct version to import involves OSGi filters Greg Logan February 15, 2018 17 / 22
  57. 57. Bags of onions: Opencast in a cluster If each Opencast server is an onion, how do they talk to eachother? Greg Logan February 15, 2018 18 / 22
  58. 58. Bags of onions: Opencast in a cluster If each Opencast server is an onion, how do they talk to eachother? Your admin server does not have the code to run encodes(*) Greg Logan February 15, 2018 18 / 22
  59. 59. Bags of onions: Opencast in a cluster If each Opencast server is an onion, how do they talk to eachother? Your admin server does not have the code to run encodes(*) Wait, but OSGi wires the service dependencies together... Greg Logan February 15, 2018 18 / 22
  60. 60. Bags of onions: Opencast in a cluster If each Opencast server is an onion, how do they talk to eachother? Your admin server does not have the code to run encodes(*) Wait, but OSGi wires the service dependencies together... How does the admin encode? Greg Logan February 15, 2018 18 / 22
  61. 61. Bags of onions: Opencast in a cluster If each Opencast server is an onion, how do they talk to eachother? Your admin server does not have the code to run encodes(*) Wait, but OSGi wires the service dependencies together... How does the admin encode? It must use a shared registry across the nodes, right? OSGi can do this natively, but we do not use that Greg Logan February 15, 2018 18 / 22
  62. 62. Bags of onions: Opencast in a cluster If each Opencast server is an onion, how do they talk to eachother? Your admin server does not have the code to run encodes(*) Wait, but OSGi wires the service dependencies together... How does the admin encode? It must use a shared registry across the nodes, right? OSGi can do this natively, but we do not use that Opencast uses remote proxies Greg Logan February 15, 2018 18 / 22
  63. 63. Opencast Modules: Remotes Opencast Remotes Register themselves as the real thing Instead of doing the work themselves they send the data to a real impl This whole process is hidden from the caller Greg Logan February 15, 2018 19 / 22
  64. 64. Opencast Modules: Remotes Opencast Remotes Register themselves as the real thing Instead of doing the work themselves they send the data to a real impl This whole process is hidden from the caller How does the remote know who to talk to?! It uses the Opencast service registry! When a service starts up it registers with the service registry This is why the first support question is ’what are you db settings’ Greg Logan February 15, 2018 19 / 22
  65. 65. Opencast: Service Registry The service registers itself with the service registry But the service registry lives in one place, right? Greg Logan February 15, 2018 20 / 22
  66. 66. Opencast: Service Registry The service registers itself with the service registry But the service registry lives in one place, right? Wrong Greg Logan February 15, 2018 20 / 22
  67. 67. Opencast: Service Registry The service registers itself with the service registry But the service registry lives in one place, right? Wrong Each node has its own service registry Shared database allows information about services to pass through the system Job dispatching is only done from one node For the sake of sanity, this is the admin node Greg Logan February 15, 2018 20 / 22
  68. 68. Opencast: Service Registry The service registers itself with the service registry But the service registry lives in one place, right? Wrong Each node has its own service registry Shared database allows information about services to pass through the system Job dispatching is only done from one node For the sake of sanity, this is the admin node It does not have to be... Please do not do this Greg Logan February 15, 2018 20 / 22
  69. 69. Opencast: Profiles Given what we know, how do we assemble Opencast? Greg Logan February 15, 2018 21 / 22
  70. 70. Opencast: Profiles Given what we know, how do we assemble Opencast? Opencast profiles live in the assemblies Specifically, feature.xml in the the karaf-features assembly Greg Logan February 15, 2018 21 / 22
  71. 71. Opencast: Profiles Given what we know, how do we assemble Opencast? Opencast profiles live in the assemblies Specifically, feature.xml in the the karaf-features assembly Each profile is a list of bundles to include These lists are absolute Not guaranteed to work, but Karaf will do its best Easy(*) to modify Greg Logan February 15, 2018 21 / 22
  72. 72. Summary Opencast is a bunch services inside of bundles inside of Karaf Greg Logan February 15, 2018 22 / 22
  73. 73. Summary Opencast is a bunch services inside of bundles inside of Karaf Maven builds the sevices and bundles Greg Logan February 15, 2018 22 / 22
  74. 74. Summary Opencast is a bunch services inside of bundles inside of Karaf Maven builds the sevices and bundles Karaf wires things together so they work Greg Logan February 15, 2018 22 / 22
  75. 75. Summary Opencast is a bunch services inside of bundles inside of Karaf Maven builds the sevices and bundles Karaf wires things together so they work Opencast handles the inter-node communication Greg Logan February 15, 2018 22 / 22
  76. 76. Summary Opencast is a bunch services inside of bundles inside of Karaf Maven builds the sevices and bundles Karaf wires things together so they work Opencast handles the inter-node communication There are a lot of moving parts, do not try to understand it all at once Greg Logan February 15, 2018 22 / 22
  77. 77. Summary Opencast is a bunch services inside of bundles inside of Karaf Maven builds the sevices and bundles Karaf wires things together so they work Opencast handles the inter-node communication There are a lot of moving parts, do not try to understand it all at once Ask questions if you are lost, confused, or having a tough time Greg Logan February 15, 2018 22 / 22
  78. 78. Summary Opencast is a bunch services inside of bundles inside of Karaf Maven builds the sevices and bundles Karaf wires things together so they work Opencast handles the inter-node communication There are a lot of moving parts, do not try to understand it all at once Ask questions if you are lost, confused, or having a tough time I would bet money that a lot of our developers do not really understand all of the OSGi magic Greg Logan February 15, 2018 22 / 22
  79. 79. Summary Opencast is a bunch services inside of bundles inside of Karaf Maven builds the sevices and bundles Karaf wires things together so they work Opencast handles the inter-node communication There are a lot of moving parts, do not try to understand it all at once Ask questions if you are lost, confused, or having a tough time I would bet money that a lot of our developers do not really understand all of the OSGi magic We do not generally need to right now Greg Logan February 15, 2018 22 / 22

×