LM
Uploaded byLorenzo Miniero
PDF, PPTX1,412 views

Janus/HOMER/HEPIC @ OpenSIPS18

The document discusses troubleshooting and monitoring the Janus WebRTC server, focusing on event handling mechanisms such as admin APIs and custom event handlers. It highlights the advantages of various event handler implementations and challenges in processing and correlating events to improve monitoring and analysis. The presentation also addresses integration with tools like Homer and potential improvements for event handling through MQTT.

Embed presentation

Download as PDF, PPTX
Troubleshooting and monitoring Janus: a HEPIC journey! Lorenzo Miniero @elminiero OpenSIPS Summit 2nd May 2018,
“Which Lorenzo are you, exactly?”
You may remember me from “Revenge of the Mutant Janus Instances” (not really...)
No, seriously, remember Janus? • General purpose, open source WebRTC server • https://github.com/meetecho/janus-gateway • Demos and documentation: https://janus.conf.meetecho.com • Community: https://groups.google.com/forum/#!forum/meetecho-janus Past OpenSIPS presentations: on Janus, SIP, and various salads • https://www.youtube.com/watch?v=SFeWYewoL7Q • https://www.youtube.com/watch?v=anmyMC6Ovl8&t=25112s
No, seriously, remember Janus? • General purpose, open source WebRTC server • https://github.com/meetecho/janus-gateway • Demos and documentation: https://janus.conf.meetecho.com • Community: https://groups.google.com/forum/#!forum/meetecho-janus Past OpenSIPS presentations: on Janus, SIP, and various salads • https://www.youtube.com/watch?v=SFeWYewoL7Q • https://www.youtube.com/watch?v=anmyMC6Ovl8&t=25112s
Troubleshooting Janus: a bit of history • First approach (still widely used) was the Admin API • Request/response protocol available on different transports • Allows to inspect handles and WebRTC “internals” from the Janus perspective • Can tweak some settings too (e.g., enable/disable debugging) • A different, asynchronous mechanism then followed: Event Handlers • Core and plugins generate events of different types • Custom modules can subscribe to and handle them • e.g., save to DB, send to external service, CDR, etc. Fun fact: Event Handlers were conceived during OpenSIPS 2016! • You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that
Troubleshooting Janus: a bit of history • First approach (still widely used) was the Admin API • Request/response protocol available on different transports • Allows to inspect handles and WebRTC “internals” from the Janus perspective • Can tweak some settings too (e.g., enable/disable debugging) • A different, asynchronous mechanism then followed: Event Handlers • Core and plugins generate events of different types • Custom modules can subscribe to and handle them • e.g., save to DB, send to external service, CDR, etc. Fun fact: Event Handlers were conceived during OpenSIPS 2016! • You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that
Troubleshooting Janus: a bit of history • First approach (still widely used) was the Admin API • Request/response protocol available on different transports • Allows to inspect handles and WebRTC “internals” from the Janus perspective • Can tweak some settings too (e.g., enable/disable debugging) • A different, asynchronous mechanism then followed: Event Handlers • Core and plugins generate events of different types • Custom modules can subscribe to and handle them • e.g., save to DB, send to external service, CDR, etc. Fun fact: Event Handlers were conceived during OpenSIPS 2016! • You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that
Routing and managing asynchronous events
“Sample Event Handler”: HTTP as a notifier • Simply forwards all events as JSON to an HTTP backend • Supports basic authentication • Can group events (i.e., JSON array vs. multiple JSON objects) • Implements basic retransmissions (exponential back-off) • Does nothing more than that: logic needs to be elsewhere • HTTP backend decides what to do with events, if anything • Behaviour can be tweaked via Admin API calls Need something else? Check the alternatives or write your own! A few other event handlers also available • RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)
“Sample Event Handler”: HTTP as a notifier • Simply forwards all events as JSON to an HTTP backend • Supports basic authentication • Can group events (i.e., JSON array vs. multiple JSON objects) • Implements basic retransmissions (exponential back-off) • Does nothing more than that: logic needs to be elsewhere • HTTP backend decides what to do with events, if anything • Behaviour can be tweaked via Admin API calls Need something else? Check the alternatives or write your own! A few other event handlers also available • RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)
“Sample Event Handler”: HTTP as a notifier • Simply forwards all events as JSON to an HTTP backend • Supports basic authentication • Can group events (i.e., JSON array vs. multiple JSON objects) • Implements basic retransmissions (exponential back-off) • Does nothing more than that: logic needs to be elsewhere • HTTP backend decides what to do with events, if anything • Behaviour can be tweaked via Admin API calls Need something else? Check the alternatives or write your own! A few other event handlers also available • RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)
How do we handle events, now? • Generating events is easy, evaluating them is another matter... • Event Handlers typically just relay events, and don’t do processing themselves • Analyzing and correlating tons of events is complicated • A few different approaches, from easiest to trickiest 1 https://github.com/stirlab/janus-event-server (dumping events to file) 2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite) 3 Write your own backend, e.g., http://www.meetecho.com/blog/event-handlers-a-practical-example/ http://www.meetecho.com/blog/correlating-janus-event-handlers/ • A much better approach: trust the smart guys! • http://www.sipcapture.org/ • http://hepic.tel/
How do we handle events, now? • Generating events is easy, evaluating them is another matter... • Event Handlers typically just relay events, and don’t do processing themselves • Analyzing and correlating tons of events is complicated • A few different approaches, from easiest to trickiest 1 https://github.com/stirlab/janus-event-server (dumping events to file) 2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite) 3 Write your own backend, e.g., http://www.meetecho.com/blog/event-handlers-a-practical-example/ http://www.meetecho.com/blog/correlating-janus-event-handlers/ • A much better approach: trust the smart guys! • http://www.sipcapture.org/ • http://hepic.tel/
How do we handle events, now? • Generating events is easy, evaluating them is another matter... • Event Handlers typically just relay events, and don’t do processing themselves • Analyzing and correlating tons of events is complicated • A few different approaches, from easiest to trickiest 1 https://github.com/stirlab/janus-event-server (dumping events to file) 2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite) 3 Write your own backend, e.g., http://www.meetecho.com/blog/event-handlers-a-practical-example/ http://www.meetecho.com/blog/correlating-janus-event-handlers/ • A much better approach: trust the smart guys! • http://www.sipcapture.org/ • http://hepic.tel/
Event Handlers in HOMER since “day one”(*)
(*A trip down Memory Lane: FOSDEM 2017!) https://archive.fosdem.org/2017/schedule/event/janus/ https://archive.fosdem.org/2017/schedule/event/homer/
From SIP to WebRTC • As anticipated, Event Handlers were conceived in a chat with QXIP • “How can I dump SIP messages with Sofia SIP?” • “Which other events would be useful to have?” • “What about a new modular approach, where HEP could be a plugin?” • The only “caveat”: it was SIP only • Only events coming from the SIP plugin were tracked, to follow the signalling • Other events related to SIP handles only saved as “related info” • Still very used and widespread, but limited to WebRTC/SIP scenarios • Lately, big step to start tracking generic WebRTC applications instead • How to handle users with more than one PeerConnection? • How to relate a PeerConnection to another, for issues and more? • How to reconstruct a session topology from the media perspective?
From SIP to WebRTC • As anticipated, Event Handlers were conceived in a chat with QXIP • “How can I dump SIP messages with Sofia SIP?” • “Which other events would be useful to have?” • “What about a new modular approach, where HEP could be a plugin?” • The only “caveat”: it was SIP only • Only events coming from the SIP plugin were tracked, to follow the signalling • Other events related to SIP handles only saved as “related info” • Still very used and widespread, but limited to WebRTC/SIP scenarios • Lately, big step to start tracking generic WebRTC applications instead • How to handle users with more than one PeerConnection? • How to relate a PeerConnection to another, for issues and more? • How to reconstruct a session topology from the media perspective?
From SIP to WebRTC • As anticipated, Event Handlers were conceived in a chat with QXIP • “How can I dump SIP messages with Sofia SIP?” • “Which other events would be useful to have?” • “What about a new modular approach, where HEP could be a plugin?” • The only “caveat”: it was SIP only • Only events coming from the SIP plugin were tracked, to follow the signalling • Other events related to SIP handles only saved as “related info” • Still very used and widespread, but limited to WebRTC/SIP scenarios • Lately, big step to start tracking generic WebRTC applications instead • How to handle users with more than one PeerConnection? • How to relate a PeerConnection to another, for issues and more? • How to reconstruct a session topology from the media perspective?
Challenge #1: getting events in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
Challenge #1: getting events in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
Challenge #1: getting events in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
Challenge #1: getting events in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
Challenge #2: storing and processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
Challenge #2: storing and processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
Challenge #2: storing and processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
Challenge #2: storing and processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
Challenge #3: correlating events • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
Challenge #3: correlating events • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
Challenge #3: correlating events • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
Challenge #3: correlating events • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
Challenge #3: storing and processing events
Challenge #3: storing and processing events
Challenge #3: storing and processing events
Challenge #4: client-side events • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
Challenge #4: client-side events • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
Challenge #4: client-side events • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
Challenge #4: client-side events • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
A real use case: IETF meetings!
Remote participation at the IETF https://www.slideshare.net/LorenzoMiniero/ietf-remote-participation-via-meetecho-webrtc-meetup-stockholm https://www.vuc.me/2017/vuc640-ietf-remote-participation-with-lorenzo-miniero/
Studying data from IETF 101 in London • IETF remote participation based on multiple Janus plugins • SIP plugin for mixed audio • Streaming plugin for static sources • VideoRoom plugin for remote speakers • Collected events related to 10 different Working Group sessions • https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000 • MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING, IPSECME, LAMPS • Good mix of heterogeneous sessions • Some shorter, some longer (DETNET lasted 5 hours!) • Some had many active remote speakers • All had attendees from all over the world (different network conditions)
Studying data from IETF 101 in London • IETF remote participation based on multiple Janus plugins • SIP plugin for mixed audio • Streaming plugin for static sources • VideoRoom plugin for remote speakers • Collected events related to 10 different Working Group sessions • https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000 • MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING, IPSECME, LAMPS • Good mix of heterogeneous sessions • Some shorter, some longer (DETNET lasted 5 hours!) • Some had many active remote speakers • All had attendees from all over the world (different network conditions)
Studying data from IETF 101 in London • IETF remote participation based on multiple Janus plugins • SIP plugin for mixed audio • Streaming plugin for static sources • VideoRoom plugin for remote speakers • Collected events related to 10 different Working Group sessions • https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000 • MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING, IPSECME, LAMPS • Good mix of heterogeneous sessions • Some shorter, some longer (DETNET lasted 5 hours!) • Some had many active remote speakers • All had attendees from all over the world (different network conditions)
First experiments: dumping events to Postgres • Postgres supports JSON natively, and is used by HOMER/HEPIC too • Why not play with this locally, first? • Created a single database, and a different table per each WG • Each table just has a unique index and the JSON value • All queries can be done on JSON fields • Played with different queries to identify different events • How many concurrent participants? How many were active? • Who subscribed to what? (SIP, slides, camera, remotees) • Any network errors? If so, caused by what? • What can we extract from the statistics?
First experiments: dumping events to Postgres • Postgres supports JSON natively, and is used by HOMER/HEPIC too • Why not play with this locally, first? • Created a single database, and a different table per each WG • Each table just has a unique index and the JSON value • All queries can be done on JSON fields • Played with different queries to identify different events • How many concurrent participants? How many were active? • Who subscribed to what? (SIP, slides, camera, remotees) • Any network errors? If so, caused by what? • What can we extract from the statistics?
First experiments: dumping events to Postgres • Postgres supports JSON natively, and is used by HOMER/HEPIC too • Why not play with this locally, first? • Created a single database, and a different table per each WG • Each table just has a unique index and the JSON value • All queries can be done on JSON fields • Played with different queries to identify different events • How many concurrent participants? How many were active? • Who subscribed to what? (SIP, slides, camera, remotees) • Any network errors? If so, caused by what? • What can we extract from the statistics?
Summarizing relevant user/handle events
Summarizing relevant user/handle events
Summarizing relevant user/handle events
Summarizing relevant user/handle events
The ugliest charts you’ll ever see!
Some bugs to be fixed in Janus too, I guess... • Analysing dumps, one thing popped to the eye: RTCP RTT > 4M?! • RTT was either quite low, or super-high, never in the middle • Turned out to be a bug in the RTCP parsing code, though... (that we fixed, now!)
How does it look like in HOMER/HEPIC?
How does it look like in HOMER/HEPIC?
Cool demo with Lorenzo tomorrow!
Next steps • Collecting real data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
Next steps • Collecting real data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
Next steps • Collecting real data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
Next steps • Collecting real data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
Next steps • Collecting real data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
Thanks! Questions? Comments? Get in touch! • https://twitter.com/elminiero • https://twitter.com/meetecho • http://www.meetecho.com

More Related Content

PPTX
Ceph Introduction 2017
byKaran Singh
 
PDF
SIP/WebRTC load testing @ KamailioWorld 2017
byLorenzo Miniero
 
PPTX
Apache Arrow Flight Overview
byJacques Nadeau
 
PDF
SIP - The Basics
byJonas Borjesson
 
PDF
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
byDataWorks Summit
 
PPTX
Junos vs ios Troubleshooting comands
bysandeep kumar
 
PPT
Nfs
bytmavroidis
 
PDF
Apache Kafka - Martin Podval
byMartin Podval
 
Ceph Introduction 2017
byKaran Singh
 
SIP/WebRTC load testing @ KamailioWorld 2017
byLorenzo Miniero
 
Apache Arrow Flight Overview
byJacques Nadeau
 
SIP - The Basics
byJonas Borjesson
 
Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise
byDataWorks Summit
 
Junos vs ios Troubleshooting comands
bysandeep kumar
 
Nfs
bytmavroidis
 
Apache Kafka - Martin Podval
byMartin Podval
 

What's hot

PDF
Alfresco Backup and Disaster Recovery White Paper
byToni de la Fuente
 
PPTX
Introduction to DPDK
byKernel TLV
 
DOCX
SDN Training - Open daylight installation + example with mininet
bySAMeh Zaghloul
 
PPTX
Apache NiFi in the Hadoop Ecosystem
byDataWorks Summit/Hadoop Summit
 
PPTX
Alfresco tuning part1
byLuis Cabaceira
 
PDF
Janus SFU cascading @ IIT-RTC 2022
byLorenzo Miniero
 
PDF
Scaling WebRTC applications with Janus
byLorenzo Miniero
 
PPTX
BGP and Traffic Engineering with Akamai
byInternet Society
 
PDF
5 p9 pnor and open bmc overview - final
byYutaka Kawai
 
PPTX
The RabbitMQ Message Broker
byMartin Toshev
 
PDF
Launch the First Process in Linux System
byJian-Hong Pan
 
PDF
Consumer offset management in Kafka
byJoel Koshy
 
PPTX
Apache kafka
byViswanath J
 
PDF
美团技术团队 - KVM性能优化
by美团点评技术团队
 
PPTX
OpenvSwitch Deep Dive
byrajdeep
 
ODP
Stream processing using Kafka
byKnoldus Inc.
 
PDF
Intro to Telegraf
byInfluxData
 
PDF
Introduction to gRPC
byPrakash Divy
 
PDF
Java features. Java 8, 9, 10, 11
byIvelin Yanev
 
PPTX
Introduction to Apache Kafka
byJeff Holoman
 
Alfresco Backup and Disaster Recovery White Paper
byToni de la Fuente
 
Introduction to DPDK
byKernel TLV
 
SDN Training - Open daylight installation + example with mininet
bySAMeh Zaghloul
 
Apache NiFi in the Hadoop Ecosystem
byDataWorks Summit/Hadoop Summit
 
Alfresco tuning part1
byLuis Cabaceira
 
Janus SFU cascading @ IIT-RTC 2022
byLorenzo Miniero
 
Scaling WebRTC applications with Janus
byLorenzo Miniero
 
BGP and Traffic Engineering with Akamai
byInternet Society
 
5 p9 pnor and open bmc overview - final
byYutaka Kawai
 
The RabbitMQ Message Broker
byMartin Toshev
 
Launch the First Process in Linux System
byJian-Hong Pan
 
Consumer offset management in Kafka
byJoel Koshy
 
Apache kafka
byViswanath J
 
美团技术团队 - KVM性能优化
by美团点评技术团队
 
OpenvSwitch Deep Dive
byrajdeep
 
Stream processing using Kafka
byKnoldus Inc.
 
Intro to Telegraf
byInfluxData
 
Introduction to gRPC
byPrakash Divy
 
Java features. Java 8, 9, 10, 11
byIvelin Yanev
 
Introduction to Apache Kafka
byJeff Holoman
 

Similar to Janus/HOMER/HEPIC @ OpenSIPS18

PDF
FOSDEM2017 - Janus Event Handlers
byLorenzo Miniero
 
PDF
Server-Sent Events (real-time HTTP push for HTML5 browsers)
byyay w00t
 
PPTX
Multi-Process JavaScript Architectures
byMark Trostler
 
PDF
HTML5 Server Sent Events/JSF JAX 2011 Conference
byRoger Kitain
 
PDF
Building an Event-driven Web @ Impact
byJeff Lindsay
 
PDF
Unlocking Realtime Web Applications - 4Developers Katowice 2023
byPatryk Omiotek
 
PDF
APIs That Make Things Happen
byJeff Lindsay
 
PDF
Netty training
byJackson dos Santos Olveira
 
PDF
Server-Side Programming Primer
byIvano Malavolta
 
PDF
Reactive Hypermedia
byKevin Swiber
 
PDF
Netty training
byMarcelo Serpa
 
PDF
ServerSentEventsV2.pdf
byAlessandro Minoccheri
 
PDF
WebHooks in 10 Minutes
byJeff Lindsay
 
PPTX
Event and signal driven programming techniques
byElizabeth Smith
 
PPTX
Event and Signal Driven Programming Zendcon 2012
byElizabeth Smith
 
PPTX
Event and signal driven programming
byElizabeth Smith
 
PDF
Server-Sent Events in Action
byAndrei Rusu
 
PPTX
Research Topics in Machine Hypermedia
byMichael Koster
 
PDF
Server Sent Events
byMohanad Obaid
 
PDF
The Event Mesh: real-time, event-driven, responsive APIs and beyond
bySolace
 
FOSDEM2017 - Janus Event Handlers
byLorenzo Miniero
 
Server-Sent Events (real-time HTTP push for HTML5 browsers)
byyay w00t
 
Multi-Process JavaScript Architectures
byMark Trostler
 
HTML5 Server Sent Events/JSF JAX 2011 Conference
byRoger Kitain
 
Building an Event-driven Web @ Impact
byJeff Lindsay
 
Unlocking Realtime Web Applications - 4Developers Katowice 2023
byPatryk Omiotek
 
APIs That Make Things Happen
byJeff Lindsay
 
Netty training
byJackson dos Santos Olveira
 
Server-Side Programming Primer
byIvano Malavolta
 
Reactive Hypermedia
byKevin Swiber
 
Netty training
byMarcelo Serpa
 
ServerSentEventsV2.pdf
byAlessandro Minoccheri
 
WebHooks in 10 Minutes
byJeff Lindsay
 
Event and signal driven programming techniques
byElizabeth Smith
 
Event and Signal Driven Programming Zendcon 2012
byElizabeth Smith
 
Event and signal driven programming
byElizabeth Smith
 
Server-Sent Events in Action
byAndrei Rusu
 
Research Topics in Machine Hypermedia
byMichael Koster
 
Server Sent Events
byMohanad Obaid
 
The Event Mesh: real-time, event-driven, responsive APIs and beyond
bySolace
 

More from Lorenzo Miniero

PDF
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
byLorenzo Miniero
 
PDF
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
byLorenzo Miniero
 
PDF
WebRTC and QUIC: how hard can it be? @ RTC.ON 2024
byLorenzo Miniero
 
PDF
WebRTC and SIP not just audio and video @ OpenSIPS 2024
byLorenzo Miniero
 
PDF
SIP trunking in Janus @ Kamailio World 2024
byLorenzo Miniero
 
PDF
Getting AV1/SVC to work in the Janus WebRTC Server
byLorenzo Miniero
 
PDF
WebRTC Broadcasting @ TADSummit 2023
byLorenzo Miniero
 
PDF
BWE in Janus
byLorenzo Miniero
 
PDF
The challenges of hybrid meetings @ CommCon 2023
byLorenzo Miniero
 
PDF
Real-Time Text and WebRTC @ Kamailio World 2023
byLorenzo Miniero
 
PDF
Become a rockstar using FOSS!
byLorenzo Miniero
 
PDF
SIP transfer with Janus/WebRTC @ OpenSIPS 2022
byLorenzo Miniero
 
PDF
WHIP WebRTC Broadcasting @ FOSDEM 2022
byLorenzo Miniero
 
PDF
WebRTC, RED and Janus @ ClueCon21
byLorenzo Miniero
 
PDF
WHIP and Janus @ IIT-RTC 2021
byLorenzo Miniero
 
PDF
Write a SocialTV app @ OpenSIPS 2021
byLorenzo Miniero
 
PDF
Janus + Audio @ Open Source World
byLorenzo Miniero
 
PDF
JamRTC @ Wonder WebRTC unConference
byLorenzo Miniero
 
PDF
Scaling WebRTC deployments with multicast @ IETF 110 MBONED
byLorenzo Miniero
 
PDF
Janus Workshop pt.2 @ ClueCon 2021
byLorenzo Miniero
 
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
byLorenzo Miniero
 
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
byLorenzo Miniero
 
WebRTC and QUIC: how hard can it be? @ RTC.ON 2024
byLorenzo Miniero
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
byLorenzo Miniero
 
SIP trunking in Janus @ Kamailio World 2024
byLorenzo Miniero
 
Getting AV1/SVC to work in the Janus WebRTC Server
byLorenzo Miniero
 
WebRTC Broadcasting @ TADSummit 2023
byLorenzo Miniero
 
BWE in Janus
byLorenzo Miniero
 
The challenges of hybrid meetings @ CommCon 2023
byLorenzo Miniero
 
Real-Time Text and WebRTC @ Kamailio World 2023
byLorenzo Miniero
 
Become a rockstar using FOSS!
byLorenzo Miniero
 
SIP transfer with Janus/WebRTC @ OpenSIPS 2022
byLorenzo Miniero
 
WHIP WebRTC Broadcasting @ FOSDEM 2022
byLorenzo Miniero
 
WebRTC, RED and Janus @ ClueCon21
byLorenzo Miniero
 
WHIP and Janus @ IIT-RTC 2021
byLorenzo Miniero
 
Write a SocialTV app @ OpenSIPS 2021
byLorenzo Miniero
 
Janus + Audio @ Open Source World
byLorenzo Miniero
 
JamRTC @ Wonder WebRTC unConference
byLorenzo Miniero
 
Scaling WebRTC deployments with multicast @ IETF 110 MBONED
byLorenzo Miniero
 
Janus Workshop pt.2 @ ClueCon 2021
byLorenzo Miniero
 

Recently uploaded

PPTX
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
PDF
100 Insights After the 200th Issue of NewMind AI Journal
byNewMind AI
 
PDF
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
DOCX
iRobot Post‑Mortem and Alternative Paths - Discussion Document for Boards and...
byDave Litwiller
 
PPTX
Conversational Agents – Building Intelligent Assistants [Virtual Hands-on Wor...
byUiPathCommunity
 
PDF
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
PDF
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PPTX
Why Most GenAI Projects Fail to Scale and How to Become One of the Success St...
byEarley Information Science
 
PDF
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PDF
Our Digital Tribe_ Cultivating Connection and Growth in Our Slack Community 🌿...
bysanjeetmishra30
 
PDF
Exam Prep Plan Overview: Amazon Web Services (AWS) Certified
byVICTOR MAESTRE RAMIREZ
 
PDF
DevFest El Jadida 2025 - Product Thinking
byElmehdi AMLOU
 
PDF
Access Control 2025: From Security Silo to Software-Defined Ecosystem
byMemoori
 
PDF
How to Evaluate a High Performance Database
byScyllaDB
 
PPTX
Emancipatory Information Retrieval: Radically Reorienting Information Retriev...
byBhaskar Mitra
 
PDF
Internet_of_Things_IoT_for_Next_Generation_Smart_Systems_Utilizing.pdf
byRoshanSyed12
 
PPTX
DYNAMICALLY.pptx good for the teachers or students to do seminars and for tea...
bympoorvika031
 
PDF
WCAG Analyzer Tools and Techniques for User-Friendly Websites
byAccessibility Analyzer
 
PDF
AI Powered Document Processing and Data Extraction
byRobert McDermott
 
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
100 Insights After the 200th Issue of NewMind AI Journal
byNewMind AI
 
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
iRobot Post‑Mortem and Alternative Paths - Discussion Document for Boards and...
byDave Litwiller
 
Conversational Agents – Building Intelligent Assistants [Virtual Hands-on Wor...
byUiPathCommunity
 
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
Why Most GenAI Projects Fail to Scale and How to Become One of the Success St...
byEarley Information Science
 
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
Our Digital Tribe_ Cultivating Connection and Growth in Our Slack Community 🌿...
bysanjeetmishra30
 
Exam Prep Plan Overview: Amazon Web Services (AWS) Certified
byVICTOR MAESTRE RAMIREZ
 
DevFest El Jadida 2025 - Product Thinking
byElmehdi AMLOU
 
Access Control 2025: From Security Silo to Software-Defined Ecosystem
byMemoori
 
How to Evaluate a High Performance Database
byScyllaDB
 
Emancipatory Information Retrieval: Radically Reorienting Information Retriev...
byBhaskar Mitra
 
Internet_of_Things_IoT_for_Next_Generation_Smart_Systems_Utilizing.pdf
byRoshanSyed12
 
DYNAMICALLY.pptx good for the teachers or students to do seminars and for tea...
bympoorvika031
 
WCAG Analyzer Tools and Techniques for User-Friendly Websites
byAccessibility Analyzer
 
AI Powered Document Processing and Data Extraction
byRobert McDermott
 

Janus/HOMER/HEPIC @ OpenSIPS18

  • 1.
    Troubleshooting and monitoringJanus: a HEPIC journey! Lorenzo Miniero @elminiero OpenSIPS Summit 2nd May 2018,
  • 2.
    “Which Lorenzo areyou, exactly?”
  • 3.
    You may rememberme from “Revenge of the Mutant Janus Instances” (not really...)
  • 4.
    No, seriously, rememberJanus? • General purpose, open source WebRTC server • https://github.com/meetecho/janus-gateway • Demos and documentation: https://janus.conf.meetecho.com • Community: https://groups.google.com/forum/#!forum/meetecho-janus Past OpenSIPS presentations: on Janus, SIP, and various salads • https://www.youtube.com/watch?v=SFeWYewoL7Q • https://www.youtube.com/watch?v=anmyMC6Ovl8&t=25112s
  • 5.
    No, seriously, rememberJanus? • General purpose, open source WebRTC server • https://github.com/meetecho/janus-gateway • Demos and documentation: https://janus.conf.meetecho.com • Community: https://groups.google.com/forum/#!forum/meetecho-janus Past OpenSIPS presentations: on Janus, SIP, and various salads • https://www.youtube.com/watch?v=SFeWYewoL7Q • https://www.youtube.com/watch?v=anmyMC6Ovl8&t=25112s
  • 6.
    Troubleshooting Janus: abit of history • First approach (still widely used) was the Admin API • Request/response protocol available on different transports • Allows to inspect handles and WebRTC “internals” from the Janus perspective • Can tweak some settings too (e.g., enable/disable debugging) • A different, asynchronous mechanism then followed: Event Handlers • Core and plugins generate events of different types • Custom modules can subscribe to and handle them • e.g., save to DB, send to external service, CDR, etc. Fun fact: Event Handlers were conceived during OpenSIPS 2016! • You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that
  • 7.
    Troubleshooting Janus: abit of history • First approach (still widely used) was the Admin API • Request/response protocol available on different transports • Allows to inspect handles and WebRTC “internals” from the Janus perspective • Can tweak some settings too (e.g., enable/disable debugging) • A different, asynchronous mechanism then followed: Event Handlers • Core and plugins generate events of different types • Custom modules can subscribe to and handle them • e.g., save to DB, send to external service, CDR, etc. Fun fact: Event Handlers were conceived during OpenSIPS 2016! • You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that
  • 8.
    Troubleshooting Janus: abit of history • First approach (still widely used) was the Admin API • Request/response protocol available on different transports • Allows to inspect handles and WebRTC “internals” from the Janus perspective • Can tweak some settings too (e.g., enable/disable debugging) • A different, asynchronous mechanism then followed: Event Handlers • Core and plugins generate events of different types • Custom modules can subscribe to and handle them • e.g., save to DB, send to external service, CDR, etc. Fun fact: Event Handlers were conceived during OpenSIPS 2016! • You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that
  • 9.
    Routing and managingasynchronous events
  • 10.
    “Sample Event Handler”:HTTP as a notifier • Simply forwards all events as JSON to an HTTP backend • Supports basic authentication • Can group events (i.e., JSON array vs. multiple JSON objects) • Implements basic retransmissions (exponential back-off) • Does nothing more than that: logic needs to be elsewhere • HTTP backend decides what to do with events, if anything • Behaviour can be tweaked via Admin API calls Need something else? Check the alternatives or write your own! A few other event handlers also available • RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)
  • 11.
    “Sample Event Handler”:HTTP as a notifier • Simply forwards all events as JSON to an HTTP backend • Supports basic authentication • Can group events (i.e., JSON array vs. multiple JSON objects) • Implements basic retransmissions (exponential back-off) • Does nothing more than that: logic needs to be elsewhere • HTTP backend decides what to do with events, if anything • Behaviour can be tweaked via Admin API calls Need something else? Check the alternatives or write your own! A few other event handlers also available • RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)
  • 12.
    “Sample Event Handler”:HTTP as a notifier • Simply forwards all events as JSON to an HTTP backend • Supports basic authentication • Can group events (i.e., JSON array vs. multiple JSON objects) • Implements basic retransmissions (exponential back-off) • Does nothing more than that: logic needs to be elsewhere • HTTP backend decides what to do with events, if anything • Behaviour can be tweaked via Admin API calls Need something else? Check the alternatives or write your own! A few other event handlers also available • RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)
  • 13.
    How do wehandle events, now? • Generating events is easy, evaluating them is another matter... • Event Handlers typically just relay events, and don’t do processing themselves • Analyzing and correlating tons of events is complicated • A few different approaches, from easiest to trickiest 1 https://github.com/stirlab/janus-event-server (dumping events to file) 2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite) 3 Write your own backend, e.g., http://www.meetecho.com/blog/event-handlers-a-practical-example/ http://www.meetecho.com/blog/correlating-janus-event-handlers/ • A much better approach: trust the smart guys! • http://www.sipcapture.org/ • http://hepic.tel/
  • 14.
    How do wehandle events, now? • Generating events is easy, evaluating them is another matter... • Event Handlers typically just relay events, and don’t do processing themselves • Analyzing and correlating tons of events is complicated • A few different approaches, from easiest to trickiest 1 https://github.com/stirlab/janus-event-server (dumping events to file) 2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite) 3 Write your own backend, e.g., http://www.meetecho.com/blog/event-handlers-a-practical-example/ http://www.meetecho.com/blog/correlating-janus-event-handlers/ • A much better approach: trust the smart guys! • http://www.sipcapture.org/ • http://hepic.tel/
  • 15.
    How do wehandle events, now? • Generating events is easy, evaluating them is another matter... • Event Handlers typically just relay events, and don’t do processing themselves • Analyzing and correlating tons of events is complicated • A few different approaches, from easiest to trickiest 1 https://github.com/stirlab/janus-event-server (dumping events to file) 2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite) 3 Write your own backend, e.g., http://www.meetecho.com/blog/event-handlers-a-practical-example/ http://www.meetecho.com/blog/correlating-janus-event-handlers/ • A much better approach: trust the smart guys! • http://www.sipcapture.org/ • http://hepic.tel/
  • 16.
    Event Handlers inHOMER since “day one”(*)
  • 17.
    (*A trip downMemory Lane: FOSDEM 2017!) https://archive.fosdem.org/2017/schedule/event/janus/ https://archive.fosdem.org/2017/schedule/event/homer/
  • 18.
    From SIP toWebRTC • As anticipated, Event Handlers were conceived in a chat with QXIP • “How can I dump SIP messages with Sofia SIP?” • “Which other events would be useful to have?” • “What about a new modular approach, where HEP could be a plugin?” • The only “caveat”: it was SIP only • Only events coming from the SIP plugin were tracked, to follow the signalling • Other events related to SIP handles only saved as “related info” • Still very used and widespread, but limited to WebRTC/SIP scenarios • Lately, big step to start tracking generic WebRTC applications instead • How to handle users with more than one PeerConnection? • How to relate a PeerConnection to another, for issues and more? • How to reconstruct a session topology from the media perspective?
  • 19.
    From SIP toWebRTC • As anticipated, Event Handlers were conceived in a chat with QXIP • “How can I dump SIP messages with Sofia SIP?” • “Which other events would be useful to have?” • “What about a new modular approach, where HEP could be a plugin?” • The only “caveat”: it was SIP only • Only events coming from the SIP plugin were tracked, to follow the signalling • Other events related to SIP handles only saved as “related info” • Still very used and widespread, but limited to WebRTC/SIP scenarios • Lately, big step to start tracking generic WebRTC applications instead • How to handle users with more than one PeerConnection? • How to relate a PeerConnection to another, for issues and more? • How to reconstruct a session topology from the media perspective?
  • 20.
    From SIP toWebRTC • As anticipated, Event Handlers were conceived in a chat with QXIP • “How can I dump SIP messages with Sofia SIP?” • “Which other events would be useful to have?” • “What about a new modular approach, where HEP could be a plugin?” • The only “caveat”: it was SIP only • Only events coming from the SIP plugin were tracked, to follow the signalling • Other events related to SIP handles only saved as “related info” • Still very used and widespread, but limited to WebRTC/SIP scenarios • Lately, big step to start tracking generic WebRTC applications instead • How to handle users with more than one PeerConnection? • How to relate a PeerConnection to another, for issues and more? • How to reconstruct a session topology from the media perspective?
  • 21.
    Challenge #1: gettingevents in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
  • 22.
    Challenge #1: gettingevents in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
  • 23.
    Challenge #1: gettingevents in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
  • 24.
    Challenge #1: gettingevents in HOMER/HEPIC • Basically, this was there already in HOMER 5.x • HTTP Event Handler + hepipe.js = Magic! • hepipe.js as an Events server, statically injecting events in the platform • Anyway, considerable refactoring done for HOMER 7.x • Modular protocols and backends (e.g., Prometheus vs. InfluxDB) • The solution: HEPop as a way to index/tag incoming heterogeneous data • Support for several different data sources (e.g., Janus and mediasoup) • For Janus, it acts as an events server exactly as hepipe.js did A potential Janus-specific improvement? As anticipated, a MQTT event handler is on the way (thanks Olle!) • Might be a much better alternative to HTTP as a way to channel events
  • 25.
    Challenge #2: storingand processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
  • 26.
    Challenge #2: storingand processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
  • 27.
    Challenge #2: storingand processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
  • 28.
    Challenge #2: storingand processing events • Ex-post evaluations vs. live monitoring • Analyzing a dump after the session ended is different from live monitoring • Postgres, MySQL, InfluxDB, Prometheus, etc. • Several options in the new HOMER/HEPIC, which is exciting! • JSON databases and queries • Janus events are in JSON format, so why not use that effectively? • HOMER/HEPIC backends support JSON as a native format Storing and tagging events Smart choice is to store events AND tag them as they flow • Good way to create quick correlations and recap what happened • Media stats can then be turned in Time Series (monitoring/alerting)
  • 29.
    Challenge #3: correlatingevents • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
  • 30.
    Challenge #3: correlatingevents • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
  • 31.
    Challenge #3: correlatingevents • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
  • 32.
    Challenge #3: correlatingevents • Sessions might be used to identify users, but can’t always be “trusted” • Works fine when users talk to Janus directly (1↔1 relationship) • There are server-side applications with a single session for all users, though • Enter opaque_id: a client-provided opaque identifier • Just set the same opaque ID for all handles belonging to the same user • The opaque ID can be whatever you want, Janus doesn’t care • Useful for intra-plugin, but also inter-plugin, correlations • e.g., to know a user is publishing and subscribing in a VideoRoom • ... but also, for instance, in a SIP call in the same application Want to learn more? http://www.meetecho.com/blog/correlating-janus-event-handlers/
  • 33.
    Challenge #3: storingand processing events
  • 34.
    Challenge #3: storingand processing events
  • 35.
    Challenge #3: storingand processing events
  • 36.
    Challenge #4: client-sideevents • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
  • 37.
    Challenge #4: client-sideevents • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
  • 38.
    Challenge #4: client-sideevents • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
  • 39.
    Challenge #4: client-sideevents • All the discussion so far has been for server-side events • Janus originates events for what happens there • What about the client side perspective? • In WebRTC, standard mechanism for getting statistics on a PeerConnection • https://www.w3.org/TR/webrtc-stats/ • Calls to getStats() return useful info, but in JavaScript • How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well? Solution: a new Janus plugin! • Client attaches and sends stats on a regular basis to the plugin via Janus API • Plugin relays stats via Event Handlers (same opaque ID as media channels) • Note: still WIP (working demo using rtcstats), but not open source yet
  • 40.
    A real usecase: IETF meetings!
  • 41.
    Remote participation atthe IETF https://www.slideshare.net/LorenzoMiniero/ietf-remote-participation-via-meetecho-webrtc-meetup-stockholm https://www.vuc.me/2017/vuc640-ietf-remote-participation-with-lorenzo-miniero/
  • 42.
    Studying data fromIETF 101 in London • IETF remote participation based on multiple Janus plugins • SIP plugin for mixed audio • Streaming plugin for static sources • VideoRoom plugin for remote speakers • Collected events related to 10 different Working Group sessions • https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000 • MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING, IPSECME, LAMPS • Good mix of heterogeneous sessions • Some shorter, some longer (DETNET lasted 5 hours!) • Some had many active remote speakers • All had attendees from all over the world (different network conditions)
  • 43.
    Studying data fromIETF 101 in London • IETF remote participation based on multiple Janus plugins • SIP plugin for mixed audio • Streaming plugin for static sources • VideoRoom plugin for remote speakers • Collected events related to 10 different Working Group sessions • https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000 • MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING, IPSECME, LAMPS • Good mix of heterogeneous sessions • Some shorter, some longer (DETNET lasted 5 hours!) • Some had many active remote speakers • All had attendees from all over the world (different network conditions)
  • 44.
    Studying data fromIETF 101 in London • IETF remote participation based on multiple Janus plugins • SIP plugin for mixed audio • Streaming plugin for static sources • VideoRoom plugin for remote speakers • Collected events related to 10 different Working Group sessions • https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000 • MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING, IPSECME, LAMPS • Good mix of heterogeneous sessions • Some shorter, some longer (DETNET lasted 5 hours!) • Some had many active remote speakers • All had attendees from all over the world (different network conditions)
  • 45.
    First experiments: dumpingevents to Postgres • Postgres supports JSON natively, and is used by HOMER/HEPIC too • Why not play with this locally, first? • Created a single database, and a different table per each WG • Each table just has a unique index and the JSON value • All queries can be done on JSON fields • Played with different queries to identify different events • How many concurrent participants? How many were active? • Who subscribed to what? (SIP, slides, camera, remotees) • Any network errors? If so, caused by what? • What can we extract from the statistics?
  • 46.
    First experiments: dumpingevents to Postgres • Postgres supports JSON natively, and is used by HOMER/HEPIC too • Why not play with this locally, first? • Created a single database, and a different table per each WG • Each table just has a unique index and the JSON value • All queries can be done on JSON fields • Played with different queries to identify different events • How many concurrent participants? How many were active? • Who subscribed to what? (SIP, slides, camera, remotees) • Any network errors? If so, caused by what? • What can we extract from the statistics?
  • 47.
    First experiments: dumpingevents to Postgres • Postgres supports JSON natively, and is used by HOMER/HEPIC too • Why not play with this locally, first? • Created a single database, and a different table per each WG • Each table just has a unique index and the JSON value • All queries can be done on JSON fields • Played with different queries to identify different events • How many concurrent participants? How many were active? • Who subscribed to what? (SIP, slides, camera, remotees) • Any network errors? If so, caused by what? • What can we extract from the statistics?
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
    The ugliest chartsyou’ll ever see!
  • 53.
    Some bugs tobe fixed in Janus too, I guess... • Analysing dumps, one thing popped to the eye: RTCP RTT > 4M?! • RTT was either quite low, or super-high, never in the middle • Turned out to be a bug in the RTCP parsing code, though... (that we fixed, now!)
  • 54.
    How does itlook like in HOMER/HEPIC?
  • 55.
    How does itlook like in HOMER/HEPIC?
  • 56.
    Cool demo withLorenzo tomorrow!
  • 57.
    Next steps • Collectingreal data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
  • 58.
    Next steps • Collectingreal data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
  • 59.
    Next steps • Collectingreal data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
  • 60.
    Next steps • Collectingreal data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
  • 61.
    Next steps • Collectingreal data and analysing it was very useful • Helped fix a couple of bugs (I’m looking at you, RTT!) • Showed we needed better info in some events (e.g., in selected-pair) • Very helpful in terms of HOMER/HEPIC integration as well • What were we looking for? • Was the available information enough? • Did we have all the instruments to monitor/navigate/search? What’s next? Not the end of the journey, just the beginning! • Need to generalize IETF meeting considerations to other use cases • Live monitoring and troubleshooting is definitely of interest • Most of the analysis was done on ex-post processing • Alerting is ready, though, so looking forward to that!
  • 62.
    Thanks! Questions? Comments? Getin touch! • https://twitter.com/elminiero • https://twitter.com/meetecho • http://www.meetecho.com