The document discusses the Routing Information Service (RIS) maintained by RIPE NCC, which collects and stores BGP routing data from routers located at Internet exchange points worldwide. It has evolved over 15+ years from a single server to a large distributed system using Apache Hadoop to store and process exabytes of routing data. The RIS data is freely available to network operators and researchers through raw data downloads, APIs, and web interfaces like RIPEstat to enable analysis of routing behavior, anomalies, and internet infrastructure trends over time.
ION Bucharest, 12 October 2016 - Today, we can say that IPv6 is already happening all around the world. It’s interesting to see the main reasons that made it happen, how it’s happening, and to make the audience think about their deployment status and strategy. Statistics from different sources are showed, including data from RIPE NCC measurements.
ION Bucharest, 12 October 2016 - Today, we can say that IPv6 is already happening all around the world. It’s interesting to see the main reasons that made it happen, how it’s happening, and to make the audience think about their deployment status and strategy. Statistics from different sources are showed, including data from RIPE NCC measurements.
Presentation given by Mirjam Kühne and Ivo Dijkhuis at TF-CSIRT (the computer security incident response teams task force) in Poznań on 21st & 22nd May 2015
32nd TWNIC IP OPM: ROA+ROV deployment & industry developmentAPNIC
APNIC Infrastructure & Development Director Che-Hoo Cheng gives a presentation on ROA and ROV deployment and why routing security is becoming more important than ever at the 32nd TWNIC IP OPM in Taipei from 20 to 21 June 2019.
What would you do if you had access to all the routing data from the Internet? In this talk, we will introduce a new framework for collecting, storing, and parsing routing data in a way that can be made available to network engineers and application developers through a simple and clean REST API. This API presents a new opportunity for network engineers to understand, visualize, and analyze their network in a way consistent with today’s software engineering practices.
ThaiNOG Day 2019: Internet Number Registry Services, the Next GenerationAPNIC
APNIC Director General Paul Wilson gives a presentation on Internet number registry services - the next generation at ThaiNOG 2019, held with BKNIX 2019 in Bangkok, Thailand from 7 to 8 May 2019.
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksMapR Technologies
From the Hadoop Summit 2015 Session with Nick Amato.
This session examines practical ways you can begin leveraging network data sources in Hadoop using familiar technologies like SQL and BI tools. Using the diverse sets of sources available, such as traces, routing protocol data, and direct packet captures from critical network locations, we will examine the capabilities of BI tools in the network context and examine cases for extracting value from data collected from the network infrastructure.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
3. Colin Petrie | RON++ | 14/12/2016 3
What is RIS?
• Worldwide network of BGP collectors
• Deployed at Internet Exchange Points
- Including at AMS-IX and NL-IX
• Collects raw BGP data from peers
• Stores BGP routing table dumps
• 15+ years of history
• Used by network operators and researchers
every day!
5. Colin Petrie | RON++ | 14/12/2016 5
What is RIS?
• A huge archive of useful data about BGP
routing activity
• A database where you can look up almost
anything you want to know about routing
• We provide APIs to query all the data
• And of course, a nice shiny web interface!
- RIPEstat
- https://stat.ripe.net/
7. Colin Petrie | RON++ | 14/12/2016 7
Why RIS?
• Original project was defined in RIPE-200
- in 1999, when the BGP table was 64,000 routes!!
• Looking glasses are instantaneous
• Routing problems are also instantaneous
• BGP needs to be recorded, to track what is
happening and what has happened.
• Also to provide statistics and reporting on
routing table metrics
8. Colin Petrie | RON++ | 14/12/2016 8
Why the RIPE NCC RIS?
• RIPE NCC is a neutral body
• Experience running measurement platforms
- Test Traffic Measurement project
- RIPE Atlas
• Supporting our own members
- who are mainly network operators
• Supporting the community
- researchers
- operators
10. Colin Petrie | RON++ | 14/12/2016 10
Raw data!
• 15+ years of data available to download and
analyse yourself :)
• https://www.ripe.net/analyse/internet-
measurements/routing-information-service-ris/
ris-raw-data
• Data stored in MRT (RFC6396) format
• Readable using BGPdump utility
- Open source, maintained by RIPE NCC
- https://bitbucket.org/ripencc/bgpdump
11. Colin Petrie | RON++ | 14/12/2016 11
Web interfaces and APIs
• Of course, if all we did was store the raw data,
we’d just need a bunch of hard disks and an
FTP server
• But you want to query all our lovely datasets!
• RIPEstat
- https://stat.ripe.net/
• Our portal for everything you ever wanted to
know!
12. Colin Petrie | RON++ | 14/12/2016 12
RIPEstat
• RIPEstat is a web-based interface that
provides everything you ever wanted to know
about IP address space, Autonomous System
Numbers (ASNs), and related information for
hostnames and countries in one place.
13. Colin Petrie | RON++ | 14/12/2016 13
RIPEstat
• What can you search for?
- ASN (autonomous system number)
- IPv4 address
- IPv4 prefix
- IPv6 address
- IPv6 prefix
- country code (ISO - e.g. NL, ES, US)
22. Colin Petrie | RON++ | 14/12/2016 22
Live stream - demo
• Prototype!!
• Let’s see if it works
• http://stream-dev.ris.ripe.net/demo
• Live stream enables new applications
- BGP Hijack detection
- real time anomaly analysis
- live monitoring of your routes
24. Colin Petrie | RON++ | 14/12/2016 24
RIPEstat Data API
• All these queries are available through an API
• Actually, all those shiny web interfaces use the
API anyway
• You can use it too!! Write your own scripts etc
• https://stat.ripe.net/docs/data_api
• There are also some extra API calls which are
not yet visualised
25. Colin Petrie | RON++ | 14/12/2016 25
RIPEstat Data API
• Remember this started because looking glasses
are instantaneous?
• BGP State
- https://stat.ripe.net/docs/data_api#BGPState
• This data call returns the state of BGP routes for a
resource at a certain point in time, as observed by
all the RIS collectors
• This is derived by applying a computation of state
to the RIB dump (granularity=8h) that occurred
exactly before that time, using the BGP updates
observed between the RIB time and the query time.
26. Colin Petrie | RON++ | 14/12/2016 26
RIPEstat Data API - BGP State
• https://stat.ripe.net/data/
bgp-state/data.json?
resource=193.0.24.0/21&
timestamp=2016-05-19T
00:33:21
• Show me what this prefix
looked like at exactly this
time!
State of AS2121 (RIPE meeting) prefix after we
plugged in the router at RIPE72 in Copenhagen
27. What else can you do?
Lots of analysis that this data allows
34. Colin Petrie | RON++ | 14/12/2016 34
How can you help?
• Peer with us!!!
- AS12654 @ AMS-IX, NL-IX
- RRC03
- http://www.ris.ripe.net/cgi-bin/peerreg.cgi
• Send us your routes
- If you can send us your full BGP table, please do
- If not, send us what you can!
- We will be recording them forever ;-)
36. Colin Petrie | RON++ | 14/12/2016 36
Collector history
Collector Location IXP Deployed Removed
RRC00 Amsterdam Multi-hop 1999 -
RRC01 London LINX 2000 -
RRC02 Paris SFINX 2001 2008
RRC03 Amsterdam AMS-IX 2001 -
RRC04 Geneva CIXP 2001 -
RRC05 Vienna VIX 2001 -
RRC06 Tokyo DIX-IE 2001 -
RRC07 Stockholm Netnod 2002 -
RRC08 San Jose MAE-West 2002 2004
RRC09 Zurich TIX 2003 2004
RRC10 Milan MIX 2003 -
RRC11 New York NYIIX 2004 -
RRC12 Frankfurt DE-CIX 2004 -
RRC13 Moscow MSK-IX 2005 -
RRC14 Palo Alto PAIX 2005 -
RRC15 Sao Paulo PTT-Metro SP 2006 -
RRC16 Miami NOTA 2008 -
RRC18 Barcelona CATNIX 2015 -
RRC19 Johannesburg NAPAfrica JB 2016 -
RRC20 Zurich SwissIX 2015 -
RRC21 Paris FranceIX 2015 -
37. Colin Petrie | RON++ | 14/12/2016 37
RRC01 data production rate
38. Colin Petrie | RON++ | 14/12/2016 38
RRC01 cumulative data
39. Colin Petrie | RON++ | 14/12/2016 39
Data growth
• More BGP routes
- BGP table has grown from 60,000 to 600,000 routes
- more BGP updates
- larger RIB (table) dumps
• More RIS collectors
• More peers at each collector
•Non-linear growth curve ;)
41. Colin Petrie | RON++ | 14/12/2016 41
Original architecture (1999)
• Diagram from
RIPE-200 (original
concept)
• Note ‘RIS Server’
- singular!
• Also, the
‘database’
- this becomes the
hardest part!!
Original RIS design (RIPE-200)
circa 1999
43. Colin Petrie | RON++ | 14/12/2016 43
Classic architecture
• MySQL database
- many millions of BGP updates
- hundreds of thousands of BGP routes, seen by hundreds of
peers
- route attributes (communities, AS_PATH, origin, med, etc)
- ASN adjacencies
- more/less specific matching
- complex data schema
44. Colin Petrie | RON++ | 14/12/2016 44
Scaling MySQL
• Splitting and sharding
- 8 MySQL servers
- some collectors were so big they needed their own MySQL
server!
• Data retention
- database was only query-able for 3 months worth of data
- the references grew too large, that every 3 months we
basically had to drop all the data, and let it start again!!
45. Colin Petrie | RON++ | 14/12/2016 45
Scaling the collectors
• Quagga used as BGP collector
• Single-threaded
- Not as scalable on modern multi-core CPUs
• Locks updates during table-dump process
- Requires that dump completes before the hold timer
expires, or BGP session will drop
• Some data consistency issues
- Sometimes updates are missing from the update dumps at
the time of a table dump
- This makes it difficult to accurately rebuild BGP state at a
intermediate time, if updates are not reliable in-between
48. Colin Petrie | RON++ | 14/12/2016 48
Back-end data distribution
49. Colin Petrie | RON++ | 14/12/2016 49
Data processing
• Apache Hadoop
- An open-source software framework for distributed storage
and distributed processing of very large data sets on
computer clusters built from commodity hardware.
• “Big Data” storage and analytics
• Allows us to build a scalable storage and
processing cluster
• Currently over 150 servers in the cluster!
- Although the cluster is not only used for RIS!
- Also used by RIPE Atlas and other projects
50. Colin Petrie | RON++ | 14/12/2016 50
Data processing - components
• HDFS
- distributed, replicated, cluster filesystem
• YARN
- compute resource manager and application scheduler
• Map/Reduce
- massive batch job processing
• HBase
- non-relational distributed database
- large tables - billions of rows X millions of columns
51. Colin Petrie | RON++ | 14/12/2016 51
Data processing - components
• Spark
- Cluster computing used for data stream processing
- i.e. non-batch computing
• Azkaban
- batch workflow job scheduler, dependency tracking etc
• Kafka
- BGP/Atlas messaging bus
52. Colin Petrie | RON++ | 14/12/2016 52
Data processing - RIS
• Raw data inputs:
- BGP updates events - everything must start from a BGP
message!
- BGP table dumps (which can also be derived from updates)
• Derived datasets
- update-counts, first-last-seen, prefixes-transited-by-asn,
peers-list, asn-stats, asn-adjacencies
- country-code mapping
- aggregated counts for historical overviews
- distributed looking-glass processing
53. Colin Petrie | RON++ | 14/12/2016 53
Data processing - Atlas
• Raw data inputs:
- Atlas measurement messages!
• Derived datasets
- Mainly aggregated counters for different zoom levels
- Probe traffic stats
- Ping measurement stats (loss %, min, max, med etc)
- DNS query stats (loss %, latency, etc)