Large scale-olap-with-kobayashi

•

0 likes•730 views

Boundary

Technology Business

Monitoring is an
analytics problem

Friday, May 18, 12

Historical Perspective

Friday, May 18, 12

1 minute collection
intervals

Arbitrary OLAP

Friday, May 18, 12

Cassandra

bitset indexes per
dimension

query-time sampling

Friday, May 18, 12

apply intelligence to
the problem

Friday, May 18, 12

Arbitrary OLAP
requires 2n data

cubes
where n is dimensionality

Friday, May 18, 12

dimensions (11) measurements (4)
epoch seconds egress packets
epoch minutes egress octets
epoch hours ingress packets
meter id ingress octets
source ip
source port
dest ip
dest port
interface
country
network

Friday, May 18, 12

Total Volume.
by
Host
Port/Protocol
Country
Network
+ meter
For each aggregation period

Friday, May 18, 12

24 hours 2 Months ~10 years

86,400
Observations
(per monitored host per query)

Friday, May 18, 12

86,400*15 ≈ 1.3M
Observations
(per monitored host )

Friday, May 18, 12

Total Observations
(for half a million meters)

Friday, May 18, 12

Riak Key 10 seconds

{ {
Layout

100 meters < 80KB

Friday, May 18, 12

Bitcask would have
been nice

LevelDB backend

Use leveldb cache to
bound memory

Friday, May 18, 12

Compute your keys

Use secondary
indexes sparingly

Friday, May 18, 12

How do I query the
database?

Friday, May 18, 12

Find 45 minutes of
total traffic seen on
meters 1, 2, 226, &
301 starting 18 hours
ago broken down by
traffic type
Friday, May 18, 12

Atomic 10 seconds

{ {
Unit of
Storage
100 meters < 80KB

Friday, May 18, 12

Step 1: fetch appropriate blocks (riak) 45 min Time
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t18 t19
Meter Id
1

0
2

(0,99)

100
(100,199)

200 226

(200,299)
301

300
(300,399)

400
(400,499)

Friday, May 18, 12

Step 2: filter 45 min Time
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t18 t19
Meter Id
1

0
2

(0,99)

100
(100,199)

200 226

(200,299)
301

300
(300,399)

400
(400,499)

Friday, May 18, 12

Step 3: aggregate and perform top-k 45 min

topk( , 10)
1

+
2
226
301

{
epochMillis: 1337230140000
portProtocol: "4740:6"
ingressPackets: 370482
ingressOctets: 3113782199
egressPackets: 343780
egressOctets: 37126033
},
{
epochMillis: 1337230140000
portProtocol: "9092:6"
ingressPackets: 440915
ingressOctets: 1816615857
egressPackets: 481237
egressOctets: 1312198133
},
...

Friday, May 18, 12

In URL Form
http://computers-r-terrible/
volume_1m_meter_port_protocol/
data?
from=-18h&
duration=45m
parts=1,2,226,301&
aggregations=observationDomainId

Friday, May 18, 12

Arbitrary Aggregations
http://computers-r-terrible/
volume_1m_meter_port_protocol/
data?
from=-18h&
duration=45m&
parts=1,2,226,301&
aggregations=
observationDomainId,
epochMillis

Friday, May 18, 12

“unfortunately the
project has been
blocked for weeks
choosing a name”

Friday, May 18, 12

Send expired data to
cold storage

output in arbitrary
time resolution

Friday, May 18, 12

Open source the data
cubing and predicate
matching code

Query grammar for
kobayashi
Friday, May 18, 12

Recently uploaded

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Scalable LLM APIs for AI and Generative AI Application Development Ettikan Karuppiah, Director/Technologist - NVIDIA Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

apidays

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Recently uploaded (20)

A Year of the Servo Reboot: Where Are We Now?

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Corporate and higher education May webinar.pptx

Apidays New York 2024 - The value of a flexible API Management solution for O...

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

presentation ICT roal in 21st century education

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Why Teams call analytics are critical to your entire business

Artificial Intelligence Chap.5 : Uncertainty

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Axa Assurance Maroc - Insurer Innovation Award 2024

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Data Cloud, More than a CDP by Matt Robison

MS Copilot expands with MS Graph connectors

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Featured

https://www.hubspot.com/state-of-marketing · Scaling relationships and proving ROI · Social media is the place for search, sales, and service · Authentic influencer partnerships fuel brand growth · The strongest connections happen via call, click, chat, and camera. · Time saved with AI leads to more creative work · Seeking: A single source of truth · TLDR; Get on social, try AI, and align your systems. · More human marketing, powered by robots

2024 State of Marketing Report – by Hubspot

Marius Sescu

ChatGPT is a revolutionary addition to the world since its introduction in 2022. A big shift in the sector of information gathering and processing happened because of this chatbot. What is the story of ChatGPT? How is the bot responding to prompts and generating contents? Swipe through these slides prepared by Expeed Software, a web development company regarding the development and technical intricacies of ChatGPT!

Everything You Need To Know About ChatGPT

Expeed Software

Product Design Trends in 2024 | Teenage Engineerings

Pixeldarts

Mental health has been in the news quite a bit lately. Dozens of U.S. states are currently suing Meta for contributing to the youth mental health crisis by inserting addictive features into their products, while the U.S. Surgeon General is touring the nation to bring awareness to the growing epidemic of loneliness and isolation. The country has endured periods of low national morale, such as in the 1970s when high inflation and the energy crisis worsened public sentiment following the Vietnam War. The current mood, however, feels different. Gallup recently reported that national mental health is at an all-time low, with few bright spots to lift spirits. To better understand how Americans are feeling and their attitudes towards mental health in general, ThinkNow conducted a nationally representative quantitative survey of 1,500 respondents and found some interesting differences among ethnic, age and gender groups. Technology For example, 52% agree that technology and social media have a negative impact on mental health, but when broken out by race, 61% of Whites felt technology had a negative effect, and only 48% of Hispanics thought it did. While technology has helped us keep in touch with friends and family in faraway places, it appears to have degraded our ability to connect in person. Staying connected online is a double-edged sword since the same news feed that brings us pictures of the grandkids and fluffy kittens also feeds us news about the wars in Israel and Ukraine, the dysfunction in Washington, the latest mass shooting and the climate crisis. Hispanics may have a built-in defense against the isolation technology breeds, owing to their large, multigenerational households, strong social support systems, and tendency to use social media to stay connected with relatives abroad. Age and Gender When asked how individuals rate their mental health, men rate it higher than women by 11 percentage points, and Baby Boomers rank it highest at 83%, saying it’s good or excellent vs. 57% of Gen Z saying the same. Gen Z spends the most amount of time on social media, so the notion that social media negatively affects mental health appears to be correlated. Unfortunately, Gen Z is also the generation that’s least comfortable discussing mental health concerns with healthcare professionals. Only 40% of them state they’re comfortable discussing their issues with a professional compared to 60% of Millennials and 65% of Boomers. Race Affects Attitudes As seen in previous research conducted by ThinkNow, Asian Americans lag other groups when it comes to awareness of mental health issues. Twenty-four percent of Asian Americans believe that having a mental health issue is a sign of weakness compared to the 16% average for all groups. Asians are also considerably less likely to be aware of mental health services in their communities (42% vs. 55%) and most likely to seek out information on social media (51% vs. 35%).

How Race, Age and Gender Shape Attitudes Towards Mental Health

ThinkNow

AI Trends in Creative Operations 2024 by Artwork Flow.pdf

marketingartwork

Skeleton Culture Code

Skeleton Technologies

PEPSICO Presentation to CAGNY Conference Feb 2024

Neil Kimberley

Content Methodology: A Best Practices Report (Webinar)

contently

How to Prepare For a Successful Job Search for 2024

Albert Qian

A report by thenetworkone and Kurio. The contributing experts and agencies are (in an alphabetical order): Sylwia Rytel, Social Media Supervisor, 180heartbeats + JUNG v MATT (PL), Sharlene Jenner, Vice President - Director of Engagement Strategy, Abelson Taylor (USA), Alex Casanovas, Digital Director, Atrevia (ES), Dora Beilin, Senior Social Strategist, Barrett Hoffher (USA), Min Seo, Campaign Director, Brand New Agency (KR), Deshé M. Gully, Associate Strategist, Day One Agency (USA), Francesca Trevisan, Strategist, Different (IT), Trevor Crossman, CX and Digital Transformation Director; Olivia Hussey, Strategic Planner; Simi Srinarula, Social Media Manager, The Hallway (AUS), James Hebbert, Managing Director, Hylink (CN / UK), Mundy Álvarez, Planning Director; Pedro Rojas, Social Media Manager; Pancho González, CCO, Inbrax (CH), Oana Oprea, Head of Digital Planning, Jam Session Agency (RO), Amy Bottrill, Social Account Director, Launch (UK), Gaby Arriaga, Founder, Leonardo1452 (MX), Shantesh S Row, Creative Director, Liwa (UAE), Rajesh Mehta, Chief Strategy Officer; Dhruv Gaur, Digital Planning Lead; Leonie Mergulhao, Account Supervisor - Social Media & PR, Medulla (IN), Aurelija Plioplytė, Head of Digital & Social, Not Perfect (LI), Daiana Khaidargaliyeva, Account Manager, Osaka Labs (UK / USA), Stefanie Söhnchen, Vice President Digital, PIABO Communications (DE), Elisabeth Winiartati, Managing Consultant, Head of Global Integrated Communications; Lydia Aprina, Account Manager, Integrated Marketing and Communications; Nita Prabowo, Account Manager, Integrated Marketing and Communications; Okhi, Web Developer, PNTR Group (ID), Kei Obusan, Insights Director; Daffi Ranandi, Insights Manager, Radarr (SG), Gautam Reghunath, Co-founder & CEO, Talented (IN), Donagh Humphreys, Head of Social and Digital Innovation, THINKHOUSE (IRE), Sarah Yim, Strategy Director, Zulu Alpha Kilo (CA).

Social Media Marketing Trends 2024 // The Global Indie Insights

Kurio // The Social Media Age(ncy)

The search marketing landscape is evolving rapidly with new technologies, and professionals, like you, rely on innovative paid search strategies to meet changing demands. It’s important that you’re ready to implement new strategies in 2024. Check this out and learn the top trends in paid search advertising that are expected to gain traction, so you can drive higher ROI more efficiently in 2024. You’ll learn: - The latest trends in AI and automation, and what this means for an evolving paid search ecosystem. - New developments in privacy and data regulation. - Emerging ad formats that are expected to make an impact next year. Watch Sreekant Lanka from iQuanti and Irina Klein from OneMain Financial as they dive into the future of paid search and explore the trends, strategies, and technologies that will shape the search marketing landscape. If you’re looking to assess your paid search strategy and design an industry-aligned plan for 2024, then this webinar is for you.

Trends In Paid Search: Navigating The Digital Landscape In 2024

Search Engine Journal

From their humble beginnings in 1984, TED has grown into the world’s most powerful amplifier for speakers and thought-leaders to share their ideas. They have over 2,400 filmed talks (not including the 30,000+ TEDx videos) freely available online, and have hosted over 17,500 events around the world. With over one billion views in a year, it’s no wonder that so many speakers are looking to TED for ideas on how to share their message more effectively. The article “5 Public-Speaking Tips TED Gives Its Speakers”, by Carmine Gallo for Forbes, gives speakers five practical ways to connect with their audience, and effectively share their ideas on stage. Whether you are gearing up to get on a TED stage yourself, or just want to master the skills that so many of their speakers possess, these tips and quotes from Chris Anderson, the TED Talks Curator, will encourage you to make the most impactful impression on your audience. See the full article and more summaries like this on SpeakerHub here: https://speakerhub.com/blog/5-presentation-tips-ted-gives-its-speakers See the original article on Forbes here: http://www.forbes.com/forbes/welcome/?toURL=http://www.forbes.com/sites/carminegallo/2016/05/06/5-public-speaking-tips-ted-gives-its-speakers/&refURL=&referrer=#5c07a8221d9b

5 Public speaking tips from TED - Visualized summary

SpeakerHub

Everyone is in agreement that ChatGPT (and other generative AI tools) will shape the future of work. Yet there is little consensus on exactly how, when, and to what extent this technology will change our world. Businesses that extract maximum value from ChatGPT will use it as a collaborative tool for everything from brainstorming to technical maintenance. For individuals, now is the time to pinpoint the skills the future professional will need to thrive in the AI age. Check out this presentation to understand what ChatGPT is, how it will shape the future of work, and how you can prepare to take advantage.

ChatGPT and the Future of Work - Clark Boyd

Clark Boyd

Getting into the tech field. what next

Tessa Mero

Google's Just Not That Into You: Understanding Core Updates & Search Intent

Lily Ray

How to have difficult conversations

Rajiv Jayarajah, MAppComm, ACC

Introduction to Data Science

Christy Abraham Joy

Time Management & Productivity - Best Practices

Vit Horky

The six step guide to practical project management If you think managing projects is too difficult, think again. We’ve stripped back project management processes to the basics – to make it quicker and easier, without sacrificing the vital ingredients for success. “If you’re looking for some real-world guidance, then The Six Step Guide to Practical Project Management will help.” Dr Andrew Makar, Tactical Project Management

The six step guide to practical project management

MindGenius

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...

RachelPearson36

Featured (20)

2024 State of Marketing Report – by Hubspot

Everything You Need To Know About ChatGPT

Product Design Trends in 2024 | Teenage Engineerings

How Race, Age and Gender Shape Attitudes Towards Mental Health

AI Trends in Creative Operations 2024 by Artwork Flow.pdf

Skeleton Culture Code

PEPSICO Presentation to CAGNY Conference Feb 2024

Content Methodology: A Best Practices Report (Webinar)

How to Prepare For a Successful Job Search for 2024

Social Media Marketing Trends 2024 // The Global Indie Insights

Trends In Paid Search: Navigating The Digital Landscape In 2024

5 Public speaking tips from TED - Visualized summary

ChatGPT and the Future of Work - Clark Boyd

Getting into the tech field. what next

Google's Just Not That Into You: Understanding Core Updates & Search Intent

How to have difficult conversations

Introduction to Data Science

Time Management & Productivity - Best Practices

The six step guide to practical project management

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...

Large scale-olap-with-kobayashi

1. Large-scale OLAP with Kobayashi Boundary Tech Talks Fri, May 18, 2012 Dietrich Featherston, Boundary @d2fn Friday, May 18, 12

2. Monitoring is an analytics problem Friday, May 18, 12

3. Historical Perspective Friday, May 18, 12

4. 1 minute collection intervals Arbitrary OLAP Friday, May 18, 12

5. Cassandra bitset indexes per dimension query-time sampling Friday, May 18, 12

6. Friday, May 18, 12

7. riak_core + fastbit Friday, May 18, 12

8. apply intelligence to the problem Friday, May 18, 12

9. Arbitrary OLAP requires 2n data cubes where n is dimensionality Friday, May 18, 12

10. dimensions (11) measurements (4) epoch seconds egress packets epoch minutes egress octets epoch hours ingress packets meter id ingress octets source ip source port dest ip dest port interface country network Friday, May 18, 12

11. Total Volume. by Host Port/Protocol Country Network + meter For each aggregation period Friday, May 18, 12

12. Friday, May 18, 12

13. 15 < 2 11 Friday, May 18, 12

14. 24 hours 2 Months ~10 years 86,400 Observations (per monitored host per query) Friday, May 18, 12

15. 86,400*15 ≈ 1.3M Observations (per monitored host ) Friday, May 18, 12

16. Total Observations (for half a million meters) Friday, May 18, 12

17. Riak Key 10 seconds { { Layout 100 meters < 80KB Friday, May 18, 12

18. Total Observations Friday, May 18, 12

19. Friday, May 18, 12

20. Friday, May 18, 12

21. Friday, May 18, 12

22. Friday, May 18, 12

23. Friday, May 18, 12

24. Bitcask would have been nice LevelDB backend Use leveldb cache to bound memory Friday, May 18, 12

25. Compute your keys Use secondary indexes sparingly Friday, May 18, 12

26. Friday, May 18, 12

27. How do I query the database? Friday, May 18, 12

28. Find 45 minutes of total traffic seen on meters 1, 2, 226, & 301 starting 18 hours ago broken down by traffic type Friday, May 18, 12

29. Atomic 10 seconds { { Unit of Storage 100 meters < 80KB Friday, May 18, 12

30. Step 1: fetch appropriate blocks (riak) 45 min Time t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t18 t19 Meter Id 1 0 2 (0,99) 100 (100,199) 200 226 (200,299) 301 300 (300,399) 400 (400,499) Friday, May 18, 12

31. Step 2: filter 45 min Time t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t18 t19 Meter Id 1 0 2 (0,99) 100 (100,199) 200 226 (200,299) 301 300 (300,399) 400 (400,499) Friday, May 18, 12

32. Step 3: aggregate and perform top-k 45 min topk( , 10) 1 + 2 226 301 { epochMillis: 1337230140000 portProtocol: "4740:6" ingressPackets: 370482 ingressOctets: 3113782199 egressPackets: 343780 egressOctets: 37126033 }, { epochMillis: 1337230140000 portProtocol: "9092:6" ingressPackets: 440915 ingressOctets: 1816615857 egressPackets: 481237 egressOctets: 1312198133 }, ... Friday, May 18, 12

33. In URL Form http://computers-r-terrible/ volume_1m_meter_port_protocol/ data? from=-18h& duration=45m parts=1,2,226,301& aggregations=observationDomainId Friday, May 18, 12

34. Arbitrary Aggregations http://computers-r-terrible/ volume_1m_meter_port_protocol/ data? from=-18h& duration=45m& parts=1,2,226,301& aggregations= observationDomainId, epochMillis Friday, May 18, 12

35. “unfortunately the project has been blocked for weeks choosing a name” Friday, May 18, 12

36. V V ʹ′ V ≃ Vʹ′ Friday, May 18, 12

37. V V ʹ′ V ≃ Vʹ′ Friday, May 18, 12

38. V V ʹ′ V ≃ Vʹ′ Friday, May 18, 12

39. Future --> Friday, May 18, 12

40. Send expired data to cold storage output in arbitrary time resolution Friday, May 18, 12

41. Open source the data cubing and predicate matching code Query grammar for kobayashi Friday, May 18, 12

42. questions? Friday, May 18, 12

Large scale-olap-with-kobayashi

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Large scale-olap-with-kobayashi