Boost on!!next generation big data platform
•Download as PPTX, PDF•
2 likes•236 views
This document provides an overview of BOOST ON!!'s next generation big data platform. It introduces Neil Tu and his background in data engineering. The agenda covers Data Labs, their work in big data analysis using techniques like machine learning and deep learning. It then details the next generation data platform which includes a Datalake using Nifi for real-time ETL and a Datalake using Hadoop for storage, authentication, and authorization. For the Nifi datalake, it describes the data processing workflow including script execution, data merging, format conversion and partitioning to reduce storage size and boost query performance. For the Hadoop datalake, it outlines the Kerberos and LDAP authentication and Apache Ranger authorization security measures.
Report
Share
Report
Share
Recommended
Microsoft cosmos
This document provides an overview of Cosmos, Microsoft's petabyte-scale data storage and computation system. Cosmos ingests and stores huge amounts of data daily from sources like Bing, MSN, and web crawls. It uses a massively parallel processing architecture based on Dryad to run SQL-like SCOPE queries over this data at scale. Queries are automatically optimized and deployed across Cosmos' tens of thousands of servers to analyze the data and power services like Bing search relevance.
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...
This talk will describe his research into using Hadoop to query and manage big geographic datasets, specifically OpenStreetMap(OSM). OSM is an “open-source” map of the world, growing at a large rate, currently around 5TB of data. The talk will introduce OSM, detail some aspects of the research, but also discuss his experiences with using the SpatialHadoop stack on Azure and Google Cloud.
InfluxDB Internals
Ryan will expand on his popular blog series and drill down into the internals of the database. Ryan will discuss optimizing query performance, best indexing schemes, how to manage clustering (including meta and data nodes), the impact of IFQL on the database, the impact of cardinality on performance, TSI, and other internals that will help you architect better solutions around InfluxDB.
Speed Up Uber's Presto with Alluxio
Alluxio Day X
March 3, 2022
https://www.alluxio.io/alluxio-day/
Speaker:
Liang Chen, Uber
Beinan Wang, Alluxio
Webinar 2017. Supercharge your analytics with ClickHouse. Alexander Zaitsev
Alexander Zaitsev presented on LifeStreet's experience implementing ClickHouse for their ad analytics platform. Some key points:
- LifeStreet processes over 10 billion events per day from their ad exchange and needed a high performance analytics solution.
- They tried various databases but migrated fully to ClickHouse due to its performance for analytics workloads.
- Major challenges included designing an efficient schema, sharding and replication strategy, and reliable data ingestion.
- ClickHouse's dictionary feature allowed them to implement normalized dimensions tables while supporting updates, improving storage efficiency and query performance.
Why You Definitely Don’t Want to Build Your Own Time Series Database
At Outlyer, an infrastructure monitoring tool, we had to build our own TSDB back in 2015 to support our service. Two years later, we decided to take a different direction after seeing for ourselves how hard it is to build and scale a TSDB. This talk will review our journey, the challenges we hit trying to scale a TSDB for large customers and hopefully talk some people out of trying to build one themselves because it is not easy!
Going from three nines to four nines using Kafka | Tejas Chopra, Netflix
The document discusses using Apache Kafka to improve data upload availability from 99.9% to 99.99% when moving data between on-premise and cloud storage. It describes using Kafka to trigger uploads to the cloud from on-premise storage with 99.9% availability and using Kafka to split uploads between cloud and on-premise storage as well as rehydrating failed on-premise uploads from the cloud to achieve 99.99% availability. The presentation concludes that Kafka provides high throughput and persistence needed to design effective data rehydration strategies across cloud and on-premise storage for very high availability.
Presto@Uber
Presto is Uber's distributed SQL query engine for their Hadoop data warehouse. Some key points:
- Presto allows interactive SQL queries directly on Uber's petabyte-scale Hadoop data lake without needing to first load the data into another database.
- It provides fast performance at scale by leveraging columnar data formats like Parquet and optimizing for distributed execution across many nodes.
- Uber deployed a 200 node Presto cluster that handles 30,000 queries per day, serving both ad hoc queries and real-time applications accessing data in Hadoop and improving on the performance of alternative solutions like Hive.
Recommended
Microsoft cosmos
This document provides an overview of Cosmos, Microsoft's petabyte-scale data storage and computation system. Cosmos ingests and stores huge amounts of data daily from sources like Bing, MSN, and web crawls. It uses a massively parallel processing architecture based on Dryad to run SQL-like SCOPE queries over this data at scale. Queries are automatically optimized and deployed across Cosmos' tens of thousands of servers to analyze the data and power services like Bing search relevance.
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...
This talk will describe his research into using Hadoop to query and manage big geographic datasets, specifically OpenStreetMap(OSM). OSM is an “open-source” map of the world, growing at a large rate, currently around 5TB of data. The talk will introduce OSM, detail some aspects of the research, but also discuss his experiences with using the SpatialHadoop stack on Azure and Google Cloud.
InfluxDB Internals
Ryan will expand on his popular blog series and drill down into the internals of the database. Ryan will discuss optimizing query performance, best indexing schemes, how to manage clustering (including meta and data nodes), the impact of IFQL on the database, the impact of cardinality on performance, TSI, and other internals that will help you architect better solutions around InfluxDB.
Speed Up Uber's Presto with Alluxio
Alluxio Day X
March 3, 2022
https://www.alluxio.io/alluxio-day/
Speaker:
Liang Chen, Uber
Beinan Wang, Alluxio
Webinar 2017. Supercharge your analytics with ClickHouse. Alexander Zaitsev
Alexander Zaitsev presented on LifeStreet's experience implementing ClickHouse for their ad analytics platform. Some key points:
- LifeStreet processes over 10 billion events per day from their ad exchange and needed a high performance analytics solution.
- They tried various databases but migrated fully to ClickHouse due to its performance for analytics workloads.
- Major challenges included designing an efficient schema, sharding and replication strategy, and reliable data ingestion.
- ClickHouse's dictionary feature allowed them to implement normalized dimensions tables while supporting updates, improving storage efficiency and query performance.
Why You Definitely Don’t Want to Build Your Own Time Series Database
At Outlyer, an infrastructure monitoring tool, we had to build our own TSDB back in 2015 to support our service. Two years later, we decided to take a different direction after seeing for ourselves how hard it is to build and scale a TSDB. This talk will review our journey, the challenges we hit trying to scale a TSDB for large customers and hopefully talk some people out of trying to build one themselves because it is not easy!
Going from three nines to four nines using Kafka | Tejas Chopra, Netflix
The document discusses using Apache Kafka to improve data upload availability from 99.9% to 99.99% when moving data between on-premise and cloud storage. It describes using Kafka to trigger uploads to the cloud from on-premise storage with 99.9% availability and using Kafka to split uploads between cloud and on-premise storage as well as rehydrating failed on-premise uploads from the cloud to achieve 99.99% availability. The presentation concludes that Kafka provides high throughput and persistence needed to design effective data rehydration strategies across cloud and on-premise storage for very high availability.
Presto@Uber
Presto is Uber's distributed SQL query engine for their Hadoop data warehouse. Some key points:
- Presto allows interactive SQL queries directly on Uber's petabyte-scale Hadoop data lake without needing to first load the data into another database.
- It provides fast performance at scale by leveraging columnar data formats like Parquet and optimizing for distributed execution across many nodes.
- Uber deployed a 200 node Presto cluster that handles 30,000 queries per day, serving both ad hoc queries and real-time applications accessing data in Hadoop and improving on the performance of alternative solutions like Hive.
ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...
Pragma Innovation is an IT services company focused on time series data solutions. Their PASS (Pragma Analytics Software Suite) allows companies to analyze, report on, and make decisions from time series network data using open source software. It is designed for ISPs, hosting providers, and telecom companies. The solution ingests network and log data, standardizes it, enriches it using tools like GeoIP, and stores it in a time series database. This allows customers to build applications for traffic engineering, security, and business intelligence use cases. Key challenges addressed in version 2.0 of the solution include data sampling, IPv4/IPv6 support, and using ClickHouse as the time series database for its performance and simplicity
Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...
The RCP/Rakuten Catalog Platform has been growing at a brisk speed over the last couple of years. Our original backbone was Cassandra. However, as they continued their growth, they internally started realizing that it was not suitable for our next stage of growth.
As such, they started looking into ScyllaDB as a better ROI solution as well as a much more stable backend. The migration itself was challenging since this has to be done for a production live data processing pipeline with minimal impact on customers. In this talk, Hitesh Shah, Engineering Manager at Rakuten USA will dive deeper into challenges and takeaways.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Kubernetes at Spreadshirt - First steps to production
This presentation describes how we at Spreadshirt got started on our adventure into Docker-land which finally led to introducing Kubernetes for container orchestration.
Scylla Summit 2022: New AWS Instances Perfect for ScyllaDB
In this talk AWS’ Ken Krupa, Head of Specialized Solutions Architecture, will describe the architecture and capabilities of two new AWS EC2 instance types perfect for data-intensive storage and IO-heavy workloads like ScyllaDB: the Intel-based I4i and the Graviton2-based I4g series.
The Intel Xeon Ice Lake-based I4i series provides unparalleled raw horsepower for your most demanding workloads. Meanwhile, the Graviton2-powered I4g instances provide lower cost per storage on a power-efficient platform to deploy your cloud-native applications.
Ken will also describe the AWS Nitro SSD, a new form of high-speed NVMe storage with a Flash Translation Layer built with Nitro controllers, which powers both of these instance families.
ScyllaDB VP of Product Tzach Livyatan will then share benchmarking results showing how ScyllaDB behaves under load on these two instance types, providing maximum system utility and efficiency.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
The Data Lake Engine Data Microservices in Spark using Apache Arrow Flight
Machine learning pipelines are a hot topic at the moment. Moving data through the pipeline in an efficient and predictable way is one of the most important aspects of running machine learning models in production.
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
The Home Depot is transforming its architecture to use microservices and polyglot persistence to handle increasing online order volumes of 250,000 lines per hour. Redis is being used to turbo charge existing monolithic applications by offloading pieces to new processes using patterns like caching, concurrency management, and powering algorithms. This improves performance by reducing database degradation and wait times by over 95%. Next steps include setting up Redis clusters on-premises and off-premises to further reduce database CPU usage and onboard more patterns.
Graph Databases at Netflix
In the slide deck, we describe how graph databases are used at Netflix. Graph databases can be faster than relational databases for deeply-connected data - a strength of the underlying model. We have used JanusGraph on top of Cassandra. Both technologies are Open Source.
Presto @ Uber Hadoop summit2017
This document discusses Presto, an interactive SQL query engine for big data. It describes how Presto is optimized to quickly query data stored in Parquet format at Uber. Key optimizations for Parquet include nested column pruning, columnar reads, predicate pushdown, dictionary pushdown, and lazy reads. Benchmark results show these optimizations improve Presto query performance. The document also provides an overview of Uber's analytics infrastructure, applications of Presto, and ongoing work to further optimize Presto and Hadoop.
Роман Новиков "Best Practices for MySQL Performance & Troubleshooting with th...
Troubleshooting performance issues can be a bit tricky, especially when you’re given a broad statement that the database is slow.
Learn to direct your attention to the correct moving pieces and fix what needs your attention.
Learn how all this is done at Percona, what we monitor and track, and the tools we use.
Understanding Storage I/O Under Load
There’s a popular misconception about I/O that (modern) SSDs are easy to deal with; they work pretty much like RAM but use a “legacy” submit-complete API. And other than keeping in mind a disk’s possible peak performance and maybe maintaining priorities of different IO streams there’s not much to care about. This is not quite the case – SSDs do show non-linear behavior and understanding the disk’s real abilities is crucial when it comes to squeezing as much performance from it as possible.
Diskplorer is an open-source disk latency/bandwidth exploring toolset. By using Linux fio under the hood it runs a battery of measurements to discover performance characteristics for a specific hardware configuration, giving you an at-a-glance view of how server storage I/O will behave under load.
ScyllaDB CTO Avi Kivity will share an interesting approach to measuring disk behavior under load, give a walkthrough of Diskplorer and explain how it’s used.
With the elaborated model of a disk at hand, it becomes possible to build latency-oriented I/O scheduling that cherry-picks requests from the incoming queue keeping the disk load perfectly Balanced.
ScyllaDB engineer Pavel Emelyanov will also present the scheduling algorithm developed for the Seastar framework and share results achieved using it.
MongoDB Capacity Planning
Deploying any software can be a challenge if you don't understand how resources are used or how to plan for the capacity of your systems. Whether you need to deploy or grow a single MongoDB instance, replica set, or tens of sharded clusters then you probably share the same challenges in trying to size that deployment.
Distributed Crypto-Currency Trading with Apache Pulsar
Apache Pulsar was developed to address several shortcomings of existing messaging systems including geo-replication, message durability, and lower message latency.
We will implement a multi-currency quoting application that feeds pricing information to a crypto-currency trading platform that is deployed around the globe. Given the volatility of the crypto-currency prices, sub-second message latency is critical to traders. Equally important is ensuring consistent quotes are available to all geographical locations, i.e the price of Bitcoin shown to a user in the USA should be the same as it to a trader in Hong Kong.
We will highlight the advantages of Apache Pulsar over traditional messaging systems and show how its low latency and replication across multiple geographies make it ideally suited for globally distributed, real-time applications.
Operationalizing MongoDB at AOL
The document summarizes Michael DelNegro's work operationalizing MongoDB at AOL over several years. Key aspects included establishing standards for host setup, directory structure, and build scripts; implementing monitoring with Argus, Nagios, and MMS; performing backups; sharing information through internal documentation; and addressing challenges like requiring more planning from developers. The operationalization effort helped support over 500 MongoDB servers storing data for 30+ active projects at AOL.
Presto GeoSpatial @ Strata New York 2017
The document discusses geospatial analytics at Uber using Presto. Presto is an interactive SQL query engine that allows for fast querying of large datasets. The document outlines how Presto was optimized for geospatial queries using quadtree indexing to efficiently filter spatial data and speed up queries involving spatial functions like ST_Contains. This improved query performance for geospatial queries in Presto by 60x compared to the previous approach using Hive.
Case study- Real-time OLAP Cubes
This case study describes an approach to building quasi real-time OLAP cubes in Microsoft SQL Server Analysis Services to enable daily comparisons of production forecasts and outcomes. The cubes are partitioned by time to allow independent and frequent updates. Initial attempts failed due to deadlocks from simultaneous partition updates. The working solution takes advantage of a 6 day work week by switching partition dates on Saturdays only and reprocessing partitions then. This allows real-time and historical partition updates without gaps or overlaps in the data.
Hadoop And Big Data - My Presentation To Selective Audience
This document provides an overview of Hadoop and big data. It discusses what Hadoop is, how it stores and processes large datasets using MapReduce, and its architecture. The presenter notes that Hadoop provides cost savings and productivity benefits by allowing companies to answer questions within large datasets. It also highlights the ROI of Hadoop, showing it can reduce storage and processing costs compared to traditional systems. Resources for learning more about Hadoop are provided at the end.
Enabling Presto Caching at Uber with Alluxio
Alluxio Day VI
October 12, 2021
https://www.alluxio.io/alluxio-day/
Speaker:
Curt Hu, Uber
Beinan Wang, Alluxio
Accelerating analytics in a new era of data
Organizations today produce exponentially more data than they did just a few years ago, but their databases weren’t built to handle these new volumes. As a result, reporting takes way too long, and some complex analytics simply cannot be done. The Era of Massive Data is upon us, and a new approach is required to overcome the limitations of traditional CPU-based data stores.
KEY TAKEAWAYS
- Flexible data exploration with minimal preparation
- Unrestricted access to your organization’s full scope of data
- Access to previously unobtainable insights, for smarter business decisions
Introduction to SQream and the IoT environment
A short talk given by Arnon Shimoni from SQream Technologies at the Internet Of Things (IoT) Israel (Tel Aviv-Yafo) meetup at the AWS loft
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook. One key feature in Presto is the ability to query data where it lives via an uniform ANSI SQL interface. Presto’s connector architecture creates an abstraction layer for anything that can be represented in a columnar or row-like format, such as HDFS, Amazon S3, Azure Storage, NoSQL stores, relational databases, Kafka streams and even proprietary data stores. Furthermore, a single Presto query can combine data from multiple sources, allowing for analytics across an entire organization.
Webinar - DreamObjects/Ceph Case Study
This document summarizes DreamObjects, an object storage platform powered by Ceph. It discusses the hardware used in storage and support nodes, including Intel and AMD processors, RAM, disks, and networking components. The document also provides details on Ceph configuration including replication, CRUSH mapping, OSD configuration, and application tuning. Monitoring tools discussed include Chef, pdsh, Sensu, collectd, graphite, logstash, Jenkins and future plans.
PyData Boston 2013
A description of Continuum Analytics projects focusing on the open source work. Includes descriptions of Wakari, Numba, Conda, Bokeh, Blaze, and CDX.
More Related Content
What's hot
ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...
Pragma Innovation is an IT services company focused on time series data solutions. Their PASS (Pragma Analytics Software Suite) allows companies to analyze, report on, and make decisions from time series network data using open source software. It is designed for ISPs, hosting providers, and telecom companies. The solution ingests network and log data, standardizes it, enriches it using tools like GeoIP, and stores it in a time series database. This allows customers to build applications for traffic engineering, security, and business intelligence use cases. Key challenges addressed in version 2.0 of the solution include data sampling, IPv4/IPv6 support, and using ClickHouse as the time series database for its performance and simplicity
Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...
The RCP/Rakuten Catalog Platform has been growing at a brisk speed over the last couple of years. Our original backbone was Cassandra. However, as they continued their growth, they internally started realizing that it was not suitable for our next stage of growth.
As such, they started looking into ScyllaDB as a better ROI solution as well as a much more stable backend. The migration itself was challenging since this has to be done for a production live data processing pipeline with minimal impact on customers. In this talk, Hitesh Shah, Engineering Manager at Rakuten USA will dive deeper into challenges and takeaways.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Kubernetes at Spreadshirt - First steps to production
This presentation describes how we at Spreadshirt got started on our adventure into Docker-land which finally led to introducing Kubernetes for container orchestration.
Scylla Summit 2022: New AWS Instances Perfect for ScyllaDB
In this talk AWS’ Ken Krupa, Head of Specialized Solutions Architecture, will describe the architecture and capabilities of two new AWS EC2 instance types perfect for data-intensive storage and IO-heavy workloads like ScyllaDB: the Intel-based I4i and the Graviton2-based I4g series.
The Intel Xeon Ice Lake-based I4i series provides unparalleled raw horsepower for your most demanding workloads. Meanwhile, the Graviton2-powered I4g instances provide lower cost per storage on a power-efficient platform to deploy your cloud-native applications.
Ken will also describe the AWS Nitro SSD, a new form of high-speed NVMe storage with a Flash Translation Layer built with Nitro controllers, which powers both of these instance families.
ScyllaDB VP of Product Tzach Livyatan will then share benchmarking results showing how ScyllaDB behaves under load on these two instance types, providing maximum system utility and efficiency.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
The Data Lake Engine Data Microservices in Spark using Apache Arrow Flight
Machine learning pipelines are a hot topic at the moment. Moving data through the pipeline in an efficient and predictable way is one of the most important aspects of running machine learning models in production.
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
The Home Depot is transforming its architecture to use microservices and polyglot persistence to handle increasing online order volumes of 250,000 lines per hour. Redis is being used to turbo charge existing monolithic applications by offloading pieces to new processes using patterns like caching, concurrency management, and powering algorithms. This improves performance by reducing database degradation and wait times by over 95%. Next steps include setting up Redis clusters on-premises and off-premises to further reduce database CPU usage and onboard more patterns.
Graph Databases at Netflix
In the slide deck, we describe how graph databases are used at Netflix. Graph databases can be faster than relational databases for deeply-connected data - a strength of the underlying model. We have used JanusGraph on top of Cassandra. Both technologies are Open Source.
Presto @ Uber Hadoop summit2017
This document discusses Presto, an interactive SQL query engine for big data. It describes how Presto is optimized to quickly query data stored in Parquet format at Uber. Key optimizations for Parquet include nested column pruning, columnar reads, predicate pushdown, dictionary pushdown, and lazy reads. Benchmark results show these optimizations improve Presto query performance. The document also provides an overview of Uber's analytics infrastructure, applications of Presto, and ongoing work to further optimize Presto and Hadoop.
Роман Новиков "Best Practices for MySQL Performance & Troubleshooting with th...
Troubleshooting performance issues can be a bit tricky, especially when you’re given a broad statement that the database is slow.
Learn to direct your attention to the correct moving pieces and fix what needs your attention.
Learn how all this is done at Percona, what we monitor and track, and the tools we use.
Understanding Storage I/O Under Load
There’s a popular misconception about I/O that (modern) SSDs are easy to deal with; they work pretty much like RAM but use a “legacy” submit-complete API. And other than keeping in mind a disk’s possible peak performance and maybe maintaining priorities of different IO streams there’s not much to care about. This is not quite the case – SSDs do show non-linear behavior and understanding the disk’s real abilities is crucial when it comes to squeezing as much performance from it as possible.
Diskplorer is an open-source disk latency/bandwidth exploring toolset. By using Linux fio under the hood it runs a battery of measurements to discover performance characteristics for a specific hardware configuration, giving you an at-a-glance view of how server storage I/O will behave under load.
ScyllaDB CTO Avi Kivity will share an interesting approach to measuring disk behavior under load, give a walkthrough of Diskplorer and explain how it’s used.
With the elaborated model of a disk at hand, it becomes possible to build latency-oriented I/O scheduling that cherry-picks requests from the incoming queue keeping the disk load perfectly Balanced.
ScyllaDB engineer Pavel Emelyanov will also present the scheduling algorithm developed for the Seastar framework and share results achieved using it.
MongoDB Capacity Planning
Deploying any software can be a challenge if you don't understand how resources are used or how to plan for the capacity of your systems. Whether you need to deploy or grow a single MongoDB instance, replica set, or tens of sharded clusters then you probably share the same challenges in trying to size that deployment.
Distributed Crypto-Currency Trading with Apache Pulsar
Apache Pulsar was developed to address several shortcomings of existing messaging systems including geo-replication, message durability, and lower message latency.
We will implement a multi-currency quoting application that feeds pricing information to a crypto-currency trading platform that is deployed around the globe. Given the volatility of the crypto-currency prices, sub-second message latency is critical to traders. Equally important is ensuring consistent quotes are available to all geographical locations, i.e the price of Bitcoin shown to a user in the USA should be the same as it to a trader in Hong Kong.
We will highlight the advantages of Apache Pulsar over traditional messaging systems and show how its low latency and replication across multiple geographies make it ideally suited for globally distributed, real-time applications.
Operationalizing MongoDB at AOL
The document summarizes Michael DelNegro's work operationalizing MongoDB at AOL over several years. Key aspects included establishing standards for host setup, directory structure, and build scripts; implementing monitoring with Argus, Nagios, and MMS; performing backups; sharing information through internal documentation; and addressing challenges like requiring more planning from developers. The operationalization effort helped support over 500 MongoDB servers storing data for 30+ active projects at AOL.
Presto GeoSpatial @ Strata New York 2017
The document discusses geospatial analytics at Uber using Presto. Presto is an interactive SQL query engine that allows for fast querying of large datasets. The document outlines how Presto was optimized for geospatial queries using quadtree indexing to efficiently filter spatial data and speed up queries involving spatial functions like ST_Contains. This improved query performance for geospatial queries in Presto by 60x compared to the previous approach using Hive.
Case study- Real-time OLAP Cubes
This case study describes an approach to building quasi real-time OLAP cubes in Microsoft SQL Server Analysis Services to enable daily comparisons of production forecasts and outcomes. The cubes are partitioned by time to allow independent and frequent updates. Initial attempts failed due to deadlocks from simultaneous partition updates. The working solution takes advantage of a 6 day work week by switching partition dates on Saturdays only and reprocessing partitions then. This allows real-time and historical partition updates without gaps or overlaps in the data.
Hadoop And Big Data - My Presentation To Selective Audience
This document provides an overview of Hadoop and big data. It discusses what Hadoop is, how it stores and processes large datasets using MapReduce, and its architecture. The presenter notes that Hadoop provides cost savings and productivity benefits by allowing companies to answer questions within large datasets. It also highlights the ROI of Hadoop, showing it can reduce storage and processing costs compared to traditional systems. Resources for learning more about Hadoop are provided at the end.
Enabling Presto Caching at Uber with Alluxio
Alluxio Day VI
October 12, 2021
https://www.alluxio.io/alluxio-day/
Speaker:
Curt Hu, Uber
Beinan Wang, Alluxio
Accelerating analytics in a new era of data
Organizations today produce exponentially more data than they did just a few years ago, but their databases weren’t built to handle these new volumes. As a result, reporting takes way too long, and some complex analytics simply cannot be done. The Era of Massive Data is upon us, and a new approach is required to overcome the limitations of traditional CPU-based data stores.
KEY TAKEAWAYS
- Flexible data exploration with minimal preparation
- Unrestricted access to your organization’s full scope of data
- Access to previously unobtainable insights, for smarter business decisions
Introduction to SQream and the IoT environment
A short talk given by Arnon Shimoni from SQream Technologies at the Internet Of Things (IoT) Israel (Tel Aviv-Yafo) meetup at the AWS loft
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook. One key feature in Presto is the ability to query data where it lives via an uniform ANSI SQL interface. Presto’s connector architecture creates an abstraction layer for anything that can be represented in a columnar or row-like format, such as HDFS, Amazon S3, Azure Storage, NoSQL stores, relational databases, Kafka streams and even proprietary data stores. Furthermore, a single Presto query can combine data from multiple sources, allowing for analytics across an entire organization.
What's hot (20)
ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...
ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...
Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...
Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...
Kubernetes at Spreadshirt - First steps to production
Kubernetes at Spreadshirt - First steps to production
Scylla Summit 2022: New AWS Instances Perfect for ScyllaDB
Scylla Summit 2022: New AWS Instances Perfect for ScyllaDB
The Data Lake Engine Data Microservices in Spark using Apache Arrow Flight
The Data Lake Engine Data Microservices in Spark using Apache Arrow Flight
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
Роман Новиков "Best Practices for MySQL Performance & Troubleshooting with th...
Роман Новиков "Best Practices for MySQL Performance & Troubleshooting with th...
Distributed Crypto-Currency Trading with Apache Pulsar
Distributed Crypto-Currency Trading with Apache Pulsar
Hadoop And Big Data - My Presentation To Selective Audience
Hadoop And Big Data - My Presentation To Selective Audience
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Presto: SQL-on-Anything. Netherlands Hadoop User Group Meetup
Similar to Boost on!!next generation big data platform
Webinar - DreamObjects/Ceph Case Study
This document summarizes DreamObjects, an object storage platform powered by Ceph. It discusses the hardware used in storage and support nodes, including Intel and AMD processors, RAM, disks, and networking components. The document also provides details on Ceph configuration including replication, CRUSH mapping, OSD configuration, and application tuning. Monitoring tools discussed include Chef, pdsh, Sensu, collectd, graphite, logstash, Jenkins and future plans.
PyData Boston 2013
A description of Continuum Analytics projects focusing on the open source work. Includes descriptions of Wakari, Numba, Conda, Bokeh, Blaze, and CDX.
John adams talk cloudy
My presentation at Microsoft's Mountain View Conference Center during the "Talk Cloudy to me" day.
Thanks to Sebastian and Scalr for a great day.
Chirp 2010: Scaling Twitter
This document summarizes a keynote speech given by John Adams, an early Twitter engineer, about scaling Twitter operations from 2008-2009. Some key points:
1) Twitter saw exponential growth rates from 2008-2009, processing over 55 million tweets per day and 600 million searches per day.
2) Operations focused on improving performance, reducing errors and outages, and using metrics to identify weaknesses and bottlenecks like network latency and database delays.
3) Technologies like Unicorn, memcached, Flock, Cassandra, and daemons were implemented to improve scalability beyond a traditional RDBMS and handle Twitter's data volumes and real-time needs.
4) Caching,
GCP for AWS Professionals
This meeting we'll host a discussion on Google Cloud Platform and Amazon Web Services to bring light to similarities and differences between platforms. If you have questions about how our platforms compare this is the meeting to attend!
Know thy cost (or where performance problems lurk)
Performance happens. Whether you're designed for it or not it doesn’t matter, she is always invited to the party (and you better find her in a good mood). Knowing the cost of every operation, and how it distributes on every subsystem will ensure that when you are building that proof-of-concept (that always ends up in production) or designing the latest’s enterprise-grade application; you will know where those pesky performance bugs like to inhabit. In this session, we will go deep into the inner working of every performance sensitive subsystem. From the relative safety of the client to the binary world of Voron.
Atom: A cloud native deep learning platform at Supremind
Supremind developed a deep learning platform called Atom to manage large-scale training data and models across multiple clouds. Atom uses Alluxio for data orchestration to provide a unified view of distributed data and optimize for data locality. Atom addresses challenges of metadata management, data optimization, and supporting both training and inference workloads. Future work includes improving distributed training, model optimization, and adapting Atom to various hardware platforms.
Netflix oss season 2 episode 1 - meetup Lightning talks
The lightning talks covered various Netflix OSS projects including S3mper, PigPen, STAASH, Dynomite, Aegisthus, Suro, Zeno, Lipstick on GCE, AnsWerS, and IBM. 41 projects were discussed and the need for a cohesive Netflix OSS platform was highlighted. Matt Bookman then gave a presentation on running Lipstick and Hadoop on Google Cloud Platform using Google Compute Engine and Cloud Storage. He demonstrated running Pig jobs on Compute Engine and discussed design considerations for cloud-based Hadoop deployments. Finally, Peter Sankauskas from @Answers4AWS discussed initial ideas around CloudFormation for Asgard and deploying various Netflix OSS
Building a High Performance Analytics Platform
The document discusses using flash memory to build a high performance data platform. It notes that flash memory is faster than disk storage and cheaper than RAM. The platform utilizes NVMe flash drives connected via PCIe for high speed performance. This allows it to provide in-memory database speeds at the cost and density of solid state drives. It can scale independently by adding compute nodes or storage nodes. The platform offers a unified database for both real-time and analytical workloads through common APIs.
From a student to an apache committer practice of apache io tdb
This talk is introduce by Xiangdong Huang, who is a PPMC of Apache IoTDB (incubating) project, at Apache Event at Tsinghua University in China.
About the Event:
The open source ecosystem plays more and more important role in the world. Open source software is widely used in operating systems, cloud computing, big data, artificial intelligence, and industrial Internet. Many companies have gradually increased their participation in the open source community. Developers with open source experience are increasingly valued and favored by large enterprises. The Apache Software Foundation is one of the most important open source communities, contributing a large number of valuable open source software and communities to the world.
The invited guests of this lecture are all from ASF community, including the chairman of the Apache Software Foundation, three Apache members, Top 5 Apache code committers (according to Apache annual report), the first Committer in the Hadoop project in China, several Apache project mentors or VPs, and many Apache Committers. They will tell you what the open source culture is, how to join the Apache open source community, and the Apache Way.
Silicon Valley Code Camp 2016 - MongoDB in production
What to look for when deploying MongoDB in Production, and a list of the most common issues reported by customers to the support team
IPv4 IPv6 Media Player
Introduction of IPv4 IPv4 Multi CDN Media Player and Status of speed comparisons of IPv4 and IPv6 in Japan
Spotify: Data center & Backend buildout
In this talk Emil Fredriksson and David Poblador i Garcia explain how Spotify builds its infrastructure in order to deliver millions of songs to millions of users.
We explain how we manage to support our development teams to build features by developing a highly scalable infrastructure.
Games Industry Analytics Forum 2 - Plumbee
This document outlines the six steps Plumbee took to build their analytics capabilities from scratch. They started with third-party analytics to get data fast but found it lacked flexibility. They then collected everything, processing vast amounts of raw data with Elastic MapReduce and Hive. Over time, they automated processing and built a relational data mart for faster queries. While making progress, they have more work ahead, such as replacing custom tools with open-source solutions like Flume and Cascading to improve scalability and enable data science capabilities.
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Twitter's operations team manages software performance, availability, capacity planning, and configuration management for Twitter. They use metrics, logs, and analysis to find weak points and take corrective action. Some techniques include caching everything possible, moving operations to asynchronous daemons, and optimizing databases to reduce replication delay and locks. The team also created several open source projects like CacheMoney for caching and Kestrel for asynchronous messaging.
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter and Finding your own Fail Whale document discusses Twitter operations. The operations team manages software performance, availability, capacity planning, and configuration management using metrics, logs, and data-driven analysis to find weak points and take corrective action. They use managed services for infrastructure to focus on computer science problems. The document outlines Twitter's rapid growth and challenges in maintaining performance as traffic increases. It provides recommendations around caching, databases, asynchronous processing, and other techniques Twitter uses to optimize performance under heavy load.
Fixing twitter
Fixing Twitter and Finding your own Fail Whale document discusses Twitter operations. The Twitter operations team focuses on software performance, availability, capacity planning, and configuration management using metrics, logs, and science. They use a dedicated managed services team and run their own servers instead of cloud services. The document outlines Twitter's rapid growth and challenges in maintaining performance. It discusses strategies for monitoring, analyzing metrics to find weak points, deploying changes, and improving processes through configuration management and peer reviews.
Fixing_Twitter
Twitter's operations team manages software performance, availability, capacity planning, and configuration management. They use metrics, logs, and analysis to find weak points and take corrective action. Some techniques include caching everything possible, moving operations to asynchronous daemons, optimizing databases, and instrumenting all systems. Their goal is to process requests asynchronously when possible and avoid overloading relational databases.
Nagios XI Best Practices
Best Practices? That’s like asking how long is a piece of string! While every environment is different, there are however a number of configurations, tweaks and methods that can be of great benefit for your Nagios XI environment. This talk will cover a variety of Best Practice topics for Nagios XI ranging from flexible object configurations through to back end performance enhancements.
Rubyslava + PyVo #48
Skypicker is a flight ticket search and booking engine that processes hundreds of terabytes of airline data monthly and sells thousands of tickets daily in Europe, Russia, and China. It uses PostgreSQL extensively to power its API and databases. The document discusses Skypicker's PostgreSQL infrastructure, which includes 5 database clusters with over 0.5TB of memory each that handle 20 million updates per hour. It also covers optimizations made to scale the databases, such as table partitioning, cascading replication, and query tuning. Skypicker is currently preparing for a large increase in data and is hiring database masters and developers.
Similar to Boost on!!next generation big data platform (20)
Know thy cost (or where performance problems lurk)
Know thy cost (or where performance problems lurk)
Atom: A cloud native deep learning platform at Supremind
Atom: A cloud native deep learning platform at Supremind
Netflix oss season 2 episode 1 - meetup Lightning talks
Netflix oss season 2 episode 1 - meetup Lightning talks
From a student to an apache committer practice of apache io tdb
From a student to an apache committer practice of apache io tdb
Silicon Valley Code Camp 2016 - MongoDB in production
Silicon Valley Code Camp 2016 - MongoDB in production
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
More from LINE Corporation
JJUG CCC 2018 Fall 懇親会LT
JJUG CCC 2018 Fall was held in Fukuoka on December 15th. The event featured talks about LINE KYOTO, Java, DateTimeFormatter, JavaDoc, Java generics, and LINE Growth Technology. LINE Growth Technology aims to help LINE grow through technology.
Reduce dependency on Rx with Kotlin Coroutines
This document discusses replacing RxJava with Kotlin Coroutines for asynchronous programming. It provides an overview of RxJava and Coroutines, compares their approaches, and shows how to write asynchronous code using Coroutines instead of RxJava. It also discusses how to integrate Coroutines with Retrofit and the MVVM pattern.
Use Kotlin scripts and Clova SDK to build your Clova extension
The document discusses using Kotlin scripts to create a Clova client. It shows how to evaluate Kotlin code from a script using a ScriptEngine to define a Clova client configuration with launch, intent, and session ended handlers. The Clova client created in the script can then be used to handle Clova requests and responses.
The Magic of LINE 購物 Testing
LINE Shopping provides an e-commerce platform in Taiwan. It has over 9 million monthly visitors, a 40% repurchase rate, and lists over 26 million products from over 1,300 brands. The document discusses how to test the LINE Shopping platform, including unit, API, and UI tests. It also describes tools like Just-API and Pyresttest that can be used to test GraphQL and REST APIs respectively using YAML configuration files.
GA Test Automation
This document discusses automating Google Analytics (GA) testing for LINE TODAY. It proposes using Robotframework with Appium to simulate user actions in the LINE app and confirm that GA events are recorded correctly. It provides details on initializing the GA Reporting API to retrieve reports and examples of dimensions, metrics, and sample report requests. Code snippets demonstrate how to set up a service account and get credentials to access the GA Reporting API. The goal is to test new features for side effects and avoid human errors by automating the process of validating GA events.
UI Automation Test with JUnit5
This document provides an overview of UI automation testing with JUnit 5. It introduces the JUnit 5 framework, including its architecture, annotations, extension model, parameter resolver, and life cycle. It also discusses how to configure JUnit 5 in Gradle projects. Additionally, it briefly mentions other tools that can be used for UI testing, such as Appium, Ayachan, Ayavue, and image recognition libraries. The document aims to help people understand and get started with JUnit 5 for UI test automation.
Feature Detection for UI Testing
The document summarizes a test night event held by LINE Fukuoka to discuss UI test automation. The event covered testing browser, iOS, and Android applications using Selenium for browsers and Appium for mobile. Attendees learned about template matching and feature detection techniques for UI testing, including pixel-perfect template matching versus feature detection which is scale invariant and can match elements that are 30-200% different in size. The techniques discussed were demonstrated using A-KAZE feature detection with OpenCV3 in Java.
日本開發者大會短講分享
This document proposes building a LINE app that provides a customized interface for the LINE TODAY news service using web views. It discusses three versions of the app with increasing features:
v1.0 uses customized web views for all pages except onboarding and login. v2.0 adds easier navigation with a bottom navigation bar. v3.0 enhances video with native video pages and a player. The document also discusses using Apache Kafka to build secondary indices for the app's database to enable features like retrieving a user's past posts.
LINE Chatbot - 活動報名報到設計分享
This document discusses using a LINE registration chatbot for an event. The chatbot allows registration, check-in, and provides information about event activities online or offline. Users can access the chatbot through the LINE app on their mobile device or desktop website. It uses technologies like beacons, QR codes, and rich menus to enable simple registration and check-in as well as interact with information booths at the event.
在 LINE 私有雲中使用 Managed Kubernetes
This document introduces a managed Kubernetes as a service (KAAS) that provides concise summaries in 3 sentences or less that provide the high level and essential information from the document. The KAAS addresses problems with different tooling, versions, and configurations across clusters by providing a standardized Kubernetes platform. It aims to reduce operation costs and improve quality by ensuring high availability, performance optimization, and private cloud integration. The KAAS leverages Rancher for declarative operations and integrates custom controllers to enable load balancing, persistent storage, and other private cloud services within Kubernetes clusters.
LINE TODAY高效率的敏捷測試開發技巧
This document discusses DevOps practices for software testing. It emphasizes the importance of continuous testing throughout the development lifecycle to reduce risk and ensure new features do not break existing functionality. Testing approaches like unit testing, integration testing, and automation are recommended to support faster release cycles and more agile workflows. The document concludes by advertising open roles for QA engineers.
LINE 區塊鏈平台及代幣經濟 - LINK Chain及LINK介紹
This document discusses LINE's plans to introduce a token economy using its own cryptocurrency called LINK. It proposes that LINK can help evolve the relationship between users and services by creating a global platform not restricted by national borders. The three key aspects of the LINK ecosystem are that it will use a single token for all dApps and services, LINK tokens will be issued as rewards for contributions to the ecosystem, and LINE will offer a blockchain platform called LINK Chain to make dApp development and use more user-friendly. The goal is for LINK to facilitate a connected digital economy across LINE's various services and applications.
LINE Things - LINE IoT平台新技術分享
This document summarizes LINE Things, a platform that allows devices to connect and communicate through LINE using Bluetooth Low Energy (BLE). It discusses how LINE Things supports both online and offline devices. For offline devices, the LINE app acts as a proxy to allow communication between devices and services via BLE and web APIs. It also introduces LINE Things LIFF BLE, which allows BLE communication between devices and LIFF apps using the LIFF SDK BLE plugin. Developers can use LIFF BLE to easily build apps to read, write, and receive notifications from connected BLE devices.
LINE Pay - 一卡通支付新體驗
1. LINE Pay is a digital wallet service that allows users to make payments and transfers between accounts without fees.
2. The service has over 17 million registered electronic payment cards.
3. Users can make transfers between LINE Pay accounts, split bills with groups of people, and pay for transportation, goods, services, and bills directly from their LINE Pay accounts.
LINE Platform API Update - 打造一個更好的Chatbot服務
The document summarizes new features and services released by LINE in 2018 to improve messaging experiences and build better bots and services. Key releases included Flex Messages, LIFF apps, quick replies, and video messages. It also discusses how developers can utilize social APIs, personalization, and audiences to engage and notify users.
More from LINE Corporation (20)
Use Kotlin scripts and Clova SDK to build your Clova extension
Use Kotlin scripts and Clova SDK to build your Clova extension
Recently uploaded
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentationsPresentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Mind map of terminologies used in context of Generative AI
Mind map of common terms used in context of Generative AI.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
20 Comprehensive Checklist of Designing and Developing a Website
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Recently uploaded (20)
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Boost on!!next generation big data platform
- 1. BOOST ON!! Next Generation Big Data Platform Neil Tu Data Architect, Data Labs LINE
- 2. Neil Tu (杜佐民) • Expertise in Distributed Platform, Data Lake, Data Warehouse...etc. • Image Processing, Computer Vision, Pattern Recognition with 5+ years experience • Career Experience: tsmc, Rakuten
- 3. Agenda • Data Labs • Next Generation Data Platform 1. Datalake Nifi 2. Datalake Hadoop
- 4. Data Labs What We Do • Big Data • Data Analysis How We Do • Mathematic Modeling • A/B Test • Machine Learning • Deep Learning • Etc.
- 5. Data Labs
- 6. Data Labs
- 7. What We Use
- 8. Next Generation Data Platform
- 9. Next Generation Data Platform Data Storage ETL
- 11. Datalake Nifi Benefit • UI Interface • Authentication and Authorization • Local Backup Platform Info • 5 Servers • Each Server - CPU: 20 - Mem: 64GB - Network: 10G - Disk: SAS 2.4TB The reason we use nifi
- 12. Execute Script • Python → Groovy 3 Times Merging • Reduce NN’s burden • Reduce I/O JSON → Avro • Formatted Avro → ORC • Boost Up • Runtime Query File Size • Hourly(peak): 210GB → 15GB • Daily(avg.): 3500GB → 250GB Datalake NifiTuning Execute Script 1st Merging 2nd Merging 3rd Merging JSON to Avro Avro to ORC Put into HDFS Consume Kafka
- 13. Datalake Hadoop
- 14. Identification Security • Kerberos (Authentication) • LDAP (Authentication) • Apache Ranger (Authorization) Identity • Natural Person • Group • System User • System Group
- 15. SecurityTop View
- 16. SecuritySideView Gateway Server Client Server Local PC Office Authentication Datalake Authentication Hadoop cli Hive cli Hiveserver2 Presto
- 17. Thank You
Editor's Notes
- 皆さんこんばんは、近年ビッグデータが人気になっております。世の中にも色んなソルーションが増えて来て、今夜うち組織が利用してる基盤技術を紹介したいと思います。 オンプレ
- 台灣台北出生 2012年まで画像処理などの研究開発が行われました。 2012年楽天入社、全社最大級レベルクラスタを作りました、今まだ利用されてます。 2016にLINE入社、今はdata lake, data warehouseなどの分散システムを構築運用してます 2分
- 写真を撮っても良いですけど、終わったらすぐにスライドを公開します。 3分
- データ研究室 色なデータ分析の専門家、DSがいます。今大体20名以上のDSが所属してます。 What we do? 色々データから価値があるもの、サービスを改善できるものを探してます。 How we do? 数学モデリング…etc. 5分
- 色々LINE family serviceから、色な国からデータを収集してます。
- サービスからの生log、生データを集計しながら貯まっています。 エンジニアがデータを加工整理してdata warehouseの形で保存してます。 このdata wareshouseのデータを利用して、レポートを出すことが出来ます。 DSもこのデータを分析することが出来ます。 7分
- これは現在のアーキテクチャです。 黄色のどころが今夜紹介した部分です。 破線はバッチ処理です、hourly, daily, weekly, monthly、いくつ種類があります。 実線はrealtime処理です。 8分
- 従来のデータ集計方法は二つがあります。 一つ目はサービスDBからバッチ処理でデータを取ります。 このサービスDBはmysqlだけではなく、mongoDB、HBASE、HADOOPなど 二つ目はkafkaのreal time処理です。 でも今夜は三つ目を紹介したいお思います。 Apache nifi~ Real time hive query 10分
- Flinkはreal time queryが出来ません。 ESはreal timeが出来ますけど、でもtable間のjoinなどのuse caseはやや難しい もしnifiを利用して、real time hive tableを作ると、上に二つ方法が足りない部分をカバー出来ます。 Hive tableがreal time化を実現すると、real time dataをreporting or data analysisに提供が出来ます。 12分
- 14分
- 17分
- 個人単位で全社公開してます Tableau, cognos 18分
- 独自のKerberosとLDAP 20分
- 21分
- 23分