M3D - Metadata Driven Development

•

0 likes•46 views

M3D stands for Metadata Driven Development and is a cloud and platform-agnostic framework for the automated creation, management, and governance of metadata and data flows from multiple sources to multiple target systems. It is a tool comprised of two components: m3d-API and m3d-engine. At Adidas the framework is used for the creation of data lake environments, management and governance of metadata, data flows from multiple sources and algorithms as data frame transformations. It is feeding 500+ tables and over 900 views. Author: Jorge Cespedes (Adidas)

Technology

Content
• What is M3D?
• M3D Components
• Why we need it?
• M3D @ Adidas
• What’s to come?

What is it?
Framework
Cloud and platform agnostic
Easy development of new feats
Multiple source/target system
Creation of data lake envs

M3D Components
M3D Engine M3D API
https://github.com/adidas/m3d-engine https://github.com/adidas/m3d-api

Features in action!
Creation of data lake envs
Inbound
Raw
File
Raw
File
Raw
File
Raw
File
Raw
File
Raw
File

Features in action!
Creation of data lake envs
Inbound
Landing
Raw
File
Raw
File
Raw
File
Raw
File
Raw
File
Raw
File

Features in action!
Creation of data lake envs
Inbound
Landing
Lake
Parqu
et
Parqu
et
Parqu
et
Parqu
et
Parqu
et
Parqu
et

Features in action!
Creation of data lake envs
Inbound
Landing
Lake Out
View
View View
View
Lake

Features in action!
Inbound
Landing
Lake Out
Lake
Multiple source/target system
M3D
Target
Exasol
Target Y
Target
Target Z
AWS
Cloud and platform agnostic
Cloudera

Features in action!
Inbound
Landing
Lake
Parqu
et
Parqu
et
Parqu
et
Parqu
et
Parqu
et
Parqu
et
Easy development of new feats
Easily define new
readers/writers based on traits
Introduce transformations on
datasets
Add new APIs to M3D Api

Why We Need It?
Simplification of definition of new tables in lake
Easy way to schedule loads
Clear semantics defined globally
Single extraction from source systems
Simple definition of views as interface for applications

M3D @ Adidas
We have the so called “Cockpit” for definition of global semantics
Orchestration with Jenkins as a Service
Loading data to lake (example):
Place files in
Inbound layer
python m3d_main.py -function
create_emr_cluster -
core_instance_type m4.large -
master_instance_type m4.large -
core_instance_count 3 -
destination_system emr -
destination_database emr_database
-destination_environment test -
config config.json -emr_version emr-
5.23.0
python m3d_main.py -function
create_table -config config.json -
destination_system emr -
destination_database emr_database -
destination_environment test -
destination_table table_name -
emr_cluster_id id-of-started-cluster
python m3d_main.py -function load_table -config config.json -
destination_system emr -destination_database emr_database -
destination_environment test -destination_table table_name -
load_type FullLoad -emr_cluster_id id-of-started-cluster

M3D @ Adidas
9 TB In Memory Lake Size
150 TB Lake Size on Big Data Platform
500+ tables in Data Lake
> 900 views consumed by Products

What’s to come?
Airflow as orchestration for M3D
Publish M3D Api as library
UI for quick setup of M3D Api/Engine
PySpark version for M3D engine

This document discusses using Docker and Docker Compose to build resilient data pipelines. It provides an example use case of processing news articles from different data sources and categories. The system is designed with microservices running in Docker containers that can interact and scale easily. Docker Compose is used to define and run the multi-container application. The keys to the design are separating each component into its own container to enable independent development, testing, and deployment across cloud infrastructure using Docker images.

(BDT205) Your First Big Data Application on AWS | AWS re:Invent 2014

Amazon Web Services

This document outlines steps for collecting, processing, and analyzing log data using Amazon Web Services. It describes launching an EMR cluster with Hive, creating a Kinesis stream, ingesting log data into Hive tables from Kinesis, running queries, and exporting results to S3 and Redshift. The process includes automating multiple iterations of querying and partitioning output by date or other fields.

Natural Language Query and Conversational Interface to Apache Spark

Databricks

Apache Spark has been a great technology for processing and analyzing Big Data. However, it is not accessible to business users, who don’t have technical or programming skills. In this talk, I’ll talk about recent efforts in the space of “Conversational analytics”. This paradigm allows any user to ask text and voice questions, in natural language, of their data to a bot and receive back a natural language and visual result. A key technology is natural language to SQL translation, where we translate natural language queries from a user into Spark SQL queries that can go against a Databricks system, and that can be easily trained on different schemas and databases. This NLP technology needs to be further combined with dialog management, natural-language generation/narration, data understanding and modeling, augmented analytics and automated visualization generation in order to achieve the goal of “Conversational Analytics”. Using such a technology, a user can ask, in plain English, “How many cases of Covid were there in the last 2 months in states that had no social distancing mandates by type of transmission”, and then dig deeper into the results in a conversational manner to uncover hidden insights from Covid datasets in a Spark instance. We believe that having access to such data and insights at their fingertips can help users make appropriate decisions quickly, improve data literacy and even overcome the scourge of fake news for the general public.

Configuration management

Luca De Vitis

This document discusses configuration management and provides an overview of Saltstack. It describes how configuration management addresses the challenges of managing hundreds of assets across multiple datacenters. It then summarizes the key components of Saltstack, including states, grains, pillars, roster, mine, job management, and reactors/events. States define the desired configuration of hosts, grains contain static host data, pillars provide variable host data, and roster facilitates remote management. Mine collects arbitrary host data, jobs manage scheduled tasks, and reactors respond to events. The document also briefly discusses syndication, proxies, Salt Virt for virtual machines, and Salt Cloud for cloud infrastructure.

Making KVS 10x Scalable

Sadayuki Furuhashi

1) The document proposes making a key-value storage system (CDP KVS) 10 times more scalable to support real-time data delivery. 2) Three ideas are presented: using an alternative distributed KVS, implementing a storage hierarchy on the existing KVS, and shipping edit logs to indexed archives. 3) The storage hierarchy approach of partitioning, compressing, and writing data to DynamoDB in batches is selected as it improves write performance and reduces storage costs while remaining stateless.

Fluentd and Docker - running fluentd within a docker container

Treasure Data, Inc.

Up and running with pyspark

Krishna Sangeeth KS

This document provides an overview of Apache Spark, including why it was created, how it works, and how to get started with it. Some key points: - Spark was initially developed at UC Berkeley as a class project in 2009 to test cluster management systems like Mesos, and was later open sourced in 2010. It became an Apache project in 2014. - Spark is faster than Hadoop for machine learning tasks because it keeps data in-memory between jobs rather than writing to disk, and has a smaller codebase. - The basic unit of data in Spark is the resilient distributed dataset (RDD), which allows immutable, distributed collections across a cluster. RDDs support transformations and actions. -

RedisConf17 - Doing More With Redis - Ofer Bengal and Yiftach Shoolman

Redis Labs

This document summarizes RedisConf 2017, covering several topics: 1. Running Redis on Flash in a DBaaS model for improved performance and cost savings compared to other NoSQL databases. 2. Redis modules gaining momentum with over 50 created so far, and the importance of multi-threading for high performance. Useful modules highlighted include RediSearch, ReJSON, Redis-ML, and Redis-Graph. 3. Using Redis for IoT applications, with challenges around small edge devices and clusters, high throughput from thousands of devices, and varied functionality needs addressed through Redis modules.

This document summarizes a presentation about using Embulk for machine learning and natural language processing. The presenter introduced themselves and their company, which uses Embulk for tasks like creating machine learning data, executing machine learning on the cloud, and natural language processing. They have developed several Embulk plugins for tasks like morphological analysis, Unicode normalization, and integrating with APIs from Amazon Machine Learning, Google Vision, and other cloud services. They discussed use cases for various machine learning and NLP tasks as well as limitations and capabilities of different cloud APIs.

Fluentd - Flexible, Stable, Scalable

Shu Ting Tseng

Overview of the Hive Stinger Initiative

Modern Data Stack France

Meet Solr For The Tirst Again

Varun Thacker

Ali Asad Lotia (DevOps at Beamly) - Riemann Stream Processing at #DOXLON

Outlyer

Video: http://youtu.be/a1r2bpGQbBQ A talk about using Riemann for stream processing. This is the metric aficionados on-premise tool of choice currently. Ali will talk about how they are using it to process the metrics coming out of their cloud service. For more info see : http://riemann.io Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London/ Follow DOXLON on twitter: twitter.com/doxlon

Keeping Spark on Track: Productionizing Spark for ETL

Databricks

ETL is the first phase when building a big data processing platform. Data is available from various sources and formats, and transforming the data into a compact binary format (Parquet, ORC, etc.) allows Apache Spark to process it in the most efficient manner. This talk will discuss common issues and best practices for speeding up your ETL workflows, handling dirty data, and debugging tips for identifying errors. Speakers: Kyle Pistor & Miklos Christine This talk was originally presented at Spark Summit East 2017.

RethinkDB - the open-source database for the realtime web

Alex Ivanov

This document summarizes RethinkDB, an open-source database for building realtime web applications. It describes key features of RethinkDB such as its push model that allows applications to receive live updates from the database via changefeeds without polling, its query language, data modeling options using embedded documents or references, and how it can be used to build collaborative, multiplayer and analytics applications.

Fluentd at Bay Area Kubernetes Meetup

Sadayuki Furuhashi

Fluentd is a log collection tool that is well-suited for container environments. It allows for flexible log collection from containers through its variety of input plugins. Logs can be aggregated and buffered by Fluentd before being sent to output destinations like Elasticsearch. This addresses problems with traditional log collection in container environments by decoupling log collection from applications and making the infrastructure more scalable and reliable.

SparkR - Play Spark Using R (20160909 HadoopCon)

wqchen

DOD 2016 - Rafał Kuć - Building a Resilient Log Aggregation Pipeline Using El...

PROIDEA

The document discusses building a resilient log aggregation pipeline using Elasticsearch and Kafka. It recommends using Kafka as a centralized buffer due to its scalability, fault tolerance, and streaming capabilities. Daily or size-based indices in Elasticsearch are preferable to a single large index. The document also provides optimization strategies for Elasticsearch, Kafka, and log shipping, including maintaining separate hot and cold tiers and properly configuring resources for data, master and ingest nodes.

Redis: REmote DIctionary Server

Ezra Zygmuntowicz

Distcp gobblin

Vasanth Rajamani

Distcp-ng is a tool for replicating massive datasets between Hadoop compatible file systems using Gobblin. It provides features like continuous replication of datasets, efficient file listing to reduce filesystem calls, dataset awareness for prioritization and notifications, failure isolation, and operational metrics. The architecture allows for copying between different source and target systems like HDFS, SFTP, and Hive. It also supports transformations, atomic publishing, and Hive table registration during copies. Future work includes building Distcp as a continuous replication service.

MySQL Slow Query log Monitoring using Beats & ELK

YoungHeon (Roy) Kim

This document provides instructions for using Filebeat, Logstash, Elasticsearch, and Kibana to monitor and visualize MySQL slow query logs. It describes installing and configuring each component on appropriate servers to ship MySQL slow logs from database servers to Logstash for processing, indexing to Elasticsearch for search and analysis, and visualization of slow query trends and details in Kibana dashboards and graphs.

MongoFr : MongoDB as a log Collector

Pierre Baillet

Automating Workflows for Analytics Pipelines

Sadayuki Furuhashi

This document discusses automating analytics pipelines and workflows using a workflow engine. It describes the challenges of managing workflows across multiple cloud services and database technologies. It then introduces a multi-cloud workflow engine called Digdag that can automate workflows, handle errors, enable parallel execution, support modularization and parameterization. Examples are given of using Digdag to define and run workflows across services like BigQuery, Treasure Data, Redshift, and Tableau. Key features of Digdag like loops, parameters, parallel tasks and pushing workflows to servers with Docker are also summarized.

Hw09 Building Data Intensive Apps A Closer Look At Trending Topics.Org

Cloudera, Inc.

The document describes TrendingTopics.org, a website that detects trending topics using Wikipedia page view data analyzed with Hadoop on Amazon EC2. It loads over 1TB of Wikipedia page view logs into Hadoop on EC2, uses Hive and Python to perform daily timelines and trend detection, and hosts the results on a Ruby on Rails front end also running on EC2. All the code is open source and hosted on Github.

Big data Lambda Architecture - Batch Layer Hands On

hkbhadraa

Building Hadoop Data Applications with Kite

huguk

ETL with SPARK - First Spark London meetup

Rafal Kwasny

The document discusses how Spark can be used to supercharge ETL workflows by running them faster and with less code compared to traditional Hadoop approaches. It provides examples of using Spark for tasks like sessionization of user clickstream data. Best practices are covered like optimizing for JVM issues, avoiding full GC pauses, and tips for deployment on EC2. Future improvements to Spark like SQL support and Java 8 are also mentioned.

Fast real-time approximations using Spark streaming

huguk

By Kevin Schmidt (Head of Data Science at Mind Candy) Luis Vicente (Senior Data Engineer at Mind Candy) For mobile games, constant tweaks are the difference between success and failure. Data and analytics have to be available in real-time, but calculating, for example, uniqueness or newness of a data point requires a list of seen data points - both memory intensive and tricky when using real-time stream processing like Spark Streaming. Probabilistic data structures allow approximation of these properties with a fixed memory representation, and are very well suited for this kind of stream processing. Getting from the theory of approximation to a useful metric at a low error rate even for many millions of users is another story. In our talk we will look at practical ways of achieving this: which approximation we used for selection of useful metrics, why we picked a specific probabilistic data structure, how we stored it in Cassandra as a time series and how we implemented it in Spark Streaming.

Building a Scalable Asset Management (DAM) Platform in the AWS

Rahul Shukla

This document outlines the architecture for building a scalable digital asset management (DAM) platform in the cloud. Key components include Amazon S3 for storage, auto scaling EC2 instances for processing metadata and generating renditions, DynamoDB for the catalog, CloudSearch for search, and Elastic Transcoder for transcoding. The architecture provides ingest of assets from S3, metadata extraction using EC2 workers, generation of renditions, building the catalog in DynamoDB and CloudSearch, and delivery of assets through CloudFront.

Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...

Amazon Web Services

With the breadth of AWS services available that are relevant to digital media, organizations can readily build out complete content/asset management (DAM/MAM/CMS) solutions in the cloud. This session provides a detailed walkthrough for implementing a scalable, rich-media asset management platform capable of supporting a variety of industry use cases. The session includes code-level walkthrough, AWS architecture strategies, and integration best practices for content storage, metadata processing, discovery, and overall library management functionality—with particular focus on the use of Amazon S3, Amazon Elastic Transcoder, Amazon DynamoDB and Amazon CloudSearch. Customer case study will highlight successful usage of Amazon CloudSearch by PBS to enable rich discovery of programming content across the breadth of their network catalog.

What's hot

Embulk and Machine Learning infrastructure

Hiroshi Toyama

Fluentd - Flexible, Stable, Scalable

Shu Ting Tseng

Overview of the Hive Stinger Initiative

Modern Data Stack France

Meet Solr For The Tirst Again

Varun Thacker

Ali Asad Lotia (DevOps at Beamly) - Riemann Stream Processing at #DOXLON

Outlyer

Keeping Spark on Track: Productionizing Spark for ETL

Databricks

RethinkDB - the open-source database for the realtime web

Alex Ivanov

Fluentd at Bay Area Kubernetes Meetup

Sadayuki Furuhashi

SparkR - Play Spark Using R (20160909 HadoopCon)

wqchen

DOD 2016 - Rafał Kuć - Building a Resilient Log Aggregation Pipeline Using El...

PROIDEA

Redis: REmote DIctionary Server

Ezra Zygmuntowicz

Distcp gobblin

Vasanth Rajamani

MySQL Slow Query log Monitoring using Beats & ELK

YoungHeon (Roy) Kim

MongoFr : MongoDB as a log Collector

Pierre Baillet

Automating Workflows for Analytics Pipelines

Sadayuki Furuhashi

Hw09 Building Data Intensive Apps A Closer Look At Trending Topics.Org

Cloudera, Inc.

Big data Lambda Architecture - Batch Layer Hands On

hkbhadraa

Building Hadoop Data Applications with Kite

huguk

ETL with SPARK - First Spark London meetup

Rafal Kwasny

Fast real-time approximations using Spark streaming

huguk

What's hot (20)

Embulk and Machine Learning infrastructure

Fluentd - Flexible, Stable, Scalable

Overview of the Hive Stinger Initiative

Meet Solr For The Tirst Again

Ali Asad Lotia (DevOps at Beamly) - Riemann Stream Processing at #DOXLON

Keeping Spark on Track: Productionizing Spark for ETL

RethinkDB - the open-source database for the realtime web

Fluentd at Bay Area Kubernetes Meetup

SparkR - Play Spark Using R (20160909 HadoopCon)

DOD 2016 - Rafał Kuć - Building a Resilient Log Aggregation Pipeline Using El...

Redis: REmote DIctionary Server

Distcp gobblin

MySQL Slow Query log Monitoring using Beats & ELK

MongoFr : MongoDB as a log Collector

Automating Workflows for Analytics Pipelines

Hw09 Building Data Intensive Apps A Closer Look At Trending Topics.Org

Big data Lambda Architecture - Batch Layer Hands On

Building Hadoop Data Applications with Kite

ETL with SPARK - First Spark London meetup

Fast real-time approximations using Spark streaming

Similar to M3D - Metadata Driven Development

Building a Scalable Asset Management (DAM) Platform in the AWS

Rahul Shukla

Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...

Amazon Web Services

Architetture serverless e pattern avanzati per AWS Lambda

Amazon Web Services

This document summarizes an AWS webinar on advanced serverless patterns for AWS Lambda. The webinar covered 4 main patterns: 1) Web applications using API Gateway, Lambda, and DynamoDB; 2) Stream processing using Kinesis and Lambda; 3) Data lakes using services like S3, Athena, and Glue; 4) Machine learning workflows using services like SageMaker, Rekognition, and Lex. The webinar provided best practices for each pattern including techniques for performance optimization, security, and architecture design.

Running Presto and Spark on the Netflix Big Data Platform

Eva Tse

This document summarizes Netflix's big data platform, which uses Presto and Spark on Amazon EMR and S3. Key points: - Netflix processes over 50 billion hours of streaming per quarter from 65+ million members across over 1000 devices. - Their data warehouse contains over 25PB stored on S3. They read 10% daily and write 10% of reads. - They use Presto for interactive queries and Spark for both batch and iterative jobs. - They have customized Presto and Spark for better performance on S3 and Parquet, and contributed code back to open source projects. - Their architecture leverages dynamic EMR clusters with Presto and Spark deployed via bootstrap actions for scalability.

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

Amazon Web Services

In this session, we discuss how Spark and Presto complement the Netflix big data platform stack that started with Hadoop, and the use cases that Spark and Presto address. Also, we discuss how we run Spark and Presto on top of the Amazon EMR infrastructure; specifically, how we use Amazon S3 as our data warehouse and how we leverage Amazon EMR as a generic framework for data-processing cluster management.

(BAC404) Deploying High Availability and Disaster Recovery Architectures with...

Amazon Web Services

The document discusses disaster recovery strategies for AWS including backup and restore, pilot light, and warm standby approaches. It provides examples of architectures using these approaches including replicating databases across Availability Zones and regions for high availability and disaster recovery. CloudFormation templates are shown that can automate the deployment of load balanced auto-scaled web servers across Availability Zones for disaster recovery.

(BDT208) A Technical Introduction to Amazon Elastic MapReduce

Amazon Web Services

"Amazon EMR provides a managed framework which makes it easy, cost effective, and secure to run data processing frameworks such as Apache Hadoop, Apache Spark, and Presto on AWS. In this session, you learn the key design principles behind running these frameworks on the cloud and the feature set that Amazon EMR offers. We discuss the benefits of decoupling compute and storage and strategies to take advantage of the scale and the parallelism that the cloud offers, while lowering costs. Additionally, you hear from AOL’s Senior Software Engineer on how they used these strategies to migrate their Hadoop workloads to the AWS cloud and lessons learned along the way. In this session, you learn the benefits of decoupling storage and compute and allowing them to scale independently; how to run Hadoop, Spark, Presto and other supported Hadoop Applications on Amazon EMR; how to use Amazon S3 as a persistent data-store and process data directly from Amazon S3; dDeployment strategies and how to avoid common mistakes when deploying at scale; and how to use Spot instances to scale your transient infrastructure effectively."

SnapDish AWS

Fumikazu Kiyota

SnapDish 1. Web / App Server - instance m1.large - python tornado - python memcached - python pymongo - nginx 2. Image Server - instance c3.large - nginx gridfs 3. ElastiCache - Job Queue - python-rq + Redis - Data Cache - Memcached 4. Job Queue Woker Server - m1.small 5. DB Server - MongoDB - Production h1.4xlarge - Backup m1.large 6. Search Engine - CloudSearch - SQS to update index

Running Spark In Production in the Cloud is Not Easy with Nayur Khan

Databricks

Apache Spark is the engine powering many data-driven use cases, from data engineering to data science and machine learning applications. At QuantumBlack, Spark is considered a key technology and used in a number of client engagements, from a Data Engineering, Data Science and Platform Engineering point of view. This talk will be around the lessons learned after running successfully Apache Spark workloads in production in the cloud for a number of years. As public cloud adoption grows in the enterprise, more and more organizations are choosing to run Apache Spark workloads on cloud infrastructure. While the cloud presents many benefits, there are a number of challenges that aren’t obvious until you start and require sometimes different approaches or thinking. This talk will look into a few different areas, starting with the Jigsaw pieces you face with Open Source software, balancing a platform for stability along with allowing innovation. The talk will then look at approaches used to combat the not so obvious challenges and trade-offs of using cloud scalable storage backends for storing/retrieving data. Finally, there’ll be a section on the considerations needed for reliability and manageability of robust analytic pipelines.

AWS re:Invent 2016: ↑↑↓↓←→←→ BA Lambda Start (SVR305)

Amazon Web Services

Ever wished you had a list of cheat codes to unleash the full power of AWS Lambda for your production workload? Come learn how to build a robust, scalable, and highly available serverless application using AWS Lambda. In this session, we discuss hacks and tricks for maximizing your AWS Lambda performance, such as leveraging customer reuse, using the 500 MB scratch space and local cache, creating custom metrics for managing operations, aligning upstream and downstream services to scale along with Lambda, and many other workarounds and optimizations across your entire function lifecycle. You also learn how Hearst converted its real-time clickstream analytics data pipeline from a server-based model to a serverless one. The infrastructure of the data pipeline relied on Amazon EC2 instances and cron jobs to shepherd data through the process. In 2016, Hearst converted its data pipeline architecture to a serverless process that relies on event triggers and the power of AWS Lambda. By moving from a time-based process to a trigger-based process, Hearst improved its pipeline latency times by 50%.

대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016

Amazon Web Services Korea

1. The document demonstrates how to use various AWS services like Kinesis, Redshift, Elasticsearch to analyze streaming game log data. 2. It shows setting up an EC2 instance to generate logs, creating a Kinesis stream to ingest the logs, and building Redshift tables to run queries on the logs. 3. The document also explores loading the logs from Kinesis into Elasticsearch for search and linking Kinesis and Redshift with Kinesis Analytics for real-time SQL queries on streams.

My First Big Data Application

Amazon Web Services

This document provides instructions for setting up a big data application on AWS using various AWS services. It describes using Amazon Kinesis Firehose to collect web server logs from an EC2 instance into an S3 bucket. It then describes using Amazon EMR with Spark and Hive to process the data, Amazon Redshift for data analysis, and Amazon QuickSight for visualization. The document contains detailed steps for setting up IAM roles, security groups, launching the EC2 instance and EMR cluster, and ingesting and exploring the log data with Spark SQL and Zeppelin notebooks.

REPEAT_1_Deep_dive_on_new_features_in_Amazon_RDS_for_SQL_Server_DAT364-R1(1).pdf

AkashGoel82

The document discusses new features in Amazon RDS for SQL Server. It provides an overview of infrastructure improvements including new instance types and storage options. It also covers security features like SQL Server audit and SQLAgentOperatorRole. Monitoring capabilities through CloudWatch Logs integration and Performance Insights are reviewed. Migration methods using native backups and restores and Change Data Capture are presented. Additional features such as increased database limits and S3 integration are also summarized.

Effectively Scale and Operate AEM with MongoDB by Norberto Leite

AEM HUB

The document discusses MongoDB and operational best practices for effectively scaling and operating AEM with MongoDB. It provides an overview of MongoDB and MongoMK, how to size deployments based on availability, volume, and working data sets. It also covers deployment, monitoring performance, and addressing fragmentation and I/O issues. The presentation emphasizes the importance of provisioning, monitoring, and working closely with Adobe to ensure a scalable solution.

M|18 What's New in the MariaDB AX Platform

MariaDB plc

The document discusses new features in MariaDB AX, an open source analytics platform. Key new features include: (1) Improved high availability and disaster recovery options using GlusterFS for local storage and parallel backup/restore. (2) Ability to perform custom analytics through user-defined aggregate and window functions. (3) Streamlined data ingestion with bulk and streaming data adapters that simplify and speed the process of loading data.

M|18 Analyzing Data with the MariaDB AX Platform

MariaDB plc

The document summarizes new features in MariaDB AX, an open-source analytics platform. Key updates include: improved high availability and disaster recovery with GlusterFS support and parallel backup/restore; enhanced analytics capabilities like user-defined aggregate and window functions; and streamlined data ingestion with streaming and bulk data adapters for loading data from sources like Kafka and applications in real-time or batch. The platform provides scalable analytics on MariaDB ColumnStore through features like distributed storage, parallel queries, and automatic partitioning.

Introduction to Apache Amaterasu (Incubating): CD Framework For Your Big Data...

DataWorks Summit

In the last few years, the DevOps movement has introduced ground breaking approaches to the way we manage the lifecycle of software development and deployment. Today organisations aspire to fully automate the deployment of microservices and web applications with tools such as Chef, Puppet and Ansible. However, the deployment of data-processing pipelines remains a relic from the dark-ages of software development. Processing large-scale data pipelines is the main engineering task of the Big Data era, and it should be treated with the same respect and craftsmanship as any other piece of software. That is why we created Apache Amaterasu (Incubating) - an open source framework that takes care of the specific needs of Big Data applications in the world of continuous delivery. In this session, we will take a close look at Apache Amaterasu (Incubating) a simple and powerful framework to build and dispense pipelines. Amaterasu aims to help data engineers and data scientists to compose, configure, test, package, deploy and execute data pipelines written using multiple tools, languages and frameworks. We will see what Amaterasu provides today, and how it can help existing Big Data application and demo some of the new bits that are coming in the near future. Speaker: Yaniv Rodenski, Senior Solutions Architect, Couchbase

Dok Talks #124 - Intro to Druid on Kubernetes

DoKC

Link to the full talk - https://youtu.be/2Rf5t2Eh6IQ https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK This talk will provide a high-level overview of Kubernetes, Helm charts and how they can be used to deploy Apache Druid clusters of any size. We'll review how Kubernetes functionality enables resilience and self-healing, historical tiers through node group affinity, middle manager scaling through Kubernetes autoscaling to optimize ingestion capacity and some of the gotchas along the way. BIO Sergio Ferragut is a database veteran turned Developer Advocate at Imply. His experience includes 16 years at Teradata in professional services and engineering roles. He has direct experience in building analytics applications spanning the retail, supply chain, pricing optimization and IoT spaces. Sergio has worked at multiple technology start-ups including APL and Splice Machine where he helped guide product design and field messaging.

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3

Databricks

This introductory workshop is aimed at data analysts & data engineers new to Apache Spark and exposes them how to analyze big data with Spark SQL and DataFrames. In this partly instructor-led and self-paced labs, we will cover Spark concepts and you’ll do labs for Spark SQL and DataFrames in Databricks Community Edition. Toward the end, you’ll get a glimpse into newly minted Databricks Developer Certification for Apache Spark: what to expect & how to prepare for it. * Apache Spark Basics & Architecture * Spark SQL * DataFrames * Brief Overview of Databricks Certified Developer for Apache Spark

Effectively Deploying MongoDB on AEM

Norberto Leite

The document discusses using MongoDB as a scalable storage solution for Adobe Experience Manager (AEM). It introduces MongoDB and the MongoMK storage component that allows AEM to use MongoDB. The rest of the document covers best practices for sizing, deploying, and operating an AEM and MongoDB configuration including considerations for availability, volume, working set, latency, deployment automation, and operational monitoring.

Similar to M3D - Metadata Driven Development (20)

Building a Scalable Asset Management (DAM) Platform in the AWS

Building a Scalable Digital Asset Management Platform in the Cloud (MED402) |...

Architetture serverless e pattern avanzati per AWS Lambda

Running Presto and Spark on the Netflix Big Data Platform

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

(BAC404) Deploying High Availability and Disaster Recovery Architectures with...

(BDT208) A Technical Introduction to Amazon Elastic MapReduce

SnapDish AWS

Running Spark In Production in the Cloud is Not Easy with Nayur Khan

AWS re:Invent 2016: ↑↑↓↓←→←→ BA Lambda Start (SVR305)

대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016

My First Big Data Application

REPEAT_1_Deep_dive_on_new_features_in_Amazon_RDS_for_SQL_Server_DAT364-R1(1).pdf

Effectively Scale and Operate AEM with MongoDB by Norberto Leite

M|18 What's New in the MariaDB AX Platform

M|18 Analyzing Data with the MariaDB AX Platform

Introduction to Apache Amaterasu (Incubating): CD Framework For Your Big Data...

Dok Talks #124 - Intro to Druid on Kubernetes

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3

Effectively Deploying MongoDB on AEM

Recently uploaded

9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...

saastr

Harnessing the Power of NLP and Knowledge Graphs for Opioid Research

Neo4j

"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk

Fwdays

At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience

Y-Combinator seed pitch deck template PP

c5vrf27qcz

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

Energy Efficient Video Encoding for Cloud and Edge Computing Instances

Alpen-Adria-Universität

The Microsoft 365 Migration Tutorial For Beginner.pptx

operationspcvita

Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency

ScyllaDB

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Alex Pruden

Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security. Paper Link: https://eprint.iacr.org/2024/257

Main news related to the CCS TSI 2023 (2023/1695)

Jakub Marek

An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers. The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 . The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .

Introduction of Cybersecurity with OSS at Code Europe 2024

Hiroshi SHIBATA

I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems. The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS. Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application. I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.

Dandelion Hashtable: beyond billion requests per second on a commodity server

Antonios Katsarakis

This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).

Leveraging the Graph for Clinical Trials and Standards

Neo4j

Northern Engraving | Nameplate Manufacturing Process - 2024

Northern Engraving

Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

Chart Kalyan

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

DianaGray10

Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more. The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications. We’ll discuss and demo the benefits of UiPath Apps and connectors including: Creating a compelling user experience for any software, without the limitations of APIs. Accelerating the app creation process, saving time and effort Enjoying high-performance CRUD (create, read, update, delete) operations, for seamless data management. Speakers: Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP Charlie Greenberg, host

AppSec PNW: Android and iOS Application Security with MobSF

Ajin Abraham

Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application. This talk covers: Using MobSF for static analysis of mobile applications. Interactive dynamic security assessment of Android and iOS applications. Solving Mobile app CTF challenges. Reverse engineering and runtime analysis of Mobile malware. How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.

"Choosing proper type of scaling", Olena Syrota

Fwdays

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...

saastr

Recently uploaded (20)

9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...

Harnessing the Power of NLP and Knowledge Graphs for Opioid Research

"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk

Y-Combinator seed pitch deck template PP

Columbus Data & Analytics Wednesdays - June 2024

Energy Efficient Video Encoding for Cloud and Edge Computing Instances

The Microsoft 365 Migration Tutorial For Beginner.pptx

Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Main news related to the CCS TSI 2023 (2023/1695)

Introduction of Cybersecurity with OSS at Code Europe 2024

Dandelion Hashtable: beyond billion requests per second on a commodity server

Leveraging the Graph for Clinical Trials and Standards

Northern Engraving | Nameplate Manufacturing Process - 2024

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

AppSec PNW: Android and iOS Application Security with MobSF

"Choosing proper type of scaling", Olena Syrota

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...

M3D - Metadata Driven Development

1. Metadata Driven Development

2. Content • What is M3D? • M3D Components • Why we need it? • M3D @ Adidas • What’s to come?

3. What is it? Framework Cloud and platform agnostic Easy development of new feats Multiple source/target system Creation of data lake envs

4. What is it?

5. M3D Components M3D Engine M3D API https://github.com/adidas/m3d-engine https://github.com/adidas/m3d-api

6. Features in action! Creation of data lake envs Inbound Raw File Raw File Raw File Raw File Raw File Raw File

7. Features in action! Creation of data lake envs Inbound Landing Raw File Raw File Raw File Raw File Raw File Raw File

8. Features in action! Creation of data lake envs Inbound Landing Lake Parqu et Parqu et Parqu et Parqu et Parqu et Parqu et

9. Features in action! Creation of data lake envs Inbound Landing Lake Out View View View View Lake

10. Features in action! Inbound Landing Lake Out Lake Multiple source/target system M3D Target Exasol Target Y Target Target Z AWS Cloud and platform agnostic Cloudera

11. Features in action! Inbound Landing Lake Parqu et Parqu et Parqu et Parqu et Parqu et Parqu et Easy development of new feats Easily define new readers/writers based on traits Introduce transformations on datasets Add new APIs to M3D Api

12. Why We Need It? Simplification of definition of new tables in lake Easy way to schedule loads Clear semantics defined globally Single extraction from source systems Simple definition of views as interface for applications

13. M3D @ Adidas We have the so called “Cockpit” for definition of global semantics Orchestration with Jenkins as a Service Loading data to lake (example): Place files in Inbound layer python m3d_main.py -function create_emr_cluster - core_instance_type m4.large - master_instance_type m4.large - core_instance_count 3 - destination_system emr - destination_database emr_database -destination_environment test - config config.json -emr_version emr- 5.23.0 python m3d_main.py -function create_table -config config.json - destination_system emr - destination_database emr_database - destination_environment test - destination_table table_name - emr_cluster_id id-of-started-cluster python m3d_main.py -function load_table -config config.json - destination_system emr -destination_database emr_database - destination_environment test -destination_table table_name - load_type FullLoad -emr_cluster_id id-of-started-cluster

14. M3D @ Adidas 9 TB In Memory Lake Size 150 TB Lake Size on Big Data Platform 500+ tables in Data Lake > 900 views consumed by Products

15. M3D @ Adidas Table

16. M3D @ Adidas

17. M3D @ Adidas

18. What’s to come? Airflow as orchestration for M3D Publish M3D Api as library UI for quick setup of M3D Api/Engine PySpark version for M3D engine

19. Thank you! Questions?