.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure

•

0 likes•484 views

Wir leben in einem Datenzeitalter! Nach Schätzungen in 2006, betrug das weltweite "Datenuniversum" ca. 0.18 Zettabytes (1 ZB => 10 hoch 21 Bytes bzw. 1 Mrd. Terabytes). In 2011 hatte sich dieses Volumen sogar verzehnfacht (1,8 Zettabytes). Somit wird in vielen Anwendungsszenarien das Thema Big Data und Big Processing immer wichtiger. Klassische relationale Datenbanksysteme, sowie Statistik und Visualisierungstools, sind oft nicht in der Lage, derart große Datenmengen zu verarbeiten. Für Big Data kommt daher eine neue Art von Software zum Einsatz, die massiv parallel auf bis zu hunderten oder tausenden von Prozessoren bzw. Servern arbeitet.

Technology

Apache Hadoop-based Services für Windows Azure

Sascha Dittmann
Software Developer / Solution Architect
Twitter: @SaschaDittmann
Blog: http://www.sascha-dittmann.de

Hadoop Distributed File System

Cluster Startvorgang

Hadoop Distributed File System
Ausfall des Namenodes (Failover)

Hadoop Distributed File System
Benuteranfrage

①

② ② ②

Hadoop Distributed File System
 Portable Operating System Interface (POSIX)
 Replikation auf mehrere Datenknoten
js> #ls input/ncdc
Found 9 items
drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/_distcp_logs_g0dedn
drwxr-xr-x - Sascha supergroup 0 2012-04-24 12:04 /user/Sascha/input/ncdc/_distcp_logs_ofj0u6
drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:09 /user/Sascha/input/ncdc/all
drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/all2
drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/metadata
drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro
drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro-tab
-rw-r--r-- 3 Sascha supergroup 529 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt
-rw-r--r-- 3 Sascha supergroup 168 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt.gz

Map/Reduce
DataNode DataNode DataNode 0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300

1949,0
1952,-11
1950,22
Map Map Map 1950,55
1950,33

Sort Sort Sort 1949,0
1950,[22,33,55]
Shuffle Shuffle Shuffle 1952,-11

Reduce
1949,0
1950,55
1952,-11

RDBMS vs. Map/Reduce
RDBMS Map/Reduce
Datenmenge Gigabytes Petabytes
Zugriff Interaktiv und Batch Batch
Lese- / Schreibzugriffe Viele Lese- und Einmaliges Schreiben
Schreibzugriffe Viele Lesezugriffe
Datenstruktur Statisches Schema Dynamisches Schema
Datenintegrität Hoch Niedrig
Skalierverhalten Nicht-Linear Linear

Demos
 Hadoop Dashboard
 Interactive Console
 Remote Desktop
 Nutzung des WA Storage
 Map/Reduce via JavaScript
 C# Streaming
 Power Pivot

Cloud Bloggers

Die Blogs der deutschen Cloud Computing-Community

Link: http://cloudbloggers.de

Im zweiten Teil unserer Microsoft Big Data Session geht es darum, wie Big Data Informationen über "klassisches" SQL zugänglich gemacht werden können und wie sich mit der neuen PolyBase-Engine unstrukturierte Hadoop-Daten mit relationalen Data Warehouse-Daten einfach verknüpfen lassen. In der Hadoop-Welt wird der SQL-Zugriff über die Komponente Hive ermöglicht. Über den Microsoft Hive ODBC-Konnektor können die üblichen BI-Tools, wie PowerPivot, diesen Zugriff direkt nutzen. Die PolyBase-Engine schließlich wird ein Bestandteil des SQL Server 2012 Parallel Data Warehouse werden und erlaubt einem transparenten SQL-Zugriff, egal, wo sich die Daten befinden.

Hadoop Einführung @codecentric

imalik8088

Wer gewinnt das SQL-Rennen auf der Hadoop-Strecke?

inovex GmbH

Nachdem in den letzten Jahren NoSQL ein beherrschendes Thema im Kontext von Big Data war, gewinnt SQL als Anfragesprache wieder große Bedeutung im Hadoop-Umfeld. Dabei steht mit Hive ein SQL-Dialekt zur Verfügung, mit dem zwar leicht Batch-orientierte ETL-Strecken für Hadoop gebaut werden können, der aber bisher für interaktive Analysen nicht geeignet war Mit Impala, Presto, Spark SQL und dem Stinger-Projekt ändert sich das nun rapide. Schnelle verteilte Query Engines erlauben interaktive analytische Anfragen auf großen Datenmengen. Dazu kommen neue Speicherformate wie Parquet und ORC, die effizientere Repräsentation und schnelleren Zugriff versprechen. In dieser Session gebe ich einen Überblick über Stärken und Schwächen der verschiedenen Ansätze und Erfahrungen aus dem praktischen Einsatz.

Hadoop 2.0 - The Next Level

Sascha Dittmann

In 2010 stellten die Entwickler von Hadoop fest, dass bei sehr große Clustern (4.000 Knoten und mehr) das bisherige MapReduce Framework nicht mehr richtig skaliert. Deshalb wurde dieses komplett überarbeitet. Das Ergebnis war YARN (Yet Another Resource Negotiator). Neben einer besseren Skalierbarkeit erzeugte YARN weitere positive Nebeneffekte. Im Oktober 2013 wurde YARN mit dem Hadoop 2.0 Release veröffentlicht. Was es mit YARN auf sich hat - und welche zusätzlichen Änderungen in Hadoop 2.0 eingeflossen sind - zeigt diese Session.

Crawl the entire web in 10 minutes...and just 100€

Danny Linden

This document describes how to build a web search engine using Common Crawl data at a low cost. Common Crawl crawls over 2 billion URLs per month and stores the data in an accessible format on AWS S3. The data can be processed using Hadoop and Pig on AWS EMR for scalability. Examples show how to load Common Crawl data into Pig and extract metadata like page titles. OnPage offers services to build scalable big data solutions using these techniques.

Big Data Bullshit Bingo

Danny Linden

This document provides an overview of Azure Data Lake Analytics (ADLA) and its capabilities. It describes the key components of ADLA including U-SQL for querying data, job execution, custom extractors and operators, external data sources, and job diagnostics. The document also includes examples of creating credentials and data sources for querying external databases, implementing a custom extractor, and job execution details.

Hochskalierbare, relationale Datenbanken in Microsoft Azure

Sascha Dittmann

Die meisten Softwareprojekte fangen mit einer einfachen relationalen Datenbank an. Doch was soll man machen, wenn die Anforderungen an die Datenbank wachsen? Was, wenn die Datenbankgröße und die Anzahl der gleichzeitigen Abfragen stetig wächst? Was, wenn aus einzelnen Datenbanken ein Data Warehouse entsteht? Sascha Dittmann zeigt in dieser Session wie solche und ähnliche Szenarien mit Microsoft Azure umsetzbar sind. Er stellt dabei Technologien wie Azure SQL Database, Azure Elastic Pools und Azure SQL Data Warehouse im Detail vor und gibt Beispiele, sowie Tipps & Tricks aus der Praxis.

Microsoft R - Data Science at Scale

Sascha Dittmann

Spark auf Hadoop ist hochskalierbar. Cloud Computing ist hochskalierbar. R, die erweiterbare Open Source Data Science Software, eher nicht. Aber was passiert, wenn wir Spark auf Hadoop, Cloud Computing und den Microsoft R Server zu einer skalierbaren Data Science-Plattform zusammenfügen? Stellen Sie sich vor wie es sein könnte, wenn Sie das Erkunden, Transformieren und Modellieren von Daten in jeder beliebigen Größe aus Ihrer Lieblings-R-Umgebung durchführen könnten. Stellen Sie sich nun vor, wie man anschließend die erzeugten Modelle - mit wenigen Klicks - als skalierbare, cloud basierte Web-Services-API bereitstellt. In dieser Session zeigt Sascha Dittmann, wie Sie Ihren R-Code, tausende von Open-Source-R-Pakete sowie die verteilte Implementierungen der beliebtesten Maschine-Learning-Algorithmen nutzen können, um genau dies umzusetzen. Dabei zeigt er wie man ein HDInsight Spark-Cluster inkl. eines Microsoft R Server-Clusters erstellt, sowie das daraus entstandene Model im SQL Server oder als swagger-based API für Anwendungsentwickler bereitstellt.

SQL Server vs. Azure DocumentDB – Ein Battle zwischen XML und JSON

Sascha Dittmann

Seit dem SQL Server 2000 hielt Stück für Stück die XML-Unterstützung Einzug in die Microsoft RDBMS Welt. Mit der Azure DocumentDB kam die zweite, hauseigene NoSQL-Datenbank in der Microsoft Cloud hinzu, welche die Daten im JSON-Format verarbeitet. In dieser Session werden wir anhand eines Praxisbeispiels step-by-step, d.h. von den vorbereitenden Schritten, über das Schreiben bis hin zum Lesen der Daten, diese beiden Technologien gegenüberstellen. Dabei arbeiten wir die Vor- und Nachteile der einzelnen Ansätze heraus und zeigen Best Practices auf.

dotnet Cologne 2015 - Azure Service Fabric

Sascha Dittmann

Wer hoch-skalierbare Anwendungen in Microsoft Azure bauen wollte, kam bislang an den Azure Cloud Services nicht vorbei. Doch diese Technologie hatte ihre Ecken und Kanten. Die an der BUILD 2015 veröffentlichte Azure Service Fabric Preview soll hierbei Abhilfe schaffen. Mit ihr sollen Entwickler schneller und unkomplizierter Anwendungen für die Cloud, wie auch fürs eigene Rechenzentrum, konstruieren können. Sascha Dittmann durfte bereits seit Ende letzen Jahres mit dem Dienst Erfahrungen sammeln und stellt diesen, anhand einiger Code-Beispiele, in dieser Session vor.

SQL Saturday #313 Rheinland - MapReduce in der Praxis

Sascha Dittmann

Das Programmiermodel MapReduce, welches vor einigen Jahren von Google veröffentlicht wurde, hat Einzug in zahlreiche Systeme erhalten. Dabei wurde es sowohl als eigenständiges System, wie beispielweise bei Hadoop, Disco oder Amazon Elastic MapReduce, aber auch als Abfragesprache innerhalb größerer Systeme, wie beispielweise bei MongoDB, Greenplum DB oder Aster Data, implementiert. Diese Session stellt gängige Problemstellungen aus der Praxis vor und wie diese mit dem MapReduce Framework von Microsoft HDInsight umgesetzt werden können.

Microsoft HDInsight Podcast #001 - Was ist HDInsight

Sascha Dittmann

dotnet Cologne 2013 - Windows Azure Mobile ServicesSascha Dittmann

dotnet Cologne 2013 - Microsoft HD Insight für .NET Entwickler

Sascha Dittmann

This document discusses Big Data concepts and tools including Hadoop, HDFS, MapReduce, Pig Latin, Hive, and analyzing JSON data with Hive. It provides examples of using MapReduce with JavaScript and C# to count words in text data. It also summarizes HDFS, showing how it provides fault tolerance and handles user requests through replication across data nodes. Finally, it discusses analyzing Twitter data stored in JSON format using Hive to extract and interpret fields.

Developer Open Space 2012 - Cloud Computing Workshop

Sascha Dittmann

Der Einbezug von Cloud Computing in die tägliche Entwicklungsarbeit gewinnt weiter an Bedeutung. Ob hybride Ansätze oder Greenfield-Engineering, Windows Azure bietet die entsprechenden Building Blocks um eine Anforderung zielführend umzusetzen. In dem Workshop werden wir gemeinsam eine klassische on-premise Anwendung „cloudifizieren“ und Schritt für Schritt Cloud-Ressourcen zielführend einsetzen.

PASS Camp 2012 - Big Data mit Microsoft (Teil 1)

Sascha Dittmann

CloudOps Summit 2012 - 3 Wege in die CloudSascha Dittmann

Big Data & NoSQLSascha Dittmann

NoSQL mit RavenDB und Azure

Sascha Dittmann

Windows Azure für Entwickler V1Sascha Dittmann

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

Skybuffer SAM4U tool for SAP license adoption

Tatiana Kojar

Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool. SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Leveraging the Graph for Clinical Trials and Standards

Neo4j

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Alex Pruden

Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security. Paper Link: https://eprint.iacr.org/2024/257

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

Recently uploaded

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

Skybuffer SAM4U tool for SAP license adoption

Tatiana Kojar

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Leveraging the Graph for Clinical Trials and Standards

Neo4j

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Alex Pruden

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Introduction of Cybersecurity with OSS at Code Europe 2024

Hiroshi SHIBATA

I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems. The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS. Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application. I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.

Nordic Marketo Engage User Group_June 13_ 2024.pptx

MichaelKnudsen27

Dandelion Hashtable: beyond billion requests per second on a commodity server

Antonios Katsarakis

This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).

Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency

ScyllaDB

[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...

Jason Yip

The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.

Taking AI to the Next Level in Manufacturing.pdf

ssuserfac0301

Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as: 1. How quickly AI is being implemented in manufacturing. 2. Which barriers stand in the way of AI adoption. 3. How data quality and governance form the backbone of AI. 4. Organizational processes and structures that may inhibit effective AI adoption. 6. Ideas and approaches to help build your organization's AI strategy.

GNSS spoofing via SDR (Criptored Talks 2024)

Javier Junquera

In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security. This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing. The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.

Digital Marketing Trends in 2024 | Guide for Staying Ahead

Wask

https://www.wask.co/ebooks/digital-marketing-trends-in-2024 Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.

What is an RPA CoE? Session 1 – CoE Vision

DianaGray10

Serial Arm Control in Real Time Presentation

tolgahangng

GraphRAG for Life Science to increase LLM accuracy

Tomaz Bratanic

FREE A4 Cyber Security Awareness Posters-Social Engineering part 3

Data Hops

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Safe Software

Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency. During the hour, we’ll take you through: Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board. Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes. Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI. We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI. This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!

Recently uploaded (20)

Columbus Data & Analytics Wednesdays - June 2024

Skybuffer SAM4U tool for SAP license adoption

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

Leveraging the Graph for Clinical Trials and Standards

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Presentation of the OECD Artificial Intelligence Review of Germany

HCL Notes and Domino License Cost Reduction in the World of DLAU

Introduction of Cybersecurity with OSS at Code Europe 2024

Nordic Marketo Engage User Group_June 13_ 2024.pptx

Dandelion Hashtable: beyond billion requests per second on a commodity server

Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency

[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...

Taking AI to the Next Level in Manufacturing.pdf

GNSS spoofing via SDR (Criptored Talks 2024)

Digital Marketing Trends in 2024 | Guide for Staying Ahead

What is an RPA CoE? Session 1 – CoE Vision

Serial Arm Control in Real Time Presentation

GraphRAG for Life Science to increase LLM accuracy

FREE A4 Cyber Security Awareness Posters-Social Engineering part 3

Driving Business Innovation: Latest Generative AI Advancements & Success Story

.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure

1. Apache Hadoop-based Services für Windows Azure Sascha Dittmann Software Developer / Solution Architect Twitter: @SaschaDittmann Blog: http://www.sascha-dittmann.de

2. Apache Hadoop & Co Zookeeper Pig

3. Hadoop Distributed File System Cluster Startvorgang

4. Hadoop Distributed File System Ausfall des Namenodes (Failover)

5. Hadoop Distributed File System Benuteranfrage ① ② ② ②

6. Hadoop Distributed File System  Portable Operating System Interface (POSIX)  Replikation auf mehrere Datenknoten js> #ls input/ncdc Found 9 items drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/_distcp_logs_g0dedn drwxr-xr-x - Sascha supergroup 0 2012-04-24 12:04 /user/Sascha/input/ncdc/_distcp_logs_ofj0u6 drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:09 /user/Sascha/input/ncdc/all drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/all2 drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/metadata drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro-tab -rw-r--r-- 3 Sascha supergroup 529 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt -rw-r--r-- 3 Sascha supergroup 168 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt.gz

7. Map/Reduce DataNode DataNode DataNode 0067011990999991950051507004+68750 0043011990999991950051512004+68750 0043011990999991950051518004+68750 0043012650999991949032412004+62300 0043012650999991949032418004+62300 1949,0 1952,-11 1950,22 Map Map Map 1950,55 1950,33 Sort Sort Sort 1949,0 1950,[22,33,55] Shuffle Shuffle Shuffle 1952,-11 Reduce 1949,0 1950,55 1952,-11

8. Map/Reduce DataNode DataNode DataNode 0067011990999991950051507004+68750 0043011990999991950051512004+68750 0043011990999991950051518004+68750 0043012650999991949032412004+62300 0043012650999991949032418004+62300 1949,0 1952,-11 1950,22 Map Map Map 1950,55 1950,33 1949,0 1952,-11 Combine Combine Combine 1950,55 1950,33 Sort Sort Sort 1949,0 1950,[33,55] Shuffle Shuffle Shuffle 1952,-11 Reduce 1949,0 1950,55 1952,-11

9. RDBMS vs. Map/Reduce RDBMS Map/Reduce Datenmenge Gigabytes Petabytes Zugriff Interaktiv und Batch Batch Lese- / Schreibzugriffe Viele Lese- und Einmaliges Schreiben Schreibzugriffe Viele Lesezugriffe Datenstruktur Statisches Schema Dynamisches Schema Datenintegrität Hoch Niedrig Skalierverhalten Nicht-Linear Linear

10. Apache Hadoop & Co Zookeeper Pig

11. Demos  Hadoop Dashboard  Interactive Console  Remote Desktop  Nutzung des WA Storage  Map/Reduce via JavaScript  C# Streaming  Power Pivot

12. Cloud Bloggers Die Blogs der deutschen Cloud Computing-Community Link: http://cloudbloggers.de

.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure

Recommended

Recommended

More Related Content

More from Sascha Dittmann

More from Sascha Dittmann (15)

Recently uploaded

Recently uploaded (20)

.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure