Hands on Hadoop
•Download as PPT, PDF•
7 likes•1,408 views
My intro talk for hadoop and how to use it with python streaming. Code is here : http://github.com/ptarjan/hands-on-hadoop-tutorial/
Report
Share
Report
Share
![Data is ugly ,[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Git Sensitive Data
Git can unintentionally store sensitive data like passwords or keys in its commit history. While commands like filter-branch can remove sensitive data from the local history, it may still be retrievable from remote repositories unless the history is rewritten there as well. The safest approach is to change all credentials and remove the local git repository entirely.
R版Getopt::Longを作ってみた
The document discusses the Getopt::Long module for parsing command line arguments in R. It provides an example of using Getopt::Long to parse arguments in an R script called getopt_long_sample.R, where the --save.path argument is parsed and assigned to the save.path variable. Getopt::Long allows defining argument types like strings, integers, and flags to parse command line options consistently in both Perl and R scripts.
vfsStream - effective filesystem mocking
The document discusses using vfsStream to mock the filesystem in unit tests. vfsStream provides a virtual filesystem that uses PHP streams, allowing tests to manipulate files and directories without interacting with the real filesystem. It describes how to set up vfsStream, create and interact with virtual files and directories, and a vfsStream PHPUnit helper that simplifies its integration with PHPUnit tests.
vfsStream - a better approach for file system dependent tests
Have you ever been annoyed by testing classes or functions operating on the file system? Be it tests that rely on presence of physical files, the problem of not cleaning up correctly after the test run or checking that your algorithm creates the correct directories and files with correct file permissions. Then this is for you: vfsStream to the rescue!
Web Application Security 101 - 05 Enumeration
This document discusses techniques for enumerating information from a target website or application, including:
1. Using search engines like Google to find publicly available information and hidden features.
2. Bruteforcing files, directories, and parameters to locate hidden areas. Tools like DirBuster can automate this process.
3. Analyzing error messages and response codes to infer application details and find vulnerabilities.
4. Fingerprinting server configuration details like virtual hosts, load balancers, alternative ports and access points.
5. The document provides examples of commands and techniques to practice these enumeration methods.
spug_2008-08
This document describes using the File::Find and MP3::Tag modules in Perl to search directories of MP3 files, find duplicates, and extract metadata. It starts by explaining how File::Find searches directories and calls a callback subroutine. It then shows incrementally building a script that uses File::Find to search for MP3 files, uses MP3::Tag to extract metadata, stores it in a database, and finds duplicate songs based on normalized artist/title fields. Security techniques like taint mode and untainting are also demonstrated to safeguard against malicious file paths.
JIP Pipeline System Introduction
This talks covers some of the basic aspects of the JIP pipeline system (http://pyjip.readthedocs.org) and it's command line interface.
JIP is a system to manage jobs on a Cluster system and simplify the process of building computational pipelines. JIP can interact with Slurm, SGE, PBS/Torque, or LSF clusters and comes with a small local scheduler to run without any remote grid engine.
Create LED lightened Wapuu (LEDパネルで光るわぷーを)
1. The document discusses setting up a Raspberry Pi to display images on an LED matrix using the REST API in WordPress.
2. Code is provided to register routes to set and get the image ID and functions to update the image ID in the database and return it.
3. A Python script is shown to call the API, get the image ID, and display the corresponding image on the LED matrix periodically.
Recommended
Git Sensitive Data
Git can unintentionally store sensitive data like passwords or keys in its commit history. While commands like filter-branch can remove sensitive data from the local history, it may still be retrievable from remote repositories unless the history is rewritten there as well. The safest approach is to change all credentials and remove the local git repository entirely.
R版Getopt::Longを作ってみた
The document discusses the Getopt::Long module for parsing command line arguments in R. It provides an example of using Getopt::Long to parse arguments in an R script called getopt_long_sample.R, where the --save.path argument is parsed and assigned to the save.path variable. Getopt::Long allows defining argument types like strings, integers, and flags to parse command line options consistently in both Perl and R scripts.
vfsStream - effective filesystem mocking
The document discusses using vfsStream to mock the filesystem in unit tests. vfsStream provides a virtual filesystem that uses PHP streams, allowing tests to manipulate files and directories without interacting with the real filesystem. It describes how to set up vfsStream, create and interact with virtual files and directories, and a vfsStream PHPUnit helper that simplifies its integration with PHPUnit tests.
vfsStream - a better approach for file system dependent tests
Have you ever been annoyed by testing classes or functions operating on the file system? Be it tests that rely on presence of physical files, the problem of not cleaning up correctly after the test run or checking that your algorithm creates the correct directories and files with correct file permissions. Then this is for you: vfsStream to the rescue!
Web Application Security 101 - 05 Enumeration
This document discusses techniques for enumerating information from a target website or application, including:
1. Using search engines like Google to find publicly available information and hidden features.
2. Bruteforcing files, directories, and parameters to locate hidden areas. Tools like DirBuster can automate this process.
3. Analyzing error messages and response codes to infer application details and find vulnerabilities.
4. Fingerprinting server configuration details like virtual hosts, load balancers, alternative ports and access points.
5. The document provides examples of commands and techniques to practice these enumeration methods.
spug_2008-08
This document describes using the File::Find and MP3::Tag modules in Perl to search directories of MP3 files, find duplicates, and extract metadata. It starts by explaining how File::Find searches directories and calls a callback subroutine. It then shows incrementally building a script that uses File::Find to search for MP3 files, uses MP3::Tag to extract metadata, stores it in a database, and finds duplicate songs based on normalized artist/title fields. Security techniques like taint mode and untainting are also demonstrated to safeguard against malicious file paths.
JIP Pipeline System Introduction
This talks covers some of the basic aspects of the JIP pipeline system (http://pyjip.readthedocs.org) and it's command line interface.
JIP is a system to manage jobs on a Cluster system and simplify the process of building computational pipelines. JIP can interact with Slurm, SGE, PBS/Torque, or LSF clusters and comes with a small local scheduler to run without any remote grid engine.
Create LED lightened Wapuu (LEDパネルで光るわぷーを)
1. The document discusses setting up a Raspberry Pi to display images on an LED matrix using the REST API in WordPress.
2. Code is provided to register routes to set and get the image ID and functions to update the image ID in the database and return it.
3. A Python script is shown to call the API, get the image ID, and display the corresponding image on the LED matrix periodically.
Perl 5.28 new features
Perl 5.28 introduces several new features including auto referencing of subroutine arguments, fancier signatures that allow type constraints, distributed map and grep functions to parallelize work, an upgraded safe module, three-state test outcomes in TAP, close enough comparators, new array manipulation operators, upgradable assignment with =^, better support for Git conflicts, and improved diagnostics.
Tests unitaires pour PostgreSQL avec pgTap
Les tests unitaires se sont pas limités au code des applications, des tests peuvent également être effectués sur les données et les schémas des bases de données.
Conférence donnée lors du meetup PostgreSQL le 22 juin 2016 à Nantes
6 things about perl 6
The document discusses several new features in Perl 6, including phasers for controlling program flow, sets and sequences, types, subsets for defining custom types, grammars, and the MAIN subroutine. It provides examples of using phasers to control block and loop execution, built-in set operations like union and intersection, sequence syntax for ranges, type checking for variables and parameters, defining subsets for things like positive numbers, and using grammars and the MAIN subroutine for command line apps.
git internals
This document provides an overview of Git internals and data structures including blobs, trees, commits and refs. It demonstrates how Git objects are stored and referenced in the Git database. Various Git commands are shown such as add, commit, branch, checkout, merge, reset, rebase and reflog. The document explains the relationships between blobs, trees, commits and branches during normal workflows as well as examples of fast-forward merges, rebases and resetting.
Openpear Project の紹介
Openpear is a PHP-based open source content management system and package repository. It provides features such as a dashboard, content management, favorites, and logging. Openpear uses Rhaco as its framework and can be installed by cloning its GitHub repository and running the setup script. Support is available on Google Groups and IRC channel #openpear on Freenode. Openpear aims to be a full-featured CMS and PEAR-like repository hosting service.
Dataiku - Paris JUG 2013 - Hadoop is a batch
This document provides an overview and comparison of Pig, Hive, and Cascading tools for Hadoop. It begins with brief histories of each tool's development: Pig was created at Yahoo Research in 2006 to enable log analytics; Hive was developed by Facebook in 2007 to provide SQL-like queries over Hadoop data; and Cascading was authored in 2008 and associated with Scalding and Cascalog projects. The document then compares features of the tools such as their procedural versus declarative programming models, data typing approaches, integration capabilities, and performance/optimization characteristics to help users choose the best technology.
Development and practical use of CLI in perl 6
YAPC::Tokyo 2019 トーク資料
Slide of my talk about development of command line interface in perl 6.
https://yapcjapan.org/2019tokyo/talks.html#/detail/12
Shell Scripting
Sexta parte da apresentação sobre Shell Scripting realizada na Bluesoft
Confira o video em http://www.vimeo.com/
Fun with processes - lightning talk
"The other day, I was playing with this idea... I wanted to be able to programmatically download a file..."
These are slides from my lightning talk that I gave on https://twitter.com/CodeElixirIO in August 2018.
Practical pig
This document provides an overview of the Pig Latin data flow language for Hadoop. It discusses why Pig is useful for increasing productivity and insulating users from complexity when working with MapReduce. The document provides examples of simple Pig Latin scripts for common tasks like filtering, joining, grouping and aggregation. It also covers performance considerations, user defined functions, common pitfalls, and recommendations.
Pig workshop
This document provides information about a Pig workshop being conducted by Sudar Muthu. It introduces Pig, describing it as a platform for analyzing large data sets. It outlines some key Pig components like the Pig Shell, Pig Latin language, libraries, and user defined functions. It also discusses why Pig is useful, highlighting aspects like its data flow model and ability to increase programmer productivity. Finally, it previews topics that will be covered in the workshop, such as loading and storing data, Pig Latin operators, and writing user defined functions.
Crack.ba
This batch script cracks passwords for RAR files by repeatedly using the Unrar.exe tool to extract files with incrementing password guesses, deleting the extraction directory and Unrar.exe file upon finding the correct password. It gets the RAR file name and path from the user, checks for validity, and then starts an infinite loop extracting with passwords from 0 to infinity until extraction succeeds without errors.
GIT rozproszony system kontroli wersji
This document provides an overview of Git and its basic commands and data structures. It begins with initializing and configuring a Git repository, then covers tracking and committing files, viewing the commit history and differences between versions, branching and merging, tagging releases, and collaborating remotely through cloning and pushing/pulling. Key objects like blobs, trees, commits and tags are also explained in relation to the Git object database structure.
Perl v5.26 Features (AmsterdamX.pm)
This document summarizes some new features in Perl 5.26, including:
- Scalar hashes now return the number of keys instead of "hash statistics"
- POSIX::tmpnam is deprecated and replaced by File::Temp for creating temporary files
- All opening braces ('{') in regular expressions are now special instead of literal
- The /xx flag can now be used multiple times and applies regex rules inside character classes
- Heredoc syntax now allows indentation of content without affecting output
- The '.' directory is no longer implicitly included in @INC for security reasons
- A new @{^CAPTURE} array provides access to matched regular expression captures
Sp ch05
This document provides information about file I/O functions in C including fopen(), fprintf(), fseek(), ftell(), feof(), ferror(), setvbuf(), tmpfile(), mktemp(), and functions for handling wide characters and internationalization. It includes code examples demonstrating the use of these functions for opening, reading, writing, positioning within, and checking for errors on files. The focus is on file I/O in C and Unicode/internationalization support for file operations.
Best Practices in Handling Performance Issues
The document provides best practices for handling performance issues in an Odoo deployment. It recommends gathering deployment information, such as hardware specs, number of machines, and integration with web services. It also suggests monitoring tools to analyze system performance and important log details like CPU time, memory limits, and request processing times. The document further discusses optimizing PostgreSQL settings, using tools like pg_activity, pg_stat_statements, and pgbadger to analyze database queries and performance. It emphasizes reproducing issues, profiling code with tools like the Odoo profiler, and fixing problems in an iterative process.
Productivity tips for developers
This document provides productivity tips for developers, including customizing dotfiles for Bash, Vim, and Git configurations. It also discusses using text expanders, customizing the MacBook touch bar, and invites readers to share their own productivity tricks. The document recommends dotfiles like .bashrc, .vimrc and .gitconfig to store customizations and aliases. It provides examples of aliases and settings for Bash, Vim and Git configurations.
6 more things about Perl 6
The document discusses various features of Perl 6, including:
- Rat arithmetic handles fractions exactly in Perl 6 unlike Perl 5.
- Catch blocks allow catching exceptions and resuming execution after failures.
- Interpolation syntax in Perl 6 allows interpolating variables, expressions and types into strings.
- Formatting methods like fmt allow formatting numbers to a given precision.
- Lists can contain other lists, and lists are flattened using the Slip method.
- Buffers can represent binary data and support operations like rotating bytes.
- Deeply nested lists can be flattened using various approaches like gathering, shifting, or prefix operators.
GettingStartedWithPHP
This document provides an introduction to getting started with PHP by discussing how to install a web server, writing basic PHP scripts to output text, using PHP variables and arrays, creating PHP functions, and utilizing standard PHP functions. Key topics covered include installing XAMPP on Linux, writing a basic "Hello World" PHP script embedded in HTML, declaring and referencing variables, iterating through arrays with for loops and foreach loops, creating functions that can modify global variables using the global statement, getting user input with fgets from standard input, and examples of common PHP functions like array_push and trim.
Profiling PL/pgSQL
The document discusses using PL/pgSQL functions and stored procedures to implement fantasy football scoring rules and statistics in PostgreSQL. It provides examples of functions to calculate scores for rushing, passing, receiving and touchdowns. It also shows how to use these functions to calculate average yearly scores for players and debug the functions using RAISE notices. The end discusses using the PLProfiler extension to monitor and analyze the performance of PL/pgSQL functions.
Apache Hadoop Shell Rewrite
This is a slightly updated draft of a talk I was planning on giving at Hadoop Summit in 2015. However the abstract was rejected. Rather than toss it, I'm going to share it with all of you on the (almost) 1 year anniversary of the first big commit of this feature!
Keep in mind that this is (currently) locked away in trunk. If you ever want to see this see the light of day, bug your vendors....
OSTU - Sake Blok on Scripting with TShark
Sake Blok, a Wireshark/Ethereal devotee since 1999, works as a Research & Development Engineer for ion-ip in the Netherlands (http://www.ionip.com) . His company provides solutions to customers who want to deliver their applications to users in a fast, secure, efficient and scalable manner. Sake\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'s main focus is to take new products for a spin in their test environment, design custom solutions for customers and troubleshoot the problems customers might encounter while using ion-ip solutions. Two years ago (2006), Sake started to add the functionality he was missing to Wireshark. He also started to fix Wireshark-bugs that were reported on Bugzilla. This work on Wireshark resulted in an invitation from Gerald Combs to join the Wireshark Core Development Team in 2007.
More Related Content
What's hot
Perl 5.28 new features
Perl 5.28 introduces several new features including auto referencing of subroutine arguments, fancier signatures that allow type constraints, distributed map and grep functions to parallelize work, an upgraded safe module, three-state test outcomes in TAP, close enough comparators, new array manipulation operators, upgradable assignment with =^, better support for Git conflicts, and improved diagnostics.
Tests unitaires pour PostgreSQL avec pgTap
Les tests unitaires se sont pas limités au code des applications, des tests peuvent également être effectués sur les données et les schémas des bases de données.
Conférence donnée lors du meetup PostgreSQL le 22 juin 2016 à Nantes
6 things about perl 6
The document discusses several new features in Perl 6, including phasers for controlling program flow, sets and sequences, types, subsets for defining custom types, grammars, and the MAIN subroutine. It provides examples of using phasers to control block and loop execution, built-in set operations like union and intersection, sequence syntax for ranges, type checking for variables and parameters, defining subsets for things like positive numbers, and using grammars and the MAIN subroutine for command line apps.
git internals
This document provides an overview of Git internals and data structures including blobs, trees, commits and refs. It demonstrates how Git objects are stored and referenced in the Git database. Various Git commands are shown such as add, commit, branch, checkout, merge, reset, rebase and reflog. The document explains the relationships between blobs, trees, commits and branches during normal workflows as well as examples of fast-forward merges, rebases and resetting.
Openpear Project の紹介
Openpear is a PHP-based open source content management system and package repository. It provides features such as a dashboard, content management, favorites, and logging. Openpear uses Rhaco as its framework and can be installed by cloning its GitHub repository and running the setup script. Support is available on Google Groups and IRC channel #openpear on Freenode. Openpear aims to be a full-featured CMS and PEAR-like repository hosting service.
Dataiku - Paris JUG 2013 - Hadoop is a batch
This document provides an overview and comparison of Pig, Hive, and Cascading tools for Hadoop. It begins with brief histories of each tool's development: Pig was created at Yahoo Research in 2006 to enable log analytics; Hive was developed by Facebook in 2007 to provide SQL-like queries over Hadoop data; and Cascading was authored in 2008 and associated with Scalding and Cascalog projects. The document then compares features of the tools such as their procedural versus declarative programming models, data typing approaches, integration capabilities, and performance/optimization characteristics to help users choose the best technology.
Development and practical use of CLI in perl 6
YAPC::Tokyo 2019 トーク資料
Slide of my talk about development of command line interface in perl 6.
https://yapcjapan.org/2019tokyo/talks.html#/detail/12
Shell Scripting
Sexta parte da apresentação sobre Shell Scripting realizada na Bluesoft
Confira o video em http://www.vimeo.com/
Fun with processes - lightning talk
"The other day, I was playing with this idea... I wanted to be able to programmatically download a file..."
These are slides from my lightning talk that I gave on https://twitter.com/CodeElixirIO in August 2018.
Practical pig
This document provides an overview of the Pig Latin data flow language for Hadoop. It discusses why Pig is useful for increasing productivity and insulating users from complexity when working with MapReduce. The document provides examples of simple Pig Latin scripts for common tasks like filtering, joining, grouping and aggregation. It also covers performance considerations, user defined functions, common pitfalls, and recommendations.
Pig workshop
This document provides information about a Pig workshop being conducted by Sudar Muthu. It introduces Pig, describing it as a platform for analyzing large data sets. It outlines some key Pig components like the Pig Shell, Pig Latin language, libraries, and user defined functions. It also discusses why Pig is useful, highlighting aspects like its data flow model and ability to increase programmer productivity. Finally, it previews topics that will be covered in the workshop, such as loading and storing data, Pig Latin operators, and writing user defined functions.
Crack.ba
This batch script cracks passwords for RAR files by repeatedly using the Unrar.exe tool to extract files with incrementing password guesses, deleting the extraction directory and Unrar.exe file upon finding the correct password. It gets the RAR file name and path from the user, checks for validity, and then starts an infinite loop extracting with passwords from 0 to infinity until extraction succeeds without errors.
GIT rozproszony system kontroli wersji
This document provides an overview of Git and its basic commands and data structures. It begins with initializing and configuring a Git repository, then covers tracking and committing files, viewing the commit history and differences between versions, branching and merging, tagging releases, and collaborating remotely through cloning and pushing/pulling. Key objects like blobs, trees, commits and tags are also explained in relation to the Git object database structure.
Perl v5.26 Features (AmsterdamX.pm)
This document summarizes some new features in Perl 5.26, including:
- Scalar hashes now return the number of keys instead of "hash statistics"
- POSIX::tmpnam is deprecated and replaced by File::Temp for creating temporary files
- All opening braces ('{') in regular expressions are now special instead of literal
- The /xx flag can now be used multiple times and applies regex rules inside character classes
- Heredoc syntax now allows indentation of content without affecting output
- The '.' directory is no longer implicitly included in @INC for security reasons
- A new @{^CAPTURE} array provides access to matched regular expression captures
Sp ch05
This document provides information about file I/O functions in C including fopen(), fprintf(), fseek(), ftell(), feof(), ferror(), setvbuf(), tmpfile(), mktemp(), and functions for handling wide characters and internationalization. It includes code examples demonstrating the use of these functions for opening, reading, writing, positioning within, and checking for errors on files. The focus is on file I/O in C and Unicode/internationalization support for file operations.
Best Practices in Handling Performance Issues
The document provides best practices for handling performance issues in an Odoo deployment. It recommends gathering deployment information, such as hardware specs, number of machines, and integration with web services. It also suggests monitoring tools to analyze system performance and important log details like CPU time, memory limits, and request processing times. The document further discusses optimizing PostgreSQL settings, using tools like pg_activity, pg_stat_statements, and pgbadger to analyze database queries and performance. It emphasizes reproducing issues, profiling code with tools like the Odoo profiler, and fixing problems in an iterative process.
Productivity tips for developers
This document provides productivity tips for developers, including customizing dotfiles for Bash, Vim, and Git configurations. It also discusses using text expanders, customizing the MacBook touch bar, and invites readers to share their own productivity tricks. The document recommends dotfiles like .bashrc, .vimrc and .gitconfig to store customizations and aliases. It provides examples of aliases and settings for Bash, Vim and Git configurations.
6 more things about Perl 6
The document discusses various features of Perl 6, including:
- Rat arithmetic handles fractions exactly in Perl 6 unlike Perl 5.
- Catch blocks allow catching exceptions and resuming execution after failures.
- Interpolation syntax in Perl 6 allows interpolating variables, expressions and types into strings.
- Formatting methods like fmt allow formatting numbers to a given precision.
- Lists can contain other lists, and lists are flattened using the Slip method.
- Buffers can represent binary data and support operations like rotating bytes.
- Deeply nested lists can be flattened using various approaches like gathering, shifting, or prefix operators.
GettingStartedWithPHP
This document provides an introduction to getting started with PHP by discussing how to install a web server, writing basic PHP scripts to output text, using PHP variables and arrays, creating PHP functions, and utilizing standard PHP functions. Key topics covered include installing XAMPP on Linux, writing a basic "Hello World" PHP script embedded in HTML, declaring and referencing variables, iterating through arrays with for loops and foreach loops, creating functions that can modify global variables using the global statement, getting user input with fgets from standard input, and examples of common PHP functions like array_push and trim.
Profiling PL/pgSQL
The document discusses using PL/pgSQL functions and stored procedures to implement fantasy football scoring rules and statistics in PostgreSQL. It provides examples of functions to calculate scores for rushing, passing, receiving and touchdowns. It also shows how to use these functions to calculate average yearly scores for players and debug the functions using RAISE notices. The end discusses using the PLProfiler extension to monitor and analyze the performance of PL/pgSQL functions.
What's hot (20)
Similar to Hands on Hadoop
Apache Hadoop Shell Rewrite
This is a slightly updated draft of a talk I was planning on giving at Hadoop Summit in 2015. However the abstract was rejected. Rather than toss it, I'm going to share it with all of you on the (almost) 1 year anniversary of the first big commit of this feature!
Keep in mind that this is (currently) locked away in trunk. If you ever want to see this see the light of day, bug your vendors....
OSTU - Sake Blok on Scripting with TShark
Sake Blok, a Wireshark/Ethereal devotee since 1999, works as a Research & Development Engineer for ion-ip in the Netherlands (http://www.ionip.com) . His company provides solutions to customers who want to deliver their applications to users in a fast, secure, efficient and scalable manner. Sake\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'s main focus is to take new products for a spin in their test environment, design custom solutions for customers and troubleshoot the problems customers might encounter while using ion-ip solutions. Two years ago (2006), Sake started to add the functionality he was missing to Wireshark. He also started to fix Wireshark-bugs that were reported on Bugzilla. This work on Wireshark resulted in an invitation from Gerald Combs to join the Wireshark Core Development Team in 2007.
Linux 系統管理與安全：基本 Linux 系統知識
The document provides information on systemd service management commands. It shows examples of using systemctl to start, stop, restart, and check the status of the httpd service. It also displays the output of systemctl status httpd which shows details about the loaded unit, active state, process IDs, and log entries for the Apache HTTP Server service.
Commands documentaion
The document describes steps to load CSV files into Pig and Hive, perform data cleaning and merging, and calculate TF-IDF scores for terms in posts by top users to identify top terms. Multiple CSV files are loaded into Pig, cleaned, merged into a single file, and loaded into Hive tables. Python MapReduce programs are then used to calculate TF-IDF scores to find the most important terms per user.
New text document (2)
This document contains the configuration settings for a high performance proxy server. It defines ports, directories, caching parameters, access controls, refresh patterns and other settings to optimize the proxy's performance and security. Caching is enabled for many media types including videos, images and files. Access and caching is specially configured for sites like YouTube, Facebook and ads.
HDFS metadata (fsimage and edits) difference CDH3 and CDH4
The document compares HDFS metadata management between CDH3 and CDH4. In CDH3, fsimage, edits, and fstime files are used to track metadata changes. In CDH4, edits are separated into files based on transaction IDs rather than time, and metadata is tracked solely through transaction IDs rather than a separate fstime file. Checkpointing behavior also differs, with CDH4 generating new fsimage files and renaming edits files after checkpoints.
Puppet Data Mining
Talk given at the 6th PuppetCamp in Edinburgh.
Covers some of the useful information puppet stores and how to get at it.
Tajo Seoul Meetup-201501
This document provides an introduction and quick start guide for Apache Tajo. It outlines how to install Tajo in local or distributed mode, configure basic settings, and launch a Tajo cluster on a single machine or across multiple machines. The document also introduces the speaker and provides contact information.
2012 coscup - Build your PHP application on Heroku
The document discusses deploying PHP applications on Heroku. It provides an overview of Heroku, including that it is a Platform-as-a-Service, was launched in 2007, uses Amazon Web Services, offers many add-ons, allows easy scaling, supports PostgreSQL, and offers some free usage. It then walks through deploying a basic "Hello World" PHP app on Heroku, including creating an app, adding code, committing and pushing to Heroku, and viewing the deployed app.
Install tomcat 5.5 in debian os and deploy war file
This document provides instructions for installing and configuring Tomcat and PostgreSQL on a Debian server. It includes steps to install Tomcat 5 and 7, add users, configure Java version 1.7, copy files to the server, configure properties, recompile for the Debian environment, deploy WAR files, test, and check services. It also provides commands for managing files, directories, and users in Linux.
Batch is Back: Critical for Agile Application Adoption
Batch processes are critical for agile application development but existing batch scheduling tools are not keeping up with modern approaches like DevOps, containers, cloud, and big data. BMC Control-M is a batch scheduling solution that addresses these issues by providing enterprise-scale workflow scheduling that integrates with the entire technology ecosystem, supports DevOps methodologies, and offers self-service, monitoring, and automation capabilities.
Hadoop on aws amazon
The document describes the steps to configure a Hadoop cluster on Amazon EC2, including launching 1 master and 10 slave EC2 instances, configuring SSH access between nodes, editing the /etc/hosts files to include all node IP addresses and names, installing and configuring Hadoop by editing configuration files like hdfs-site.xml and mapred-site.xml, and defining the master and slave nodes in the masters and slaves files.
Hadoop on aws amazon
The document describes the steps to configure a Hadoop cluster on Amazon EC2, including launching 11 EC2 instances (1 master and 10 slaves), configuring SSH access, modifying hosts files, installing and configuring Hadoop, and starting the Hadoop services. Key steps include separating the instances into a master and slaves, configuring environment variables and HDFS directories, copying the Hadoop installation to all nodes, and modifying security groups to allow communication between nodes.
Instrumentación de entrega continua con Gitlab
Mostraremos el caso real de cómo tenemos implementado en nuestra empresa el flujo de desarrollo para integración y entrega continua, instrumentado con GitLab.
Sesión presentada por David Padilla en SG Next 2017
Learning the command line
An introduction to the Unix command line with a focus on uses for web developers given at Code for Miami MeetUp in April 2014.
Logstash for SEO: come monitorare i Log del Web Server in realtime
This document discusses using Logstash to collect, parse, and analyze log files. It begins with an introduction to logs and Logstash. It then covers installing and configuring Logstash - including using inputs to collect logs, filters to parse and transform data, and outputs to send parsed logs to a storage system. The document demonstrates a Logstash configuration to collect Apache access logs, parse fields using Grok, and output to Elasticsearch for analysis with Kibana. It concludes with tips on using Logstash for SEO-related tasks like analyzing crawler behavior and page load speeds.
Using Nix and Docker as automated deployment solutions
Explains how Docker and Nix work as deployment solutions, in what ways they are similar and different, and how they can be combined to achieve interesting results.
Apache Tomcat Shutdown Startup Script Shell
This script is used to start and stop the Apache Tomcat application server. It checks environment variables and Java installation locations. It then executes the specified command, such as "start", "stop", "debug" passing in JVM options and system properties. The script handles tasks like setting up the classpath, checking for required files, and running Java in the background for startup.
Танки_в_Лунапарке: нагрузочное_тестирование_в_Яндексе
Презентация Андрея Кузьмичева, Руководителя группы нагрузочного тестирования
Конференция SQA Days Spb, ноябрь 2010
Rush, a shell that will yield to you
Rush is a shell with ruby syntax and idioms. This presentation goes over how to use it and embed it in other applications.
Similar to Hands on Hadoop (20)
HDFS metadata (fsimage and edits) difference CDH3 and CDH4
HDFS metadata (fsimage and edits) difference CDH3 and CDH4
2012 coscup - Build your PHP application on Heroku
2012 coscup - Build your PHP application on Heroku
Install tomcat 5.5 in debian os and deploy war file
Install tomcat 5.5 in debian os and deploy war file
Batch is Back: Critical for Agile Application Adoption
Batch is Back: Critical for Agile Application Adoption
Logstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtime
Using Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutions
Танки_в_Лунапарке: нагрузочное_тестирование_в_Яндексе
Танки_в_Лунапарке: нагрузочное_тестирование_в_Яндексе
More from Paul Tarjan
Yahoo! HackU 2010
The document provides information about Yahoo! HackU 2010 including a list of past winning hacks and technologies used. It encourages hackers to fix something they use daily or build something for themselves, focusing on mobile, video, social, local or geo applications. Ideas, sample hacks, and resources are given to help hackers be successful. The schedule and normal process for hacking are outlined as well.
Hadoop Jute Record Python
This document discusses a Python library for parsing Hadoop Record files.
The library includes a parser that can parse Hadoop's Data Definition Language into generic Python data types. It outputs the data structure, but the user must transform it into a class structure.
The parsing library is only part of what is needed - a DDL translator is still needed to fully convert the data definition language into Python classes. Feedback is welcomed to improve the library.
How To Be A Hacker
If you want to build cool stuff and not just be a code monkey in a cubicle, then I recommend you start hacking today.
This is my intro talk for Yahoo's HackU program.
Hacku Intro 2009
This document provides information about HackU, a 24-hour hack day event hosted by Yahoo where participants can build projects using various Yahoo technologies and APIs for prizes. It encourages attendees to come with project ideas and basic coding skills in PHP, JavaScript, and tools like YQL and YUI. The event will include talks on different hacking topics throughout the week leading up to the hack day on Friday where participants can get assistance in areas like Linux, web hosting, APIs, and more to implement their ideas.
Semantic Searchmonkey
Semantic Search + SeachMonkey talk given at Yahoo! Hacku event.
http://developer.yahoo.com/hacku
http://developer.yahoo.com/searchmonkey
SearchMonkey
The document discusses SearchMonkey, an open platform from Yahoo! that allows developers to build structured data into search results. It presents several approaches for providing structured data to SearchMonkey, including embedding RDF or microformats directly into web pages, generating a DataRSS feed from a database, extracting data via XSLT, or calling a remote web service. The document encourages developers to prototype with XSLT initially and provides resources for learning more about SearchMonkey and structured data standards.
More from Paul Tarjan (6)
Recently uploaded
The simplified electron and muon model, Oscillating Spacetime: The Foundation...
Discover the Simplified Electron and Muon Model: A New Wave-Based Approach to Understanding Particles delves into a groundbreaking theory that presents electrons and muons as rotating soliton waves within oscillating spacetime. Geared towards students, researchers, and science buffs, this book breaks down complex ideas into simple explanations. It covers topics such as electron waves, temporal dynamics, and the implications of this model on particle physics. With clear illustrations and easy-to-follow explanations, readers will gain a new outlook on the universe's fundamental nature.
Pengantar Penggunaan Flutter - Dart programming language1.pptx
Pengantar Penggunaan Flutter - Dart programming language1.pptx
Walmart Business+ and Spark Good for Nonprofits.pdf
"Learn about all the ways Walmart supports nonprofit organizations.
You will hear from Liz Willett, the Head of Nonprofits, and hear about what Walmart is doing to help nonprofits, including Walmart Business and Spark Good. Walmart Business+ is a new offer for nonprofits that offers discounts and also streamlines nonprofits order and expense tracking, saving time and money.
The webinar may also give some examples on how nonprofits can best leverage Walmart Business+.
The event will cover the following::
Walmart Business + (https://business.walmart.com/plus) is a new shopping experience for nonprofits, schools, and local business customers that connects an exclusive online shopping experience to stores. Benefits include free delivery and shipping, a 'Spend Analytics” feature, special discounts, deals and tax-exempt shopping.
Special TechSoup offer for a free 180 days membership, and up to $150 in discounts on eligible orders.
Spark Good (walmart.com/sparkgood) is a charitable platform that enables nonprofits to receive donations directly from customers and associates.
Answers about how you can do more with Walmart!"
clinical examination of hip joint (1).pdf
described clinical examination all orthopeadic conditions .
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
Denis is a dynamic and results-driven Chief Information Officer (CIO) with a distinguished career spanning information systems analysis and technical project management. With a proven track record of spearheading the design and delivery of cutting-edge Information Management solutions, he has consistently elevated business operations, streamlined reporting functions, and maximized process efficiency.
Certified as an ISO/IEC 27001: Information Security Management Systems (ISMS) Lead Implementer, Data Protection Officer, and Cyber Risks Analyst, Denis brings a heightened focus on data security, privacy, and cyber resilience to every endeavor.
His expertise extends across a diverse spectrum of reporting, database, and web development applications, underpinned by an exceptional grasp of data storage and virtualization technologies. His proficiency in application testing, database administration, and data cleansing ensures seamless execution of complex projects.
What sets Denis apart is his comprehensive understanding of Business and Systems Analysis technologies, honed through involvement in all phases of the Software Development Lifecycle (SDLC). From meticulous requirements gathering to precise analysis, innovative design, rigorous development, thorough testing, and successful implementation, he has consistently delivered exceptional results.
Throughout his career, he has taken on multifaceted roles, from leading technical project management teams to owning solutions that drive operational excellence. His conscientious and proactive approach is unwavering, whether he is working independently or collaboratively within a team. His ability to connect with colleagues on a personal level underscores his commitment to fostering a harmonious and productive workplace environment.
Date: May 29, 2024
Tags: Information Security, ISO/IEC 27001, ISO/IEC 42001, Artificial Intelligence, GDPR
-------------------------------------------------------------------------------
Find out more about ISO training and certification services
Training: ISO/IEC 27001 Information Security Management System - EN | PECB
ISO/IEC 42001 Artificial Intelligence Management System - EN | PECB
General Data Protection Regulation (GDPR) - Training Courses - EN | PECB
Webinars: https://pecb.com/webinars
Article: https://pecb.com/article
-------------------------------------------------------------------------------
For more information about PECB:
Website: https://pecb.com/
LinkedIn: https://www.linkedin.com/company/pecb/
Facebook: https://www.facebook.com/PECBInternational/
Slideshare: http://www.slideshare.net/PECBCERTIFICATION
How to Fix the Import Error in the Odoo 17
An import error occurs when a program fails to import a module or library, disrupting its execution. In languages like Python, this issue arises when the specified module cannot be found or accessed, hindering the program's functionality. Resolving import errors is crucial for maintaining smooth software operation and uninterrupted development processes.
The History of Stoke Newington Street Names
Presented at the Stoke Newington Literary Festival on 9th June 2024
www.StokeNewingtonHistory.com
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
RPMS Template 2023-2024 by: Irene S. Rueco
Main Java[All of the Base Concepts}.docx
This is part 1 of my Java Learning Journey. This Contains Custom methods, classes, constructors, packages, multithreading , try- catch block, finally block and more.
PCOS corelations and management through Ayurveda.
This presentation includes basic of PCOS their pathology and treatment and also Ayurveda correlation of PCOS and Ayurvedic line of treatment mentioned in classics.
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
বাংলাদেশের অর্থনৈতিক সমীক্ষা ২০২৪ [Bangladesh Economic Review 2024 Bangla.pdf] কম্পিউটার , ট্যাব ও স্মার্ট ফোন ভার্সন সহ সম্পূর্ণ বাংলা ই-বুক বা pdf বই " সুচিপত্র ...বুকমার্ক মেনু 🔖 ও হাইপার লিংক মেনু 📝👆 যুক্ত ..
আমাদের সবার জন্য খুব খুব গুরুত্বপূর্ণ একটি বই ..বিসিএস, ব্যাংক, ইউনিভার্সিটি ভর্তি ও যে কোন প্রতিযোগিতা মূলক পরীক্ষার জন্য এর খুব ইম্পরট্যান্ট একটি বিষয় ...তাছাড়া বাংলাদেশের সাম্প্রতিক যে কোন ডাটা বা তথ্য এই বইতে পাবেন ...
তাই একজন নাগরিক হিসাবে এই তথ্য গুলো আপনার জানা প্রয়োজন ...।
বিসিএস ও ব্যাংক এর লিখিত পরীক্ষা ...+এছাড়া মাধ্যমিক ও উচ্চমাধ্যমিকের স্টুডেন্টদের জন্য অনেক কাজে আসবে ...
How to Add Chatter in the odoo 17 ERP Module
In Odoo, the chatter is like a chat tool that helps you work together on records. You can leave notes and track things, making it easier to talk with your team and partners. Inside chatter, all communication history, activity, and changes will be displayed.
Recently uploaded (20)
The simplified electron and muon model, Oscillating Spacetime: The Foundation...
The simplified electron and muon model, Oscillating Spacetime: The Foundation...
Digital Artefact 1 - Tiny Home Environmental Design
Digital Artefact 1 - Tiny Home Environmental Design
Pengantar Penggunaan Flutter - Dart programming language1.pptx
Pengantar Penggunaan Flutter - Dart programming language1.pptx
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
বাংলাদেশ অর্থনৈতিক সমীক্ষা (Economic Review) ২০২৪ UJS App.pdf
Hands on Hadoop
- 1. Paul Tarjan Chief Technical Monkey Yahoo! http://paulisageek.com
- 6. Map/Reduce
- 8. Python wordcount mapper http://github.com/ptarjan/hands-on-hadoop-tutorial/blob/master/mapper.py
- 9. Python wordcount reducer http://github.com/ptarjan/hands-on-hadoop-tutorial/blob/master/reducer.py