Buildoop is an open-source collaboration project that aids in creating custom Linux-based systems using the Hadoop ecosystem. It includes features like build recipes and configuration artifacts essential for software development, as well as a list of available Bill of Materials (BOM) for various versions of Hadoop components. Additionally, it supports integration and testing of software modules through virtualized infrastructure and provides commands for building and deploying packages.

1. buildoop

2. The Buildoop Project
“
Open source collaboration project that
provides templates and tools to help
you create custom Linux-based systems
based on Hadoop ecosystem.

3. Software Development life-cycle

4. Buildoop dissection
●
●
●
“buildoop recipes” specify how a particular package is built. It
includes all the package dependencies, source code locations,
configuration, compilation, build, install and remove instructions.
“Configuration artifacts” set of configuration management files
for operating system deploy and software configuration.
“SIT” the individual software modules are combined and tested
as a group by means of virtualized infrastructure.

5. Buildoop: Build package 1/4
●
List of available BOM – Bill Of Materials. Set of components to
manufacture an end product.
$ buildoop -bom
Available BOM targets:
test.bom
openbus-0.0.1.bom
stable.bom
bigtop-0.7.0.bom
TARGET=CentOS-6.x
conf/bom/
├── bigtop-0.7.0.bom
├── openbus-0.0.1.bom
├── stable.bom
└── test.bom
HADOOP_VERSION=2.2.0_openbus-0.0.1-r1
ZOOKEEPER_VERSION=3.4.5_openbus-0.0.1-r1
HBASE_VERSION=0.94.16_openbus-0.0.1-r1
PIG_VERSION=0.11.1_openbus-0.0.1-r1
HIVE_VERSION=0.11.0_openbus-0.0.1-r1
SQOOP_VERSION=1.4.4_openbus-0.0.1-r1
OOZIE_VERSION=4.0.0_openbus-0.0.1-r1
FLUME_VERSION=1.4.0_openbus-0.0.1-r1
STORM_VERSION=0.9.0.1_openbus-0.0.1-r1
KAFKA_VERSION=0.8.0_openbus-0.0.1-r1
HUE_VERSION=3.5.0_openbus-0.0.1-r1
PROTOBUF_VERSION=2.5.0_r1

6. Buildoop: Build package 2/4
$ buildoop openbus-0.0.1 -info
Target Platform:
CentOS-6.x
Ecosystem versions:
HADOOP: 2.2.0_openbus-0.0.1-r1
ZOOKEEPER: 3.4.5_openbus-0.0.1-r1
HBASE: 0.94.16_openbus-0.0.1-r1
PIG: 0.11.1_openbus-0.0.1-r1
HIVE: 0.11.0_openbus-0.0.1-r1
SQOOP: 1.4.4_openbus-0.0.1-r1
OOZIE: 4.0.0_openbus-0.0.1-r1
FLUME: 1.4.0_openbus-0.0.1-r1
STORM: 0.9.0.1_openbus-0.0.1-r1
KAFKA: 0.8.0_openbus-0.0.1-r1
HUE: 3.5.0_openbus-0.0.1-r1
PROTOBUF: 2.5.0_r1

7. Buildoop: Build package 3/4
$ buildoop openbus-0.0.1 flume -info
Recipe name : flume-1.4.0_openbus-0.0.1-r1.bd
Description : Apache Flume
Home site : http://flume.apache.org/
License
: Apache-2.0
URL base : http://www.eu.apache.org/dist/flume/1.4.0/apache-flume1.4.0-src.tar.gz
MD5SUM hash : a932da09b9edb95b0c4bf52e6480fe21
{
JSON recipe
]
"do_info": {
"description": "Apache Flume",
"homepage": "http://flume.apache.org/",
"license": "Apache-2.0",
"filename": "flume-1.4.0_openbus-0.0.1-r1.bd"
},
"do_download": {
"src_uri": "http://www.eu.apache.org/dist/flume/1.4.0/apache-flume-1.4.0-src.tar.gz",
"src_md5sum": "a932da09b9edb95b0c4bf52e6480fe21"
},
"do_fetch": {
"download_cmd": "wget"
}

8. Buildoop: Build package 4/4
$ buildoop openbus-0.0.1 flume -build
$ buildoop openbus-0.0.1 hadoop -build
$ buildoop openbus-0.0.1 -build
build/deploy/
├── bin
│ ├── flume-1.4.0-openbus0.0.1.noarch.rpm
│ ├── flume-agent-1.4.0-openbus0.0.1.noarch.rpm
│ ├── hadoop-2.2.0-openbus0.0.1.x86_64.rpm
│ ├── hadoop-hdfs-2.2.0-openbus0.0.1.x86_64.rpm
│ ├── hadoop-hdfs-zkfc-2.2.0-openbus0.0.1.x86_64.rpm
│ └── repodata
│
├── filelists.sqlite.bz2
│
├── filelists.xml.gz
│
├── other.sqlite.bz2
│
├── other.xml.gz
│
└── repomd.xml
└── src
├── flume-1.4.0-openbus0.0.1.src.rpm
└── hadoop-2.2.0-openbus0.0.1.src.rpm

9. SIT: System Integration Tests
$ buildoop openbus-tests -info
Available tests for openbus set:
hdfs-test
zookeeper-test
Flume-test
$ buildoop openbus-tests hdfs-test -info
$ buildoop openbus-tests hdfs-test -run
$ buildoop openbus-tests all-test -run
Test scope:
●
●
●
Hadoop package repository deployment.
Puppet configuration deployment.
Smoke testing for Hadoop ecosystem.

10. buildoop
https://github.com/buildoop