More Related Content

Slideshows for you(20)


Manta Unleashed BigDataSG talk 2 July 2013

  1. Manta Unleashed BigDataSg Meetup 2 July 2013 Christopher W. V. Hogue Ph.D.
  2. Big Data in 2002 – NBLAST - Computing 361,249,575,000 Protein Sequence Alignments & storing significant hits
  3. Big Data in 2003 – Distributed Computing, Tiered Architecture for 10 Billion Protein 3D structure samples Volunteer Computing Blueprint Data Center
  4. What is Manta? • Manta is a new operating-system level component of the IaaS platform of Joyent released June 26 2013. • Manta is an object store system for big-data that you can compute on without moving your data • Manta provides map-reduce capability for executing POSIX standard, arbitrary compute jobs directly on cloud storage servers • Manta allows map-reduce operations formed by any standard UNIX command or application in any run-time language without moving stored data without Hadoop or Java code without loading raw data into a database
  5. What Operating System? • Manta is built on SmartOS, using the illumos kernel, which is open-source UNIX • SmartOS is Not GNU/LINUX • SmartOS is a very lightweight illumos distro for cloud hypervisors with KVM and storage that runs in RAM from PXE/CD/USB boot media • Derived from Sun Microsystem’s Open Solaris • Over 10,000 packages supported via pkgsrc system
  6. illumos is the Open Source Unix kernel forked from Solaris Cloud OS Server OS Storage OS Kernel DTrace Crossbow Zones ZFS SMF MDB CDDL Oracle closed its Solaris source… Aug 2010 Database OS Jan 2010 and more… Kernel Innovations Bugfixes GCC build ZFS feature flags ZFS background delete ZFS LZ4 compression KVM Type 1 hypervisor UNIX System V Release 4 Four years of legal work to open-source Solaris. 2004-2008 1992
  7. Manta – What is SmartOS? • SmartOS is Joyent’s lightweight illumos kernel based operating system optimized for high-performance cloud computing. • illumos is an open-source fork of Open Solaris, supported by Joyent, Nexenta, OmniTI, DEY systems, and Delphix and other core committers. • After Oracle bought Sun Microsystems, many Solaris software engineers, those who built ZFS, Dtrace and other components, left Oracle and joined the illumos effort. • illumos distros that you can experiment with include SmartOS, OmniOS, OpenIndiana, and NexentaStor. • Prerequisite for Manta Use: Your code needs to run/tested on (x86) illumos!
  8. • Started in 2004 • IaaS hosting: – Windows, Linux, FreeBSD KVM images – LinkedIn , Wanelo, Voxer, Storify, Geeklist, Tripshare … many others – Singapore’s Reebonz ( • 4 Primary Data Centers -> • 3rd Party Smart Data Center Licensees who run Joyent-Powered Clouds, e.g.: – Telefonica – Spain – – MiCloud – Taiwan – – Libero – Italy – • Class-1 DC Operators • SSAE 16 Certified • Multi-layered Physical Security • Highly-Redundant Power • Early Warning Fire Suppression • All Tier-1 ISP Connectivity • 10gb/40gb Fully-Meshed Network • Full Peering, Fiber Connectivity May 20 2013 – Dell drops Open Stack Cloud, Partners with Joyent for high-performance, high- availability IaaS service provision.
  9. Joyent as an IaaS provider • Has full development control of the entire operating system stack • Is the corporate steward of the Node.js Javascript run-time language • Community friendly - provides SmartOS image downloads, source for free, and support • You can deploy a private cloud for free with 3rd party management software “Project Fifo”
  10. SmartOS Storage Implementation • All SmartOS storage is local, on ZFS – Integrated disk/volume management – Copy-on-write – Self-healing – Protection against silent data corruption – No hardware RAID dependency – Striping, RAID-Z with no write hole – No fsck resilvering – Built-in filesystem compression options – Compress a subdirectory – Snapshots – zfs send / receive – Integrated SSD IO caching – Add drives with one command, while in production • Manta in the Joyent Datacenter is built on ZFS – no SAN, no NAS head nodes – no tiered layers – standard commodity Intel servers – 4 U servers with 73 TiB of user data – basic SAS HBA technology – Every object is stored on 2 ZFS pools by policy default, local to the server on which it is accessed – Architecture leads me to speculate that Manta stands for “Manta is Not Tiered Architecture”
  11. Manta Features • A multi-datacenter object store • Fine-grained replication commands • No object size limits • Per-object replication policies • Filesystem-like namespace including directory queries • Up to 1 million files per directory • Public folders for CDN data delivery • Read-after write consistency • SnapLinks – a file hard-link (ln) and snapshot mash- up, allowing alternate file naming and versioning in place. Use to mimic data movement. • REST with JSON API • Interactive shell access through Node.js driven SDK and commands • Compute in place with map-reduce processing with arbitrary code and scripts without data movement • GuardTime keyless data signatures and validation
  12. Manta’s Compute-on-Storage • On AWS E3 • Move the “big data” into – EC2 – Hadoop • Then orchestrate a method to run the query • Then clean up additional big data instances • On Manta • grep in place on the storage servers • Manta hands back your job output in a new folder For a simple grep style text query in a big-data collection of server logs:
  13. How does Manta work? End User • Install Node.js package with mlogin() and local Manta commands • Local Node.js environment includes Manta interactive shell and fast I/O data and command transfers up to the Manta Data Center . • Commands transit via REST APIs with JSON encoding. These can be called directly. Data Center • Connects to End User • Distributes and commits data uploads according to replication policy (2 by default) • Fast consistency, data is ready to use without waiting for synch • Jobs are launched in SmartOS Zone VM images on the server. • The hashed UID of the Zone that is launched becomes the job number/directory for output data
  14. Manta Commands Client-Side Utilities Installed locally as Joyent Manta Node.js SDK. Also available to your jobs on the data-side -- -> • mls - Lists directory contents • mput - Uploads data to an object • mget - Downloads an object from the service • mjob - Creates and runs a computational job on the service • mfind - Walks a hierarchy to find names of objects by name, size, or type • mlogin - Interactive session client • mln - Makes SnapLinks between objects • mmkdir - Make directories • mrm - Remove objects or directories • mrmdir - Remove empty directories • msign - Create a signed URL to a object stored in the service • muntar - Create a directory hierarchy from a tar file Data-Side Utilities Additional commands are available to your jobs in the data- side compute environment: • maggr - Performs key-wise aggregation on plain text files. • mcat - Emits the named object as an output for the current task. • mpipe - Output pipe for the current task. • msplit - Split the output stream for the current task to many reducers. • mtee - Capture stdin and write to both stdout and a object. Control interactively via shell-like SDK, OR automate with REST + JSON APIs.
  15. Manta patterns for job creation • $ mjob create –m ’command-to- map’ –r ‘command to reduce’ • Big Data Map Reduce version of grep: – (GNU grep –H prints name of file matching pattern, so you know what file is matched) • $ mjob create -m ’grep -H --label=$MANTA_INPUT_OBJECT pattern’ -r cat
  16. Manta Documentation – Total Word Count in text file collection with map-reduce of wc + awk 1-liner Interactive REST + JSON API
  17. Manta Documentation – Image conversion with ImageMagik “convert”
  18. What software can I run on Manta? Thousands of ready to use UNIX packages on the VM image: • Python • Perl • R • Node.js • Java • ImageMagik • ffmpeg • OpenSSL • Sqlite • MySQL client • Postgres client Or run custom software that is not on the VM image: • These are called Assets • Can be interpretable code or SmartOS compatible binaries • Upload a SmartOS compatible package (e.g. tarball as tgz or a script file) on Manta • Use a job script that unpacks the custom asset inside the Manta VM, and executes it. • Use standard Unix approaches for error loging, output, pipes and tees.
  19. Use Cases • Democratization of BIG DATA – No longer in the hands of a few • Mass market self-logging devices – Transportation/Automotive – E-health monitoring systems – Sensor networks • Scientific paper PDF collections – Federate collections – Allow large scale text mining • Genomic Sequence Analysis – Store Raw Data – Move compute pipeline to data – Meta-pipelines in parallel for computing over old data with new knowledge • Running a checksum over your data to assure its integrity • Log processing: clickstream analysis, MapReduce on logs • Text processing including search • Image processing: converting formats, generating thumbnails, resizing • Video processing: transcoding, extracting segments, resizing • Data Analysis, Mining and Graphing with NumPy, SciPy and R
  20. Manta Pricing Manta compute charges are by the second: $0.00004/GB DRAM * sec If you run 1000 parallel tasks in 32GB DRAM instances on 1000 objects and they each take a second, then you've used 32000 seconds of time and the cost for this job would be $1.28. Storage charges are slightly less than Amazon E3 Bandwidth IN is free Bandwith OUT has tiered charges. Request Type Price per unit of requests Delete Free POST, PUT, LIST (“GET DIR”) $0.005/1000 requests GET, OPTION, HEAD $0.004/10000 requests Storage Tier Default (2 copies)Price per GB (per individual copy) First 1 TB/mo $0.086 $0.043 Next 49 TB/mo $0.072 $0.036 Next 450 TB/mo $0.064 $0.032 Next 500 TB/mo $0.058 $0.029 Next 4000 TB/mo$0.054 $0.027 Next 5000 TB/mo$0.050 $0.025 Default is 2 copies. When submitting an object to the service, you can specify the number of copies stored, from one (1) to six (6).
  21. Deploy a Fast, Scalable, Free, Open Source Private IaaS Cloud Today. • SmartOS • Project FiFO My PXE boot 2-node desktop IaaS Cloud setup Fifo Web Console managing SmartOS KVM Type 1 (bare metal) Hypervisor