The document discusses the SMACK stack 1.1, which includes tools for streaming, Mesos, analytics, Cassandra, and Kafka. It describes how SMACK stack 1.1 adds capabilities for dynamic compute, microservices, orchestration, and microsegmentation. It also provides examples of running Storm on Mesos and using Apache Kafka for decoupling data pipelines.
2. Elodina is a big data as a service platform built on top
of open source software.
The Elodina platform solves today’s data
analytics needs by providing the tools and
support necessary to utilize open source
technologies.
http://www.elodina.net/
3. Whats SMACK Stack?
SMACK stack 1.0 has been traditionally Spark, Mesos, Akka, Cassandra and
Kafka lots https://dzone.com/articles/smack-stack-guide and lots lots more https:
//www.google.com/webhp?q=smack%20stack
Now we are going to introduce SMACK Stack 1.1 and talk more about dynamic
compute, micro services, orchestration, micro segmentation all part of what you
can do now with Streaming, Mesos, Analytics, Cassandra and Kafka
4. The free lunch is over!
http://www.gotw.ca/publications/concurrency-ddj.htm
5. Many industries still don’t get it
XML is everywhere but we have alternatives!
We can support XML interface but don’t have to take on the burden of the extra
data. You can save A LOT of overheard just by having a pre-processing step
taking the XML, turning it into Avro and processing and storing that.
It works https://github.com/elodina/xml-avro
You can even process the response in Avro but return the result in XML, more on
that later though!
6. You need to be running Mesos. Lots of options here!
What is most important is that you abstract your “Provider” from your “Grid”.
What is “The Grid”?
It is your PaaS layer you deploy too that runs your software. (aka your new
awesome super computer)
The grid is your mesos cluster. You are likely going to have more than one so plan
accordingly. Think of it as immutable infrastructure, the computer does.
Step 1
8. The Grid … 2.0 ...
https://github.com/elodina/sawfly/blob/master/cloud-deploy-grid.md
Program against your datacenter like it’s a single pool of resources Apache Mesos abstracts CPU,
memory, storage, and other compute resources away from machines (physical or virtual), enabling
fault-tolerant and elastic distributed systems to easily be built and run effectively. Mesosphere’s Data
Center Operating System (DCOS) is an operating system that spans all of the machines in a datacenter
or cloud and treats them as a single computer, providing a highly elastic and highly scalable way of
deploying applications, services, and big data infrastructure on shared resources. DCOS is based on
Apache Mesos and includes a distributed systems kernel with enterprise-grade security.
13. But there is more!
● Provisioning
● Micro Segmentation
● Orchestration
● Configuration Management
● Service Discovery
● Deployment Isolation and Identification
● Telemetry, Tracing, Ops Stuff, Etc
● Oh My!
It boils back down into stacks! https://github.com/elodina/stack-deploy and how
you are working with your schedulers in your cluster ultimatlly.
16. In the Grid you need Schedulers!
● Kafka – Producer/Consumer-based message queue management
● Exhibitor – Supervisor for distributed persistence (like ZooKeeper)
● Cassandra/DSE – HA, scalable, distributed NoSQL data storage
● Storm – Topology-based Real-time distributed data streaming
● Monarch – Distributed Remote Procedure Calls, Kafka REST interface and schema repository
● Zipkin – Configure, launch and manage Zipkin distributed trace on Mesos
● HDFS – Configure, launch and manage HDFS on Mesos (coming soon)
● Stockpile – Consumer to “stock pile” data into persistent storage (mesos scheduler only for c* now)
● MirrorMaker – Consumer to make a mirror copy of data to destination
● StatsD – Producer to pump StatsD on Mesos into Kafka for consumption, preserves layers
● SysLog – Producer to pump Syslog on Mesos into Kafka for consumption, preserves layers
https://github.com/elodina/
34. Apache Kafka
• Apache Kafka
o http://kafka.apache.org
• Apache Kafka Source Code
o https://github.com/apache/kafka
• Documentation
o http://kafka.apache.org/documentation.html
• Wiki
o https://cwiki.apache.org/confluence/display/KAFKA/Index
49. Streaming & Analytics
● The landscape of streaming is about to get more fragmented and harder to
navigate. This is not all bad news and it is not much different than where we
were with NoSQL 6 years ago or so.
● Different systems are getting really (really (really)) good at different things.
○ Dag based systems
○ Event based systems
○ Query & Execution Engines
○ Streaming Engines
○ Etc!
60. Go Kafka Client - Fan Out Processing
https://github.com/elodina/go-kafka-client-mesos
● Dynamic Kafka Log workers
● Blue/Green Deploy Support
● Fan Out Processing
● Auditable
● Batches
● Scalable/Auto-Scalable