Ran Silberman, Big Data, Technical Leader at LivePerson, participated in "Fullstack Developers Israel" meetup: "Never-Ending Data Streams" (http://www.meetup.com/full-stack-developer-il/events/166864612/).
In LivePerson we process great amounts of data.
The data is stored in Hadoop and can be used for batch processing and querying.
LivePerson use Kafka and Storm to complete a Big-Data solution for Real-time processing in addition to batch processing.
In his lecture, Ran provides and overview of the implementation of real-time data streaming at LivePerson and provides some tips and tricks from his experience, covering:
1) High Availability
2) Data consistency
3) Data format and schema enforcement
4) Auditing data integrity