Downloaded 142 times
![Info
Analysis
Filter
[ json ]
Neural Network
How can it be addressed?](https://image.slidesharecdn.com/presoapacheconv2-150423190859-conversion-gate02/75/A-Stock-Prediction-System-using-Open-Source-Software-13-2048.jpg)
Uploaded byFred Melo
A Stock Prediction System using Open-Source Software
The document discusses a stock prediction system utilizing open-source software that incorporates machine learning methodologies such as neural networks and clustering for analyzing historical datasets. It highlights the challenges of integrating real-time data processing into traditional models and proposes a solution using a data lake architecture with continuous learning capabilities. The system emphasizes extensibility, fault tolerance, and real-time analytics, as well as various integration options with existing tools and frameworks.
More Related Content
![DevOps, CD and [Data] Microservices](https://cdn.slidesharecdn.com/ss_thumbnails/qconsp-devopsfordata-160405174944-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to A Stock Prediction System using Open-Source Software
A Stock Prediction System using Open-Source Software
- 2. A Stock PredictionSystem using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito wmarkito@pivotal.io @william_markito
- 4. It's all aboutDATA Data Sources Look for patterns Prediction
- 7. Machine Learning isthe answer Neural Networks Clustering Genetic Algorithms
- 8. Train with historicaldataset Apply model to the new input Applying Machine Learning
- 9. Hard to addnew data sources Why? Hard to scale Why so hard? Hard to make it real-time
- 10. HDFS Data Lake Store Analytics Hardto change Labor intensive Inefficient No real-time information ETL based Data-source specific Traditional models are reactive and static
- 11. HDFSData Lake Expert System/ Machine Learning In-Memory Real- Time Data Continuous Learning Continuous Improvement Continuous Adapting Data Stream Pipeline Multiple Data Sources Real-Time Processing Store Everything Stream-based, real-time closed-loop analytics are needed
- 12. Info Analysis Look at pasttrends (for similar input) Evaluate current input Score / Predict Neural Network How can it be addressed?
- 13. Info Analysis Filter [ json ] NeuralNetwork How can it be addressed?
- 14. Info Analysis Filter Enrich Neural Network Howcan it be addressed?
- 15. Info Analysis Neural Network Filter EnrichTransform How can it be addressed?
- 16. Info Analysis Filter Enrich Transform NeuralNetwork How can it be addressed?
- 17. Info Analysis Filter Enrich Transform Transform NeuralNetwork How can it be addressed?
- 18. Neural Network In-Memory DataGrid Real-time scoring How can it be addressed? Train
- 19. Neural Network In-Memory DataGrid Front-end Update Push How can it be addressed?
- 20. Ingest Transform Sink SpringXD Store/ Analyze Fast Data Distributed Computing Predict / Machine Learning Other Sources and Destinations JMS Streaming real-time analytics architecture
- 21. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 22.
- 23. SpringXD INGEST / SINKPROCESS ANALYZE • Little or no coding required • Dozens of built-in connectors • Seamless integration with Kafka, Sqoop • Create new connectors easily using Spring • Call Spark, Reactor or RxJava • Built-in configurable filtering, splitting and transformation • Out-of-box configurable jobs for batch processing • Import and invoke PMML jobs easily • Call Python, R, Madlib and other tools • Built-in configurable counters and gauges Data Stream Pipelining
- 24. SpringXD XD NodesXD NodesXDNodesXD Nodes Ingest SpringXD Split Filter Transform Sink XD admin XD Nodes Ingest Split Filter Transform Sink Stream Deployment Messaging Scale-Out and HA Architecture
- 25. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 26.
- 27.
- 28.
- 29. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 30.
- 31.
- 32. medium avg (x+1) relative strength (x) mediumavg (x) price(x) Neural Network
- 33.
- 34. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 36.
- 37.
- 38.
- 40. Follow-up: In-Memory Unconference "Aplace for all things in-memory: projects, people, ideas, roadmaps, discussions." Location: Hill Country A/B” Weds 4:15pm - 6pm. (after this talk) The demo code is on GitHub! @fredmelo_br @william_markito