Course project for STA525 at Cal Poly Pomona.
Financial data is often to be a subject of time series analysis. Recently, cryptocurrency markets in the world have been rapidly increasing their presence enough to interact with legal currency markets. However, due to a relatively insufficient trading infrastructure and people’s understanding about cryptocurrencies when compared to legal currencies, occasionally some of cryptocurrency show a strong dependency on another currency in terms of their price. Given this, we tried to analyze the relationship between the price of Bitcoin in Japan and the one of Bitcoin Cash in the United States with ARMA (+GARCH) model and correlation analysis.
Paper work: https://www.slideshare.net/KeitaMakino/time-series-analysis-in-cryptocurrency-markets-the-bitcoin-brothers-paper-work
#AcademiaDatathon Finlists' Solution of Crypto Datathon CaseData Science Society
Team UNWE, one of the finalists from #AcademiaDatathon will present their solution to the #cryptocurrency data case. Explore how to perform data modeling with ARIMA and Neural Network.
To learn more visit: https://bit.ly/2uhfF37
Video from the presentation: https://bit.ly/2LlaeYd
Dynamic Binary Analysis and Obfuscated Codes Jonathan Salwan
At this presentation we will talk about how a DBA (Dynamic Binary Analysis) may help a reverse engineer to reverse obfuscated code. We will first introduce some basic obfuscation techniques and then expose how it's possible to break some stuffs (using our open-source DBA framework - Triton) like detect opaque predicates, reconstruct CFG, find the original algorithm, isolate sensible data and many more... Then, we will conclude with a demo and few words about our future work.
#AcademiaDatathon Finlists' Solution of Crypto Datathon CaseData Science Society
Team UNWE, one of the finalists from #AcademiaDatathon will present their solution to the #cryptocurrency data case. Explore how to perform data modeling with ARIMA and Neural Network.
To learn more visit: https://bit.ly/2uhfF37
Video from the presentation: https://bit.ly/2LlaeYd
Dynamic Binary Analysis and Obfuscated Codes Jonathan Salwan
At this presentation we will talk about how a DBA (Dynamic Binary Analysis) may help a reverse engineer to reverse obfuscated code. We will first introduce some basic obfuscation techniques and then expose how it's possible to break some stuffs (using our open-source DBA framework - Triton) like detect opaque predicates, reconstruct CFG, find the original algorithm, isolate sensible data and many more... Then, we will conclude with a demo and few words about our future work.
Bitcoin protocol for developerBitcoin Protocol for DevelopersParadigma Digital
Introducción de Alberto Gómez al protocolo de Bitcoin y al lenguaje Bitcoin Scripting, el cual permite desarrollar características y comportamiento sobre el dinero y las transferencias de valor.
Presentation with Pere Villega (https://www.linkedin.com/in/perevillega/) about Blockchain (esp Bitcoin and Ethereum). Relating Blockchain to FP through foldLeft in Event Sourcing, then adding Crypto hashing and Distributed systems thinking and consensus.
Long hidden in the shadow of daddy bitcoin, its offsprings are now more and more aggressive and eager to dethrone it!
Indeed the last generation of blockchain has undeniable advantages: No more mining, faster or instant transactions, user-friendly interface, and even no more blockchain!
In this talk, we’ll have a look at the problems of the first blockchains and understand how the new generations are trying to solve them ... with varying degrees of success!
TensorFlow London 11: Pierre Harvey Richemond 'Trends and Developments in Rei...Seldon
Speaker: Pierre Harvey Richemond, PhD student at the Data Science Institute with Imperial College
After a successful career in quantitative finance, Pierre is researching deep learning and reinforcement learning at Data Science Institute. He holds several degrees in mathematics and engineering.
Abstract:
In this high-level talk, he will go through the latest recent and significant developments in the theory of reinforcement learning. Topics will range from soft Q-learning to proximal policy optimization and the Monte-Carlo tree search used in AlphaGo Zero. He will discuss strategies to implement these methods in Tensorflow, combine and replicate them in practice, and highlight connections with other related fields such as convex optimization and optimal transport.
Thanks to all TensorFlow London meetup organisers and supporters:
Seldon.io
Altoros
Rewired
Google Developers
Rise London
[Data Meetup] Data Science in Finance - Building a Quant ML pipelineData Science Society
Georgi Kirov shares a common market-neutral statistical arbitrage framework. It will help showcase the many different ways to structure a systematic research project. From data reconciliation and signal backtesting to optimization and execution, what are some principled ways to evaluate and compare ML ideas? This process inevitably depends on the characteristics of a specific strategy, for instance, if it is liquidity-taking or liquidity-making.
This presentation is about world's hot trending topic known as "Cryptocurrency". This presentation covers a general knowledge about cryptocurrency, crypto coins, bitcoin, coin mining. It specifically shows people about how to start mining and what are the basic requirements.
These slides are about detection method for game bots, I presented the slides in NDSS 2016
You can find the original paper in https://www.internetsociety.org/sites/default/files/blogs-media/you-are-game-bot-uncovering-game-bots-mmorpgs-via-self-similarity-wild.pdf
2020/11/19 PRIMA2020: Implementation of Real Data for Financial Market Simula...Masanori HIRANO
Masanori HIRANO, Hiroyasu MATSUSHIMA, Kiyoshi IZUMI, and Hiroki SAKAJI,
"Implementation of Real Data for Financial Market Simulation using Clustering, Deep Learning, and Artificial Financial Market,"
The 23rd International Conference on Principles and Practice of Multi-Agent Systems (PRIMA 2020), Aichi, Nagoya, Japan, Nov. 18-20th, 2020. (Online)
Arbitrage is principally described as the buying and selling of items in respect to the inequity in prices. It is normally used by traders who take advantage of different prices of the same goods in different geographical locations. The profit made is from the price difference. The arbitrage traders buy goods from cheaper places and sell the goods in regions with higher prices and keep the extra money as their profit.
Bitcoin protocol for developerBitcoin Protocol for DevelopersParadigma Digital
Introducción de Alberto Gómez al protocolo de Bitcoin y al lenguaje Bitcoin Scripting, el cual permite desarrollar características y comportamiento sobre el dinero y las transferencias de valor.
Presentation with Pere Villega (https://www.linkedin.com/in/perevillega/) about Blockchain (esp Bitcoin and Ethereum). Relating Blockchain to FP through foldLeft in Event Sourcing, then adding Crypto hashing and Distributed systems thinking and consensus.
Long hidden in the shadow of daddy bitcoin, its offsprings are now more and more aggressive and eager to dethrone it!
Indeed the last generation of blockchain has undeniable advantages: No more mining, faster or instant transactions, user-friendly interface, and even no more blockchain!
In this talk, we’ll have a look at the problems of the first blockchains and understand how the new generations are trying to solve them ... with varying degrees of success!
TensorFlow London 11: Pierre Harvey Richemond 'Trends and Developments in Rei...Seldon
Speaker: Pierre Harvey Richemond, PhD student at the Data Science Institute with Imperial College
After a successful career in quantitative finance, Pierre is researching deep learning and reinforcement learning at Data Science Institute. He holds several degrees in mathematics and engineering.
Abstract:
In this high-level talk, he will go through the latest recent and significant developments in the theory of reinforcement learning. Topics will range from soft Q-learning to proximal policy optimization and the Monte-Carlo tree search used in AlphaGo Zero. He will discuss strategies to implement these methods in Tensorflow, combine and replicate them in practice, and highlight connections with other related fields such as convex optimization and optimal transport.
Thanks to all TensorFlow London meetup organisers and supporters:
Seldon.io
Altoros
Rewired
Google Developers
Rise London
[Data Meetup] Data Science in Finance - Building a Quant ML pipelineData Science Society
Georgi Kirov shares a common market-neutral statistical arbitrage framework. It will help showcase the many different ways to structure a systematic research project. From data reconciliation and signal backtesting to optimization and execution, what are some principled ways to evaluate and compare ML ideas? This process inevitably depends on the characteristics of a specific strategy, for instance, if it is liquidity-taking or liquidity-making.
This presentation is about world's hot trending topic known as "Cryptocurrency". This presentation covers a general knowledge about cryptocurrency, crypto coins, bitcoin, coin mining. It specifically shows people about how to start mining and what are the basic requirements.
These slides are about detection method for game bots, I presented the slides in NDSS 2016
You can find the original paper in https://www.internetsociety.org/sites/default/files/blogs-media/you-are-game-bot-uncovering-game-bots-mmorpgs-via-self-similarity-wild.pdf
2020/11/19 PRIMA2020: Implementation of Real Data for Financial Market Simula...Masanori HIRANO
Masanori HIRANO, Hiroyasu MATSUSHIMA, Kiyoshi IZUMI, and Hiroki SAKAJI,
"Implementation of Real Data for Financial Market Simulation using Clustering, Deep Learning, and Artificial Financial Market,"
The 23rd International Conference on Principles and Practice of Multi-Agent Systems (PRIMA 2020), Aichi, Nagoya, Japan, Nov. 18-20th, 2020. (Online)
Arbitrage is principally described as the buying and selling of items in respect to the inequity in prices. It is normally used by traders who take advantage of different prices of the same goods in different geographical locations. The profit made is from the price difference. The arbitrage traders buy goods from cheaper places and sell the goods in regions with higher prices and keep the extra money as their profit.
Similar to Time Series Analysis in Cryptocurrency Markets (20)
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Unleashing the Power of Data_ Choosing a Trusted Analytics Platform.pdfEnterprise Wired
In this guide, we'll explore the key considerations and features to look for when choosing a Trusted analytics platform that meets your organization's needs and delivers actionable intelligence you can trust.
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
2. Cryptocurrency
• Bitcoin (BTC)
• Starts from 2009, originally introduced by “Satoshi
Nakamoto”
• Blockchain system enables us to make online
transactions without any third-party authorization
process
A→B
1.0BTC
B→C
3.0BTC
A→C
0.5BTC
Bitcoin Blockchain
New block in every 10 minutes
2
3. Dataset
• Price of BTC/JPY at coincheck.com
• Collected on 11/13/2017
• Data every 10 seconds → in total 8640 data points
3
4. Analysis
• Is this a Random Walk?
• Financial data is often said to be a random walk data
• ADF test to determine the probability where the data is
a R.W.[1]
[1] RでGARCHモデル - TokyoR #21
https://www.slideshare.net/horihorio/garch-by-r
4
5. Analysis
• Is this a Random Walk?
• Take a return to make it non R.W.
• 𝑦 𝑡 =
𝑥 𝑡
𝑥 𝑡−1
− 1
5
6. Analysis
• ACF and PACF
• Possibly (P,Q) = (2,0), (0,2), (5,0) or (0,5)?
Otherwise P,Q>0
6
8. Analysis
• sarima
• (P,Q) = (0,3) and (2,0) have a problem in p-values
• The others have some problem in variance and QQ
• Try GARCH model to reduce these issues
8
9. Analysis
• sarima
• (P,Q) = (0,3) and (2,0) have a problem in p-values
• The others have some problem in variance and QQ
• Try GARCH model to reduce these issues
9
10. Analysis
• GARCH
• What should be for GARCH(P,Q)?
• Difficult to find them → for loop[1]
• Determine the best model based on BIC.
[1] RでGARCHモデル - TokyoR #21
https://www.slideshare.net/horihorio/garch-by-r
10
11. Analysis
• GARCH
• Assume t distribution for the residual as it’s a financial
data[2]
not normal
[2]資産価格の実証分析/金融経済論 II
http://www.ier.hit-u.ac.jp/~iwaisako/class/Efinance2008/Efinance04_08dist.pdf
11
22. Cryptocurrencies
• Bitcoin Cash (BCH)
• Starts from August 1st, 2017, having been hard forked
from original BTC.
A→B
1.0BTC
B→C
3.0BTC
A→C
0.5BTC
Bitcoin Blockchain
A→D
1.5BCH
E→B
2.3BCH
Bitcoin Cash Blockchain
Fork on 08/01
22
27. Strategy
• Use a bias in the markets
• JPY has much less market share on BCH
Drastic change in BCH/USD price can trigger a negative
effect to BCH,/JPY but the other way would not be
happen
https://www.cryptocompare.com/coins/
27
28. Strategy
• Is that true?
• Let’s see the CCF
→The least CCF values appear at lag=2 and 3
28
30. Strategy
• Base Algorithm
Look at the high and low price of
BCH/USD in recent 10 minutes
Look at the average price of
BCH/USD in recent 30 seconds
Is that much higher/lower than the
average of past 30 seconds?
Is the difference high?
Buy/Sell BTC/JPY
F
T
T
F
30
34. Conclusion
• The price of bitcoin can be fit with ARMA + GARCH
model, assuming the residual follow a t distribution.
• However, it is virtually useless to forecast a market
trend.
• Comparison with another market can enable us to
predict a bias in the short time ahead if there is a
special condition, such as a currency is drastically
increasing its market share.
• To win in this field, it is most important to keep up
to latest trends and information.
34
This topic.
Have you ever heard about bitcoin recently?
Yes, cryptocurrency is a hot topic this year.
Bitcoin is a kind of currency, originally introduced by Satoshi Nakamoto and that is powered by a blockchain system.
Blockchain is a continuously growing set of records. People can view all the history of the records, but cannot edit it anymore.
Using this, we now able to make a online payment without any 3rd party authorization such as governments, banks, public companies and so forth.
Because it is regarded as a currency, there is a exchange of bitcoin.
So today’s objective is this time series. The records were collected via coincheck.com on 11/13/2017, showing the price of BTC in JPY, frequency is every 10 seconds thus the total amount is 8640 records.
Firstly, it is often said that financial data is actually behaves as a random walk.So let’s see if the data is so. Here we have a ADF-test and get these p-values, indicating that there are some possibility of being a random walk.
Then, taking a return of the price make it non-random walk data, as you see the p-values are now very small.
Next, look at ACF and PACF, it should be reasonable to consider these four, either AR or MA models, or otherwise ARMA model.
Using auto arima gives us some possibility of ARMA model here, ARMA 2,3 or 4,1 .
Then examine the diagnosis of these models. As you see, there is a problem in p-values when we select 0,3 or 2,0.
The others have a problem in residuals and QQ Plots. Then considering GARCH may help us at finding a better model.
However, it is difficult to say generally which P,Q for GARCH is good for this data. Therefore, we have create all the models with all possible combinations of P,Q from 1,1 to 4,3 and compared their BIC to find the best model.
At this procedure, we have assumed that the residual follows a t distribution. It is because in this dataset most of the records are very close to 0.
After the step of for loop, we indeed got P,Q = 1,1 for all ARMA models. This is 2,3 + 1,1.
4,1 + 1,1, looks good.
0,5 + 1,1. You see there is a fault here.
And 5,0 1,1. It has some problem here too.
So, our best model should be ARMA 4,1 + GARCH 1,1.
Then let’s see the diagnostics.
Residuals with and without GARCH look identical at a glance, but
Actually there is a difference. And… it probably make some positive effects for stabilize the variance.
QQ plot. Actually it is difficult to say something. Which one do you think better?
Histogram of the residuals. It is also hard to find a significance, but at least not too far from the distribution.
However, when it comes to prediction, we clearly see it says little about the future.
Yes, forecasting a price of bitcoin could never be so easy that we can do for the term project, unfortunately.
But, you know, there is another way.
Yes, there are more and more currencies than bitcoin in the world.
From now on, we’ll talk on this, bitcoin cash, a brother of the bitcoin.
Bitcoin cash is a relatively new cryptocurrency which was born in this august being hard forked from the original bitcoin. It means the blockchain of bitcoin cash has all the history of bitcoin blockchain before this july.
And because it is 8 years newer than bitcoin, its specification was greatly improved to endure increasing transactions in these days.
So, this younger brother is much better than the older one. Some people welcomed this currency and enthusiastically invested, then.
The price of BCH suddenly exploded in this November. The ratio between BTC and BCH radically increased up to 4 to 5 times in 48 hours.
And this is our second data, collected via bittrex.com and shows the price of BCH in USD. Does anybody notice something? So
These trends have a negative relationship.
And here is our strategy. This pie chart explains the currency share in the cryptocurrencies. You see now JPY has the greatest volume in trading BTC, but never appear in the right chart, the share in BCH. It indicates that changes in BCH market can trigger a following change in BTC market, but the other way would not be happen.
How to confirm this assumption? See this CCF in the prices.
The bottom, least values on the CCF lie around lag=2 or 3.
It proves that there is about 30 seconds delay in propagating BCH/USD effect to BTC/JPY market.
And this is the scatterplot of the prices with 30 seconds lag. Showing there are typically negative relations when BCH gets high or low price.
Then try this flow chart.
First look at relatively long trend in BCH, if it finds a high volatility, then see in shorter span and detect an abrupt change in the price of BCH to trade BTC.
So we improved this... And let it run on Microsoft Azure web app.
And here is the result. In the day, we made about 30 closed trades and succeeded in increasing the fund about 6%.
This chart illustrates how the system bought and sold. Although there might be some suggestion, overall sell point is relatively higher than buy point,
In conclusion, we have evaluated an ARMA + GARCH model that can explain the bitcoin price. However, it is not enough to predict the future trend.
Then, comparison between two currencies, especially either of them is drastically changing its price, would help us at finding the short-term prediction easily.
Therefore, you must be up-to-date if you want to win in this field.
The end?
Well, can you guess which one finally won this war?
[Okay, so imagine today we have a new currency USD Next, which doesn’t have any physical object and there is no store which accept it yet, but it is completely secure, easy to use on the web, high anonymity, brah brah brah...
Do you wanna investigate all your asset to this currency?]
Yes, the name value is much much overwhelming...