This document discusses how to build an agent for automated trading using reinforcement learning. It covers modeling the market environment and agent, training the agent using backtesting and paper trading, and integrating with real trading APIs. Specific techniques mentioned include neural networks, gym environments, rule-based strategies, and the Interactive Brokers API.

1. Training the agent for trading use
Interactive Broker python api

2. How to build your own agent
● Modeling
● Training
● Backtesting
● Real-time paper trading

3. How to build the model
Market
Environment
Observation
Agent
Action
Reward

4. How to build the model
● Input : State t (Observation)
● Output : Action (Price Order, Limit Order, long or short, ask or bid)
● Reward
● St, At, Rt, St+1, At+1, Rt+1……

5. How to build the model
● Model architecture
○ FNN
○ CNN
○ RNN(LSTM, GRU)
○ CNN + RNN(LSTM, GRU)
○ UFCNN

6. How to build the model
● Objective Function
● Maximize Total Reward
● Set correct answer using historical data (ex : future>3% buy...)
○ RL way Rt+1*gamma + Rt+2*gamma^2 …
● Reinforcement Learning

7. Open Ai Gym
● https://github.com/openai/gym
● https://gym.openai.com/evaluations/eval_glkKKInTm6GlmcOQRZuhQ
import gym
env = gym.make("Breakout-v0")
observation = env.reset()
for _ in range(1000):
env.render()
action = env.action_space.sample() # your agent here (this takes random actions)
observation, reward, done, info = env.step(action)

8. Trading Gym
● https://github.com/Yvictor/TradingGym
import random
import pandas as pd
import trading_env
df = pd.read_hdf('dataset/SGXTW.h5', 'STW')
env = trading_env.make(obs_data_len=256, step_len=128,
df=df, fee=0.1, max_position=5, deal_col_name='Price',
feature_names=['Price', 'Volume', 'Ask_price','Bid_price',
'Ask_deal_vol','Bid_deal_vol', 'Bid/Ask_deal', 'Updown'])
env.reset()
env.render()
state, reward, done, info = env.step(random.randrange(3))
### random choice action and show the transaction detail
for i in range(500):
state, reward, done, info = env.step(random.randrange(3))
env.render()
if done:
break
env.transaction_details

9. Trading Gym
● Build your own agent
class YourAgent:
def __init__(self):
# build your network and so on
pass
def choice_action(self, state):
## your rule base condition or your max Qvalue action or Policy Gradient action
# action=0 -> do nothing
# action=1 -> buy 1 share
# action=2 -> sell 1 share
## in this testing case we just build a simple random policy
return np.random.randint(3)

10. Trading Gym
● backtesting
agent = YourAgent()
state = env.backtest()
done = False
while not done:
state, reward, done, info = env.step(agent.choice_action(state))
#print(state, reward)
#env.render()
if done:
break
env.transaction_details

11. Trading Gym
● Rule base usage
env = trading_env.make(obs_data_len=10, step_len=1,
df=df, fee=0.1, max_position=5, deal_col_name='Price',
feature_names=['Price', 'MA'],
fluc_div=100.0)
class MaAgent:
def __init__(self):
pass
def choice_action(self, state):
if state[-1][0] > state[-1][1] and state[-2][0] <= state[-2][1]:
return 1
elif state[-1][0] < state[-1][1] and state[-2][0] >= state[-2][1]:
return 2
else:
return 0
# then same as above

12. Real-time paper trading
● Interactive Broker API
○ Cross platform
○ Need to open TWS or IB gateway
○ C++, C#, Java, Python, VB …
○ http://interactivebrokers.github.io/#

13. Real-time paper trading
● Official IB python API
○ Requirements : python 3.1 or higher, TWS 952.x or higher
○ Install API for python https://git.io/vHjng
wget http://interactivebrokers.github.io/downloads/twsapi_macunix.973.02.zip
unzip twsapi_macunix.973.02.zip
cd IBJts/source/pythonclient
python setup.py install
Cd ../../smaples/Python/Testbed/
python Program.py -p 4002

14. Real-time paper trading
● ibapi.client.EClient, ibapi.wrapper.EWrapper
from ibapi.client import EClient
from ibapi.wrapper import EWrapper, iswrapper
class TWSClient(EWrapper, EClient):
def __init__(self):
EClient.__init__(self, wrapper=self)
client = TWSClient()
client.connect("127.0.0.1", 4002, clientId=1)
print("serverVersion:%s connectionTime:%s" % (client.serverVersion(),
client.twsConnectionTime()))

15. Real-time paper trading
● Contract
○ contract = ibapi.contract.Contract()
contract.symbol = "EUR"
contract.secType = "CASH"
contract.currency = "GBP"
contract.exchange = "IDEALPRO"
○ contract = ibapi.contract.Contract()
contract.symbol = 'TSLA'
contract.secType = 'STK'
contract.currency = 'USD'
contract.exchange = 'SMART'

16. Real-time paper trading
● Order
○ Market
○ Market If Touched
○ Market On Close, Open
○ Limit Order
○ Limit if Touched
○ https://interactivebrokers.github.io/tws-api/basic_orders.html#gsc.tab=
0

17. Real-time paper trading
● Market Data
○ Live Market
○ L1
■ Stocks, Futures and others , Frequency : 250 ms
■ US Options Frequency : 100 ms
○ L2
○ Historical Data
○ Real Time Bars

18. Real-time paper trading
● Accounts
○ Positions
○ Summary
○ Updates
○ https://interactivebrokers.github.io/tws-api/account_portfolio.html#g
sc.tab=0

19. Real-time paper trading with Trading GYM
WIP

20. DEMO