黒河内 倫 / Osamu Kurokouchi No-Full Route Changed Our Lives @AS55394 APRICOT-APAN 2015にて、LT枠で弊社の黒河内が発表した資料です http://jp.apricot-apan.asia/

1. Copyright © GREE, Inc. All Rights Reserved.Copyright © GREE, Inc. All Rights Reserved.
No-Full Route Changed Our Lives
@AS55394
Osamu Kurokochi, Data Center Team, Infrastructure Headquarters

2. Copyright © GREE, Inc. All Rights Reserved.
Name
Osamu Kurokochi
Dept.
Data Center Team,
Infrastructure Headquarters,
GREE, Inc
Self Introduction

3. Copyright © GREE, Inc. All Rights Reserved.
There were 8 border gateway protocol (BGP) routers.
Full route reception was in operation on all routers,
and each BGP router was connected as iBGP peers
in a full mesh topology.
Summer in 2012 (Configuration at the time)
GREE environment
R R
R R
R
R
R
R

4. Copyright © GREE, Inc. All Rights Reserved.
Summer in 2012 (occurrence of fault)
One time, a fault on the Transit side occurred and
caused peers to crash.
Convergence of the routes at the time took time
and the Router CPU froze.
iBGP peers also began to crash which caused chaos.
A shutdown of about 5 minutes lasted
intermittently until convergence. (The only thing
we could do was to watch what happened.)

5. Copyright © GREE, Inc. All Rights Reserved.
Summer in 2012 (cause of fault)
There were 3 main factors.
1. Insufficient hardware processing capability
2. Increased number of routes
3. Too many iBGP peers (not that many)
Caused by one or a mixture of 3 factors above.

6. Copyright © GREE, Inc. All Rights Reserved.
Solution 1. Reinforced hardware
Buy hardware having better performance.
Solution 2. Configuration change
Introduce RR and reduce the number of iBGP-
Peers.
Solution 3. Decreased number of routes
Decrease the number of routes with a mechanism
to reduce the load during a BGP update.
Breakthrough Solutions Considered at the Time

7. Copyright © GREE, Inc. All Rights Reserved.
Key Judgment Point
Replacement of BGP routers at all bases also was
considered but is difficult in terms of effort.
When a procedure
“verification → order →
delivery → maintenance arrangement”
was considered, this remedy was too slow…
It was judged that the problem was difficult to
solve by introducing new hardware.

8. Copyright © GREE, Inc. All Rights Reserved.
Key Judgment Point
We narrowed the solutions down to solution 3.
In our company’s business model, 99% of
accesses were from mobile devices.
The necessity of full route itself was reconsidered
resulting in as follows:
Full route → Partial route + Default route
*Partial Route = 3 domestic mobile carriers and 5
ASs.

9. Copyright © GREE, Inc. All Rights Reserved.
Transit Router
Own router
Transit Router
1 In-house filtering
2 TransitFilter method
Full route
Default route
Partial route
Default route
RIB FIB
Own router
RIB FIB
Partial Route
Default Route
Transparent
GREE adopts this solution.
There Are Two Partial Routes

10. Copyright © GREE, Inc. All Rights Reserved.
Summary of Solutions
Solution 1. Reinforced hardware
Buy hardware having better performance.
→ Verification required and it takes time for delivery.
Solution 2. Configuration change
Implement RR to reduce the number of iBGP-Peers.
→ Verification required, it takes time for delivery and
no conclusive evidence that the problem will be rectified.
Solution 3. Decreased number of routes
Lower the number of routes with a mechanism to
reduce the load during a BGP update.
→ This can solve the problem in a short time and is
reliable.

11. Copyright © GREE, Inc. All Rights Reserved.
Solution of Problem
Number of routes:
At the time, 400,000 routes
→ reduced to approx. 2600 routes
Come on, line trouble!
We actually tried the solution.
These have been further reduced to approx.
1800 routes.

12. Copyright © GREE, Inc. All Rights Reserved.
I thought...
Even without full route,
it is possible to continue the our business.
Do not you think the meaning of
that you have a complete route again?

13. Copyright © GREE, Inc. All Rights Reserved.