Hybrid Planner for Smart Charging of Electric Fleets

1. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal | Copyright © 2021 Tata Consultancy Services Limited
A Hybrid Planning System for Smart Charging of Electric Fleets
Kshitij Garg, Ajay Narayanan, Prasant Misra, Arunchandar Vasan, Vivek Bandhu, Debarupa Das
Tata Consultancy Services – Research, India
AI-ML Systems 2022

2. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Background
2
Electric vehicle (EV) adoption is increasing for last-mile deliveries (especially in the e-commerce segment) !
Key Factors:
1. Reduced OpEx
2. Sustainability benefits, which directly or indirectly impact (1)
3. Better control over the fleet transition roadmap from ICEVs to EVs
▪ Demand-side: reasonably predictable due to routine/repeating nature of delivery operations
(each trip is around 80-100 km with 30 deliveries)
▪ Supply-side: uncertainty due to public charging can be handled by operating captive chargers at the depot/warehouse
Strong business case
for e-commerce
“last-mile” mobility
electrification!

3. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Problem Overview
3
Objective (What?): Design an intelligent charging management system using captive chargers for a fleet of “last-mile” EVs that reduces the fleet
charging cost while meeting the operational constraints!
Motivation (Why?)
▪ Charging management is non-trivial, even with captive chargers and reasonable energy demand estimates; which has a cost bearing on the OpEx!
– Supply-side: Charging capacity often lags the demand-side energy needs
▪ Charger deployment is capital intensive (tendency to under-provision chargers)
▪ Pressure to meet TCO parity with non-EVs through high utilization of vehicles and low idle-time of chargers
▪ Grid (in many cases) becomes the bottleneck
▪ capacity limitations of the grid connection line to the depot location could restrict charging
▪ non-availability of chargers due to scheduled black-outs or brown-outs of the grid in developing economies
– Demand-side: EV charge should be sufficient to handle planned routes and any associated uncertainty
Requirement: Intelligent Charging Management system:
✓ Day-ahead planning that assigns EVs to compatible chargers at specific points in time
✓ Real-time handling of any deviations from the planned assignment
Challenges:
▪ Limited flexibility in the planning process due to constraints on the available supply capacity and length of vehicle stay in the depot
▪ The scope of planning is not limited to a single trip, but spans over multiple trips in the day that may have knock-on effect from previous trips
(multi-period planning)
▪ Developing plans for fleet-level charging is a computationally difficult problem; which becomes even harder for a fleet of heterogeneous
vehicles (with different battery capacities) and charger types (AC/DC with different power ratings and connectors)
▪ Time-of-day effects in electricity pricing further increases planning complexity

4. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Solution Approach
4
Hybrid planning system that combines day-ahead planning (learning agent) with online replanning (heuristic)

5. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
System Model
5
Parameters
𝜃𝑡
𝑖 1: if charger 𝑗 is available at time 𝑡;
0: otherwise
𝐿𝑗 minimum charging rate of charger 𝑗 (A)
𝑈𝑗 maximum charging rate of charger 𝑗 (A)
𝑉𝑗 operating voltage of charger 𝑗
𝜓𝑡
𝑖 1: if EV 𝑖 is present in the depot at time 𝑡;
0: otherwise
𝜇𝑡
(𝑖,𝑗) 1: if EV 𝑖 is compatible with charger 𝑗;
0: otherwise
𝑒𝑡
𝑖 energy demand of EV 𝑖 at time 𝑡 (kWh)
H avg. charge consumption rate (kWh/unit time)
𝑄𝑖 total battery capacity of EV 𝑖 (kWh)
𝑞𝑖𝑛𝑖𝑡
𝑖 initial SoC of EV 𝑖 (kWh)
Decision Variables
𝛾𝑡
𝑖 1: if EV 𝑖 switched chargers between time (𝑡 − 1) & 𝑡;
0: otherwise
𝜆𝑖 1: if EV 𝑖 does not depart on time;
0: otherwise
𝛼𝑡
(𝑖,𝑗) 1: if EV 𝑖 is charging at charger 𝑗 at time 𝑡;
0: otherwise
𝑟 𝑘 𝑡
𝑖,𝑗
rate of charging (A): EV 𝑖; charger 𝑗; time t
𝑏𝑡
𝑖 SoC of EV 𝑖 at time 𝑡 (kWh)
𝑝𝑡 price of electricity at time 𝑡 ($/kWh)
𝑝𝑎𝑣𝑔 avg. price of electricity at time 𝑡 ($/kWh)
EV Set V |V| = 𝑖
Charger Set C
|C| = 𝑗
Charging Rate Set R
|R| = 𝑘

6. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Optimization Problem
6
ℳ = 𝑚𝑖𝑛 ( ෍
𝑡∈𝑇
෍
𝑗∈𝐶
෍
𝑖∈𝑉
𝑟𝑡
𝑖,𝑗
∗ 𝑝𝑡 + 𝒫 ) Penalty:
(i) not meeting the energy demand of vehicles
by their departure deadlines +
(ii) vehicles switching chargers across
consecutive charging sessions)
Cost of charging all vehicles in the fleet
[C1]: Ensure that every EV charges at most one charger at any given time
෍
𝑖 ∈ 𝑉
𝛼𝑡
𝑖,𝑗
≤ 1 ∀𝑗 ∈ 𝐶
Objective: Minimize the Charging Cost of the EV Fleet
Constraints

7. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Optimization Problem
7
[C3]: Ensure that, at any given time, the battery level of an EV is greater than the energy required to complete its assigned trip;
except when the trip has been delayed and penalty is incurred at 𝜆𝑖
= 1
𝑏𝑡
𝑖
+ M . 𝜆𝑖
≥ 𝑒𝑡
𝑖
∀𝑖 ∈ 𝑉, ∀𝑡 ∈ 𝑇
[C4]: Ensure that a vehicle cannot charge when it is not in the depot
෍
𝑗 ∈ 𝐶
[1 − 𝜓𝑡
𝑖
] . 𝛼𝑡
(𝑖,𝑗)
− 𝜆𝑖
≤ 1 ∀𝑖 ∈ 𝑉, ∀𝑡 ∈ 𝑇
[C5]: Ensure that a vehicle can charge only at a compatible charger
෍
𝑗 ∈ 𝐶
[1 − 𝜇𝑡
(𝑖,𝑗)
] . 𝛼𝑡
(𝑖,𝑗)
< 1 ∀𝑖 ∈ 𝑉, ∀𝑡 ∈ 𝑇
0 ≤ 𝑏𝑡
𝑖
≤ 𝑄𝑖
∀𝑖 ∈ 𝑉, ∀𝑡 ∈ 𝑇
[C2]: Ensure that the battery level of a vehicle is within its allowed limits at any given time (battery capacity constraint)

8. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Optimization Problem
8
[C8]: Set the SoC of each vehicle at the start of the planning horizon
𝑏0
𝑖
= 𝑞𝑖𝑛𝑖𝑡
𝑖
∀𝑖 ∈ 𝑉
[C9]: Calculate the vehicle shifts (# instances when a vehicle switched chargers in consecutive time steps)
𝛾𝑡
𝑖
= ෍
𝑗∈ 𝐶, 𝑚𝜖 𝐶 −{𝑗}
𝛼𝑡−1
(𝑖,𝑗)
. 𝛼𝑡
(𝑖,𝑗)
∀𝑖 ∈ 𝑃, ∀𝑗 ∈ 𝑉, 𝑖 ≠ 𝑗
[C7]: Ensure chargers are not utilized when they are down for maintenance
෍
𝑗∈ 𝐶
[1 − 𝜃𝑡
𝑖
] . 𝛼𝑡
(𝑖,𝑗)
≤ 1 ∀𝑖 ∈ 𝑉, ∀𝑡 ∈ 𝑇
[C6]: Ensure that the change in battery level from t to (t+1) is satisfied
𝑏𝑡+1
𝑖
= 𝑏𝑡
𝑖
+ 𝜓𝑡
𝑖
. 𝑟 𝑘 𝑡
𝑖,𝑗
− 𝐻 1 − 𝜓𝑡
𝑖
. 1 − 𝜆𝑡
𝑖
∀𝑖 ∈ 𝑉, ∀𝑗 ∈ 𝐶, ∀𝑘 ∈ 𝑅

9. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Learning Model for Day-ahead Planning
9
Planning decision := which vehicle 𝑖 -> which charger 𝑗 -> what charging rate 𝑟 -> which charging time slot ℎ ?
▪ Vehicle 𝑖 is not in the depot at 𝑡 [C4]
▪ Vehicle 𝑖 has the required SoC to complete all trips for the day [C3]
▪ Vehicle 𝑖 and charger 𝑗 have incompatible charging protocols [C5]
▪ The charging rate 𝑟 is not enough to reach the required SoC level in the stipulated time frame.
▪ The charging time slot ℎ is not within the of the current decision step 𝑡 and departure time.
▪ The maximum value of 𝑟 exceeds the charging rate limit of charger 𝑗.
▪ The charger 𝑗 is not available at charging time slot ℎ.
▪ There is an existing vehicle assignment of charger 𝑗 at time slot ℎ [C1]
Masking scheme for finding FEASIBLE (Vehicle -> Charger -> Charging Rate -> Charging Time Slot) quadruples

10. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Leaning Agent Representation
10
State Variables
𝑧𝑡
𝑖 (𝑒𝑡
𝑖
− 𝑏𝑡
𝑖
) | Energy (charge) needed for trip completion
𝑟𝑡
(𝑖,𝑗)
charging rate of vehicle 𝑖 at charger 𝑗 at time t
𝑣𝑠𝑡
𝑖 shifts in charger allocation for vehicle 𝑖 at time t
𝑑ℎ
𝑖 Urgency factor: time difference between the departure time of
vehicle 𝑖 and the charging time slot ℎ
𝑝ℎ Electricity price at charging time slot h
State 𝑺𝒕 Action 𝑨𝒕
assign: vehicle 𝑖 -> charger 𝑗 -> charging rate 𝑟 ->
charging time slot ℎ
Decision := from all feasible 𝑖, 𝑗, 𝑟, ℎ quadruples, which quadruple is the best choice?
Reward 𝑹𝒕
− 𝐴1 ∗ 𝑝ℎ ∗ 𝑟𝑡
(𝑖,𝑗)
𝐴1 = 0.06 -VE reward for the cost of the charging operation
− 𝐴2 ∗ 𝑣𝑠𝑡
𝑖 𝐴2 = 0.005 -VE reward for consecutive charger shifts of a vehicle
+ 𝐴3 ∗ 𝑒𝑡
𝑖
− 𝑏𝑡
𝑖 𝐴3 = 20
+VE reward for priotizing assignments that reduce the
gap between the required and the current SoC

11. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Training for Learning Approach
11
1) Initialize the neural network with weights 𝜙
2) Initialize batch size 𝛽, replay buffer 𝐵
3) :FOR episode = 1 :TO total-num-episodes :DO
i. Randomly choose data instance from training set
ii. Initialize (reset) environment and get initial states
iii. :FOR t < T :DO
a. priority = (𝑒𝑡
𝑖
− 𝑏𝑡
𝑖
) / 𝑑𝑡
𝑖
, ∀𝑖 ∈ 𝑉
b. veh_to_charge = sorted array of vehicles needing charge at t in order of priority
c. :WHILE len(veh_to_charge ) != 0 :DO
i. 𝑖 = pop (veh_to_charge)
ii. 𝑡𝑑𝑒𝑎𝑑𝑙𝑖𝑛𝑒 = time slot when 𝑖 has to depart for next trip
iii. req_soc = charge needed by 𝑖 for next trip
iv. Find feasible_quadruples = feasible combinations of (vehicle 𝑖, charger 𝑗, charging rate 𝑟, time slot ℎ)
v. :WHILE len(feasible_quadruples) >= 0 :DO
a. Calculate 𝑞𝑡 = 𝜙(𝑆𝑡) ∀ feasible (𝑖, 𝑗, 𝑟, ℎ) quadruples
b. Choose quadruple (𝑖, 𝑗, 𝑟, ℎ) with max 𝑞𝑡 with probability 𝜖 and random with probability (1 - 𝜖)
c. Execute/Apply (𝑖, 𝑗, 𝑟, ℎ) assignment in/to the environment and get reward 𝑅𝑡
d. Add [𝑆𝑡; 𝑅𝑡; 𝑞𝑡] for quadruple (𝑖, 𝑗, 𝑟, ℎ) to replay buffer 𝐵
vi. If req_soc is not met, then shift 𝑡𝑑𝑒𝑎𝑑𝑙𝑖𝑛𝑒 of 𝑖 by 1 time slot and calculate penality
iv. Delete oldest entries in B if size exceeds buffer capacity
v. Draw 𝛽 samples from B
vi. Update 𝜙 by minimizing MSE loss between 𝑞𝑡 and 𝑅𝑡

12. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Heuristic for Online Replanning
12
1) Initialize the environment delivery_env
2) Initialize offline_plan = LearningAgent (delivery_env)
3) Initialize online_plan = offline_plan
4) :FOR t < T :DO
i. veh_arrive = vehicles scheduled to arrive at time step (t+1)
ii. veh_to_charge = sorted array of vehicles needing charge at t in order of priority
iii. Initialize delay to store delays in each vehicle
iv. :FOR 𝑖 in veh_arrive :DO
a. delay[𝑖] = calculate delay in arrival of vehicle 𝑖
b. delivery_env = shift delivery_env of vehicle 𝑖 based on the delay
c. :IF delivery_env does not violate offline_plan :THEN
i. break
d. avail_chargers = available chargers at time t
e. Find feasible_triple = feasible combinations of (vehicle 𝑖, charger 𝑗, charging rate 𝑟)
f. :FOR j 𝜖 avail_chargers :DO
a. :FOR triple 𝜖 feasible_triple[𝑗] :DO
i. priority1 = (𝑒𝑡
𝑖
− 𝑏𝑡
𝑖
) / 𝑑ℎ
𝑖
ii. priority2 = -(𝑝ℎ ∗ 𝑟𝑡
𝑖,𝑗
∗ 𝑉𝑗)
iii. priority3 = 𝑝𝑎𝑣𝑔 - 𝑝𝑡
iv. priority4 = 𝑟𝑡
𝑖,𝑗
v. priority = C1* priority1 + C2* priority2 + C3* priority3 + C4* priority4
g. Choose triple with highest priority and perform the action
h. :FOR 𝑖 𝜖 veh_to_charge :DO
i. 𝑡𝑑𝑒𝑎𝑑𝑙𝑖𝑛𝑒 = time slot that 𝑖 has to depart for next trip
ii. :IF 𝑡𝑑𝑒𝑎𝑑𝑙𝑖𝑛𝑒 == t AND 𝑏𝑡
𝑖
< 𝑒𝑡
𝑖
∶ THEN
i. Shift delivery schedule of 𝑖 time slot by 1 and
calculate penalty

13. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
Avg. 8.5%
Avg. 10.5%
Avg. 14%
Avg. 14%
1. In terms of solution accuracy, LA outperforms GH in general, but becomes more cost-efficient with increasing scale
2. In terms of computation speed, GH is faster than RL in general, but this gap closes with increasing scale
LA has a better optimality gap than GH; wherein the average gap is 21.36% in the case of LA,
while it is 26.26% for GH
Evaluation - I

14. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
1. Uncoordinated (FCFS) charging results in the highest
fleet charging cost, irrespective of the magnitude of
delay in arrival at the depot (or randomness).
2. Online replanning is 7-20% better than uncoordinated
and day-ahead planning approaches.
Evaluation - II

15. | Copyright © 2021 Tata Consultancy Services Limited
TCS Internal
▪ We model the electric vehicle charging planning problem with constraints on energy demand; fulfilment
deadlines; charging protocol compatibility; electricity prices; and formulate the optimization problem to
minimize the operational cost of the fleet.
▪ We propose a hybrid planning system for charging of EV fleets by combining day-ahead planning with online
replanning.
▪ For day-ahead planning, we design a value-based learning algorithm where we define the (state, action) space, and
engineer the reward signal for the agent to find cost-effective charging plans.
▪ For online replanning, we design a heuristic that tracks the feasibility of the offline plan, and makes dynamic
adjustments by prioritizing vehicles using a greedy approach.
▪ We evaluate the planning performance using datasets for a range of EV delivery routes and time requirements.
▪ In the offline case, our study shows that the solution quality of the proposed learning agent is 8.5-14% better than the
greedy heuristic baseline, and is within 22% of the optimal obtained through ILP
▪ In the online case, the proposed approach yields 7-20% better results than uncoordinated and day-ahead planning
baselines.
▪ In general, the quality of the proposed solution improves with increasing scale and size of the problem.
15
Final Remarks