6. #MMAMD
Multi-Source Inventory (MSI)
MSI is designed to enable stock management in multiple locations so that
merchants can properly reflect their physical warehouses in Magento
system without external extensions or customization
The following terms are important as you work with MSI:
• Sources
• Stocks
• Aggregate Saleable Quantity
• Reservations
16. #MMAMD
Separate Index Per Stock
SKU QTY
Product A 300
SKU QTY
Product A 300
Inventory_Stock_1
Inventory_Stock_2
17. #MMAMD
Multi Dimensional Indexation
• When Source is Disabled
• When Source is added/removed from Stock
• When Source item changes quantity (Order processed, ERP sent update
etc)
Indexer should be triggered in 3 cases:
18. #MMAMD
Reservation Mechanism
Reservation – allows to reserve some product quantity in stock when the
order is placed
Reservations are append only operations and help us to prevent blocking
operations and race conditions at the time of checkout.
20. #MMAMD
Order Placement – Initial State
NYC Warehouse
SKU-1: 200 qty Atlantic Stock
SKU-1: 300 qty
BOS Warehouse
SKU-1: 100 qty
Raw data Index Reservations
No records
Available Qty: 300 qty (data from index, empty reservations)
22. #MMAMD
Order Placement – Step 2
NYC Warehouse
SKU-1: 200 qty Atlantic Stock
SKU-1: 300 qty
BOS Warehouse
SKU-1: 100 qty
Raw data Index Reservations
SKU-1: -5
Available Qty: 295 qty (data from index 300 qty, apply reservations -5)
Data is NOT changed Reservation has been added
23. #MMAMD
Can I Buy 300 Items of SKU?
Atlantic Stock
SKU-1: 300 qty
300
Pacific Stock
SKU-1: 300 qty
400
-5
-11
Website A
Website B
< 300 => NO
> 300 => YES
25. #MMAMD
Order Placement – Step 4
NYC Warehouse
SKU-1: 200 qty Atlantic Stock
SKU-1: 300 qty
BOS Warehouse
SKU-1: 100 qty
Raw data Index Reservations
SKU-1: -5
Available Qty: 298 qty (data from index 300 qty, apply reservations -5 +3)
Data is NOT changed Reservation has been added
SKU-1: +3
27. #MMAMD
Order Placement – Step 6
NYC Warehouse
SKU-1: 198 qty Atlantic Stock
SKU-1: 298 qty
BOS Warehouse
SKU-1: 100 qty
Raw data Index Reservations
SKU-1: -5
Available Qty: 298 qty (data from index 298 qty, apply reservations -5+3+2=0)
Data is CHANGED Reservation has been added
SKU-1: +3
SKU-1: +2
28. #MMAMD
Order Placement – Step 7
Action: Reservation cleaning
Looping through these reservations we could find reservations which in sum gives 0 (Zero) and
remove them.
29. #MMAMD
Order Placement – Step 8
NYC Warehouse
SKU-1: 198 qty Atlantic Stock
SKU-1: 298 qty
BOS Warehouse
SKU-1: 100 qty
Raw data Index Reservations
No Records
Available Qty: 298 qty (data from index, empty reservations)
Data is NOT changed Reservation has been removed
Good Morning
Hope you guys doing good !!
Thank you everyone to present here in weekend morning and give me an opportunity to contribute towards community.
Let me introduce myself…. Myself Bhavesh Surani and working as a Head of Ecommerce & Mobile in Brainvire Infotech Pvt. Ltd.
Twitter Handle: @bhaveshsurani
Agenda
Single warehouse Inventory system means when any new order come then you don’t need to much worry about to how process an order in terms of selection of warehouse or where to package those items.
Any orders request or Any SKU request which comes through the magento to your system which do fulfilled by single warehouse.
This is the really 1 source address and 1 shipment address. Magento 1.x, Magento 2.1.x, 2.2.x all system consider as a Single Source Inventory System.
Here customer uses some backend ERP system to manage multiple sources inventory and pass global stock to Magento.
The following terms are important as you work with Inventory Management:
Sources represent physical locations that store and ship available products. These locations can include warehouses, brick-and-mortar stores, distribution centers, and drop shippers. (Any location can be designated as a source for virtual products.)
Stocks map a sales channel (currently limited to websites) to source locations and on-hand inventory. A stock can map to multiple sales channels, but a sales channel can be assigned to only one stock.
Aggregate Salable Quantity is the total virtual inventory that can be sold through a sales channel. The amount is calculated across all sources assigned to a stock.
Reservations track deductions from the saleable quantity as customers add products to carts and complete checkout. When an order ships, the reservation clears and deducts the shipped amounts from specific source inventory quantities.
Basic ideas for multi sourcing is to split inventory between sources using 1 magento installation. So we can have several warehouse which located in different location/different country/different states.
When customer making an order, it should be the actual the same and wont affect user experience. All complexity will do taking care from magento.
Some products of order located in one place and another products of order located in another place…Here how to optimize logistic, how to optimize delivery time and cost of delivery, Its responsibility of magento. To support big change, magento have consider big change in architecture.
In this diagram, a bicycle merchant has inventory for a touring bike in a warehouse and two stores.
He has 2 stocks with configured website sales channels and sources.
When a customer shops through the UK website, Magento aggregates bike inventory from the warehouse and flagship store sources for a saleable quantity of 95. The touring bike can be shipped from either the warehouse or the flagship store, but not the US store.
Amazon Marketplace web store same stock draws from the same aggregate stock as the UK web store.
Here you can see diagram of entity relationship where you can see the source, actual represents physical locations and source items. So source items actually shows how many of particular products stored on particular physical stores/source/warehouse or locations.
Now you have Inventory_Source_Stock_Links, which help you to combine and making an Virtual Aggregation. Virtual Aggregation actually happen use indexation time.
You can consider stock index…. Inventory_stick_Id stock index entity which created while stock indexation. When we create index entity which mark them with sales channel which we have.
Ideally default out of the box implementation we implement sales channel as a country, stores, customer group.
To support such big changes with Architecture, Magento Introduce to Command/Query Responsibility Segregation Concept to achieve MSI. This very known concept and many big/enterprise system applies this concept.
Basic of this concept, it will separate read and write operation. Why we make this separation, we make this separation to provide ability of independent scaling of read and write operation. In depend scaling read and write operation provide more boost for performance.
For both reading and writing. For Example: if we are talking about some particular operation like data changing… to change some data we actual use some pretty sophisticated module and that module use some complex validation to make sure all data integrity and data consistency actually in valid state. All of that, we need to do write operation and if you understand, all of checks or validation constrains take sometime.
Actually When you make read operation, just to render data on frontend like list of products, product page… actually no need to do any kind of validation…all we need to do just grab data from data storage and put it to the template. Actually its could be even pure php array and no need any kind of validation happen. It would be pretty performance friendly way. That’s why we have separation between read and write environment
If you go further steps, think more about this architecture, we come up with idea that… we have actually have different data storage for reading and writing. For example, ,we have data storage optimization for writing with mysql database and readying it will be elastic search.. Which is really fast for retrieving data and rendering very fast and render it on frontend.
This concept is not new and if u have magento 1 experience, you aware about EAV architecture, we are storing attribute value in dedicated table based on attribute type.. Varchar, int and text. If we want to get some projections of project, we need to make lots of joins with all data combine and its not really performance friendly way..
If we need to Make it render on frontend. What actually representation need to do
if we change data for the attribute. We need to do in the write scenario, but not need to do anything with read scenario.
That’s why we have indexation process and we create flat table which is performance friendly. It contains all products attributes and we just need to query only single table to retrieve data.
Here you see these segregations, so write interface and read interface. In read interface, for example if you got to frontend and just need to render data regular product list… here just use read interface.
All we need to do is just to get this performance friendly read interfaces. We will use them for example to describe particular amount of products which available for sales with some particular scope.
We see here stock_items and its describes the items IN Stock or Out of stock. Its pretty performance friendly and if you need to make any additional aggregation or pre-calculation as a run time.
Another scenario with write request, In write request in our scenario will be the stock deduction during the checkout process. Because When we make the checkout process we need to make stock deduction from our physical sources. Because some products are ordered, so we need to make change with global stock.
This purpose we have the write interfaces and also used during the syncs of inventory with third party ERP or product management system.
We ask segregation interfaces depend on scenario where that going to use weather its write interface or read interface.
Multi Source Inventory Source System is very complex feature with magento. First of all before placing an order, we need to make sure that we have enough inventory in all sources to fulfil particular quotes.
Even before creating an order, you can query each sources to understand that, how many items each sources have. If you see that, there are enough products on all sources and you can have potential to find combination of the sources which you can full fill quote, you are able to place an order.
When you place an order, now complication happen during the shipping process and you need to calculate, how much it cost to sent all the items from all the sources to shipping address. In order to calculate, you need to know which origin address you send them, know the combine all products from all those orders items together somehow and kind of packages for different sources and you potential will have much more long package to send on shipping address. It will be at least some package some time have multiple sources to fulfil your all the orders items with multiple packages.
If you are going to send and now need to combine all the orders items gather combine in one packages from different sources.
Lets start with indexation. Here is example which shows complexity which can happen with your system when you have multiple sources.
Multiple sources with multiple websites. Here is showing many to many relationship.
One website can have multiple sources to stores inventory but beside one warehouse can assigned to different website to fulfil order.
In this use cases, NYC & BOS warehouse are assigned to Atlantic stock which is assigned to Website A, where LA and SF warehouse assigned to Pacific stock which is assigned to Website B.
Also you can see SF warehouse assigned to both Pacific stock and Bay Area stock as well as assigned to website B and Website C at a same time. here this is kind of complexity which need to deal and because of that, it may cause lots of performance issues with your system if you would like to make real-time stock check query when you place an order and do check that, will able to accept that order or declined order.
Now order to reduce this complexity related to this stock, stock now allow you to aggregate all qty information from all warehouse assign to particular website. When you actually take an order, stock will be one which allow you to give an answer, do you have stock for particular item to accept orders. As you see, stock is aggregation of certain SKU from all the warehouse assign to stock.
When we talk about indexation, we have indexer which creates flat structures results information about stock for all the warehouse which assigned to particular stocks. So really separated index for the stock confirmation in different tables per stock. If you have multiple stocks in your system, then it will create multiple index table which allow you to query very fast about multiple products which you have there which do you would like to sell.
As separation of stocks for different table allow you to actually reduce complexity even more. Now you don’t have multiplication of information for different stocks in one table and you query become much more efficient.
Indexation process get triggered on 3 cases:
When source is disabled, you need to kind of reduce the amount of qty from particular stocks where the sources was assigned
Same situation when source was added/removed from particular stocks, then you need to add qty to warehouse and from stocks or remove qty from warehouse then need to reduce from stocks
Also when item got sold, then also need to reduce qty information from stocks
So we built an all the system which allow to have multiple index table stock exists very new in magento, before magento allow single dimension indexation system… now we have multidimensional indexation system… besides multiple triggers which allows you to update index table.
Now would like to talk about reservations API… Which allows you to reserve certain amount of products in certain stocks. Why we need that?
As we want to avoid overwrite index table real time… Don't want to write in index when you place an order or when you fulfil order. Because this separation sounds to be complex and you need to insure all peace of system get updated when you do that as well as besides complexity its not performance friendly write same table which currently been used to read information when other order placed.
It’s the separation of read and write sources of data its called CQRS… Its followed strictly CQRS as directly related to checkout and checkout functionality is very Stress under performance when you an lots of formula coming in system.
Because of that, we have decided to avoid write in index table real-time. When only time index table updated when indexer run and reduce or add certain information from the warehouse inventory.
If you don’t try to write to the table, how its actually place an order? In warehouse, we create an reservation, reservation is append only API its allow to just write some new records to reservation table. Records should contains information which allow to understand ladger with how many products left in system. In our case, reservation contains SKU of product and qty of SKU which was reserve in particular stocks.
Reservation per stock, certain stocks allow to create new records which do add/reduce certain amount of products its based on situation.
Lets see this example to explain more… We have Atlantic Stock with couple of warehouse. So total number of Atlantic Stocks is begging of scenario with 300 which 200 with NY warehouse and 100 with Boston warehouse.
Initial State of application before processing an order and showing No Records with Reservation table.
Same Warehouse information, same stock information and reversion table which contain all reservations for certain products and certain warehouse. Total available number of products is 300
Now we have customers placed an 5 products orders on website A which come to atlantics stock.
What happen here.. We don’t touch any data here in warehouse data, stock data or website data. We create just create record in reservation table with -5 products appended to reservation table. So basically deducted 5 products from the system.
Now total available qty of products 295. how this happen, we can take all records from reservation for particular stock and for particular SKU, just sum up how much sum up qty in reservation table.
Just sum up Qty from reservataion and deduct qty from stock qty.
If you would like to accept an order, just would like to give an answers that, if you would like to accept an order qty 300, wont able to accept on Atlantic stock where in pacific stock, yes will able to accept ti.
So couple of fraction related to reservation when admin complete order and indexer run for indexation..
First of all we will reduce qty of the products from warehouse which was used to ship the products. NYC shows -2 products and showing 198 products. BOS warehouse wont updated and showing 100 qty. Total SKU information get deducted after indexation run and update stock information. It will get updated to 298 products.
Now we talk about reservation depend only mechanism.. So here showing reservation with +2 amount of qty.. Because we cant actually go back and update reservation record. Its only things we can do is to create similar reservation with opposite amount of qty. before we had a -5, after indexation run. New record created with +2 as well as +3 which shows when you have cancelled items from orders.
So total when you calculate, total number available qty in your system with sum up all stock information as well as all reservation…
It will be 298 -5+3+2… total qty is 298.. So this is the case where just update records in reservation tables and you actually optimize the performance… because you don’t need to update records which currently been used read information from the system.
You separate read and write concerns in very well manner.
Questions is now how do you clean up reservation table… Magento having inventory_cleanup_reservations Cron jobs which process which keep flow information & check SKU for the stocks which have SUM up equale 0 and remove those records. Eventually we have index which keep this table small enough and doesn’t grow big.
After this clean up happen, you can see no records in reservation table.
