This document discusses using LINQ for data manipulation when performing machine learning tasks. It describes creating a custom lightweight data table using dictionaries to store row and field data in order to easily query the data using LINQ. Sample LINQ queries are provided to select data based on conditions, calculate averages and sums, group data, and get distinct values and counts. LINQPad is also mentioned as a useful tool for testing LINQ and SQL queries.

1. Using LINQ
ML Learning Notes – FH2020

2. Why Do I need to use LINQ
 During my attempt to learn Machine Learning, I figured that I need to be able to easily select
data based on certain conditions and just focus on the ML algorithms rather than the data
manipulation itself.
 When I searched about SQL like query in C#, I noticed LINQ
 The syntax and concept is easy to understand (having SQL query knowledge definitely helps)
 It is available using System.Linq

3. Data Design
 I want to create my own light weight data table (instead of using DataTable class from
System.Data)
 My Data Table (called MLDataTable) is basically Dictionary of Row Data.
 RowData class is a dictionary of FieldDatas
 FieldData contains 3 members: Name, Value and Masked

4. Class: MLDataTable

5. Class: RowData

6. Struct: FieldData

7. Populate Data
 Data loaded from fast CSV reader
 Each Field Data is added into the RowData
 Then each Row data is added into the MLDataTable

8. Notes
 I can use LINQ to query dictionary of custom class
 In a table – like implementation I can use Dictionary in Dictionary to accommodate dynamic
structure from training CSV (number of columns)
 I like the Dictionary Key pair, where I can use the column name as Key string and pass it
dynamically when creating the query
 Can use Aggregation such as sum and average, Group By

9. Sample Query #1
 Process numerical data pair an Attribute Field and a Decision Field where Attribute value is not
disabled (Masked=1)
 Then get data where attribute value is < and >= Treshold value (to be used to calculate RSS)

10. Sample Query #2
 Get Average Value from a group of data
 Get calculated RSS Value = SUM(SQR(Value – Avg))

11. Sample Query #3
 Get Distinct values and counts of a particular field that is not disabled

12. Some interesting stuff
 LINQPad is useful not only to test the LINQ query, but also can be used to test SQL query
 It has small database that can be used for testing purpose.
 Download from: https://www.linqpad.net/Download.aspx