2. Content
I. About Alasql
II. SQL data language
III. JavaScript API
IV. Persistence and external databases
V. JSON, TXT, CSV, TSV, and Excel data processing
VI. JavaScript frameworks: Angular.js, d3.js
VII. Command-line utilities: Alacon, Alaserver
4. Alasql
• JavaScript SQL database library designed for:
• Client-side SQL database with persistence
• Fast data processing for BI and ERP applications
• JS data manipulation and advanced filtering, grouping and joining
• Easy ETL (extract, transfer, and loading) data in CSV and XLSX
formats
• Works in browser, Node.js, mobile applications
5. Alasql in Internet
• GitHub
• http://github.com/agershun/alasql
• Official site
• http://alasql.org
6. Installation and Usage
• Installation:
• In the browser
• Copy file (production)
• dist/alasql.min.js
• Or (debug)
• dist/alasql.js
• dist/alasql.js.map
• In Node.js
• npm install alasql
• Usage:
• In the browser
• <script src=‘alasql.js’></script>
• AMD module
• require([‘alasql’],
function(alasql){ /* body */ });
• In Node.js
• var alasql = require(‘alasql’);
7. Quick Start
// Advanced JavaScript data processing (sync. with parameters)
var data = [{a:1,b:1,c:1},{a:1,b:2,c:1},{a:1,b:3,c:1}, {a:2,b:1,c:1}];
var res = alasql('SELECT a, COUNT(*) AS b FROM ? GROUP BY a',[data]);
console.log(res);
// Work with IndexedDB database with SQL (async, multiple SQL statements)
alasql(’ATTACH INDEXEDDB DATABASE MyBase;
USE MyBase;
SELECT City.*
FROM City
JOIN Country USING CountryCode
WHERE Country.Continent = ”Asia”’, [], function (res) {
console.log(res.pop());
});
9. Alasql SQL statements
• Data query
• SELECT
• Data manipulation
• INSERT
• UPDATE
• DELETE
• Data definition
• CREATE TABLE
• ALTER TABLE
• DROP TABLE
• Database
• USE DATABASE
• CREATE DATABASE
• DROP DATABASE
• External database
• ATTACH DATABASE
• DETACH DATABASE
• Transactions
• BEGIN
• COMMIT
• ROLLBACK
• Show
• SHOW DATABASES
• SHOW TABLES
• SHOW CREATE TABLE
• Program
• SET
• SOURCE
• Debug
• ASSERT
• Information
• HELP
10. Statements
• Single statement
• Return value
• Query result
• [{a:1},{a:2}]
• Number of rows processed
• 362
• Number of database object
processed (e.g. tables
dropped)
• 1 / 0
• Multiple statements
• Separated by semicolon
• “CREATE DATABASE test;
USE test1”
• Return value
• Array of return values of
each of statements
• [1,0, [{a:1},{a:2}]]
11. Case-Sensitive
• Case insensitive
• SQL Keywords (SELECT)
• Standard functions (LEN)
• Aggregators (SUM)
• Engines (INDEXEDDB)
• FROM-functions (TXT)
• INTO-functions (XLSX)
• Same:
• SELECT * FROM city
• select * from city
• Case sensitive
• Database names
• Table names
• Columns
• User-defined functions
• JSON properties and
functions
• JavaScript classes
• Different:
• SELECT * FROM city
• SELECT * FROM City
• SELECT * FROM CITY
12. SELECT
• SELECT
• TOP / LIMIT FETCH
• DISTINCT
• INTO
• FROM
• JOIN ON / USING
• GROUP BY
• HAVING
• WHERE
• ORDER BY
• UNION / INTERSECT /
EXCEPT
• Value modifiers
• VALUE, COLUMN, ROW,
MATRIX …
• Columns
• City.Name, City.*, Population
AS p
• Operators
• w*h+20
• Aggregators
• SUM(), COUNT(),..
• Function
• LCASE(), LEN(), ..
13. Return Value Modifier
• SELECT By default
• returns array of objects
• [{a:1,b:10},{a:2,b:20}]
• SELECT VALUE
• returns first value of first row
• 1
• SELECT COLUMN
• returns first column from all rows
• [1,2]
• SELECT ROW
• returns values of all columns of first
row
• [1,10]
• SELECT MATRIX
• returns array of arrays
• [[1,10],[2,20]]
• Return number of all lines in
README.md
• SELECT VALUE COUNT(*) FROM
TXT(‘README.md’)
• Return array of arrays:
• SELECT MATRIX * FROM one
14. SELECT columns
• Columns
• SELECT size
• SELECT City.Name,
City.Population
• Expressions
• SELECT LCASE(City), 2+2
• Aggregators
• SELECT COUNT(*),
SUM(Population)
• Alias
• SELECT City+” “+Country
AS LongName
• All columns from table
• SELECT *, City.*
• Columns of arrays
• SELECT [0],[1]
• Column names with
spaces, etc
• [My Column]
• `My Column`
15. Operators
• Number
• +,-,*,/
• String
• +
• Logic
• AND, OR, NOT
• =, !=, >, >=, <, <=
• Complex
• v BETWEEN a AND b
• v NOT BETWEEN a AND b
• v IN (10,20,30)
• v NOT IN (SELECT *
FROM Ages)
• v >= ANY (20,30,40)
16. Aggregators
• SQL Standard
• SUM()
• AVG()
• COUNT()
• MAX()
• MIN()
• FIRST()
• LAST()
• Non-standard
• AGGR()
• COUNT
• COUNT(one)
• COUNT(*)
• AGGR – operations on
aggregated values
• SELECT SUM(a) AS sm,
COUNT(*) AS cnt,
AGGR(sm/cnt) AS avg1,
AVG(a) AS avg2
FROM data
• Here: avg1 = avg2
18. TOP / LIMIT FETCH
// Select top 10 records
SELECT TOP 10 * FROM Cities ORDER BY Name
// Select 20 records starting from record number 5
SELECT * FROM Cities ORDER BY Name LIMIT 20 FETCH 5
20. INTO
• Into table
• SELECT * INTO City
FROM Capital WHERE
• SELECT * INTO
• Into external file (into-
functions)
• SELECT * INTO
CSV(‘city.csv’) FROM City
• Into stdout (for Node.js)
• SELECT * INTO TXT()
FROM City
• Into-functions
• TXT()
• JSON()
• CSV()
• TSV() / TAB()
• XLSX()
21. FROM
• From table
• SELECT * FROM albums
• SELECT * FROM mydb.test
• From parameter
• alasql(‘SELECT * FROM
?’,[singers]);
• From file (from function)
• SELECT * FROM
XLSX(“medals.xlsx”)
• From stdin (for Node.js)
• SELECT * FROM TXT()
• FROM table alias
• SELECT * FROM ? City
• SELECT * FROM album AS a
• From SELECT
• SELECT * FROM
(SELECT * FROM
(SELECT * FROM City))
• From functions
• TXT()
• JSON()
• CSV()
• TSV() / TAB()
• XLSX() / XLS()
22. From Parameter
• Array of objects
• alasql(‘SELECT
city.population FROM ? AS
city’,[city]);
• Array of arrays
• alasql(‘SELECT [0]+[1]*[2]
FROM ?’, [data]);
• Object
• alasql(“SELECT [1] FROM
? WHERE [0] =
‘one’”,[{one:1,two:2}])
• String
• alasql(“SELECT LEN([0])
FROM ?”,[“Multi n line n text”])
• Parameter data type
conversion
• String => array of lines
• SELECT * FROM ? WHERE
LEN([0]) > 10
• “abcncde” =>
[[“abc”],[“cde”]]
• Objects => array of pairs
key-value
• {a:1,b:2} => [[“a”,1],[“b”,2]]
23. JOIN
• Joins
• [INNER] JOIN
• LEFT JOIN
• RIGHT JOIN
• [FULL] OUTER JOIN
• ANTI JOIN
• SEMI JOIN
• CROSS JOIN
• NATURAL JOIN
• USING
• SELECT city.*, country.*
FROM city
JOIN country
USING countryid
• ON
• SELECT city.*, country.*
FROM city
JOIN country
ON city.countryid =
country.countryid
24. WHERE
• Expression
• SELECT * FROM City
WHERE Population >
1000000
• EXIST() / NOT EXIST()
• SELECT * FROM City
WHERE EXIST(SELECT
* FROM Capital WHERE
City.Name =
Capital.Name)
25. GROUP BY
• Grouping
• SELECT * FROM City
GROUP BY Contient,
Country
• Grouping functions
• CUBE()
• ROLLUP()
• GROUPING SETS()
• SELECT * FROM City
GROUP BY
ROLLUP(Continent,
Country)
27. ORDER BY
• Ascending
• SELECT * FROM City
ORDER BY Population
• SELECT * FROM City
ORDER BY Population
ASC
• Descending
• SELECT * FROM City
ORDER BY Name
DESC
28. UNION / INTERSECT / EXCEPT
• SELECT 10
UNION ALL
SELECT 20
• UNION
• UNION ALL
• INTERSECT
• EXCEPT / MINUS
29. INSERT values
• VALUES
• INSERT INTO city (name, population) VALUES
(“Moscow”,11500000), (“Kyiv”,5000000)
• INSERT INTO city VALUES (“Paris”,3500000)
• INSERT INTO city VALUES {name:”Berlin”, population:4000000}
• DEFAULT VALUES
• INSERT INTO city DEFAULT VALUES
• SELECT (= SELECT INTO)
• INSERT INTO city SELECT capital AS name FROM country
GROUP BY capital;
32. CREATE TABLE
DROP TABLE
• CREATE TABLE star (
one INT DEFAULT 100,
two STRING,
three BOOL PRIMARY KEY
);
• DROP TABLE star;
33. ALTER TABLE
• ADD COLUMN
• ALTER TABLE City ADD COLUMN Continent STRING
• RENAME COLUMN
• ALTER TABLE City RENAME COLUMN Continent TO WorldPart
• DROP COLUMN
• ALTER TABLE City DROP COLUMN Continent
• RENAME TO
• ALTER TABLE City RENAME TO Capital
34. CREATE DATABASE /
DROP DATABASE / USE DATABASE
• Create database
• CREATE DATABASE mydb
• Select default database
• USE DATABASE mydb
• USE mydb
• Drop database
• DROP DATABASE mydb
35. Transaction
• Begin
• BEGIN
• Commit
• COMMIT
• Rollback
• ROLLBACK
• In version 0.0.35 Alasql supports transactions only for
Local Storage and DOM-storage database. Full support
for other databases will be available in future versions
36. SHOW
• SHOW DATABASES – list of all databases in memory
• SHOW DATABASES LIKE ‘A%’
• SHOW TABLES – list of tables in database
• SHOW TABLES FROM mydb
• SHOW CREATE TABLE table – show CREATE TABLE
statement from the table
• SHOW CREATE TABLE City
37. SET, SOURCE, ASSERT, HELP
• SET - now used only for one option:
• SET AUTOCOMMIT ON / OFF
• SOURCE “file.sql” – read and execute all SQL statements from
file
• SOURCE ‘world.sql’
• ASSERT value – throws error if result of last operation is not
equal to value (Alasql uses equalDeep() function for
comparision)
• ASSERT 1
• ASSERT “Wrong Value”, [{a:1,b:”Odessa”}]
• HELP
• Show list of available commands
39. SQL and JavaScript:
Better Together!
SQL way
alasql(‘CREATE DATABASE test01’);
alasql(‘USE test01’);
alasql(‘CREATE TABLE one (a INT)’);
alasql(‘INSERT INTO one VALUES (10)’):
var res = alasql(‘SELECT * FROM one’);
JavaScript way
data = [{a:1}, {a:2}, {a:3}];
alasql(‘SELECT * FROM ? WHERE a >=
?’, [data, 2]);
or
var db = new alasql.Database();
db.exec(“select * from one”, function(data)
{
console.log(data.length);
});
40. alasql - main library object and function
• alasql(sql,params,callback) – execute sql
• alasql.exec(sql,params,callback) – execute sql
• alasql.parse(sql) – parse to AST (abstract syntax tree)
• ast.compile(databaseid) – compile statement and cache it
in database cache
• alasql.exec(sql) – execute statement
• alasql.use(databaseid) – use database
• alasql.pretty(sql) – pretty SQL output in HTML and TXT
• alasql.options - options
41. alasql() - main function
• alasql(sql,[params],[callback])
• sql – one or some SQL-statements separated by ‘;’
• If one statement – alasql() returns one value
• USE test12 => 1
• SELECT * FROM one => [{a:1}, {a:2}]
• If some statements – alasql() returns array of values, one for each
statement
• USE test12; SELECT * FROM one => [1,[{a:1}, {a:2}]]
• params – an array of parameters of SQL statement
• You can use ? in SQL statement
• alasql(‘SELECT a FROM ? WHERE b = ?’,[[{a:1,b:1}, {a:2,b:2}],2])
• callback – a callback function
• Without callback alasql() runs synchroniously
• With callback alasql() runs asynchroniously with callbacks
42. alasql(): sync and async
• Sync version
• var result = alasql(sql,
params)
• Async version
• alasql(sql, params,
function(result) {
// do something
//with result
});
• It is impossible to use
sync version with
async operations like:
• IndexedDB functions
• INTO- and FROM-
functions
44. Alasql options
• alasql.options
• alasql.options.valueof (true/false) – convert all values with
.valueOf() function before comparing
• alasql.options.angularjs (true/false) – remove $$hashKey from
result arrays if angular.js library loaded
45. How Alasql stores data?
• alasql.databases – list of all current databases in memory
• alasql.engines – list of all alasql available engines (like
localStorage, IndexedDB)
46. Database class
• var db = new alasql.Database(‘mydb’)
• db.databaseid – database name
• db.tables – list of tables
• db.engineid – engine (Local Storage, IndexedDB, etc.)
• db.exec(sql) – execute sql in mydb database
48. User-defined functions
and compiled statements
Custom functions:
alasql.fn.myfn = function(a,b) {
return a*b+1;
}
alasql(‘SELECT myfn(a,b) FROM
one’);
Compiled statements:
var ins = alasql.compile(‘INSERT
INTO one VALUES (?,?)’);
ins(1,10);
ins(2,20);
Compiled functions:
var bigSum =
alasql.compile(‘SELECT SUM(a)
FROM one WHERE a>3’, ‘value’);
var res = bigSum([10]) + 10;
49. JavaScript classes as SQL data types
• alasql.fn.Date = Date;
• alasql(‘CREATE order (
orderno INT,
orderdate Date
)’);
• Classes are case-sensitive
50. NEW (like JavaScript ‘new’ operator)
• Register class as alasql type
• alasql.fn.Date = Date;
• Use NEW
• alasql(‘SELECT NEW Date(yr,mn-1,dy) FROM orderdates’);
51. Property
• Property operator ->
• INSERT INTO one VALUES @{a:5, b:{c:[4,5]}}
• SELECT * FROM one WHERE a->b->0 = 4
• Expression
• SELECT * FROM one WHERE a->(LCASE(“B”))->(1-1) = 4
52. Call JavaScript object function
• Arrow function ->
• object -> function(parameters)
• Select lengths of all lines from text file
• alasql(‘SELECT [0]->length FROM TXT(“mytext.txt”)’
• alasql(‘SELECT LEN([0]) FROM TXT(“mytext.txt”)’
53. JavaScript object properties
• Arrow function -> property
• var data = [{a:{b:1,c:1}, {a:{b:2}}}]
• alasql(‘SELECT a->b FROM ?’,[data]);
• Array members
• SELECT a->(0) FROM data
• Calculated property names
• SELECT a->(“mon”+moid), b->(2+2) FROM data
54. Object Properties & Functions
• Object property
• a -> b
• a -> b -> c
• Array member
• a -> 1
• a -> 1 -> 2
• Calculated property
name
• a -> (1+2)
• a -> (“text” + “ “ + ”more”)
• Functions
• myTime -> getFullYear()
• s -> substr(1,2)
• JavaScript string
functions
• “ABCDE”->length
• SELECT s->length
FROM mytext
55. JSON objects
• @ prefixes (like Objective-C NSObjects)
• @1
• @”string”
• @{a:1,b:2} or {a:1,b:2}
• @[1,2,3] – conflct with column names with spaces [My Column]
• Three equal operators
• a = b like == in JavaScript
• a == b compare a.valueOf() and b.valueOf() – for dates
• a === b uses equalDeep() – for JSON objects
56. JSON with expressions
• CREATE TABLE one;
• INSERT INTO one VALUES @{b:1}, @{b:2}
• SELECT @{a:@[2014,(2014+1),(2014+b)]} FROM one
• [{a:[2014,2015,2015]}, {a:[2014,2015,2016]}]
57. CREATE TABLE AND INSERT JSON
VALUES
• JSON table
• CREATE TABLE one;
• INSERT INTO one VALUES @{a:1}, @{b:2}, @{a:1,b:2}, @1,
@”String”
• JSON object
• CREATE TABLE two (a JSON);
• INSERT INTO one VALUES (1), (‘two’), (@{b:’three’}),
@[‘F’,’O’,’U’,’R’]
58. SELECT JSON
• SELECT * FROM one
• [{a:1}, {b:2}, {a:1,b:2}, 1, ”String”]
• SELECT a FROM one
• [{a:1}, {a:undefined}, {a:1}, {a:undefined},{a:undefined}]
• SELECT * FROM one WHERE a=1
• [{a:1},{a:1,b:2}]
59. Deep equal (==, !==)
• SELECT @{a:1} == @{a:1}
• True
• SELECT * FROM one WHERE a=1
• INSERT INTO one VALUES {a:[5,6]}
• SELECT * FROM one WHERE a==@[5,6]
60. Deep Clone JSON object
• SELECT a FROM one
• SELECT deepClone(a) FROM one
61. ? parameter value
• ? operator
• alasql(‘INSERT INTO one VALUES @{year:?, b:1}’,[2014]);
• alasql(“select * from sales where dt == @{year:?}”, [2014])
• Parameter object property by name
• alasql(‘SELECT $a FROM ?’, [{a:1}]’)
• alasql(‘SELECT :b FROM ?’, [{b:1}]’)
• Array member
• alasql(‘SELECT $2 FROM ?’, [[0,1,2,3,4]]’)
62. Date and Time in Alasql
• Usual realization of date and
time types in different SQL
databases:
• DATE
• DATETIME
• TIMEDIFF
• TIME
• Constants
• “YYYY-MM-DD”
• “YYYY-MM-DD hh:mm:ss”
• “YYYY-MM-DD hh:mm:ss.SSS”
• Definition
• CREATE TABLE orders (
ordered INT,
orderdate DATE
);
• SELECT DAY(orderdate),
MONTH(orderdate),
YEAR(orderdate)
FROM orders
WHERE orderdate
BETWEEN
“2014-01-01” AND
“2014-06-31”
63. JavaScript Date object
• Definition:
• var now = new Date();
• Constants
• No, only new object:
• new Date(2014,0,1)
• How to compare
• new Date(2014,0,1) != new
Date(2014,0,1)
• BUT!
• new Date(2014,0,1).valueOf() !=
new Date(2014,0,1) .valueOf()
• getTime() = valueOf()
64. Alasql approach
• DATE
• “2014-12-01”
• DATETIME
• “2014-12-01”
• “2014-12-01 23:01:23”
• “2014-12-01 12:34:56.123”
• Define class
• alasql.fn.Number = Number;
• alasql.fn.Date = Date;
• Table
• CREATE TABLE orders (
orderid Number,
orderdate Date
);
• Date
• new Date(“2014-12-01”)
• Compare
• new Date(a) == new Date(b)
66. Supported external databases
• Browser
• Local Storage (LOCALSTORAGE)
• IndexedDB (INDEXEDDB)
• Node.js
• DOM-storage (analog of Local Storage) – (LOCALSTORAGE)
• Browser and Node.js
• SQLite (SQLITE)
67. CREATE DATABASE
DROP DATABASE / SHOW DATABASES
• Engines
• CREATE INDEXEDDB DATABASE MyBase
• DROP INDEXEDDB DATABASE MyBase
• Created databases are not attached automatically
• Show databases in Local Storage
• SHOW LOCALSTORAGE DATABASE
68. ATTACH DATABASE
DETACH DATABASE
• Attach database
• ATTACH INDEXEDDB DATABASE Stars
• Attach database as alias
• ATTACH INDEXEDDB DATABASE Stars AS Astras
• Attach database from file (with parameters)
• ATTACH SQLITE DATABASE Stars(“stars.sqlite”)
• Detach database
• DETACH DATABASE Astras
• Attached database is not set as default (use USE
DATABASE statement)
• It is not necessary to USE database to use it (use
database prefixe)
• SELECT * FROM Sky.Stars
69. AUTOCOMMIT option
• Local Storage can work in two modes
• SET AUTOCOMMIT ON (default)
• Alasql stores results of each SQL statement to LocalStorage
• SET AUTOCOMMIT OFF
• Use BEGIN, COMMIT, and ROLLBACK to copy data from memory from
and to Local Storage
74. Angular.js and Alasql
// Export data to Excel file from Angular.js array
function MyCtrl($scope) {
$scope.items = [{City:”Moscow”, Population: 1150000},
{City: ‘New York’, Population: 16000000}];
function exportToExcel() {
alasql(‘SELECT * INTO XLSX(“mydata.xlsx”, {headers:true})
FROM ?’,[$scope.items]);
}
}
75. d3.js and Alasql
// Load data from cities.csv file and create a list with city names.
alasql(‘SELECT * FROM CSV(“cities.csv”,{headers:true})’,[],function(cities){
d3.select(“#cities”)
.append(‘ul’)
.data(cities)
.entry()
.append(‘li’)
.text(function(city){return city.name});
});
77. Alacon – command-line SQL
for data file processing
• Purpose
• Complex text processing
• Batch file format conversion
• Join data files on keys
• Usage:
• node alacon sql param1 param2…
• node –f file param1 param2…
78. Alacon samples
• Convert Excel file
• node alacon “select [0], [1] from xls(‘mytext.txt’)”
• Count number of lines from stdin
• node alacon ‘SELECT VALUE COUNT(*) FROM TXT()’ <a.txt
• Select long lines
• node alacon ‘SELECT * FROM TXT() WHERE LEN([0])>60’ <a.txt
• Grep
• node alacon “SELECT * FROM TXT() WHERE [0] LIKE ’M%’” <a.txt
• Filter lines with ‘one’ word:
• alacon “select line into txt() from txt() where line like ‘%one%’” <a.a
>b.b
• Calculator
• node alacon ‘2*2’
79. Alaserver - very simple SQL server
• Run
• node alaserver –p 8081
• Enter in browser line:
• localhost:8081?SELECT * FROM TXT(‘README.md’)
• or GET
• $.get(“localhost:8081?SELECT * FROM TXT(‘README.md’)”)
• Warning: Alaserver is not multithreaded, not secured, not
protected