The document outlines the use of Node.js within a heterogeneous system for big data analytics, emphasizing its fast performance and compatibility with AWS. It covers project structure using Maven and NPM, testing frameworks, and the deployment process managed by orchestration tools. Additionally, it includes details on setting up a test environment, build processes, and continuous integration practices.

1. Node.js
in a heterogeneous system
Alexey Migutsky : Senior Software Developer @ Lohika

2. What do we do?

3. What do we do?
Ubix.io
PaaS for BigData analytics in cloud
environments.

4. What do we use?

5. Scala
Python
What do we use?
Bash
Javascript

6. BDAS (Spark + Shark)
AWS
Black Voodoo Magic™
What do we use?
Velocipede™
Brains

7. What’s next?

8. Node & Maven
Jenkins & Node testing
What’s next?
Velocipede™
CI

9. ?

10. Why Node.js?

11. Why Node.js?
Fast and easy
Easy websockets support
Plays nice with AWS
We use it as a web-tier orchestrator

12. Node & Maven

13. Node & Maven
We use maven (pom.xml)
We use npm (packege.json)
We have node+maven projects!

14. Project structure

16. Project structure
+ Subprojects are maven modules
+ Subprojects use common code base
+ Easy « build process »
+ Easy deployments
- Uber root package.json
- Mangled dependencies

17. Custom modules

18. Custom modules
Live in root /node_modules right now
Planning to move to separate git repos
Tested with subprojects

19. Tests

20. Tests
Mocha + Sinon + Chai
Sinon-chai + mocha-sinon
Chai-as-promised
Rewired
Jenkins TAP plugin

21. Test example
describe('##getConfiguration', function () {
it('should return promise and resolve with data ', function (done) {
var promise = sysStore.getConfiguration('somekey');
promise.should.eventually.become({key: 'somekey'})
.then(function () {
mockDb.getItem.should.be.calledWithMatch(
mockTables.system,
{key: sinon.match.string}
);
})
.should.notify(done);
});
});

22. Test Env How-to

23. Test Env How-to
Goals:
-
run from root using mvn test
run from sub using npm test
human-readable result logs
result >> in console and file
Jenkins support
http://www.mindmeister.com/346337860

24. Test Env How-to
package.json
"devDependencies": {
"mocha": "*",
"sinon": "1.6.*",
"chai": "*",
"sinon-chai": "*",
"mocha-sinon": "*",
"chai-as-promised" : "*",
"rewire" : "*"
},
"scripts" : {
"test" : "./runtest.sh"
}

25. Test Env How-to
runtest.sh
#!/bin/bash
mocha $(find test -name '*.js') -R tap |
tee test/testresults.tap; (exit ${PIPESTATUS[0]})
The whole command runs in a single line. Wrapped here for display purpose

26. Test Env How-to
pom.xml (maven-antrun-plugin)
<execution>
<id>test-nodejs</id>
<phase>test</phase>
<configuration>
<tasks name="Run mocha tests">
<exec dir="${basedir}”
executable="npm”
failonerror="true">
<arg value="test"/>
</exec>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>

27. Build process

28. Build process
-
No npm install on build server (too slooow)
-
Fixed deps versions (update by-hand)
-
All deployable deps are checked in!
-
Build phase only copies deps in subprojects
http://www.futurealoof.com/posts/nodemodules-in-git.html

29. Deployment

30. Deployment
-
Builds are stored at S3
-
Copied to instances by orchestration tools
-
Managed by upstart

31. Deployment
upstart (server.conf)
#this should be put in /etc/init/server.conf on Ubuntu
description ”Node.js server"
author
"Alexey Migutsky"
# used to be: start on startup
# until we found some mounts weren't ready yet while booting:
start on started mountall
stop on shutdown
# Automatically Respawn:
respawn
respawn limit 20 5
# Max open files are @ 1024 by default. Bit few.
limit nofile 32768 32768
script
# Not sure why $HOME is needed, but we found that it is:
export HOME="/root”
exec /usr/local/bin/node /usr/local/server/app.js >> /var/log/server-err.log 2>&1
end script
post-start script
end script
http://kvz.io/blog/2009/12/15/run-nodejs-as-a-service-on-ubuntu-karmic/

32. Deployment hint
Have a look at http://pm2.io/

33. Continuous Integration

35. « hack some node.js »
npm test

38. build & test

39. build & test
store

40. build & test
store

41. build & test
store

42. build & test
store

43. That’s it!

44. That’s it!
Thanks
@mr_mig_by
bit.ly/mr_mig