GAA Presents "goMongo" and HayStack

www.gaaconsulting.com

Building a Smarter Enterprise
In a Services and Knowledge-based Economy

Migrating Legacy Data: goMongo

Dr. Naveen Kapoor
GAA Consulting
Naveen.kapoor@gaaconsulting.com

GAA CONFIDENTIAL
March 2013 © Copyright GAA Corporation


Objectives
 Demonstrate how GAA is converting legacy databases from SQL to
mongoDB.
 Discuss lessons learnt using multiple conversion projects
 Share knowledge of embedding and linking
 Maintaining referential integrity in mongoDB
Objectives and
Topics Topics
 Background on GAA
 Experience of transformation from relational to mongoDB
 GAA’s solution to ease the transformation process
 goMongo: Conversion tool from relational databases
 Joins, embedding and linking in mongoDB
 Appendix
 Background on migration projects
 GAA Solutions
 AAS Audit Graphs
 On Demand Reporting (ODR)
 Dashboard

2 March 2013 GAA CONFIDENTIAL © Copyright GAA Corporation

GAA Integration, Architecture and On Demand Reporting

Company Overview
Who is GAA Consulting
We are a small business with over four years of experience serving Federal and Commercial clients

Our People
• Services ranging from architecture, system support, on demand reporting and analytics
• Resources skilled in Oracle, Java, Web services, SOA, Javascript, Web development
• Dedicated team that supports its clients on a 24X7 basis

Our Clients

Department of Commerce
US Economic Census
GSA, USGS, VA, DOT

Our Differentiators
• Proven, well established production level systems, system support
• Seven (7) outstanding ready-to-integrate solutions at various Federal Agencies
• Highly skilled resources to support legacy and new application builds
• Dedicated team to support 24X7 data center operations

JAN 2013
3 GAA CONFIDENTIAL © Copyright GAA Corporation
JAN 2013


Authentication and Authorization Solutions

Authentication
Converted to
mongoDB from
- AAS
Oracle and MySQL
Using goMongo

Authorization
- CAMS

Enterprise
- SSO

GAA


Demonstrations

CAMS
Relational Database
mongoDB

goMongo
Conversion
Embedding and Linking



Converting CAMS: Our experiences with goMongo
CAMS is written using JavaScript, PHP, AJAX, Web Services

Data Storage: Oracle RAC
High Availability: Yes
Current Loads: 50K transactions/day
JavaScript: Over 4000 lines
PHP Script: Over 2000 lines

CAMS on mongoDB

Data Storage: mongoDB

High Availability: Replicated

Current Loads: New release

JavaScript: Over 4000 lines – made 10 changes

PHP Script: Less than 500 lines

Time to Convert: 1 week



Transformation: GAA provides goMongo to ease the
transformation process

Step1: Transfer all
relational content to
mongoDB

Step2: Identify PK/FK for
tables in mongoDB

Step3: Convert and Test
using “Embed or Link”
PK/FK relationships



goMongo – Web based tool

Apply
• Connect to the Node/Leaf Rule • Generate foreign
database associations as
• Oracle, mySQL • Convert the tables to embedded array
• Provide u/p mongoDB elements
• Select DB • _id’s are generated • Generate foreign
automatically associations as links

Select tables Test the result



Connect to Relational Database (Oracle)

1. Select
Connection
stored in
CAMS 4. Click
Convert.
Done!!!

2. Tables
loaded from
the
database

Conversion maintains
data types for strings,
integers, floats and date
time.

3. Tables to
collections



View the mongoDB

1. View all
mongoDB
databases
and
collections

2. Select a
collection to
view
documents

3. Click to
edit it in
Document
Editor



goMongo Document Editor

JSON is
loaded. Click
on any field
to view and
edit



Embedding and Linking with goMongo

Preserving the capability for:
Self referencing
 Application has sub-applications

1-many relationships
 A Classroom has many students

Many-many: would like to embed or link all
users in the group collection.
 USER (ID)
 GROUP (ID)
 USERGROUP(USERID, GID)



goMongo addresses all types
Over 4K embeds or linkages in under a second

1. Identify 2. Identify 4. Select
3. Identify
Primary, Fields in option for
Brokerage
Foreign and Primary and embed or
Collection
Brokerage Foreign link and label
fields
Collections Collections in FC



Node/Leaf Rule: 2 Simple rules for converting Associated entities
Rule 1: If entities are leaf then Embed the associations
Rule 2: If entities are Nodes Link the associations

Application Permissions

AID = 1 ID = 1
PID = 1 Permission = Create Role
Application Name = ODR ID = 1
Permission = Create User
Embed the Permissions Entity
Since it has no sub entities.
Permissions entity is a leaf

mongoApplication
_id : 1
PID : [{“0”:{“id”:0,”perm”:”create role”}, “1”:{“id: 1,”perm”:”create user”}}]

Application Name : ODR



Node/Leaf Rule: 2 Simple rules for converting Associated entities
Rule 1: If entities are leaf then Embed the associations
Groups
Rule 2: If entities are Nodes Link the associations
GID = 1

RID = 1

Application Roles Group Name = Admin Group

AID = 1 Groups
RID
GID = 2
RID have multiple roles GID have multiple
Groups RID = 1

Group Name = Admin Group
Application Name = ODR

Link the Roles Entity
Since it has sub entities ie Groups.

mongoApplication mongoRoles mongoGroups
_id _role_1 _id _group1
_id : 1
groups: Group name: Admin Group
roles : [_id_role1,_id_role2] [_id_group1,_id_group2]



Joins in MongoDB
Function: JoinTables

No. of Arguments: 6
Arg Name Type Description
1 $PT String Primary Table which has the unique set of values
2 $FT String Secondary Table which has the repeating set of values
3 $IF String Link Field. This is the string value that is used in the $FT to link to the
$PT. The $FT will have an array of values which is stored under the $lf
label
4 $match Array Matching value for example array('USERNAME'=>'kapoo304'). The
Array USERNAME field is in the $PT collection
5 $PTC Array Array of columns that you want from the $PT. for example:
array('USERNAME'=>'User') (Actual table column and display column)
6 $FTC Array Array of columns that you want from the $FT. for example:
array('GROUPSNAME'=>'Group') (Actual table column and display
column)

Response: An array of results which shows the values that match the matchArray. Use
the response from this to pass to the first argument of the joinArray function.



Joins in MongoDB – Cont…
Function: JoinArray

No. of Arguments: 6
Arg Name Type Description
1 $PA Array Primary Array which has the unique set of values
2 $FT String Secondary Table which has the repeating set of values
3 $IF String Link Field. This is the string value that is used in the $FT to link to the
$PT. The $FT will have an array of values which is stored under the $lf
label
4 $colum String Name of column from the $PA that matches in value in the lined values
nMatch in $FT under the $lf label
5 $PTC Array Array of columns that you want from the $PA. for example:
array('USERNAME'=>'User') (Actual table column and display column)
6 $FTC Array Array of columns that you want from the $FT. for example:
array('GROUPSNAME'=>'Group') (Actual table column and display
column)

Response:



Data Abstraction Layer and GAA driver for mongoDB: A logical
representation of mediation to support noSQL

Application Server

Data Abstraction Layer

Database Specific Driver
GAA driver for goMongo
Example: mySQL driver

MySql mongoDB



Appendix
Node/Leaf Rules
GAA Products using goMongo



AAS – Authentication and Authorization Service

Forms/
PL/SQL

SOAP
Web Service
Browser requests
APP/DB Session
Request Server Information
Pages
AdHoc
Queries

AAS Manage Authorization
Authorization
Manage Authorization Server Information
RCP

CAMS SOAP Service requests
Web
based
LDAP
Server



Single Sign-On using SAML and Kerberos Certificates

Value Creation:
1) Application Users can
authenticate against single
enterprise source of identity.
2) Provides high levels of security
3) No passwords are transferred
over the wire
4) Integrate SSO easily with new
and legacy applications using
.jar, and web services that can
be leveraged across multiple
software languages.
5) One call to authenticate and
authorize user.
6) Sessions are maintained in the
database as opposed to files;
easing scaling and high volume
traffic
7) User transactions can be
logged and integrated with
analytics engine
8) CAMS provides the historical
charts on transactions by user,
by application and time periods



Authorization - Increasing reusability with CAMS
Governance, Security, Control, Monitoring, Management
• As an enterprise how do we manage the growing interconnections amongst applications?
• What if I move the end point of the service providers?
• Rapid provisioning of new business applications, services and service providers.
• Monitoring, Security, Flexibility and Resiliency needs to be ensured

Deliver architecture using services
• Provide Security, Governance, Monitoring, Auditing and Control
• Load balancing to support a Federated service delivery platform
• Identify and deliver new services by leveraging SOA suite tools: BPEL

Business Entities SOA Governance LDAP
1. Security Policies (HTTPS, SLA) Authentication Service
2. Business Rules (Content Based Routing)
3. Monitoring
Monitoring and rapid growth:
1. Monitoring SLAs
2. Integrate business services through ESB’s exposed end points to
Business Applications
create new business services

ESB AAS
Route dynamically Authentication and Load balanced, highly
Transport formats Authorization available authorization
Mediate and DR, LB, HA Services information
Legacy applications

Logs

Service Registry & Logs
Stores WSDLs , End Points for HA, LB, DRM & Logs

MARCH 2013


CAMS: Business Owners can manage and govern consolidations,
applications and entitlement
Value Creation:
1) Multiple applications and their
owners can assign permissions
to users via groups and roles.
2) Application space can be
further broken down into sub-
applications
3) Supports single sign-on.
4) Supports web services with
adapters using REST or SOAP
protocols
5) Configurations for applications
can be centralized
6) CAMS as an application
introspects to provide access to Click a group
manage privileges for other to manage Roles,
applications RLP, Users
7) Business Owners manage who
can do what
8) Rapid deployment
9) Business owners, developers,
testers and deployment staff
are consumers of this Click on user to
application remove from group
10) Proven solution
Click user to add to
the group



AAS Web services that provide flexible integration across the
enterprise. Supports both REST and SOAP based protocols.

Value Creation:
1) Supports single sign-on for
various applications written in
Java, PHP, Python, JavaScript,
Oracle PL/SQL
2) Core web services based
technology that supports both
SOAP and REST based web
services call
3) Supports authentication by
integrating with LDAP servers
4) Load balanced and highly
available to address disaster
and performance issues
5) Sessions are managed in
MongoDB for high performance
as opposed to a file based
management system
6) Scales up to 100,000+
transactions per day



GAA Solutions



AAS audit graphs: Monitor and track usage of applications

• View delivered
services
• Governance
• Management
• SLA
Compliance
• PDF Reports
• Authorization
Network



CAMS delivers On-demand reporting: Mashup content from diverse
platforms while delivering single-view to the Client

Problem:
No centralized reporting system
that will integrate with systems and
applications

Solution:
On demand reporting that
integrates with applications, LDAP,
Directory and servers

Value Creation:
1. Users can formulate their own
request for information via an
email
2. Information can be retrieved on
an on-demand basis.
3. Integrates with any
applications, system support,
Remedy, COTS products.
4. Services both mobile and
standard platforms

MARCH 2013

GAA Presents "goMongo" and HayStack

More Related Content

Viewers also liked

Similar to GAA Presents "goMongo" and HayStack

More from MongoDB

GAA Presents "goMongo" and HayStack