Subject: Integral Solutions for Batch Processes
Written by: Walter Valiente
Title: Lead Developer at VMN+
Company: http://www.vmnplus.com
Introduction: This paper focuses on the necessity to find a generic product that provides
support in developing applications in a batch process for high volume information in spite
of a high pressure timetable
Brief history: In my current job, I recently had the experience of being able to develop a
completely new product. We ended up developing two new products to complete the
solution. It wasn’t until almost the end of the project that the need to join these two
products in order for them to function properly surfaced. We achieved this by generating
a huge chain of shell scripts, tables and store procedures. It was almost as difficult to
sustain, as we needed the central processes to interrelate. For this reason, I will try to
explain the usefulness of having a single process to join these 2 skeletal development
batches.
Problem to solve: In the first phase of the project there is a comprehensive analysis of the
system. After several meetings with various modules, some common features between
revenues and expenditures arose. These features relate to how they were obtained or
how the information was saved. For example: the external systems are related to ours by
means of archives, while the internal relationship of the system was given by future
boards that had yet to exist. From this idea we discovered that different modules can
read and write files in tables and tables that can read and write files. In summary the
module may need a combination of files and tables to your posts and / or outputs.
Figure 1 shows:
(Module)
Archive Archive
s s
o Entrance Exit o
Tables Tables
Illustration 1: Distinct Entrance and Exits
We agreed that the best solution would be to make an abstract process to undertake
“something” so that different modules can do their job without taking into account the
nature of entry and/or exit.
On the other hand we need a process – this was the first to be developed that can
manage requests for processing to be resolved by stored procedures or
http://www.vmnplus.com Distributed Agile Development On-Demand
All content in this paper is protected copyright laws 2009

Subject: Integral Solutions for Batch Processes
Written by: Walter Valiente
Title: Lead Developer at VMN+
Company: http://www.vmnplus.com
shell scripts, for a given amount of time. For example, sending an e-mail every 1st of the
month that contains reports to the users.
Almost finishing the stage of development, we realized that the only thing that was
necessary to execute, was the modules of entry/exit that are generically explained in the
proceeding paragraph. Do to time constraints, both processes were kept separate and
are connected by scripts, tables and store procedure.
This paper explains how the sections from the union of these two processes would be in
only one administrator of generic modules (AGM)
Overview:: This product AGM, should consist of 6 units which are described below:
Entrance Exit
AGM
Map Uploader Map
Archives Archives
Entrance Departure
o Virtual o
Virtual Virtual
Tables Registry E S Registry Tables
Administrator
Module 1 Module 2 Module N
E S E S E S
Illustration 2: incide AGM.
Uploader: This is the starting point of the application; its job is to assign the configuration
of modules within AGM. The configuration applies to both its style of execution as well as
the type of entry and exit to use. The system expected to meet its running time later
on.
Administrator: The Administrator is responsible for being the trigger of the modules to
perform independently from the rest. The types of implementation can be at this time or
for a specific date. The implementation period can be one-time, a number of times. It’s
usually every half hour or,
another method would be every 31 days at a time. It’s a vital section that begins to
prepare this process to support the batch used batch window with saturated servers
http://www.vmnplus.com Distributed Agile Development On-Demand
All content in this paper is protected copyright laws 2009

Subject: Integral Solutions for Batch Processes
Written by: Walter Valiente
Title: Lead Developer at VMN+
Company: http://www.vmnplus.com
Entrance Map: This has the algorithms to interpret the different types of files. The files
can be encolumnated, or binary XML. In addition they may contain a header, body and
foot, or be flat. It also makes all necessary checks to see if a file does not exist, it’s
badly formatted or simply has bad information.
Virtual Registry: This is the point of connection between a module and its input /
output. It represents the information in the abstract. To make understandable the
different fields and values of this registry, I used a map type object. It’s what relates the
variable name with its type, value and features that come from a configuration file style
CFS.
Virtual Module: AGM only understands and orchestrates with this module. The
development of the final modules is where the business logic must adapt to this
particular structure. This module is actually a class type interface in which the other
modules inherit their virtual methods. Among its main methods are:
• PARAMETERS (): Allows you to pass parameters from external development.
• BEGIN (): To make the module perform everything you need before starting to process.
• VALIDATE (): checks for registry entry from the user logic. For example, age is not
greater than 130.
• PROCCES (): It is for the bulk of the module. It is where the data is processed and
prosecuted.
• END (): All references to the word processing, commits this conduct in the base if
everything was okay.
• EXCEPTION (): what makes the module in case of any error detected by AGM or
invoked in the same module.
Map Output: Has the inverse algorithms of Map Input since it becomes a virtual record in
almost any type of file.
Map Departure Has the inverse algorithms of the Map Entrance since it becomes a
virtual record in almost any type of file
Collectively, these sections achieve a mechanism for rapid development (AGM performs
heavy work) like processing, since data is handled in the buffer memory (virtual log).
http://www.vmnplus.com Distributed Agile Development On-Demand
All content in this paper is protected copyright laws 2009

Subject: Integral Solutions for Batch Processes
Written by: Walter Valiente
Title: Lead Developer at VMN+
Company: http://www.vmnplus.com
Upcoming developments: In future, this product can bring new features, both functional
and performance.
• A unit of performance. Enable performance measurements in metric and processing
volume time.
• While the AGM in itself is efficient as a product, Serial processing row by row may give
rise to bottlenecks. The solution to this problem is to shield the virtual module with
multiprocessing functions.
• To configure the map entrance, it would be useful to use regular expressions in
filenames. This way, the modules can have more than one input file for processing.
• Another benefit is that the mapper can handle input and output transfer protocols like
FTP. In this position, it would have a chance not only to interpret any type of file, but
also look or leave it on any server.
• For the administrator section, it would be useful to have the possibility to configure a
chain of execution, with the end being that one could come to depend on the result.
The connection between them can be given by connecting the virtual records of both
modules. The output register would be a check on the other module. .
• A new entry and exit such as graphical interface would be desirable if the processing
module is lightweight.
Conclusion: It’s been shown that one can build an integral product of the design of batch
processes, adaptable to practically any business logic required. It’s used to process large
amounts of information, for joining an execution process manager to another processor in
a single scalable product and generating an advantage in strength and speed of
implementation and development. It offers good performance to the modules to work
entirely in memory and also features maintainability.
Give us a call
415-992-5388 SFO
708-441-4124 Chicago
http://www.vmnplus.com Distributed Agile Development On-Demand
All content in this paper is protected copyright laws 2009