Process & Tools to incorporate agility and collaborative Development model in Projects, Python based project as use-case

1. Ingredients of Agile and
Collaborative Development Model
Python Based Projects
msreejith@yahoo.com

2. Agenda
• Introduction
– Why adopt agile & collaborative model
• Source Code Structure
• Testing Considerations
• Sample Packages & Tools

3. Introduction
• Why Agility needs to be factored in
– When you have to respond to adhoc new/changing requirements
during the project sustenance lifecycle
– When you need fast iterative go-to-prod cycles
• Why Collaborative model needs to be factored in
– When development needs to be de-centralized and multiple disparate
teams contribute
– Organizations may have pockets of “SMEs” that could collaborate on
functionality in parallel
• Testing Considerations
– Testing & Automation has to be integrated at several stages
– Localized unit testing of framework, utils and contributed modules

4. Source Code Structure
• Identify and Segregate what constitutes “Framework” from rest of the components
– Core Framework component, for instance, should be the only part of your code that follows “closed”
development model
– i.e a centralized team owns and maintains the “Framework” modules
– “Framework” changes could be less frequent than rest of the components
– Good example is a Framework that abstracts the “data store” interactions and prepares a Data-model to be
consumed by other collaborative developers
• Identify and segregate Non-framework modules
– Non-framework modules should be “open” for collaborative development
– Non-framework modules should be consumers of the Framework and can follow agile development schedules &
practices independently
– Good example of Non-framework module is “Rules” development. Rules could be developed collaboratively by
“SMEs” from disparate teams. They could do so by subscribing and adhering to the core Framework published
Interfaces (data-model, or getter APIs for instance)
• Testing Considerations
– Framework should provide unit-test modules and templates for both core framework and the non-
framework modules
– Framework should also provide “Integration testing” hooks – where the new changes could be executed on
a target environment representing production-like dataset. This is crucial to help identify quality issues early
in development considering short agile deployment cycles

5. Source Code Structure
project_root/
framework/
rules_root/
framework/
src/
utils/
unittest/
conf/
rules_root/
data/
rules/
unittest/
- Closed, waterfall development
- separate framework repo
- Framework code packages, and runs the
rules, abstracts data-store details
- Collaborative, agile development
- separate rules repo
- Rules code only focuses on business
logic using data-model/interfaces
published by framework runtime

6. Sample Packages & Tools
Category Package/Tool Name Purpose
Deploy & Setup Virtualenv Tool to create Isolated Python env that would provide
consistent library versioning deployment of project
Dev Automation Jinja2.Template Templatize and auto-generate Class definitions, Unit test
module skeleton, Html report format generation etc to
minimize manual repetitive work
Dev Workflow git Quick project setup time.
Source control for collaborative & agile development, code
review processes
Dev Json;
Avro
Input/Output data persistence and movement;
To maintain versioning across releases.
Localized
Development and
Unit testing
[Connectors to data sources to
fetch sample data into project
framework]
Nosetest
For each unit-test “fetch” and persist sample data
representive of Positive & Negative test-cases for unit-test
local execution.
Automatically discover and run tests.
Continuous Testing Jenkins Continuous Integration – Run regression tests with every
check-in. Also run full functional-test with complete run-
time env and complete input data mimic’ing Production
Documentation Pydoc Major functions and util packages should be documented
with pydoc
Documentation &
Reporting
Sqllite database Any documents and report generation (for results analysis,
metadata analysis)