Business intelligence, web analytics and FOSS Using the Mail & Guardian Online and Amatomu.com as examples

Background 0.5 million readers, 100 000 visits a day, various sources of BI, vendor lock-in, new territories that lack BI/web analytics

Background Publishing business models dependent on analytics, changes in the Web: Web 2, social media and Semantic Web

Reality The business of a media company is to create sell-able statistics, whereas the public mandate is to inform and encourage democratic governance

Rethinking Watershed moment, great risk and great opportunity

Catalyst Rebuilding our publishing system from the ground up + experience with Amatomu.com

More goals than Bafana - A comprehensive series of dashboards - Real-time reports of user activity - Rich visualisation - A product responsive to key metrics

Specific Challenges Internally, data is not centralised, standardised or aligned to our knowledge management goals

Specific Challenges Web 2, server logging and auditing

Amatomu Growing blogging community, social media

Amatomu Inspired by a lack of information and the potential to facilitate growth of the SA blogosphere

Amatomu Aggregates blog content, gathers analytics and presents personalised statistics, global statistics and rankings

Amatomu Collect statistics for 5.6 million PI per month, 2.5k active bloggers producing 1 05 million db transactions per month

Lessons learned Real-time analytics systems are difficult to scale, data increases in value as it ages, FOSS can do the job

FOSS architecture LAMP stack

The great debate Packaged software or create custom code?

Packaged software Easy to evaluate, low risk, ongoing development and active community, medium reward, hard to tailor to specific business requirements

Custom code Hard to evaluate, high risk, high reward because closely aligned to business processes

BI Packages 2 biggest are Pentaho and JasperSoft, both very powerful solutions for out-the-box BI, both offer FOSS servers with paid support

The thorny path We have chosen to develop custom code because we’re confident we can do it successfully and no packaged software does what we want it to

External factors Systems need to be overhauled every 2 years and rebuilt every 5 years

Guiding principles Data must be reusable, interoperability via web services and XML, scalability via the cloud - Amazon EC2, SimpleDB, S3 storage

Insurance policy Development using frameworks for PHP, Ajax and CSS: CodeIgniter, JQuery/ExtJS, Google Blueprint

Insurance policy Protects us against change in the development team and common mistakes

Emphasis Rich media graphing, real-time visualisation, enabling knowledge creation through discovery

Inspiration Edward R Tufte Digg Labs

Helpful tools FusionCharts - visually attractive graphing, mapping, widgets

Thank You Demo and discussion