Software development used to be like Advertising. You knew 50% of the spend worked, but you didn't which 50%. Today both advertising and software development are becoming more data-driven, as the breadcrumbs become ever more accessible.
Insights: July 26 big data workshop for causesLarry Eason
Insights: Big Data was hosted by IBM at their Innovation Center in Waltham, MA July 26, 2013. Sponsors were my firm DotOrgPower, Findability Sciences and Mind Over Media, Inc.
The simplest definition of Big Data is large and complex unstructured data (images posted on Facebook, email, text messages, GPS signals from mobile phones, tweets, and other social media updates…etc.) that cannot be processed by traditional database tools.
Great quotes from historical leading thinkers (Einstein) to contemporary (Tim O’Reilly & DJ Patil) have to say about the power, use and analysis of data.
Insights: July 26 big data workshop for causesLarry Eason
Insights: Big Data was hosted by IBM at their Innovation Center in Waltham, MA July 26, 2013. Sponsors were my firm DotOrgPower, Findability Sciences and Mind Over Media, Inc.
The simplest definition of Big Data is large and complex unstructured data (images posted on Facebook, email, text messages, GPS signals from mobile phones, tweets, and other social media updates…etc.) that cannot be processed by traditional database tools.
Great quotes from historical leading thinkers (Einstein) to contemporary (Tim O’Reilly & DJ Patil) have to say about the power, use and analysis of data.
Linked Water Data For Water Information ManagementEdward Curry
The management of water consumption is hindered by low general awareness and absence of precise historical and contextual information. Effective and efficiency management of water resources requires a holistic approach considering all the stages of water usage. A decision support tool for water management services requires access to a number of different data domains and different data providers. The design of next-generation water information management systems poses significant technical challenges in terms of information management, integration of heterogeneous data, and real-time processing of dynamic data. Linked Data is a set of web technologies that enables integration of different data sources. This work investigates the usage of Linked Data technologies in the Water Management domain, describes the fundamental concepts of the approach, details an architecture, and discusses possible water management applications.
This presentation offers a basic understanding of Big Data. It does this by defining Big Data, offers a History of Big Data, Big Data by the Numbers and the 8 Laws of Big Data
Big Data, Trends,opportunities and some case studies( Mahmoud Khosravi)Mahmood Khosravi
Humans have been generating data for thousands of years. More recently we have seen
an amazing progression in the amount of data produced from the advent of mainframes
to client server to ERP and now everything digital. For years the overwhelming amount
of data produced was deemed useless
Within the operational phase buildings are now producing more data than ever before, from energy usage, utility information, occupancy patterns, weather data, etc. In order to manage a building holistically it is important to use knowledge from across these information sources. However, many barriers exist to their interoperability and there is little interaction between these islands of information. As part of moving building data to the cloud there is a critical need to reflect on the design of cloud-based data services and how they are designed from an interoperability perspective. If new cloud data services are designed in the same manner as traditional building management systems they will suffer from the data interoperability problems. Linked data technology leverages the existing open protocols and W3C standards of the Web architecture for sharing structured data on the web. In this paper we propose the use of linked data as an enabling technology for cloud-based building data services. The objective of linking building data in the cloud is to create an integrated well-connected graph of relevant information for managing a building. This paper describes the fundamentals of the approach and demonstrates the concept within a Small Medium sized Enterprise (SME) with an owner-occupied office building.
Trends in Big Data & Business Challenges Experian_US
Join our #DataTalk on Thursdays at 5 p.m. ET. This week, we tweeted with Sushil Pramanick – who is the founder and president of the The Big Data Institute (TBDI).
You can learn about upcoming chats and see the archive of past big data tweetchats here
http://www.experian.com/blogs/news/about/datadriven
Matt Sadler Infomagination Fs Forum 061009mattsadler
A presentation by Matt Sadler, Senior Planner at Euro RSCG London, to the Financial Services Forum on the 6th of October 2009, as part of their Age of Infomagination workshop event.
Visit www.infomagination.co.uk for more info.
Improving Policy Coherence and Accessibility through Semantic Web Technologie...Edward Curry
The complexity, volume and diversity of government policies and regulations raises significant burden on both the complying parties and government itself. On the one hand, businesses, civil organizations and other societal entities are required to simultaneously comply with and interpret different and possibly conflicting or inconsistent regulations. On the other hand, government as a whole must ensure policy and regulatory coherence across its various policy domains. While the recent wave of open government initiatives have led to significantly more public access to these documents, features allowing cross-referencing related documents and linking to less formal documents or comments on other media more understandable and accessible to the public are not common if at all available today. As a solution to this challenge, we propose an Open Government-wide Policy and Regulation Information Space consisting of documents that are “semantically” annotated and cross-linked to other documents in the information space as well as to external resources such as interpretations, comments and blogs on the social web.
Our approach is three-fold. First, we identify the requirements for the infrastructure. Second, we eloborate a Reference Architecture identifying the various elements needed within the infrastructure. Third, we show how such infrastructure may be realised as a linked data portal where policies and regulations are published as linked open data. Finally, we present a case study involving environmental policy and regulations; discuss the potential impact of such infrastructure on coherency and accessibility of policies and regulations and concludes with challenges associated with provisioning a linked open policy and regulatory information infrastructure.
Linked Water Data For Water Information ManagementEdward Curry
The management of water consumption is hindered by low general awareness and absence of precise historical and contextual information. Effective and efficiency management of water resources requires a holistic approach considering all the stages of water usage. A decision support tool for water management services requires access to a number of different data domains and different data providers. The design of next-generation water information management systems poses significant technical challenges in terms of information management, integration of heterogeneous data, and real-time processing of dynamic data. Linked Data is a set of web technologies that enables integration of different data sources. This work investigates the usage of Linked Data technologies in the Water Management domain, describes the fundamental concepts of the approach, details an architecture, and discusses possible water management applications.
This presentation offers a basic understanding of Big Data. It does this by defining Big Data, offers a History of Big Data, Big Data by the Numbers and the 8 Laws of Big Data
Big Data, Trends,opportunities and some case studies( Mahmoud Khosravi)Mahmood Khosravi
Humans have been generating data for thousands of years. More recently we have seen
an amazing progression in the amount of data produced from the advent of mainframes
to client server to ERP and now everything digital. For years the overwhelming amount
of data produced was deemed useless
Within the operational phase buildings are now producing more data than ever before, from energy usage, utility information, occupancy patterns, weather data, etc. In order to manage a building holistically it is important to use knowledge from across these information sources. However, many barriers exist to their interoperability and there is little interaction between these islands of information. As part of moving building data to the cloud there is a critical need to reflect on the design of cloud-based data services and how they are designed from an interoperability perspective. If new cloud data services are designed in the same manner as traditional building management systems they will suffer from the data interoperability problems. Linked data technology leverages the existing open protocols and W3C standards of the Web architecture for sharing structured data on the web. In this paper we propose the use of linked data as an enabling technology for cloud-based building data services. The objective of linking building data in the cloud is to create an integrated well-connected graph of relevant information for managing a building. This paper describes the fundamentals of the approach and demonstrates the concept within a Small Medium sized Enterprise (SME) with an owner-occupied office building.
Trends in Big Data & Business Challenges Experian_US
Join our #DataTalk on Thursdays at 5 p.m. ET. This week, we tweeted with Sushil Pramanick – who is the founder and president of the The Big Data Institute (TBDI).
You can learn about upcoming chats and see the archive of past big data tweetchats here
http://www.experian.com/blogs/news/about/datadriven
Matt Sadler Infomagination Fs Forum 061009mattsadler
A presentation by Matt Sadler, Senior Planner at Euro RSCG London, to the Financial Services Forum on the 6th of October 2009, as part of their Age of Infomagination workshop event.
Visit www.infomagination.co.uk for more info.
Improving Policy Coherence and Accessibility through Semantic Web Technologie...Edward Curry
The complexity, volume and diversity of government policies and regulations raises significant burden on both the complying parties and government itself. On the one hand, businesses, civil organizations and other societal entities are required to simultaneously comply with and interpret different and possibly conflicting or inconsistent regulations. On the other hand, government as a whole must ensure policy and regulatory coherence across its various policy domains. While the recent wave of open government initiatives have led to significantly more public access to these documents, features allowing cross-referencing related documents and linking to less formal documents or comments on other media more understandable and accessible to the public are not common if at all available today. As a solution to this challenge, we propose an Open Government-wide Policy and Regulation Information Space consisting of documents that are “semantically” annotated and cross-linked to other documents in the information space as well as to external resources such as interpretations, comments and blogs on the social web.
Our approach is three-fold. First, we identify the requirements for the infrastructure. Second, we eloborate a Reference Architecture identifying the various elements needed within the infrastructure. Third, we show how such infrastructure may be realised as a linked data portal where policies and regulations are published as linked open data. Finally, we present a case study involving environmental policy and regulations; discuss the potential impact of such infrastructure on coherency and accessibility of policies and regulations and concludes with challenges associated with provisioning a linked open policy and regulatory information infrastructure.
A talk laying out the near future of intelligent conversational interfaces – programming for the mr robot era. Bots and machines talking to humans talking to machines. In this talk RedMonk industry analyst James Governor will lay out the emerging landscape for bot-driven apps and machine learning. As we move into voice-driven programmable platforms like Google Now, Siri and Cortana , and intelligent agent-based systems, developers are getting rid of the old UIs completely. What will the world look like when we start talking to the machines, and no longer need to walk around looking down at our phones all the time? A number of trends are changing as machine learning become a reality and we see new platform abstractions and approaches emerge. Big Data, pervasive telemetry, augmented reality, machine learning, integration of every service– what’s not to like?
Digital london digital breadcrumbs presentationJames Governor
my deck for Digital London. Not sure I nailed the data science angle, but i got good feedback. i look at the economics of valuation based on data, for example.
This speech will talk to the major trends shaping the business of business analytics. From compliance to carbon accounting, from cloud computing to open source software. We're in an era of unprecedented change. Every Google search is in effect an ad-hoc query, which has utterly changed expectations for business analytics. IT economics continue on the trend to abundance, while business is now being designed around constraints - analytics and reporting will be at the heart of the change.
I originally intended to present on how we should be careful not to sleepwalk into closed systems on the web that can be switched off at a moments notice- what I call the Permission-based web. But then the Arab Spring happened. Then the London Riots happened. I decided to take a much deeper look at the Open Society, and how all this ties together.
Disposability has been a long time coming – I remember the aha when I first heard how Google dealt with hardware failures – by simply throwing out the offending boxes every once in a while, with no need to do anything immediate. At Google the software architecture meant that hardware became disposable. Today that architectural idea(l) is becoming a core design principle for Cloud Native software – 12 Factor Apps. Everything should be disposable. Except of course the data… Apps are like fish, but data is like wine. Stateless apps still require a persistence layer.
Disruption and the New Kingmakers: Software Developers and Payment ServicesJames Governor
Talk at Payments International 2015 in which I talked to a bunch of payments experts about what's going on in software and disruption. The revolution thus far is not being driven by public APIs.
rethinking testing - what happens after we kill waterfall with fire? how cloud native companies do what enterprises can learn from them? on agile, TDD, but most importantly making engineering tradeoffs
OpenStack is the most widely supported open source cloud technology platform, making it ideal for enterprise organizations with preferred technologies and unique requirements. With its standard APIs, broad ecosystem of integrated technologies, and ability to span both private, public and hybrid clouds, OpenStack is providing many organizations with new agility in their datacenters. Learn how large enterprise users like Disney, Best Buy and Wells Fargo are approaching the shift to the cloud, moving to infrastructure consolidation under a single management layer with OpenStack.
The talk will cover in broad strokes the building blocks, facilitators and challenges for big data based decision making.
Using examples from two projects from very dissimilar domains (High tech manufacturing and Public Health) Dr. Vinze will present possibilities for Data Science for both practitioners and academic researchers.
Big Data Trends and Challenges Report - WhitepaperVasu S
In this whitepaper read How companies address common big data trends & challenges to gain greater value from their data.
https://www.qubole.com/resources/report/big-data-trends-and-challenges-report
A l'occasion de l'eGov Innovation Day 2014 - DONNÉES DE L’ADMINISTRATION, UNE MINE (qui) D’OR(t) - Philippe Cudré-Mauroux présente Big Data et eGovernment.
Big data. Small data. All data. You have access to an ever-expanding volume of data inside the walls of your business and out across the web. The potential in data is endless – from predicting election results to preventing the spread of epidemics. But how can you use it to your advantage to help move your business forward?
Data is growing exponentially and it’s now possible to mine and unlock insights from data in new and unexpected ways. Empower your business to take advantage of this data by harnessing the rich capabilities of Microsoft SQL Server and the familiarity of Microsoft Office to help organize, analyze, and make sense of your data—no matter the size.
Advanced Analytics and Machine Learning with Data VirtualizationDenodo
Watch full webinar here: https://bit.ly/3aXysas
Advanced data science techniques, like machine learning, have proven to be extremely useful to derive valuable insights from your data. Data Science platforms have become more approachable and user friendly. With all the advancements in the technology space, the Data Scientist is still spending most of the time massaging and manipulating the data into a usable data asset. How can we empower the data scientist? How can we make data more accessible, and foster a data sharing culture?
Join us, and we will show you how Data Virtualization can do just that, with an agile and AI/ML laced data management platform. It can empower your organization, foster a data sharing culture, and simplify the life of the data scientist.
Watch this webinar to learn:
- How data virtualization simplifies the life of the data scientist, by overcoming data access and manipulation hurdles.
- How integrated Denodo Data Science notebook provides for a unified environment
- How Denodo uses AI/ML internally to drive the value of the data and expose insights
- How customers have used Data Virtualization in their Data Science initiatives.
DataEd Slides: Data Management Best PracticesDATAVERSITY
It is clear that Data Management best practices exist and so does a useful process for improving existing Data Management practices. The question arises: Since we understand the goal, how does one design a process for Data Management goal achievement? This approach combines the DM BoK and the CMMI/DMM, permitting organizations with the opportunity to benefit from the best of both. The approach permits organizations to understand current Data Management practices, strengths to leverage, and remediation opportunities. In a nutshell, it describes what must be done at the programmatic level to achieve better data use.
Everyone knows there's too much big data. But what's the best way to harness the power of big data? This presentation discusses three analytic engines that companies big and small are using to capture, store, transform and use big data. Also included are case studies of big data in action.
Connected Farms ...and the Digital CatapultChandan Rajah
Presented at the "Towards a Connected Farm" workshop conducted by the Satellite Applications Catapult, this presentation highlights the ideas and trends while moving towards a connected farm.
2020 progressive delivery, git ops, observability James Governor
GitOpsDays keynote. Tech trends in 2020. Progressive Delivery, GitOps and Observability. Moving to a product management driven culture of experimentation. But what underpins that. Automation, collaboration through GitHub. all the companies making it easier to experiment with features before rolling them out to more users. Tools used and built by Nike, CapitalOne, Target, Uber.
Talk about Progressive Delivery to Aginext, a community conference for agile practitioners and coaches. Progressive Delivery is a take on CI/CD with a particular focus on Experimentation, with blue/green deployments, canarying, and feature flags
session from devops world covering everything from accessibility and inclusion to sustainability and the future of work and making better renewable energy choices.
CI/CD and Progressive Delivery. Reframing velocity vs risk. James Governor
DevOps practices and agile development have been transformational for companies that have successfully adopted them, leading to far greater development velocity, higher code quality, with fewer outages. So if you're planning to do multiple production deploys per day what they might look like in an enterprise context? What would it take and where can you start?
Leveraging CI/CD. Progressive delivery: What it is and how it can benefit you?
Infrastructure as code deployment with pipelines: Employing DevOps in your IT processes
A talk at DevOps World/Jenkins World about growth in the world software developer populations, with a global scale take on the coming software/industrial transformation. Look at platforms, geographies, diversity&inclusion and sustainability.
Goto Copenhagen: How beauteous technology is! O brave new world. James Governor
In this Brave New World with massive software-driven disruption, take advantage of new technologies like serverless and Kubernetes and approaches including Continuous Delivery and DevOps. What are the new skills and technologies that will lead to the first billion dollar one person company. But power should also mean responsibility - a look at the ethics of new software-driven capabilities.
Developer Relations is hard. Long hours, tons of emotional labour, absurd travel schedules, and then developers just complain about what feature they did or didn't get. In this talk we used Rolling Stones songs to kick off a conversation about the good, the bad, and the ugly of Developer Relations. Do you work for the man, are they treating you right, and how are smart companies scaling, while still showing empathy for the end user- the developer.
A Tale of Two ITs. Tech, Power, ResponsibilityJames Governor
It's a cliche to say it's the best time in history to be a software developer. We have so much capability at our disposal and we don't need to ask for permission any more. The technology landscape, and the world at large, is changing faster than ever - parts of the stack are well defined, but innovating on top of that will creates new challenges. In this talk RedMonk I outline developer trends, directions, and choices, as cloud native technologies and approaches become ubiquitous. Kubernetes won - so now what? We examine adoption trends around containers and associated technology including service meshes, but also consider the role of serverless technology in a container native world. New stacks are enabling progressive deployment, but we need approaches such as chaos engineering, creating resilience by designing for failure.
While it's true that developers are the new kingmakers, with great power comes great responsibility. Are developers ready for that responsibility? We'll need new ethical frameworks to make better, more socially responsible choices. Just because you can doesn't mean you should.
A talk about what I call Progressive Delivery: a basket of skills and technologies concerned with modern software development, testing and deployment: including canarying, Feature Flags, A/B testing at scale. We're seeing Advances in approaches to application and service. On the technology side, Kubernetes and Istio bring new management challenges, but also opportunities – service mesh approaches can enable a lot more sophistication in routing of new application functions to particular user communities, managing and mitigating the blast ratio of a service.
Progressive Delivery at Spring One PlatformJames Governor
Talk at Pivotal's Spring One Platform event. Progressive Delivery: a term to describe a new basket of skills and technologies concerned with modern software development, testing and deployment. Canarying, Feature Flags, A/B testing at scale. Advances in approaches to application and service Observability. On the technology side, Kubernetes and Istio bring new management challenges, but also opportunities – service mesh approaches can enable a lot more sophistication in routing of new application functions to particular user communities.
Presentation at New Relic's FutureStack 2018 examining trends and new disciplines in modern software development, driven by the need to increase code and feature velocity while improving code quality.
So I had 10 minutes to talk about The Future in the Thursday morning keynote at Spring One 2017. On the same bill as Eric Brewer... no pressure then. Funnily enough in many ways it's harder to prepare a 10 minute talk than one that lasts 30. Machine learning, AI, empathy-based ops, taking responsibility, voice recognition and the emerging event-based 2 tier architectures. Spring, Cloud Foundry and Node.js. What should we build and why?
Logging and machine data at Scale. re:Invent 2017James Governor
Talk about the tapping into the unrealised value of log and machine data in order to create new products and services and improved customer experiences. The value of Splunk and rise of ELK for logging and monitoring. Scale at Facebook. Observability as the new hotness, with thanks to Charity Majors. Use cases and customer examples (bosch etc)
convenience is a powerful force in consumer choices, but also plays a huge role when it comes to developer experience and tech adoption. From Uber to deliveroo, docker to mongoDB, convenience is the killer app. Convenience can have a downside though. A talk on convenience at mongodb world europe
keynote for MongoDB Europe. the awesome power of convenience. making things easy for developers and they'll build services that are more convenient for your citizens, customers and and end users.
Data transformation is the new digital transformationJames Governor
A short talk for International Payments summit in London - looking at the how and why of data culture, the value of data, and the importance of new approaches to information security. Will data ever be a balance sheet item?
a talk i gave about labs approaches to digital transformation, how corporates like going to different spaces, working alongside startups and or web developers, to learn new skills
Space cluster disrupt, Open Innovation labsJames Governor
talk about open source culture, disruption and modern software development, how to be more like Web Companies in terms of delivering new digital services to market faster
9. 9
“The lumber industry sells what used to be waste
— sawdust, chips, and shredded wood — for a
pretty profit. Today you’ll find these by-products in
synthetic fireplace logs, concrete, mulch, particle
board, fuel, livestock and pet bedding, winter road
traction, weed killing and more.”
Jason Fried, 37signals
10. 10
“In contrast to most traditional survey methods,
search data are collected as a by-product of normal
activity”
- the Bank of England
13. 13
Wrapping Up
The business is demanding that you deliver more products to market faster.
response: Agile, DevOps, and Continuous Integration/Deployment
The War for Talent hasn't even started yet - use data for resource allocation and human resources.
Software development is an art but also science – data is a bridge.
Let developers do their jobs and make infrastructure choices, but measure the results.
Watch, don’t ask.
Agile wins, precisely because it is data-driven. Developers thrive on good data.
Break big problems and projects into smaller pieces run by smaller, more responsible, teams.
(At Netflix) the Developer is responsible if they break the build.
With large, legacy code bases, data allows you to find the best likely fix(er).
Use data to reduce organisational disagreements.
Open source is central to good software development engineering practice
Higher quality apps lower support costs.