With the broader adoption of digital technologies and AI, organisations face the emerging risks of AI, the unfamiliar, and the intensified risk of cybersecurity, the familiar. AI and cybersecurity are intertwined, but risk silos are often created when they are dealt with at the technology and governance levels. This talk will explore the interactions between responsible AI and cybersecurity risks via industry case studies. It will show how we can break down the risk silos and use emerging trust-enhancing technologies, architecture and end-to-end software engineering/DevOps practices to connect the two worlds and uplift the risk management posture for both.

1. Australia’s National Science Agency
Responsible AI &
Cybersecurity
A tale of two
technology risks
Liming Zhu
Research Director, CSIRO’s Data61
Chair, Blockchain & Distributed Ledger
Technology, Standards Australia
Expert on working groups:
ISO/IEC JTC 1/WG 13 Trustworthiness
ISO/IEC JTC 1/SC 42/WG 3 - Artificial intelligence – Trustworthiness

2. CSIRO’s Data61: Australia’s Largest Data & Digital
Innovation R&D Organisation
1000+
talented people
(including
affiliates/students)
Home of
Australia’s
National AI
Centre
Data61
Generated
18+ Spin-outs
130+ Patent
groups
200+
Gov &
Corporate
partners
Facilities
Mixed-Reality Lab
Robotics Inno. Centre
AI4Cyber HPC Enclave
300+
PhD students
30+
University collaborators
Responsible
Tech/AI
Privacy & RegTech
Engineering & Design of
AI Systems
Resilient &
Recovery Tech
Cybersecurity
Digital Twin
Spark (bushfire) toolkit
2 |

3. § More sources & types from public & partners
§ Intergovernmental data sharing
§ Access and use of sensitive data from another
organization/country
§ Privacy but also commercial and other sensitivity
§ Data analytics over encrypted data -
”sharing/use without access”
§ Open data/innovation (anonymized or
desensitized data)
Trend: Value Arises from Data Sharing & Joint Analytics
Data sharing, Data-as-a-service & AI/ML/Model-as-a-Service
3 |

4. Trend: Regulation/Ethic Overlay
Data Economy: Balancing Innovation & Regulation Burden
Legislations
• GDPR, EU AI Act
• Australia
• AU Privacy Act
• Data Breach Notification Scheme
• Consumer Data Right (CDR): Open Banking, Energy..
Increasing Concerns
• Cybersecurity: Data (increasingly integrity) and AI
• Responsible AI – Trust Data/AI-powered Service
- Fairness, Accountability, Transparency, Privacy, Civil liberties…
- Rights to explanation and redress
- Right to be forgotten
4 |

5. Tech Trend: Trust Architecture- AI and Security
5 |
Systems Operating in the Context of
• Zero Trust Environment
• Trustless Machines/Protocols
• Distributed Trust/Blockchain
• Distributed Infrastructure
• Data, Compute/Code, Models

6. Distributed Trust Architecture in AI Engineering/Systems
6 |
• Entanglements, Correction Cascades,
Undeclared Customers
• Data (Model, Code, Config..) Dependencies
• Anti-patterns
• Debt: Abstraction, Reproducibility, Process
Management, Culture
Circa 2014-15 2020-2021/Today
• ”federated data collection, storage, model,
and infrastructure”
• “co-design and co-versioning”…
• implication of foundation models

7. Distributed Trust in Software Supply Chain
7 |
Circa 2014-15 2020-2021/Today

8. Why we need to manage AI risks
8 |

9. • Human, societal and environmental
wellbeing
• Human-centred values
• Fairness
• Privacy protection and security
• Reliability and safety
• Transparency and explainability
• Contestability
• Accountability
Australian AI Ethics Principles
Security is part of it
9 |

10. • Different stakeholder interests & complex landscape of risk assessment
• Industry level vs. org level vs. team level
Challenge: Diverse stakeholders and risk landscape
10 |

11. • Risk silos competing for resources
• CISO vs. CIO: security team vs. Dev team
• Board risk committees: financial, legal, reputation
– + HSE + privacy + security + ethics + AI + ….
• Limited connections between risks assessed separately
• Forced and meaningless roll-up
• Risk mgt perceived as a barrier – a separate thing dreaded doing
Challenge: Competing risk silos
11 |

12. • Each org has existing and different governance/risk approaches
– Shortage of expertise to assess new risks e.g. AI risks
– No capacity to examine each project deeply
– Checklist, conversations, info sheet
– Not underpinned by formal or technical approaches
• Treating risk analysis as hazard/threat analysis, omitting
– System vulnerability, exposure risks and response/mitigation risk
Challenge: Risk integration and expertise
12 |

13. • Lift the boat - Solutions that benefit multiple risk management, e.g.
– End-to-end provenance across data, code and AI models
– Control intercepts, federated learning, distributed trust
• Connected risks – meaningful technical trade-offs/mitigation, e.g.
– Patterns with multi-risk consequences and trade-offs
• Whole-of-system risks – meaningful aggregation, e.g.
– Connected patterns across process, governance and product
• Integration with existing processes
– Product development processes & governance processes
– Most efficient use of specialised expertise
Solution Principles: lift the boat, connect the risks
13 |

14. Responsible & Secure AI System
Responsible and Secure (AI) Systems
AI ethics principles
Trusted user interaction
Responsible/
Secure-AI-by-
design
AI pipelines
Accountable
DevSecOps
Non-AI components
Responsible/
Secure data
management
Fair &
secure AI
DevOps
AI components
Multi-level governance
Cybersecurity
14 |

15. Process/Governance patterns for connected and
integrated risk assessment
15 |

16. • Connect multiple technical risks when possible
• Focus on mitigations that help address multiple risks first
• Then consider single-risk mitigations
• Mitigation/response introduce overlooked new risks - must assess
Connected Risk Assessment
AI4M Operationalising Responsible AI Project: https://research.csiro.au/ai4m/operationalising-responsible-ai/
16 |

17. Context-Specific Risk Assessment
Operationalising Responsible AI Project: https://research.csiro.au/ai4m/operationalising-responsible-ai/
17 |

18. Pattern template
• Summary
• Type of pattern
• Type of objective
• Target users
• Impacted stakeholders
• Relevant principles
• Context
• Problem
• Solution
• Benefits
• Drawbacks
• Related patterns
• Known uses
https://research.csiro.au/ss/science/projects/responsible-ai-pattern-catalogue/
Pattern Catalogue – extra key info
• In software engineering, a pattern is a
reusable solution to a recurring problem
in a given context
• capture the experience of experts about best
practices
• document in an accessible and structured way
for stakeholders (e.g. developers)
• Pattern catalogue
• a collection of patterns that are related to
some extend
• used together or independently of each other
18 |

19. Pattern Example
https://research.csiro.au/ss/science/projects
/responsible-ai-pattern-catalogue/
19 |

20. Pattern Catalogue – cross aspects
[1] https://research.csiro.au/ss/science/projects/responsible-ai-pattern-catalogue/
20 |

21. Pattern Catalogue – cross levels
21 |

22. Pattern Catalogue – cross life cycle
22 |

23. AI
Ecosystem
Pattern Catalogue – cross supply chain
& system layers
23 |

24. Pattern-oriented
responsible-AI-by-design
reference architecture
24 |

25. Product/Tech patterns for embedding multi-risk
mitigations – lift the boat
25 |

26. Analytics/Simulation to Data: Data Airlock
Not Data to Analytics/Simulation
• Analytics/Simulation requests to
data -> Insights back
• No data sharing
• Automated vetting of insights
• Risks mitigated: security, privacy,
emotional harm, accountability…
• Case Studies: Major government
agency
26 |
Data is kept away in vaults.
All analytics models and simulation results are vetted.

27. Trust Architecture at Scale: Consumer-Driven Sharing
Enabling FinTechs including blockchain-based ones
• Consumer Data Right (CDR): Australia’s legislation impacting
consumer data and its services
• Consumers can authorise 3rd parties to access their data
• Currently designated sectors: Banking, Energy…
• Data61’s (Recent) Role
• Setting Architecture/Data API standards
• Security profiles standards
• Trust Architecture Trade-offs
• Trusted gateway vs. peer-to-peer trust
• Trust in Nodes: Processing-only vs. Processing + Use
• Risks mitigated: security, privacy, over-regulation,
accountability, irresponsible data/analytics
https://consumerdatastandards.gov.au
27 |
ACCC Consumer Data Right in Energy Consultation paper:
data access models for energy data, 2019

28. When there are cultural or legislative restrictions
in place to data sharing, consider alternatives!
Federated Model: “Data Co-Ops”
• No centralised data repositories
• Edge AI and Analytics
Scientific Approaches
• Zero-knowledge proofs, homomorphic
encryption, secure-multi-party computation
• Risks mitigated: security, privacy, accountability,
explainability
Trust Architecture: Federated ML/Data Analytics
From limited access to full encryption during use
28 |
Other Case Studies at Data61
• Bank + Telco for fraud analytics
• Two gov departments for joint insights
Other Supported Scenarios
• Innovation in secure transactions
• Access to data by regulators
• Cross-border data flow

29. Use Cases
- keyboard prediction
- browser history recommendation
- visual object detection
- diagnosis and treatment prediction
- drug discovery (across facilities involving IP)
- meta-analysis over distributed medical databases
- augmented reality
More Data61 case studies
• name entity resolution
• fraud/anomaly detection (bank + telco)
• crop yield prediction - federated transfer learning
• IIoT fault detection
More Federated Learning Architecture & Use Cases
Data61 work: SK Lo, Q Lu, L Zhu, HY Paik, X Xu, C Wang: Architectural patterns for the
design of federated learning systems, Journal of Systems and Software (2021)
Data61 work: SK Lo, Q Lu, HY Paik, L Zhu, FLRA: A Reference Architecture for Federated
Learning Systems, European Conference on Software Architecture (2021)
Data61 work: Wei, K., Li, J., Ding, M., Ma, C., Yang, H.H., Farokhi, F., Jin, S., Quek, T.Q.S., Poor,
H.V., 2020. Federated Learning With Differential Privacy: Algorithms and Performance Analysis.
IEEE Transactions on Information Forensics and Security 15, 3454–3469.
29 |

30. Federated Learning: Trust Architecture and Patterns
30 |

31. Trustworthiness: Model/Data Integrity & Provenance
31 |
Data61 work: X Xu, C. Wang, J. Wang, et. al. “Improving Trustworthiness of AI-
based Dynamic Digital-Physical Parity” , 2021 (submitted)
• Blockchain improves trust in data integrity
and model integrity
• Provenance is the key

32. Trust Architecture Patterns: Privacy-by-Design
32 |
•
Data61 work: Su Yen Chia, Xiwei Xu, Hye-Young Paik, Liming Zhu: Analysing and
extending privacy patterns with architectural context. SAC 2021
GDPR &
Australian Privacy
Principles

33. Safe Data Sharing: Provable Desensitization & Synthetic Data
Quantified risks assessment, mitigation and compliance, synthetic data sets
§ Provably desensitized data sharing/release for joint analytics and simulation
§ Synthetic datasets that balance authenticity and obfustication
§ Quantified risks and mitigation
§ Case Studies: Worked with 30+ Gov agencies
R4: Re-identification Risks Ready-Reckoner
33 |

34. KG-based automated tools to assist humans
34 |

35. • Knowledge Graphs across AI and security risks
• uses a graph-structured data model or topology to integrate data
• Graphically present semantic relationship between entities
• Responsible/Secure AI Knowledge Graph
• Incorporating unstructured data
• AI ethics principles, security standards, policy documents…
• AI and security incidents…
• Pattern catalogues, online solutions…
• Dark pattern datasets…
• Supplemented with GPT
• …
Our Approach: Automated tools assisting human
35 |

36. Knowledge
provenance and
explainability
• Aspect extraction
• rule based (TOSEM 2022)
• Supervised NER+QA (TOSEM revision)
• Unsupervised clustering (ASE 2021)
• Vulnerability KG
• Four heterogeneous sources (NVD,
IBM X-Force, ExploitDB, Openwall)
• Seven vulnerability aspects
• Link to CWE+CAPEC
• Integrate CVSS classifications
• Add aspect synonyms
• A web interface to access the knowledge
• http://vbom.org/#/home
36 |

37. Integrating API knowledge
37 |

38. Integrating user tasks/failures – better test
System KG Construction
Proof-of-Concept Tool Implementation
KG Meta-Model Design
Test Scenario Generation
Bug Reports
Static Part Dynamic Part
Manual Categories
Definition
Manual Action
Definition
Automatic Concept
Extraction
Entity Linking
Configuration files
Step Normalization
Step Splitting
Step Clustering
Scenario Extraction
Soap Opera Test Generation
Relevant Bug Reports Finding Test Scenario Generation
Seed Bug Report Test Scenarios
Static Dynamic
Category
Concept
Action
presentedIn
synonymOf
antonymOf
Step
hasConcept
hasAction
nextStep
actionOn
Expected
Results
Actual
Results
Preconditions
satisfy
leadTo
leadTo
synonymOf Scenario
execute
Cluster
belongTo
actionOn
Constructing a System Knowledge Graph of User Tasks and Failures from Bug Reports to Support Soap Opera
Testing (Su et al., ASE 2022)
38 |

39. KG uses: Multi-faceted Search
39 |

40. KG uses: Dark Pattern- ethical, security and privacy risks
• Dark pattern: a type of user interface designed to trick users into
doing things that they did not mean to do
• disguised ad, preselection, hidden information, trick questions, forced action,
false hierarchy, etc.
40 |

41. KG Uses: Dark Pattern Detector
Knowledge graph +
Natural language processing
Computer vision
Input: a user interface
Output: locate the dark pattern, explain and give examples
• Dark Pattern: Privacy Zuckering
• Description: You are tricked into publicly sharing more information
about yourself than you really intended to.
• Possible Solution: Allow users to disable the permission
• Similar Examples
41 |

42. KG Uses: Supplement AIBOM Generator
• Many organizations procure AI
technologies/solutions from third
parties to build AI systems
• Software Bill of Materials (SBOM):
ensure transparency and security
of software supply chain
• Component name, version, supplier,
dependency relationship, author of
SBOM, timestamp, etc.
• AI/Data BOM
42 |

43. • Despite a struggling tale of two siloed risks: Cybersecurity and AI
• Solution principles
– Lift the boat - solutions that benefit multiple risk management
– Connected risks – meaningful technical trade-offs/mitigation
– Whole-of-system risks – meaningful aggregation.
– Integration with existing processes
• Solutions
– Process/Governance patterns for connected/integrated risk mgt
– Product/Tech patterns for embedding multi-risk mitigations
– KG-based Automated tools to assist humans
For more: https://research.csiro.au/scs/ liming.zhu@data61.csiro.au
Summary: lift the boat, connect the risks
43 |