AI Transformation in highly regulated industry - Challenges and Opportunities

1. Australia’s National Science Agency
AI Transformation
- in highly regulated Industry
Dr/Prof Liming Zhu, Research Director
Expert in working groups
• ISO/IEC JTC 1/WG 13 Trustworthiness
• ISO/IEC JTC 1/SC 42/WG 3 – AI Trustworthiness

2. ”Directional” Combinatorial Innovation
AI/ML Cyber
Human
Distributed
Trust
Trustworthy AI/Data
Ecosystem
Human-Centred
Cybersecurity
Trustworthy/Federated
AI/Agents/Identity
Responsible AI
Sovereign Identity &
Transaction
Trustworthy AI Practices
& Supply Chains
Trusted Collaborative
Intelligence
Trustworthy Digital
Infrastructure

3. Generative AI
•Text
•Image/video
•Code/Scripts
•Data
Predictive
Diagnostic
Concepts Clarified
3 |
Prescriptive
Discriminative
Model
Generative
Model
Hybrid

4. (Generative) AI Transformation - Business
4 |
• AI as Generic Capability/Digital Expert
• human resources or tools/functionality
• Blurring/Moving Division Boundaries
• reverse Conway's Law on org & strategy
• AI Supply Chain
• Where is your AI? Who, How trained?
Risk-based approach & experimentation

5. Australia’s National Science Agency
Challenges

6. Australia’s AI ethics framework OECD AI principles
Principles
Standards
Frameworks NIST AI RMF ISO Standards
Algorithms
Models
Data61’s
SE for RAI
……
…
1. The Vertical Gap – Alignment & Practices
Model Alignment != System Alignment
Principles/Standards != Eng. Practices
Lu, Q., Luo, Y., Zhu, L., Tang, M., Xu, X., Whittle, J., 2023. Operationalising Responsible AI Using a
Pattern-Oriented Approach: A Case Study on Chatbots in Financial Services. IEEE Intelligent Systems.
6 |

7. 2. The Understanding Gap - Inscrutable
Do we have to fully understand the AI model?
Can system-level understanding help?
7 |

8. One More Thing – Here Come the LLMs
8 |
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. Towards Responsible AI in the Era of ChatGPT: A Reference
Architecture for Designing Foundation Model-based AI Systems. https://arxiv.org/abs/2304.11090

9. Australia’s National Science Agency
Directions
&
Questions

10. 1. Close the Gaps – engineering practices
10 |
Lu, Q., Zhu, L., Xu, X., Whittle, J., Xing, Z., 2022. Towards a Roadmap on Software Engineering for
Responsible AI, in: 1st International Conference on AI Engineering (CAIN)
Measurements/Metrics, Evaluation/Verification/Validation Methods

11. Close the Gaps – operationalisable
11 |
Xia, B., Lu, Q., Perera, H., Zhu, L., Xing, Z., Liu, Y., Whittle, J., 2023. Towards Concrete and
Connected AI Risk Assessment (C2AIRA). 2nd International Conference on AI Engineering (CAIN)
Dozens of Frameworks
Which methods & tools
for which stakeholders?

12. Close the Gaps – Connected Patterns
12 |
Lu, Q., Zhu, L., Xu, X., Whittle, J., 2023. Responsible-AI-by-Design: A Pattern Collection for Designing Responsible
AI Systems. IEEE Software https://research.csiro.au/ss/science/projects/responsible-ai-pattern-catalogue/
Lee, S.U., Perera, H., Xia, B., Liu, Y., Lu, Q., Zhu, L., Salvado, O., Whittle, J., 2023. QB4AIRA: A Question Bank for AI
Risk Assessment. https://doi.org/10.48550/arXiv.2305.09300

13. 2. Understand at the System Level
Increasingly, the study of these trained
(but un-designed) systems seems
destined to become a kind of natural
science…
… they are similar to the grand goals
of biology, which is to "figure out"
while being content to get by without
proofs or guarantees …
“AI as (an Ersatz) Natural Science?”
by Subbarao Kambhampati
13 |

14. Understanding via “Testing”
Zhuo, T.Y., Huang, Y., Chen, C., Xing, Z., 2023. Exploring AI Ethics of ChatGPT: A
Diagnostic Analysis https://arxiv.org/abs/2301.12867
14 |
ICSE23 DeepTest Keynote - Testing Generative Large Language Model:
Mission Impossible or Where Lies the Path? Zhenchang Xing, CSIRO’s Data61
Capability +/-/⊥ Alignment
Waluigi Effect prevents
model-level solution

15. Understanding via Accountability
15 |
No Agreed Best Practices
No Agreed Safety Test
Verifiable investment in safety
Accountability enforced by law/market

16. Understanding via Accountability
16 |
Xu, X., Wang, C., Wang, Jeff, Lu, Q., Zhu, L., 2022. Dependency tracking for risk
mitigation in machine learning systems, in: 44th ICSE
Xia, B., Bi, T., Xing, Z., Lu, Q., Zhu, L., 2023. An Empirical Study on Software
Bill of Materials: Where We Stand and the Road Ahead, in: 45th ICSE
Software Bills of Materials (SBOM)/AIBOM

17. 3. Design Foundation Model-based Systems
| Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. A Framework for Designing
Foundation Model based Systems https://arxiv.org/abs/2305.05352v1
LLM eating the traditional system functions
Moving boundaries ex emerging capabilities
• Design with capabilities, not functionalities
• Design for capability evolution and agility
Tools being optimized for LLM/Agents
• Selected/Used by both human and LLM/Agents
• Trusted by human and LLM/Agents

18. (Generative) AI Transformation - Infrastructure
18 |
“Talk to Your Data” --> FM-based Innovation/Operational excellence
“Zero-gradient Infrastructure” (few AI models) --> FM tuning and FMOps
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. A Framework for Designing
Foundation Model based Systems https://arxiv.org/abs/2305.05352v1

19. Responsible AI for LLM-based Applications
19 |
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. Towards Responsible AI in the Era of ChatGPT: A Reference
Architecture for Designing Foundation Model-based AI Systems. http://arxiv.org/abs/2304.11090

20. RAI in the Era of Foundation Models
AI Engineering Directions
• (Learned) Guardrails
• Radical observability
• Understand rather than build
Responsible AI Engineering
• Close the principle-alg. gaps
• Engineering practices/methods
• Measurement/metrics
• Connected patterns
• Understand at the system level
• AIBOM & accountability
More info & Contact
https://research.csiro.au/ss/
Liming.Zhu@data61.csiro.au
Brendan.Omalley@data61.csiro.au
Coming out late 2023
Foundation Models
• Design with capabilities, not func.
• Design for system evolution
• Tools optimised for LLM/Agents
• Special RAI patterns
Collaborate with CSIRO’s Data61 on
• RAI Engineering best practices & evaluation
• LLM/Foundation model-based system design/eval
For the latest, follow me on
Twitter: @limingz
LinkedIn: Liming Zhu