Liming Zhu presented on software engineering for responsible AI. Some key points: 1. There are gaps between principles/standards and engineering practices for building ethical AI systems, and between different teams working in silos. 2. Software engineering can play a connecting role by closing these gaps and operationalizing responsible practices at the system level through approaches like guardrails, observability, and understanding AI systems rather than just building them. 3. As large language models become more common, software engineering for responsible AI will need to adapt, such as through reference architectures for designing systems using foundation models responsibly. The talk discussed directions for software engineering to help close gaps and connect silos in responsible AI development

1. Australia’s National Science Agency
Liming Zhu
Research Director, CSIRO’s Data61
Conjoint Professor, UNSW
Software Engineering
as the Linchpin of
Responsible AI
All pencil drawings in this presentation are created by AI

2. Australia’s National Science Agency
Thank You!
https://research.csiro.au/ss/team/se4ai/
https://research.csiro.au/ss/

3. A personal confession
3 |
learning the most important future skill creating an “intelligent” chatbot
Homo Sapiens
Homo Sentients

4. A professional confession
4 |
I will deliver a keynote at ICSE. The following is my
talk outline. Imagine you are a member of the ICSE
community, could you critique the outline?
Now imagine you are Professor Jon Whittle, an
influential software engineering researcher, could
you critique the outline? Overall, do you like it?

5. What’s Responsible AI?
5 |
Responsible AI is the practice of developing
and using AI systems in a way that provides
benefits to individuals, groups, and wider
society, while minimizing the risk of
negative consequences.
(Non) Functional Requirements
or
Existential Risk

6. What are these requirements?
Classical requirements, but HARD
Model/Algorithm-level challenges
• Alignment
• Inscrutable
• Data drift
• Attribution
…
Australian AI Ethics Principles
• Human, societal & env. wellbeing
• Human-centred values
• Fairness
• Privacy protection and security
• Reliability and safety
• Transparency and explainability
• Contestability
• Accountability
6 |

7. What about the System/SE Level?
7 |
2014-2015 2020-2022
ICSE23 TechDebt Keynote - Technical Debt in AI-based
Software Systems: Challenges and Approaches. Sherry Xu
ICSE23 DeepTest Keynote - Testing Generative Large Language
Model: Mission Impossible or Where Lies the Path? Zhenchang Xing
ICSE23 CHASE Keynote - Humans of AI. Jon Whittle
Trust Debt
Architecture Debt
Explainability Debt
Prompt Controllability/Testability
Modular/Testable AI Chains
Beyond Accuracy

8. Shaw vs Zhu Debate (Aug/22)
8 |
https://www.goodtechthings.com/pile-of-complexity/

9. Shaw vs Zhu Debate (Aug/22)
9 |
intentions -> agents
• data foraging/synthesis
• emerging capabilities
• scalable (AI) oversights
https://medium.com/@itamar_f/software-3-0-the-era-of-intelligent-software-
development-acd3cafe6cd7
https://karpathy.medium.com/software-2-0-a64152b37c35
requirements -> build
examples -> discover
Future directions
• Guardrails
• Radical observability
• Understand rather than build
at the system-level

10. Australia’s National Science Agency
Challenges
&
Trends

11. Australia’s AI ethics framework OECD AI principles
Principles
Standards
Frameworks NIST AI RMF ISO Standards
Algorithms
Models
SE for RAI
……
…
1. The Vertical Gap – Alignment
We have no engineering practices and tools for
building an ethical and trustworthy AI chatbot!
Head of Customer Chatbot from a Major Bank
Model Alignment != System Alignment
Principles/Standards != Eng. Practices
Lu, Q., Luo, Y., Zhu, L., Tang, M., Xu, X., Whittle, J., 2023. Operationalising Responsible AI Using a
Pattern-Oriented Approach: A Case Study on Chatbots in Financial Services. IEEE Intelligent Systems.
11 |

12. 2. The Horizontal Gap - Silos
12 |
The board does not have the bandwidth for
one more risk...
- Executive from a Major Telco
Powell, S., 2023. Risk-based AI Governance, in: Lu, Q., Zhu, L., Whittle, J., Xu, X. , Responsible
AI: Best Practices for Creating Trustworthy System, Addison-Wesley Professional.

13. 13 | Powell, S., 2023. Risk-based AI Governance, in: Lu, Q., Zhu, L., Whittle, J., Xu, X. , Responsible
AI: Best Practices for Creating Trustworthy System, Addison-Wesley Professional.
• Competing for resources
• Software Dev vs Sec vs ML
• Board risk committee: financial, legal, reputation
+ HSE + ethics + ESG + security + privacy + AI + ….
• Risks assessed/mitigated separately
• Asking for Software Engineering’s connecting role
2. The Horizontal Gap - Silos

14. 3. The Understanding Gap - Inscrutable
14 |
90% of our AI solutions can not see the light of
the day as our risk and compliance people do
not understand them….
- Head of AI Lab at a Major Bank
I don’t either.

15. 3. The Understanding Gap - Inscrutable
Do we have to fully understand the AI model?
Can SE-driven system-level understanding help?
15 |

16. One More Thing – Here Come the LLMs
16 |
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. Towards Responsible AI in the Era of ChatGPT: A Reference
Architecture for Designing Foundation Model-based AI Systems. https://arxiv.org/abs/2304.11090

17. Australia’s National Science Agency
Directions
&
Questions

18. 1. Close the Gaps – outward looking
18 |
Lu, Q., Zhu, L., Xu, X., Whittle, J., Xing, Z., 2022. Towards a Roadmap on Software Engineering for
Responsible AI, in: 1st International Conference on AI Engineering (CAIN)

19. 1. Close the Gaps – operationalisable
19 |
Xia, B., Lu, Q., Perera, H., Zhu, L., Xing, Z., Liu, Y., Whittle, J., 2023. Towards Concrete and
Connected AI Risk Assessment (C2AIRA). 2nd International Conference on AI Engineering (CAIN)
Dozens of Frameworks
Which SE/non-SE methods &
tools for which stakeholders?

20. 2. Connect the Silos – Connected Patterns
20 |
Lu, Q., Zhu, L., Xu, X., Whittle, J., 2023. Responsible-AI-by-Design: A Pattern Collection for Designing Responsible
AI Systems. IEEE Software https://research.csiro.au/ss/science/projects/responsible-ai-pattern-catalogue/
Lee, S.U., Perera, H., Xia, B., Liu, Y., Lu, Q., Zhu, L., Salvado, O., Whittle, J., 2023. QB4AIRA: A Question Bank for AI
Risk Assessment. https://doi.org/10.48550/arXiv.2305.09300

21. 21 |
Lu, Q., Zhu, L., Xu, X., Whittle, J., 2023. Responsible-AI-by-Design: A Pattern Collection for Designing Responsible
AI Systems. IEEE Software https://research.csiro.au/ss/science/projects/responsible-ai-pattern-catalogue/
2. Connect the Silos – cross levels

22. 22 |
AI Supply Chain
AI System
AI Operation
Lu, Q., Zhu, L., Xu, X., Whittle, J., 2023. Responsible-AI-by-Design: A Pattern Collection for Designing Responsible
AI Systems. IEEE Software https://research.csiro.au/ss/science/projects/responsible-ai-pattern-catalogue/
2. Connect the Silos – cross systems

23. Identified Risks & SE Mitigations
The Story with the Chatbot Team at a Major Bank
Lu, Q., Luo, Y., Zhu, L., Tang, M., Xu, X., Whittle, J., 2023. Operationalising Responsible AI Using a
Pattern-Oriented Approach: A Case Study on Chatbots in Financial Services. IEEE Intelligent Systems.
23 |
Verifiable ethical requirements/testing
Ethical black box
Human feedback could make things worse
…

24. 3. Understand at the System Level
Increasingly, the study of these trained
(but un-designed) systems seems
destined to become a kind of natural
science…
… they are similar to the grand goals
of biology, which is to "figure out"
while being content to get by without
proofs or guarantees …
“AI as (an Ersatz) Natural Science?”
by Subbarao Kambhampati
24 |
The application of a systematic, disciplined,
quantifiable approach to the development,
operation and maintenance of software; that is,
the application of engineering to software. -IEEE

25. Understanding via “Testing”
Zhuo, T.Y., Huang, Y., Chen, C., Xing, Z., 2023. Exploring AI Ethics of ChatGPT: A
Diagnostic Analysis https://arxiv.org/abs/2301.12867
25 |
ICSE23 DeepTest Keynote - Testing Generative Large Language Model:
Mission Impossible or Where Lies the Path? Zhenchang Xing, CSIRO’s Data61
Capability +/-/⊥ Alignment
Waluigi Effect

26. Understanding via Accountability
26 |

27. Understanding via Accountability
27 |
Xu, X., Wang, C., Wang, Jeff, Lu, Q., Zhu, L., 2022. Dependency tracking for risk
mitigation in machine learning systems, in: 44th ICSE
Xia, B., Bi, T., Xing, Z., Lu, Q., Zhu, L., 2023. An Empirical Study on Software
Bill of Materials: Where We Stand and the Road Ahead, in: 45th ICSE
Software Bills of Materials (SBOM)/AIBOM

28. 4. SE for Foundation Model-based Systems?
28 |
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. A Framework for Designing
Foundation Model based Systems https://arxiv.org/abs/2305.05352v1
LLM eating the traditional SE world
Moving boundaries ex emerging capabilities
Tools being optimized for LLM/Agents

29. Responsible AI for LLM-based Applications
29 |
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. Towards Responsible AI in the Era of ChatGPT: A Reference
Architecture for Designing Foundation Model-based AI Systems. http://arxiv.org/abs/2304.11090

30. Australia’s National Science Agency
Conclusion
&
Reflections

31. SE as the Linchpin of Responsible AI
31 |
Lu, Q., Zhu, L., Xu, X., Whittle, J., Xing, Z., 2022. Towards a Roadmap on Software Engineering for
Responsible AI, in: 1st International Conference on AI Engineering (CAIN)
Responsible AI Engineering
• Close the gaps
• Connect the silos
• Understand at the system level
• SE4AI & RAI in the era of
foundation models

32. Call to Action: Responsible AI Engineering Community
32 |
https://raiengineering.github.io/RAIE/RAIESI/
Global Research Alliance
for Responsible AI

33. Where to Go from Here?
33 |
At the system level
• Guardrails
• Radical observability
• Understand rather than build
Morphing into a science discipline
Study AI-generated systems to understand
• engineered systems
• natural systems
• human intelligence

34. Tool
Partner
Legacy
Different Perspectives
34 |
Unleash
Humanity

35. SE as the Linchpin of Responsible AI
SE4AI Directions
• guardrails
• radical observability
• understand rather than build
SE4AI as a Science Discipline
study AI-generated systems to understand
• engineered systems
• natural systems
• human intelligence
Lu, Q., Zhu, L., Xu, X., Whittle, J., Xing, Z., 2022. Towards a Roadmap
on Software Engineering for Responsible AI, in: 1st International
Conference on AI Engineering (CAIN)
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. Towards Responsible AI in the Era
of ChatGPT: A Reference Architecture for Designing Foundation Model-based AI
Systems. http://arxiv.org/abs/2304.11090
Lu, Q., Zhu, L., Xu, X., Xing, Z., Whittle, J., 2023. A Framework for Designing
Foundation Model based Systems https://arxiv.org/abs/2305.05352v1
Responsible AI Engineering
• Close the vertical gaps
• Connect the silos
• Understand at the system level
• SE4AI & RAI in the era of
foundation models
Zhu, L., Xu, X., Lu, Q., Governatori, G., Whittle, J., 2022. AI and
Ethics—Operationalizing Responsible AI, in: Chen, F., Zhou, J.
(Eds.), Humanity Driven AI. Springer
Twitter: @limingz
https://research.csiro.au/ss/
Shaw, M., Zhu, L., 2022. Can Software Engineering Harness the
Benefits of Advanced AI? IEEE Software 39, 99–104.
Coming out late 2023