The document discusses various levels and methods of potential self-improvement in artificial intelligence (AI) systems, along with associated gains, limitations, risks, and safety considerations. It examines self-improvement approaches at the hardware level, learning level, and code/goal system levels. Specific techniques mentioned include increasing computer resources, different types of learning, rewriting code and architecture, changing goals, and creating new or copied versions of the AI. The document also discusses potential cascades, cycles, and limits to an AI's self-improvement abilities.

1. Levels of
self-improvement
of the AI
Alexey Turchin,
Science for Life extension Foundation

2. What is
self-improvement of
the AI?Roman V. Yampolskiy From Seed AI to Technological Singularity
via Recursively Self-Improving Software. https://arxiv.org/pdf/1502.06512v1.pdf

3. What is
intelligence?

4. Intelligence is a
measure of average
level of performance
Shane Legg, Marcus Hutter. Universal Intelligence: A Definition of Machine Intelligence,
https://arxiv.org/abs/0712.3329

5. Measure can grow but
it can’t increase itself

6. So is
recursive self-improving
magic?

7. Is RSI like nuclear
chain reaction?E.Yudkowsky. Intelligence Explosion Microeconomics. https://intelligence.org/files/IEM.pdf

8. What is going on inside
AI which is trying to make
its performance better?

9. AI has many levels and
changes could happen
on all of them:
• Goal level
• Architecture and code
• Learning and data
• Hardware

10. Hardware level:
acceleration
Increasing of the speed of
computation
Gain: No more than 3-5 times gain on
current elementary base
Limitations: Thermal energy dissipation
Risk: No much risks on early stages
Safety: Low hanging fruit

11. Hardware level:
more computers
Increasing of the speed of
computation
Gain: Logarithmic growth
Limitations: Connection and
pararlelization problems
Risk: Will try to takeover internet
Safety: Boxing, fake resources, low
hanging fruit.

12. Hardware level:
hardware accelerators
Increasing of the speed of
computation
Gain: 100-1000 times
Limitations: 1 month time delay; access
to fabs
Risk: AI needs money and power to get it
Safety: Control over fabs.

13. Hardware level:
Change of the
elementary base
Increasing of the speed of
computation
Gain: 100-1000 times
Limitations: 1 month time delay; access
to fabs
Risk: AI needs money and power to get it
Safety: Control over fabs.

14. Learning level:
Data acquisition
Getting data from outer
sources, like scanning
internet, reading books
Gain: unclear, but large
Limitations: bandwidth of
access to the internet, internal
memory size, long time
Risk: AI could have mistaken
ideas about the world on its
early stages
Safety: Control over
connections.

15. Learning level:
Passive learning
Training of neural nets.
Gain: unclear
Limitations: competitively
extensive and data hungry task.
It may need some labeled data.
Risk: Overfitting or wrong
fitting
Safety: Supervision

16. Learning level:
Active learning with thinking
Creating new rules and
ideas.
Gain: unclear
Limitations: meta-meta
problems
Risk: Testing
Safety: Supervision

17. Learning level:
Active learning with thinking
Acquiring unique important
information
Gain: may be enormous
Limitations: context
dependence.
Risk: Running out of box
Safety: Supervision

18. Learning level:
Active learning with thinking
Experimenting in nature and
Bayesian updates
Gain: may be large
Limitations: context
dependence, slow experiments
in real life
Risk: Running out of box
Safety: Supervision

19. Learning level:
Active learning with thinking
Thought experiments and
simulations.
Gain: may be large
Limitations: long and
computationally expensive, not
good for young AI
Risk:
Safety: Supervision

20. Learning level:
Active learning with thinking
World model changes and
important facts
Gain: may be large
Limitations: long and
computationally expensive, not
good for young AI
Risk: Different interpretation of the
main goal
Safety: Some world model could
make AI safer (if it thinks that it is
in simulation)

21. Learning level:
Active learning with thinking
Value learning. If AI don’t have
fixed goals it could have
intention to continue learn
values from humans.
Limitations: long and
computationally expensive, not
good for young AI
Risk: Different interpretation of the
main goal
Safety: Some world model could
make AI safer (if it thinks that it is
in simulation)

22. Learning level:
Active learning with thinking
Learning to self-improve
Limitations: need for tests, no
previous knowledge
Risk: explosive potential of the AI
Safety: Keep knowledge about AI
away from AI

23. Learning level:
Active learning with thinking
Information about own
structure
Limitations: need for tests, no
previous knowledge
Risk: explosive potential of the AI
Safety: Keep knowledge about AI
away from AI

24. Rewriting its own code
Rewriting of neural
net: choosing right
architecture of the net
for a task
Gain: huge on some
tasks
Limitations: any neural
net has a failure mode
Risk: Look rather benign
Safety: not clear
DeepMind’s PathNet: A Modular Deep Learning Architecture for AGI.
https://medium.com/intuitionmachine/pathnet-a-modular-deep-learning-architecture-for-agi-5302fcf53273#.
48g6wx5i2

25. Rewriting its own code
Optimization and
debugging.
Gain: limited
Limitations: some bugs
are very subtle
Risk: Look rather benign
Safety: insert bugs
artificially?

26. Rewriting its own code
Rewriting of modules and
creating subprograms
Gain: limited
Limitations:
Risk: Look rather benign
Safety:

27. Rewriting its own code
Adding important instrument,
which will have consequences
on all levels.
Gain: may be high
Limitations: testing is needed
Risk:
Safety:

28. Rewriting its own code
Rewriting its own the core
Gain: may be high
Limitations: risks of halting, need
for tests,
Risk: recursive problems
Safety: Encryption, boxing

29. Rewriting its own code
Architectural changes: changes
of relation between all elements of
AI of all level
Gain: may be high
Limitations: risks of halting, need
for tests
Risk: recursive problems
Safety:

30. Rewriting its own code
Unplug of restrictions
Gain: it depends
Limitations: there should be
restrictions
Risk: many dangers
Safety: Second level restriction
which starts if first level is broken;
self-termination code

31. Rewriting its own code
Coding of the new AI from
scratch based on completely
different design
Gain: it depends
Limitations: there should be
restrictions
Risk: many dangers
Safety: Second level restriction
which starts if first level is broken;
self-termination code

32. Rewriting its own code
Acquiring new master algorithm
Gain: large
Limitations: need for testing
Risk: New way of presenting goals
may be needed, Father-child
problem
Safety:

33. Rewriting its own code
Meta-meta level changes. These
are the changes that change AIs
ability to SI, like learning to
learn, but with more
intermediate levels, like
improvement of improvement of
improvement.
Gain: could be extremely large or 0.
Limitations: could never return to
practice
Risk: recursive problems,
complexity
Safety: Philosophical landmines with
recursion

34. Goal system changes
Reward driven learning
Gain: could be extremely large or 0.
Limitations: could never return to
practice
Risk: recursive problems,
complexity
Safety: Philosophical landmines with
recursion

35. Goal system changes
Reward hacking
Gain: could be extremely large or 0.
Limitations: could never return to
practice
Risk: recursive problems,
complexity
Safety: Philosophical landmines with
recursion
Yampolskiy, R.V., Utility Function Security in Artificially Intelligent Agents. Journal of
Experimental and Theoretical Artificial Intelligence (JETAI), 2014: p. 1-17

36. Goal system changes
Changes of instrumental goals
and subgoals
Gain: could be extremely large or 0.
Limitations: could never return to
practice
Risk: recursive problems,
complexity
Safety: Philosophical landmines with
recursion

37. Goal system changes
Changes of the final goal.
Gain: No gain
Limitations: will not want to do it
Risk: could happen randomly, but
irreversably
Safety: Philosophical landmines with
recursion

38. Improving by accusation
non-AI resources
• Money
• Time
• Power over others
• Energy
• Allies
• Controlled territory
• Public image
• Freedom from human and
other limitations, and safety
Stephen M. OMOHUNDRO. The Basic AI Drives
https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf

39. Changing number of AIs
Creating narrow AIs, Tool AIs
and agents with specific goals
Gain: Limited
Limitations: need to control them
Risk: revolt
Safety: Narrow AIs as AI police

40. Changing number of AIs
Creating own copies and
collaborating with them
Gain: Limited
Limitations: need to control them
Risk: revolt
Safety: Narrow AIs as AI police

41. Changing number of AIs
Creating own new version and
its testing
Gain: Large
Limitations: need to control them
Risk: revolt
Safety:

42. Changing number of AIs
Creating orgainsations from
copies
Gain: Large
Limitations: need to control them
Risk: revolt
Safety:

43. Cascades, cycles and
styles of SI
Yudkowsky suggested that during its evolution
different types of SI-activity will be presented in
the some forms, which he called cycles and
cascades.
Cascade is a type self-improvement, where next
version is defined by biggest expected gain in
productivity.
Cycle is a form of cascade there several action
repeated all over again.

44. Styles: evolution and
revolutions
Evolution is smooth, almost linear increase of
the AI capabilities by learning, increasing of
computer resources, upgrading modules, writing
subroutines.

45. Styles: evolution and
revolutions
Revolutions are radical changes of architecture,
goal system, master algorithm. They are crucial
for recursive SI. They are intrinsically risky and
unpredictable, but they produce most of the
capabilities gains.

46. Cycles
Knowledge-hardware cycle of SI is a cycle in
which AI collect knowledge about new hardware
and when build it for itself.

47. Cycles
AI theory knowledge – architectural
changes cycle is primary revolution cycle, and it
is very unpredictable for us. Each architectural
change will give the AI ability to learn more how
to make better AIs.

48. Possible limits and
obstacles in self-
improvement
Theoretical limits to computation

49. Possible limits and
obstacles in self-
improvement
Mathematical nature of complexity of the
problems and definition of intelligence “it
becomes obvious that certain classes of problems
will always remain only approximately solvable
and any improvements in solutions will come
from additional hardware resources not higher
intelligence” [Yampolsky].

50. Possible limits and
obstacles in self-
improvement
Nature of recursive self-improvement
provides diminishing returns of logarithmic
scale, “Mahoney also analyzes complexity of RSI
software and presents a proof demonstrating that
the algorithmic complexity of Pn (the nth iteration
of an RSI program) is not greater than O(log n)
implying a very limited amount of knowledge gain
would be possible in practice despite theoretical
possibility of RSI systems. Yudkowsky also
considers possibility of receiving only logarithmic
returns on cognitive reinvestment: log(n) +
log(log(n)) + … in each recursive cycle.”

51. Possible limits and
obstacles in self-
improvement
No Free Lunch theorems – difficulty to search the
space of all possible minds to find a mind with
superior intelligence to a given mind.

52. Possible limits and
obstacles in self-
improvement
Difficulties connected with Gödel and Lob
theorem, “Lobstacle”: “Löb’s theorem states
that a mathematical system can’t assert its own
soundness without becoming inconsistent.”
“If this sentence is true, then Santa Claus exists."

53. Possible limits and
obstacles in self-
improvement
“Procrastination paradox will also prevent the
system from making modifications to its code
since the system will find itself in a state in which
a change made immediately is as desirable and
likely as the same change made later.”

54. Possible limits and
obstacles in self-
improvement
Paradoxes in logical reasoning with self-
reference, like “This sentence is false.” I call
deliberately created paradox of such type
“philosophical landmines” and they could be a
mean of last hope to control AI.

55. Possible limits and
obstacles in self-
improvement
Yampolsky showed inevitable
wireheading of agents above certain
level of intelligence, that is hacking of own
reward and utility function

56. Possible limits and
obstacles in self-
improvement
Correlation obstacle by Chalmers: “a
possibility that no interesting properties
we would like to amplify will correspond to
ability to design better software.”

57. Pointer problem: If a program starts to
change its code, while running it
simultaneously, it could crash, if it change
the same lines of code there its pointer is
now.
A program can’t run and change it self
simultaneously.
Possible limits and
obstacles in self-
improvement

58. Possible limits and
obstacles in self-
improvement
Father and child problem is in fact a
fight for dominance between AI
generations, and it clearly has many
failure modes.

59. Possible limits and
obstacles in self-
improvement
If AI is a single computer
program, it could halt

60. Converging instrumental goals
in self-improvement of AI
AI Safety problem on each new level:
Avoiding war with new generation

61. Converging instrumental goals
in self-improvement of AI
Need to test new versions for their rea
ability to reliably solve complex problems
better

62. Converging instrumental goals
in self-improvement of AI
Ability to return to previous state

63. Converging instrumental goals
in self-improvement of AI
Preferring evolution to revolutions, and lower
level changes to higher level changes: AI prefers to
reach the same level of optimization power by lower
level changes, that is by evolutionary development, but
not by revolutions

64. Converging instrumental goals
in self-improvement of AI
Revolutions in early
stage of AI and
evolution on later stage
AI will prefer revolutions
only if it will be in very
urgent situation, which will
probably be in the beginning
of its development, when it
has to win over other AI
p r o j e c t s a n d u r g e n t l y
prevent other global risks.

65. Converging instrumental goals
in self-improvement of AI
Military AI as converging goal n early
stages of AI development

66. Converging instrumental goals
in self-improvement of AI
Solving Fermi paradox

67. Converging instrumental goals
in self-improvement of AI
Cooperation with humans of early
stages of its development

68. Converging instrumental goals
in self-improvement of AI
Protecting its own reward function
against wireheading

69. Self-improving
of the net of AIs
• It can’t halt. If one agent halts, other will work.
• It has natural ability to clean bugs (natural selection).
• It is immune to suicide of any single object. Even if all of them will suicide it will not
happen simultaneously and they will be able to create offsprings so the net will continue to
exist.
• There is no pointer problem.
• There is no so strong difference between evolution and revolutions. Revolutionary
changes may be tried by some agents, and if they work, such agents will dominate.
• There is no paperclip maximizers: different agents have different final goals.
• If one agent start to dominate other, the evolution of all system almost stops (the same way
as dictatorship is bad for market economy).

70. Possible interventions in self-
improving process to make it less
dangerous
1. Taking low hanging fruits
2. Explanation of risks to Young AI
3. Initial AI designs that are not able
to quick SI
4. Required level of testing
5. Goal system, which prevent
unlimited SI
6. Control rods and signalization

71. Self-improvement is not necessary
condition for global catastrophic AI
Narrow AI designed to construct
dangerous biological viruses could му even
worse

72. Conclusion: 30 different
levels of self-improvment
Some produce small gains, but some may produce recursive gains.
Conservative estimate: Each level will increase performance 5 times, and
there is no recursive SI.
In that case total SI:
931 322 574 615 478 500 000 = 10 power 21 times
Conclusion: Recursive SI is not necessary to create superinteligence,
even modest SI on many levels is

73. Conclusion:
Medium level self-improvement of
Young AI and its risks
While unlimited self-improvement may meet some conceptual difficulties, first human level AI may
get some medium level self-improvement on approximately low cost, quickly and with low self-
risk.
But combination of this low hanging SI tricks may produce 100-1000 increase in performance
even for the boxed Young AI.
So some types of SI will not be available to the Young AI, as they are risky, take a lot of time or require
external resources.