We present a novel alternative method (other than using compression algorithms) to approximate the algorithmic complexity of a string by calculating its algorithmic probability and applying Chaitin-Levin's coding theorem.
Convolutional Neural Networks and Natural Language ProcessingThomas Delteil
Presentation on Convolutional Neural Networks and their application to Natural Language Processing. In-depth walk-through the Crepe architecture from Xiang Zhang, Junbo Zhao, Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015).
Loosely based on ODSC London 2016 talk: https://www.slideshare.net/MiguelFierro1/deep-learning-for-nlp-67182819
Code: https://github.com/ThomasDelteil/TextClassificationCNNs_MXNet
Demo: https://thomasdelteil.github.io/TextClassificationCNNs_MXNet/
(flattened pdf, no animation, email author for .pptx)
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Preferred Networks
This presentation explains basic ideas of graph neural networks (GNNs) and their common applications. Primary target audiences are students, engineers and researchers who are new to GNNs but interested in using GNNs for their projects. This is a modified version of the course material for a special lecture on Data Science at Nara Institute of Science and Technology (NAIST), given by Preferred Networks researcher Katsuhiko Ishiguro, PhD.
Spelling correction systems for e-commerce platformsAnjan Goswami
This is a presentation on building a scalable machine learned spell correction system for an e-commerce site. However, most of the techniques are also generally applicable for any large consumer site.
Convolutional Neural Networks and Natural Language ProcessingThomas Delteil
Presentation on Convolutional Neural Networks and their application to Natural Language Processing. In-depth walk-through the Crepe architecture from Xiang Zhang, Junbo Zhao, Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015).
Loosely based on ODSC London 2016 talk: https://www.slideshare.net/MiguelFierro1/deep-learning-for-nlp-67182819
Code: https://github.com/ThomasDelteil/TextClassificationCNNs_MXNet
Demo: https://thomasdelteil.github.io/TextClassificationCNNs_MXNet/
(flattened pdf, no animation, email author for .pptx)
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Preferred Networks
This presentation explains basic ideas of graph neural networks (GNNs) and their common applications. Primary target audiences are students, engineers and researchers who are new to GNNs but interested in using GNNs for their projects. This is a modified version of the course material for a special lecture on Data Science at Nara Institute of Science and Technology (NAIST), given by Preferred Networks researcher Katsuhiko Ishiguro, PhD.
Spelling correction systems for e-commerce platformsAnjan Goswami
This is a presentation on building a scalable machine learned spell correction system for an e-commerce site. However, most of the techniques are also generally applicable for any large consumer site.
Parallel Algorithm for Graph Coloring Heman Pathak
The graph coloring problem is an assignment of colors to the vertices such that no two adjacent vertices are assigned the same color. A k-coloring of a graph G is a coloring of G using k colors.
TensorFlow Tutorial | Deep Learning With TensorFlow | TensorFlow Tutorial For...Simplilearn
This presentation on TensorFlow will help you understand what is Deep Learning and it's libraries, why use TensorFlow, what is TensorFlow, how to build a computational graph, programming using elements in TensorFlow, what are Recurrent Neural Networks along with a use case implementation on TensorFlow. TensorFlow is a software library developed by Google for the purposes of conducting machine learning and deep neural network research. In this video, you will learn the fundamentals of TensorFlow concepts, functions and operations required to implement deep learning algorithms and leverage data like never before. Now let's get started in mastering the concept of Deep Learning using TensorFlow.
Below topics are explained in this TensorFlow presentation:
1. What is Deep Learning?
2. Top Deep Learning libraries?
3. Why use TensorFlow?
4. What is TensorFlow?
5. Building a computational graph
6. Programming elements in TensorFlow
7. Introducing Recurrent Neural Networks
8. Use case implementation of RNN using TensorFlow
Why Deep Learning?
It is one of the most popular software platforms used for deep learning and contains powerful tools to help you build and implement artificial neural networks. Advancements in deep learning are being seen in smartphone applications, creating efficiencies in the power grid, driving advancements in healthcare, improving agricultural yields, and helping us find solutions to climate change.
You can gain in-depth knowledge of Deep Learning by taking our Deep Learning certification training course. With Simplilearn’s Deep Learning course, you will prepare for a career as a Deep Learning engineer as you master concepts and techniques including supervised and unsupervised learning, mathematical and heuristic aspects, and hands-on modeling to develop algorithms. Those who complete the course will be able to:
1. Understand the concepts of TensorFlow, its main functions, operations and the execution pipeline
2. Implement deep learning algorithms, understand neural networks and traverse the layers of data abstraction which will empower you to understand data like never before
3. Master and comprehend advanced topics such as convolutional neural networks, recurrent neural networks, training deep networks and high-level interfaces
4. Build deep learning models in TensorFlow and interpret the results
5. Understand the language and fundamental concepts of artificial neural networks
6. Troubleshoot and improve deep learning models
7. Build your own deep learning project
8. Differentiate between machine learning, deep learning and artificial intelligence
Learn more at: https://www.simplilearn.com
[Paper] Multiscale Vision Transformers(MVit)Susang Kim
multiscale feature hierarchies with the transformer model.
a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information visual pathway with a hierarchical model
Slides for a talk about Graph Neural Networks architectures, overview taken from very good paper by Zonghan Wu et al. (https://arxiv.org/pdf/1901.00596.pdf)
Quantum computing description in short. History about quantum computers. Hero's of quantum computers,. introductions abstract what are quantum computers
https://telecombcn-dl.github.io/2018-dlai/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks or Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles of deep learning from both an algorithmic and computational perspectives.
Parallel Algorithm for Graph Coloring Heman Pathak
The graph coloring problem is an assignment of colors to the vertices such that no two adjacent vertices are assigned the same color. A k-coloring of a graph G is a coloring of G using k colors.
TensorFlow Tutorial | Deep Learning With TensorFlow | TensorFlow Tutorial For...Simplilearn
This presentation on TensorFlow will help you understand what is Deep Learning and it's libraries, why use TensorFlow, what is TensorFlow, how to build a computational graph, programming using elements in TensorFlow, what are Recurrent Neural Networks along with a use case implementation on TensorFlow. TensorFlow is a software library developed by Google for the purposes of conducting machine learning and deep neural network research. In this video, you will learn the fundamentals of TensorFlow concepts, functions and operations required to implement deep learning algorithms and leverage data like never before. Now let's get started in mastering the concept of Deep Learning using TensorFlow.
Below topics are explained in this TensorFlow presentation:
1. What is Deep Learning?
2. Top Deep Learning libraries?
3. Why use TensorFlow?
4. What is TensorFlow?
5. Building a computational graph
6. Programming elements in TensorFlow
7. Introducing Recurrent Neural Networks
8. Use case implementation of RNN using TensorFlow
Why Deep Learning?
It is one of the most popular software platforms used for deep learning and contains powerful tools to help you build and implement artificial neural networks. Advancements in deep learning are being seen in smartphone applications, creating efficiencies in the power grid, driving advancements in healthcare, improving agricultural yields, and helping us find solutions to climate change.
You can gain in-depth knowledge of Deep Learning by taking our Deep Learning certification training course. With Simplilearn’s Deep Learning course, you will prepare for a career as a Deep Learning engineer as you master concepts and techniques including supervised and unsupervised learning, mathematical and heuristic aspects, and hands-on modeling to develop algorithms. Those who complete the course will be able to:
1. Understand the concepts of TensorFlow, its main functions, operations and the execution pipeline
2. Implement deep learning algorithms, understand neural networks and traverse the layers of data abstraction which will empower you to understand data like never before
3. Master and comprehend advanced topics such as convolutional neural networks, recurrent neural networks, training deep networks and high-level interfaces
4. Build deep learning models in TensorFlow and interpret the results
5. Understand the language and fundamental concepts of artificial neural networks
6. Troubleshoot and improve deep learning models
7. Build your own deep learning project
8. Differentiate between machine learning, deep learning and artificial intelligence
Learn more at: https://www.simplilearn.com
[Paper] Multiscale Vision Transformers(MVit)Susang Kim
multiscale feature hierarchies with the transformer model.
a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information visual pathway with a hierarchical model
Slides for a talk about Graph Neural Networks architectures, overview taken from very good paper by Zonghan Wu et al. (https://arxiv.org/pdf/1901.00596.pdf)
Quantum computing description in short. History about quantum computers. Hero's of quantum computers,. introductions abstract what are quantum computers
https://telecombcn-dl.github.io/2018-dlai/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks or Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles of deep learning from both an algorithmic and computational perspectives.
An introduction to coding using Python’s on-screen ‘turtle’ that can be commanded with a few simple instructions including forward, backward, left and right. The turtle leaves a trace that can be used to draw geometric figures.
Towards a stable definition of Algorithmic RandomnessHector Zenil
Although information content is invariant up to an additive constant, the range of possible additive constants applicable to programming languages is so large that in practice it plays a major role in the actual evaluation of K(s), the Kolmogorov complexity of a string s. We present a summary of the approach we've developed to overcome the problem by calculating its algorithmic probability and evaluating the algorithmic complexity via the coding theorem, thereby providing a stable framework for Kolmogorov complexity even for short strings. We also show that reasonable formalisms produce reasonable complexity classifications.
This paper reviews algorithmic information theory, which is an attempt to apply information-theoretic and probabilistic
ideas to recursive function theory. Typical concerns in this approach are, for example, the number of bits of information required to
specify an algorithm, or the probability that a program whose bits are chosen by coin flipping produces a given output. During the past
few years the definitions of algorithmic information theory have been reformulated. The basic features of the new formalism are presented
here and certain results of R. M. Solovay are reported.
Time series anomaly discovery with grammar-based compressionPavel Senin
We propose two algorithms that use grammar induction to aid anomaly detection without any prior knowledge. Our algorithm discretizes continuous time series values into symbolic form, infers a context free grammar, and exploits its hierarchical structure to effectively and efficiently discover algorithmic irregularities that we relate to anomalies. The approach taken is based on the general principle of Kolmogorov complexity where the randomness in a sequence is a function of its algorithmic incompressibility. Since a grammar induction process naturally compresses the input sequence by learning regularities and encoding them compactly with grammar rules, the algorithm's inability to compress a subsequence indicates its Kolmogorov (algorithmic) randomness and correspondence to an anomaly.
An Optimal Iterative Algorithm for Extracting MUCs in a Black-box Constraint ...Philippe Laborie
We present a non-intrusive iterative algorithm for extracting Minimal Unsatisfiable Cores in black-box constraint networks. The problem can be generalized as the one of finding a minimal subset satisfying an upward-closed property P. If performance is measured as the number of infeasibility property checks, we show that the proposed algorithm, ADEL, is optimal both for small and for large MUCs and that it consistently outperforms existing approaches in between those two extremal cases.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
ON THE DUALITY FEATURE OF P-CLASS PROBLEMS AND NP COMPLETE PROBLEMScscpconf
In term of computational complexity, P-class (abbreviated as P) problems are polynomial-time
solvable by deterministic Turing machine while NP complete (abbreviated as NPC) problems
are polynomial-time solvable by nondeterministic Turing machine. P and NPC problems are
regularly treated in different classes. Determining whether or not it is possible to solve NPC
problems quickly, is one of the principal unsolved problems in computer science today. In this
paper, a new perspective is provided: both P problems and NPC problems have the duality
feature in terms of computational complexity of asymptotic efficiency of algorithms.
Basic Computer Engineering Unit II as per RGPV SyllabusNANDINI SHARMA
Algorithm, Flowchart, Categories of Programming Languages, OOPs vs POP, concepts of OOPs, Inheritance, C++ Programming, How to write C++ program as a beginner, Array, Structure, etc
Information Content of Complex NetworksHector Zenil
This short talk given in Stockholm, Sweden, explains how algorithmic complexity measures, notably Kolmogorov complexity approximated both by lossless compression algorithms and the Block Decomposition Method (BDM) are capable of characterizing graphs and networks by some of their group-theoretic and topological properties, notably graph automorphism group size and clustering coefficients of complex networks. The method distinguished between models of networks such as regular, random, small-world and scale-free.
A new RSA public key encryption scheme with chaotic maps IJECEIAES
Public key cryptography has received great attention in the field of information exchange through insecure channels. In this paper, we combine the Dependent-RSA (DRSA) and chaotic maps (CM) to get a new secure cryptosystem, which depends on both integer factorization and chaotic maps discrete logarithm (CMDL). Using this new system, the scammer has to go through two levels of reverse engineering, concurrently, so as to perform the recovery of original text from the cipher-text has been received. Thus, this new system is supposed to be more sophisticated and more secure than other systems. We prove that our new cryptosystem does not increase the overhead in performing the encryption process or the decryption process considering that it requires minimum operations in both. We show that this new cryptosystem is more efficient in terms of performance compared with other encryption systems, which makes it more suitable for nodes with limited computational ability.
Introduction to complexity theory that solves your assignment problem it contains about complexity class,deterministic class,big- O notation ,proof by mathematical induction, L-Space ,N-Space and characteristics functions of set and so on
Application of Fuzzy Algebra in Coding TheoryIJERA Editor
Fuzziness means different things depending upon the domain of application and the way it is measured. By
means of fuzzy sets, vague notions can be described mathematically now a vigorous area of research with
manifold applications. It should be mentioned that there are natural ways (not necessarily trivial) to fuzzily
various mathematical structures such as topological spaces, algebraic structure etc. The notion of L-fuzzy sets
later more generalizations were also made using various membership sets and operations. In this section we let F
denote the field of integers module 2, we define a fuzzy code as a fuzzy subset of Fn where Fn = {(a1, ....an | a i
F, i = 1, ...n} and n is a fixed arbitrary positive integers we recall that Fn is a vector space over F. We give an
analysis of the Hamming distance between two fuzzy code words and the error – correcting capability of a code
in terms of its corresponding fuzzy codes. The results appearing in the first part of this section are from [17].
Application of Fuzzy Algebra in Coding TheoryIJERA Editor
Fuzziness means different things depending upon the domain of application and the way it is measured. By
means of fuzzy sets, vague notions can be described mathematically now a vigorous area of research with
manifold applications. It should be mentioned that there are natural ways (not necessarily trivial) to fuzzily
various mathematical structures such as topological spaces, algebraic structure etc. The notion of L-fuzzy sets
later more generalizations were also made using various membership sets and operations. In this section we let F
denote the field of integers module 2, we define a fuzzy code as a fuzzy subset of Fn where F
n
= {(a1, ....an | a i
F, i = 1, ...n} and n is a fixed arbitrary positive integers we recall that Fn
is a vector space over F. We give an
analysis of the Hamming distance between two fuzzy code words and the error – correcting capability of a code
in terms of its corresponding fuzzy codes. The results appearing in the first part of this section are from [17].
Similar to A Numerical Method for the Evaluation of Kolmogorov Complexity, An alternative to lossless compression algorithms (20)
Francesca Gottschalk - How can education support child empowerment.pptxEduSkills OECD
Francesca Gottschalk from the OECD’s Centre for Educational Research and Innovation presents at the Ask an Expert Webinar: How can education support child empowerment?
Model Attribute Check Company Auto PropertyCeline George
In Odoo, the multi-company feature allows you to manage multiple companies within a single Odoo database instance. Each company can have its own configurations while still sharing common resources such as products, customers, and suppliers.
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
Biological screening of herbal drugs: Introduction and Need for
Phyto-Pharmacological Screening, New Strategies for evaluating
Natural Products, In vitro evaluation techniques for Antioxidants, Antimicrobial and Anticancer drugs. In vivo evaluation techniques
for Anti-inflammatory, Antiulcer, Anticancer, Wound healing, Antidiabetic, Hepatoprotective, Cardio protective, Diuretics and
Antifertility, Toxicity studies as per OECD guidelines
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdfTechSoup
In this webinar you will learn how your organization can access TechSoup's wide variety of product discount and donation programs. From hardware to software, we'll give you a tour of the tools available to help your nonprofit with productivity, collaboration, financial management, donor tracking, security, and more.
A Numerical Method for the Evaluation of Kolmogorov Complexity, An alternative to lossless compression algorithms
1. A Numerical Method for the Evaluation
of Kolmogorov Complexity
Hector Zenil
Amphith´ˆtre Alan M. Turing
ea
Laboratoire d’Informatique Fondamentale de Lille
(UMR CNRS 8022)
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 1 / 39
2. Foundational Axis
As pointed out by Greg Chaitin (thesis report of H. Zenil):
The theory of algorithmic complexity is of course now widely
accepted, but was initially rejected by many because of the fact
that algorithmic complexity is on the one hand uncomputable
and on the other hand dependable on the choice of universal
Turing machine.
This last drawback is specially restrictive for real world applications
because the dependency is specially true for short strings, and a solution
to this problem is at the core of this work.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 2 / 39
3. Foundational Axis (cont.)
The foundational departure point of the thesis is based in a rather but
apparent contradiction, pointed out by Greg Chaitin (same thesis report):
... the fact that algorithmic complexity is extremely, dare I say
violently, uncomputable, but nevertheless often irresistible to
apply ...
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 3 / 39
4. Algorithmic Complexity
Foundational Notion
A string is random if it is hard to describe.
A string is not random if it is easy to describe.
Main Idea
The theory of computation replaces descriptions with programs. It
constitutes the framework of algorithmic complexity:
description ⇐⇒ computer program
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 4 / 39
5. Algorithmic Complexity (cont.)
Definition
[Kolmogorov(1965), Chaitin(1966)]
K (s) = min{|p|, U(p) = s}
The algorithmic complexity K (s) of a string s is the length of the shortest
program p that produces s running on a universal Turing machine U.
The formula conveys the following idea: a string with low algorithmic
complexity is highly compressible, as the information that it contains can
be encoded in a program much shorter in length than the length of the
string itself.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 5 / 39
6. Algorithmic Randomness
Example
The string 010101010101010101 has low algorithmic complexity because it
can be described as 18 times 01, and no matter how long it grows in
length, if the pattern repeats the description (k times 01) increases only
by about log (k), remaining much shorter than the length of the string.
Example
The string 010010110110001010 has high algorithmic complexity because
it doesn’t seem to allow a (much) shorter description other than the string
itself, so a shorter description may not exist.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 6 / 39
7. Example of an evaluation of K
The string 01010101...01 can be produced by the following program:
Program A:
1: n:= 0
2: Print n
3: n:= n+1 mod 2
4: Goto 2
The length of A (in bits) is an upper bound of K (010101...01).
Connections to predictability: The program A trivially allows a shortcut to
the value of an arbitrary digit through the following function f(n):
if n = 2m then f (n) = 1, f (n) = 0 otherwise.
Predictability characterization (Shnorr) [Downey(2010)]
simple ⇐⇒ predictable
random ⇐⇒ unpredictable
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 7 / 39
8. Noncomputability of K
The main drawback of K is that it is not computable and thus can only be
approximated in practice.
Important
No algorithm can tell whether a program p generating s is the shortest
(due to the undecidability of the halting problem of Turing machines).
No absolute notion of randomness
It is impossible to prove that a program p generating s is the shortest
possible, also implying that if a program is about the length of the original
string one cannot tell whether there is a shorter program producing s.
Hence, there is no way to declare a string to be truly algorithmic random.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 8 / 39
9. Structure vs. randomness
Formal notion of structure
One can exhibit, however, a short program generating s (much) shorter
than s itself. So even though one cannot tell whether a string is random
one can declare s not random if a program generating s is (much) shorter
than the length of s.
As a result, one can only find upper bounds of K and s cannot be more
complex than the length of that shortest known program producing s.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 9 / 39
10. Most strings have maximal algorithmic complexity
Even if one cannot tell when a string is truly random it is known most
strings cannot have much shorter generating programs by a simple
combinatoric argument:
There are exactly 2n bit strings of length n,
But there are only 20 + 21 + 22 + . . . + 2(n−1) = 2n − 1 bit strings of
fewer bits. (in fact there is one that cannot be compressed even by a
single bit)
Hence, there are considerably less short programs than long programs.
Basic notion
One can’t pair-up all n-length strings with programs of much shorter length
(there simply aren’t enough short programs to encode all longer strings).
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 10 / 39
11. The choice of U matters
A major criticism brought forward against K is its dependence of universal
Turing machine U. From the definition:
K (s) = min{|p|, U(p) = s}
It may turn out that:
KU1 (s) = KU2 (s) when evaluated respectively using U1 and U2 .
Basic notion
This dependency is particularly troubling for short strings, shorter than for
example the length of the universal Turing machine on which K of the
string is evaluated (typically in the order of hundreds of bits as originally
suggested by Kolmogorov himself).
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 11 / 39
12. The Invariance theorem
A theorem guarantees that in the long term different algorithmic
complexity evaluations will converge to the same values as the length of
the strings grow.
Theorem
Invariance theorem If U1 and U2 are two (universal) Turing machines and
KU1 (s) and KU2 (s) the algorithmic complexity of a binary string s when
U1 or U2 are used respectively, there exists a constant c such that for all
binary string s:
|KU1 (s) − KU2 (s)| < c
(think of a compiler between 2 programming languages)
Yet, the additive constant can be arbitrarily large, making unstable (if not
impossible) to evaluate K (s) for short strings.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 12 / 39
13. Theoretical holes
1 Finding a stable framework for calculating the complexity of short
strings (one wants to have short strings like 000...0 to be always
among the less algorithmic random despite any choice of machine.
2 Pathological cases: Theory says that a single bit has maximal random
complexity because the greatest possible compression is evidently the
bit itself (paradoxically it is the only finite string for which one can be
sure it cannot be compressed further), yet one would intuitively say
that a single bit is among the simplest strings.
We try to fill these holes by introducing the concept of algorithmic
probability as an alternative evaluation tool for calculating K (s).
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 13 / 39
14. Algorithmic Probability
There is a measure that describes the expected output of a random
program running on a universal Turing machine.
Definition
[Levin(1977)]
m(s) = Σp:U(p)=s 1/2|p| i.e. the sum over all the programs for which U (a
prefix free universal Turing machine) with p outputs the string s and halts.
m is traditionally called Levin’s semi-measure, Solomonof-Levin’s
semi-measure or the Universal distribution [Kirchherr and Li(1997)].
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 14 / 39
15. The motivation for Solomonoff-Levin’s m(s)
Borel’s typewriting monkey metaphor1 is useful to explain the intuition
behind m(s):
If you were going to produce the digits of a mathematical constant like π
by throwing digits at random, you would have to produce every digit of its
infinite irrational decimal expansion.
If you place a monkey on a typewriter (with say a 50 keys typewriter), the
probability of the monkey typing an initial segment of 2400 digits of π by
chance is (1/502400 ).
1´
Emile Borel (1913) “M´canique Statistique et Irr´versibilit´” and (1914)
e e e
“Le hasard”.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 15 / 39
16. The motivation for Solomonoff-Levin’s m(s) (cont.)
But if instead, the monkey is placed on a computer, the chances of
producing a program generating the digits of π are of only 1/50158
because it would take the monkey only 158 characters to produce the first
2400 digits of π using, for example, this C language code:
int a = 10000, b, c = 8400, d, e, f[8401], g; main(){for(; b-c; )
f[b + +] = a/5; for(; d = 0, g = c ∗ 2; c- = 14, printf(“%.4d”, e + d/a),
e = d%a)for(b = c; d+ = f[b] ∗ a, f[b] = d%–g, d/ = g–, –b; d∗ = b);
Implementations in any programming language, of any of the many known
formulae of π are shorter than the expansions of π and have therefore
greater chances to be produced by chance than producing the digits of π
one by one.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 16 / 39
17. More formally said
Randomly picking a binary string s of length k among all (uniformly
distributed) strings of the same length has probability 1/2k .
But the probability to find a binary program p producing s (upon halting),
among binary programs running on a Turing machine U is at least 1/2|p|
such that U(p) = s (we know that such a program exists because U is a
universal Turing machine)
Because |p| ≤ k (e.g. the example for π described before), a string s with
a short generating program will have greater chances to have been
produced by p rather than by writing down all k bits of s one by one.
The less random a string the more likely to be produced by a short
program.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 17 / 39
18. Towards a semi-measure
However, there is an infinite number of programs producing s, so the
probability of picking a program producing s among all possible programs
is ΣU(p)=s 1/2|p| , the sum of all the programs producing s running on the
universal Turing machine U.
Nevertheless, for a measure to be a probability measure, the sum of all
possible events should add up 1. So ΣU(p)=s 1/2|p| cannot be a probability
measure given that there is an infinite number of programs contributing to
the overall sum. For example, the following two programs 1 and 2 produce
the string 0.
1: Print 0
and:
1: Print 0
2: Print 1
3: Erase the previous 1
and there are (countably) infinitely many more.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 18 / 39
19. Towards a semi-measure (cont.)
So for m(s) to be a probability measure, the universal Turing machine U
has to be a prefix-free Turing machine, that is a machine that does not
accept as a valid program one that has another valid program in its
beginning, e.g. program 2 starts with program 1, so if program 1 is a valid
program then program 2 cannot be a valid one.
The set of valid programs is said to form a prefix-free set, that is no
element is a prefix of any other, a property necessary to keep
0 < m(s) < 1. For more details see (Kraft’s inequality [Calude(2002)]).
However, some programs halt or some others don’t (actually, most do not
halt), so one can only run U and see what programs produce s
contributing to the sum. It is said then, that m(s) is semi-computable
from below, and therefore is considered a probability semi-measure (as
opposed to a full measure).
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 19 / 39
20. Some properties of m(s)
Solomonoff and Levin proved that, in absence of any other information,
m(s) dominates any other semi-measure and is, therefore, optimal in this
sense (hence also its universal adjective).
On the other hand, the greatest contributor in the summation of programs
ΣU(p)=s 1/2|p| is the shortest program p, given that it is when the
denominator 2|p| reaches its smallest value and therefore 1/2|p| its greatest
value. The shortest program p producing s is nothing but K (s), the
algorithmic complexity of s.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 20 / 39
21. The coding theorem
The greatest contributor in the summation of programs ΣU(p)=s 1/2|p| is
the shortest program p, given that it is when the denominator 2|p| reaches
its smallest value and therefore 1/2|p| its greatest value. The shortest
program p producing s is nothing but K (s), the algorithmic complexity of
s. The coding theorem [Levin(1977), Calude(2002)] describes this
connection between m(s) and K (s):
Theorem
K (s) = −log2 (m(s)) + c
Notice that the coding theorem reintroduces an additive constant! One
may not get rid of it, but the choices related to m(s) are much less
arbitrary than picking a universal Turing machine directly for K (s).
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 21 / 39
22. An additive constant in exchange for a massive
computation
The trade-off this is, however, that the calculation of m(s) requires an
extraordinary power of computation.
As pointed out by J.-P. Delahaye concerning our method (Pour La
Science, No. 405 July 2011 issue):
Comme les dur´es ou les longueurs tr`s petites, les faibles
e e
complexit´s sont d´licates ` ´valuer. Paradoxalement, les
e e ae
m´thodes d’´valuation demandent des calculs colossaux.
e e
The first description of our approach was published in Greg Chaitin’s
festchrift volume for his 60th. anniversary: J-P. Delahaye & H. Zenil,
“On the Kolmogorov-Chaitin complexity for short sequences,” Randomness and
Complexity: From Leibniz to Chaitin, edited by C.S. Calude, World Scientific,
2007.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 22 / 39
23. Calculating an experimental m
Main idea
To evaluate K (s) one can calculate m(s). m(s) is more stable than K (s)
because one makes less arbitrary choices on a Turing machine U.
Definition
D(n) = the function that assigns to every finite binary string s the
quotient:
(# of times that a machine (n,2) produces s) / (# of machines in (n,2)).
D(n) is the probability distribution of the strings produced by all n-state
2-symbol Turing machines (denoted by (n,2)).
Examples for n = 1, n = 2 (normalized by the # of machines that
halt)
D(1) = 0 → 0.5; 1 → 0.5
D(2) = 0 → 0.328; 1 → 0.328; 00 → .0834 . . .
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 23 / 39
24. Calculating an experimental m (cont.)
Definition
[T. Rad´(1962)]
o
A busy beaver is a n-state, 2-color Turing machine which writes a
maximum number of 1s before halting or performs a maximum number of
steps when started on an initially blank tape before halting.
Given that the Busy Beaver function values are known for n-state 2-symbol
Turing machines for n = 2, 3, 4 we could compute D(n) for n = 2, 3, 4.
We ran all 22 039 921 152 two-way tape Turing machines starting with a
tape filled with 0s and 1s in order to calculate D(4)2
Theorem
D(n) is noncomputable (by reduction to Rado’s Busy Beaver problem).
2
A 9-day calculation on a single 2.26 Core Duo Intel CPU.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 24 / 39
25. Complexity Tables
Table: The 22 bit-strings in D(2) from 6 088 (2,2)-Turing machines that halt.
[Delahaye and Zenil(2011)]
0 → .328 010 → .00065
1 → .328 101 → .00065
00 → .0834 111 → .00065
01 → .0834 0000 → .00032
10 → .0834 0010 → .00032
11 → .0834 0100 → .00032
001 → .00098 0110 → .00032
011 → .00098 1001 → .00032
100 → .00098 1011 → .00032
110 → .00098 1101 → .00032
000 → .00065 1111 → .00032
Solving degenerate cases
“0” is the simplest string (together with “1”) according to D.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 25 / 39
26. Partial D(4) (top strings)
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 26 / 39
27. From a Prior to an Empirical Distribution
We see algorithmic complexity emerging:
1 The classification goes according to our intuition of what complexity
should be.
2 Strings are almost always classified by length except in cases in which
intuition justifies they should not. For ex. even though 0101010 is of
length 7, it came better ranked than some strings of length shorter
than 7. One sees emerging the low random complexity of 010101...
as a simple string.
From m to D
Unlike m, D is an empirical distribution and no longer a prior. D
experimentally confirms the intuition behind Solomonoff and Levin’s
measure.
Full tables are available online: www.algorithmicnature.org
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 27 / 39
28. Miscellaneous facts from D(3) and D(4)
There are 5 970 768 960 machines that halt among the 22 039 921 152
in (4,2). That is a fraction of 0.27 halt.
Among the most random looking group strings from D(4) there are :
0, 00, 000..., 01, 010, 0101, etc.
Among the most random looking strings one can find:
1101010101010101, 1101010100010101, 1010101010101011 and
1010100010101011, each with frequency of 5.4447×10−10 .
As in D(3), where we reported that one string group (0101010 and its
reversion) climbed positions, in D(4) 399 strings climbed to the top
and were not sorted among their length groups.
In D(4) string length was no longer a classification determinant. For
example, between positions 780 and 790, string lengths are: 11, 10,
10, 11, 9, 10, 9, 9, 9, 10 and 9 bits.
D(4) preserves the string order of D(3) except in 17 places out of 128
strings in D(3) ordered from highest to lowest string frequency.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 28 / 39
29. Connecting D back to m
To get m we replaced a uniform distribution of bits composing strings to a
uniform distribution bits composing programs. Imagine that your
(Turing-complete) programming language allows a monkey to produce
rules of Turing machines at random, every time that the monkey types a
valid program it is executed.
At the limit, the monkey (which is just a random source of programs) will
end up covering a sample of the space of all possible Turing machine rules.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 29 / 39
30. Connecting D back to m
On the other hand, D(n) for a fixed n is the result of running all n-state
2-symbol Turing machines according to an enumeration.
An enumeration is just a thorough sample of the space of all n-state
2-symbol Turing machines each with fixed probability
1/(# of Turing machines in (n,2)) (by definition of enumeration).
D(n) is therefore, a legitimate programmer monkey experiment. The
additional advantage of performing a thorough sample of Turing machines
by following an enumeration is that the order in which the machines are
traversed in the enumeration is irrelevant as long as one covers all the
elements of a (n,2) space.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 30 / 39
31. Connecting D back to m (cont.)
One may ask why shorter programs are favored.
The answer, in analogy to the monkey experiment, is based on the uniform
random distribution of keystrokes: programs cannot be that long without
eventually containing the ending program keystroke. One can still think
that one can impose a different distribution of the program instructions,
for ex. changing the keyboard distribution repeating certain keys.
Choices other than the uniform are more arbitrary than just assuming no
additional information, and therefore a uniform distribution (a keyboard
with two or more letter “a”’s rather than the usual one seems more
arbitrary than having a key per letter).
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 31 / 39
32. Connecting D back to m (cont.)
Every D(n) is a sample of D(n + 1) because (n + 1, 2) contains all
machines in (n, 2). We have empirically tested that strings sorted by
frequency in D(4) preserve the order of D(3) which preserves the order of
D(2), meaning that longer programs do not produce completely different
classifications. One can think of the sequence D(1), D(2), D(3), D(4), . . .
as samples which values are approximations to m.
One may also ask, how can we know whether a monkey provided with a
different programming language would produce a completely different D,
and therefore yet another experimental version of m. That may be the
case, but we have also shown that reasonable programming languages
(e.g. based on cellular automata and Post tag systems) produce
reasonable (correlated) distributions.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 32 / 39
33. Connecting D back to m (cont.)
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 33 / 39
34. m(s) provides a formalization for Occam’s razor
The immediate consequence of algorithmic probability is simple but
powerful (and surprising):
Basic notion
Type-writing monkeys (Borel)
garbage in → garbage out
Programmer monkeys: (Bennett, Chaitin)
garbage in → structure out
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 34 / 39
35. What m(s) may tell us about the physical world?
Basic notion
m(s) tells that it is unlikely that a Rube Goldberg machine produces a
string if the string can be produced by a much simpler process.
Physical hypothesis
m(s) would tell that, if processes in the world are computer-like, it is
unlikely that structures are the result of the computation of a Rube
Goldberg machine. Instead, they would rather be the result of the shortest
programs producing that structures and patterns would follow the
distribution suggested by m(s).
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 35 / 39
36. On the algorithmic nature of the world
Could it be that m(s) tells us how structure in the world has come to be
and how is it distributed all around? Could m(s) reveal the machinery
behind?
What happens in the world is often the result of an ongoing (mechanical)
process (e.g. the Sun rising due to the mechanical celestial dynamics of
the solar system).
Can m(s) tell something about the distribution of patterns in the world?
We decided to see so we got some empirical datasets from the physical
world and made a comparison against data produced by pure computation
that by definition should follow m(s).
The results were published in H. Zenil & J-P. Delahaye, “On the
Algorithmic Nature of the World”, in G. Dodig-Crnkovic and M. Burgin (eds),
Information and Computation, World Scientific, 2010.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 36 / 39
37. On the algorithmic nature of the world
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 37 / 39
38. Conclusions
Our method aimed to show that reasonable choices of formalisms for
evaluating the complexity of short strings through m(s) give consistent
measures of algorithmic complexity.
[Greg Chaitin (w.r.t our method)] ...the dreaded theoretical hole
in the foundations of algorithmic complexity turns out, in
practice, not to be as serious as was previously assumed.
Our method also seems notable in that it is an experimental approach that
comes into the rescue of the apparent holes left by the theory.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 38 / 39
39. Bibliography
C.S. Calude, Information and Randomness: An Algorithmic
Perspective (Texts in Theoretical Computer Science. An EATCS
Series), Springer, 2nd. edition, 2002.
G. J. Chaitin. On the length of programs for computing finite binary
sequences. Journal of the ACM, 13(4):547–569, 1966.
G. Chaitin, Meta Math!, Pantheon, 2005.
R.G. Downey and D. Hirschfeldt, Algorithmic Randomness and
Complexity, Springer Verlag, to appear, 2010.
J.P. Delahaye and H. Zenil, On the Kolmogorov-Chaitin complexity for
short sequences, in Cristian Calude (eds) Complexity and Randomness:
From Leibniz to Chaitin. World Scientific, 2007.
J.P. Delahaye and H. Zenil, Numerical Evaluation of Algorithmic
Complexity for Short Strings: A Glance into the Innermost Structure
of Randomness, arXiv:1101.4795v4 [cs.IT].
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 39 / 39
40. C.S. Calude, M.A. Stay, Most Programs Stop Quickly or Never Halt,
2007.
W. Kirchherr and M. Li, The miraculous universal distribution,
Mathematical Intelligencer , 1997.
A. N. Kolmogorov. Three approaches to the quantitative definition of
information. Problems of Information and Transmission, 1(1):1–7,
1965.
P. Martin-L¨f. The definition of random sequences. Information and
o
Control, 9:602–619, 1966.
L. Levin, On a concrete method of Assigning Complexity Measures,
Doklady Akademii nauk SSSR, vol.18(3), pp. 727-731, 1977.
L. Levin., Universal Search Problems., 9(3):265-266, 1973.
(submitted: 1972, reported in talks: 1971). English translation in:
B.A.Trakhtenbrot. A Survey of Russian Approaches to Perebor
(Brute-force Search) Algorithms. Annals of the History of Computing
6(4):384-400, 1984.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 39 / 39
41. M. Li, P. Vit´nyi, An Introduction to Kolmogorov Complexity and Its
a
Applications,, Springer, 3rd. Revised edition, 2008.
S. Lloyd, Programming the Universe: A Quantum Computer Scientist
Takes On the Cosmos, Knopf Publishing Group, 2006.
T. Rad´, On non-computable functions, Bell System Technical
o
Journal, Vol. 41, No. 3, 1962.
R. J. Solomonoff. A formal theory of inductive inference: Parts 1 and
2. Information and Control, 7:1–22 and 224–254, 1964.
H. Zenil and J.P. Delahaye, On the Algorithmic Nature of the World,
in G. Dodig-Crnkovic and M. Burgin (eds), Information and
Computation, World Scientific, 2010.
S. Wolfram, A New Kind of Science, Wolfram Media, 2002.
Hector Zenil (LIFL) A Numerical Method for the Evaluation of Kolmogorov Complexity 39 / 39