Learning Is Not Just Compression

Thesis

Compression is an important lens on learning because useful models capture regularities rather than memorize every detail. But learning also requires grounding, generalization, feedback, and goals. A shorter description is not automatically a better model, and a predictive model is not automatically a knower.

The compactness intuition

Supporting/contextual references: [learn-solomonoff-1964] [learn-rissanen-1978]

A learner encounters many observations and seeks a structure that makes them less surprising. A short rule can replace a long list: the motion of a planet can be described by a law rather than by every position, and a grammar can summarize many sentences. Minimum description length and related ideas formalize the intuition that a good model captures regularities while leaving noise unmemorized.

Compression is therefore a powerful part of learning. It exposes overfitting, rewards explanatory economy, and connects prediction to representation. But the shortest description depends on a language and a model class. A lookup table can be compact if the codebook is hidden in advance; a simple polynomial can compress training data while making absurd predictions outside its range. Compactness needs a task and a test.

Prediction is not understanding

Supporting/contextual references: [learn-solomonoff-1964] [learn-clark-2013]

A system can predict the next token in a sequence by exploiting correlations without representing the causes a human would cite. A thermostat predicts that heat should switch on below a threshold, but it does not construct a weather theory. A language model can produce a fluent answer while relying on brittle associations. Successful prediction is evidence of learned structure; it does not by itself identify the structure or its meaning.

Understanding is a contested word, so it helps to unpack it. Does it require a causal model, the ability to answer counterfactual questions, grounded reference, flexible transfer, or conscious experience? Different tasks demand different capacities. Compression can support all of them by finding reusable representations, but none follows from compression alone. The distinction protects both scientific modesty and the possibility that machines learn in genuinely new ways.

Generalization tests the model

Supporting/contextual references: [learn-vapnik-1995] [learn-bartlett-2002]

A learner has learned when it can handle relevant cases beyond its training examples. Statistical learning theory studies the conditions under which low empirical error predicts low future error, using assumptions about hypothesis classes, data distributions, and complexity. Cross-validation and held-out tests operationalize part of this discipline. They do not guarantee robustness when the environment shifts or when the test set fails to represent the deployment world.

Compression can conflict with generalization. A model that memorizes exceptions may fit training data perfectly but fail on a new domain. A model that compresses aggressively may erase rare but important events. The right tradeoff depends on costs: a medical screening system should treat unusual signals differently from a recommender optimizing convenience. Learning is not simply the removal of detail; it is the preservation of distinctions that matter for future action.

Causation enters through intervention

Supporting/contextual references: [learn-pearl-2000] [learn-vapnik-1995]

Correlation can compress observations without revealing what would happen if we changed one variable. Ice-cream sales and drowning may rise together because of temperature, not because one causes the other. A learner that can predict under the same conditions may fail when an intervention changes the system. Causal models add assumptions about mechanisms, interventions, and invariances across environments.

Experiments supply a special kind of information because the learner changes a system and observes consequences. Animals learn by acting; children test hidden objects; scientists manipulate variables; robots probe surfaces. This active loop cannot be reduced to passively finding a short code for a fixed dataset. Compression remains useful for representing a causal model, but action supplies the evidence that separates competing explanations.

The body gives a signal meaning

Supporting/contextual references: [learn-clark-2013] [learn-bengio-2013]

Learning in organisms is situated. A smell matters because it guides approach or avoidance; a visual edge matters because it predicts surfaces and affordances; a word matters because a community uses it in coordinated activity. The body supplies needs, sensors, and consequences that turn patterns into relevant information. A detached dataset can contain structure, but relevance arrives through an agent’s position and goals.

Embodied cognition does not imply that silicon systems cannot learn. It implies that the question must name the loops. A robot with sensors, actuators, and changing internal models has opportunities for grounding that a text-only predictor lacks. A software system can acquire social grounding through interaction, but the evidence must show more than statistical fluency. Learning is a relation between a system and an environment, not only a transformation of files.

Compression and representation

Supporting/contextual references: [learn-rissanen-1978] [learn-bengio-2013]

Deep networks often learn internal representations that make a task easier. Early layers may capture local regularities; later layers can separate classes or model long-range dependence. Information bottleneck language describes a tension between retaining predictive features and discarding irrelevant variation. These descriptions can illuminate training, but mutual information in high-dimensional deterministic systems is technically delicate, and no single bottleneck story explains every architecture.

A representation is useful because it supports operations. Two images can be compressed similarly while differing in the feature needed for a safety decision. A word embedding can capture usage patterns while missing reference, social stakes, or a fact that changes with time. Evaluating learning therefore requires behavioral and causal tests, not only a lower loss or shorter code. The model must show what it can preserve and when it breaks.

Open research directions

Supporting/contextual references: [learn-bengio-2013] [learn-clark-2013] [learn-sutskever-2014]

Researchers are studying active learning, causal representation learning, out-of-distribution robustness, and methods that attach uncertainty and provenance to predictions. In artificial intelligence, a central question is whether scalable systems can acquire grounded concepts through interaction rather than only absorb correlations from static corpora. In neuroscience, researchers ask how brains trade compression against metabolic cost, plasticity, and rare-event sensitivity.

These programs have no single settled answer. A compact representation may be necessary for flexible intelligence without being sufficient for meaning or consciousness. A system may learn causal structure without having a human-like self. Better benchmarks should test intervention, transfer, explanation, and calibrated uncertainty instead of rewarding fluent compression alone. What counts as learning remains a scientific design question as well as a philosophical one.

A model that can meet the world

Supporting/contextual references: [learn-pearl-2000] [learn-clark-2013] [learn-sutskever-2014]

The difference becomes visible under intervention. A model that predicts only the observations it has already seen may be a compact summary, while a model that anticipates what changes when an action is taken has captured something closer to causal structure. Learning systems can be compared by how they update after surprise, transfer across environments, and expose uncertainty. These tests do not settle whether a system understands, but they keep that word connected to capacities rather than impressions.

Compression earns its place in the story of learning because it separates regularity from accidental detail. Yet a learner must do more than shorten a record. It must preserve distinctions that support prediction, survive relevant changes, use feedback, and—where the task demands it—connect symbols to causes and consequences. Those capacities can be measured even when the word understanding remains disputed.

The invitation is to ask what has been learned, for whom, and under which intervention. A compact model is not a soul, a proof, or a world. It is a tool that earns trust when its abstractions continue to work beyond the data that made them. Learning begins with compression and becomes knowledge only through contact with consequences.

The practical boundary

Supporting/contextual references: [learn-vapnik-1995] [learn-pearl-2000] [learn-bengio-2013]

A compact model deserves trust only when its success travels beyond training conditions. Intervention, transfer, uncertainty, and failure analysis reveal whether it has found a relation in the world or merely a shortcut in the data. Calibration and adversarial evaluation cannot guarantee robustness, but a portfolio of tests can show whether a capability survives changes in wording, setting, and goal.

Human learning also includes forgetting, attention, social correction, and commitments that are not reducible to a single objective function. A learner is changed by what it treats as relevant, by whom it trusts, and by the consequences of being wrong. Those facts make learning developmental and normative as well as statistical.

Compression is one instrument in a larger epistemic practice: it finds regularity, action tests causes, feedback revises expectations, and communities decide which distinctions matter. The result is not a perfect internal copy of the world but a situated competence that can meet new consequences without collapsing at the first surprise.

Sources & references

Supporting/contextual references, not claim-level proof.

1. Ray Solomonoff — A Formal Theory of Inductive InferenceInformation and Control 7(1), 1–22; 7(2), 224–254, 1964.
2. Jorma Rissanen — Modeling by Shortest Data DescriptionAutomatica 14(5), 465–471, 1978.
3. Vladimir Vapnik — The Nature of Statistical Learning TheorySpringer, 1995.
4. Judea Pearl — Causality: Models, Reasoning, and InferenceCambridge University Press, 2000.
5. Peter L. Bartlett and Shahar Mendelson — Rademacher and Gaussian Complexities: Risk Bounds and Structural ResultsJournal of Machine Learning Research 3, 463–482, 2002.
6. Yoshua Bengio, Aaron Courville, and Pascal Vincent — Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence 35(8), 1798–1828, 2013.
7. Andy Clark — Whatever Next? Predictive Brains, Situated Agents, and the Future of Cognitive ScienceBehavioral and Brain Sciences 36(3), 345–355, 2013.
8. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le — Sequence to Sequence Learning with Neural NetworksAdvances in Neural Information Processing Systems 27, 3104–3112, 2014.