Social Equity and Public Administration
Origins, Developments, and Applications
Introduction: Language Models as Compressors
Introduction: Language Models as Compressors
Nova: Welcome to our podcast on language models as compressors. Today we will explore how large language models can be seen as powerful compressors of text data, and what this implies for the future of AI.
What is a Compressor?
What is a Compressor?
Nova: To start, could you explain the basic idea behind viewing a language model as a compresser?
The Math Behind It
The Math Behind It
Nova: A language model learns the probability distribution over sequences of tokens. Once you have that distribution, you can use it for lossless compression: for a given text, a better predictor yields a shorter arithmetic encoding. So the cross-entropy loss used in training is closely related to the compressed size. Essentially, a good model implies a good compresser.
Perplexity and Bits per Token
Perplexity and Bits per Token
Nova: That makes sense. So the perplexity metric is effectively an exponentiated measure of bits per token?
Quantifying Compression Performance
Quantifying Compression Performance
Nova: Exactly. Perplexity is 2 raised to the power of the cross-entropy loss. For example, if the loss is 3 bits per token, perplexity is 2^3 = 8. So the model is as confused as if it were choosing uniformly among 8 options. This is directly proportional to the compressed size; a lower perplexity means fewer bits needed.
Model Size vs Compression Gain
Model Size vs Compression Gain
Nova: But how does this relate to the model size? A giant model like GPT-4 has billions of parameters. Isn't the model itself huge, so it's not a practical compresser?
The Kolmogorov Complexity Connection
The Kolmogorov Complexity Connection
Nova: Right. In practical terms, you have to account for the decompresser size. If you're compressing a dataset, you need to factor in the model parameters as part of the compressed representation. This is the idea behind the
Approximating Kolmogorov Complexity
Approximating Kolmogorov Complexity
Nova: So large language models can be seen as approximating Kolmogorov complexity?
Program Length and Model Size
Program Length and Model Size
Nova: Yes, in a way. Kolmogorov complexity of a string is the length of the shortest program that outputs it. A language model is a program that can generate text, so when used as a compresser with arithmetic coding, the model plus the compressed data approximate the shortest description. The model itself is the 'program' part.
Compression and Generalization
Compression and Generalization
Nova: This has interesting implications for generalization. If a model can compress well, it must have captured structure.
Minimum Description Length
Minimum Description Length
Nova: Exactly. This is related to the minimum description length principle and Occam's razor. A model that achieves lower compression of the training data relative to its size is likely to have found patterns that generalize. However, we must be careful: a model can also just memorize and overfit, but that would show up as a large model size relative to the compression gain.
Wrap-up: Training as Compression
Wrap-up: Training as Compression
Nova: So in a way, training is just finding a short program that explains the data. Thank you for this discussion.
Conclusion
Nova: Thanks for tuning in. We hope this episode gave you a new lens on language models. Join us next time.