› 1D Linear Autoencoder Simulator Back
Machine Learning Simulator

1D Linear Autoencoder Simulator — Compression Equivalent to PCA

Compress a 10-dim signal through a K-dim bottleneck and decode it linearly. After SGD training a linear AE behaves like PCA on the top K principal components.

Parameters
Latent (bottleneck) dimension K
—
Learning rate η
—
Iterations (epochs)
Input signal type
Noise σ
—

The input dimension is fixed at D = 10. The chosen signal sets the intrinsic dimension; noise is added and the model is trained with SGD on N = 50 samples. As the epochs advance, the loss falls and the reconstruction x̂ comes to overlap the input x.

Results
0
Epoch
—
Reconstruction loss MSE
—
Latent dimension K
—
Compression ratio K/D
Encoding, bottleneck, reconstruction and learning curve

Top: input signal x (solid blue) and reconstruction x̂ (dashed orange) / bottom left: bottleneck z (K values) / bottom right: training step vs loss (MSE). As the epochs advance, x̂ overlaps x and the loss falls.

Theory & Key Formulas

A 1D linear autoencoder compresses the input $x \in \mathbb{R}^D$ into a low-dimensional latent representation $z$ with the encoder matrix $W_e \in \mathbb{R}^{K \times D}$ and restores the original dimension with the decoder matrix $W_d \in \mathbb{R}^{D \times K}$.

Encoding, decoding and reconstruction error:

$$z = W_e\,x, \qquad \hat{x} = W_d\,z = W_d\,W_e\,x$$ $$L = \lVert x - \hat{x} \rVert^2$$

Stochastic gradient descent update (learning rate $\eta$):

$$W_e \leftarrow W_e + 2\eta\,W_d^{\top}(x - \hat{x})\,x^{\top}$$ $$W_d \leftarrow W_d + 2\eta\,(x - \hat{x})\,z^{\top}$$

When the optimisation converges, $W_d W_e \approx V V^{\top}$ ($V$ = the top $K$ principal components), so the linear AE becomes a subspace projection equivalent to PCA.

What Is the 1D Linear Autoencoder Simulator?

🙋
An autoencoder is just a network that outputs its input as it is, right? What is interesting about that?
🎓
Roughly speaking, the trick is a narrow layer in the middle called the "bottleneck". For example, it is trained to compress a 10-dimensional input to just 3 dimensions and then restore the original 10. If the input really had 10 degrees of freedom, restoring it would be impossible, but if the data actually lie on a "thin 3-dimensional space", it can be restored almost perfectly. Press "Retrain" with K = 3 in the simulator and the reconstruction loss MSE should fall to about the size of the noise (about 0.007 for σ = 0.1).
🙋
True! So when I dropped K to 1, the MSE grew a lot (from about 0.007 to about 0.19). Does that mean you cannot squeeze 3-dimensional information into a single number?
🎓
Exactly. In information-theory terms, representing an intrinsically 3-dimensional signal with one number means throwing information away, and the variance in the remaining 2 directions shows up as "reconstruction error". For a linear AE it has been proved that the optimal way to squeeze into K dimensions is the "projection onto the top K principal components". So a linear AE is essentially PCA.
🙋
What, it is the same as PCA? Then why use an autoencoder at all?
🎓
Good question. As long as it stays linear, PCA is indeed enough. But the moment you put non-linear activation functions (ReLU, tanh and so on) in the encoder and decoder, it can compress along a "curved manifold". For example, MNIST handwritten digits lie on a "thin curved surface inside 784 dimensions", which a linear space cannot capture. In practice autoencoders are applied to image compression, anomaly detection, denoising, generative models (VAE) and much more.
🙋
When I raise the noise slider, the MSE keeps growing even though the data are 3-dimensional. What is happening?
🎓
That is the "noise floor". Of the variance in the data, the part due to the intrinsic 3-dimensional components is fixed, and adding noise makes the total variance larger than that. A K = 3 autoencoder captures only the intrinsic components, and the noise remains as reconstruction error. So the reconstruction MSE is the part of the noise that does not lie in the K-dimensional subspace, about $\sigma^2(1-K/D)$ (about 0.007 for σ = 0.1 and K = 3). Put the other way round, this is also how "an autoencoder removes noise automatically": the mechanism of denoising.

Frequently Asked Questions

If the learning rate is too large, gradient descent jumps across the valley of the loss to the other side, and the MSE increases or oscillates. A linear AE is a quadratic minimisation problem, so an extreme η can diverge to NaN. In this simulator, about η = 0.05 is the starting value, and η of 0.3 or more can show a tendency to diverge. If it becomes unstable, return η to the range 0.01–0.05.
SGD is stochastic, so with few iterations it does not get close enough to the optimum. In this simulator, reducing the iterations to 10 shows that the MSE does not fall far enough even with K = 3. Aim for about 500–1000 iterations and check that the learning curve has flattened. A linear AE can also be solved in closed form (SVD), but here it is trained with SGD, which carries over to deep learning.
Not necessarily. The optimum of a linear AE is unique only up to "the subspace spanned by the top K principal components"; there is freedom of rotation within it, because projection onto the same subspace can be achieved by countless (We, Wd) pairs. To extract the principal components themselves, a post-processing step that diagonalises the covariance matrix of We·X after training is needed.
If you make K larger than 3 (for example 5 or 9), the reconstruction MSE falls a little further (about 0.005 for σ = 0.1 and K = 5). But since there is only intrinsically 3-dimensional information plus noise, the decrease is just the extra dimensions learning part of the noise. An excessive bottleneck dimension can lead to "memorising noise" and reduce generalisation to new data. That is one reason to keep K small as a form of regularisation for autoencoders.

Real-World Applications

Anomaly detection:If an autoencoder is trained only on normal data, normal inputs are restored with low reconstruction error while abnormal inputs give large errors. It is a widely used method in situations with few labelled anomalies, such as vibration data from production lines, medical images and network traffic logs. The mechanism of learning a data structure of "intrinsic dimension + noise", as in this simulator, is its foundation.

Image and video compression:Learned image compression with deep autoencoders is being studied as an alternative to JPEG/MPEG. The encoder converts the image into a compact latent representation and the decoder restores it. The linear AE is its starting point and is mathematically equivalent to PCA compression and the eigenface method in face recognition.

Representation learning and feature extraction:Autoencoders learn "meaningful features" from large amounts of unlabelled data, which are then used as pre-training for downstream tasks (classification, regression). Modern self-supervised learning such as BERT and GPT also inherits, in a broad sense, the autoencoder-like idea of "predicting part of the input".

Generative models (the basis of VAEs and diffusion models):The variational autoencoder (VAE) is an extension that introduces a probability distribution in the latent space so that new samples can be generated. Recent diffusion models are also closely related to autoencoder-like ideas in latent space. The linear AE is the most basic foundation of these state-of-the-art models.

Common Misconceptions and Points to Note

The most common misconception isbelieving that "an autoencoder can compress information like magic". In information-theory terms, there is an upper limit on how much information can pass through a K-dimensional bottleneck, and with K below the intrinsic dimension of the data, loss of information is unavoidable. Set K = 1 or K = 2 in this simulator and you will see at once that the reconstruction MSE increases greatly. "Identifying the intrinsic dimension of the data" is the starting point of autoencoder design.

The next most common isthinking that a linear AE and PCA are different things. As mentioned, the optimum of a linear AE trained with MSE loss coincides with projection onto the top K principal components of PCA. So the point of "using a linear AE" lies in its educational value as an entry point to deep, non-linear AEs and in the scalability of SGD to large data. For feeding raw data into a linear AE, SVD is often enough in terms of computational cost.

Finally,thinking that "a reconstruction error approaching zero is always desirable"is also dangerous. Driving the error to zero on noisy data makes the network learn the noise too, breaking generalisation to new data. If you raise K in this simulator while keeping a noise of about σ = 0.5, the error keeps falling while "noise memorisation" takes place. In practice it is essential to choose the bottleneck dimension, learning rate and number of iterations appropriately and to look at the reconstruction error on validation data as well.

How to Use

  1. Set the bottleneck dimension K in the range 1 to D−1. With K = 2 the original signal is compressed to two dimensions
  2. Choose the learning rate (eta) between 0.001 and 0.5 to control how fast SGD training converges. The default 0.05 is stable
  3. Set the number of training iterations between 10 and 2000. 500 iterations are often enough to converge; increase to 2000 when the compression is strong
  4. Adjust the noise standard deviation (sigma) between 0.00 and 1.00 to control the noise level of the input signal
  5. When training finishes, check the reconstruction loss MSE and the compression ratio K/D to judge whether the value of K is appropriate

Worked Example

Compressing the "Smooth (intrinsically 3-D)" signal with input dimension D = 10 (noise σ = 0.10) to K = 3: with η = 0.05 and 500 iterations (10 epochs), the reconstruction loss converges to MSE ≈ 7.4×10⁻³. The compression ratio is K/D = 3/10 = 0.30. Setting K = 1 raises the error floor to MSE ≈ 0.19, confirming that a K smaller than the intrinsic dimension loses information. Because a linear autoencoder learns the same subspace as PCA, choosing K at or above the intrinsic dimension keeps the main signal characteristics in fewer dimensions.

Practical Notes

  1. Preventing overfitting in machine learning: even if the MSE on the training data falls, validate on test data. A K smaller than the intrinsic dimension loses information, while a K that is too large reconstructs the noise as well
  2. SGD convergence instability: too large an η causes oscillation or divergence, while η < 0.001 needs 1000 or more iterations. Lowering the learning rate helps stability when K is large
  3. Signal-processing implementation: if the reconstruction error exceeds the tolerance, increase K or consider combining with other methods such as a Kalman filter. For anomaly detection, check the distribution of reconstruction errors on normal data before setting a threshold