The input dimension is fixed at D = 10. The chosen signal sets the intrinsic dimension; noise is added and the model is trained with SGD on N = 50 samples. As the epochs advance, the loss falls and the reconstruction x̂ comes to overlap the input x.
Top: input signal x (solid blue) and reconstruction x̂ (dashed orange) / bottom left: bottleneck z (K values) / bottom right: training step vs loss (MSE). As the epochs advance, x̂ overlaps x and the loss falls.
A 1D linear autoencoder compresses the input $x \in \mathbb{R}^D$ into a low-dimensional latent representation $z$ with the encoder matrix $W_e \in \mathbb{R}^{K \times D}$ and restores the original dimension with the decoder matrix $W_d \in \mathbb{R}^{D \times K}$.
Encoding, decoding and reconstruction error:
$$z = W_e\,x, \qquad \hat{x} = W_d\,z = W_d\,W_e\,x$$ $$L = \lVert x - \hat{x} \rVert^2$$Stochastic gradient descent update (learning rate $\eta$):
$$W_e \leftarrow W_e + 2\eta\,W_d^{\top}(x - \hat{x})\,x^{\top}$$ $$W_d \leftarrow W_d + 2\eta\,(x - \hat{x})\,z^{\top}$$When the optimisation converges, $W_d W_e \approx V V^{\top}$ ($V$ = the top $K$ principal components), so the linear AE becomes a subspace projection equivalent to PCA.