Available at: https://digitalcommons.calpoly.edu/theses/3229
Date of Award
3-2026
Degree Name
MS in Computer Science
Department/Program
Computer Science
College
College of Engineering
Advisor
Paul Anderson
Advisor Department
Computer Science
Advisor College
College of Engineering
Abstract
Modern deep neural networks are heavily overparameterized, with learned representations occupying lower-dimensional subspaces than their ambient dimensions suggest. This thesis investigates whether fixed random Johnson-Lindenstrauss (JL) projections can exploit this redundancy by functioning as architectural bottlenecks that reduce model size and training cost while preserving task performance.
We systematically evaluate JL-based compression across multiple architectures (convolutional networks, multilayer perceptrons, vision transformers, autoregressive language models) and tasks (reconstruction, classification, language modeling). Experiments reveal that compression tolerance depends critically on placement strategy and compression ratio.
Linear probe analysis quantifies intrinsic dimensionality at different network depths, revealing that CIFAR-10 final representations require only 5-10 dimensions despite ambient dimensions of 256-512. Intermediate representations require approximately 50 dimensions to maintain transformation capacity. This 25-100x gap between intrinsic and ambient dimensionality explains why aggressive compression is possible and provides principled targets for layer-specific compression strategies.
The success of completely random, non-learned projections in preserving task performance demonstrates that neural networks learn robust geometric structures where distance preservation is sufficient for representation quality. This work establishes random projections as viable architectural components and opens a design space for efficient neural networks through principled dimensionality reduction.