Phase Transition in Artificial Neural Networks

Kumar, Aakash (2025) Phase Transition in Artificial Neural Networks. Masters thesis, Indian Institute of Science Education and Research Kolkata.

Text (MS Dissertation of Aakash Kumar (20MS209)) 20MS209_Thesis_file.pdf - Submitted Version Restricted to Repository staff only Download (1MB)

Abstract

Artificial neural networks have demonstrated remarkable utility across various domains; however, their training often demands significant computational resources. The Lottery Ticket Hypothesis suggests that within every fully connected neural network, there exists a smaller subnetwork that can be trained from scratch to achieve similar performance. This thesis builds on recent work related to the Strong Lottery Ticket Hypothesis (SLTH), an even stronger conjecture, which states that sufficiently overparameterized randomly initialized neural networks contain sparse subnetworks that will perform as well as a small trained network on a given dataset—without any training. This has motivated a considerable amount of research trying to prove that a given smaller network can be approximated by pruning a larger network. While previous studies have tried to answer how large a network needs to be to approximate a given target network through pruning, we go further by investigating both the required network size and the precision of its weights. We assume a target network of a given size, whose weights are represented with a certain precision, and a large a network whose weights are represented with more precision than the target, and explore the relationship that must hold between size and precision of the large network, that must hold in order for the larger network to represent the target network. In one of our results, the required network size is almost exact, mostly free of any arbitrary constants unlike other pervious works. Additionally, we show that the upper bound on the parameter count required to approximate a given network matches the lower bound asymptotically for a one layered network by parameter counting argument, hinting at the optimality of the solution

Actions (login required)

View Item