University of Twente Student Theses
Towards understanding and modelling sparse training algorithms at extreme sparsity regime
Engers, Vincent P.G. van (2021) Towards understanding and modelling sparse training algorithms at extreme sparsity regime.
|
PDF
1MB |
| Abstract: | Deep neural networks have proven useful in practical applications. However, many redundant connections unnecessarily inflate network size and computational complexity. Inspired by pruning in biological brains, sparse training methods such as Sparse Evolutionary Training (SET) and Accuracy based Sparse Evolutionary Training (AccSET) prove more efficient than fully connected counterparts by dynamically adding and removing connections. Contrasting to the great amount of research on deep neural networks, little research exists on the effect of varying hyperparameters and structure on the performance and behaviour of sparse neural networks. This paper investigates the influence of varying sparsity levels on the behaviour and performance of Sparse Evolutionary Training and provides new insights related to sparse training over a large sparsity horizon. This paper categorizes different levels of sparsity and defines the extreme sparse contour. It further delivers a systematic analysis of extremely sparse neural networks and a mathematical formulation of the relation between sparsity and accuracy which can estimate with great accuracy the expected accuracy of a model at a given sparsity level. The experiments in this research show that certain sparse neural networks can be trained at extreme sparsity levels. With these results, this paper contributes to the understanding of sparse training in artificial neural networks and underlines the importance of sparse neural networks to the implementation of machine learning in limited computing devices. |
| Item Type: | Essay (Bachelor) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Business & IT BSc (56066) |
| Link to this item: | https://purl.utwente.nl/essays/86887 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page