Opening the 'Black Box' of Neural Networks

In the field of Artificial Intelligence, the neural network (NN) has become an incredibly popular tool of late, achieving impressive accuracy on previously very challenging applications, including object recognition, image classification and prediction in high-dimensional spaces. Systems have been developed which independently learn intuitively reasonable data features, and the methodology has been found to generally excel at processing features in this way. However, very little intuition exists for the workings of NNs, especially in very deep multi-layer applications (Deep Learning); frequently, they are deployed by practitioners as 'black box' function approximators, with little fundamental understanding of their abilities and limitations. This is problematic for two reasons.
Firstly, the performance of an NN depends heavily on the selection of a suitable architecture, with the number of layers, number of perceptron 'units' per layer, unit interconnection structure, use of convolutional filters (and their associated sizes) and use of generalisation aids such as batch normalisation and dropout all impacting the network's test accuracy. Currently, appropriate architectures can only be identified by referencing designs used in other successful systems and evolving these by trial and error. The field would massively benefit from additional intuition for this task, which would permit more rigorous architecture optimisation and possibly the development of more accurate systems.
Secondly, by our non-understanding of how NNs work at a fundamental level, we are similarly blind to why they might break in certain situations. Recent research has considered adversarial examples - carefully perturbed sample inputs which are almost indistinguishable from the original by humans, but cause a classifier NN to confidently assign a completely incorrect classification - and found that many networks are highly susceptible to failing under this kind of attack. There have also been cases of networks exhibiting racial bias in their decisions, presumably learned from training data subject to ever-present human biases. NNs are beginning to be investigated in socially and economically important contexts, where there is potential for substantial harm to be done by an improperly functioning system; it is thus very dangerous to proceed down this route without thoroughly understanding networks' underlying behaviour. Consequently, the development of this kind of knowledge is a strong prerequisite for continuing research on this path.
I propose to investigate these issues by carefully analysing NNs of various scales, deployed in various applications. This could partly comprise a naïve grid search over possible architectures to investigate the resulting accuracy of each, or alternatively the exploration of mathematically provable aspects of the network's computations. One potentially fruitful avenue for immediate exploration is the somewhat paradoxical behaviour of NN weights - particularly in large-scale networks, the trained weight collection is curiously sparse, with relatively few weights differing non-trivially from zero, suggesting large swathes of the network contribute very little to the overall objective. Observing that additional weights increase the dimensionality of the weight-training search space, it seems intuitively reasonable to slim down large networks; indeed, existing work has noted minimal accuracy loss from substantial post-training weight pruning. However, further studies have found these seemingly superfluous weights to be surprisingly essential to the optimiser's ability to find a performant collection of weights, with training markedly inhibited in slimmed-down networks, even where we know a good-quality solution to exist. Understanding the mechanisms behind this phenomenon could provide valuable insight into this aspect of NNs' operation, thus advancing progress towards fuller general understanding of NNs.