AI brain gardening, why pruning can yield sweeter fruit

On the journey to explainable, understandable, unbiased and wholly business-applicable AI, knowing what order machine brains learn in is key to being able to access machine intelligence and so gauge its worth when applied to modern business functions - knowing our sparsification from our quantization is now part of our AI awareness responsibility.


If there is one major challenge associated with the combined fields of Artificial Intelligence (AI), Machine Learning (ML) and Deep Learning (DL), it is perhaps the inconvenient truth that to make these systems fully functional, we humans are still doing a whole lot of learning.

With issues like AI bias clouding our view of what's artificially right or not, things aren't always easy to fathom. With the need for so-called 'explainable AI' now coming to the fore, people want to know how and why our AI engines make the decisions they come to in order for us to be able to trust them.

On the journey to explainable, understandable, unbiased and wholly business-applicable AI, knowing what order machine brains learn in is key to being able to access machine intelligence and so gauge its worth when applied to modern business functions.

On the slightly esoteric end of the AI learning discussion at the moment is sparsification, a concept close to mind for Henrik Nordmark in his role as director of science, data and innovation at data science company Profusion.

Sparsification, for the record, in the case of AI is the process of 'pruning' the size of datasets to reduce them as AI brains are in the process of restructuring, training and optimising in order to make them perform effectively - due to the smaller dataset in use, its application can be extremely well suited to edge Internet of Things (IoT) deployments.

Sparsification (in AI terms) often sits close to its sister discipline of quantization, which in mathematics and digital signal processing is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set.

When AI borrows from nature

"The idea behind sparsification is very interesting. It is essentially borrowed from nature. For example, in humans, the brain starts out with an explosion of synapse formation which can be quickly reconfigured to adapt to whatever children need to learn. However, over time the brain is able to prune itself and just leave the connections that are truly important. Sparsification mimics that process of pruning connections that the neural network doesn't actually need to accomplish the tasks it has learned," said Nordmark.

It seems plausible that sparsification could lead to the development of what the industry likes to call Artificial Neural Networks (ANN), collective masses of AI power that are leaner in terms of the space they occupy and faster to run. It follows that they could be more environmentally friendly - provided that the pruning process itself was not so computationally expensive that it would outweigh the benefits of getting the leaner ANN.

"The idea of combining sparsification with edge computing [for the Internet of Things (IoT)] is also very intriguing because we don't always have the luxury of doing all of our computations in one centralised location. Thus, if we could create very lightweight ANNs that do the clever stuff we want them to do but whose computations could be performed on an edge device like a mobile phone or some IoT device this could be quite powerful," said Nordmark.

This could be especially important in certain sectors such as agriculture. For example, if we created a deep learning ANN to detect whether certain crops have early signs of disease by just analysing images of the plant, it would be useful if that could be done using the camera on your phone and a lightweight app that does the detection.

There may also be interesting applications in terms of enhancing data privacy. For example, Nordmark advises that we think about when applying machine learning to sensitive medical records to help find cures for diseases, we might use an approach called federated learning in which the analysis is not done in one central location but rather is distributed across locations, which could potentially be done more effectively using more lightweight ANNs created using sparsification.

Also here, considering current trends towards remote and disparate workforces, there may be other less obvious applications where combining sparsification and edge computing may solve valuable problems.

Technically possible, but commercially questionable

"What is not so clear is whether there is a strong commercially validated need for sparsification now. There are a lot of great ideas out there that make sense from a technical perspective, however, they don't all become innovations we get to enjoy unless you can hook it up to a business model that works. In this specific instance, cloud computing has brought down very dramatically the cost of both storing data and performing calculations on data," said Nordmark.

Nordmark concludes by saying that, personally, he would be surprised if there were all that many companies that currently felt the need to create a very lean ANN. In fact, in his experience, most companies are only beginning to embrace statistics, machine learning and other mathematical methods for driving value out of their data.

There may of course be some niche scenarios in which creating a lean ANN is exactly what is needed to create commercial value. It is probably reasonable to expect some of these commercially viable niches would exist when combining these lean ANNs with edge computing in which the value being created depends on edge devices like mobile phones, drones or other IoT devices.

To say that machine intelligence is still in its formative years is not unreasonable. It is probably past elementary school level and well into its adolescent teenage scholastic studies. When we get past university degree level and beyond, then perhaps we'll have made the grade.