The last decade has been a time of change in the world of machine learning. A field that was once more heavy on hype than practical applications grew and began to deliver major breakthroughs that revolutionized industrial processes and consumer products alike. But for the field to continue to deliver major breakthroughs in these areas and beyond, more progress will be needed in the tinyML space. Conventional methods of implementing machine learning algorithms – small computing devices that rely on powerful computing resources in the cloud to perform inferences – are limited in their effectiveness due to issues of privacy, latency, and cost. TinyML offers the promise of eliminating these problems and opening new classes of problems to be solved by artificial intelligence algorithms.
Of course running a state-of-the-art machine learning model, with billions of parameters, isn’t exactly easy when memory is measured in kilobytes. But with some creative thinking and a hybrid approach that uses the power of the cloud and mixes it with the benefits of tinyML, it’s just possible. A team of researchers at MIT has shown how this is possible with their system method called Netcast which relies on high-resource cloud computing to rapidly retrieve model weights from memory, then transmits them almost instantaneously to the tinyML hardware via a fiber optic network. Once those weights are transmitted, an optical device called a broadband “Mach-Zehnder” modulator combines the sensor data to make lightning-fast calculations on the spot.
The team’s solution uses a cloud computer with a large amount of memory to store the weights of the full neural network in RAM. Those weights are transmitted to the connected device as needed over an optical pipe with enough bandwidth to transmit an entire feature-length movie in one millisecond. This is one of the major limiting factors that prevent tinyML tools from using large models, but it is not the only one. Processing power is also at the forefront of these devices, so the researchers also proposed a solution to this problem in the form of a box-like receiver that performs very fast analog calculations by including input data in transmitted weights.
This program makes it possible to perform billions of iterations per second on a device that has been resourced like a desktop computer since the early 1990s. In this process, on-device machine learning that ensures privacy, reduces latency, and is more energy efficient is made possible. Netcast has been tested for image classification and digit recognition tasks with more than 50 miles separating the tinyML device and cloud resources. After only a small amount of calibration work, average accuracy rates above 98% were observed. Results of this quality are good enough to be used in commercial products.
Before that happens, the team is working to improve its methods to achieve even better performance. They also want to shrink the shoebox-sized receiver down to the size of a single chip for inclusion in other devices such as smartphones. With further development of Netcast, big things may be on the way for tinyML.