Edge AI

Most AI systems assume that data is sent to a cloud server for inference: your photo goes to Google's servers, voice audio goes to Amazon's servers, medical sensor data goes to a hospital's cloud. Edge AI inverts this: inference runs on the device itself, whether that is a phone, a surveillance camera, an industrial sensor, a car's onboard computer, or a microcontroller in a wearable.

The case for edge inference is compelling along several dimensions. Latency: sending data to the cloud and receiving a response takes 50-200ms under good network conditions. Autonomous vehicle perception, industrial robot control, and augmented reality applications require sub-10ms inference - impossible with cloud round-trip latency. Bandwidth: video and sensor data is enormous; streaming it continuously to the cloud is expensive and sometimes impossible (rural IoT deployments, spacecraft, submarines). Privacy: many users and regulators are uncomfortable with audio, video, and biometric data leaving the device. Reliability: cloud-dependent systems fail when connectivity is lost; edge inference continues offline.

Deploying models on edge devices requires overcoming severe resource constraints. Even a flagship smartphone has 8-12GB of RAM and a mobile GPU a fraction as powerful as a data centre GPU. An IoT microcontroller might have 256KB of RAM and no GPU at all. Models designed for cloud inference typically require gigabytes of memory and millions of compute operations per inference - far beyond what edge hardware can accommodate.

Model compression techniques are the toolbox for edge deployment. Quantisation (reducing from FP32 to INT8 or even 4-bit) reduces model size and enables integer arithmetic that runs efficiently on edge hardware. Pruning removes unnecessary weights, reducing computation. Knowledge distillation trains small student models that mimic large teacher models, producing compact models optimised for the target task. Neural architecture search can discover architectures specifically efficient for the target hardware (MobileNet, EfficientNet, ShuffleNet are architectures specifically designed for mobile inference).

Edge ML frameworks support deployment across diverse hardware. TensorFlow Lite (mobile and embedded), ONNX Runtime (cross-platform), Core ML (Apple devices), and SNPE (Snapdragon Neural Processing Engine for Qualcomm) each optimise inference for specific hardware targets. Neural Processing Units (NPUs) - dedicated silicon for ML operations - are now standard in flagship smartphones (Apple Neural Engine, Google Tensor chip, Qualcomm Hexagon), dramatically improving edge inference performance per watt.

TinyML is the extreme end of edge AI: running inference on microcontrollers with kilobytes of memory. TensorFlow Lite Micro and Edge Impulse enable voice keyword detection, gesture recognition, and anomaly detection on hardware like Arduino, STM32, and Nordic nRF chips - enabling ML capabilities in devices that cost a few dollars and run on coin cell batteries for months.