On-Sensor AI can reduce data transfer, latency, and reliance on external computing by performing AI inference directly on the image sensor. This approach can be useful for machine vision applications where real-time processing, bandwidth, power consumption, or system size are important considerations.
Traditionally, image data is captured by a camera and transferred to a host computer, where AI inference and additional processing are performed. With On-Sensor AI, some of this processing takes place directly on the image sensor.
1. Reducing Data Transfer
High-resolution image sensors generate large amounts of image data. In a conventional architecture, this data is transferred from the camera to a host system for processing.
With On-Sensor AI, inference can take place directly on the sensor. This can reduce the amount of data that needs to be transferred to the host, particularly when the application primarily requires AI results rather than the complete full-resolution image.
LUCID’s Triton Smart, for example, can provide the regular image, the AI input tensor, and the inference output tensor as camera data. The image output can also be reduced to as little as 4 × 4 pixels when appropriate for the application.
2. Reducing Latency
Moving image data to an external processing system introduces additional steps between image capture and inference.
Performing inference directly on the sensor can shorten this processing path. LUCID describes the Triton Smart’s on-sensor architecture as reducing latency and enabling real-time inference directly on the camera.
The actual performance depends on factors such as the AI model and image output resolution.
3. Reducing Dependence on Host Computing
On-Sensor AI can also change the computing requirements of a machine vision system.
The Triton Smart uses Sony’s IMX501 intelligent vision sensor, which incorporates an AI inference engine and 8 MB of on-sensor SRAM. This allows supported AI models to run directly on the sensor rather than requiring inference to be performed by an external CPU or GPU.
This can be particularly relevant when designing systems where computing resources, physical space, or system complexity are important considerations.
4. Power and System Efficiency
Reducing the amount of data transferred and the need for external processing can also affect overall system efficiency.
LUCID positions the Triton Smart as a self-contained edge vision solution designed to reduce system complexity and power requirements. The camera’s typical power consumption is specified as 3.1 W when powered via PoE, or 2.5 W when powered externally.
These figures apply specifically to the Triton Smart and should not be interpreted as a general power advantage for every On-Sensor AI architecture.
5. Local AI Processing
On-Sensor AI also enables AI inference to take place locally, without relying on cloud connectivity.
The Triton Smart performs AI inference locally on the camera. LUCID states that the camera can operate without external processing or cloud connectivity, while its ArenaView MP software can also operate entirely offline and locally.
For applications where local processing is preferred, this can simplify the system architecture.
6. Model Development and Deployment
The hardware architecture is only part of implementing AI in a machine vision system. Developers also need a practical way to train, optimize, convert, and deploy models.
An effective development workflow should support the transition from model training to deployment on the target imaging hardware. This can include tools for model training, quantization, conversion, and deployment, as well as ready-to-use training examples.
Triton Smart: An Example of On-Sensor AI
The Triton Smart from LUCID Vision Labs combines a 12.3 MP Sony IMX501 CMOS sensor with on-sensor AI processing in a compact industrial camera.
The sensor provides a resolution of 4056 × 3040 pixels, while the camera supports AI applications including image classification and object detection. Performance varies according to the AI model and image output resolution. For example, LUCID specifies up to 7.4 FPS at 12.3 MP with object detection and up to 25 FPS when using a 4 × 4 image output. For classification, the corresponding figures are 8.1 FPS and 30 FPS.
Choosing the Right Architecture
On-Sensor AI provides an alternative way to distribute image processing and AI inference within a machine vision system. It can be particularly relevant when latency, bandwidth, power consumption, or external computing requirements are important considerations.
However, the right architecture depends on the application. AI model requirements, image resolution, processing speed, and system integration should all be considered when evaluating whether On-Sensor AI is suitable for a particular machine vision system.
Need help evaluating machine vision and imaging solutions for your application? Prolog Optics can help assess your requirements and identify suitable solutions.