Inference at the Edge: Deploying Vision Models to Manufacturing PLCs

A vision model trained in the cloud runs at 800ms latency on a manufacturing PLC. The line moves a part every 400ms. The model is unusable. Welcome to edge AI deployment — where every assumption from cloud-native ML gets violated.

What makes edge deployment hard

Four constraints that don't exist in cloud inference. Power — most edge devices run on 25-50W envelopes, an A100 needs 400W. Compute — edge accelerators run integer math at 1-10 TOPS; cloud GPUs run float math at 300+ TFLOPS. Memory — edge devices have 2-8GB RAM; cloud has 80GB+. Latency — edge inference needs to be deterministic in milliseconds, not best-effort in hundreds.

These constraints mean the model architecture, the precision, the deployment toolchain, and the runtime all differ from cloud.

The model has to shrink

A YOLOv8x at 68M parameters runs at 60ms on an A100. The same model on a Jetson Orin Nano runs at 400ms. The fix isn't a bigger Jetson — it's a smaller model. YOLOv8s at 11M parameters, or distilled MobileNet variants, deliver 30-50ms inference on edge hardware with accuracy that's often within 2 points of the full-size model on domain-specific tasks.

Pruning, distillation, and architecture search are the techniques. Each is a 1-2 day project for teams that have done it before, a 2-3 week project for teams that haven't.

Quantization is non-optional

FP32 inference on edge is rare. INT8 quantization, with post-training calibration, typically reduces model size by 4x and increases inference speed by 2-3x with 1-3 points of accuracy loss. INT4 is more aggressive and has more accuracy cost; viable on simpler models.

TensorRT, OpenVINO, and ONNX Runtime are the toolchains. Pick based on your target hardware — TensorRT for NVIDIA Jetson, OpenVINO for Intel-based industrial PCs, ONNX Runtime as the more portable fallback.

The runtime matters as much as the model

A well-quantized model deployed through a poor runtime gives up all the gains. The right pattern: model exported through the runtime's optimizer, hardware-specific kernel selection, batching disabled (singleframe latency is what matters), warm-start initialization (avoid cold-start latency in production).

Most edge inference failures we see are not model failures — they're runtime configuration failures.

Connectivity is a separate problem

Manufacturing networks are not cloud networks. Plan for intermittent connectivity, slow uploads, and the requirement to operate offline for hours or days. The model needs to run without phoning home. The audit trail and monitoring data need to buffer locally and sync when connectivity returns.

Update strategy is hard

You'll need to push new model versions to edge devices on a regular cadence. Rolling out via SSH to 40 plants does not scale. OTA update mechanisms — secure, atomic, rollbackable — are required. This is where most edge AI deployments fall over: the model works, the inference works, the update path is broken.

Where integrated platforms help

Intellabel's Manufacturing vertical pack includes an edge deployment SDK, OTA update tooling, and runtime configurations for Jetson, Coral, and industrial PCs. The MLOps tier handles the cloud side — training, optimization, registry — feeding the edge side through versioned deployments. The integration matters because the alternative is stitching SageMaker + custom OTA + custom monitoring, which is multi-month work that most manufacturing AI teams don't have the engineering bandwidth for.

Edge inference at scale is one of the few places where the platform layer determines whether the project ships or doesn't. Get the platform right and the rest is engineering. Get it wrong and the model never leaves the lab.

From Labeling to Structured AI Data Pipelines