Jetson and Edge GPU Boards: Power Modes, Toolchains and Deployment

An edge GPU board gives you data-centre-shaped software inside a fixed-wattage box. Two things follow that catch teams out: performance is set by a power mode you choose rather than by the hardware alone, and the software stack is a coupled bundle where you do not get to pick component versions independently.

The power envelope is the design constraint

These modules are specified at a small number of power budgets, and the budget you select determines how many CPU cores are online and how fast the GPU and memory controller are allowed to run. Selecting a lower budget does not throttle you unpredictably — it configures you deliberately, which is much better.

Budget the whole system, not the module:

system_power = module_budget
             + camera(s)
             + storage
             + networking (wifi radios are not free)
             + any carrier-board peripherals

Worked, a module capped at 15 W:

  module                    15.0 W
  MIPI camera                0.8 W
  NVMe SSD, active           3.5 W
  wifi, transmitting         1.5 W
  ---------------------------------
  total draw                20.8 W

A supply rated at 20 W browns out under load — and the symptom
is a board that reboots during inference, not a warning.

Size the supply generously and measure at the wall under a sustained load, not at idle. Intermittent reboots under load are the single most common deployment failure on these boards, and they present as software instability.

Power modes and clocks

Two tools, doing different things, routinely confused:

sudo nvpmodel -q          # which mode am I in?
sudo nvpmodel -m 0        # select a mode (numbering is board-specific)
sudo jetson_clocks        # pin clocks for a reproducible benchmark
sudo jetson_clocks --show # what they are pinned to
tegrastats               # live utilisation, temperature and power

Mode numbering differs between modules, so do not copy a mode number from a tutorial written for a different board. Query the available modes on your own hardware and record which one your benchmark and your production deployment each use. A benchmark taken in one mode and a deployment running in another is a discrepancy that takes days to find.

Why the versions are coupled

On a desktop you install a driver, then a CUDA version, then whatever libraries you want. On these boards the board support package, the kernel, the GPU driver, CUDA, the deep-learning libraries and the inference runtime ship as one versioned bundle. You do not upgrade one component; you move the whole board to a new bundle version.

The practical consequences:

• Container images are tagged to the board’s base version. A container built for a newer base will not necessarily run on an older board, because the GPU driver lives outside the container and is mounted in from the host. “It works in Docker” is not portability here.
• Python wheels are architecture- and stack-specific. The frameworks you install from a public index are usually built for desktop CUDA and will not work. Use the builds published for this platform, matched to your base version.
• Upgrading is a flash, not an update. Plan for it. A fleet on an old base version that cannot run your new model is a logistics problem, not a software one.

Establish what you actually have before debugging anything:

cat /etc/nv_tegra_release              # board support package / L4T version
dpkg -l | grep -i nvidia-l4t-core      # the same, as a package version
nvcc --version                          # CUDA toolkit, if installed
python3 -c "import tensorrt; print(tensorrt.__version__)"
uname -a

Put those five lines in your deployment script and log the output with every build. Half the difficult problems on these boards are a version mismatch that nobody recorded.

Optimised engines are not portable

This is the single most important operational fact about deploying to these boards, and it is the one that ruins release plans.

The high-performance inference path compiles your network into an optimised engine: kernels selected, layers fused, precision chosen, all specialised to a particular GPU architecture, a particular runtime version and often a particular set of input shapes. That engine is a build artefact for one target, not a portable model file. Move it to a board with a different GPU generation or a different runtime version and it will refuse to load.

1. Ship the portable model, build the engine on the target. Distribute an ONNX file — see ONNX as a portability layer — and compile the engine on the device during installation or first run.
2. Expect the build to be slow. Engine construction searches over kernel implementations and can take many minutes on edge hardware. Do it once, in a provisioning step or a background task, never in a request path.
3. Cache the engine, keyed by everything it depends on. Model hash, runtime version, GPU architecture, precision, input shape profile. If any key changes, rebuild. A cache keyed only on the model hash will happily load an engine built for a different runtime and fail confusingly.
4. Verify numerically after building. Precision reduction happens during engine construction, so the engine can be measurably different from the model you exported. Run the same golden-input comparison you would run for any conversion, with a tolerance chosen for the precision you asked for.
5. Keep a fallback path. If engine construction fails on a device — out of memory during the build is common — fall back to a slower generic runtime rather than to a non-functional product.

Deployment realities

• Storage fills up. The stack, the containers, the cached engines and the models add up quickly on the module’s onboard storage. Plan external storage from the start; discovering it mid-deployment means re-flashing devices in the field.
• Thermals depend on the enclosure, not the module. A module with a passive heatsink on a bench behaves nothing like the same module in a sealed weatherproof box on a wall. Characterise sustained throughput in the actual enclosure at the actual ambient temperature — the same sustained-load method as everywhere else in this cluster.
• Headless recovery matters. Devices in the field have no monitor. Ensure serial console access, a watchdog that reboots on hang, and a way to recover from a failed update that does not require a site visit.
• Camera integration is board-specific. Whether a given camera works depends on kernel drivers in the board support package. Confirm the exact camera module against the exact base version before designing around it.
• Clock and time synchronisation. Many of these boards have no battery-backed real-time clock. Timestamps on sensor data will be wrong after a power cycle until the network provides time, and downstream systems that assume monotonic timestamps will behave strangely.

Monitoring in the field

Report a small, fixed set of signals from every deployed board, and choose them so that the common failures are distinguishable from one another:

• Power mode and pinned-clock state. So that a slow board can be diagnosed as misconfigured rather than faulty.
• Sustained inference rate, as a rolling median. The metric that degrades first when thermals or power are wrong.
• Temperature and any throttling indicator. The cause, paired with the symptom above.
• Base version, runtime version, model version, engine cache key. So that a fleet-wide regression can be attributed to a version rather than guessed at.
• Reboot count. A board rebooting under load is a power supply problem and it will otherwise be reported as “the software crashes sometimes”.