There is a predictable moment in connected product development where someone says the update mechanism can wait until after the pilot. It is a reasonable-sounding position and it is the most expensive decision available on the project.
What actually goes wrong
OTA is not a feature you bolt on. It constrains the flash layout, the bootloader, the power budget, the key storage and the manufacturing process. Deciding it late means either accepting whatever the existing design allows, or respinning hardware.
The failure we see most often is a device with no room for a second image slot. Without one, an update means erasing the running firmware and hoping the write completes. A brownout halfway through leaves a brick, and the fix is a site visit for every affected unit.
The minimum that belongs in the prototype
- Two image slots, or an external flash part with room for one, decided before the board is laid out.
- A bootloader that verifies a signature before it will run an image.
- Signing keys generated and stored properly from the very first build, so the process is exercised, not theoretical.
- A health check after boot, with automatic rollback if the new image fails it.
- A rollout that can be halted from the console once it is going wrong.
Practise the rollback, not just the update
Every team tests that an update works. Very few test that a rollback works, which is unfortunate, because rollback is the mechanism you will need under pressure. We run at least one deliberate bad-image rollout on every pilot fleet, and it has never once been uneventful the first time.
An update path that has never been used in anger is not an update path. It is an assumption with a version number.
Staged rollout is a product decision
Once the mechanism exists, the interesting question is policy. Which units are the canary cohort, how long do you wait, what failure rate halts the campaign automatically. Those decisions belong to whoever owns the fleet, which means they need to live in the console rather than in a deploy script only two engineers can run.