Testing a connected product again after its software changes

A connected product is a moving target. A door sensor that worked yesterday may behave differently today, not because its hardware changed, but because its software did. An update can change an alert tone, a timer, a threshold, or the way the sensor reconnects after a power cut, and each of those changes can affect whether a user notices an open door.

Testing after an update should therefore start with a list of behaviours that must still work, not with a general hope that the update is safe. For a door sensor, that list includes opening alerts, closing alerts, low battery alerts, timer behaviour, and app display, each tested separately after the update, not assumed from the launch test.

Timer failure is a telling example. Suppose an update resets a schedule that silences alerts at night, or changes a delay before an alert is sent. A user who relied on the old schedule may miss an open door, while a user who relied on the old delay may receive alerts at the wrong time and learn to ignore them. Both failures stem from a software change, not from a hardware fault.

Compatibility adds another layer. An update may work on a new sensor but fail on an older sensor still in homes, or may work on one phone operating system but not another. The test plan should therefore include old sensor versions and common phone types, rather than testing only the newest combination in the office.

Illustration: a brand updates 60 door sensors and imagines that 20 of them, in round illustration terms, lose a timer setting until the user opens the app again. Those figures, as presented in Testing a connected product again after its software changes, are invented, not measured. They show why update testing must include the user journey after the update: does the sensor work without the user opening the app, does it warn the user if a setting was lost, and does it recover gracefully if the update fails halfway.

If the answer to any of those questions is no, the update should not be released until the journey is fixed or clearly explained to users, because a silent loss of a setting is a safety problem, not a minor inconvenience.

An update test plan should be written before the update is released, and should list the behaviours to retest, the versions to test on, and the criteria for a pass. It should also include a rollback plan: what happens if the update fails, how the sensor returns to a working state, and how users are told.

• List alerts, timers, readings, and app display that must still work after the update.
• Test old sensor versions and common phone types, not only the newest setup.
• Check the user journey after the update, including settings that may be lost.
• Prepare a rollback plan and a user message for a failed update.

Connected testing often fails by testing only the new feature. A brand excited about a new alert sound may test that sound thoroughly while forgetting to retest the low battery alert, the timer, or the reconnection after a power cut. Regression testing, which retests old behaviours after a change, is the remedy, and it should be routine, not exceptional.

Another failure is to test in the office only, on a strong network, with a new phone, and a new sensor. Real homes have weak signals, old phones, and old sensors, and an update that works in the office may fail in those conditions. The test plan should include realistic network and device conditions.

Close by filing the update test record with the update itself: what changed, what was retested, what passed, what failed, and what was fixed before release. That record lets a future developer see why an update was released and what was checked, rather than guessing from a version number alone.

Revisit update testing for every release, however small it seems. A small change to an alert tone can have a large effect on whether a user notices an open door, and a routine retest is the only way to catch that effect before users do.