Big project: rollout

The base is running, the people inside are breathing. Now rip out the old control and insert five new chips at once? If something goes wrong, someone sits in the dark without air. That is why a big project is installed like a new water pipe in an inhabited house: piece by piece, with a tap to turn back.

Commissioning step by step#

One subsystem after the other, and the most important one last. Logistics and production first – if something sticks there, at worst an order waits. Power afterwards, life support at the very end, once everything else has been running for days. The central unit comes as soon as two subsystems serve their interfaces.

Shadow mode#

Before a new chip may control a device, it runs along in the shadow: it reads all sensors, computes everything, shows its decision on a display – and leaves the device alone. The old control or the player's hand stays in charge. Only a lever “Freigabe” hands the device over to it:

select tmp ok new -1
sbn ANZ HASH("Neu_Entscheidung") Setting tmp   # what the new control would do
beqz release start                        # without release: hands off the device
sbn HEIZUNG HASH("Werkstatt") On new

That way you see for a day whether the new control would do the same as the old one, without it being able to do any harm.

Fallback#

For every part of the system there is a way back: the old chip in a box next to it, a lever for manual operation, the release that you throw back again. If the new chip detects an error itself, it reports it – and with release it switches its device into the safe state.

Watching#

After switching over the work is not done. For a few days you look at probes and counters: how often does the heater switch? How often was there an alarm? Does every chip's heartbeat keep moving? Only when nothing surprises you any more does the next subsystem get its turn.