It fails at boot but works by hand: environment first
Updated
Manual runs inherit your shell; boot runs get almost nothing: compare environments line by line.
Works by hand, fails at boot is a whole genre of incident with one plot: the manual run borrows riches from your interactive shell, and the boot run starts poor. The fix is a comparison, not a guess.
Diff the two worlds
Capture the manual environment. The variables, the path, the user, the working directory. Write them down from the working run.
Capture the boot environment. Log the same list from inside the unit at start. The diff between the lists is a short list of suspects.
Pin everything boot lacks. Unit file gains the user, directory, variables and absolute paths. Rerun at boot, not by hand: reboot or restart the unit cleanly.
Worked example: a fictional morning failure
The context below is fictional. Fictional service ParcelTrack (fictional) starts fine from a terminal, fails at every boot with a missing-config error. The manual shell has a variable pointing at the config; boot has nothing.
The diff names one variable. The unit file pins it, plus the working directory found missing next. A reboot test proves the fix; the prevention is a unit template with no inherited anything.
Checklist: boot-proof services
- Manual and boot environments captured and diffed.
- Unit pins user, directory, variables, absolute paths.
- Fix proven by a real reboot, not a manual start.
- Template updated so the next service inherits the discipline.
Related reading
- Hands on: Boot Failure from Environment and Unit.
- The service companion: Systemd services that stay up.
Straight answers
Frequently asked questions
Why does it work manually but not at boot?
Your shell provides variables, paths, secrets and working directory that boot does not. The difference is the bug.
What should a unit file pin?
User, working directory, every required variable, and absolute paths. Nothing inherited, everything stated.
How do I see the boot environment?
Log it from the unit itself at start: dump the variables that matter before doing anything else.