⏱ Safe2Go — Jobs

• Europe address import (running since 25 Sep, about 16% through the "ways" phase). All 50.6 M address points and 5.06 M locality points from the node phase were saved; the rest stopped. It sat dead for ~28 h until found on 28 Sep.
• Weekly GTFS re-ingest (had been running 25 h 52 min, reloading all 2,813 feeds). Stopped part-way.
• GTFS freshness check.

What is backed up, and where

On the external drive only (too big for the PC) - copied and read back 29 Sep, 0 mismatches

Also on the external drive: the full Safe2Go code working copy (git history, iOS certificates, Android release keystore, local settings files; 4,194 files, 0 mismatches), SmartPark keystores and the Claude history backup. Not backed up anywhere yet: Railway app variables - the owner copies them into a password manager.

Schedule

• 21:00 UTC nightly (server, safe2go-backup.timer → full_backup.sh nightly): all sets above, validated, listed with size + md5 in MANIFEST.txt; server keeps the last 3 days.
• After every country import: that country's tables (country:<name>), before the next country starts.
• 07:30 German time daily (owner's PC, Task Scheduler "Safe2Go backup pull"): copies every new file to D:\Safe2Go-Backups\<date>\, compares md5 with the server, keeps the newest 3 nightly folders (the manual 28–29 Sep folders are kept for good).

How to restore

• Timetables / addresses / mobility graph / Railway Postgres: pg_restore -d <database> <file>.dump (gtfs_* → gtfs, osm_* / europe_* / country_* → osm_transit).
• Europe backup of 29 Sep 07:59 predates Europe's map-location indexes: after restoring, create them again (the exact two commands are in the backup's MANIFEST.txt and in D:\Safe2Go-Backups\README.txt).
• App user data: mongorestore --uri=<MongoDB address> --archive=<file> --gzip.
• Configs: tar -xzf configs_*.tar.gz -C / (restores scripts, timers and settings to their original paths).

Risks still open

• LOW The external drive (WD 5 TB, USB) had connection dropouts during the first copy; every file was re-copied where needed and read back after an unplug/replug (41/41 match). Use a direct USB port on the back of the PC.
• MED The PC copy only happens when the PC is on; a week offline means a week without a fresh off-server copy.
• MED Restore tests (restoring into a scratch database and comparing counts) are scheduled after the search-index rebuild; until then the backups are verified by row counts and file checks, not by a full restore.
• LOW The configs backup includes the server's settings files, which contain passwords: keep the PC and the external drive private.

gtfs-freshness — daily feed freshness check + auto-refresh HIGH RISK

What it does

1. For every GTFS feed, reads how long its timetable is valid (latest calendar.end_date, or latest calendar_dates.date for feeds that only use exception dates).
2. Any feed that has expired or expires within 14 days gets re-downloaded and reloaded, but only if its source_url is a direct file. Landing-page URLs are logged as FEEDS_NEEDING_MANUAL_URL instead of being retried.
3. Reloading uses the shared loader ingestOne() in gtfs_ingest_batch.js: delete the feed first, insert the feed row, stream stops/trips/stop_times in, then write row_counts.

Risks

• UPDATE 1 Oct This job, together with gtfs-refresh's start-up cleanup (next card), also removed 7 feeds that had real data: OVapi Netherlands, De Lijn Flanders, HSL Helsinki, Mexico City, Fluo Grand Est, SIMUR Bogotá and AMT Italy. OVapi failed here every night from 3 Sep onwards (log lines 237, 751, 3339–6211), and each failure left it empty. The 28 Sep restore missed these 7 because their records were already gone. Now restoring (Task Queue, Feeds Monitor → Lost feeds). A database guard now rejects any delete that would leave a feed missing or empty (section 8b).
• HIGH — happening now A failed reload leaves the feed empty. The delete and the reload are not one transaction, so when a load fails the old timetable is already gone. Verified 28 Sep: 12 feeds have no completion marker. 11 of them have 0 stop_times: Île-de-France Mobilités (Paris, tdg-80921), Warsaw ZTM (mdb-2092), Lisbon TML (mdb-3412), Carris Metropolitana (mdb-2027), Budapest BKK (mdb-990), a large Finland feed (mdb-1090), Metra Chicago (mdb-2854), 2× FGV Valencia, Sofia (mdb-2848), AECFA Spain (mdb-2727). The 12th, Barrie Transit (mdb-3), is empty in every table. Users in those cities currently get no real timetable routes.
• HIGH Large feeds hit statement timeout when the disk is busy (Europe import, or another GTFS job at the same time). They then fall into the risk above.
• MED A feed with bad source data (e.g. dates like " 202-50-10", null stop_id) fails every day and is deleted every day. The daily "refresh" turns a feed that was working into an empty one.
• MED 111 feeds (mostly Canadian ca-*, whose URL is a Statistics Canada landing page) are never refreshed. Their timetables will expire.
• MED A ~6 h run of heavy writes to the gtfs database slows live timetable queries. It overlaps gtfs-refresh on Mondays and gtfs-reingest at weekends.
• LOW The log is not in /etc/logrotate.d, so it grows forever.
• LOW Nobody is alerted about failures. They are only written to the log.

gtfs-refresh — weekly catalog status check + new-feed ingestion HIGH RISK

What it does

1. Re-checks each feed's current status (active / inactive / deprecated) against the MobilityDatabase catalog and updates the feeds table. The app only serves status='active' feeds.
2. Runs the normal ingestion orchestrator to pick up brand-new active feeds. Feeds already present are skipped.
3. Known gap, stated in the script: it does not notice when a still-active feed republishes a new timetable. That is gtfs-freshness's and gtfs-reingest's job.

Risks

• FIXED 1 Oct HIGH — caused data loss Its start-up cleanup deleted feed records. gtfs_ingest_all.js ran DELETE FROM feeds WHERE row_counts IS NULL every time it started (added 13 Aug, commit ab15d16). That removed every feed left empty by a failed freshness reload, including OVapi (Netherlands). The clean-up ran 8 times, removing 3–111 records each time (CLEANED_ORPHANS in /var/log/gtfs-refresh.log and the August ingest logs). Now it only reports unfinished feeds and retries them; deployed 1 Oct, server md5 = git.
• HIGH Overlaps with gtfs-freshness every Monday (seen 28 Sep, 03:14–06:37). Two writers on one database means more timeouts, and those lead to emptied feeds (see freshness risk 1).
• HIGH 108 errors in the last run and nobody alerted. The errors are only in the log and in gtfs_ingest_failures.json.
• MED If the catalog wrongly marks a feed inactive, that city silently disappears from timetable routing the same day.
• MED Depends on the MobilityDatabase API and its token. If the API is down or the token expires, the run fails.
• MED On 5 Oct (the 5th is a Monday) it overlaps the monthly osm-transit-reimport.

gtfs-reingest — weekly full reload of every feed HIGH RISK

What it does

1. Downloads and reloads every active and stale feed, whether or not it changed. This catches agencies that republish a new timetable under the same feed ID.
2. Uses the same delete-then-load ingestOne() for each feed.
3. Runs VACUUM ANALYZE on the GTFS tables at the end to reclaim the space the deletes free.

Risks

• HIGH Same empty-on-failure problem as freshness, but for all ~2,800 feeds every week.
• HIGH Takes more than a day, so it always overlaps at least one daily gtfs-freshness run (both delete and reload the same feeds).
• HIGH Not resumable. If it is killed it starts again from feed 1 the next Saturday, and the VACUUM at the end never runs, so dead rows build up and the disk fills.
• MED More than a day of heavy disk load slows live timetable searches for the whole weekend.
• MED Next run is Sat 3 Oct 04:00, very likely while the Europe import is still running (disk contention).

osm-transit-reimport — monthly OSM transit-stop refresh HIGH RISK — next run Mon 5 Oct

What it does

For each of 8 regions in order (Germany, India, Malaysia/Singapore/Brunei, Australia-Oceania, France, Canada, Africa, USA): download the Geofabrik extract to /tmp, import its stations/stops/airports into a scratch table with osm2pgsql, then in one transaction delete that region's rows from transit_stops and insert the new ones. A failure leaves that region's old data untouched (this part is safe).

Risks

• HIGH Downloads go to /tmp, which is a 7.7 GB RAM disk, not the data disk. Extracts like USA and Africa are larger than 7.7 GB, so those downloads will fill /tmp and fail, and USA/Africa stops will never refresh. While /tmp is full it holds that much RAM. That can push free memory under 2.5 GB, which triggers mem_watchdog.sh to kill a Valhalla routing server (see below). Region sizes are approximate; the 7.7 GB /tmp size is verified.
• HIGH Clashes with the Europe address import. Every osm2pgsql import into the osm_transit database writes the shared osm2pgsql_properties table (prefix planet_osm). The Europe import runs in --append mode and relies on that table (it currently records the Europe import's style and updatable=true). If this job runs before Europe finishes, the Europe import can fail or be refused. Not yet tested; needs checking before 5 Oct.
• MED Runs on a Monday morning together with gtfs-refresh and gtfs-freshness.
• MED The rewritten script has never actually run from the timer.

fabrication-watch — live check for fake routes MED

What it does

Calls the live production API (/api/journey/_test_plan) for 5 fixed real trips and checks each answer for the known fake-data signatures: a flight leg that isn't a real scheduled flight, a banned wrong-region booking provider, or a placeholder "X International Airport" name without a real IATA code. Logs PASS or FAIL.

Risks

• MED Only writes a log line; nobody gets alerted. 57 FAIL lines are in the log to date. The 28 Sep 00:16 and 06:17 empty responses (Toronto→NY, Paris) haven't been investigated. They line up with the disk load and with Paris's emptied GTFS feed.
• LOW It only covers 5 routes, so a fake leg on any other route isn't caught.
• LOW It adds 5 real requests to production every 6 hours (negligible).

valhalla-usa-watchdog — routing-server memory guard MED

What it does

Despite the name, it covers 8 Valhalla routing servers: ports 8002 Germany, 8003 India, 8004 USA, 8005 Malaysia/SG/BN, 8006 Australia, 8007 Canada, 8008 Africa, 8009 Europe. If a port has no process, it starts one. If a process stays over its memory limit for 2 checks in a row (10 min), it kills it with kill -9 and starts it again. Limits: Germany 4.5 GB, India 4 GB, USA 8 GB, Africa 2 GB, Europe 4.5 GB, others 2 GB.

Risks

• MED Each kill means a few seconds with no routing in that region.
• MED Ports 8011–8016 (Brazil, Argentina, Mexico, China, Russia) are not covered. If one of those servers dies, or the server reboots, nothing restarts it and routing in those countries stays down.
• LOW Because of KillMode=none, restarted servers stay inside this unit's group, and systemd logs "left-over process" warnings on every run. On 27 Sep, needrestart also listed this unit for restart together with Postgres.

These run under nohup, not as systemd services. None of them comes back by itself after a reboot, except that valhalla-usa-watchdog restarts ports 8002–8009.

Europe address import (resumed 28 Sep) DONE

Risks

• HIGH Keeps the data disk 90%+ busy (measured 93%). Live timetable searches time out and fall back to taxi/estimates.
• HIGH Clashes with osm-transit-reimport on 5 Oct (see its card) and with gtfs-reingest on 3 Oct.
• MED osm2pgsql cannot pick up mid-file, but this run can safely be run again: in append mode each way's old rows are replaced, not duplicated, and indexes on (osm_type, osm_id) were added 28 Sep to make that fast.
• MED Not a service, so it doesn't restart after a reboot.

europe_periodic_backup.sh — 2-hourly Europe snapshot STOPPED

Risks

• MED Stops for good after a reboot.
• MED The database password is written in plain text inside the server copy of the script (the git copy reads it from the environment).
• LOW ~5 min of extra disk reads every 2 h. Uses ~8 GB of data disk.

Canada & USA address import (started 29 Sep) MED

Risks

• MED Each new country rebuilds osm2pgsql's staging tables (create mode); a crash mid-country means that country restarts from zero. Already-finished countries are never touched (separate tables per country).
• MED Must finish before the monthly osm-transit-reimport on 5 Oct (both use the same staging tables); USA is expected well before then.
• LOW The "already imported?" check counts through the combined view (all countries, 130 M+ rows) and takes several minutes per country; to be replaced by a per-country table check.

mem_watchdog.sh — emergency RAM guard HIGH — 7 copies running

What it does

If available RAM drops below 2,500 MB, it kills the biggest Valhalla routing server with kill -9 and starts it again.

Risks

• HIGH 7 copies run at once. The single-copy lock only remembers the newest one. At one low-memory moment, several copies can each kill a routing server, taking down several regions together.
• HIGH Memory pressure caused by other jobs (Postgres during imports, a full /tmp) makes it kill routing servers that did nothing wrong. Swap is already 100% used (9 of 9 GB, 28 Sep).
• MED Its port map has no Europe (8009) or the 8011–8016 regions. It still restarts them from the config name, but logs an empty port.

13 Valhalla routing servers MED

Ports 8002–8009 (see watchdog) plus 8011 Brazil, 8012 Argentina, 8013 Mexico, 8015 China, 8016 Russia. All run under nohup, started between 10 and 26 Sep. They aren't scheduled, but they are the other two jobs' targets and are affected by every memory event. Risk: not systemd services; 8011–8016 have no auto-restart at all.

PostgreSQL autovacuum (inside Postgres) LOW–MED

Postgres cleans up deleted rows in the background. It is switched off on planet_osm_nodes for the Europe import (must be switched back on afterwards; it's on the after-Europe checklist). On the gtfs database an autovacuum worker has been running since 27 Sep 06:21, cleaning up after the killed re-ingest. Risk: extra disk load at the same time as the imports.

apt-daily-upgrade / unattended-upgrades — automatic security updates FIXED 28 Sep MED

What it does

Installs Ubuntu security updates automatically. Until 28 Sep, its helper needrestart then restarted every service using an updated library, including PostgreSQL. That caused the 27 Sep incident.

Changes made 28 Sep

• /etc/needrestart/conf.d/90-no-auto-restart.conf: services are only listed as needing a restart, never restarted; PostgreSQL is explicitly excluded.
• /etc/apt/apt.conf.d/51-no-auto-postgres: postgresql* and libpq5 are held out of automatic updates, because a PostgreSQL package update restarts the database from its own installer.
• /etc/apt/apt.conf.d/99-no-auto-reboot: automatic reboots are off.

Remaining risks

• MED PostgreSQL security fixes now need a manual update and restart, done between jobs.
• MED A new kernel (7.0.0-34) is installed but not active. The server needs a planned reboot at some point, and that reboot stops all the nohup processes in section 5.
• LOW Services keep running on the old library copies until restarted by hand.

Other OS housekeeping LOW

These live inside the Node server process (setInterval). They restart whenever the app is redeployed or restarted, and everything they keep only in memory is lost at that moment.

Health monitor MED

Risks

• MED Only watches place search and quotas. It does not watch timetable routing, the Hetzner jobs, Valhalla, or the imports. None of the problems on this page would have triggered an email.
• LOW The "last problem" memory resets on every deploy, so an ongoing problem gets emailed again.

Safety check-in sweeper MED

Risks

• MED While the app is restarting or redeploying (seconds to a minute), no check-in is escalated. It catches up after the restart only if the sessions are still stored.
• MED SMS/phone escalation needs Twilio credentials, which are not set yet, so alerts go by email only.

Live-tracking session cleanup MED

Risks

• MED Tracking sessions are kept only in memory, so every deploy or restart ends every live-tracking link that people have shared.

Small in-memory cleanups LOW

• server/middleware/searchRateLimit.js: drops old per-IP search counters every rate-limit window. Risk: counters reset on every deploy.
• server/services/overpass.js: drops expired Overpass cache entries every 10 min. No risk worth noting.

Nothing was removed before the data it belonged to was verified and backed up, per the owner's standing rule.

Owner switch for the guard (only when a deletion is really wanted): INSERT INTO gtfs_guard_override (until, reason) VALUES (now() + interval '1 hour', 'why');. The row stays as a record. Limit: a database superuser can still switch the triggers off. The daily inventory counts the real rows, so that would still show up.

1. New 1 Oct: finish restoring the 7 lost feeds and compare their counts with the 3 Sep loads. Then decide (owner) whether to reload the 39 "empty when lost" and 8 active "no record" feeds (Feeds Monitor → Lost feeds).
2. New 1 Oct: gtfs-reingest.service is in "failed" state since the automatic Postgres restart killed its 26 Sep run. Next run Sat 3 Oct.
3. Restore the 11 emptied feeds and make reloads keep the old timetable on failure DONE 28 Sep (AECFA left out by the owner).
4. Connect the external drive and copy the map tiles, routing data, catalog and building footprints plus a third copy of the PC backups (section 2b).
5. Restore tests of the backups into a scratch database (after the search-index rebuild).
6. Rotate the credentials that appeared in a Railway screenshot during the 29 Sep backup work (MongoDB password, Stripe secret key, Gmail app password).
7. Restrict who can connect to the server's database (currently any address on the internet with the password) and change that password.
8. Before 5 Oct: make osm-transit-reimport download to the data disk instead of /tmp, and keep it from running while the Europe import is active.
9. Stop the GTFS jobs running on top of each other (freshness + refresh on Mondays, reingest across the weekend). Let only one GTFS writer run at a time.
10. Reduce mem_watchdog.sh to a single copy (7 running now).
11. Add auto-restart for Valhalla 8011–8016 (Brazil, Argentina, Mexico, China, Russia).
12. Send FAIL alerts (fabrication-watch, GTFS errors, import death) by email instead of only writing them to logs.
13. Add the Safe2Go job logs to logrotate.
14. Plan a maintenance window for the pending kernel reboot and manual PostgreSQL updates, after Europe finishes.