HSLG — complete-loss disaster recovery (Ubuntu / Webmin / Virtualmin / MariaDB)

Canonical public documentation page: https://media.hslg.uk/app/disaster-recovery/

Scope: Rebuild hosting, WordPress, API, documentation publication, development tooling, Android signing/builds and external iOS builds after the entire server and all backups are lost.

Status: Rebuild instructions reviewed against source and operator-reported setup; not an executed bare-metal recovery drill. Vendor versions, hosting contracts, DNS and plugin availability must be independently confirmed at recovery time.

0. What can and cannot be recovered

A clean Ubuntu installation can recreate the operating environment, WordPress, empty HSLG MariaDB tables, plugins, UI and development tooling if the source and package archives survive elsewhere. If every copy of the HSLG source also disappears, the current application cannot be reconstructed exactly from an empty WordPress installation. If all databases and backups are lost, customer records, sessions, bookings, orders, audit records, uploaded media, WordPress editorial revisions and original MariaDB write timestamps cannot be recreated. A new database must be labelled a new empty instance; do not synthesize old records, retroactively claim successful payments, or back-date timestamps. Payment providers and email services might retain limited independent records; confirm what they hold and what privacy law permits restoring. Obtain professional advice before communicating a personal-data incident.

If the development Android keystore and all encrypted offsite copies are lost, its original signing identity cannot be recreated. Future APKs signed by a new identity generally cannot update installed development apps in place. A Play-managed production signing key may be recoverable only under its provider's rules. Apple certificates/provisioning require access to the owner's Apple developer account and macOS; lost account access can prevent iOS updates. Never commit keys, credentials, recovery codes or database dumps into the source, AI library or publicly served directories.

Dependency order: physical/virtual host → OS network/time → DNS and TLS → Webmin/Virtualmin → PHP-FPM/Apache or Nginx → MariaDB → isolated virtual server → WordPress → HSLG plugins/schema → customer content and documents → private build tooling → native builds → domain and user acceptance → backup drills. Recovery pauses if a preceding validation fails.

1. Evidence to keep off the failed server

Preserve encrypted, independently restorable source/release archives with SHA-256 hashes, WordPress plugins, DB backups, uploads/media, all required secrets in a separate credential manager, DNS/registrar access, TLS certificate inventory, Virtualmin domain configuration exports, database users/grants, mail provider settings, dependency lockfiles/caches, Android dev keystore and fingerprints, Apple Developer access and provisioning, Stripe webhook endpoint IDs and live/test secrets, and a signed recovery test result. Preserve audit evidence in an independent storage account or physical medium. Merely generating backups on /home/media does not qualify as disaster recovery.

This guide's commands do not contain secrets. Restore passwords via private interactive prompts or a secret manager. Never paste secrets into shell history or a support conversation.

2. Before making changes — inventory, snapshot, confirmation

Use docs/disaster-recovery/commands/01-inspect-host.sh on the existing host or new clean host. It is strictly read-only and collects OS, kernels, PHP SAPIs, installed PHP modules, FPM pool ownership and status, Virtualmin domain entries, MariaDB version and the website's actual document root. Review the output and redact domain/host information before publishing it.

On the existing shared server do not run any installer, apt upgrade, MariaDB initialization, or recreation action. The Webmin/Virtualmin recovery installer is for a new empty server only. Before anything destructive, verify the exact new machine, out-of-band access, fresh disk/snapshot and operator approval. Shared /etc/php, /etc/apache2, /etc/mysql, systemd, apt, ufw, and system-wide services are owned by Virtualmin/OS, not HSLG. PHP module packages are inherently OS-wide; on a shared server, inventory them and request maintenance approval rather than silently installing them.

3. Provision a fresh Ubuntu server safely

Use supported Ubuntu Server LTS matching the tested deployment baseline (operator reported Ubuntu 24.04, Linux 7.0.0-31-generic, x86_64). Set the management hostname to hq.hslg.uk only if the recovered server is intentionally replacing that host. Confirm hostname, static/public IP, IPv4/IPv6, firewall, SSH key access, security updates, disk capacity and time synchronisation. Never disable SSH before confirming a second working login. DNS A/AAAA and reverse DNS updates should be staged before cutover. IPv6 must point to a working host or be removed intentionally.

Example inspection only on the new machine:

hostname -f; cat /etc/os-release; uname -m; date -u; timedatectl status

ip -brief address; ip route; df -h /; free -h; systemctl is-active ssh

Do not copy an old root filesystem onto an already running Virtualmin host. Install and secure the OS under the hosting provider's documented process. Set NTP/chrony so MariaDB sessions can operate in UTC. Record original disk layout and approved recovery identity.

4. Install and validate Webmin + Virtualmin — new machine only

Virtualmin manages the web server, PHP-FPM pools, users, certificates, domains, MySQL/MariaDB grants, mail and backups. On the new clean supported host, fetch the current official Virtualmin installer from the vendor's documented download path, inspect it and independently verify its authenticity and current installation prerequisites. A commonly documented GPL installer path is https://software.virtualmin.com/gpl/scripts/virtualmin-install.sh; its present availability and digest are not verified in this project. Do not pipe a remote script directly into root shell or execute a downloaded installer without operator review. A new install may choose Apache or Nginx and may deploy/upgrade PHP, MariaDB and mail components; select architecture before installation. Do not run the installer on the currently working hq.hslg.uk installation.

# On NEW, BLANK Ubuntu host only. This step DOWNLOADS for review; it does not execute.

curl -fL --proto '=https' --tlsv1.2 \

  'https://software.virtualmin.com/gpl/scripts/virtualmin-install.sh' \

  -o /root/virtualmin-install.review.sh

sha256sum /root/virtualmin-install.review.sh

head -n 50 /root/virtualmin-install.review.sh

# STOP: compare the script/hash with the vendor's current official installation instructions.

# AFTER manual verification on the NEW host, an administrator may run the vendor-approved installer.

After approved installation: validate Webmin login at the configured management port through a protected network; run Virtualmin configuration validation; confirm Virtualmin knows the right default IP/hostname; check service health and MariaDB connectivity. Record the actual configured web server and PHP mode. Do not assume Apache+PHP-FPM if Virtualmin has selected another configuration. Restrict the Webmin administrative port to trusted addresses. Enable HTTPS for both management and the hosted website.

sudo virtualmin list-domains --name-only

sudo systemctl --no-pager --type=service --state=running | grep -E 'webmin|mariadb|apache2|nginx|php.*fpm' || true

sudo mariadb --version

5. Recreate media.hslg.uk Virtualmin virtual server

Create a dedicated Virtualmin virtual server for media.hslg.uk under a dedicated Unix owner media. On new Virtualmin, prefer the domain-creation UI and explicitly select the website, PHP-FPM and MariaDB features; use virtualmin create-domain --help to verify CLI flags before scripting them. Do not run create-domain against the old host or duplicate an existing domain. Verify the document root is precisely /home/media/public_html/; Virtualmin's actual home path, PHP pool and website owner are authoritative rather than assumptions. Recreate SSL certificates through Virtualmin using a resolvable DNS record and provider-supported ACME flow; don't reuse lost private keys.

sudo virtualmin list-domains --name-only

sudo stat -c '%U:%G %a %n' /home/media /home/media/public_html

sudo virtualmin list-php-versions --domain media.hslg.uk 2>/dev/null || true

Keep private files (/home/media/.hslg-build, /home/media/.local, docs backup copies, keys, archives, Composer caches) outside public_html. Never use chmod 777 or blanket chown -R on a shared Virtualmin host. New files should be owned by media; directory permissions must be verified rather than forced recursively.

6. PHP inventory and version discipline (critical)

Operator-reported baseline: PHP CLI 8.3.6 at /usr/bin/php8.3, MariaDB 10.11.14, PHP-FPM pool /etc/php/8.3/fpm/pool.d/17926959921219725.conf. These are historical observations, not guaranteed after rebuild. Record PHP CLI and the site-selected FPM version; they can differ. Check active extensions for both SAPIs before attempting WP/plugin activation. The following are typical candidates; install only those actually needed by the audited current plugin manifests and WordPress on the newly provisioned host:

| PHP need | Ubuntu 24.04 package typically providing it | Why / checks |

|—|—|—|

| PHP-FPM/CLI | php8.3-fpm, php8.3-cli | Process manager and private WP-CLI runtime |

| MariaDB driver | php8.3-mysql | mysqli, pdo_mysql for WordPress |

| JSON / sodium / openssl | Usually provided by PHP core/common | php -m, php -i; never replace with custom crypto |

| XML / DOM / SimpleXML | php8.3-xml | WordPress extensions and document processing |

| cURL | php8.3-curl | HTTPS transport and API calls |

| mbstring | php8.3-mbstring | Multibyte input and mail/content processing |

| ZIP | php8.3-zip | Plugin/archive handling |

| GD | php8.3-gd | Image processing |

| Intl | php8.3-intl | International formatting if code requires it |

| BC Math | php8.3-bcmath | Only if audited libraries require it |

| OPcache | php8.3-opcache | Production caching; inspect FPM settings |

| Imagick | php-imagick | Optional; only if actually used and approved |

IMPORTANT: apt install php8.3-* is a system-wide change, even if the app itself lives under /home/media. Virtualmin hosts may run different PHP versions per site. Never install or remove packages on an existing multi-tenant host without Virtualmin compatibility checks and administrator approval. On a *new, dedicated, empty* recovery host, a maintainer may dry-run exact packages first:

sudo apt-get update

sudo apt-get -s install php8.3-fpm php8.3-cli php8.3-mysql php8.3-curl php8.3-xml php8.3-mbstring php8.3-zip php8.3-gd php8.3-intl php8.3-opcache

# Read every simulated addition/removal, especially Apache/FPM/Virtualmin integration.

# Then seek explicit maintenance approval before running the same command without -s.

Version-specific CLI inspection:

php -v; php --ini; php -m; php -i | grep -Ei '^Loaded Configuration File|^Scan this dir|^upload_max_filesize|^post_max_size|^memory_limit|^max_execution_time'

/usr/bin/php8.3 -m 2>/dev/null || true

sudo systemctl is-active php8.3-fpm

sudo /usr/sbin/php-fpm8.3 -t

Record extensions as a manifest in the recovery evidence, and distinguish missing but optional from missing and required. php --ini shows CLI configuration, not the effective per-website FPM values. Never use a CLI output alone to claim the upload limit is fixed.

7. php.ini, per-domain PHP-FPM pools and upload limits

Goal: Match the current HSLG site configuration without altering other Virtualmin websites. Web uploads initially failed because upload_max_filesize in the effective FPM pool was too low. The operator then confirmed the following HSLG-only pool values on 22 October 2026:

php_admin_value[upload_max_filesize] = 64M

php_admin_value[post_max_size] = 80M

Review other important PHP settings for the HSLG site only: memory_limit (WordPress/plugin operations), max_execution_time, max_input_time, max_file_uploads, upload_tmp_dir, session.save_path, date.timezone, display_errors, log_errors, error_log, expose_php, opcache.enable, FPM pm, pm.max_children, pm.max_requests, socket access and owner/group. Do not impose guessed values. Keep display_errors=Off for public pages and use nonpublic logging. Verify post_max_size is larger than upload_max_filesize, and web-server request-body limits independently if upload errors persist. Make sure the site owner can use the upload temp directory and that neither it nor the log path is world writable.

Safe inspection workflow (no changes):

1. Run docs/disaster-recovery/commands/01-inspect-host.sh and 02-inspect-php.sh as root. They report all FPM pool ownership matches without printing secrets.

2. Identify the unique pool serving the media Unix user and media.hslg.uk; do not edit the first numbered filename by guesswork.

3. Inspect /etc/php/8.3/fpm/php.ini, conf.d/*.ini, the chosen pool and Virtualmin domain PHP overrides; search for php_value[...] and php_admin_value[...] duplicates.

4. Prefer Virtualmin's per-domain PHP Settings screen or supported CLI. A prior operator used virtualmin modify-php-ini --domain media.hslg.uk --ini-name upload_max_filesize --ini-value 64M and similarly post_max_size 80M, but it did not immediately fix the live upload failure. Check supported CLI flags using virtualmin modify-php-ini --help; do not assume undocumented flags work.

5. When the chosen Virtualmin installation stores overrides only in its FPM pool, make a timestamped pool-specific backup, edit only that pool, syntax-test PHP-FPM and reload the relevant version after checking that other tenants share that service. Prefer Virtualmin's managed interface to direct file edits, which could be regenerated by Virtualmin later.

6. Reinspect the target FPM pool and compare the effective web request. Use a temporary access-controlled diagnostic on media.hslg.uk if necessary; delete it immediately. Never expose phpinfo() publicly; it leaks environment paths and sensitive runtime details.

7. Confirm WordPress upload UI and HSLG API functionality. On rollback restore only the backed-up target pool, run php-fpm8.3 -t, and reload only the affected PHP-FPM service in an approved maintenance window. Never restart MariaDB or the whole web server to change an HSLG upload limit.

For a carefully guarded, reversible per-domain recovery action, use docs/disaster-recovery/commands/03-stage-media-php-limits.sh. It performs a dry run unless --apply is supplied after an explicit operator review and host/domain verification. It will stop if it cannot identify one relevant media FPM pool, fail syntax checks, or see the expected Virtualmin domain. It does not edit global php.ini, another domain's pool, or the system PHP version.

Verify site limits using a temporary authenticated WP-CLI check plus the WordPress Site Health admin view:

sudo -u media -H wp --path=/home/media/public_html eval 'echo "PHP ".PHP_VERSION."; upload=".ini_get("upload_max_filesize")."; post=".ini_get("post_max_size").PHP_EOL;'

# This WP-CLI result uses CLI PHP; use Site Health > Info > Server for web-SAPI confirmation.

FPM pool filenames are opaque Virtualmin IDs. Hardcoding /etc/php/8.3/fpm/pool.d/17926959921219725.conf into an unattended installer is unsafe; identify the current target by both owner and domain or stop.

8. MariaDB installation, schema and timestamps

Virtualmin may manage the MariaDB service on the new host. Inspect first: sudo mariadb --version, systemctl is-active mariadb, and sudo mariadb -e 'SELECT VERSION(), @@global.time_zone, @@session.time_zone, UTC_TIMESTAMP(6)'. No global DB config edits on an existing shared host. Configure and verify UTC sessions in the HSLG application DB connection so DEFAULT CURRENT_TIMESTAMP(6) at original insert is a comparable UTC timestamp; server clock skew must be addressed if multi-node sync is later implemented.

Create a database and limited database user through Virtualmin's domain database facility or audited SQL. Do not place database passwords in a pasted CREATE USER ... IDENTIFIED BY ... shell history. Use a private credential manager and least privilege. Validate WordPress $table_prefix on the recovered site, then deploy schema migrations in order (foundation → record schema 2 → account/order schema 3 → payment schema 4) only after confirming the actual plugin migration plan on this release. Do not mark tables as migrated by editing options manually.

Customer fields use stable customer UUID, fresh record UUID, origin node ID and original MariaDB written_at DATETIME(6) DEFAULT CURRENT_TIMESTAMP(6). Ordinary inserts omit written_at; imported rows preserve their original source timestamp. Latest-value reads order by written_at DESC, record_id DESC. WordPress content, credentials, orders and payments are *not* latest-write-wins data. Do not import fabricated rows or create payment confirmations based on a timestamp. Document legal retention/erasure obligations.

If every database backup failed: create only fresh empty business tables. Mark previously issued tokens and sessions invalid. Old app installations must treat identities/drafts as unavailable until registered/reconciled again. Preserve all independent payment-provider receipts separately and do not charge anyone again until reconciliation is verified. Account recovery cannot resurrect lost profiles without another lawful surviving source.

9. WordPress reinstall and safe revision-preserving docs recovery

After Virtualmin creates /home/media/public_html/ with correct ownership, ensure the document root is empty or contains only expected Virtualmin placeholders. The following is the previously used direct-root install; it must never overwrite an existing installation:

sudo -u media -H bash -c 'cd /home/media/public_html && test ! -e wp-config.php && test ! -e wp-includes && curl -fsSL https://wordpress.org/latest.tar.gz | tar -xz --strip-components=1'

Use official WordPress setup wizard; configure an HSLG-specific MariaDB DB/user and administrator using a fresh secret. Install WP-CLI privately (recorded reference: docs/history/commands/02-wp-cli-install.sh), verify its signature/checksum, and then run sudo -u media -H wp --path=/home/media/public_html core version to validate runtime. Disable file editing from the dashboard as appropriate; keep admin accounts protected by MFA and least privilege.

Restore only vetted presenter ZIP, independent App API ZIP and independent docs publisher ZIP, checking SHA-256 manifests and their minimum PHP/WordPress requirements. HSLG presentation is not a mobile build input; the WordPress API is not a substitute for native Capacitor. Do not enable staging-only M5/M6 API flags on a public host without authentication and HTTPS checks. Test wp_mail() using the configured WordPress SMTP/provider plugin; never add a second hidden mail transport. Review webserver file owners, PHP-FPM isolation and logs before activation.

The docs publisher owns /app/code/, /app/architecture/, /app/user-guide/, /app/licensing/, /app/disaster-recovery/ and /devlog/. Generate documentation from the reviewed source bundle. Install the documentation publisher release ZIP through the WordPress administrator. The plugin automatically creates reviewed pages on activation and revises owned pages on subsequent ZIP updates; no routine SSH or WP-CLI publishing command is needed. The publisher must stop on unowned-page path collisions or unreviewed manual edits. It uses wp_insert_post() for new pages and wp_update_post() for changes, preserving post IDs and the WordPress revision history. Require unlimited revisions (WP_POST_REVISIONS=true) and verify the revision list after each release. Never delete posts or overwrite the documentation tables directly. If all WordPress backups are lost, historical on-site page revisions cannot be restored. The source documentation still recreates the most recently archived canonical content.

10. Rebuild the private Composer/PHP dependency environment

Composer on the operator-reported HSLG site was installed privately at /home/media/.local/bin/composer with COMPOSER_HOME=/home/media/.config/composer and COMPOSER_CACHE_DIR=/home/media/.cache/composer. On a rebuilt server use the verified Composer official installer and SHA-384 installer signature from getcomposer.org / composer.github.io/installer.sig as captured in docs/history/commands/05-composer-private-install.sh. Never run Composer as root against the HSLG project or install it globally for app deployment. Install only with audited composer.json and composer.lock, using --no-dev --prefer-dist --no-interaction and retained vendor notices; don't invent a Stripe SDK installation if manifests are incomplete. Avoid executing Composer scripts/plugins from unreviewed archives.

11. Rebuild the private Android toolchain

Use /home/media/.hslg-build/ for all private build downloads and caches, not public_html. Previous operator-reported versions: Node 22.23.3, pnpm 10.15.0, Temurin JDK 21.0.13, Android Command-Line Tools, API 36, Build Tools 36.0.0, Platform Tools, and Gradle 8.14.3 for direct testing. See docs/history/commands/06-private-build-toolchain.sh, 07-build-toolchain-repair.sh and docs/disaster-recovery/commands/04-verify-build-environment.sh. Download official packages again, verify upstream checksums, and maintain an approved offline package cache. Do not blindly trust old version URLs forever; ensure current Capacitor project/Gradle plugin compatibility from the checked-in lockfiles before installing anything.

Build the real apps/mobile Vite entry, run pnpm installation from a reviewed lockfile, cap sync android, project Gradle wrapper and device-level tests. A WebView-only preview APK is not a genuine Capacitor app. Keep Android dev keystore encrypted outside source and WP uploads; restore the same key/certificate fingerprint to support in-place updates. Without a surviving key, stop rather than silently signing an "upgrade" with a new identity. Never bundle signing passwords in release ZIPs.

12. Restore the AI development workspace and iOS build arrangement

The AI workspace is ephemeral; restore HSLG source from the latest surviving versioned archive and verify its hash. Reintroduce only audited npm tarballs, SDK ZIPs and lockfiles. Read root/mirrored AGENTS.md, README, three documentation hierarchies, licensing, history, disaster recovery and current TODO/CHANGELOG. Run source/docs/history/licensing/native gates independently. The AI environment cannot be assumed to possess server SSH access, all Maven caches, a signing key or a physical device. Claims must match real evidence.

Native iOS IPA compilation requires a supported macOS host with Xcode. No GitHub dependency is mandated. Use an independently managed Mac or an approved macOS build provider with encrypted Apple signing credentials and reviewed API. A PHP-triggered build service may enqueue jobs only with authenticated/authorised requests, anti-CSRF protections, constrained commands, queuing, audit logs and off-site build runners. Never expose exec() or a shell endpoint through public WordPress for arbitrary commands; never compile via PHP-FPM request workers. Apple ID and App Store Connect access must be recovered through the account owner's official process.

13. Mail, payments, domain, TLS and compliance

Recreate DNS records and WordPress wp_mail() configuration (SMTP/mail-provider plugin, SPF/DKIM/DMARC and verified sending domain). Send a test verification challenge, record receipt, verify expiry/rate limits and don't log codes/secrets. Confirm HTTPS for mobile API routes and host/domain verification for Android App Links and iOS Universal Links. Restore Stripe accounts and webhooks from the external provider's real account records, install fresh webhook secret through secure configuration, validate signatures in test mode, and reconcile payment and order ledgers before enabling charging. Never assume one Stripe customer/payment method is shareable across five venue accounts. Keep commercial software and media licensing evidence, third-party notices and external service terms up to date; unresolved commercial clearance must block production release.

14. Mandatory end-to-end recovery validation (separate evidence per layer)

Virtualmin/OS: correct hostname/IP, SSH recovery account, DNS, TLS, Virtualmin domain list, management access and no unrelated-site changes. PHP: correct media FPM pool, required modules, active website upload limit, no public phpinfo/error dumping, web and CLI modes noted. MariaDB: connectivity, empty-vs-restored dataset labelled honestly, HSLG schema migration status, timestamp UTC check and least-privilege grants. WordPress: fresh core, owner/path, plugin activation, login, site health, canonical docs navigation and revision history. API: protected authenticated endpoints, email test, profile timestamp/UUID semantics, staging-only feature gates and no fake financial success. Android: true Capacitor bridge, installed native plugins, APK signatures, physical-device QR/network/links and session safety. iOS: actual macOS/Xcode archive, signing, test devices and links. Licensing: direct/transitive package evidence, rights to distribute all assets, notices and service terms.

Run docs/disaster-recovery/commands/05-verify-site.sh as a read-only start. A successful script proves only its reported checks; it does not substitute for an operator's browser/phone tests, provider access or DB restore drill. Archive output outside public_html with review timestamp and responsible operator.

15. Recurring backup and drill design for one operator

Use layered backups: Virtualmin domain backup plus independent MariaDB logical dump plus encrypted uploaded media/config archive plus offsite immutable copies of source/releases and signing-key backup. Define backup cadence and RPO/RTO by actual operating needs; do not invent zero data loss. Test restoration into an isolated blank Virtualmin domain or separately created VM, using dummy accounts, without overwriting production. Periodically prove WordPress revisions and HSLG app API tables can be restored and tested. A backup reported "successful" is not a recovery plan until a restore drill works. Record backup provider/cost, retention, privacy erasure and who can access recovery secrets. Budget for an independent copy outside this server.

16. Release-by-release review obligation

This is one authoritative disaster-recovery WordPress page, not a dispersed set of public notes. All SSH command text files are part of the reviewed private source under docs/disaster-recovery/commands/ and are referenced from this page. Every release, including docs-only, must record an explicit disaster-recovery review linked to the newest chronological history entry and the current source/documentation checksum. Review when dependencies, tool versions, virtual hosting, PHP/FPM, MariaDB schemas, names/paths, ports, DNS/TLS, mail, signing, licence terms or plugin publication methods change. The gate must reject a missing/stale review, an outdated WordPress generated page, or command files that have changed without review. Publication on media.hslg.uk is a separate approval-and-evidence step; a local gate pass does not assert the WordPress page is published.

Current measured status: documentation source and publisher can be checked locally; no full disaster recovery, remote Virtualmin environment recreation, restore, email delivery, iOS or native release drill has been executed here. Do not label this guide as field-validated until that evidence exists.

Restore the HSLG restricted Android build worker and release artifacts

The HSLG WordPress presentation update can queue a genuine Capacitor APK build. The website does not invoke a shell. After restoring Ubuntu 24.04, Webmin, Virtualmin, the media virtual server, its WordPress instance, MariaDB and the isolated Android toolchain, restore the HSLG Presentation plugin and confirm PHP CLI is runnable as media (WP-CLI is optional; the release includes a local PHP-CLI queue bridge). First inspect /home/media/.hslg-build/env.sh and verify Gradle 8.14.3, pnpm, Android SDK build tools and Java 21. Do not change the system-wide Virtualmin PHP configuration to fix a failed Android build.

The worker is installed once as the media Unix user using the versioned plugin's installer:

bash /home/media/public_html/wp-content/plugins/hslg-presentation/worker/setup-worker.sh

It registers an entry in that user's crontab to poll WordPress for a reviewed native payload. Confirm the entry with crontab -l as media. Run the read-only docs/disaster-recovery/commands/06-verify-worker-registration.sh from the recovered source as media and confirm that it finds exactly one scheduled worker with the current plugin path. The installer remains versioned at wordpress/hslg-presentation/worker/setup-worker.sh; preserve it in project source and recovery archives. The worker's runtime code resides in the updated presentation plugin; builds, source staging, caches, debug ZIPs and APKs remain under /home/media/.hslg-build/. release/native-app.php inside the plugin is an ABSPATH-protected ZIP-bearing PHP payload, not an instruction to let PHP execute Gradle. The plugin checks a SHA-256 source checksum before queueing, and the worker rechecks it after claiming. A queued job whose artifact has changed must fail without compiling.

23 October 2026 field evidence: The site operator ran worker/setup-worker.sh as the non-root media account and reported the installer message confirming cron registration. This confirms installer completion was reported, not that cron will survive future restore or that an HSLG APK was compiled. Recheck cron with 06-verify-worker-registration.sh after each disaster restore and verify the compiled APK independently.

A successful native debug build puts HSLG-debug.apk, HSLG-build-debug.zip, build.log and status.json inside /home/media/.hslg-build/releases/<random-job-id>/. The WP-admin download endpoints require manage_options and a nonce; these artifacts must not be moved into public downloads. Native build logs may contain private infrastructure paths and should be reviewed before sending externally. The worker deletes its temporary compilation tree after archiving diagnostics, but leaves reusable npm/Gradle caches and the generated debug signing identity intact. Never package a debug keystore, token, MariaDB dump or .env into the WordPress plugin or diagnostics.

If the entire host and all backups are gone, you can reinstall the build toolchain and recompile source that exists in another surviving repository, ChatGPT artifact or WordPress plugin archive. You cannot recreate lost customer data, a lost signing key, lost certificates or erased release history from a clean Ubuntu install. A replacement debug signing key may prevent an APK from upgrading an old on-device installation without uninstalling it. Perform a real install/update test on a physical Android phone after recovering the signing identity and domain app-link association files. This paragraph is a recovery procedure, not evidence of a completed recovery drill.

Verified APK identity, failed-build recovery and safe retries

A WordPress build queue record is not a compiled release, even if the worker has claimed it. The Android build counts as successful only after a signed HSLG-debug.apk exists in the private release directory and matches the checksum in its worker status.json. Never set the successful payload marker while queueing or claim-starting a job. Do not edit MariaDB/WordPress options manually to force completion.

If a build fails, the WordPress administrator must be able to download the preserved private HSLG-build-debug.zip. Use Retry Android build in HSLG Presentation to queue a fresh attempt for the same reviewed payload. A newer presenter plugin release must also retry automatically if the last payload has no verified APK. Normal administration page refreshes must not create endless retries. A currently active queued/running job must not be duplicated; the worker uses a media-only filesystem lock to avoid concurrent compilers.

If the worker appears inactive, inspect the media account's cron registration with the read-only 06-verify-worker-registration.sh command; rebuild missing registration using the versioned setup-worker.sh as media, not root. If the job has been running longer than the defined timeout and no artifact exists, preserve any logs, then invoke the explicit retry action. Do not change PHP-FPM, WordPress database content, Virtualmin global services, MariaDB tables or another tenant's crontab to solve Android build errors.

Rebuilding after a native-scanner API mismatch

The operator's first real queued APK build stopped at the Vite stage on 22 October 2026, not during MariaDB, PHP, Virtualmin or Gradle startup. The reviewed @capacitor/barcode-scanner 3.1.3 package exports CapacitorBarcodeScanner, and the source must call CapacitorBarcodeScanner.scanBarcode(...); the older BarcodeScanner import cannot be bundled. The correction is deployed in presenter v2.9.3 as a changed, hash-verified native payload. Do not alter global PHP settings, install packages system-wide, or clear WordPress posts to fix a TypeScript/Vite symbol error. Preserve a failed debug ZIP, use the normal WordPress plugin upgrade and inspect the fresh job result. If the machine was rebuilt from scratch, run 06-verify-worker-registration.sh as the restricted media user before requesting another APK.