GoatFlow 0.10.0 is the largest release since the platform/product split: 93 commits and 263 changelog entries. The release announcement covers what users get. This post covers five pieces of engineering underneath it, each of which started as a narrow fix and turned out to be about a seam in the system: where a token meets a session, where one SQL string meets two databases, where a plugin meets the sandbox, where a version number meets a build, and where an upgrade meets a database it didn’t create.

Sessions That Actually End

The Problem

GoatFlow issues JWT access and refresh tokens at login, and it also keeps a session row so that admins can see who is signed in under Admin → Sessions. Those two things were not connected. Deleting a session, from the admin screen or by logging out, only had an effect on UI pages, and only when the browser happened to send the session_id cookie. Every /api/v1 route kept accepting the token until it expired on its own.

The gap was wider than it first looked. OIDC, SAML and POST /api/v1/auth/login logins created no session at all, so those users never appeared in Admin → Sessions and there was nothing to kill even if you wanted to.

The Solution

Every login path now creates a session, and both tokens carry its id as a sid claim. Authentication middleware checks that the session still exists on every route: API, plugin and customer. Refresh checks it too, so a killed session can’t mint a new access token.

The SSO flows got a related fix along the way. OIDC state and SAML RelayState are now bound to the browser that started the login by a short-lived HttpOnly cookie, so a callback completed in another browser is refused.

The Benefits

“Log out” and “kill session” now mean what an operator assumes they mean, on every surface, including tokens handed to scripts. The cost is one-off and stated up front in the upgrade notes: tokens issued before 0.10.0 have no sid, so everyone signs in again once. Accepting session-less tokens for a grace period would have kept the exact hole this change closes, so there is no grace period.

One SQL Codebase, Two Databases

The Problem

GoatFlow inherits the OTRS schema and was written against MariaDB first. PostgreSQL support existed on paper, but 0.9.0 could not even finish migrating a fresh PostgreSQL database. Three migrations had been renumbered on the MySQL side to keep the two sets aligned, and the old PostgreSQL copies were never deleted. golang-migrate saw two files with the same version and stopped with “duplicate migration file”.

Behind that one blocker sat the usual MySQL-isms scattered through handler code: backtick identifiers, INSERT IGNORE, ON DUPLICATE KEY UPDATE, UUID(). And behind those sat a class of bug no linter caught: SQL naming a table or column that exists in one migration set but not the other.

The Solution

Three layers, from cheapest to most thorough.

First, the migration sets were made identical in version numbers, and migrate up now runs every migration on an empty PostgreSQL database. The test database init scripts apply every *.up.sql in order rather than a hand-maintained list, so the test schema can’t fall behind the real one.

Second, a conversion layer. database.ConvertUpsert(query, conflictCols...) lets you write the upsert the MySQL way and name its unique key; on PostgreSQL it becomes ON CONFLICT (cols) DO UPDATE SET ... EXCLUDED.c. ConvertPlaceholders also rewrites backticks, INSERT IGNORE and UUID() for PostgreSQL.

Third, the schema itself became a lint input. Two new gk-lint rules, sql-unknown-table and sql-unknown-column, read both migrations/mysql and migrations/postgres and reject SQL that names a table, or an INSERT/UPDATE column, that doesn’t exist on both. The rules run in the pre-commit gate, so the check happens before review, not after deploy.

The Benefits

The full unit suite now runs against both databases. A new PostgreSQL install works from the first migrate up. And the lint rules turn “it works on my database” from a production discovery into a commit-time error. The one honest caveat stays in the upgrade notes: since 0.9.0 could never migrate on PostgreSQL, there is no 0.9.0 PostgreSQL database to upgrade, and PostgreSQL users start fresh on 0.10.0.

The Plugin Sandbox Stops Trusting Plugins to Be Polite

The Problem

Plugins declare the resources they need in resources.permissions, and the host scopes their SQL to the caller’s organisation. In practice there were ways around both. gRPC plugins bypassed the permission checks that WASM plugins went through. And the organisation scoping could be escaped by the plugin’s own SQL: a query that already mentioned org_id, an ... OR 1=1 predicate, a JOIN or subquery reaching an org-aware table, or an INSERT for another organisation.

The version handshake was also wrong. GKRegistration.MinHostVersion was never checked, and the gRPC runtime told every plugin it was running on GoatFlow 0.6.4, because that value was hard-coded.

The Solution

Every HostAPI call and every SQL statement is now checked against the declared permissions, for both runtimes. Every reference to an org-aware table gets the plugin’s org predicate wrapped in parentheses, so a trailing OR can’t widen it. UPDATE may not change org_id, INSERT must target the caller’s organisation, and SQL shapes that can’t be scoped safely are refused outright instead of passed through.

Plugins now receive the real host version, and a plugin whose MinHostVersion is newer than the host is refused before Init, at boot, on hot reload, on upload and on marketplace install. A refused hot reload leaves the running version in place.

The Benefits

The sandbox is now a property of the host rather than a convention plugins follow. It also turned out to be a useful audit tool: once the host enforced declarations, all 16 first-party plugins needed a commit declaring the permissions they actually use. With those in place, deploying the plugins that weren’t yet on the dev stack against the enforcing host logged zero sandbox refusals.

A Version Number You Can Trust

The Problem

The plugin host-version check is only as good as the version it compares against. While wiring it up, we found the Docker build’s -X linker flags pointed at a package path that had moved, so commit, branch and build date were all compiled in as unknown. Fixing the path naively would have been worse: the build also passed the CI-computed version, which for branch builds is the branch name. A host reporting its version as main or dev treats every plugin as compatible, which silently switches the new check off on exactly the images people test with.

The marketplace had a related shape problem. Each index entry described one version. Once a plugin’s latest release needed 0.10.0, every older host was locked out, and Update didn’t check compatibility at all. It also deleted the installed plugin before downloading the new one, so a failed download or signature check left nothing installed.

The Solution

The version now has exactly one source: the constant in internal/platform/version/version.go, set by make prepare-release and pinned against the Helm chart, TrueNAS app, README and CHANGELOG by a consistency test. The build injects only commit, branch and date, at the correct package path. The release job fails if the pushed tag doesn’t match the constant.

Marketplace entries gained an optional versions list with a min_host_version per release. Older clients ignore the field, so the index schema version didn’t change. Install picks the newest release the host can run, Update offers the newest compatible one, and both download and verify into a staging folder, swapping the new version in only after the signature checks out.

The Benefits

A host that can’t run the newest release of a plugin still finds one it can. A failed update leaves the working plugin alone. And the version a build reports is now the version it is, which is what lets the plugin compatibility check be strict without being a trap.

Upgrades That Stop When They Should

The Problem

Three separate upgrade paths each had the same flaw: when something didn’t match, they guessed.

When schema_migrations was left dirty by a failed migration, every GoatFlow process forced that version as applied at startup and kept migrating. The failed migration was skipped for good, and the upgrade came up on a half-migrated schema without saying so.

Docker Compose keeps config/plugins on a volume, so bundled WASM plugins stayed at whatever version first filled the volume. A 0.9.0 install upgraded to 0.10.0 kept running the old stats plugin without its queue-permission fixes.

And GOATFLOW_SECURE_KEY had an empty default, so each process generated its own key. The backend and the runner ended up with different keys, and the runner couldn’t decrypt webhook signing secrets.

The Solution

A dirty migration now stops migration, logs the version and the recovery command (migrate force <version> once it’s fully applied, or the previous version to re-run it), and leaves the flag for an operator.

The image carries a pristine copy of the bundled plugins. At startup GoatFlow installs each bundled file that is missing or is still an untouched bundled copy, tracked by hash in a manifest. A file an admin changed is kept and logged. The sync uses Go’s os.Root, so a symlink in the operator-writable plugin volume that points elsewhere stops the sync with a warning instead of being written through. gosec flagged the original path-joining version in CI. Rewriting it on os.Root closed the real issue, where a #nosec comment would only have silenced the warning.

Every compose file now requires GOATFLOW_SECURE_KEY and refuses to start without it. CI generates a random key per test run, because its databases are thrown away anyway.

The Benefits

The first real test was the public demo. Before tagging, we copied the demo’s database dump and compose file to a workstation and ran the old setup and the new deploy side by side. The rehearsal found that reloading the seed over a newer schema broke migration 18 with “Duplicate column name”, and that demo mode was never passed to the containers. With migrations that refuse to guess, the failure was loud and specific instead of a quietly broken schema. The reset script now drops each seeded database before loading the dump, the release workflow writes a persistent secure key the first time it deploys, and 0.10.0 deployed to the demo cleanly on the tag.

Lessons

The common thread in this release is that the dangerous failures were the quiet ones. A killed session that kept working, a migration marked done that wasn’t, a version string that read dev, a plugin that kept its old code after an upgrade: none of these crashed. Each one produced a system that looked healthy and wasn’t. The fixes share a shape too. When the system can’t be sure, it stops and says why, rather than picking the answer that keeps things running.

The second lesson is about where checks live. Every guarantee in this release that held up is enforced at a boundary the code under test can’t skip: middleware on every route, a sandbox in the host rather than the plugin, a lint rule that reads the schema, a release job that compares the tag with the constant. Conventions such as “plugins should scope their queries” or “remember to bump the version” fail the first time someone is in a hurry. Boundaries don’t.

The last lesson is the cheapest one. Rehearsing the production upgrade against a copy of production data found two failures that the full test suite never would have, because the test suite starts from an empty database and production never does. It is now part of our routine before every tag.

Photos: CC0 from Wikimedia Commons. Asian elephants by Mammalwatcher, playground by Bernard Spragg, stop sign by Matthew Paul Argall. Screenshots from the GoatFlow dev stack.