← Back to articles
Technology

Why Nobody Trusts Staging

Why Nobody Trusts Staging

"It passed on staging, but broke in production." After this happens a few times, the team stops trusting staging. People skip checks because "it differs from production anyway," and bugs reach users. This article is about the release stage: why staging loses trust, and the order in which to rebuild it.

Four ways staging loses trust

1. The data is stale or empty

The data is a copy from six months ago, so the records a new feature needs don't exist. Or it is empty, and lists and reports can't be checked at all. Either way, you get "verified" results that mean nothing.

2. It is down or occupied

It doesn't respond in the morning, or another project's branch is still deployed and nobody knows what is running. Being made to wait is a reason not to come back.

3. It differs from production in unknown ways

One instance instead of several, every external service mocked, some environment variables missing. If nobody can say what differs, nobody can explain what a pass actually means.

4. Results are not shared

The outcome of a check lives only in one person's head. The next person either repeats it or skips it.

The order to rebuild in

You don't need to make everything identical to production. That costs too much. Work in this order:

  1. List the current differences. Write down what differs between production and staging.
  2. Decide which differences are acceptable. Instance count and size may be fine. Database engine and version, the authentication mechanism, and the set of environment variables are not.
  3. Automate data refresh. Make it a scheduled job, not a manual task.
  4. Set rules for occupancy. Make it visible who has deployed what.
  5. Pick one place to record results.

A template for the difference table

Filling in this table is the first step.

  • Database: engine and version, order of magnitude of record counts
  • External services: real, sandbox, or mock
  • Environment variables and settings: missing or extra keys (compare names, not values)
  • Infrastructure: instance count, memory, timeout values
  • Authentication and permissions: role types and test accounts
  • Email and notifications: confirm nothing reaches real recipients

You can compare key names mechanically:

# Compare only the variable NAMES between production and staging (no values printed)
diff <(cut -d= -f1 prod-vars.txt | sort) <(cut -d= -f1 staging-vars.txt | sort)

Values will differ, which is expected. A name that exists on one side only is a misconfiguration waiting to become a bug.

Keeping data trustworthy

Copying production data as-is risks leaking personal information, which is a different incident. Choose one of these:

  • An anonymized copy of production: replace names, email addresses, phone numbers and free-text fields, then regenerate it on a schedule such as weekly
  • Generated data: a script that produces data close to production in volume, skew, and missing values, the same way every time

In either case, show the "last refreshed" date somewhere visible. It reduces the chance of testing against stale data.

Balancing cost

A full-size environment running all the time is expensive. There are three common compromises:

  • Stop it at night and on weekends, and start it automatically in the morning
  • Scale it up to production size only during the pre-release verification window
  • Create a temporary environment per change and delete it automatically after merge

Whichever you pick, everyone must know when it will be off. An environment that stops without notice gets treated like one that is broken.

A failure example: "it should have passed"

Staging mocked an external payment service. In production, the real service sometimes timed out, and the code for that case had never been written. The mock always returned success, so staging could never reveal it. If you use mocks, add modes that return failures and delays as a minimum.

What to do this week

  1. Build the difference table and mark the differences you will not accept
  2. Show the data's last refresh date somewhere
  3. Decide where verification results are recorded, for example in tickets

Putting it into practice with Bugoon

A rebuilt staging environment only stays in use if reporting what you find is easy. Embed the Bugoon widget in your staging site, and the person verifying can report straight from the screen with a screenshot and annotations, along with a recording of the steps they took. Reports can be linked to GitHub Issues and tracked on the kanban board, so results no longer live only in someone's head. Each report also carries environment details such as browser and screen size, which helps you tell whether a problem comes from a difference from production.

Streamline bug reporting for your team.

Bugoon is free to get started. Add one line of code to your site and transform how your team handles bugs.

Get Started