← Back to articles
Technology

Test Data Like Production

Test Data Like Production

It worked on your machine, and then the screen broke the moment it hit production. Many of these bugs come from differences in test data, not from the code. This article covers four things to match when you build test data for the testing phase, and how to use anonymized production data safely.

Why clean data is not enough

Development data is mostly a handful of rows typed by hand: short names, complete addresses, around ten records. Real users match none of that. Your tests only prove the inputs you imagined, and the inputs that exist only in production have never been tried.

Four things to match

1. Volume

Problems invisible at 10 rows appear at 10,000: slow lists, a row lost at a page boundary, searches that time out.

  • Seed main lists with 1x to 2x the expected production count
  • Create the pagination boundaries (exactly 20 and 21 records)
  • Check the empty state with zero records

2. Skewed distribution

Production data is never uniform. A few users own most of the data and most users own little. Evenly generated data cannot reproduce "slow only for one customer".

  • Make 10% of users hold most of the records
  • Deliberately include one record with the longest and largest values

3. Missing values

Optional fields are empty in production. Code written on the assumption that a value "can't be null" fails right there.

  • Cover three cases for optional fields: missing, null, and empty string
  • Include users with no avatar or profile set

4. Mixed legacy formats

After years of spec changes, production holds old formats: phone numbers with and without hyphens, full-width and half-width characters, dates saved by an older version. Data created only under the current spec cannot reproduce this mix.

Try it: a seed script

Do not build data by hand. Encode the four points in a generator. Here is a Ruby example.

users = 10_000.times.map do |i|
  {
    name: i % 50 == 0 ? "a" * 200 : "user#{i}",
    phone: [nil, "", "03-1234-5678", "0312345678"].sample,
    posts_count: i < 1000 ? rand(500..5000) : rand(0..5)
  }
end

This one includes long names (extreme values), empty and full-width phone numbers (missing and mixed formats), and counts skewed to the top 10% (distribution), all inside 10,000 records (volume).

Using anonymized production data

A copy of production is the most realistic source, but never copy it as is.

  1. Replace names, emails, phone numbers, and addresses with other values that keep the same format
  2. Replace consistently, so the same person always maps to the same fake value, or related records break
  3. Delete or template free-text fields, because they often contain personal data
  4. Keep only the anonymized data in development and never bring the original

Common mistakes

  • Anonymizing every email to one value and missing a duplicate-check bug
  • Creating data once and never refreshing it, so new columns stay empty
  • Giving only the test account clean data, so nobody notices the gap with production

Putting it into practice with Bugoon

Some production-only bugs will still slip through. What matters then is capturing the situation at that moment. With the Bugoon widget embedded, a reporter can annotate a screenshot and send it with the recorded steps. The report becomes a GitHub Issue you can manage on the kanban board, and the data that triggered it can be added to your four checks for next time.

Streamline bug reporting for your team.

Bugoon is free to get started. Add one line of code to your site and transform how your team handles bugs.

Get Started