how to resolve flaky rspecs?

I have this Ruby Rspec randomly failing on my CICD (doesn't fail locally).

This is a minified part of my RSpec:

require "rails_helper"

RSpec.describe "Admin::Vendors", type: :request do
  let!(:vendors) {
    create_list(:vendor, 5, type: 1) do |vendor|
      create(:vendor_user, vendor_id: vendor.id, role: "vendor")
    end

    create_list(:vendor, 5, type: 2) do |vendor|
      create(:vendor_user, vendor_id: vendor.id, role: "vendor")
    end

    create_list(:vendor, 5, status: "waiting_for_approval") do |vendor|
      create(:vendor_user, vendor_id: vendor.id, role: "vendor")
    end

    create_list(:vendor, 5, status: "inactive") do |vendor|
      create(:vendor_user, vendor_id: vendor.id, role: "vendor")
    end
  }
  
  describe "index" do
    context "(success - display the list of vendors by vendor name asc)" do
      before { get "/api/v1/vendor_management/admin/vendors?sort_column=vendor_name&sort_by=asc", headers: headers[:auth] }

      it "returns list of vendors in asc order" do
        asc_vendor_names = json["vendors"].pluck("name")
        expect(json).not_to be_empty
        expect(json["vendors"]).not_to be_empty
        expect(response).to have_http_status(200)
        expect(asc_vendor_names).to eq(asc_vendor_names.sort)
      end
    end

    context "(success - display the list of vendors by vendor name desc)" do
      before { get "/api/v1/vendor_management/admin/vendors?sort_column=vendor_name&sort_by=desc", headers: headers[:auth] }

      it "returns list of vendors in desc order" do
        desc_vendor_names = json["vendors"].pluck("name")
        expect(json).not_to be_empty
        expect(json["vendors"]).not_to be_empty
        expect(response).to have_http_status(200)
        expect(desc_vendor_names).to eq(desc_vendor_names.sort.reverse)
      end
    end
  end

end

and for some reason, 70% of the times, I get this failure in my pipeline:

Failures:

  1) Admin::VendorManagement::VendorController index (success - display the list of vendors by vendor name desc) returns list of vendors in desc order
     Failure/Error: expect(desc_vendor_names).to eq(desc_vendor_names.sort.reverse)
     
       expected: ["Wyatt Conn", "Vikki Rowe VM", "Rocco Bauch", "Randal Muller", "Noe Koch", "Natashia Stark", "Miss L...oe Walter", "Dwain Stoltenberg", "Chara Orn", "Benito Hegmann", "Armida Kuphal", "Agustin Mosciski"]
            got: ["Wyatt Conn", "Vikki Rowe VM", "Rocco Bauch", "Randal Muller", "Noe Koch", "Natashia Stark", "Miss L...h Murazik", "Dwain Stoltenberg", "Chara Orn", "Benito Hegmann", "Armida Kuphal", "Agustin Mosciski"]
     
       (compared using ==)

I can not figure out what can possibly cause this corrupted data? how to solve this? Thanks!

Solution

It's difficult to give a concrete answer to this, because your question lacks key information. In particular, the error message says:

..., "Natashia Stark", "Miss L...oe Walter", "Dwain Stoltenberg", ...
..., "Natashia Stark", "Miss L...h Murazik", "Dwain Stoltenberg", ...

But what is that critical difference between ...oe Walter and ...h Murazik? Which value(s) are actually in a different order, and therefore causing the failure?

You could log the full lines to see exactly what's changed. My prediction is that it's something like mr Tom... vs Mr Bob... where the first letter is lower-case.

Anyway, the failing line in the test is:

expect(desc_vendor_names).to eq(desc_vendor_names.sort.reverse)

The only way I think this can fail (unless something really weird is happening) is that the left side is sorting alphabetically in SQL, and the right side is sorting alphabetically in ruby.

Why does this matter? Because ruby's .sort is based on UTF8 codepoints, whereas your database is probably using some other collation. This can cause various subtle differences in the sort order when the strings contain accented characters (é, ø, į, ...) or upper-case vs lower-case characters.

For example, given the strings: ['test 1', 'Test 2'], these will (probably) be sorted in a different order in ruby vs the database.

So how can you fix the test?

My recommendation would be to just ensure your factories/fixtures/initialiser is using names that won't cause issues - i.e. nothing that starts with a lower-case letter or "unusual" accented character, so that your ruby and SQL sort orders are the same.

Alternatively, you could use something a bit more robust to make ruby's .sort behave the same as SQL, such as:

# probably good enough!
.sort_by(&:downcase)

# Also accounts for most accented character issues:
.sort do |a, b|
  I18n.transliterate(a.downcase) <=> I18n.transliterate(b.downcase)
end

# Hardcore solution:
https://github.com/ninjudd/icunicode