Attempting to rescue a PG::UniqueViolation error from a block in a Sidekiq (7.2.1) job in Rails (7.0.8) dev env, running on Ruby 3.2.2 revision e51014f9c0 with a local PostgreSQL 14.10 db on Ubuntu 20.04 fails. I'm launching the job from the console with InitThingsJob.new.perform_now while testing, with a known duplicate at the first position.
Attempting to rescue ActiveRecord::RecordNotUnique or rescue PG::UniqueViolation, ActiveRecord::RecordNotUnique similarly fails.
The failing function operates on responses received from an external API that contain collections of objects. Sometimes, but very rarely, the external API has duplicate entries. We can't have those, so the DB has a unique index on a reliable id field.
It is intended to create an array from the collection in the response, then iterate through the array in a block and create a record each element. If an element is successfully created from an element then it shifts the array to remove that element. If a PG::UniqueViolation error is raised, then it rescues it, shifts the array to remove the element that caused the error, and retries the block with the truncated array.
def create_x_from_y_query
response_objs = @response.body["data"]["objects"]
begin
response_objs.each do |obj|
@obj = Obj.new(k_id: obj["k_id"], etc: obj["etc"])
@obj.save
#... other things...
response_objs.shift
end
rescue PG::UniqueViolation
logger.warn "Warning stuff."
response_objs.shift
retry
end
end
Expected behaviour is to rescue from the PG::UniqueViolation error, shift the element of the enumerable that creates the error, then retry the block with the truncated array.
What occurs is the job exiting with error PG::UniqueViolation when a duplicate is encountered, returning:
E, [2024-01-23T13:57:36.755697 #1213779] ERROR -- : Error performing InitThingsJob (Job ID: 1312b3a9-8359-4cb6-a16d-8442370d815b) from Sidekiq(default) in 1548.0ms: ActiveRecord::RecordNotUnique (PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_obj_on_k_id"
DETAIL: Key (k_id)=(<value>) already exists.
):
/home/xxx/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params'
... Down the trace until we arrive at:
/home/xxx/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_obj_on_k_id" (ActiveRecord::RecordNotUnique)
DETAIL: Key (k_id)=(<value>) already exists.
/home/xxx/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': ERROR: duplicate key value violates unique constraint "index_objs_on_k_id" (PG::UniqueViolation)
DETAIL: Key (k_id)=(<value>) already exists.
Some replies to Handling unique constraint exceptions inside transaction suggest that their problem arises from concurrency and to handle the rescue outside the transaction block to resolve it or to rescue the ActiveRecord error instead of the PG error, but neither appear to work here and the error output isn't the same.
Rescuing PG::UniqueViolation will definitely fail, because the actual exception you'll see is ActiveRecord::RecordNotUnique. I'm guessing that when you added that to the code, you didn't restart your sidekiq server, so the change wasn't picked up because that WILL rescue the exception.
That being said, your whole .shift thing won't work as you expect. By doing the .shift inside the each loop you'll actually be skipping every second element of the array:
> arr = (0..5).to_a
> arr.each { |e| puts e; arr.shift }
0
2
4