"duplicate key value violates unique constraint " error in CDN and CDN pods continously restarting #7155
Description
Hello,
Occassionally we face this error:
CDS fails with CDN pods continuously restarting. In the logs for the failing CDN pods, error messages similar to the following can be seen:
Data conflict (from: pq: duplicate key value violates unique constraint "idx_service_auth_consumer_id", request_id: [...]
In addition, the API logs contain lines similar to the following:
service.get> service <SERVICE_ID> data corrupted
service.getAll> service <SERVICE_ID> data corrupted
NAME READY STATUS RESTARTS AGE
cds-api-6b7b5ddbc6-b2rvz 2/2 Running 59 (5h32m ago) 7d6h
cds-api-6b7b5ddbc6-c2fqd 2/2 Running 56 (17h ago) 7d6h
cds-api-6b7b5ddbc6-zmhtb 2/2 Running 59 (17h ago) 7d6h
cds-cdn-66f957f49-28jw8 1/2 CrashLoopBackOff 1343 (3m11s ago) 7d6h
cds-cdn-66f957f49-2hdr9 1/2 CrashLoopBackOff 1345 (3m10s ago) 7d6h
cds-cdn-66f957f49-lh5q8 1/2 Running 1343 (5m21s ago) 7d6h
We use Postgres in HA mode (3 pods for server, 3 pods for sentinel and 3 pods for proxy).
The keeper pods write the same errors as CDN.
Can you please look into it?