Database service degradation due to storage saturation

A database node reached full storage capacity, which caused the underlying database system to enter a degraded state.

This affected multiple dependent services relying on database access, including API operations and security-related services.

The storage growth was caused by two combined issues:

• Backup jobs were executed more frequently than intended due to a scheduling configuration issue
• Automated retention cleanup did not execute correctly due to a logic error in timestamp handling

As a result, backup data accumulated over time without being removed.

During the incident:

• Database-backed services experienced outages and degraded performance
• Some operations failed or were delayed
• In certain cases, duplicate operations could occur due to retries during degraded system state

No customer data was lost.

The issue was resolved through the following steps:

• Storage capacity was restored by removing excess backup data
• Backup scheduling was corrected to the intended frequency
• Retention logic was fixed to ensure proper cleanup of old backups
• All affected services were verified and returned to normal operation

The following improvements have been implemented:

• Validation of backup scheduling configuration to prevent unintended execution frequency
• Correction of timestamp handling in retention logic
• Improved monitoring of storage utilization
• Introduction of safeguards to prevent uncontrolled data growth
• Strengthened validation of system health during backup operations