All reports

Redis provisioning inconsistency: firewall allowlist not enforced on new instances

WAYSCLOUD-TR-2026-0005Operational DeviationmediumResolved
Published: 2026-04-03 11:01:47 UTC Updated: 2026-04-03 11:12:15 UTC
Event: Apr 3, 2026 — Apr 3, 2026

Summary

A provisioning inconsistency caused firewall allowlist rules to not be enforced at the network level, despite appearing correctly configured in the dashboard. The issue primarily affected newly provisioned Redis instances where custom firewall rules were applied. Existing instances and instances without custom allowlist configuration were not affected.

What Happened

On April 3, 2026, a connection timeout was reported on a newly provisioned Redis instance in the Norway region. The IP address had been correctly added to the firewall allowlist through the dashboard, but the connection could not be established.

Investigation revealed a multi-layer inconsistency between the control plane database, the host-level firewall agent, and the runtime enforcement layer. The firewall synchronization script used strict error handling that caused it to terminate silently when processing a new instance with no pre-existing rules. The agent layer did not validate the script outcome and returned a success response regardless. The control plane committed the database change despite the failed enforcement, resulting in a silent divergence between the recorded configuration and the actual network state.

The issue was scoped to the provisioning and firewall enforcement pipeline for newly created instances. Existing established connections were not disrupted. There was no data exposure, no unauthorized access, and no integrity impact.

Impact

Newly provisioned Redis instances where custom firewall rules were applied during or after creation were affected. The allowlist appeared correctly configured in the dashboard, but was not enforced at the network level, preventing any connection from being established. Instances without custom allowlist entries and previously established connections were not affected. No data was exposed and no unauthorized access occurred.

Actions Taken

The firewall synchronization script was corrected to handle the edge case that caused early termination. The host agent was updated to validate execution results and surface failures explicitly. The control plane API was modified to roll back database changes when downstream enforcement fails, ensuring the recorded state and actual network state remain consistent at all times.

Firewall rules were verified consistent across the database, the agent state, and the live enforcement layer for all active instances.

Preventive Measures

Silent failure paths have been eliminated across all three layers of the provisioning stack. Database transactions are now atomically tied to successful downstream enforcement — if any layer fails, the entire operation is rolled back and the failure is surfaced to the caller.

Boot persistence has been hardened with a dedicated service that restores firewall rules from the persisted state on every system restart, with correct service ordering dependencies. The provisioning pipeline now applies the same enforcement guarantees to all firewall operations, for both new and existing instances.

Affected Services

redis

Timeline

Apr 3, 2026, 09:21 UTC
All times are shown in UTC. Live updates published during the incident are available on the WAYSCloud Status page.
Apr 3, 2026, 09:21 UTC
Investigating
Connection issues reported affecting newly provisioned Redis instances. Customers unable to establish connections despite correctly configured allowlists.
Apr 3, 2026, 09:22 UTC
Investigating
Issue isolated to Redis instances in the Norway region. Existing instances and other regions not affected. Investigation confirmed allowlist rules present in the control plane but not enforced at the network level.
Apr 3, 2026, 09:24 UTC
Identified
Root cause identified as a multi-layer enforcement failure in the provisioning pipeline. The firewall synchronization script, host agent, and control plane API each contained independent failure modes that combined to create a silent divergence between configured and actual state.
Apr 3, 2026, 09:33 UTC
Resolved
All fixes deployed and verified. Firewall enforcement corrected across all layers with explicit failure propagation and transactional rollback. Firewall state verified consistent across control plane, agent, and network enforcement layer. All Redis instances in the Norway region are fully operational.