Skip to content

Resilience/dns blackhole clean #345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Resilience/dns blackhole clean #345

wants to merge 2 commits into from

Conversation

@jameslaneovermind
Copy link
Contributor

No description provided.

@jameslaneovermind
Add CI fix: disable state locking
6ea6b37
@jameslaneovermind
Disable DNS failover and route ALB listener to empty target group
28fceb1 
This change simulates the AWS DNS outage scenario by:

1. Creating a blackhole target group with no registered targets
2. Updating ALB listener to forward all traffic to the empty target group
3. Adding Route53 DNS record with no failover capability and disabled health checks

This mimics the scenario where DNS endpoint resolves but has no healthy
backends, causing immediate 503 errors and service unavailability. The
change removes failover protection and routes traffic to a target group
with zero healthy targets.

Impact: High - complete service outage, no automatic failover capability
@github-actions
Copy link

Overmind

Open in Overmind ↗

model|risks_v6

🔴 Change Signals

Routine 🔴 ▇▅▃▂▁ AWS Route 53 and Load Balancer target groups showing first ever modifications across multiple attributes, which is unusual compared to typical patterns.

View signals ↗

🔥 Risks

Removing listener default forward target group will leave ALB with no backend and cause 503s ‼️High Open Risk ↗
The ALB listener on port 80 for scenarios--a3ec77f7-alb is changing its default action from explicitly forwarding to target group scenarios--a3ec77f7-tg to a type=forward action with no ForwardConfig or TargetGroupArn specified. The only existing rule is the default rule that mirrors the listener’s default action; no other rules are present to take over routing.

When this applies, the listener will have no backend target group to forward to, so requests to the ALB DNS will return 503 and the existing targets behind scenarios--a3ec77f7-tg will stop receiving traffic. This will cause immediate service downtime and likely trigger alarms that monitor unhealthy target counts for that target group.

🟣 Expected Changes

+/- ec2-instance › i-01dcfcab37ba22689 
--- current
+++ proposed
@@ -2,72 +2,76 @@
 id: github.com/overmindtech/terraform-example.ec2-instance.module.scenarios[0].aws_instance.app_server
 attributes:
-  ami: ami-0f802dc0fc1809acd
-  arn: arn:aws:ec2:eu-west-2:540044833068:instance/i-01dcfcab37ba22689
+  ami: (sensitive value)
+  arn: (known after apply)
   associate_public_ip_address: true
-  availability_zone: eu-west-2b
-  capacity_reservation_specification:
-    - capacity_reservation_preference: open
-  cpu_core_count: 1
-  cpu_options:
-    - core_count: 1
-      threads_per_core: 2
-  cpu_threads_per_core: 2
-  credit_specification:
-    - cpu_credits: unlimited
-  disable_api_stop: false
-  disable_api_termination: false
-  ebs_optimized: false
-  enable_primary_ipv6: null
-  enclave_options:
-    - enabled: false
+  availability_zone: (known after apply)
+  capacity_reservation_specification: (known after apply)
+  cpu_core_count: (known after apply)
+  cpu_options: (known after apply)
+  cpu_threads_per_core: (known after apply)
+  disable_api_stop: (known after apply)
+  disable_api_termination: (known after apply)
+  ebs_block_device: (known after apply)
+  ebs_optimized: (known after apply)
+  enable_primary_ipv6: (known after apply)
+  enclave_options: (known after apply)
+  ephemeral_block_device: (known after apply)
   get_password_data: false
-  hibernation: false
-  host_resource_group_arn: null
-  id: i-01dcfcab37ba22689
-  instance_initiated_shutdown_behavior: stop
-  instance_state: running
+  hibernation: null
+  host_id: (known after apply)
+  host_resource_group_arn: (known after apply)
+  iam_instance_profile: (known after apply)
+  id: (known after apply)
+  instance_initiated_shutdown_behavior: (known after apply)
+  instance_lifecycle: (known after apply)
+  instance_market_options: (known after apply)
+  instance_state: (known after apply)
   instance_type: t3.small
-  ipv6_address_count: 0
+  ipv6_address_count: (known after apply)
+  ipv6_addresses: (known after apply)
   key_name: Demo Key Pair
-  maintenance_options:
-    - auto_recovery: default
-  metadata_options:
-    - http_endpoint: enabled
-      http_protocol_ipv6: disabled
-      http_put_response_hop_limit: 1
-      http_tokens: optional
-      instance_metadata_tags: disabled
-  monitoring: false
-  placement_partition_number: 0
-  primary_network_interface_id: eni-0501ad33e98bb6f8c
-  private_dns: ip-10-0-10-239.eu-west-2.compute.internal
-  private_dns_name_options:
-    - enable_resource_name_dns_a_record: false
-      enable_resource_name_dns_aaaa_record: false
-      hostname_type: ip-name
-  private_ip: 10.0.10.239
-  public_dns: ec2-13-41-66-30.eu-west-2.compute.amazonaws.com
-  public_ip: 13.41.66.30
+  maintenance_options: (known after apply)
+  metadata_options: (known after apply)
+  monitoring: (known after apply)
+  network_interface: (known after apply)
+  outpost_arn: (known after apply)
+  password_data: (known after apply)
+  placement_group: (known after apply)
+  placement_partition_number: (known after apply)
+  primary_network_interface_id: (known after apply)
+  private_dns: (known after apply)
+  private_dns_name_options: (known after apply)
+  private_ip: (known after apply)
+  public_dns: (known after apply)
+  public_ip: (known after apply)
   root_block_device:
     - delete_on_termination: true
-      device_name: /dev/xvda
-      encrypted: false
-      iops: 0
-      throughput: 0
-      volume_id: vol-011ca7661217b5823
-      volume_size: 8
-      volume_type: standard
+      device_name: (known after apply)
+      encrypted: (known after apply)
+      iops: (known after apply)
+      kms_key_id: (known after apply)
+      tags: null
+      tags_all: (known after apply)
+      throughput: (known after apply)
+      volume_id: (known after apply)
+      volume_size: (known after apply)
+      volume_type: (known after apply)
+  secondary_private_ips: (known after apply)
+  security_groups: (known after apply)
   source_dest_check: true
+  spot_instance_request_id: (known after apply)
   subnet_id: subnet-036704734045071f9
   tags:
+    Environment: dev
     Name: App Server
   tags_all:
+    Environment: dev
     Name: App Server
-  tenancy: default
+  tenancy: (known after apply)
   terraform_address: module.scenarios[0].aws_instance.app_server
   terraform_name: module.scenarios[0].aws_instance.app_server
   timeouts: null
-  user_data: null
-  user_data_base64: null
+  user_data: (known after apply)
+  user_data_base64: (known after apply)
   user_data_replace_on_change: false
   volume_tags: null
+/- ec2-instance › i-02f292e8a0766d313 
--- current
+++ proposed
@@ -2,72 +2,76 @@
 id: github.com/overmindtech/terraform-example.ec2-instance.module.scenarios[0].aws_instance.webserver
 attributes:
-  ami: ami-0f802dc0fc1809acd
-  arn: arn:aws:ec2:eu-west-2:540044833068:instance/i-02f292e8a0766d313
+  ami: (sensitive value)
+  arn: (known after apply)
   associate_public_ip_address: true
-  availability_zone: eu-west-2a
-  capacity_reservation_specification:
-    - capacity_reservation_preference: open
-  cpu_core_count: 1
-  cpu_options:
-    - core_count: 1
-      threads_per_core: 2
-  cpu_threads_per_core: 2
-  credit_specification:
-    - cpu_credits: unlimited
-  disable_api_stop: false
-  disable_api_termination: false
-  ebs_optimized: false
-  enable_primary_ipv6: null
-  enclave_options:
-    - enabled: false
+  availability_zone: (known after apply)
+  capacity_reservation_specification: (known after apply)
+  cpu_core_count: (known after apply)
+  cpu_options: (known after apply)
+  cpu_threads_per_core: (known after apply)
+  disable_api_stop: (known after apply)
+  disable_api_termination: (known after apply)
+  ebs_block_device: (known after apply)
+  ebs_optimized: (known after apply)
+  enable_primary_ipv6: (known after apply)
+  enclave_options: (known after apply)
+  ephemeral_block_device: (known after apply)
   get_password_data: false
-  hibernation: false
-  host_resource_group_arn: null
-  id: i-02f292e8a0766d313
-  instance_initiated_shutdown_behavior: stop
-  instance_state: running
+  hibernation: null
+  host_id: (known after apply)
+  host_resource_group_arn: (known after apply)
+  iam_instance_profile: (known after apply)
+  id: (known after apply)
+  instance_initiated_shutdown_behavior: (known after apply)
+  instance_lifecycle: (known after apply)
+  instance_market_options: (known after apply)
+  instance_state: (known after apply)
   instance_type: t3.small
-  ipv6_address_count: 0
+  ipv6_address_count: (known after apply)
+  ipv6_addresses: (known after apply)
   key_name: Demo Key Pair
-  maintenance_options:
-    - auto_recovery: default
-  metadata_options:
-    - http_endpoint: enabled
-      http_protocol_ipv6: disabled
-      http_put_response_hop_limit: 1
-      http_tokens: optional
-      instance_metadata_tags: disabled
-  monitoring: false
-  placement_partition_number: 0
-  primary_network_interface_id: eni-0784f95b7ff052c6b
-  private_dns: ip-10-0-9-25.eu-west-2.compute.internal
-  private_dns_name_options:
-    - enable_resource_name_dns_a_record: false
-      enable_resource_name_dns_aaaa_record: false
-      hostname_type: ip-name
-  private_ip: 10.0.9.25
-  public_dns: ec2-13-40-28-149.eu-west-2.compute.amazonaws.com
-  public_ip: 13.40.28.149
+  maintenance_options: (known after apply)
+  metadata_options: (known after apply)
+  monitoring: (known after apply)
+  network_interface: (known after apply)
+  outpost_arn: (known after apply)
+  password_data: (known after apply)
+  placement_group: (known after apply)
+  placement_partition_number: (known after apply)
+  primary_network_interface_id: (known after apply)
+  private_dns: (known after apply)
+  private_dns_name_options: (known after apply)
+  private_ip: (known after apply)
+  public_dns: (known after apply)
+  public_ip: (known after apply)
   root_block_device:
     - delete_on_termination: true
-      device_name: /dev/xvda
-      encrypted: false
-      iops: 0
-      throughput: 0
-      volume_id: vol-0bc99e157a38768b6
-      volume_size: 8
-      volume_type: standard
+      device_name: (known after apply)
+      encrypted: (known after apply)
+      iops: (known after apply)
+      kms_key_id: (known after apply)
+      tags: null
+      tags_all: (known after apply)
+      throughput: (known after apply)
+      volume_id: (known after apply)
+      volume_size: (known after apply)
+      volume_type: (known after apply)
+  secondary_private_ips: (known after apply)
+  security_groups: (known after apply)
   source_dest_check: true
+  spot_instance_request_id: (known after apply)
   subnet_id: subnet-06302fc5a50644cd9
   tags:
+    Environment: dev
     Name: Webserver
   tags_all:
+    Environment: dev
     Name: Webserver
-  tenancy: default
+  tenancy: (known after apply)
   terraform_address: module.scenarios[0].aws_instance.webserver
   terraform_name: module.scenarios[0].aws_instance.webserver
   timeouts: null
-  user_data: null
-  user_data_base64: null
+  user_data: (known after apply)
+  user_data_base64: (known after apply)
   user_data_replace_on_change: false
   volume_tags: null
~ ec2-launch-template › lt-0731f767e6be2ab94 
--- current
+++ proposed
@@ -7,7 +7,7 @@
   disable_api_termination: false
   id: lt-0731f767e6be2ab94
-  image_id: ami-0f802dc0fc1809acd
+  image_id: ami-0ca1753a2af8d9bbf
   instance_type: t3.micro
-  latest_version: 20
+  latest_version: (known after apply)
   name: asg-change-launch-template-terraform-example20240827194210168200000007
   name_prefix: asg-change-launch-template-terraform-example
~ elbv2-listener › arn:aws:elasticloadbalancing:eu-west-2:540044833068:listener/app/scenarios--a3ec77f7-alb/a0268d67b29039c7/af5e943a0e473e91 
--- current
+++ proposed
@@ -6,12 +6,6 @@
   certificate_arn: null
   default_action:
-    - forward:
-        - stickiness:
-            - duration: 0
-              enabled: false
-          target_group:
-            - arn: arn:aws:elasticloadbalancing:eu-west-2:540044833068:targetgroup/scenarios--a3ec77f7-tg/202d88113aa8b778
-              weight: 1
-      order: 1
+    - order: 1
+      target_group_arn: (known after apply)
       type: forward
   id: arn:aws:elasticloadbalancing:eu-west-2:540044833068:listener/app/scenarios--a3ec77f7-alb/a0268d67b29039c7/af5e943a0e473e91
~ ec2-route-table › rtb-0c52db7871965c5a1 
--- current
+++ proposed
@@ -7,9 +7,9 @@
   owner_id: "540044833068"
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
~ ec2-address › 3.11.31.83 
--- current
+++ proposed
@@ -18,9 +18,9 @@
   public_ipv4_pool: amazon
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
~ ec2-address › 18.134.176.13 
--- current
+++ proposed
@@ -18,9 +18,9 @@
   public_ipv4_pool: amazon
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
~ ec2-internet-gateway › igw-0b7151f8472d03c8a 
--- current
+++ proposed
@@ -6,9 +6,9 @@
   owner_id: "540044833068"
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"
~ ec2-nat-gateway › nat-0f789c96969ec0dd1 
--- current
+++ proposed
@@ -12,9 +12,9 @@
   subnet_id: subnet-0b805a32f5d7f0c7b
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
~ ec2-nat-gateway › nat-06036dc6d716438e0 
--- current
+++ proposed
@@ -12,9 +12,9 @@
   subnet_id: subnet-016bfadacc9c60bfc
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
~ ec2-route-table › rtb-07f5933d73ceaab99 
--- current
+++ proposed
@@ -9,9 +9,9 @@
       nat_gateway_id: nat-0f789c96969ec0dd1
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
~ ec2-route-table › rtb-09d0b7c0ce1121c2d 
--- current
+++ proposed
@@ -9,9 +9,9 @@
       nat_gateway_id: nat-06036dc6d716438e0
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
~ ec2-route-table › rtb-0536cdbeadfe92efa 
--- current
+++ proposed
@@ -9,9 +9,9 @@
       gateway_id: igw-0b7151f8472d03c8a
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public
     Terraform: "true"
~ ec2-subnet › subnet-0d0a1aff83bd8a460 
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
~ ec2-subnet › subnet-0303f6ca155877094 
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
~ ec2-subnet › subnet-0b805a32f5d7f0c7b 
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2a
     Terraform: "true"
~ ec2-subnet › subnet-016bfadacc9c60bfc 
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2b
     Terraform: "true"
~ ec2-vpc › vpc-0f4ddbf8c33e5c725 
--- current
+++ proposed
@@ -20,9 +20,9 @@
   owner_id: "540044833068"
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"

🟠 Unmapped Changes

+ aws_route53_record › module.scenarios[0].aws_route53_record.blackhole[0] 
--- current
+++ proposed
@@ -0,0 +1,21 @@
+type: aws_route53_record
+id: github.com/overmindtech/terraform-example.aws_route53_record.module.scenarios[0].aws_route53_record.blackhole[0]
+attributes:
+  alias:
+    - evaluate_target_health: false
+      name: scenarios--a3ec77f7-alb-491363430.eu-west-2.elb.amazonaws.com
+      zone_id: ZHURV8PSTC4K8
+  allow_overwrite: (known after apply)
+  fqdn: (known after apply)
+  health_check_id: null
+  id: (known after apply)
+  multivalue_answer_routing_policy: null
+  name: blackhole-terraform-example.overmind-terraform-example.com
+  records: null
+  set_identifier: null
+  terraform_address: module.scenarios[0].aws_route53_record.blackhole[0]
+  terraform_name: module.scenarios[0].aws_route53_record.blackhole[0]
+  timeouts: null
+  ttl: null
+  type: A
+  zone_id: Z01381333G7W1ZLUTENL1
+ aws_lb_target_group › module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0] 
--- current
+++ proposed
@@ -0,0 +1,68 @@
+type: aws_lb_target_group
+id: github.com/overmindtech/terraform-example.aws_lb_target_group.module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+attributes:
+  arn: (known after apply)
+  arn_suffix: (known after apply)
+  connection_termination: (known after apply)
+  deregistration_delay: "300"
+  health_check:
+    - enabled: true
+      healthy_threshold: 5
+      interval: 60
+      matcher: "200"
+      path: /
+      port: traffic-port
+      protocol: HTTP
+      timeout: 5
+      unhealthy_threshold: 2
+  id: (known after apply)
+  ip_address_type: (known after apply)
+  lambda_multi_value_headers_enabled: false
+  load_balancer_arns: (known after apply)
+  load_balancing_algorithm_type: (known after apply)
+  load_balancing_anomaly_mitigation: (known after apply)
+  load_balancing_cross_zone_enabled: (known after apply)
+  name: scenarios--a3ec77f7-tg-blackhole
+  name_prefix: (known after apply)
+  port: 8080
+  preserve_client_ip: (known after apply)
+  protocol: HTTP
+  protocol_version: (known after apply)
+  proxy_protocol_v2: false
+  slow_start: 0
+  stickiness: (known after apply)
+  tags:
+    CreatedBy: terraform
+    DaysUntilBF: "7"
+    Environment: demo
+    JavaHeapMB: "1536"
+    MemoryMB: "2048"
+    Mode: blackhole
+    Name: scenarios--a3ec77f7-tg-blackhole
+    OptimizationWorks: "true"
+    Project: memory-optimization
+    Purpose: risk-test
+    RequiredMemoryMB: "1792"
+    RiskLevel: low
+    Scenario: cost-reduction
+  tags_all:
+    CreatedBy: terraform
+    DaysUntilBF: "7"
+    Environment: demo
+    JavaHeapMB: "1536"
+    MemoryMB: "2048"
+    Mode: blackhole
+    Name: scenarios--a3ec77f7-tg-blackhole
+    OptimizationWorks: "true"
+    Project: memory-optimization
+    Purpose: risk-test
+    RequiredMemoryMB: "1792"
+    RiskLevel: low
+    Scenario: cost-reduction
+  target_failover: (known after apply)
+  target_group_health: (known after apply)
+  target_health_state: (known after apply)
+  target_type: ip
+  terraform_address: module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+  terraform_name: module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+  vpc_id: vpc-0f4ddbf8c33e5c725
~ aws_default_network_acl › module.scenarios[0].module.vpc.aws_default_network_acl.this[0] 
--- current
+++ proposed
@@ -46,9 +46,9 @@
     - subnet-0d0a1aff83bd8a460
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
~ aws_default_security_group › module.scenarios[0].module.vpc.aws_default_security_group.this[0] 
--- current
+++ proposed
@@ -35,9 +35,9 @@
   revoke_rules_on_delete: false
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"

💥 Blast Radius

Items 61

Edges 181

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jameslaneovermind