Merge pull request #15 from davidsilva/fixes/noticed-during-video

Things noticed during recording of the video: (1) Try to improve Lamb…

Author: davidsilva

.github/workflows/ci.yml

@@ -160,8 +160,8 @@ jobs:
             outputfile.txt
         env:
           NODE_ENV: development
-          DATABASE_URL: postgres://${{ secrets.DB_USERNAME }}:${{ secrets.DB_PASSWORD }}@${{ secrets.DB_HOST }}:${{ secrets.DB_PORT }}/${{ secrets.DB_NAME }}
 
+      # sets up Docker Buildx to enable advanced build features in the workflow.
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v1
 

README.md

@@ -117,12 +117,13 @@ A lot of `ci.yml` is self-explanatory but here are some notes to help clarify so
   * Does an install and build for the Lambda function.
   * And finally zips up everything.
 * **Update Lambda migration function**: Updates the AWS Lambda function code for database migrations using the AWS CLI. The workflow waits for the update to be completed before the function is invoked.
+* **Build and push frontend Docker image**: Note that this step uses a Dockerfile in the root of the project, rather than the Dockerfile in the frontend directory. I wanted to keep the Dockerfile in the frontend directory for local development purposes, e.g., `frontend/Dockerfile` uses the Angular server and port 4200.
 
 ## Infrastructure
 
 ![Infrastructure Diagram](images/Interview-Prep-Infrastructure-v2.drawio.png)
 
-The diagram above illustrates the major parts of the application infrastructure:
+The diagram above illustrates the major parts of the application infrastructure. Here are some descriptions and notes:
 
 * **Interview Prep VPC**: "The Virtual Private Cloud (VPC) is a logically isolated network within the AWS cloud where we can launch and manage AWS resources. It provides a secure environment to group and connect related resources and services, such as EC2 instances, RDS databases, and ECS clusters. The VPC allows us to define our own IP address range, create subnets, and configure route tables and network gateways, ensuring that our infrastructure is both secure and scalable." (GitHub Copilot came up with such a great explanation here that I'm just going to use it as-is.)
 * **Availability zones A and B**: `us-east-1a` and `us-east-1b`. These zones, along with their corresponding public and private subnets, enhance the app's resilience. Currently, one task each for the ECS frontend and backend is deployed, but this can be scaled to distribute tasks across both availability zones.
@@ -134,13 +135,20 @@ The diagram above illustrates the major parts of the application infrastructure:
 * **Public route table**: The public routing table is associated with the public subnets and directs traffic to the internet through the Internet Gateway. This allows resources in the public subnets, such as the load balancer and bastion host, to communicate with the internet.
 * **Private route table**: The private routing table is associated with the private subnets and directs traffic to the internet through the NAT Gateway. This allows resources in the private subnets, such as the ECS services and RDS database, to access the internet for updates and patches while keeping them isolated from direct internet access.
 * **Internet gateway**: Allows resources within the VPC to communicate with the internet.
-* **NAT gateway**: The NAT gateway is in public subnet A but both private subnets can use it via the private route table. We *could* add a NAT gateway to public subnet B to ensure higher availability and fault tolerance.
+* **NAT gateway**: The NAT gateway is in public subnet A but both private subnets can use it via the private route table. We *could* add a NAT gateway to public subnet B to ensure higher availability and fault tolerance. The NAT gateway is used, for example, by ECS tasks to pull Docker images from the ECR. It's also used by the migrate Lambda function to get values from AWS Systems Manager Parameter Store.
 * **API gateway**: This serves as a proxy that forwards the path, data, method, and other request details to the backend API server. This allows the backend to handle the actual processing of the requests. The API gateway is set up to handle CORS, limiting web browser requests to pages served from our frontend host. In the future we'll add an API key and authorization to restrict usage of the API.
 * **ECR for hosting frontend and backend Docker images**: Our GitHub workflow builds and pushes Docker images of our frontend and backend apps to the AWS Elastic Container Registry, tagging the latest build as... "latest."
 * **ECS cluster with frontend and backend ECS services**: The GitHub workflow updates the backend and frontend services in the AWS Elastic Container Service. These services are where our frontend and backend servers actually run, providing the necessary environments for our applications to operate.
 * **RDS-hosted Postgres database instance**: The application uses an instance of Postgres hosted by the AWS Relational Database Service. It runs in the private subnets.
 * **Migrate Lambda function**: This is a function run by the GitHub workflow. A workflow step packages up the migration files with the Lambda function itself and then invokes the function.
 * **Route 53-hosted domains**: `dev.interviewprep.onyxdevtutorials.com` and `api.dev.interviewprep.onyxdevtutorials.com`. The DNS configuration in Route 53 connects the frontend and backend domains to the load balancer. This is achieved using alias records that point to the load balancer's DNS name and zone ID.
+* **CIDR Blocks**: CIDR (Classless Inter-Domain Routing) blocks are used to define IP address ranges within the VPC.
+  * **VPC CIDR Block**: This is set to `10.0.0.0/16`, allowing for 65,536 possible IP addresses -- which is plenty for this project.
+  * **Subnet CIDR Blocks**: Each subnet gets 256 IP addresses:
+    * **Public Subnet A**: 10.0.1.0/24 provides 256 IP addresses.
+    * **Public Subnet B**: 10.0.2.0/24 provides 256 IP addresses.
+    * **Private Subnet A**: 10.0.3.0/24 provides 256 IP addresses.
+    * **Private Subnet B**: 10.0.4.0/24 provides 256 IP addresses.
 
 ## Costs
 
@@ -260,6 +268,16 @@ Once you're connected to the bastion host, you can directly access the database:
 
 `psql -h <db-endpoint> -U <username> -d <db-name>`
 
+Once connected to the database, you can type `\l` to list all the databases managed by the PostgreSQL server you are connected to.
+
+Type `\dt` to list all the tables in the current database.
+
+Type `\d products` or `\d users` to get information about each of these tables.
+
+## Add New Migration File
+
+Assuming CWD is `backend`, `npx knex migrate:make <migration-file-name> --knexfile ./src/knexFile.ts --migrations-directory ../migrations`.
+
 ## Version History
 
 ### 0.1.0

backend/migrate-lambda/src/migrate.ts

@@ -5,9 +5,12 @@ import { SSMClient, GetParameterCommand } from '@aws-sdk/client-ssm';
 const ssmClient = new SSMClient({ region: 'us-east-1' });
 
 const getSSMParameter = async (name: string): Promise<string> => {
-  const command = new GetParameterCommand({ Name: name, WithDecryption: true });
-  const response = await ssmClient.send(command);
-  return response.Parameter?.Value || '';
+  if (!name) {
+    throw new Error('SSM parameter name is required');
+  }
+    const command = new GetParameterCommand({ Name: name, WithDecryption: true });
+    const response = await ssmClient.send(command);
+    return response.Parameter?.Value || '';
 };
 
 export const runMigrations = async () => {
@@ -16,11 +19,11 @@ export const runMigrations = async () => {
         const config = knexConfig[env];
 
         config.connection = {
-            host: await getSSMParameter(`/interview-prep/${env}/DB_HOST`),
-            user: await getSSMParameter(`/interview-prep/${env}/DB_USERNAME`),
-            password: await getSSMParameter(`/interview-prep/${env}/DB_PASSWORD`),
-            database: await getSSMParameter(`/interview-prep/${env}/DB_NAME`),
-            port: parseInt(await getSSMParameter(`/interview-prep/${env}/DB_PORT`), 10),
+            host: await getSSMParameter(process.env.DB_HOST_PARAM || ''),
+            user: await getSSMParameter(process.env.DB_USER_PARAM || ''),
+            password: await getSSMParameter(process.env.DB_PASS_PARAM || ''),
+            database: await getSSMParameter(process.env.DB_NAME_PARAM || ''),
+            port: parseInt(await getSSMParameter(process.env.DB_PORT_PARAM || ''), 10),
             ssl: { rejectUnauthorized: false },
         };
 

backend/migrations/20250129192259_add_favorite_color_to_users_table.ts

@@ -0,0 +1,16 @@
+import type { Knex } from "knex";
+
+
+export async function up(knex: Knex): Promise<void> {
+    return knex.schema.table('users', (table) => {
+        table.string('favorite_color').nullable();
+    });
+}
+
+
+export async function down(knex: Knex): Promise<void> {
+    return knex.schema.table('users', (table) => {
+        table.dropColumn('favorite_color');
+    });
+}
+

backend/migrations/20250130010302_add_favorite_song_to_users_table.ts

@@ -0,0 +1,16 @@
+import type { Knex } from "knex";
+
+
+export async function up(knex: Knex): Promise<void> {
+    return knex.schema.table('users', (table) => {
+        table.string('favorite_song').nullable();
+    });
+}
+
+
+export async function down(knex: Knex): Promise<void> {
+    return knex.schema.table('users', (table) => {
+        table.dropColumn('favorite_song');
+    });
+}
+

terraform/environments/development/main.tf

@@ -43,7 +43,6 @@ module "rds" {
   db_username = var.db_username
   db_password = var.db_password
   db_sg_id    = module.security_groups.db_sg_id
-  lambda_sg_id = module.security_groups.lambda_sg_id
   environment = var.environment
 }
 

terraform/environments/development/outputs.tf

@@ -48,6 +48,11 @@ output "ecs_task_execution_role_arn" {
   value       = module.iam.ecs_task_execution_role_arn
 }
 
+output "ecs_task_role_arn" {
+  description = "The ARN of the ECS task role"
+  value       = module.iam.ecs_task_role_arn
+}
+
 output "lambda_exec_role_arn" {
   description = "The ARN of the Lambda execution role"
   value       = module.iam.lambda_exec_role_arn

terraform/modules/api_gateway/main.tf

@@ -6,22 +6,24 @@ resource "aws_api_gateway_rest_api" "api" {
 resource "aws_api_gateway_resource" "proxy" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     parent_id   = aws_api_gateway_rest_api.api.root_resource_id
-    path_part   = "{proxy+}"
+    path_part   = "{proxy+}" # Path part that acts as a catch-all proxy for any request path.
 
-    depends_on = [ aws_api_gateway_rest_api.api ]
+    depends_on = [ aws_api_gateway_rest_api.api ] # Ensure the API is created before creating the resource.
 }
 
 resource "aws_api_gateway_method" "proxy_method" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     resource_id = aws_api_gateway_resource.proxy.id
-    http_method = "ANY"
-    authorization = "NONE"
-    api_key_required = false
+    http_method = "ANY" # Handle every type of HTTP request
+    authorization = "NONE" # No authorization required (yet)
+    api_key_required = false # No API key required (yet)
     request_parameters = {
       "method.request.path.proxy" = true
     }
+  # This configuration allows the API Gateway to serve as a proxy for my actual backend application, handling all types of HTTP requests and forwarding them to the backend.
 }
 
+// Define the OPTIONS method for the proxy resource (for CORS preflight requests)
 resource "aws_api_gateway_method" "proxy_options" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     resource_id = aws_api_gateway_resource.proxy.id
@@ -30,12 +32,14 @@ resource "aws_api_gateway_method" "proxy_options" {
     api_key_required = false
 }
 
+# Define the integration between the proxy resource and the backend application. Basically, the API Gateway will forward all requests to the backend application.
 resource "aws_api_gateway_integration" "proxy_integration" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     resource_id = aws_api_gateway_resource.proxy.id
     http_method = aws_api_gateway_method.proxy_method.http_method
     type = "HTTP_PROXY"
     integration_http_method = "ANY"
+    # Load balancer knows that port 3000 is the backend application
     uri = "http://${var.lb_dns_name}:3000/{proxy}"
     request_parameters = {
       "integration.request.path.proxy" = "method.request.path.proxy"
@@ -66,6 +70,7 @@ resource "aws_api_gateway_integration" "health_integration" {
     uri = "http://${var.lb_dns_name}:3000/health"
 }
 
+# Defines how API Gateway should handle the OPTIONS method for the proxy resource. In this case, it uses a MOCK integration to generate a mock response.
 resource "aws_api_gateway_integration" "proxy_options_integration" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     resource_id = aws_api_gateway_resource.proxy.id
@@ -76,6 +81,9 @@ resource "aws_api_gateway_integration" "proxy_options_integration" {
     }
 }
 
+# In Amazon API Gateway, an aws_api_gateway_method_response specifies the possible responses from an API Gateway, while an aws_api_gateway_integration_response maps the response from an integration to the API Gateway response. 
+
+# This resource specifies the response parameters (headers) that the integration should return. It is part of the integration setup and tells API Gateway what to include in the response when the OPTIONS method is called.
 resource "aws_api_gateway_integration_response" "proxy_options_integration_response" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     resource_id = aws_api_gateway_resource.proxy.id
@@ -88,6 +96,7 @@ resource "aws_api_gateway_integration_response" "proxy_options_integration_respo
     }
 }
 
+# This resource specifies the method response parameters (headers) that the method should return. It is part of the method setup and ensures that the headers specified in the integration response are actually included in the final response sent to the client.
 resource "aws_api_gateway_method_response" "proxy_options_response" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     resource_id = aws_api_gateway_resource.proxy.id
@@ -109,15 +118,18 @@ resource "aws_api_gateway_deployment" "api_deployment" {
     ]
     rest_api_id = aws_api_gateway_rest_api.api.id
 
+    # This effectively triggers a redeployment whenever I do `terraform apply`, even if there are no actual changes to the configuration. I need to experiment with this setting.
     triggers = {
       redeployment = "${timestamp()}"
     }
 
+    # Minimize downtime by creating the new deployment before destroying the old one. And... because I don't think AWS would let me destroy the API given that it's in use by the load balancer.
     lifecycle {
         create_before_destroy = true
     }
 }
 
+# An API Gateway stage is a logical reference to a lifecycle state of your API (for example, dev, test, prod). Stages are used to manage and deploy different versions of your API, allowing you to test changes in a development environment before promoting them to production.
 resource "aws_api_gateway_stage" "api_stage" {
     deployment_id = aws_api_gateway_deployment.api_deployment.id
     rest_api_id = aws_api_gateway_rest_api.api.id
@@ -132,11 +144,11 @@ resource "aws_api_gateway_stage" "api_stage" {
 resource "aws_api_gateway_method_settings" "api_method_settings" {
     rest_api_id = aws_api_gateway_rest_api.api.id
     stage_name = aws_api_gateway_stage.api_stage.stage_name
-    method_path = "*/*"
+    method_path = "*/*" # The path and method for which these settings apply. The format is HTTP_METHOD/RESOURCE_PATH. You can use */* to apply the settings to all methods and resources.
     settings {
-        metrics_enabled = true
-        logging_level = "INFO"
-        data_trace_enabled = true
+        metrics_enabled = true # Enable CloudWatch metrics for the method.
+        logging_level = "INFO" # E.g., INFO, ERROR
+        data_trace_enabled = true # Can generate a large volume of log data, especially for APIs with high traffic or large payloads.
     }
 }
 
@@ -145,6 +157,10 @@ resource "aws_cloudwatch_log_group" "api_gateway_log_group" {
     retention_in_days = 7
 }
 
+# IAM stuff should probably be in IAM module.
+
+# It could be to have a root-level configuration to enable logging for the various modules.
+
 resource "aws_iam_role" "api_gateway_cloudwatch_role" {
     name = "${var.environment}-interview-prep-api-gateway-cloudwatch-role"
     assume_role_policy = jsonencode({
@@ -185,16 +201,18 @@ resource "aws_iam_role_policy_attachment" "api_gateway_cloudwatch_policy_attachm
     role = aws_iam_role.api_gateway_cloudwatch_role.name
 }
 
+# custom_domain_name and custom_domain_zone_id are output and used in the dns module.
 resource "aws_api_gateway_domain_name" "custom_domain" {
   domain_name = "api.dev.interviewprep.onyxdevtutorials.com"
 
   endpoint_configuration {
-    types = ["EDGE"]
+    types = ["EDGE"] # The endpoint type (EDGE, REGIONAL, or PRIVATE)
   }
 
-  certificate_arn = var.certificate_arn
+  certificate_arn = var.certificate_arn # The ARN of the SSL certificate to use for the custom domain.
 }
 
+# Used to map the custom domain to the API Gateway stage.
 resource "aws_api_gateway_base_path_mapping" "custom_domain_mapping" {
   api_id = aws_api_gateway_rest_api.api.id
   stage_name = aws_api_gateway_stage.api_stage.stage_name

terraform/modules/bastion/main.tf

@@ -3,7 +3,7 @@ resource "aws_instance" "bastion" {
   instance_type = var.instance_type
   subnet_id = var.public_subnet_id
   vpc_security_group_ids = [var.bastion_sg_id]
-  key_name = var.key_name
+  key_name = var.key_name # SSH key pair name
 
   tags = {
     Name = "${var.environment}-interview-prep-bastion"

terraform/modules/ecr/main.tf

@@ -1,8 +1,9 @@
 resource "aws_ecr_repository" "frontend" {
   name = "interview-prep-frontend"
-  image_tag_mutability = "MUTABLE"
+  image_tag_mutability = "MUTABLE" # Allows using a tag like "latest" to refer to the latest version of the image.
+  # Allows you to enable or disable automatic scanning of container images for vulnerabilities when they are pushed to the repository.
   image_scanning_configuration {
-    scan_on_push = true
+    scan_on_push = true # Images are automatically scanned for vulnerabilities when they are pushed to the repository.
   }
 
     tags = {
@@ -14,8 +15,9 @@ resource "aws_ecr_repository" "frontend" {
 resource "aws_ecr_repository" "backend" {
   name = "interview-prep-backend"
   image_tag_mutability = "MUTABLE"
+  # Allows you to enable or disable automatic scanning of container images for vulnerabilities when they are pushed to the repository.
   image_scanning_configuration {
-    scan_on_push = true
+    scan_on_push = true # Images are automatically scanned for vulnerabilities when they are pushed to the repository.
   }
 
     tags = {