UP | HOME
Land of Lisp

Zhao Wei

How can man die better than facing fearful odds, for the ashes of his fathers and the temples of his Gods? -- By Horatius.

AWS Notes

1. ECS

1.2. Concepts

1.2.1. ECS

  • abbreviation for Elastic Container Service. It is a container management service for containers on a cluster.
  • Task Definition defines one or multiple containers to construct as a application.
  • A Task is an instance of task definition.
  • A Service is used to guarantee that you always have some number of Tasks running at all times. Such as, make sure there are always 3 running task across different zones.
  • A Cluster is used to manage a collection of resources such as CPU, memory, etc.

1.2.2. EC2

  • Short for Elastic Compute Cloud (Amazon EC2)

1.2.3. Farget and EC2

  • Both are cluster service types.
  • Farget doesn’t need server. It still need docker image.
  • EC2 need server and allow you to have more control on it.

1.2.4. Task Definitions

  • describes one or more containers, up to 10
  • blueprint for your application:
    • which containers/image to use
    • which launch type ?
    • which port
    • what data volumes should be used with the containers
  • Different task examples
  • A task is the instantiation of a task definition within a cluster.
  • A Service runs and maintains a specified number of tasks.

1.2.5. Cluster

  • A logical grouping of resources.
  • For Fargate type, AWS ECS manages cluster resources automatically.
  • For EC2, you need to manage a group of container instances.

1.3. Features of ECS

  • provide high available manner for
    • across multiple zones
    • within one Region
  • schedule the placement of containers across your cluster based on:
    • your resource needs
    • isolation policies
    • availability requirements

1.4. Steps to use ECS

  • Define VPC
  • Create clusters within a new or existing VPC.
  • Create tasks (from task definition) to run in Service in Cluster.
    • Running task is instance of task definition.
  • For ECS, you need to group task definition with service description.
    • Running task is instance of Service which is {task definition, Service description}
  • Docker image is created for tasks, and it is prepared in AWS ECR or Docker-Hub

1.5. TODO About AutoScaling of EC2

1.5.1. Similar example from App-worker-manager

Worker manager relies on two tables.

  • app-dummy-service-task-definitions-drdev, its schema:

    Pooling ScaleTo TaskArn TaskDef
    • It manages pool. Pool lists all the workers available to use.
    • It acts as a plan table (like how much memory I should need). It records:
      • For a specific version, how many I should keep it in hot pool.
      • For a speicif version, it is used/defined in which ECS task definition.
    • The value for TaskArn and TaskDef is generated from worker-pipeline:
      • After pushing worker into ECR, a new ECS task definition for worker is generated for new available worker. Worker manager will use this worker-task-definition to star worker.
  • app-worker-manager-hotpool-drdev, its schema:

    UUID DT Available Ip Port ServiceApiId TaskArn TaskDefArn WorkerVersion
    • It acts as a implementation/progress table for the app-dummy-service-task-definitions-drdev, plan table.
      • How to allocate those memory(worker), or which worker version is used in which ECS task instance.
      • For example, if the plan table set ScaleTo=6 and Pooling=Y for TaskDef=NX119. Then, this implementation table will add 6 records. They show where those 6 workers have been used in which ECS task instances.
    • After “the memory has been allocated”, if there are clients coming in and used those worker (consume some/all those memory). Then, in the next checking point (every 15mins), worker manager will need to allocate new workers as planed.
  • A related table, app-dummy-service-data-drdev.
    • This table records all the client activities which also includes which worker has been used for what purpose.

2. Route53

  • To make Route53 works, you need to create 2 records: one is type A, another is CNAME which is created during validate certificate.

2.1. Switch traffic to different domain name as failover

  • Task description
    • Given two domains: lb.dev.app.com as primary service and lb.drdev.app.com secondary service, we want to create another hosted zone in Route53 acs.app.com.
    • In normal case, we let dev.acs.app.com to route traffice to lb.dev.app.com.
    • In disaster case, we let dev.acs.app.com to route traffice to lb.drdev.app.com.
  • Steps
    • Create new hosted zone acs.app.com in Route53.
      • If public hosted zone is a third level domain, then add the NS records of this zone to the parent hosted zone.
    • Within acs.app.com, create record
      • Routing policy -> Simple routing, next
      • Define simple record:
        • Record name: dev.acs.app.com
        • Record type: CNAME -- Routes traffice to another domain name and to some AWS resources
        • Value/Route traffice to:
          • IP address or another value depending on the record type
          • lb.dev.app.com
  • Troubleshooting01: Could not resolve host: dev.acs.app.com
    • After created CNAME dev.acs.app.com to lb.dev.app.com, we should be able to visit resources under dev.acs.app.com as we visit lb.dev.app.com.

      curl https://lb.dev.app.com/app/health
      {"status":"ok","ts":1598240637924,"local":"8/24/2020, 3:43:57 AM","utc":"Mon, 24 Aug 2020 03:43:57 GMT"}
      
      curl https://dev.acs.app.com/app/health
      curl: (6) Could not resolve host: dev.acs.app.com
      
      • I forget to add the NS records of hosted zone acs.app.com to its parent hosted zone app.com. After doing so, the DNS resolution shows the following result.
    • Domain name resolve in Ubuntu
      • Resolve lb.dev.app.com (endpoint for app-dev)

        dig lb.dev.app.com
        
        ; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> lb.dev.app.com
        ;; global options: +cmd
        ;; Got answer:
        ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14519
        ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
        
        ;; OPT PSEUDOSECTION:
        ; EDNS: version: 0, flags:; udp: 65494
        ;; QUESTION SECTION:
        ;lb.dev.app.com.            IN      A
        
        ;; ANSWER SECTION:
        lb.dev.app.com.     0       IN      A       34.195.245.88
        lb.dev.app.com.     0       IN      A       34.197.127.0
        lb.dev.app.com.     0       IN      A       52.7.50.34
        
        ;; Query time: 80 msec
        ;; SERVER: 127.0.0.53#53(127.0.0.53)
        ;; WHEN: Mon Aug 24 13:27:16 CST 2020
        ;; MSG SIZE  rcvd: 95
        
      • Resolve dev.acs.app.com

        dig dev.acs.app.com
        
        ; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> dev.acs.app.com
        ;; global options: +cmd
        ;; Got answer:
        ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39460
        ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
        
        ;; OPT PSEUDOSECTION:
        ; EDNS: version: 0, flags:; udp: 65494
        ;; QUESTION SECTION:
        ;dev.acs.app.com.           IN      A
        
        ;; ANSWER SECTION:
        dev.acs.app.com.    0       IN      CNAME   lb.dev.app.com.
        lb.dev.app.com.     0       IN      A       52.7.50.34
        lb.dev.app.com.     0       IN      A       34.197.127.0
        lb.dev.app.com.     0       IN      A       34.195.245.88
        
        ;; Query time: 81 msec
        ;; SERVER: 127.0.0.53#53(127.0.0.53)
        ;; WHEN: Mon Aug 24 13:29:06 CST 2020
        ;; MSG SIZE  rcvd: 117
        

2.2. LoadBalancer in ECS

AppService:
  Type: AWS::ECS::Service
  DependsOn:
    - ALBListenerRule
    #- ServiceRole
  Properties:
    Cluster: !ImportValue
      Fn::Sub: ${ECSStackName}-ECSCluster
    DesiredCount: !FindInMap
      - EnvironmentMap
      - !Ref 'ConfigEnvironment'
      - DesiredCount
    LoadBalancers:
      - ContainerName: altas-service
        ContainerPort: '8080'
        TargetGroupArn: !Ref 'ECSTG'
      - ContainerName: altas-service
        ContainerPort: '8080'
        TargetGroupArn: !Ref 'ECSTGALB'
    #Role: !Ref 'ServiceRole'
    TaskDefinition: !Ref 'TaskDefinition'
    PlacementStrategies:
      - Field: attribute:ecs.availability-zone
        Type: spread
      - Field: memory
        Type: binpack        
ECSTGALB:
  Type: AWS::ElasticLoadBalancingV2::TargetGroup
  Properties:
    HealthCheckIntervalSeconds: 10
    HealthCheckPath: /health
    HealthCheckProtocol: HTTP
    HealthyThresholdCount: 2
    Port: 80
    Protocol: HTTP
    UnhealthyThresholdCount: 2
    VpcId: !ImportValue
      Fn::Sub: ${VPCStackName}-VPCId
    TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: '20'
ALBListenerRule:  
  Type: AWS::ElasticLoadBalancingV2::ListenerRule
  DependsOn:
    - ECSTGALB
  Properties:
    Actions:
      - Type: forward
        TargetGroupArn: !Ref 'ECSTGALB'
    Conditions:
      - Field: path-pattern
        Values:
          - /app/*
    ListenerArn: !ImportValue
      Fn::Sub: app-alb-${StackEnvironment}-AppPubALBListenerArn
    Priority: '5'
ALBListener:
  Type: AWS::ElasticLoadBalancingV2::Listener
  #DependsOn: ServiceRole
  Properties:
    DefaultActions:
      - Type: forward
        TargetGroupArn: !Ref 'ECSTG'
    LoadBalancerArn: !Ref 'ECSALB'
    Port: '80'
    Protocol: TCP
ECSALB:
  Type: AWS::ElasticLoadBalancingV2::LoadBalancer
  Properties:
    Type: network
    Scheme: internal
    Name: !Sub '${VPCStackName}-NLB'
    Subnets: !If
      - ThirdAZ
      - - !ImportValue
          Fn::Sub: ${VPCStackName}-PrivateSubnet1
        - !ImportValue
          Fn::Sub: ${VPCStackName}-PrivateSubnet2
        - !ImportValue
          Fn::Sub: ${VPCStackName}-PrivateSubnet3
      - - !ImportValue
          Fn::Sub: ${VPCStackName}-PrivateSubnet1
        - !ImportValue
          Fn::Sub: ${VPCStackName}-PrivateSubnet2
    LoadBalancerAttributes:
      - Key: load_balancing.cross_zone.enabled
        Value: 'true'
ECSTG:
  Type: AWS::ElasticLoadBalancingV2::TargetGroup
  DependsOn: ECSALB
  Properties:
    HealthCheckIntervalSeconds: 10
    HealthCheckPath: /health
    HealthCheckProtocol: HTTP
    HealthyThresholdCount: 2
    Port: 80
    Protocol: TCP
    UnhealthyThresholdCount: 2
    VpcId: !ImportValue
      Fn::Sub: ${VPCStackName}-VPCId
    TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: '20'                           

3. AWS Security

3.2. My notes

  • Different models have differences on how security responsibility are devided (where AWS and customer responsibility start and end)
    • model for infrastructure services
    • model for container services
    • model for abstrac services
  • In infrastructure services, customer needs to look after:
    • client and server side encryption
    • network traffic protection
    • security of the operating system
    • network
    • firewall configuration
    • application security
    • identity and access management
  • Container services model has much less requirement for customer
  • Abstract services model has least requirement for customer

3.2.1. Office building analogy

  • Think of your office as a multi-story building
  • Each floor has multiple suites
  • Each department like HR, Payroll can sit in one floor (or) multiple floors
  • Every suite has few rooms
  • Each room has couple of desk where employees work

    AWS Company
    Region Office building
    Availability Zone Each floor
    VPC Each department
    Subnet Each suite
    IP Each desk
  • When company assign desk to employee vs Launch AWS EC2 instance

    Assign desk Launch EC2 instance
    Which department this employee belongs to We need to pick which VPC this instance should be deployed to
    Then identifies which floors the department spans across Then we pick which availability zone (if not, default availability zone)
    Then identifies which suites have open desks Then we pick the subnet (if not default subnet)
    Assigns one of the open desks to the employee Then an IP from this subnet gets assigned to our EC2 instance (as private IP)
  • Other associated terminologies
    • Security groups like access card

3.2.2. VPC

  • What is it?
    • Whenever you launch an instance within your AWS account, by default it will be launched into a default VPC and gets public IP and private IP.
    • A logical virtual network spanning an entire AWS region (one region – one default VPC, can create multiple VPCs in a region).
  • Six components are created as default parts of VPC
    • VPC CIDR block
      • Every VPC is associated with an IP address range that is part of a Classless Inter-Domain Routing (CIDR) block which will be used to allocated private IP addresses to EC2 instances.
    • Subnets
      • a subnet is associated with only one availability zone (can not span multiple zones)
      • However, a zone can host multiple subnets.
      • Each subnet in a VPC is associated with an IPv4 CIDR block that is a a subset of the /16 CIDR block of its VPC.
    • Internet gateway
      • Provide connectivity to outside AWS (internet)
      • A VPC must be attached to an Internet gateway to communicate with internet. Only one IGW can be attached to a VPC at a time.
      • Bi-direction source and destination network address translation for your EC2 instances.
    • Route table
      • Every subnet must be associated with a route table.
      • If the association is not explicitly defined, then a subnet will be implicitly associated with the main route table.
    • Network access control lists (ACLs)
    • Security Group
    • VPC endpoint
  • AWS provides security mechanisms for your instances in the form of network ACLs and security groups.

3.2.3. Subnets

  • A subnet is a distinct network segment with its own IP address range within the larger VPC CIDR (Classless Inter-Domain Routing) range.
  • A VPC and have all public or public/private subnet combination.
    • A public subnet is defined by its connection (through a Routing Table) to an Internet Gateway (IGW) within your VPC.
    • Any subnet without a route to the IGW is considered private.
  • A private subnet is a subnet which doesn’t have a route to the internet gateway.
  • Steps to create a public subnet
    1. VPC –> create subnet
      • select the VPC within which you want to create the subnet
      • select availability zone
      • enter CIDR block
    2. Add an IGW
    3. Attach IGW to a VPC
    4. Add a route to the IGW from your subnet
      • Route Tables –> Create Route Table
      • Select your VPC, select “Create”
      • Edit the Route Table just created, give it a route to the outside world
        • Add another route
        • Enter ’0.0.0.0/0’ in Destination (to allow access to any Internet address)
        • Enter your IGW ID in the ’Target’ field. Save it.
    5. Associate your route table with your subnet
      • Select ’Subnet Associations’ (while selecting the route table you created, the sections below show different sections)
      • Select the subnets you wish to associate with this route table using the check box and click “save”.

3.2.4. Security Group

  • AWS security groups (SGs) are associated with EC2 instances and provide security at the protocol and port access level.
  • Working much the same way as a firewall:
    • a set of rules filter traffic coming into and out of an EC2 instance.
  • Security groups are specific to a VPC (which VPC the SG will reside).

3.2.5. Launch instance into VPC

  • Before launch
    1. identify the subnet where this instance need to be launched into.
    2. identify an available zone.
  • After launch (Once an instance is created within the VPC/Subnet/Availability Zone)
    • If the you launch the instance into a subnet that has assign public ip address attribute enabled, a public IP address is assigned to the primary network interface (eth0) of the created instance.
    • A public IP is mapped to the primary private IP through NAT. The public IP will change whenever you stop EC2 instance and start it again.
    • Notic: that public IP is not the AWS Elastic IP (EIP).

3.2.6. Bastion host

  • Bastion hosts are instances that sit within your public subnet and are typically accessed using SSH or RDP, as jump server.
  • Purpose: use SSH to log into other instances (within private subnets) deeper within your VPC.
  • If you require remote connectivity with your private instances over the public internet, then you need it.
  • Basic steps to create a bastion host
    1. Launch an EC2 instance as you normally would for any other instance.
    2. Apply OS hardening as required.
    3. Set up the appropriate security groups (SG).
    4. Implement either SSH-agent forwarding (Linux connectivity) or Remote Desktop Gateway (Windows connectivity).
    5. Deploy an AWS bastion host in each of the Availability Zones you’re using.

3.3. My summary

  • EC2 instance level security is controled by security groups which applies restrictions on instance’s protocol and port level.
  • Network level security is controled by applying network ACLs on subnets (equivalent of the security groups attached to EC2 instances).

4. ECR

  • pull image from ECR

    eval $(aws ecr get-login --no-include-email --region us-east-1 --profile=default)
    
    • Need to specify in which region you want to login.

5. TODO VPC

5.2. VPC networking components

  • Networking interfaces
  • Route tables
  • Prefix lists
  • NAT
    • NAT gateways
    • NAT instances
  • DHCP options sets
  • DNS
  • VPC peering
  • Elastic IP address
  • ClassicLink

6. CloudFormation Skills

6.1. Use FindInMap to customize environment variables based on keys

  • Goal: we want to form a aws resources string from environment map based on different conditions.
  • Example:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: Resources supporting the App Service
    
    Mappings:
      TableMap:
        ReplicateTables:
          RoleMaster: 'app-role-master'
          ApiKeyTable: 'app-dummy-service-apikey'
          UserRoleTable: 'app-dummy-service-user-role'
      EnvironmentMap:
        ConfigEnvironment:
          DRSrcEnv: 'drdev'
    
    
    Resource:
      - !Sub 'arn:aws:dynamodb:*:*:table/app-*-${StackEnvironment}/index/*'
      - Fn::Sub:
        - 'arn:aws:dynamodb:*:*:table/${table}-${srcenv}'
        - table:
            Fn::FindInMap:
              - TableMap
              - !Ref 'ReplicateTables'
              - RoleMaster
      - Fn::Sub:
        - 'arn:aws:dynamodb:*:*:table/${table}-${srcenv}/index/*'
        - table:
            Fn::FindInMap:
              - TableMap
              - !Ref 'ReplicateTables'
              - RoleMaster
          srcenv:
            Fn::FindInMap:
              - EnvironmentMap
              - !Ref 'ConfigEnvironment'
              - DRSrcEnv      
    
    • See Fn::Sub for syntax rule. In general, the values is subtituded from two variable table and srcenv and each of them is referenced by the environment map defined by Mappings.

7. My notes

7.1. Network Fundamentals