Kubespot (AWS)

Compliance Oriented Kubernetes Setup for AWS.

Kubespot is an open source terraform module that attempts to create a complete compliance-oriented Kubernetes setup on AWS, Google Cloud and Azure. These add additional security such as additional system logs, file system monitoring, hard disk encryption and access control. Further, we setup the managed Redis and SQL on each of the Cloud providers with limited access to the Kubernetes cluster so things are further locked down. All of this should lead to setting up a HIPAA / PCI / SOC2 being made straightforward and repeatable.

This covers how we setup your infrastructure on AWS, Google Cloud and Azure. These are the three Cloud Providers that we currently support to run Kubernetes. Further, we use the managed service provided by each of the Cloud Providers. This document covers everything related to how infrastructure is setup within each Cloud, how we create an isolated environment for Compliance and the commonalities between them.

Tools & Setup

brew install kubectl kubernetes-helm awscli terraform

Cluster Usage

If the infrastructure is using the opsZero infrastructure as code template then you access the resources like the following:

Add your IAM credentials in ~/.aws/credentials.

[profile_name]
aws_access_key_id=<>key>
aws_secret_access_key=<secret_key>
region=us-west-2
cd environments/<nameofenv>
make kubeconfig
export KUBECONFIG=./kubeconfig # add to a .zshrc
kubectl get pods

Cluster Setup

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

Providers

Name

Version

aws

~> 4.16.0

helm

n/a

http

n/a

kubernetes

>= 2.0

null

n/a

tls

n/a

Inputs

Name

Description

Type

Default

Required

alb_controller_version

The chart version of the ALB controller helm chart

string

"1.4.2"

no

alb_name

Release name of the ALB controller chart

string

"aws-load-balancer-controller"

no

ami_image

The EC2 image ID to launch

string

""

no

aws_load_balancer_controller_enabled

Enable ALB controller by default

bool

true

no

bastion_ec2_keypair

Key name of the Key Pair to use for the bastion host

string

"opszero"

no

bastion_eip_enabled

Whether the EIP for bastion host is enabled

bool

false

no

bastion_enabled

Whether the bastion host is enabled

bool

false

no

bastion_instance_type

The instance type to use for the bastion instance

string

"t3.micro"

no

bastion_volume_size

The volume szie to use for the bastion instance

number

20

no

bastion_vpn_allowed_cidrs

These are the IPs that the bastion and VPN allow connections from.

list

[
“0.0.0.0/0”
]

no

cidr_block

The CIDR block used by the VPC

string

"10.2.0.0/16"

no

cidr_block_private_subnet

The CIDR block used by the private subnet

list

[
“10.2.2.0/24”,
“10.2.3.0/24”
]

no

cidr_block_public_subnet

The CIDR block used by the private subnet

list

[
“10.2.0.0/24”,
“10.2.1.0/24”
]

no

cluster_logging

List of the desired control plane logging to enable

list

[
“api”,
“audit”,
“authenticator”,
“controllerManager”,
“scheduler”
]

no

cluster_private_access

Whether the Amazon EKS private API server endpoint is enabled

bool

false

no

cluster_public_access

Whether the Amazon EKS private API server endpoint is enabled

bool

true

no

cluster_public_access_cidrs

List of CIDR blocks. Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled

list

[
“0.0.0.0/0”
]

no

cluster_version

Desired Kubernetes master version

string

"1.21"

no

csi_secrets_store_enabled

Specify whether the CSI driver is enabled on the EKS cluster

bool

false

no

csi_secrets_store_version

The version of the CSI store helm chart

string

"1.2.2"

no

ec2_keypair

Key name of the Key Pair to use for the EKS nodes

string

"opszero"

no

efs_enabled

Specify whether the EFS is enabled on the EKS cluster

bool

false

no

eips

List of Elastic IPs

list

[]

no

eks_guardduty_enabled

whether the guardduty is enabled for the EKS cluster

bool

true

no

enable_egress_only_internet_gateway

Create an egress-only Internet gateway for your VPC0

bool

false

no

enable_ipv6

Enable an Amazon-provided IPv6 CIDR block with a /56 prefix length for the VPC

bool

false

no

enable_nat

Whether the NAT gateway is enabled

bool

true

no

enabled_metrics_asg

A list of metrics to collect

list

[
“GroupDesiredCapacity”,
“GroupInServiceCapacity”,
“GroupInServiceInstances”,
“GroupMaxSize”,
“GroupMinSize”,
“GroupPendingCapacity”,
“GroupPendingInstances”,
“GroupStandbyCapacity”,
“GroupStandbyInstances”,
“GroupTerminatingCapacity”,
“GroupTerminatingInstances”,
“GroupTotalCapacity”,
“GroupTotalInstances”
]

no

environment_name

Name of the environment to create AWS resources

string

n/a

yes

fargate_selector

Terraform object to create the EKS fargate profiles

map

{
“serverless”: {}
}

no

iam_users

List of IAM users

list

[]

no

instance_userdata

User data to provide when launching the instance

string

""

no

karpenter_enabled

Specify whether the karpenter is enabled

bool

false

no

karpenter_name

The release name of the karpenter helm chart

string

"karpenter-scaler"

no

karpenter_version

The version of the karpenter helm chart

string

"v0.13.2"

no

legacy_subnet

Specify how the subnets should be created

bool

true

no

memorydb_enabled

Specify whether the memorydb is enabled

bool

false

no

metrics_server_version

The version of the metric server helm chart

string

"3.8.2"

no

monitoring_role_arn

The ARN for the IAM role that permits RDS to send enhanced monitoring metrics to CloudWatch Logs

string

""

no

node_group_cpu_threshold

The value of the CPU threshold

string

"70"

no

node_groups

Terraform object to create the EKS node groups

map

{}

no

node_role_policies

A list of The ARN of the policies you want to attach

list

[]

no

nodes_blue_desired_capacity

The number of Amazon EC2 instances that should be running in the group

number

0

no

nodes_blue_instance_type

The instance type to use for the instance

string

"t3.micro"

no

nodes_blue_max_instance_lifetime

The maximum amount of time, in seconds, that an instance can be in service

number

604800

no

nodes_blue_max_size

The maximum size of the Auto Scaling Group

number

0

no

nodes_blue_min_size

The minimum size of the Auto Scaling Group

number

0

no

nodes_blue_root_device_size

Size of the volume in gibibytes (GiB)

string

"20"

no

nodes_blue_subnet_ids

A list of subnet IDs to launch resources in

list

[]

no

nodes_green_desired_capacity

The number of Amazon EC2 instances that should be running in the group

number

0

no

nodes_green_instance_type

The instance type to use for the instance

string

"t3.micro"

no

nodes_green_max_instance_lifetime

The maximum amount of time, in seconds, that an instance can be in service

number

604800

no

nodes_green_max_size

The maximum size of the Auto Scaling Group

number

0

no

nodes_green_min_size

The minimum size of the Auto Scaling Group

number

0

no

nodes_green_root_device_size

Size of the volume in gibibytes (GiB)

string

"20"

no

nodes_green_subnet_ids

A list of subnet IDs to launch resources in

list

[]

no

nodes_in_public_subnet

INSECURE! Only use this if you want to avoid paying for the NAT. Also set enable_nat to false

bool

false

no

redis_enabled

Whether the redis cluster is enabled

bool

false

no

redis_engine_version

Version number of the cache engine to be used for the cache clusters in this replication group

string

"6.x"

no

redis_node_type

Instance class of the redis cluster to be used

string

"cache.t3.micro"

no

repos

n/a

list

[]

no

sql_cluster_enabled

Whether the sql cluster is enabled

bool

false

no

sql_database_name

The name of the database to create when the DB instance is created

string

""

no

sql_encrypted

Specify whether the DB instance is encrypted

bool

true

no

sql_engine

The name of the database engine to be used for this DB cluster

string

"aurora-postgresql"

no

sql_engine_mode

The database engine mode

string

"serverless"

no

sql_engine_version

The engine version to use

string

"12.7"

no

sql_identifier

The name of the database

string

""

no

sql_instance_allocated_storage

The allocated storage in gibibytes

number

20

no

sql_instance_class

The instance type of the RDS instance.

string

"db.t3.medium"

no

sql_instance_enabled

Whether the sql instance is enabled

bool

false

no

sql_instance_engine

The database engine to use

string

"postgres"

no

sql_instance_max_allocated_storage

the upper limit to which Amazon RDS can automatically scale the storage of the DB instance

number

200

no

sql_master_password

Password for the master DB user

string

""

no

sql_master_username

Username for the master DB user

string

""

no

sql_node_count

The number of instances to be used for this DB cluster

number

0

no

sql_parameter_group_name

Name of the DB parameter group to associate

string

""

no

sql_rds_multi_az

Specify if the RDS instance is enabled multi-AZ

bool

false

no

sql_serverless_max

The maximum capacity for the DB cluster

number

2

no

sql_serverless_min

The maximum capacity for the DB cluster

number

2

no

sql_serverless_seconds_until_auto_pause

The time, in seconds, before the DB cluster in serverless mode is paused

number

300

no

sql_subnet_group_include_public

Include public subnets as part of the clusters subnet configuration.

bool

false

no

sso_roles

Terraform object of the IAM roles

map

{
“admin_roles”: [],
“dev_roles”: [],
“monitoring_roles”: [],
“readonly_roles”: []
}

no

tags

Terraform map to create custom tags for the AWS resources

map

{}

no

vpc_flow_logs_enabled

Specify whether the vpc flow log is enabled

bool

false

no

zones

AZs for the subnets

list

[
“us-west-2a”,
“us-west-2b”
]

no

Resources

Name

Type

aws_autoscaling_group.nodes_blue

resource

aws_autoscaling_group.nodes_green

resource

aws_cloudwatch_log_group.vpc

resource

aws_cloudwatch_metric_alarm.bastion_cpu_threshold

resource

aws_cloudwatch_metric_alarm.database_cpu_database

resource

aws_cloudwatch_metric_alarm.database_disk_database

resource

aws_cloudwatch_metric_alarm.database_free_disk_database

resource

aws_cloudwatch_metric_alarm.database_free_disk_database2

resource

aws_cloudwatch_metric_alarm.database_free_disk_database3

resource

aws_cloudwatch_metric_alarm.database_free_disk_database4

resource

aws_cloudwatch_metric_alarm.database_io_mysql

resource

aws_cloudwatch_metric_alarm.database_io_postgres

resource

aws_cloudwatch_metric_alarm.node_group_cpu_threshold

resource

aws_cloudwatch_metric_alarm.nodes_blue_cpu_threshold

resource

aws_cloudwatch_metric_alarm.nodes_green_cpu_threshold

resource

aws_db_instance.default

resource

aws_db_subnet_group.default

resource

aws_egress_only_internet_gateway.egress

resource

aws_eip.bastion_eip

resource

aws_eip.eips

resource

aws_eks_addon.core

resource

aws_eks_cluster.cluster

resource

aws_eks_fargate_profile.fargate

resource

aws_eks_node_group.node_group

resource

aws_elasticache_cluster.default

resource

aws_elasticache_subnet_group.default

resource

aws_flow_log.vpc

resource

aws_iam_instance_profile.node

resource

aws_iam_openid_connect_provider.cluster

resource

aws_iam_policy.alb

resource

aws_iam_policy.efs_policy

resource

aws_iam_role.cluster

resource

aws_iam_role.fargate

resource

aws_iam_role.node

resource

aws_iam_role.node_oidc

resource

aws_iam_role.vpc

resource

aws_iam_role_policy.autoscaling

resource

aws_iam_role_policy.autoscaling_oidc

resource

aws_iam_role_policy.karpenter

resource

aws_iam_role_policy.vpc

resource

aws_iam_role_policy_attachment.alb

resource

aws_iam_role_policy_attachment.aws_node_oidc

resource

aws_iam_role_policy_attachment.cluster-AmazonEKSClusterPolicy

resource

aws_iam_role_policy_attachment.cluster-AmazonEKSServicePolicy

resource

aws_iam_role_policy_attachment.fargate-AmazonEKSFargatePodExecutionRolePolicy

resource

aws_iam_role_policy_attachment.karpenter_ssm_policy

resource

aws_iam_role_policy_attachment.node-AmazonEC2ContainerRegistryReadOnly

resource

aws_iam_role_policy_attachment.node-AmazonEKSWorkerNodePolicy

resource

aws_iam_role_policy_attachment.node-AmazonEKS_CNI_Policy

resource

aws_iam_role_policy_attachment.node-EFS

resource

aws_iam_role_policy_attachment.node_role_policies

resource

aws_instance.bastion

resource

aws_internet_gateway.public

resource

aws_launch_configuration.nodes_blue

resource

aws_launch_configuration.nodes_green

resource

aws_memorydb_cluster.example

resource

aws_memorydb_subnet_group.default

resource

aws_nat_gateway.gw

resource

aws_rds_cluster.default

resource

aws_rds_cluster_instance.cluster_instances

resource

aws_route.ig

resource

aws_route.ipv6

resource

aws_route.nat

resource

aws_route_table.private

resource

aws_route_table.public

resource

aws_route_table_association.private

resource

aws_route_table_association.public

resource

aws_security_group.bastion

resource

aws_security_group.cluster

resource

aws_security_group.node

resource

aws_security_group_rule.bastion_ssh

resource

aws_security_group_rule.cluster-ingress-node-https

resource

aws_security_group_rule.eks

resource

aws_security_group_rule.node-ingress-cluster

resource

aws_security_group_rule.node-ingress-self

resource

aws_security_group_rule.private_subnet

resource

aws_security_group_rule.public_subnet

resource

aws_subnet.private

resource

aws_subnet.public

resource

aws_vpc.vpc

resource

helm_release.aws_efs_csi_driver

resource

helm_release.aws_load_balancer

resource

helm_release.csi_secrets_store

resource

helm_release.karpenter

resource

helm_release.metrics-server

resource

kubernetes_cluster_role_binding.eks_admins_binding

resource

kubernetes_cluster_role_binding.eks_readonly_binding

resource

kubernetes_config_map.aws_auth

resource

kubernetes_role.default_eks_admins

resource

kubernetes_role.default_eks_developers

resource

kubernetes_role.default_eks_monitoring_admins

resource

kubernetes_role.default_eks_readonly

resource

kubernetes_role_binding.default_eks_admins

resource

kubernetes_role_binding.default_eks_developers

resource

kubernetes_role_binding.default_eks_monitoring_admins

resource

kubernetes_role_binding.default_eks_readonly

resource

null_resource.csi_secrets_store_aws_provider

resource

null_resource.karpenter_crd

resource

aws_ami.ubuntu

data source

aws_availability_zones.available

data source

aws_caller_identity.current

data source

aws_eks_cluster_auth.cluster

data source

aws_iam_policy.ssm_managed_instance

data source

aws_region.current

data source

aws_ssm_parameter.eks_ami

data source

http_http.csi_secrets_store_aws_provider

data source

http_http.karpenter_crd

data source

tls_certificate.cluster

data source