More control with your k8 clusters

Introduction

We have created our clusters and we have a few services running. How can we control the nodes that are created using kops

What are the AWS Resources Created

• VPC
• Subnet
• 3 for the cluster master and nodes
• 3 for bastions
• Routes
• 1 route for the Bastions
• 3 routes for each region 
• Routes
• Bastion route table have routes 0.0.0.0/0 going to the internet gateway
• Cluster route table have routes 0.0.0.0/0 going to the NAT gateway
• Internet Gateway
• Elastic IP
• 3 public ip's attached to each of the 3 NAT gateways
• NAT Gateways
• 3 NAT Gateways for each region
• Security Groups
• 3 security groups for each of Bastion, Master & Nodes
• Load Balancers
• 1 for Master
• 1 for Bastion
• Launch Configurations
• 1 each for Master, Node & Bastion
• Autoscaling Groups
• 1 each corresponding to Launch configurations
• Instances
• As requested for Master, Nodes and Bastion

Advanced Configuration

kops has the following parameters which enables you to control your cluster

• master-count -> Allows you to specify the number of masters
• master-size -> Allows you to specify the size of the master machine
• node-count -> Allows you to specify the number of nodes
• node-size -> Allows you to specify the size of the node machines

Use spot instances to reduce costs

It is no secret that Spot instances are much cheaper than on-demand instances. You can have the master and/or nodes as spot instances with this hack

Master ==> kops edit ig master-eu-west-1a --name k8.shivag.io --state s3://k8-kops-cluster-state-s3
Nodes ==> kops edit ig nodes --name k8.shivag.io --state s3://k8-kops-cluster-state-s3
Bastion ==> kops edit ig bastions --name k8.shivag.io --state s3://k8-kops-cluster-state-s3

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-05-18T19:28:46Z"
  labels:
    kops.k8s.io/cluster: k8.shivag.io
  name: master-eu-west-1a
spec:
  image: kope.io/k8s-1.17-debian-stretch-amd64-hvm-ebs-2020-01-17
  machineType: t2.medium
  maxPrice: "0.05"  <=== Max cost for spot instance
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-1a
  role: Master
  subnets:
  - eu-west-1a

Ensure you have the price corresponding to the instance you are using

Additional Security Group Assignment

The following flags on kops ensure you can have additional security groups assigned to the instance groups

--master-security-groups

--node-security-groups

Note: 

• Security Groups are attached to VPC; we need to have the VPC Pre-created and specify the VPC during the create phase or we need to make this change later and propagate it; in which case we need to remove the security group and delete the security group before deleting the cluster

Additional Security Group for Bastions

ubuntu@ip-10-0-1-79:~$ kops edit ig bastions --name k8.shivag.io --state s3://shivag.kube-kops-state
piVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-05-21T21:40:33Z"
  generation: 1
  labels:
    kops.k8s.io/cluster: k8.shivag.io
  name: bastions
spec:
  additionalSecurityGroups:
  - sg-095b938fcbad614bc
  image: kope.io/k8s-1.17-debian-stretch-amd64-hvm-ebs-2020-01-17
  machineType: t2.micro
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: bastions
  role: Bastion
  subnets:
  - utility-eu-west-1a
  - utility-eu-west-1b
  - utility-eu-west-1c