[k8s] Multi-etcd Cluster - Internal

This is a practice to set up a Kubernetes environment on a VM and build a multi-etcd cluster within the cluster.

Setting up HAproxy Node

• Building a cluster with multiple master nodes.

 162  sudo apt install haproxy
  163  sudo apt upgrade
  164  sudo vi /etc/haproxy/haproxy.cfg

frontend kubernetes-master-lb
        bind 0.0.0.0:6443
        option tcplog
        mode tcp
        default_backend kubernetes-master-nodes

backend kubernetes-master-nodes
        mode tcp
        balance roundrobin
        option tcp-check
        option tcplog
        server k8s-master1 192.168.56.111:6443 check
        server k8s-master2 192.168.56.112:6443 check
        server k8s-master3 192.168.56.113:6443 check
  165  sudo systemctl restart haproxy.service
  166  sudo systemctl status haproxy.service

Cluster

• Initialization

  • If the IP is not advertised during the join, enp0s3 is captured, causing communication failure between etcd nodes.

  •     sudo kubeadm init --control-plane-endpoint etcd-proxy:6443 --pod-network-cidr 10.96.0.0/12 --service-cidr=172.100.0.0/12 --apiserver-advertise-address=192.168.56.110 --upload-certs
    

• Joining master nodes / Joining worker nodes

etcd

1. CA Certificate

ps -ef | grep kube | grep trusted-ca-file

2. Certificate

ps -ef | grep kube | grep cert-file

3. Key

ps -ef | grep kube | grep key-file

---

• Version Check

kubectl -n kube-system exec -it etcd-etcd-master1 -- sh -c "ETCDCTL_Api=3 etcdctl \\
--endpoints=127.0.0.1:2379 \\
--cacert=/etc/kubernetes/pki/etcd/ca.crt \\
--cert=/etc/kubernetes/pki/etcd/server.crt \\
--key=/etc/kubernetes/pki/etcd/server.key \\
version"

• Viewing etcd Members

kubectl -n kube-system exec -it etcd-etcd-master1 -- sh -c "ETCDCTL_Api=3 etcdctl \\
--endpoints=127.0.0.1:2379 \\
--cacert=/etc/kubernetes/pki/etcd/ca.crt \\
--cert=/etc/kubernetes/pki/etcd/server.crt \\
--key=/etc/kubernetes/pki/etcd/server.key \\
member list -w=table"
{"level":"warn","ts":"2024-04-14T02:11:37.823751Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCDCTL_Api=3"}
+------------------+---------+-------------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |    NAME     |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+-------------+-----------------------------+-----------------------------+------------+
| 20aa71ad29b6a6e5 | started | k8s-master1 | <https://---:2380> | <https://---:2379> |      false |
| 5d9ccb44c5eed81c | started | k8s-master3 | <https://---:2380> | <https://---:2379> |      false |
| ce986b79bf10fe5a | started | k8s-master2 | <https://---:2380> | <https://---:2379> |      false |
+------------------+---------+-------------+-----------------------------+-----------------------------+------------+

• Health Check

kubectl -n kube-system exec -it etcd-etcd-master1 -- sh -c "ETCDCTL_Api=3 etcdctl \\
--endpoints=127.0.0.1:2379 \\
--cacert=/etc/kubernetes/pki/etcd/ca.crt \\
--cert=/etc/kubernetes/pki/etcd/server.crt \\
--key=/etc/kubernetes/pki/etcd/server.key \\
endpoint health --cluster -w=table"

127.0.0.1:2379 is healthy: successfully committed proposal: took = 41.570972ms

• Viewing etcd Leader and Overall Status

kubectl -n kube-system exec -it etcd-etcd-master3 -- sh -c "ETCDCTL_Api=3 etcdctl \\
--endpoints=127.0.0.1:2379 \\
--cacert=/etc/kubernetes/pki/etcd/ca.crt \\
--cert=/etc/kubernetes/pki/etcd/server.crt \\
--key=/etc/kubernetes/pki/etcd/server.key \\
endpoint status --cluster -w=table"

{"level":"warn","ts":"2024-04-14T02:15:03.670079Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCDCTL_Api=3"}
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| <https://---:2379> | ---|  3.5.10 |  6.2 MB |     false |      false |        35 |      31099 |              31099 |        |
| <https://---:2379> | ---|  3.5.10 |  6.1 MB |      true |      false |        35 |      31099 |              31099 |        |
| <https://---:2379> | ---|  3.5.10 |  6.3 MB |     false |      false |        35 |      31105 |              31104 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

---

Kubernetes and etcd Cluster Leader Roles

Detailed Function Descriptions

Kubernetes Master Node Leader

• API Server: Handles API requests and manages the cluster state

• Scheduler: Schedules new pods to appropriate nodes

• Controller Manager: Runs various controllers to maintain desired state (e.g., Replica Controller)

• Cluster Coordination: Oversees cluster coordination, state monitoring, and necessary actions

etcd Cluster Leader

• Handling Write Requests: Manages all write requests within the cluster

• Maintaining Data Consistency: Ensures data consistency across all nodes

• Leading Leader Election: Oversees the election of a new leader if the current leader fails

• Managing Cluster State: Monitors and manages the overall cluster state, ensuring normal operation