Minikube on Ubuntu in Multipass VM on M1 mac

Note: If your looking for other k8s options to give a try, I can also recommend running k3s on Ubuntu in Parallels as described here.

So there I am, having made the switch to the a new M1 MacBook and trying to find myself a development setup that works. I’m coming from a i9 mac on which I ran minikube using the vmware driver for running MongoDB, Kafka and ElasticSearch, and spun up some docker containers for compiling our application frontend. Docker has never been a great experience on the Mac, but with the arm64 architecture it appears to have become even worse. That’s where things get complicated when using minikube as the docker driver is currently the only driver that’s supported on M1 macs.

Multipass

I decided to give multipass from Canonical a try. Multipass allows you to spin up headless Ubuntu instances from the command line. They even have a minikube workflow that bootstraps an Ubuntu image running minikube for you. You can spin up an instance with 4 cpus and 10G of RAM running minikube using the following command.

multipass launch minikube -c 4 -m 10G

Eventually you’ll get the "Launched: minikube" message. Let’s (re)start minikube with some additional options that will make our life easier later on.

By typing multipass list you’ll get a list of the installed Ubuntu instances. To access the shell of the Ubuntu instance type.

multipass shell minikube

The minikube v1.25.1 version shipped with this multipass instance contains a nasty bug that causes minikube to lose all configuration on a restart. So we’ll replace it with a patched version first.

curl -LO https://storage.googleapis.com/minikube-builds/13506/minikube-linux-arm64 && sudo install minikube-linux-arm64 /usr/bin/minikube

Now let’s (re)start minikube with some additional configuration that will make our life a bit easier. Specifically the api-server ips will allow us to connect with kubectl on the Mac after setting up portforwarding. To find out what the api-server api should be install the net-tools package so you can use ifconfig.

sudo apt install net-tools

Now run ifconfig and find the enp0s1 interface and note the IP. In my case that’s 192.168.64.16. Use yours for the --apiserver-ips argument in the command below.

minikube start --driver=docker  --extra-config=apiserver.service-node-port-range=1-65535 --apiserver-ips=192.168.64.16

Now run kubectl to get a list of the running pods to see if minikube is running.

kubectl get pods

Obviously there aren’t any as we didn’t deploy anothing to the minikube cluster yet. As it’s pretty hard to find a simple echo-server example that is available and working for arm64 I took the one from https://github.com/polyverse/node-echo-server, rebuilt it for arm64 and pushed it do dockerhub as uiterlix/node-echo-server.

The see if it all works, let’s deploy the ‘hello world’ echoserver application into the cluster. This echo server starts a http server on port 8080 and prints the request path when called.

kubectl create deployment hello-world --image=uiterlix/node-echo-server

kubectl will say deployment.apps/hello-node created. If we do a kubectl get pods now, well get the list of pods with a single hello node pod in there.

NAME                          READY   STATUS    RESTARTS   AGE
hello-world-ddfcb7b68-7z9ph   1/1     Running   0          18s

That’s aweseome! In order to be able to call the service we need to expose its port.

kubectl expose deployment hello-world --type=NodePort --port=8080

Now we can ask minikube for the URL of our service

minikube service hello-world --url

This returns something like:

http://192.168.49.2:30214

Now we can test whether our service works using a simple curl command.

curl http://192.168.49.2:30214/hello

Which, when all is well, gives the following response.

Requested: /hello

As we know it works, we can delete our service and deployment.

kubectl delete service hello-world
kubectl delete deployment hello-world

To be able to use kubectl from our MacOS host right into the multipass vm running minikube we still need to do a couple of things. First copy your public ssh key (probably in ~/.ssh/id_rsa.pub) to the ~/.ssh/authorized_keys file in your multipass vm. This allows use to use scp to copy the certificates needed for k8s authentication to our MacOS host.

You can use the following script on your mac to copy the required certificates to your local machine. Finally it will print a kube config for you to incorporate in ~/.kube/config. Replace the values for the UBUNTU_IP and LOCAL_USER variables with your own.

UBUNTU_IP=192.168.64.16
LOCAL_USER=uiterlix
REMOTE_USER=ubuntu
MINIKUBE_PATH=/home/$REMOTE_USER/.minikube
BASE_PATH=$MINIKUBE_PATH/profiles/minikube
HOME_DIR=/Users/$LOCAL_USER
PROFILE_DIR=$HOME_DIR/.minikube-multipass
CERTS_DIR=$PROFILE_DIR/certs
rm -rf $PROFILE_DIR
mkdir $PROFILE_DIR
mkdir $PROFILE_DIR/certs
scp $REMOTE_USER@$UBUNTU_IP:$MINIKUBE_PATH/ca.crt $CERTS_DIR
scp $REMOTE_USER@$UBUNTU_IP:$BASE_PATH/client.crt $CERTS_DIR
scp $REMOTE_USER@$UBUNTU_IP:$BASE_PATH/client.key $CERTS_DIR
scp $REMOTE_USER@$UBUNTU_IP:/home/$REMOTE_USER/.kube/config $PROFILE_DIR

sed -i '' "s/\/home\/ubuntu\/.minikube\/ca.crt/\/Users\/$LOCAL_USER\/.minikube-multipass\/certs\/ca.crt/g" $PROFILE_DIR/config
sed -i '' "s/server.*/server: https:\/\/$UBUNTU_IP:8443/g" $PROFILE_DIR/config
sed -i '' "s/\/home\/ubuntu\/.minikube\/profiles\/minikube\/client.key/\/Users\/$LOCAL_USER\/.minikube-multipass\/certs\/client.key/g" $PROFILE_DIR/config
sed -i '' "s/\/home\/ubuntu\/.minikube\/profiles\/minikube\/client.crt/\/Users\/$LOCAL_USER\/.minikube-multipass\/certs\/client.crt/g" $PROFILE_DIR/config
sed -i '' "s/: minikube/: minikube-multipass/g" $PROFILE_DIR/config
cat $PROFILE_DIR/config
cp $PROFILE_DIR/config $HOME_DIR/Desktop/

Now it’s time to forward the k8s api port. You can use the following script in the ubuntu vm.

DOCKER_NET=`ifconfig |grep br- |cut -c1-15`
HOST_IP=192.168.49.2
sudo iptables -t nat -A PREROUTING -p tcp -i enp0s1 --dport 8443 -j DNAT --to-destination $HOST_IP:8443
sudo iptables -A FORWARD -i enp0s1 -o $DOCKER_NET -j ACCEPT
sudo iptables -A FORWARD -i $DOCKER_NET -o enp0s1 -m state --state RELATED,ESTABLISHED -j ACCEPT

If all is well and you’ve included the k8s config into ~/.kube/config you should be able to use kubectl on your mac with the minikube cluster in the Ubuntu vm.

kubectl get pods --context minikube-multipass

See the script below for forwarding multiple ports.

Port forwarding

With minikube running my infrastructure services I want to be able to access these from my application running in IntelliJ on MacOS. As they run inside a docker container of an Ubuntu VM in Multipass some port forwards are needed to make it work.

#!/bin/bash
MINIKUBE_IP=192.168.49.2
MINIKUBE_NET=`ifconfig |grep br- |cut -c1-15`
HOST_NET=enp0s1
USER=`whoami`

echo "Running script as: $USER" 
echo "Forwarding to network $MINIKUBE_NET"

ports=(2181 3333 8000 8100 8443 9092 9200 9300 27017)
for port in "${ports[@]}"
do
    echo "Forwarding port ${port}"
	sudo iptables -t nat -A PREROUTING -p tcp -i $HOST_NET --dport $port -j DNAT --to-destination $MINIKUBE_IP:$port    
done

sudo iptables -A FORWARD -i $HOST_NET -o $MINIKUBE_NET -j ACCEPT
sudo iptables -A FORWARD -i $MINIKUBE_NET -o $HOST_NET -m state --state RELATED,ESTABLISHED -j ACCEPT

Uninstalling Multipass

I do encounter multipass hanging sometimes when starting the minikube VM. Unfortunately uninstalling multipass seems to be the only remedy for now. To uninstall multipass execute:

sudo sh /Library/Application\ Support/com.canonical.multipass/uninstall.sh