Secure your Kubernetes services with NGINX ingress controller, tls and more.

Disclaimer: samples provided in this post were tested both in Azure Container Services (AKS) and Kubernetes provided by Docker for Windows.

In previous posts I showed you how to Run a Precompiled .NET Core Azure Function in a Container and how to Deploy your first Service to Azure Container Services (AKS).

By now you should be able to run your own services in Kubernetes and starting to wonder about how can you give answers to questions such as:

Is there an easy way add TLS support to my services?
Can I add whitelisting and other nice features such as rate limits (“throtling”)?

Through this post I’ll walk you through a series of samples to show you how you can answer those questions by deploying a NGINX Ingress Controller to your Kubernetes cluster.

Prerequisites:

Kubernetes deployed and working experience.
kubectl installed
helm

Expose a service with public IP

Let’s start by deploying the following service to your Kubernetes cluster, by saving the following content yaml content to a file named dni-function.yaml:

 1apiVersion: apps/v1beta1
 2kind: Deployment
 3metadata:
 4  name: dni-function
 5spec:
 6  replicas: 1
 7  template:
 8    metadata:
 9      labels:
10        app: dni-function
11    spec:
12      containers:
13      - name: dni-function
14        image: cmendibl3/dni:1.0.0
15        ports:
16        - containerPort: 80
17---
18apiVersion: v1
19kind: Service
20metadata:
21  name: dni-function
22spec:
23  type: LoadBalancer
24  ports:
25  - name:
26    port: 9000
27    targetPort: 80
28  selector:
29    app: dni-function

Now deploy it to Kubernetes:

1kubectl apply -f ./dni-function.yaml

In a few seconds you’ll have a working Web API (Validates Spanish National Identification Numbers) with the service type set to LoadBalancer which means that the solution is exposed to the world using a public IP.

If you are running on localhost, you can try it executing:

1curl -k http://localhost:9000/api/validate?dni=88410248L

which should return true.

Make the service private (No public IP)

Now in order to secure the API let’s make the service private changing it’s type to ClusterIP. So update the contents of dni-function.yaml as follows:

 1apiVersion: apps/v1beta1
 2kind: Deployment
 3metadata:
 4  name: dni-function
 5spec:
 6  replicas: 1
 7  template:
 8    metadata:
 9      labels:
10        app: dni-function
11    spec:
12      containers:
13      - name: dni-function
14        image: cmendibl3/dni:1.0.0
15        ports:
16        - containerPort: 80
17---
18apiVersion: v1
19kind: Service
20metadata:
21  name: dni-function
22spec:
23  type: ClusterIP
24  ports:
25  - name:
26    port: 80
27  selector:
28    app: dni-function

and redeploy:

1kubectl delete -f ./dni-function.yaml
2kubectl apply -f ./dni-function.yaml

Now you don’t have a public endpoint and therefore any attempt to query the service will result in a Service Unavailable response.

Deploy the NGINX Ingress Controller

It’s time to deploy the NGINX Ingress Controller: a daemon, deployed as a Kubernetes Pod which provides a simple yet effective way to configure features such as TLS, Whitelisting, rate limits, etc…

Let’s use Helm to deploy the ingress controller:

1helm install stable/nginx-ingress --name my-nginx `
2    --set controller.ingressClass=nginx `
3    --set controller.service.externalTrafficPolicy=Local `
4    --set controller.service.loadBalancerIP=127.0.0.1

If you test again with:

1curl -k http://localhost/api/validate?dni=88410248L

and because there is no rule specified to route the traffic from the ingress endpoint to the private service, you’ll get a response coming from a default backend!

Deploy the first Ingress Rule

Create a file named ingress_rules.yaml with the following contents and deploy:

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: ingress-rules
 5  namespace: default
 6  annotations:
 7    kubernetes.io/ingress.class: nginx
 8    nginx.ingress.kubernetes.io/rewrite-target: /
 9spec:
10  rules:
11  - host: localhost
12    http:
13      paths:
14      - path: /
15        backend:
16          serviceName: dni-function
17          servicePort: 80

1kubectl appply -f ingress_rules.yaml

If everything is ok and you query the service again you should get a response from the Web API, so we are back to square one, but with a great difference: all traffic goes through the NGINX Ingress Controller and that fact will let us add new features to the solution.

Please note that with this rules you can use different paths to expose diferent services.

TLS

To add TLS to the service you’ll first need a certificate which you can generate with openssl. In order to speed things up save the following contents to a file named tls-secret.yaml, which provides a certificate for localhost (just for testing) and will add a secret to your Kubernetes cluster:

1apiVersion: v1
2data:
3  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMrekNDQWVPZ0F3SUJBZ0lKQVBBVlNoS1d5NWw1TUEwR0NTcUdTSWIzRFFFQkN3VUFNQlF4RWpBUUJnTlYKQkFNTUNXeHZZMkZzYUc5emREQWVGdzB4T0RBeU1qVXhOelUzTURSYUZ3MHhPVEF5TWpVeE56VTNNRFJhTUJReApFakFRQmdOVkJBTU1DV3h2WTJGc2FHOXpkRENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DCmdnRUJBT012c25IOEhmT2dSSnE5UDRLdGRudnNkWEtaa0tYMHpNclpNdURHWnppeXFucEMwVzdwRVZOV0lmK2kKcFNpcGU2RWNBeDlWakNmUWxZbHJuNnZBLzFLUTlLRW84SURicUF3ZTloVGFLK2dRV21IblJGaFhFcCs5ZmxqbQptcVRQS3JHNEgydnlZTWQwYnpacUlJQUZHMTV3NzZLODZNRzhxVUR0V0NRNWxvWFNBZUtUVzcwSVNMSXVNZ016CklpV2lsVjB4Ly9MS2RHZDJKRi9wZVVBUGhqT05SNUJ2cHArUXlhQ3gvTDNFaGFuWE56amZXUi8zZ084SUZwN0IKcmhpcWwzWWNCZHJES2xTaE8raTFDY3FnbGZUdEdkaXVENWZHYXB3QUQ5eVNwV081bTFSR1NTaHZmTktHR05XVApOeDczemQ4b0FMckFHbHR6K2I4VjlSaTMrNThDQXdFQUFhTlFNRTR3SFFZRFZSME9CQllFRlB2S1BFT0U2VUFnCnVLTFA5QjRDczkzc2d3SnRNQjhHQTFVZEl3UVlNQmFBRlB2S1BFT0U2VUFndUtMUDlCNENzOTNzZ3dKdE1Bd0cKQTFVZEV3UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFDYVM0eTFZMnVjdzRsOEJMVGJ0NW9QVApnQXpKSW5iYTVET3NWZElEcmZEa0FOOUJhcVh2Y011dzlmN0xYc0dlenNiMmdCSm1GbllpcUxLcTBYdmVBU2x2ClJoS0R5OVJhTThpeTV0eVdVZEJXSzJFZWtvUEJHOFhMVHhScnorbEpoQmpMNXhTS3dSMDN3MW45OEVQbWRFOUkKTS90OWdHKzdZYnZyaVZhdTFINURiZWtTZld0L2pIZG9UU2hwNEJFTVNxelJSaHpqbmNUamZoR0J2KzR2cGFCRAo1aHZ0WHFsekZHODdRR05LQWdHK01hdFFLY1JCTFppNndkRDk4djdMV2NROGhydWxGT2lxeXNmNDh3bzZodEhyCm14S3BnVWpHbE52dVdKZmhNWTZCdlkyeFlveHB2R0d3Z1hnNWsxQUFldzNjK0RRTWpUUjU3dnVTZ2tnTGZVaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
4  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRRGpMN0p4L0Izem9FU2EKdlQrQ3JYWjc3SFZ5bVpDbDlNeksyVExneG1jNHNxcDZRdEZ1NlJGVFZpSC9vcVVvcVh1aEhBTWZWWXduMEpXSgphNStyd1A5U2tQU2hLUENBMjZnTUh2WVUyaXZvRUZwaDUwUllWeEtmdlg1WTVwcWt6eXF4dUI5cjhtREhkRzgyCmFpQ0FCUnRlY08raXZPakJ2S2xBN1Zna09aYUYwZ0hpazF1OUNFaXlMaklETXlJbG9wVmRNZi95eW5SbmRpUmYKNlhsQUQ0WXpqVWVRYjZhZmtNbWdzZnk5eElXcDF6YzQzMWtmOTREdkNCYWV3YTRZcXBkMkhBWGF3eXBVb1R2bwp0UW5Lb0pYMDdSbllyZytYeG1xY0FBL2NrcVZqdVp0VVJra29iM3pTaGhqVmt6Y2U5ODNmS0FDNndCcGJjL20vCkZmVVl0L3VmQWdNQkFBRUNnZ0VCQUpqbEhza0xqZlRLSmFHbVA3bm9sOWJxMmxnWDlYdGE5d0NGa0hJcDFJb1oKNUJXSUpuN29LQnJYMnVXNlJrREpYMFNjSDVYVTh4QlFsbkwzbFd2MzVWMWg1T0VaTmxMaWdZUTJ5aEphaWpZUgoyMklNVExqUFVOOWtua1dpWE8wUjUzL1hsSDRIanc1czAvUGhGS0pUelltUHBCYjM0QVdTdkszUGpnUkRKWVJFCi9ybEF0SzZSaHFKNThBS0dPOGp3OEd5WEJhTzJRVkVJSTVVdUM1QTBhTGMvUjJxS1ZpdzJ6bGJ6TVRNUVpKc2QKVjRubklYS29oL1R6WWZjdk90OVZKOTRqWlRmSGtRdHBHbzgwbFVhNGtFMXY2b2gyZG5GcThtZ3M5a3ZDV2ZLSwpTTlVhNkU2UGlMenFzR01VRE85enVHRmpUQm9xMDdBamNSVHMvS1NaOEdFQ2dZRUEvUGlkTUtGQnRaMHp6TnpOClB0Q1FrMzZkZ3VMZ2xhSVl5Qk5JSXl5SG84WW5wS3dONWJ1ZjM2ZktIWmZMeUt3SnNFMkhiNWQ0aXI1RzNFaVAKYzBhYjJFQnpQMTdyaUVVb2htVkJZTUNjY1ZlMi9qWnduZ2ducjBpcFEvc3ZqSFdrYlBTcFozam9ySjV4Z1owYQpXM21yN0U1Mi85THQxTGFYeXdRMkIydDAvR2tDZ1lFQTVlZ01yVzIzMlIyS3NrQVRDWDV3UDNwYk9SUkJkaEU5Cmh0a2JDTXc0ZDFHU1hBOElWSEVKRkZVdkFjODhiY0ZlckxXQzZkd0V1N2pTRnUyRUwwcEdKWmFTQ2NoTllrUkIKQ1F1MFhOdDBzUzRoRzhoUFRiejZROVZ6RzBjVm91RUlsdlVEc0dCU3pOV2FYck5lSDY5MktPMWM5akd4RzVITgozTC9VdzM2STFzY0NnWUJYMUhHdkNxM253bmJUciszSzIxcjIrc1R4UnBnM0c1cURETDdGQjVib2M4b2IwR2phCjFIUERrVndKUGtUUW5YcVhyYk5TT1VMdTJQVjlVZXdNVi8yUDdZQ1dCZnk4eVZZeW8wRTV1R1lZckIycTBYZjAKUmx5UTdTZG5wUFJ6VGYwU256ZVo1MDdSY0FsMHVQa0h2WXpGZE5DNExhSEpjc1B0QnI5RGdEbVQwUUtCZ0NDbQpPcDZxZFRCdEpKUTV5enBPN1d2bVdXd2F0MDBvRjUrOTF6d0JuSWM5VzFhZGYrWldBeDhURmREZytFang3QnNFCnorbWNLRVBzZEZGek81Rm5yOXlJcklhZEhuZzFEek5VcVRHQ3JPaTRqMVVkdGoxbzkvV0lLNGVWS2JwdTBNUjMKV1NYRUdCNGt1MzUxWkltRlpuZGJkaGMwYVYxcjhGdElGdFFJZFRCakFvR0JBSXV1dGFzQjFFT2dUZUVtTGJqYgpKdWVlZVRtSUQvWHpLMldpSE1ydG5HOHN0TWF0ai9VWjZUcTJMQnh4SkJqdVRFOHF5dUtYeG1KTmUxSnRkbUdKClU2bkZmWXo4OW1QSmlWK2dtSWU1L1VNNGJRbnB2SFNJdEtyNHFCREgvTkY0Z21xdGQ3MDlnTm5XRS9jdk1sQWkKQndYNWh2TzFWdndzZmRNeE0rSGIwVUxPCi0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
5kind: Secret
6metadata:
7  name: tls-secret
8  namespace: default
9type: Opaque

Now modify the ingress_rules.yaml from previous steps to include the tls section as shown below:

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: ingress-rules
 5  namespace: default
 6  annotations:
 7    kubernetes.io/ingress.class: nginx
 8    nginx.ingress.kubernetes.io/rewrite-target: /
 9spec:
10  tls:
11  - hosts:
12    - localhost
13    secretName: tls-secret
14  rules:
15  - host: localhost
16    http:
17      paths:
18      - path: /
19        backend:
20          serviceName: dni-function
21          servicePort: 80

Deploy both files:

1kubectl apply -f ./tls-secret.yaml
2kubectl apply -f ./ingress_rules.yaml

In this new state, if you query the service you should get redirects to the TLS endpoint, so try:

1https://localhost/api/validate?dni=88410248L;

and yes! the service is responding and only accesible through the TLS endpoint!

Whitelisting

To restrict the service in a way that only a list of IPs can access it, modify the ingress_rules.yaml to add the whitelist-source-range annotation:

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: ingress-rules
 5  namespace: default
 6  annotations:
 7    kubernetes.io/ingress.class: nginx
 8    nginx.ingress.kubernetes.io/whitelist-source-range: '192.168.65.3/32'
 9    nginx.ingress.kubernetes.io/rewrite-target: /
10spec:
11  tls:
12  - hosts:
13    - localhost
14    secretName: tls-secret
15  rules:
16  - host: localhost
17    http:
18      paths:
19      - path: /
20        backend:
21          serviceName: dni-function
22          servicePort: 80

and deploy:

1kubectl apply -f ./ingress_rules.yaml

Feel free to try different ranges and understand how you can block or enable access to your service.

Rate limits

Now it’s time to protect the service applying some kind of throtling. Modify the ingress_rules.yaml to add the limit-connections and limit-rps annotations:

 1apiVersion: extensions/v1beta1
 2kind: Ingress
 3metadata:
 4  name: ingress-rules
 5  namespace: default
 6  annotations:
 7    kubernetes.io/ingress.class: nginx
 8    nginx.ingress.kubernetes.io/whitelist-source-range: '192.168.65.3/32'
 9    nginx.ingress.kubernetes.io/limit-connections: '10'
10    nginx.ingress.kubernetes.io/limit-rps: '1'
11    nginx.ingress.kubernetes.io/rewrite-target: /
12spec:
13  tls:
14  - hosts:
15    - localhost
16    secretName: tls-secret
17  rules:
18  - host: localhost
19    http:
20      paths:
21      - path: /
22        backend:
23          serviceName: dni-function
24          servicePort: 80

and deploy:

1kubectl apply -f ./ingress_rules.yaml

You just limited the number of concurrent connections from a given client to 10 and it’s allowed number of requests per second to 1. You see the power here don’t you?

That one wraps it up for today. Hope you enjoyed the ride!

Please download all code and files here and be sure to check the online documentation to learn more about the annotations and available features.