Kubernetes Ingress Configurations and Static Assets¶
Estimated time to read: 11 minutes
- Originally Written: July, 2020
There are a number of benefits to using an ingress.
- Centralized SSL termination
- Rule based routing
- Reduction in IP address usage
However you may run into potential issues with routing rules for some applications and designs. I haven't found a good consolidated document on the problems and fixes so have started documenting what we've found.
First let's recap the Kubernetes Ingress resource.
An Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.
Here's an example configuration.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: guestbook-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /guestbook
backend:
serviceName: guestbook
servicePort: 80
- path: /sockshop
backend:
serviceName: sockshop
servicePort: 80
You can access the services through the IP address of the Ingress Controller, in this case 10.113.105.202. The paths (/guestbook and /sockshop) are converted into the nginx.conf configuration on the controller.
When It Works Correctly¶
To see a basic example we can use the Kubernetes Guestbook example application.
https://github.com/kubernetes/examples/blob/master/guestbook/all-in-one/guestbook-all-in-one.yaml
Save the following to a file e.g. ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: guestbook
namespace: guestbook
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: frontend
servicePort: 80
Deploy this application to your cluster
Once it's running you should have access to the guestbook application through the ingress controller e.g. http://10.113.105.202/
If you open the developer console in your browser you should see a number of files downloaded as the page loads.
Let's see how it works¶
In this example, the path: / configuration in the ingress YAML file acts as a wildcard, therefore matching all assets that are required (index.html, controllers.js, guestbook.php).
Info
In Kubernetes 1.18 there are ingress enhancements (Exact and Prefix keywords) to provide more granular control.
https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/
Since there's a match, the controller will forward these to the associated service, frontend.
The frontend webserver, Apache, has been configured as default and is serving content from the root directory, /, so returns the requested files.
More than One Application¶
Question
It's great that the guestbook is working however what happens if we want to deploy a second application? Should we also use the root path?
rules:
- http:
paths:
- path: /
backend:
serviceName: frontend
servicePort: 80
- path: /
backend:
serviceName: sockshop
servicePort: 80
Question
How could we direct traffic to the right application if the same path points to two different services?
In this case we might want to configure a specific path per application.
rules:
- http:
paths:
- path: /guestbook
backend:
serviceName: frontend
servicePort: 80
- path: /sockshop
backend:
serviceName: sockshop
servicePort: 80
Here's the result when we try to reach the guestbook with our new path, /guestbook
As you can see, when it reaches the ingress controller it matches the path and so the traffic is sent to the web server. However, the web server is serving content from the root directory and we are trying to access files located in the subdirectory, /guestbook. If you monitor the logs you should see a /guestbook 404 since the content can't be located.
Rewriting the Location¶
We need to specify /guestbook in the browser but also need to have the web server understand what we need. We can use the NGINX Ingress rewrite-target annotation in Kubernetes to overcome this issue.
The annotations in the Kubernetes Ingress allow us to include additional configuration required for our environment. In this case, we can use the ingress YAML file to provide NGINX configurations which will be added to the nginx.conf file automatically.
Info
This may be different depending on which ingress controller you are using
Using the rewrite-target annotation we can specify the target URI where the traffic must be redirected.
https://kubernetes.github.io/ingress-nginx/examples/rewrite/
So in our example we first match the path, /guestbook, and then NGINX will rewrite it to / before sending it to the web server. Since it's now rewritten to the root directory, /, the index.html file is returned correctly as was the case in the first scenario.
However there's still a problem as you can see below.
It turns out that within index.html is a reference to a few other static assets (CSS and JS files). When the page loads it tries to download the required files but does not provide any subdirectory to indicate that this is for the guestbook application. It tries to access 10.113.105.202/controllers.js, however the path we've configured is looking for 10.113.105.202/guestbook.
A 404 error is returned since there's no rule to match controllers.js and forward to the web server.
Solving the problem¶
To overcome this issue you can update the rewrite-target to only append the second capture group. Here's an updated example.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: my-cluster.my-domain.com
http:
paths:
- backend:
service:
name: guestbook
port:
number: 80
path: /guestbook(/|$)(.*)
pathType: Prefix
The two key lines to note are:
nginx.ingress.kubernetes.io/rewrite-target: /$2
This annotation tells the Nginx ingress controller to rewrite the URL of incoming requests. /$2 refers to the second group in the regular expression, denoted by (.*) in the path. In essence, it means "change the URL of the incoming request to the value captured in the second group of the regular expression".
path: /guestbook(/|$)(.*)
This is a regular expression defining the paths that this ingress rule will apply to. Any URL path that starts with /guestbook will match this rule. The (/|$) means "either a slash or the end of the line". The (.*) means "any number of any characters", which is a catch-all for anything that follows /guestbook/ in the URL. Essentially, it captures any path that starts with /guestbook/ or simply /guestbook.
So, for example, if an incoming request URL is /guestbook/page1, the ingress rule will rewrite the target URL as /page1 before routing it to the appropriate service.
Info
Sometimes you might find it works when accessing https://my-domain.com/guestbook/ (note the trailing slash) but not https://my-domain.com/guestbook.
URL paths are hierarchical, similar to file paths in a file system. When you have a trailing slash like https://my-domain.com/guestbook/, it's interpreted as a directory and the browser will request https://my-domain.com/guestbook/controller.js rather than just controllers.js.
When it's received at the Kubernetes Ingress, the ingress rule will rewrite the target from guestbook/controller.js to /controllers.js before sending it to the server which returns controllers.js.
There are some other options outlined below to solve this problem however see if the rewrite syntax above will work as this may be the easiest option.
Another Option: Serve Static Content from a Different Location¶
The first option is to separate the static content from the main HTML files and host them externally.
You can see an example of this in the guestbook application by looking at the bootstrap.min.css file. Bootstrap is a popular front-end open source toolkit, and so is hosted on a CDN and available for anyone to use.
When the page loads, the Bootstrap CSS file is downloaded directly from the CDN. This means we don't need to worry about ingress or web server rules to server this file.
The drawback of this approach is having to manage these files being hosted somewhere else.
Another Option: Modify the Application Code¶
The second option is to modify the application code to prepend the path when loading the assets . In this example we would update the HTML to reference the file, guestbook/controllers.js.
When the browser attempts to download the file it will request the path, 10.113.105.202/guestbook/controllers.js. We have a rule that matches /guestbook so the ingress controller will rewrite this to the root directory, /, and request controllers.js from the web server.
Info
In the example flow diagram above you'll see the path is updated to path: /guestbook/* to match all our files and subpaths.
The drawback of this approach is having to update all the code, which in some cases can be a very large task.
Another Option: Change the Apache Serving Directory¶
The final option we'll look at is to modify the configuration on the web server, in this case Apache.
From the testing and examples above, we know that we need a subdirectory, /guestbook, for this application. However when we added this the web server returned a 404 since it was serving content from the root directory, /, and couldn't find the guestbook subdirectory files.
To solve this problem we can tell Apache to stop serving content from /, and instead use /guestbook as the root directory.
In the apache2.conf configuration file you need to modify the DocumentRoot so it reads:
On Ubuntu you should be able to find apache2.conf in the /etc/apache2/ directory.
Update this file and then restart Apache if needed. You should also remove the rewrite-target annotation from your ingress as we are no longer serving content from root, /
If everything has worked you should now be able to access your guestbook application successfully and all static assets loaded correctly.