Exposing TCP/UDP Services on Kubernetes with Ingress-Nginx

May 30, 2022

4 min read

Exposing TCP/UDP Services on Kubernetes with Ingress-Nginx

Table of Contents

Introduction

Exposing custom TCP/UDP ports on Kubernetes can be challenging, especially for those who are not heavily experienced in DevOps. I spent some time digging GitHub issues in order to figure this out, hence decided to write a short blog post.
In this article, I will guide you on how to expose a TCP port for the Redis deployment using the Ingress-Nginx controller. While Ingress-Nginx doesn't support TCP or UDP services out of the box, it provides the necessary tools to extend its functionality.

Creating a Redis Deployment

To begin, let's create a Redis deployment by defining a redis-depl.yaml file with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  selector:
    matchLabels:
      app: redis
  replicas: 1
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:latest
        ports:
        - containerPort: 6379

This YAML file creates a Redis deployment with one replica and exposes container port 6379. Apply the YAML file to your cluster using kubectl apply -f redis-depl.yaml.

Creating a ClusterIP Service for Redis Deployment

Next, create a Kubernetes service to expose the Redis deployment by creating a redis-service.yaml file with the following contents:

# redis-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  selector:
    app: redis
  ports:
  - name: tcp-redis-port
    port: 6379
    targetPort: 6379
  type: ClusterIP

This YAML file defines a Kubernetes service that exposes port 6379 for the Redis deployment within your cluster. Apply the YAML file using kubectl apply -f redis-service.yaml.

Installing Ingress-Nginx

If you haven't installed Ingress-Nginx yet, follow the official documentation specific to your cloud provider. For this blog post, we will use Scaleway as an example. Copy the contents of their deploy.yml file and save it as ingress-setup.yml. We will come back to it later to make minor modifications.

Apply Ingress-Nginx by running kubectl apply -f ingress-setup.yml.

Configuring Ingress-Nginx Controller

Assuming you have successfully installed Ingress-Nginx and confirmed that the "ingress-nginx" namespace with its active controller pods are running, we can deploy our configuration file with service routings. Create a my-ingress-routes.yaml file with the following contents:

# my-ingress-routes.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress-srv
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
  rules:
    - host: redis.example.com
      http:
        paths:
          - path: /
			pathType: Prefix
		    backend:
		      service:
		        name: redis-service
		        port:
		          name: tcp-redis-port

This YAML file creates an Ingress resource that exposes the custom TCP port for the Redis deployment.

Identifying the Issue with the Current Configuration

After completing the previous steps, you can try to connect to your Redis host or via public LoadBalancer IP redis.example.com, they will likely fail with a connection timeout. Your connection TCP request gets lost within the cluster and not properly routed.

To confirm this issue, check the nginx.conf configuration file generated by Ingress. You can access the ingress-nginx-controller-*** container pod within the "ingress-nginx" namespace and browse /etc/nginx/nginx.conf.

Ensure that within the last Stream/Server block, starting with a comment saying # TCP services, the PROXY protocol is enabled using the listen directive for port 6379.

# nginx.conf
stream {
    # ...
    # TCP services
    server {
        listen 6379 proxy_protocol;
        # ...
    }
}

Currently, there is no block with enabled TCP services. We will enable it in the next section.

Refer to the NGINX documentation on the proxy protocol and its configuration here for more information.

Configuring Ingress Controller for TCP Services

By default, Ingress does not support TCP or UDP services. However, you can enable this functionality in the Ingress controller by using the --tcp-services-configmap and --udp-services-configmap flags. These flags allow you to specify an existing config map that maps external ports to the services you want to expose.

To create the config map, follow this YAML format:

# tcp-services.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  6379: "default/redis-service:6379:PROXY"

In the example above, the service redis-service running in the default namespace is exposed on port 6379, using the same port number. We added the PROXY property to decode the incoming request parameter.

For UDP load balancing (available from NGINX version 1.9.13), you can create a similar config map for UDP services. Here's an example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: udp-services
  namespace: ingress-nginx
data:
  53: "kube-system/kube-dns:53"

In this case, the service kube-dns running in the kube-system namespace is exposed on port 53, using the same port number.

The next step to enable TCP/UDP proxy support is to expose the corresponding ports in the Service defined for the Ingress. In the ingress-setup.yml file, locate the Service block with the name ingress-nginx and add the extra configuration for our Redis ports:

# ingress-setup.yml
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
    - name: proxied-redis-tcp-6379	# added
      port: 6379 					# added
      targetPort: 6379				# added
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

Finally, indicate the config map in the deployment args of the ingress controller as follows:

args:
  - /nginx-ingress-controller
  - --tcp-services-configmap=ingress-nginx/tcp-services # added

By following these steps, you can configure the Ingress controller to support TCP and UDP services and expose them accordingly.

Now, you can apply all the above-configured files and test your Redis connection.

Conclusion

Enabling TCP and UDP services in the Ingress controller allows you to expose additional functionalities and expand the capabilities of your Kubernetes cluster. By utilizing the --tcp-services-configmap and --udp-services-configmap flags, you can configure the controller to point to existing config maps that map external ports to the services you want to expose.

However, keep in mind that if your primary goal is to simply allow public access to your Redis and not pass through your Ingress LoadBalancer, you may also choose to expose your Redis deployment directly via the corresponding LoadBalancer service of Kubernetes.

Remember to create the appropriate config maps for TCP and UDP services, specifying the desired ports and service mappings. Additionally, ensure that the necessary ports are exposed in the Service defined for the Ingress.

By following these steps, you can effectively configure the Ingress controller to handle TCP and UDP services, enabling seamless routing and load balancing for your applications.

Thank you for your time!