dev-ops-challenges

Fix Python App Deployed on Kubernetes Cluster

Technical Overview

Deploying microservices on Kubernetes relies on the synchronization of container settings and network exposure configurations. A typical Python Flask application runs an internal WSGI server that binds to a specific port inside the container (defaulting to 5000).

If a configuration error occurs during update operations, the application can fail in two primary ways:

  1. Image pull errors (ImagePullBackOff): Typing the wrong container registry image or tag stops container creation.
  2. Port forwarding mismatches (Connection refused): If the service’s targetPort does not match the port the application container is listening on, the networking rules generated by kube-proxy will route traffic to an empty port, resulting in timeout or connection errors.
graph TD
    subgraph Mismatched Routing (Broken)
        User1[User traffic] -->|Accesses NodeIP:32345| Node1[Node]
        Node1 -->|Service TargetPort: 80| Pod1[Pod: flask-deployment]
        Pod1 -->|Connection Refused| App1[Flask Container: Listening on 5000]
    end
    
    subgraph Corrected Routing (Fixed)
        User2[User traffic] -->|Accesses NodeIP:32345| Node2[Node]
        Node2 -->|Service TargetPort: 5000| Pod2[Pod: flask-deployment]
        Pod2 -->|Traffic Routed Successfully| App2[Flask Container: Listening on 5000]
    end

Troubleshooting Port Inconsistencies & Container Images

Diagnosing multi-tier communication issues requires inspecting logs and verifying endpoints sequentially.

1. Debugging Container Port Mismatches

When a client contacts a NodePort service, traffic undergoes multiple translations: \(\text{NodePort (e.g. 32345)} \longrightarrow \text{Service Port (e.g. 80)} \longrightarrow \text{TargetPort (e.g. 5000)} \longrightarrow \text{Container Port (e.g. 5000)}\)

If the targetPort in the Service spec is configured incorrectly (e.g., set to 80 instead of 5000), traffic will hit the container’s port 80. Because the Flask server inside the container is only listening on port 5000, the connection is immediately refused.

Finding the Correct Application Port:

2. Resolving Image Pull Failures

When a Pod is stuck in ImagePullBackOff, the kubelet has failed to download the image. Check the exact name and registry. In this challenge, the image was misconfigured as poroko/flask-app-demo (which does not exist) instead of the correct image poroko/flask-demo-app.

Infrastructure & Configuration Requirements

Step-by-Step Implementation

Step 1: Connect to the Kubernetes Jump Host

SSH from your workstation into the cluster command host:

ssh thor@jump_host_ip

Step 2: Diagnose the Deployment Failures

List the pods to check their statuses:

kubectl get pods

Expected Output showing ImagePullBackOff:

NAME                                READY   STATUS             RESTARTS   AGE
flask-deployment-7f8a9b0c-abcde     0/1     ImagePullBackOff   0          2m

Describe the Pod to check the event logs:

kubectl describe pod flask-deployment-7f8a9b0c-abcde

Notice the event at the bottom indicating a pull failure:

Events:
  Type     Reason   Age                   From               Message
  ----     ------   ----                  ----               -------
  Warning  Failed   2m (x3 over 2m)       kubelet            Failed to pull image "poroko/flask-app-demo": rpc error: code = NotFound desc = failed to pull and unpack image

Step 3: Correct the Container Image Spec

Open the Deployment spec in your default terminal editor:

kubectl edit deployment flask-deployment

Locate the image key and change the wrong image name to the correct one:

# BEFORE
      - name: flask-container
        image: poroko/flask-app-demo

# AFTER
      - name: flask-container
        image: poroko/flask-demo-app

Save and exit. The Deployment will automatically trigger a rolling update.

Step 4: Verify Container Logs and Port Binding

Wait for the Pod to initialize and transition into Running state:

kubectl get pods

Expected Output:

NAME                                READY   STATUS    RESTARTS   AGE
flask-deployment-6f5d4c3b-klmno     1/1     Running   0          20s

Check the logs of the running container to see which port the Python application binds to:

kubectl logs flask-deployment-6f5d4c3b-klmno

Expected Output:

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

Note that the application binds to port 5000.

Step 5: Correct the Service TargetPort

Describe the flask-service Service to inspect its ports:

kubectl describe service flask-service

Expected Output showing wrong targetPort:

Name:                     flask-service
Namespace:                default
Type:                     NodePort
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  32345/TCP

Notice that the Service’s TargetPort is set to 80, but the Flask application inside the container is running on port 5000.

Edit the service configuration:

kubectl edit service flask-service

Modify the targetPort to match the Flask container port (5000):

# BEFORE
  ports:
  - name: http
    nodePort: 32345
    port: 80
    protocol: TCP
    targetPort: 80

# AFTER
  ports:
  - name: http
    nodePort: 32345
    port: 80
    protocol: TCP
    targetPort: 5000

Save and exit the editor.

Post-Deployment Verification

1. Verify Service Port Translation

Confirm the service reflects the correct targetPort:

kubectl get svc flask-service

Expected Output:

NAME            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
flask-service   NodePort   10.96.142.11   <none>        80:32345/TCP   5m

2. Verify Application Accessibility

Send an HTTP request from outside the cluster network targeting NodePort 32345:

curl -I http://<NODE_IP>:32345

Expected Output:

HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: ...
Server: Werkzeug/0.16.x Python/3.x
Date: Sun, 19 Jul 2026 ...

The Python Flask application is now successfully debugged, routing traffic correctly, and accessible!