Troubleshooting PMF on Cloud

Following are some of the common issues you may encounter during the installation process.

mf-operator pod does not show up

Error message

You see the following error.

$ oc get rs
NAME                    DESIRED   CURRENT   READY   AGE
mf-operator-87b88494f   1         0         0       33s

$ oc describe rs mf-operator-87b88494f

Events:
Type     Reason        Age                   From                   Message
----     ------        ----                  ----                   -------
Warning  FailedCreate  96s (x14 over 2m17s)  replicaset-controller  Error creating: pods "mf-operator-87b88494f-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1000570000, 1000579999]]

Possible cause

Either the scc.yaml is not deployed or mf-operator scc assignment is not done.

Resolution

Assign mf-operator scc by using the following command:

oc adm policy add-scc-to-group mf-operator system:serviceaccounts:<project-name>

Operator pod status shows ErrImagePull

Error message

You see the following error.

$ oc get pods NAME                          READY   STATUS         RESTARTS   AGE
mf-operator-87b88494f-gtpq2   0/1     ErrImagePull   0          4s

Possible cause

Can be any of the following.

  • Image pull secret is not updated in service_account.yaml file.
  • The updated pull secret does not exist.
  • Image pull secret exists with wrong registry credentials.

Resolution

Ensure that the image pull secret is created with right registry credentials and the same secret name is updated in the service_account.yaml file.

After deploying custom resource, no pods show up

Error message

You see the following error in pods.

Failed to list mf.ibm.com/v1, Kind=MFOperator: mfoperators.mf.ibm.com is forbidden: User "system:serviceaccount:mfnew:mf-operator" cannot list resource "mfoperators" in API group "mf.ibm.com" in the namespace "mfnew"

Possible cause

The namespace name is not updated in the role_binding.yaml file or wrong namespace name is updated.

Resolution

Ensure proper namespace name is updated in the role_binding.yaml file.

The dbinit-job shows error

Error message

You see the following error.

$ oc get pods
NAME                          READY   STATUS    RESTARTS   AGE
ibm-mf-dbinit-job-9c2gb       0/3     Error     3          32s

Possible cause

Can be any of the following.

  • The database details (host, port, and name) provided in the charts_v1_mfoperator_cr.yaml (custom resource) file are not correct.
  • Databaseis not reachable.
  • The dbsecret was created with wrong credentials.
  • If db-init pod log reports error particularly for TABLE PUSH_DEVICES, then PMF database could have been created without PAGESIZE. To create a database with PAGESIZE, see Setting Up Databases to create a database with PAGESIZE.

Resolution

Correct the database details and redeploy.

PMF pods show 0/1 READY

Error message

You see the following error.

$ oc get pods
NAME                              READY   STATUS      RESTARTS   AGE
ibm-mf-defaultsecrets-job-7z42m   0/1     Completed   0          5m21s
ibm-mf-push-77fb65c758-wtg28      0/1     Running     0          5m16s
ibm-mf-server-d87ddf67f-x9n4x     0/1     Running     0          5m16s

Resolution

  • Check if the database that PMF is connected to is reachable. For more details on the issue, check the pod logs.
  • If no errors are shown on the pod logs and database is reachable, delete the existing pods so that new pods are recreated.
  • For any other issues, check the operator pod logs to get more details on the issue.

Elasticsearch data pod is not coming up properly

Possible cause

The storage class is not provided in the charts_v1_mfoperator_cr.yaml (custom resource) file or the provided PersistentVolumeClaim (PVC) does not have proper access for the Elasticsearch to write data.

Resolution

If the claimName is provided in the charts_v1_mfoperator_cr.yaml (custom resource) file, then make sure Elasticsearch data has access to the mount location by running the following commands on the mount path.

chown -R 1001:1001 <mount_path>
chmod -R ug+rwx <mount_path>

PMF routes are not created/accessible

Possible cause

The spec.ingress.hostname is not updated in the custom resource yaml file or pods are not properly running.

Resolution

Update spec.ingress.hostname and redeploy the custom resource file.

Deployment issues

For deployment issues, share the following information with PSL support.

  • The installed mf-operator version from the deploy/operator.yaml file.
  • PMF custom resource file charts_v1_mfoperator_cr.yaml).

  • Output for the following command.

     oc get pods

  • Operator pod logs by running the following command.

    oc logs <mf-operator-pod-name

  • Output for the following command for each pod.

     oc describe pod <pod-name>

  • PMF logs for all the pods.

     oc logs <pod-name>

PMF functionality issues

  • Enable PMF traces using Custom server configuration.

  • Get the logs by running the following command.

     oc cp <server-pod-name>:/logs/messages.log ./server-messages.log
 oc cp <server-pod-name>:/logs/trace.log ./server-trace.log

es-operator issues

For es-operator issues, share the following information with PSL support.

  • The installed es-operator version from the es/deploy/operator.yaml file.
  • Elasticsearch custom resource file charts_v1_esoperator_cr.yaml).

  • Output for the following command.

     oc get pods

  • Operator pod logs by running the following command.

     oc logs <es-operator-pod-name>

  • Output for the following command for each pod.

     oc describe pod <pod-name>

  • Elasticsearch logs for all the pods.

     oc logs <pod-name>
▲
Last modified on