Migrating Analytics data from On-prem MFP 8.0 to PMF Cloud 9.1 on Kubernetes

Prerequisites

  • Ensure to take snapshot of the analytics data into a directory.
  • Note the number of shards configured in the Elastic search on-prem environment.
  • Analytics data is migrated on a fresh PMF 9.1 installation.
  • PMF Elastic search component runs in the same namespace as other PMF components.

Procedure

Proceed as follows.

  1. Ensure that the elasticsearch API works.

     curl --location --request GET 'http://{ElasticSearchHost}:9200/_nodes/process?pretty'

    Where,

    {ElasticSearchHost} is IP address or hostname of the machine where the elasticsearch service is running.

  2. Add the JNDI property into the server.xml file where elasticsearch service is running. This property defines the storage path for the snapshot data.

     <jndiEntry jndiName="analytics/path.repo" value='"D:/Backup"'/>

  3. Run the following Elastic search API to create repository for backup.

     curl --location --request PUT 'http://{ElasticSearchHost}:9200/_snapshot/my_backup' --header 'Content-Type: application/json' --data '{"type": "fs","settings": {"location": "D:/Backup"}}'

  4. Run the following Elastic search API to create the snapshot data for existing Elastic search data.

     curl --location --request GET 'http://{ElasticSearchHost}:9200/_snapshot/my_backup/snapshot_1'

  5. Create a /es_backup directory on the VMs where Elastic search pods (bm-es-pr-esclient, ibm-es-pr-esmaster, ibm-es-pr-esdata) are running by using the following command.

     mkdir /es_backup

  6. Copy the snapshot data taken from on-prem elasticsearch into the /es_backup directory.

  7. Edit ibm-es-prod-* configmapby using the following command. Change the name of configmap before running the command.

     kubectl edit configmap ibm-es-prod-9-1728914385-es-configmap

    Update the following changes under “elasticsearch.yml section”.

    a. Add path.repo: /es_restore\n after script.inline: true\n. b. Change number_of_shards value to match the number of shards in your on-prem elasticsearch configuration. The default shard in PMF cloud is 3.

    Following is a sample code where path.repo was added and number_of_shards value changed to 5.

     apiVersion: v1
 data:
   elasticsearch.yml: "cluster:\n  name: ${CLUSTER_NAME}\nnode:\n  master: ${NODE_MASTER}\n
         \ data: ${NODE_DATA}\n  name: ${NODE_NAME}\n  ingest: ${NODE_INGEST}\nindex:\n
         \ number_of_replicas: 1\n  number_of_shards: 5\n  mapper.dynamic: true\npath:\n
         \ data: /data/data_${staticname}\n  logs: /data/log_${staticname}\n  plugins:
         /elasticsearch/plugins\n  work: /data/work_${staticname}      \nprocessors: ${PROCESSORS:1}\nbootstrap:\n
         \ memory_lock: false\nhttp:\n  enabled: ${HTTP_ENABLE}\n  compression: true\n
         \ cors:\n    enabled: true\n    allow-origin: \"*\"\ncloud:\n  k8s:\n    service:
         ${DISCOVERY_SERVICE}\n    namespace: ${NAMESPACE}\ndiscovery:\n  type: io.fabric8.elasticsearch.discovery.k8s.K8sDiscoveryModule\n
         \ zen:\n    ping.multicast.enabled: false\n    minimum_master_nodes: 1\nxpack.security.enabled:
         false\nxpack.ml.enabled: false\ncompress.lzf.decoder: optimal\ndiscovery.zen.ping.multicast.enabled:
         false\nbootstrap.mlockall: true\ncompress.lzf.decoder: safe\nscript.inline: true\npath.repo:
         /es_restore\n"

  8. Create hostPath volume mount in ibm-es-pr-esclient, ibm-es-pr-esmaster deployments and ibm-es-pr-esdata statefulset.

    If you are using PMF 9.1 on kubernetes,

    a. Edit the deployment ibm-es-pr-esclient by using the following command.

     kubectl edit deployment ibm-es-pr-esclient

    Add the following mountPath under “volumeMounts” and “volumes”.

     mountPath: /es_restore
 name: hp-volume

    Sample code after adding the mountPath.

     volumeMounts:
 - mountPath: /es_restore
 name: hp-volume
 - mountPath: /data
 name: storage
 - mountPath: /elasticsearch/config/elasticsearch.yml
 name: config
  subPath: elasticsearch.yml
 - mountPath: /elasticsearch/config/log4j2.properties
 name: config
 subPath: log4j2.properties

    Add the following hostPath under “volumes”.

     hostPath:
   path: /es_backup
   type: DirectoryOrCreate
 name: hp-volume

    Sample code after adding the hostPath.

     volumes:
 - hostPath:
     path: /es_backup
     type: DirectoryOrCreate
     name: hp-volume
 - emptyDir: {}
     name: storage
 - configMap:
     defaultMode: 420
     name: ibm-es-prod-9-1728914385-es-configmap
     name: config

    Save and exit.

    b. Edit the deployment ibm-es-pr-esmaster by using the following command.

     kubectl edit deployment ibm-es-pr-esmaster

    Add the following mountPath under “volumeMounts” and “volumes”.

     mountPath: /es_restore
 name: hp-volume

    Sample code after adding the mountPath.

     volumeMounts:
 - mountPath: /es_restore
 name: hp-volume
 - mountPath: /data
 name: storage
 - mountPath: /elasticsearch/config/elasticsearch.yml
 name: config
 subPath: elasticsearch.yml
 - mountPath: /elasticsearch/config/log4j2.properties
 name: config
 subPath: log4j2.properties

    Add the following hostPath under “volumes”.

     hostPath:
   path: /es_backup
   type: DirectoryOrCreate
 name: hp-volume

    Sample code after adding the hostPath.

     volumes:
 - hostPath:
 path: /es_backup
 type: DirectoryOrCreate
 name: hp-volume
 - emptyDir: {}
 name: storage
 - configMap:
 defaultMode: 420
 name: ibm-es-prod-9-1728914385-es-configmap
 name: config

    Save and exit.

    c. Edit the statefulset ibm-es-pr-esdata by using the following command.

     kubectl edit statefulset ibm-es-pr-esdata

    Add the following mountPath under “volumeMounts” and “volumes”.

     mountPath: /es_restore
 name: hp-volume

    Sample code after adding the mountPath.

     volumeMounts:
 - mountPath: /es_restore
 name: hp-volume
 - mountPath: /data
 name: storage
 - mountPath: /elasticsearch/config/elasticsearch.yml
 name: config
 subPath: elasticsearch.yml
 - mountPath: /elasticsearch/config/log4j2.properties
 name: config
 subPath: log4j2.properties

    Add the following hostPath under “volumes”.

     hostPath:
   path: /es_backup
   type: DirectoryOrCreate
 name: hp-volume

    Sample code after adding the hostPath.

     volumes:
 - hostPath:
 path: /es_backup
 type: DirectoryOrCreate
 name: hp-volume
 - emptyDir: {}
 name: storage
 - configMap:
 defaultMode: 420
 name: ibm-es-prod-9-1728914385-es-configmap
 name: config

    Save and exit.

  9. Ensure that all Elastic search pods are up and running.

  10. Ensure that the elasticsearch API works.

     curl --location --request GET 'http://{ES_MASTER_POD_IP}:9200/_nodes/process?pretty'

    Where,

    {ES_MASTER_POD_IP} is the ibm-es-pr-esmaster pod IP Address.

    To know the IP addresses of the pods, run below command:

     kubectl get pods -o wide

  11. Run the following command to restore the snapshot.

    curl --location --request POST 'http://{ES_MASTER_POD_IP}:9200/_snapshot/my_backup/snapshot_1/_restore'

    Successful response

    {"accepted":true}

    If you have already run the close index API then open the index by using the following command.

      curl --location --request POST 'http://{ES_MASTER_POD_IP}:9200/global_all_tenants/_open'

    Where,

    “global_all_tenants” is the index that was closed.

Troubleshooting

For resolving issues that you might encounter when you are migrating Analytics data from On-prem MFP 8.0 to PMF Cloud 9.1 on Kubernetes, see Troubleshooting Migrating Analytics data (On-prem MFP).

▲
Last modified on