Migrate analytics data from MFP Cloud 8.0 to PMF Cloud 9.1 on Kubernetes

Prerequisites

Ensure to take snapshot of the analytics data into a directory.
Note the number of shards configured in the Elastic search on-prem environment.
Analytics data is migrated on a fresh PMF 9.1 installation.
PMF Elastic search component runs in the same namespace as other PMF components.

Procedure

Proceed as follows.

In your existing MFP 8.0 set up package, go to es/deploy directory and modify operator.yaml file and change es-operator image to do the upgrade.

Replace existing operator image version with 8.1.32 version, and ensure you have updated version info on below lines (3 places) in operator.yaml

Note: PMF 9.1 package contains es-operator:8.1.32 image, push it to your internal docker repository before applying changes in operator.yaml file.
```
 release: es-operator-8.1.32
 ...
 release: es-operator-8.1.32
 ...
 image: es-operator:8.1.32
```
Apply operator.yaml changes by using the following command.
```
 kubectl apply -f operator.yaml
```
Wait untll all the ibm-es-* pods come up.

Check if the ibm-es-es-configmap reflects the path.repo by using the following command.

 kubectl edit configmap ibm-es-es-configmap

If the changes do not reflect in the ibm-es-es-configmap, then manually edit configMap by using the following command.

 kubectl edit configmap ibm-es-es-configmap

Update path.repo: /es_backup\n in elasticsearch.yaml section.

Example

 apiVersion: v1 
 data: 
   elasticsearch.yml: "cluster:\n  name: ${CLUSTER_NAME}\nnode:\n  master: ${NODE_MASTER}\n 
     \ data: ${NODE_DATA}\n  name: ${NODE_NAME}\n  ingest: ${NODE_INGEST}\nindex:\n 
     \ number_of_replicas: 1\n  number_of_shards: 3\n  mapper.dynamic: true\npath:\n 
     \ data: /data/data_${staticname}\n  logs: /data/log_${staticname}\n  plugins: 
     /elasticsearch/plugins\n  work: /data/work_${staticname}      \nprocessors: ${PROCESSORS:1}\nbootstrap:\n 
     \ memory_lock: false\nhttp:\n  enabled: ${HTTP_ENABLE}\n  compression: true\n 
     \ cors:\n    enabled: true\n    allow-origin: \"*\"\ncloud:\n  k8s:\n    service: 
     ${DISCOVERY_SERVICE}\n    namespace: ${NAMESPACE}\ndiscovery:\n  type: io.fabric8.elasticsearch.discovery.k8s.K8sDiscoveryModule\n 
     \ zen:\n    ping.multicast.enabled: false\n    minimum_master_nodes: 1\nxpack.security.enabled: 
     false\nxpack.ml.enabled: false\ncompress.lzf.decoder: optimal\ndiscovery.zen.ping.multicast.enabled: 
     false\nbootstrap.mlockall: true\ncompress.lzf.decoder: safe\nscript.inline: true\npath.repo: 
     /es_backup\n"

Create PersistentVolume (PV) and PersistentVolumeClaim (PVC).

a. Create PersistentVolume as follows.

es-pv-volume.yaml

 apiVersion: v1 
 kind: PersistentVolume 
 metadata: 
   name: es-pv-volume 
   labels: 
     type: local 
 spec: 
   storageClassName: manual 
   capacity: 
     storage: 2Gi 
   accessModes: 
     - ReadWriteMany 
   hostPath: 
     path: "/es_backup" 

 kubectl apply -f es-pv-volume.yaml

b. Create PersistentVolumeClaim as follows.

es-pv-claim.yaml

   apiVersion: v1 
   kind: PersistentVolumeClaim 
   metadata: 
     name: es-pv-claim 
   spec: 
     storageClassName: manual 
     accessModes: 
       - ReadWriteMany 
     resources: 
       requests: 
         storage: 2Gi 

 kubectl apply -f es-pv-claim.yaml

Change the storage size as per your requirements.

 kubectl get pv

Output

 NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
 es-pv-volume     2Gi        RWX            Retain           Bound    mfpmig2/es-pv-claim            manual         <unset>                          12s

 kubectl get pvc

Output

 NAME                   STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
 es-pv-claim            Bound    es-pv-volume     2Gi        RWX            manual         <unset>                 9s

Make volumeMounts and volumes changes in the Elastic search deployments and statefulset.

See the deployments by using the following command.

 kubectl get deployments

Output

 NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
 es-operator              1/1     1            1           40h
 ibm-es-esclient          1/1     1            1           40h
 ibm-es-esmaster          1/1     1            1           40h
 ibm-mf-analytics         1/1     1            1           41h
 ibm-mf-analytics-recvr   1/1     1            1           41h
 ibm-mf-server            1/1     1            1           41h
 mf-operator              1/1     1            1           41 

See the statefulsets by using the following command.

 kubectl get statefulsets

Output

 NAME            READY   AGE
 ibm-es-esdata   1/1     40h

Modify ibm-es-esclient and ibm-es-esmaster deployments to update volumeMounts and volumes as follows.

a. Modify ibm-es-esclient deployment by using the following command.

 kubectl edit deployment ibm-es-esclient

Add below volumeMount.

 - mountPath: /es_backup 
       name: hp-volume 

Add below volume.

 - name: hp-volume 
     persistentVolumeClaim: 
       claimName: es-pv-claim  

Example

 volumeMounts: 
   - mountPath: /es_backup 
       name: hp-volume 
   - mountPath: /data 
       name: storage 
   - mountPath: /elasticsearch/config/elasticsearch.yml 
       name: config 
       subPath: elasticsearch.yml 
   - mountPath: /elasticsearch/config/log4j2.properties 
       name: config 
       subPath: log4j2.properties 

 volumes: 
   - name: hp-volume 
       persistentVolumeClaim: 
         claimName: es-pv-claim 
   - emptyDir: {} 
       name: storage 
   - configMap: 
       defaultMode: 420 
       name: ibm-es-es-configmap 
       name: config    

b. Modify ibm-es-esmaster deployment by using the following command.

 kubectl edit deployment ibm-es-esmaster

Add below volumeMount.

 - mountPath: /es_backup 
       name: hp-volume 

Add below volume.

 - name: hp-volume
      persistentVolumeClaim: 
         claimName: es-pv-claim 

Example

 volumeMounts: 
   - mountPath: /es_backup 
       name: hp-volume 
   - mountPath: /data 
       name: storage 
   - mountPath: /elasticsearch/config/elasticsearch.yml 
       name: config 
       subPath: elasticsearch.yml 
   - mountPath: /elasticsearch/config/log4j2.properties 
       name: config 
       subPath: log4j2.properties 

 volumes: 
   - name: hp-volume 
       persistentVolumeClaim: 
         claimName: es-pv-claim 
   - emptyDir: {} 
       name: storage 
   - configMap: 
         defaultMode: 420 
         name: ibm-es-es-configmap 
       name: config   

c. Modifyibm-es-esdatastatefulset by using the following command.

 kubectl edit statefulset ibm-es-esdata

Add below volumeMount.

 - mountPath: /es_backup 
     name: hp-volume 

Add below volume.

 - name: hp-volume 
     persistentVolumeClaim: 
         claimName: es-pv-claim  

Example

 volumeMounts: 
   - mountPath: /es_backup 
       name: hp-volume 
   - mountPath: /data 
       name: analytics-data 
   - mountPath: /elasticsearch/config/elasticsearch.yml 
       name: config 
       subPath: elasticsearch.yml 
   - mountPath: /elasticsearch/config/log4j2.properties 
       name: config 
       subPath: log4j2.properties 

 volumes: 
   - name: hp-volume 
       persistentVolumeClaim: 
         claimName: es-pv-claim 
   - name: analytics-data 
       persistentVolumeClaim: 
         claimName: mfanalyticsvolclaim2 
   - configMap: 
         defaultMode: 420 
         name: ibm-es-es-configmap 
       name: config  

Verify that /es_backup path is mounted in each of the elasticsearch pods.

 ibm-es-esclient-69f4974f8f-brczt          1/1     Running     0          42h
 ibm-es-esdata-0                           1/1     Running     0          42h
 ibm-es-esmaster-6548b4ddf9-l884h          1/1     Running     0          42h

a. Verify that /es_backup path in ibm-es-esclient pod by using the following command.

 kubectl exec -it ibm-es-esclient-7cb8768cf5-kv5lc bash

 bash-4.4$ cat /elasticsearch/config/elasticsearch.yml 
 cluster:
   name: ${CLUSTER_NAME}
 ...
 ...
 ...

 xpack.ml.enabled: false
 compress.lzf.decoder: optimal
 discovery.zen.ping.multicast.enabled: false
 bootstrap.mlockall: true
 compress.lzf.decoder: safe
 script.inline: true
 path.repo: /es_backup

 bash-4.4$ ls -ld /es_backup/

You should see path.repo: /es_backup in the elasticsearch.yaml` and also a mounted path.

b. Similiarly verify /es_backup path for ibm-es-esdata-* and ibm-es-esmaster-* pods.

Take snapshot using elasticsearch API

a. Get elasticsearch master pod IP address by using the following command.

 kubectl get pods -o wide

Output

 NAME                                      READY   STATUS      RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
 es-operator-658779fbb4-xbm76              1/1     Running     0          42h   192.0.2.139   master-node   <none>           <none>
 ibm-es-esclient-69f4974f8f-brczt          1/1     Running     0          42h   192.0.2.138   master-node   <none>           <none>
 ibm-es-esdata-0                           1/1     Running     0          42h   192.0.2.135   master-node   <none>           <none>
 ibm-es-esmaster-6548b4ddf9-l884h          1/1     Running     0          42h   192.0.2.143   master-node   <none>           <none>
 ibm-mf-analytics-58574df7d6-49bft         1/1     Running     0          42h   192.0.2.142   master-node   <none>           <none>
 ibm-mf-analytics-recvr-7bf6857955-xw4ct   1/1     Running     0          42h   192.0.2.191   master-node   <none>           <none>
 ibm-mf-defaultsecrets-job-2bcg9           0/2     Completed   0          42h   192.0.2.163   master-node   <none>           <none>
 ibm-mf-server-867d785477-bmzmp            1/1     Running     0          42h   192.0.2.184   master-node   <none>           <none>
 mf-operator-5c499fd5d8-84sfr              1/1     Running     0          42h   192.0.2.154   master-node   <none>           <none> 

Elastic Search master pod IP address in this example is 192.0.2.143.

b. Execute following API to verify that elastic search cluster is up and running.

 curl --location --request GET 'http://192.0.2.143:9200/_nodes/process?pretty' 

c. Execute following API to set backup directory.

 curl --location --request PUT 'http://192.0.2.143:9200/_snapshot/my_backup' --header 'Content-Type: application/json' --data '{"type": "fs","settings": {"location": "/es_backup"}}' 

 Success response: {"acknowledged":true}

d. Execute following API to take snapshot.

 curl --location --request PUT 'http://192.0.2.143:9200/_snapshot/my_backup/snapshot_1'              

 Success response: {"accepted":true}

e. Verify that the snapshot is created by using this command on the VM/Node where /es_backup volume is created.

 ls /es_backup/
                
 index  indices  metadata-snapshot_1  snapshot-snapshot_1

Post-upgrade task

Restore the MFP 8.0 Analytics snapshot data

Create PersistentVolume and PersistentVolumeClaim and point it to MFP snapshot (/es_backup directory located on the node/vm) also ensure that elasticsearch.yaml is reflecting the path.repo location which is /es_backup.

a. Complete the Steps#5a-5c of the Procedure section.

b. Complete the Steps#6-7 of the Procedure section.
Restore snapshot by using Elastic search API.

a. Complete the Steps#8a-8d of the Procedure section.

b. Execute following API to restore the snapshot.
```
 curl --location --request POST 'http://192.0.2.137:9200/_snapshot/my_backup/snapshot_1/_restore'

 Success response: {"accepted":true}   
```
c. If you get open index error then close the index and try to run the restore command again.

Close the index by using the following API.
```
 curl --location --request POST 'http://192.0.2.173:9200/global_all_tenants/_close'
```
Where,

‘global_all_tenants’ is the index to be closed. For each open index error, run execute this API with the given open index.

d. Run the restore command after closing all the open indexes.
```
 curl --location --request POST 'http://192.0.2.137:9200/_snapshot/my_backup/snapshot_1/_restore'
        
 Success Response: {"accepted":true}
```
e. Once the restore is successful, open all the closed index.

Open an index by using the following API.
```
 curl --location --request POST 'http://192.0.2.173:9200/global_all_tenants/_open'   
```
f. Restart the analytics pod by using the following command.

bash kubectl delete pod ibm-mf-analytics-58574df7d6-49bft92

▲

Last modified on