Elasticsearch to OpenSearch data migration

  1. How to set the custom tenant name On-prem Analytics.

    See Persistent Mobile Foundation Analytics Server.

  2. How to set the custom tenant name for Cloud Analytics.

    See the Step#2b in the Migrating from MFP 8.0 (Cloud).

  3. How to verify that data was successfully exported.

    To confirm whether the data export was successful, you can re-run the following command.

    npm run migrate

    If the export has already completed, the status will display “Already exported” for each index, indicating that no further action is needed.

    Sample log output

     INFO (519026): --------------------------------------------------
 INFO (519026): Export
 INFO (519026): ElasticSearch host: http://localhost:9500/
 INFO (519026): Connection mode: insecure
 INFO (519026): Timestamp: 2025-09-22T15:37:19.770Z
 INFO (519026): --------------------------------------------------
 INFO (519026): Loaded existing metadata.json
 INFO (519026): [Index: worklight__serverlog__1726099200000] | Status: Skipping! Already exported | Document Count: 465
 INFO (519026): [Index: global_all_tenants] | Status: Skipping! Already exported | Document Count: 3
 INFO (519026): [Index: worklight__multitypes] | Status: Skipping! Already exported | Document Count: 1
 INFO (519026): [Index: worklight__device] | Status: Skipping! Already exported | Document Count: 1317789
 INFO (519026): [Index: worklight__appsessionsummarizedhourly__1726099200000] | Status: Skipping! Already exported | Document Count: 20
 INFO (519026): [Index: worklight__appsession__1726099200000] | Status: Skipping! Already exported | Document Count: 1317840
 INFO (519026): [Index: worklight__networktransactionsummarizedhourly__1726099200000] | Status: Skipping! Already exported | Document Count: 91
 INFO (519026): [Index: worklight__customdata__1726099200000] | Status: Skipping! Already exported | Document Count: 1316071
 INFO (519026): [Index: worklight__appfeedback__1726099200000] | Status: Skipping! Already exported | Document Count: 1677512
 INFO (519026): [Index: worklight__screenfeedback] | Status: Skipping! Already exported | Document Count: 1677551
 INFO (519026): [Index: worklight__applog__1726099200000] | Status: Skipping! Already exported | Document Count: 5270430
 INFO (519026): [Index: worklight__networktransaction__1726099200000] | Status: Skipping! Already exported | Document Count: 9944247
 INFO (519026): Total documents on http://localhost:9500/ : 22522020

    To ensure data consistency, the total document count shown in the logs should match the count retrieved from the Elasticsearch endpoint by using the following.

    protocol://<elastic-hostname>:<port>/_count

  4. Export or import failure due to connection error.

    If the export or import process fails with the following error message,

    Database URL not accessible: http://IP_ADDERESS:9500/. Reason: connect ECONNREFUSED IP_ADDRESS:9500

    Verify the following,

    • The Analytics server instance is up and running.
    • The Elasticsearch endpoint is accessible.
    • The configured URL for Elasticsearch is correct and accessible.

  5. Warning messages during export (Node.js Version 24).

    If you have Node.js Version 24, you can expect to see the following warning messages during the export process. You can safely ignore these messages.

    npm WARN Unknown user config "user". This will stop working in the next major version of npm.
npm WARN Unknown user config "unsafe-perm". This will stop working in the next major version of npm.
  6. No data visible on the Analytics console after migration.
    • Check whether a custom tenant was used during the import process.
    • Ensure the same tenant is correctly configured in the server.xml file.
    • See Step#1 or Step#2 for more information depending on your deployment type.

  7. Export of data is interrupted for any reason.

    Simply re-run the export command. The process will automatically resume from the point where it previously failed.

  8. Data migration or import to OpenSearch has failed or stalled with the following error.

    "Multiple attempts to send data to OpenSearch have failed. Terminating the process."

    This typically indicates that one or more nodes in the OpenSearch cluster do not have sufficient available storage to accommodate the incoming data.

    • Verify disk usage on all the OpenSearch cluster nodes using the following API:

      protocol://<open-search-ip>:<port>/_cat/nodes?v&h=name,disk.used_percent

      Ensure that disk usage on each node is below 90%.

    • Increase the disk space as needed to bring utilization below 90%, and ensure there is enough additional capacity to accommodate the data being migrated from Elasticsearch.

      Typically, OpenSearch requires approximately the same amount of storage as currently used on Elasticsearch.

  9. You need to reimport the data by cleaning up the data imported on the OpenSearch.

    • Clean up the tenant and specific data from OpenSearch by using the following command.

       curl -X DELETE http://10.44.235.91:9200/<tenant_name> -k -u '<open-search_user-name>:<open-search_password>'

    • Start the migration.

▲
Last modified on