Best Practices for setting up the Persistent Mobile Foundation Analytics production Cluster

This topic describes the list of best practices that include the DOs and the DON’Ts to be followed while setting up an Analytics Server in production.

Configuration settings

Data purging has to be applied to the production environment, mandatorily, in order to not persist the entire document set from the beginning. By setting the appropriate TTL values for various event documents, the search scope for Elasticsearch queries can be reduced considerably. Following are the TTL values that are to be set for the Persistent Mobile Foundation Analytics PMF 9.x Server:

TTL properties for Analytics Event/Documents

  • TTL_PushNotification
  • TTL_PushSubscriptionSummarizedHourly
  • TTL_ServerLog
  • TTL_AppLog
  • TTL_NetworkTransaction
  • TTL_AppSession
  • TTL_AppSessionSummarizedHourly
  • TTL_NetworkTransactionSummarizedHourly
  • TTL_CustomData
  • TTL_AppPushAction
  • TTL_AppPushActionSummarizedHourly
  • TTL_PushSubscription

Example usage:

<jndiEntry jndiName="analytics/TTL_AppLog" value= '"20d"' />

TTL value are expected to be String literals and should be passed within single quotes.
If the TTl entry for a particular document-type is not defined in JNDI property, then the TTL value is set to 90 by default.

Topology

Multi-Node Analytics Cluster

  • It is important to have a load balancer in front of the nodes to ensure that the analytics layer is offered an even load across the nodes.
  • In a two-node analytics cluster when a load balancer is not used, it is good to configure or use the Analytics console of the node that is not used for accepting the data from the Persistent Mobile Foundation Server.

Example

Consider that there are two nodes for the analytics server. In such a case, the recommendation for the Persistent Mobile Foundation server configuration for analytics is as below:

Recommended

  • mfp/mfp.analytics.url -> http://node1:9080/analytics-service/rest

  • mfp/mfp.analytics.console.url -> http://node2:9080/analytic/console

NOT recommended

  • mfp/mfp.analytics.url -> http://node1:9080/analytics-service/rest

  • mfp/mfp.analytics.console.url -> http://node1:9080/analytic/console

This allows the user to reduce the load on the nodes when the user views the analytics console.

Performance tuning - Operating System

  • Increase the allowed number of open file descriptors to 32k or 64k.
  • Increase the virtual memory map counts.

Note: Check the corresponding documentation for the Operating System.

Performance tuning - Analytics

  • Both Java Xms and Xmx has to be set (Min and Max) as same.
  • Maximum allowed Heapsize Per JVM <= RAM Size/2.
  • Number of Primary Shards = Number of Nodes of the Analytics Cluster.
  • Number of Replica per shard >= 2.

Note: If there is only one node then there is no need of a replica.

Performance tuning - Elasticsearch

Elasticsearch tuning can be performed in a separate YAML file (for example, it can named as elasticsearchconfig.yml), the path to this file can be configured in the analytics server configuration (using the JNDI properties).

Property Name: analytics/settingspath

Value: <path_to_the_ES_config_yml\>

Apply the Elasticsearch tuning parameters by adding the values to a .yml file and accessing it using the JNDI entry.

Elasticsearch tuning parameters are to be considered for a basic tuning of the environment and can be tuned further based on the infrastructure resources:

  1. Set a value for indices.fielddata.cache.size

    Example

    indices.fielddata.cache.size:  35%

    Note: Use analytics/indices.fielddata.cache.size with caution. Do not increase it to a high value, as increasing this value can cause Out of Memory. The underlying technology used by the analytics platform loads several field values into memory to provide faster access to those documents. This is known as the field cache. By default, the amount of data loaded into memory by the field cache is unbounded. If the field cache becomes too large, it can cause an out of memory exception and crash the analytics platform.

  2. Set a value for indices.fielddata.breaker.limit.

    Set indices.fielddata.breaker.limit to a value greater than indices.fielddata.cache.size.

    So, if the cache size is 35%, set breaker limit to a value greater than the cache size.

  3. Set a value for indices.fielddata.cache.expire.

    This value sets the expiry time of the Elasticsearch cache and prevents the cache from growing unbound and filling up the heap.

    indices.fielddata.cache.expire - A time-based setting that expires field data after a certain time of inactivity. Defaults to -1. For example, can be set to 5 m for a 5 minute expiry.

  4. The default setting for Analytics is to NOT purge any data.

    Configure the TTL appropriately to ensure that data is purged. Otherwise, the data store can grow in an unlimited manner.

DOs and DON’Ts

  • Avoid clearing the analyticsData directory when the analytics nodes are running.
  • In a multi node cluster, avoid using the same node for pushing the events into analytics cluster and access the console. Best practice would be to use a Load balancer in front of the analytics cluster.
  • Avoid using any other Application Server clustering methodologies for Analytics Cluster. The underlying Elasticsearch creates cluster on its own using its node discovery mechanisms.
  • Never increase the Analytics min/max heap size to value greater than half of the RAM Size on the node. For instance, if you have a node with RAM size of 16 GB then the max allowed heap size is 8 GB for analytics.
  • For analytics cluster name (analytics/clustername JNDI property), use a unique cluster name. Avoid using the default name worklight.

SDK issues

  • Cordova Applications must initialize at native platform for Lifecycle events to enable AppSession capture

    In the Persistent Mobile Foundation PMF 9.x, the app sessions are incremented/recorded when the app goes from background to foreground.

    Capturing AppSessions is enabled by adding listeners for lifecycle events. The native SDKs provide appropriate APIs for adding these listeners. However, in the case of Cordova there is no JavaScript API for adding these lifecycle event listeners. Instead, the listeners have to be added using native platform APIs even for Cordova applications.

    Excerpt from the Using PMF Analytics API in Client Applications - Client lifecycle events topic,

    After the Analytics SDK is configured, app sessions start to be recorded on the users device. A session in the Persistent Mobile Foundation Analytics is recorded when the app is moved from the foreground to the background, which creates a session on the Analytics Console. As soon as the device is set up to record sessions and you send your data, you can see the Analytics Console populated with data, as shown below.

    For example, for a Cordova app on iOS Platform (iOS) it is mandatory to add the following under AppDelegate.m:

     [[WLAnalytics sharedInstance] addDeviceEventListener:LIFECYCLE];
 [[WLAnalytics sharedInstance] send];

  • View the custom Data on Analytics console

    To quickly locate the custom data sent to Analytics server using the Analytics Client SDK APIs, following step can be performed.

    Go to Analytics Console > Dashboard > Custom Charts > Create a Custom Chart.

    Create a custom chart

    For more information refer to the documentation Creating Custom Charts.

Troubleshooting steps

  1. Customer Environment Version:
    Gather the details of the complete software stack, including OS, JDK/JRE, AppServer, the Persistent Mobile Foundation version, and the Persistent Mobile Foundation build version.
  2. Compare the environment details with the Persistent Mobile Foundation Analytics Software Compatibility Matrix/requirements.
  3. Gather the Analytics Topology & Hardware specifications used.
  4. Check if there was any performance tuning performed (in case of performance issues).
  5. Gather the Persistent Mobile Foundation Server’s server.xml (Liberty) to verify the Analytics integration configuration.
  6. Gather the screen shot of the Analytics administration console.
  7. Gather the Analytics server.xml (Liberty) to verify the Analytics integration configuration.
  8. Collect the output of the following REST APIs (listed under the section – Important Commands & APIs for troubleshooting analytics issues).

Troubleshooting utilities

Following are the open source tools which can be effectively used to visualize and administer the elasticsearch indices, data/shard allocation etc.

Troubleshooting commands and APIs

Use cURL to run the following REST APIs to capture the response for identifying various information about the cluster/shard/indices:

http://<es_node>:9500/_cluster/health
http://<es_node>:9500/_cluster/stats
http://<es_node>:9500/_cat/shards
http://<es_node>:9500/_node/status
http://<es_node>:9500/_cat/indices

Security recommendations

Secure cookies

Missing secure flag on cookies could allow attacker to read the cookies and make unintended use of it, this can be avoided by setting listed flags in WebSphere Liberty Server configuration.

  • To have secure flag on session cookies set the configuration as follows.

     <httpSession cookieSecure="true" cookieHttpOnly="true" />

  • To have secure flag on LtpaToken cookies, make sure you access the application on https protocol. Following property should be added in server configuration only if you are accessing application on https. If it is on http, you should disable the following configuration.

     <webAppSecurity logoutOnHttpSessionExpire="true" ssoRequiresSSL="true" />

References

▲
Last modified on