Scalability and Hardware Sizing for Mobile Foundation 9.0

This document contains information about scalability-related parameters of Persistent Mobile Foundation, and the scalability tests performed.
It serves as a guide to the provided hardware sizing calculator that is used to determine the required hardware for an PMF deployment.

Jump to:

• Architecture
• Scalability Tests
• Using the hardware sizing calculator
• Push Notification
• Analytics
• Direct Update
• Database usage and size
• Application Center
• Security Model

Architecture

The PMF topology is based on the following principles:

• PMF is installed in the organization LAN, connecting to various enterprise back-end systems.
• PMF can be clustered for high availability and scalability.
• The database is used for storing administration and configuration data, push notifications, registered clients data, and the security context of the clients. A single instance of the database is shared by the PMF runtime environment in all PMF cluster nodes. It is possible to cluster the database by using the tools and methods that are provided by the database vendor.
• PMF is installed behind a web authentication infrastructure, which acts as a reverse proxy and provides SSL.
• The database server and the PMF must be installed on separate host machines.

Topologies can vary depending on the corporate network architecture, disaster recovery, different back-end systems, and so on.

Scalability Tests

PMF conducted a series of scalability tests before releasing PMF. The following table contains a short summary of the tests performed.

Test Flow

The test, which is run repeatedly, involves the following actions:

1. A typical banking application is simulated. It logs in, makes multiple calls for account list and transaction lists, and then logs out.
2. The PMF application server uses an HTTP adapter to call a simulated back end.
3. The back end is called via HTTP and returns with a predefined response.

Hardware

The following hardware was used for PMF application server:

• x3250 M4, Xeon 4C E3-1220v2 69W 3.1GHz/1600MHz/8MB Note: With the specified number of cores, PMF performs thousands of transactions per second. If your throughput is low, you can use fewer cores than the specified number.
• 32 GB 1333 MHz Note: Not all RAM was used. Use the accompanying calculator to compute required RAM.
• NetExtreme II BCM5790 Gigabit Ethernet
• 2 X 500 3.5in SATA, SR C100

The following hardware was used during testing for the PMF relational database:

• 2 Intel Xeon 8C 2.3 GHz/20 MB
• 20 GB DDR3 RAM
• 6-7 500 GB 7.2 Krpm SATA disks
• 1 RAID 5 controller

PMF Client:

• Simulated by twenty virtual machines that run multiple client threads. Running Ubuntu 10.04 64-Bit 4 GB RAM.
• The client load is achieved with Apache jMeter, an open source server performance testing tool by Apache.

Test Results

Throughput was measured at 2,000 requests per second per server with an average CPU usage of 70%.

Using the hardware sizing calculator

About sizing

The hardware sizing calculator is based on a scenario that was tested in PMF Labs. This scenario is based on a real customer scenario.
However, PMF is a that is augmented with server-side developer code. Therefore, the sizing is largely influenced not only by client calls, but also by the efficiency of the server-side code. The server performance is also influenced by the security checks used by the application and the security configuration. Hence, you must consider the calculator as a baseline from which you can proceed to customer-specific scalability tests.

This document is accompanied by a Microsoft Office Excel spreadsheet. The purpose of this spreadsheet is to help organizations calculate the required hardware for a PMF deployment.

The calculator computes:

• The number of required servers. The server hardware baseline spec is defined in the preceding test.
• The required database size.
• The push notification throughput.
• The analytics server disk space and the number of servers.

Memory requirements

The PMF version 9.0 does not store data inside the client’s HTTP session. Memory is therefore not a performance bottleneck. In the scalability tests conducted by PMF, the server was run with a maximum JVM heap size set to 6GB.

Push Notifications

What are push notifications?

A push notification is an alert that indicates a change or an update that the PMF sends to users’ mobile devices. Push notifications are popular with all mobile operating systems. PMF has APIs that maintain the lists of mobile devices and their tag subscriptions, and send push notifications. PMF supports push notifications for the following vendors:

• Google's GCM for Android
• Apple APNS for iOS

PMF does not currently support push notification API for BlackBerry or Windows 8.
Possible architectures for push notifications Two architectures are possible for push notifications. They differ by the method that the enterprise back end uses to provide push notification data to the PMF:

• Push notifications are sent to the PMF by a call to a PMF procedure.
• The PMF pulls data by using a Persistent Mobile Foundation JMS adapter. The enterprise back end puts the push notifications in the JMS queue.

For more information about these alternate architectures, see the Persistent Mobile Foundation user documentation.

The primary difference between the two methods is that the first method is limited to a single server, which is pulling data from the JMS queue. With the second method, the procedures in the PMF are called through a load balancer. Therefore, the architecture can scale to more than one server.

Broadcast, tag-based, and unicast (1:1) notifications

PMF V9.0 supports broadcast, tag-based, and unicast (1:1) notifications.

• Broadcast notifications are sent to all the registered mobile devices of a specific application.
• Broadcast notifications are sent to all the registered mobile devices of a specific application.
• Unicast notifications, are sent to the devices of specific users. The "userID" (or) "deviceID" is specified in the introduced server-side REST API, which enables to send push notification messages to a specified target.

Testing assumptions

Push providers were simulated with a delay that is typical to these services:

• APNS: 0.05-0.1 milliseconds
• GCM: 20-25 milliseconds
• MPNS: 20-25 milliseconds

PMF can associate multiple devices per user. On average, each user is assumed to have 1.1 devices.
In Persistent Mobile Foundation, devices can subscribe to tags or topics of interest. Performance of tag-based notifications was tested with tags that had 600,000, 300,000, and 150,000 devices subscribed to them.

For broadcast or tag-based push notifications, a PMF can push a notification message to over 22,500 iOS devices and 50,000 Android devices per second. This is with all of the device types together.

For unicast or event-source based push notifications, an instance of PMF can push 1,750 notifications per second to APNS, and 350 notifications per second to GCM and MPNS, when it is under load. The load includes 2,400 regular transactions per second, as well as proportional device calls for subscribing to, and unsubscribing from, the push service.

The CPU and the memory are mostly unaffected for unicast and event-source based push notifications. However, for broadcast and tag-based push, if the number of mobile devices that would register for push notifications is large (for example, more than 100,000), then there could be bursts of 100% CPU usage and higher memory usage for a short duration during which the notification message would be sent to all the registered mobile devices. If your requirements are such that you do not want to delay your mobile users because of push notifications, you might want to have an instance of PMF dedicated to handle all the push notification needs, and configure your back-end server to send all the push notification requests directly to this dedicated server.

The database is unaffected, except for the storage of devices and tag subscriptions, as mentioned in section 8. Database usage and size.

Analytics

Persistent Mobile Foundation Operational Analytics

The Analytics is a WAR file that is bundled with PMF. The Analytics is based on file system storage rather than database storage. The hardware that is required to run the Analytics relies on several factors including:

• The frequency of connectivity to the server
• The verbosity set for logging
• The number of days that data is kept before being deleted

There are three distinct data types:

• Analytics collected from server network connections
• Client Logs
• Server Logs

The analytics data is always collected and forwarded to the PMF Analytics Server. PMF produces debug logs that are optionally forwarded to the PMF Analytics Server. Client logs are sent when the client app is optionally configured to collect and send logs.

An event of network connection to the analytics server is recorded in the following cases:

• A device successfully authenticates with the PMF instance.
• A device makes an adapter invocation.
• A push notification is sent to a device.
• A client-side API is called to create a custom activity.

Server network connections occur in the following situations:

• A device successfully authenticates with the PMF instance.
• A device makes an adapter invocation.
• A push notification is sent to a device.
• A client-side API is called to create a custom activity.

Server network connections are always collected and forwarded to the PMF Analytics Platform.

Server network connections occur in the following situations:

• A device successfully authenticates with the PMF instance.
• A device makes an adapter invocation.
• A push notification is sent to a device.
• A client-side API is called to create a custom activity.

Server network connections typically take up most of the disk space in the Analytics Platform. Device authentications and adapter invocations are the most common types of network connections that are recorded in the Analytics Platform.

When you estimate hardware requirements, it is important to take into account the number of devices that will be connecting to your server and the number of adapter invocations each device will be making.

Client logs are sent when the client app is optionally configured to collect and send logs.
By default, client logs are set to “FATAL”, which means that only crash logs are forwarded to the PMF instance. With this default setting, client logs use less space than analytics data that are created by server network connections.

Changing this level and adding custom client logs drastically affects the disk space used by client logs. With more custom logs and higher verbosity log levels, client logs can consume much more disk space (by several magnitudes) than server network connections. When you estimate hardware requirements, you must take into consideration the size client logs and the frequency of sending these logs.

Server logs, which are produced by PMF and are saved to disk, are also forwarded to the PMF by default. Server logs will have the least impact on the disk space sizing.

On Persistent Mobile Foundation Analytics, you can configure automatic purging of data.

For more information, see the Persistent Mobile Foundation user documentation

Different data types can be purged at different intervals. This means that if you wish to keep a year's worth of server network event data but only three weeks of log data, you can set these values separately.

Log data takes up more space than server network event data. Depending on the configured verbosity, PMF could be sending a very large amount of log data. If disk space is limited, you must consider purging data more frequently or adding nodes to your Analytics cluster.

You can add a new node to the analytics cluster with zero downtime. Therefore, you might want to start with a single node and add nodes as necessary. The total number of nodes that you can add to the cluster is equal to the number of shards that were created when the first node was installed.

For more information about shards and how to set up a cluster, see the Persistent Mobile Foundation user documentation

Direct Update

What is Direct Update?

When the administrator redeploys an app to the PMF instance without changing its version, PMF updates the web resources folder. Then, when an app connects to PMF, the server directly pushes the web resources (HTML, JavaScript, CSS and images) of the newly deployed app to the device. The push happens only when an app contains different web resources and the same native code.

Direct Update scalability challenges

When the administrator triggers direct update by redeploying an app to the PMF instance, it creates a surge of web resources download requests from the connected devices to the PMF. The IT organization has to be ready for it.

Factors that affect direct-update:

1. # of concurrent direct-update requests. The # of downloads and especially the frequency of usage of the application (e.g. once an hour vs. once a month) will determine the number of requests the PMF can handle when the new version is pushed.
2. Application size. There is an accumulative size limitation on the ability of the PMF to push data to clients.
3. Difference in content. In Mobile Foundation 9.0, the difference in content between the old and the new version is also important (see below).

Direct Update lab results

An instance of PMF can push a total of 250 MB/s. As an example, if the application size is 5 MB, PMF can serve 50 direct updates per second. This means that if the cluster contains four PMF nodes, the cluster can serve 200 direct updates per second. This is the limit for a single server rather than the network limitation.

In Persistent Mobile Foundation V9.0, direct update was improved to allow users to receive only the web resources that have changed between updates, instead of downloading the entire web resource package. This enhancement reduces download time, conserves bandwidth and improves overall user experience. When the delta size is between 10k to 300k the PMF TPS (Transactions per second) rate remains almost the same (6% decrease) with an additional 200 to 300 direct update requests per second.

The customer IT infrastructure has to be ready for both incremental direct update and full direct update requests. Subject to the exact customer scenario, most of the updates are likely to fall under the incremental scenario.
Small B2C deployments and most B2E deployments can use the above product features as is. As your application grows, you will need to use other means.
As a first approximation, you can get ready by adding servers and increasing outgoing network bandwidth. A more scalable solution would be to use a CDN.

Using a CDN

The best scalable solution for Direct Update, both full and incremental, is to use a content delivery network (CDN) in front of the PMF instance. When caching the direct update URL, the request is served from the CDN and does not get to PMF. The expiration policy must be specified in the CDN. The CDN's primary use is for full direct update requests but it may also for incremental direct update requests especially if the factors mentioned above influence your system.

For more information, see the "Serving direct update requests from CDN page" topic in the user documentation.about Direct Update, see the user documentation topic

Database Usage and Size

PMF database usage profile

The Persistent Mobile Foundation database stores administration and configuration data, registered clients data, the security context of the clients and push notifications. If the Application Center is used, its data is also stored in a database.

For a comprehensive description of the database tables, including the estimated number of rows per table, see the topic Internal runtime databases in the user documentation.

The main data items that require more database resources are the following:

• Client registration data: information is stored about every registered client instance (device and application pair) including information about the device, the application, user names associated with the client, last activity time and additional custom attributes.
• Client security context: the authentication state of the client. The size of the data depends on the number of the security checks used by the application, and the size of the state data stored by each security check.
• Push notification information: the list of devices that registered for push notifications, and the subscriptions from each device.

The hardware sizing calculator attempts to help you determine the expected size of the database, based on the usage profile.

Application Center

The Application Center is the PMF application store. The concept of the Application Center is similar to the concept of the Apple public App Store or the Google Play Store, except that it targets only private usage within a company. By using the Application Center, users from the same company or organization download applications to mobile phones or tablets from a single place that serves as a repository of mobile applications.

The Application Center and PMF can be installed on different systems. This installation choice ensures that the setup of the Application Center app store does not affect the performance of PMF applications on the server, when it is installed on a different hardware.

The performance of the Application Center is mainly tied to the quality and bandwidth of the network, and to the number of users. The distribution of large mobile applications must be done preferably through 3/4G or WiFi networks. On low networks, the Application Center client tries to solve problems that are related to timeout. If the Application Center cannot reach the server, it tries again several times. The download of an application might also be interrupted when the connection to the network is lost. In such case, the download can restart from where it stopped (for Android, iOS).

The Application Center requires one database schema to store the application catalog and the user information. The size of the database depends mainly on the average size of the applications that are stored in the catalog and the number of users that are connected to the system. This is reflected in the Hardware Calculator document.

Security model

The security model in Persistent Mobile Foundation is based on the OAuth 2.0 protocol, which has two distinct phases:

• Acquiring an access token
• Using the access token to access protected resources

In the first phase, an access token is generated by the PMF server. In the second, token validation is performed to allow access to protected resources running either on the PMF server or as external resources.

Token generation is more CPU consuming than token validation. Accordingly, the token expiration affects the overall performance of the PMF server, and should be configured taking into account both performance and security considerations.

For example, a very long token expiration will reduce the token generation overhead on the PMF server (thus improving performance), but might be less secured in the sense that client authorisation against the PMF server will be infrequent. On the other hand, a very short token expiration can be considered very secured (since it will require frequent interaction with the PMF server), but it will come at a performance cost.