Integrations > Perfecto Mobile Monitoring

Perfecto Mobile Monitoring

Perfecto MobileCloud offers automated testing on real mobile devices, connected via Wi-Fi or cellular carriers to the Internet, spread in different locations around the world. AlertSite UXM integrates with Perfecto MobileCloud, allowing you to schedule and monitor test runs on real devices. This way you can continuously monitor the key transactions on mobile devices, and get proactively notified of any availability or performance issues.

In This Guide

How AlertSite and Perfecto Mobile Work Together

Perfecto Mobile provides automated testing capabilities and real devices for monitoring, and AlertSite provides scheduling, alerting and reporting. The following image shows the integration flow between the two systems.

AlertSite and Perfecto integration

  1. User uploads app and creates tests in Perfecto MobileCloud.
  2. User schedules tests in AlertSite.
  3. Tests run on real devices around the world, in the Perfecto cloud.
  4. Test results are sent to AlertSite.
  5. AlertSite sends alerts if a test fails. User can also view test reports online in AlertSite.

Purchasing Mobile Monitoring Plan

Perfecto Mobile monitoring is an optional extra service to AlertSite Enterprise customers. Please contact your AlertSite Sales Manager for details. To view the contact information, select Settings > AlertSite 1.0 from the top menu, then go to Support > Support Center.

Configuring Perfecto

Configure Monitoring Devices

Perfecto MobileCloud provides multiple copies of the same device model for redundancy. You use some devices for script development and debugging, and others for AlertSite monitoring. For example, if your Perfecto cloud includes three iPhone 5 devices, you use two of them for monitoring, and another one for scripting.

Monitoring devices must be paired, with each pair including two identical devices with the same model, OS version, network carrier and geographical location. A pair of devices is used as a logical monitoring device, and they appear as a single device in AlertSite. A monitoring script runs automatically on both paired devices, one after another (in random order based on device availability). This improves fault tolerance: if a script fails on one device but succeeds on another, the run is reported as successful. Only errors detected on both devices during the same run may indicate an actual problem.

To designate devices for monitoring, you need to pick pairs of twin devices, and specify a unique description for each device pair.

  1. In Perfecto MobileCloud, select More > System Administration from the top menu.
  2. Switch to the Devices tab.
  3. Find two devices with the same model and OS version to use as a logical monitoring device.

    Perfecto device pairs

  4. Select one device and click Properties on the toolbar.
  5. Enter the Description for the device. Use something that reflects the device’s business function or characteristics, such as model, carrier network, or location. For example, Amazon-Monitor-iPhone5, Android-ATT-Boston and so on.

    Perfecto Mobile device description

  6. Click OK.
  7. Add the same description for another paired device.

Repeat the procedure to specify descriptions for all pairs of your monitoring devices.

Create Monitoring Scripts

You create and maintain the monitoring scripts in the Perfecto MobileCloud IDE. Monitoring scripts usually contain these elements:

If you are targeting multiple platforms and devices, follow Perfecto’s ScriptOnce methodology to create a device-agnostic script.

Save your monitoring scripts inside the Public or My (private) folder. Do not use group folders for monitoring scripts (group folders are named after the user group you are member of).

Define Transactions to Monitor

Perfecto monitoring scripts must include transactions to mark the actual things you want to monitor. A transaction is a group of commands with a transaction ID assigned. It succeeds if all of its commands succeed, and fails if at least one command fails.

Usually a transaction is a text checkpoint that measures your application’s response time and status (OK/error) in reaction to the preceding user action. This way you can monitor the application launch time, or the load time of specific screens in your application. To learn more about configuring Perfecto scripts to measure application responsiveness, see Measuring User Experience Timing in Perfecto documentation.

You can have up to 5 transactions in a Perfecto script.

For example, the following script includes two transactions, Site Load and What's new. Each transaction is a web page checkpoint that waits for a page to load, and verifies that is loaded successfully. The transaction results sent to AlertSite are the page load success/failure and the page load time.

Transactions in a Perfecto script

To define a transaction:

  1. Open your script in the Perfecto IDE.
  2. To create a new group, select a command, or several consecutive commands, and click the Group icon on the toolbar.
  3. Double-click a group to open its properties.
  4. Change the group name to something more descriptive. For example, Home page, Log in, Book a hotel, and so on. This name will appear in AlertSite reports.
  5. Select the Transaction ID, from trans1 to trans5. The IDs define the order in which the script’s transactions will appear in AlertSite reports. It is best to assign the IDs sequentially from top to bottom of your script.

    Perfecto transaction properties

  6. Click Close to save the changes.

Repeat the procedure to define other transactions, up to 5 in total.

Note: You can have as many groups as you wish, but only those with the transaction ID assigned will be reported to AlertSite.

Configure Error Policies

Commands in Perfecto scripts have error policies that specify how to handle failed commands, for example, whether to continue or stop the script. If later transactions in your script depend on earlier transactions, you may want to stop the script on error. In this case, use the Abort, Break or Continue policy (see below). If the transactions are independent, you can use Ignore to ignore the errors and continue the script.

To specify the error policy for a command:

  1. Open the script in the Perfecto IDE.
  2. Select a command in the script. You can select several commands using Shift-clicks.
  3. Click Edit on the toolbar.
  4. Switch to the On error tab.
  5. Select the error policy, and click OK.

    Perfecto error policy

The error policies work as follows:

For more information, see Error Policies in Perfecto documentation.

Using AlertSite

Create AlertSite Monitors From Perfecto Scripts

  1. On the AlertSite Dashboard, click the big +, then click + New Monitor.
  2. Select Mobile, and click Real Device Monitor.
  3. Specify your Perfecto account details and basic monitor information:
  4. Click Start Monitoring Now!.

    Note: This completes basic monitor configuration, but the monitor does not actually run yet.

  5. Click Add/Edit Script. Here you can select the script to run, and the mobile devices for monitoring.

    Selecting a Perfecto script and devices

  6. Click Add Script and select the script to run.
  7. Click Add a Device, select one or more devices to run this script, and click Continue.
  8. To enable monitoring, click the power icon power off to set it “on”: power on
  9. Click Save to save the monitor configuration.

AlertSite will now run the selected script in your Perfecto MobileCloud according to the schedule, and collect the results.

Set Up Alerts

AlertSite can send automated alerts when your mobile transactions fail, or the response time exceeds the configured limit. You need to enable alerts in the monitor settings and configure the alert recipients.

To enable alerts for a monitor:

  1. Open the monitor settings. To do this, click the monitor on the AlertSite Dashboard, then click pencil.
  2. Click Monitor Alerts.
  3. Select which alert types the monitor will send:

    Perfecto monitor alerts

    Availability alerts – sent on an error condition.

    Performance alerts – sent when the response time exceeds a threshold. You can set thresholds for the entire script or individual transactions. To configure the threshold:

    To learn more about configuring performance alerts, see Performance Alerts.

  4. Click Save in the top right to save the changes.

Now, configure recipients for alerts:

  1. From the top menu, select Alerts > Alert Recipients.
  2. To create a new recipient, click New Recipient and specify the contact information.
  3. Select the recipient in the list, and click Edit Recipient.
  4. To receive alerts about errors, switch to the Availability Alerts tab and enable these alerts.

    Important: For email recipients, also select Attach server response to e-mail alerts. This will include Perfecto execution reports in email alerts.

    Recipient configuration for availability alerts

  5. To receive response time alerts, switch to the Performance Alerts tab and enable these alerts.
  6. Configure other recipient settings as needed.
  7. Click Submit.

Note: If you have many recipients in AlertSite, but you want to send Perfecto monitor alerts only to specific recipients, you need to create a recipient group for the monitor and its alert recipients. For instructions, see Recipient Groups.

View Monitoring Status in AlertSite

You can see the status of your monitoring scripts on AlertSite’s dashboard. The response time displayed is the total UX time of all transactions in the script. If a monitor discovers an error, it turns red and is moved to the top of the dashboard.

Perfecto monitor on AlertSite dashboard

To view detailed results, you can either click the charts on the tiles, or manually navigate the following dashboards:

See Common Status Codes for status code descriptions and troubleshooting tips.

View Perfecto Execution Reports

You can receive Perfecto reports attached to email alerts when your monitoring script completes with an error. Email recipients should have the Attach server response to e-mail alerts option selected to use this feature (see above).

You can also access Perfecto reports from AlertSite. First, go to Monitors > Runs, select your monitor and choose a time period at the top. You will see all the test results for that period. You can filter the runs by status (OK/error) or sort them by execution time to find the needed results. Expand the runs to see the results of individual transactions.

Perfecto run results in AlertSite

In the expanded run, click View Perfecto Report to view the detailed execution results with device screenshots and videos.

Perfecto report

Failed runs with status 5300 may not have an associated Perfecto report. In this case, click Request Log on the Monitor Runs dashboard to view the AlertSite error log. The first error in the log indicates the reason why the test failed. See Common Status Codes below for some typical errors.

Error log for a Perfecto monitor

AlertSite Reports

AlertSite provides a set of reports that you can download, print or email.

 

To create a cumulative report:

Once the report is created you can view it, download, email, or schedule it as a recurring email report. You can also access all preciously created reports in Reports > Document Manager.

 

To view AlertSite report for a specific test run, do any of the following:

This opens the Detail report for that run, with the status and response time of all transactions in your Perfecto script.

 

Reports contain test results as a table. You can expand a test to see the transactions performed in that test, their status and response time. Click the camera icon next to a test to view the Perfecto report. If the Perfecto report is not available, click to view the AlertSite error log.

Detail report for a Perfecto monitor

Common Status Codes

This table contains some common error codes used by Perfecto monitors in AlertSite UXM. For other status codes, please see AlertSite UXM Status Codes.

Status Description Possible Causes and Solutions
0 Site is OK now The test completed successfully.
2 Test timed out Status 2 is a failed time checkpoint. This usually means your application is busy and experiencing a slowdown. If the response time looks normal, review the timers in your script and increase the timeouts.
5 Validation failed Status 5 is a failed checkpoint, meaning the expected text or image was not found in your application. You can see the checkpoint condition and results in the Perfecto execution report. If everything looks OK, review the checkpoint conditions in the script.
95 Unable to monitor Unable to run the script in Perfecto MobileCloud.
  • The Perfecto host name, user ID or password is incorrect.
  • The specified script is not found (for example, it has been moved or deleted).
  • The specified device is not found (for example, it is no longer available for monitoring).
  • A temporary out-of-resource state. This condition should clear up shortly, but if it persists, please contact SmartBear Support.
Note: Status 95 “Unable to monitor” appears only on the main Dashboard, in Monitor Summary and Monitor Availability. No new entries are added to the monitor run history, because this status usually indicates a temporary condition.
96 Invalid XML Unable to fetch script results from Perfecto MobileCloud. Please report this error to SmartBear Support.
5090 Processing exception Status 5090 is a legacy status code that was used prior to October 23, 2015. It means no device was available to run the script. You may need to adjust the monitoring schedule (run interval and monitor offset) to avoid overlaps with other monitors.
5300 Service execution error There was a problem running your Perfecto script. To view the error message, go to Monitors > Runs, expand the failed run and click Request Log. Below are some common causes and solutions.

 

Handset unavailable, or Cannot open device <device ID> for user <user@domain.com>, device is in use

No devices are available to run the script. To avoid the problem, you can:

  • Change the monitor run interval and offset to avoid overlaps with your other Perfecto monitors.
  • Make sure the monitoring devices are not used for scripting, debugging or any other purposes other than production monitoring.
  • Allocate more devices for monitoring.

 

Transaction results missing from Perfecto

The script does not have any transactions defined. Transactions mark the operations to monitor, so no transactions means no performance data sent to AlertSite. To learn how to configure transactions in your Perfecto script, see Define Transactions to Monitor.

 

<Command name> сommand failed. Reason: handset server: <error message>

A failed script command means a problem with the script or the monitored application. You can review Perfecto report to identify the failed command and troubleshoot the script in the Perfecto IDE to fix the problem.

 

Empty variable or parameter: <variable name>

Review the script in the Perfecto IDE and make sure all command parameters are specified.

If your script includes variables, make sure all variables except DUT (“device under test”) have default values and are marked as Runtime. The default values will be used when the script is run from AlertSite.

Troubleshooting

Response time is 0 seconds in AlertSite

This happens if your Perfecto script does not contain any transactions – groups with the transaction ID assigned. AlertSite monitors transactions only rather than the entire script. To learn how to configure transactions, see Define Transactions to Monitor.

Some Perfecto scripts do not show up in AlertSite

AlertSite can monitor only public and private scripts from your Perfecto repository, but not group scripts (scripts shared between members of a group). Please save your monitoring scripts to the Public or My (private) folder or subfolders.

Error when creating a Perfecto monitor: "Your primary home location does not support Mobile Monitoring"

This happens if the default primary location specified in your AlertSite account preferences is other than Fort Lauderdale or Atlanta. To avoid the issue, you need to temporarily change your default primary location:

Then, create your Perfecto monitor.

After that, you can revert to your previous location settings.

What’s Next

© 2017 SmartBear Software. All rights reserved.      Terms of Use · Privacy Policy