Open topic with navigation
Perfecto MobileCloud offers automated testing on real mobile devices, connected via Wi-Fi or cellular carriers to the Internet, spread in different locations around the world. AlertSite UXM integrates with Perfecto MobileCloud, allowing you to schedule and monitor test runs on real devices. This way you can continuously monitor the key transactions on mobile devices, and get proactively notified of any availability or performance issues.
Perfecto Mobile provides automated testing capabilities and real devices for monitoring, and AlertSite provides scheduling, alerting and reporting. The following image shows the integration flow between the two systems.
Perfecto Mobile monitoring is an optional extra service to AlertSite Enterprise customers. Please contact your AlertSite Sales Manager for details. To view the contact information, select > AlertSite 1.0 from the top menu, then go to Support > Support Center.
Perfecto MobileCloud provides multiple copies of the same device model for redundancy. You use some devices for script development and debugging, and others for AlertSite monitoring. For example, if your Perfecto cloud includes three iPhone 5 devices, you use two of them for monitoring, and another one for scripting.
Monitoring devices must be paired, with each pair including two identical devices with the same model, OS version, network carrier and geographical location. A pair of devices is used as a logical monitoring device, and they appear as a single device in AlertSite. A monitoring script runs automatically on both paired devices, one after another (in random order based on device availability). This improves fault tolerance: if a script fails on one device but succeeds on another, the run is reported as successful. Only errors detected on both devices during the same run may indicate an actual problem.
To designate devices for monitoring, you need to pick pairs of twin devices, and specify a unique description for each device pair.
Find two devices with the same model and OS version to use as a logical monitoring device.
Enter the Description for the device. Use something that reflects the device’s business function or characteristics, such as model, carrier network, or location. For example, Amazon-Monitor-iPhone5, Android-ATT-Boston and so on.
Repeat the procedure to specify descriptions for all pairs of your monitoring devices.
You create and maintain the monitoring scripts in the Perfecto MobileCloud IDE. Monitoring scripts usually contain these elements:
If you are targeting multiple platforms and devices, follow Perfecto’s ScriptOnce methodology to create a device-agnostic script.
Save your monitoring scripts inside the Public or My (private) folder. Do not use group folders for monitoring scripts (group folders are named after the user group you are member of).
Perfecto monitoring scripts must include transactions to mark the actual things you want to monitor. A transaction is a group of commands with a transaction ID assigned. It succeeds if all of its commands succeed, and fails if at least one command fails.
Usually a transaction is a text checkpoint that measures your application’s response time and status (OK/error) in reaction to the preceding user action. This way you can monitor the application launch time, or the load time of specific screens in your application. To learn more about configuring Perfecto scripts to measure application responsiveness, see Measuring User Experience Timing in Perfecto documentation.
You can have up to 5 transactions in a Perfecto script.
For example, the following script includes two transactions, Site Load and What's new. Each transaction is a web page checkpoint that waits for a page to load, and verifies that is loaded successfully. The transaction results sent to AlertSite are the page load success/failure and the page load time.
To define a transaction:
Select the Transaction ID, from trans1 to trans5. The IDs define the order in which the script’s transactions will appear in AlertSite reports. It is best to assign the IDs sequentially from top to bottom of your script.
Repeat the procedure to define other transactions, up to 5 in total.
Note: You can have as many groups as you wish, but only those with the transaction ID assigned will be reported to AlertSite.
Commands in Perfecto scripts have error policies that specify how to handle failed commands, for example, whether to continue or stop the script. If later transactions in your script depend on earlier transactions, you may want to stop the script on error. In this case, use the Abort, Break or Continue policy (see below). If the transactions are independent, you can use Ignore to ignore the errors and continue the script.
To specify the error policy for a command:
Select the error policy, and click OK.
The error policies work as follows:
For more information, see Error Policies in Perfecto documentation.
Enter your Perfecto Hostname (for example, mylab.perfectomobile.com), User ID and Password.
Run Interval and Monitor Offset – The monitoring schedule: how often to run the script, and which minute of an hour to start the run. For example, with the 60-minute interval and 10-minute offset, the script runs every hour at 8:10, 9:10, 10:10 and so on. With the 20-minute interval and 5-minute offset, the script runs every hour at 5, 25 and 45 minutes past hour. For 120-minute monitors, offsets 1-60 mean even hours (2 PM, 4 PM, etc.), offsets 61-120 – odd hours (3 PM, 5 PM, etc.)
Important: Devices in Perfecto MobileCloud can run only one script at a time. Configure the schedule to spread the monitoring over an hour between the various devices you have. Make sure that no two monitors try to access the same device at the same time.
Click Start Monitoring Now!.
Note: This completes basic monitor configuration, but the monitor does not actually run yet.
Click Add/Edit Script. Here you can select the script to run, and the mobile devices for monitoring.
AlertSite will now run the selected script in your Perfecto MobileCloud according to the schedule, and collect the results.
AlertSite can send automated alerts when your mobile transactions fail, or the response time exceeds the configured limit. You need to enable alerts in the monitor settings and configure the alert recipients.
To enable alerts for a monitor:
Select which alert types the monitor will send:
Availability alerts – sent on an error condition.
Performance alerts – sent when the response time exceeds a threshold. You can set thresholds for the entire script or individual transactions. To configure the threshold:
Select the number of devices that should be in error or warning to trigger the alert.
To learn more about configuring performance alerts, see Performance Alerts.
Now, configure recipients for alerts:
To receive alerts about errors, switch to the Availability Alerts tab and enable these alerts.
Important: For email recipients, also select Attach server response to e-mail alerts. This will include Perfecto execution reports in email alerts.
Note: If you have many recipients in AlertSite, but you want to send Perfecto monitor alerts only to specific recipients, you need to create a recipient group for the monitor and its alert recipients. For instructions, see Recipient Groups.
You can see the status of your monitoring scripts on AlertSite’s dashboard. The response time displayed is the total UX time of all transactions in the script. If a monitor discovers an error, it turns red and is moved to the top of the dashboard.
To view detailed results, you can either click the charts on the tiles, or manually navigate the following dashboards:
See Common Status Codes for status code descriptions and troubleshooting tips.
You can receive Perfecto reports attached to email alerts when your monitoring script completes with an error. Email recipients should have the Attach server response to e-mail alerts option selected to use this feature (see above).
You can also access Perfecto reports from AlertSite. First, go to Monitors > Runs, select your monitor and choose a time period at the top. You will see all the test results for that period. You can filter the runs by status (OK/error) or sort them by execution time to find the needed results. Expand the runs to see the results of individual transactions.
In the expanded run, click View Perfecto Report to view the detailed execution results with device screenshots and videos.
Failed runs with status 5300 may not have an associated Perfecto report. In this case, click Request Log on the Monitor Runs dashboard to view the AlertSite error log. The first error in the log indicates the reason why the test failed. See Common Status Codes below for some typical errors.
AlertSite provides a set of reports that you can download, print or email.
To create a cumulative report:
Once the report is created you can view it, download, email, or schedule it as a recurring email report. You can also access all preciously created reports in Reports > Document Manager.
To view AlertSite report for a specific test run, do any of the following:
This opens the Detail report for that run, with the status and response time of all transactions in your Perfecto script.
Reports contain test results as a table. You can expand a test to see the transactions performed in that test, their status and response time. Click the camera icon next to a test to view the Perfecto report. If the Perfecto report is not available, click to view the AlertSite error log.
This table contains some common error codes used by Perfecto monitors in AlertSite UXM. For other status codes, please see AlertSite Status Codes.
|Status||Description||Possible Causes and Solutions|
|0||Site is OK now||The test completed successfully.|
|2||Test timed out||Status 2 is a failed time checkpoint. This usually means your application is busy and experiencing a slowdown. If the response time looks normal, review the timers in your script and increase the timeouts.|
|5||Validation failed||Status 5 is a failed checkpoint, meaning the expected text or image was not found in your application. You can see the checkpoint condition and results in the Perfecto execution report. If everything looks OK, review the checkpoint conditions in the script.|
|95||Unable to monitor||Unable to run the script in Perfecto MobileCloud.
|96||Invalid XML||Unable to fetch script results from Perfecto MobileCloud. Please report this error to SmartBear Support.|
|5090||Processing exception||Status 5090 is a legacy status code that was used prior to October 23, 2015. It means no device was available to run the script. You may need to adjust the monitoring schedule (run interval and monitor offset) to avoid overlaps with other monitors.|
|5300||Service execution error||There was a problem running your Perfecto script. To view the error message, go to Monitors > Runs, expand the failed run and click Request Log. Below are some common causes and solutions.
Handset unavailable, or Cannot open device <device ID> for user <email@example.com>, device is in use
No devices are available to run the script. To avoid the problem, you can:
Transaction results missing from Perfecto
The script does not have any transactions defined. Transactions mark the operations to monitor, so no transactions means no performance data sent to AlertSite. To learn how to configure transactions in your Perfecto script, see Define Transactions to Monitor.
<Command name> сommand failed. Reason: handset server: <error message>
A failed script command means a problem with the script or the monitored application. You can review Perfecto report to identify the failed command and troubleshoot the script in the Perfecto IDE to fix the problem.
Empty variable or parameter: <variable name>
Review the script in the Perfecto IDE and make sure all command parameters are specified.
If your script includes variables, make sure all variables except DUT (“device under test”) have default values and are marked as Runtime. The default values will be used when the script is run from AlertSite.
This happens if your Perfecto script does not contain any transactions – groups with the transaction ID assigned. AlertSite monitors transactions only rather than the entire script. To learn how to configure transactions, see Define Transactions to Monitor.
AlertSite can monitor only public and private scripts from your Perfecto repository, but not group scripts (scripts shared between members of a group). Please save your monitoring scripts to the Public or My (private) folder or subfolders.
This happens if the default primary location specified in your AlertSite account preferences is other than Fort Lauderdale or Atlanta. To avoid the issue, you need to temporarily change your default primary location:
Then, create your Perfecto monitor.
After that, you can revert to your previous location settings.