Welcome!

Agile Computing Authors: Zakia Bouachraoui, Pat Romanski, William Schmarzo, Yeshim Deniz, Liz McMillan

Related Topics: @DevOpsSummit, Mobile IoT, Agile Computing

@DevOpsSummit: Blog Post

The Ultimate Mobile Performance Testing Checklist | @DevOpsSummit #DevOps

Align mobile performance testing objectives with corporate and business requirements

To conduct complete mobile performance testing, a process that will allow you to consistently deliver an exceptional application experience to mobile users, make sure you hit every point on our ultimate mobile performance testing checklist.

By now, hopefully you're well aware of the importance of mobile performance testing. It's essential for your organization to test for mobile users whether or not you have a dedicated mobile application. In case you aren't aware of its necessity, here are a couple statistics to cement the idea:

  • 29% of smartphone users will immediately switch to another site or app if it doesn't satisfy their needs (e.g., they can't find information or it's too slow)
  • Of the aforementioned users that switch, 70% do so because of lagging load times (Source)

To conduct complete mobile performance testing, a process that will allow you to consistently deliver an exceptional application experience to mobile users, make sure you hit every point on our ultimate mobile performance testing checklist:

✔ Align mobile performance testing objectives with corporate and business requirements

If you ignore the needs of the business, you risk testing the wrong aspects of your application. Identify which aspects of the app experience drive meaningful metrics and then determine the user experience requirements relative to these metrics.

✔ Identify KPIs for application performance

Your key performance indicators (KPIs) will depend on your business objectives. Clarify your goals early on and consider these important KPIs when initiating mobile performance testing: average requests per second, average request response time, maximum response time, average throughput, and error rate.

✔ Work directly with the codebase

Building a mobile performance testing suite without any knowledge of how the code works may cause you to overlook significant problems and bottlenecks. Work with your developers to identify parts of the application that should be tested based on the way the application is built. Also, design your tests in conjunction with Dev to gain a comprehensive understanding of the application areas that need to be tested.

✔ Prioritize test cases

Executing all of your tests across every combination of variants is both time and cost intensive. Initially, you'll want to use your KPIs and behavioral analytics to determine the 10-20 combinations you wish to run in the first round of testing. By focusing on these selected combinations, you'll have a better idea of the types of scenarios your application can and cannot handle. Obviously, you should fix any bottleneck or bug before moving on to the next mobile performance testing scenario.

✔ Load testing

Mobile traffic spikes can affect everyone on your application. Consult your analytics platform to ensure your load testsaccurately reflect realistic proportions of mobile users and web users-this is especially important if your mobile and web sites share a backend.

✔ Device simulation

To make the testing environment and user experience as realistic as possible, utilize emulators. These allow you to test various phones, tablets, etc. without having to physically own and/or manage those devices. While emulating devices, be sure to take into account parallel connections as mobile networks, devices and applications all enforce varying policies in terms of how many are allowed at a given time. You'll also need to customize the user agent header in your scripts to ensure the web or mobile server is sending the correct content to all of your users' different devices.

✔ Network virtualization

Mobile devices will access your servers over 3rd-party carrier networks that vary greatly in speed, latency, bandwidth, etc. By employing network virtualization, you can mimic the network characteristics that mobile devices actually experience. Prioritize the characteristics you see most often, but don't ignore how your application performs for users on the extreme ends of the spectrum. You'll want to make sure that every mobile user's experience is the best it can be, given their specific network limitations.

✔ Geographic location

As you well know, your mobile users won't be accessing your application from the same place. Depending on their location, mobile users will experience different levels of performance, and as a tester, this is something you'll need to account for. Create a geographic profile of your users based on historical data and run geographically-dispersed load tests from the cloud.

✔ Real-device recording

It can be easy to overlook the importance of recording from real devices when conducting mobile performance testing, especially when you're testing a web app that can be recorded from a browser. The problem comes when any call from the mobile interface is different from a call from the web app, or if there is not a corresponding web app. Recording your test scenarios directly from a mobile device will give you the most accurate test results in either situation.

✔ End-user experience testing on real devices

Quickly identify the root cause of mobile performance issues by engaging in end-user experience testing on real devices. Load testing paired with functional testing on a real device will allow you to measure rendering times on the front end of your application that you can then correlate with backend times.

✔ Testing in production

No matter how closely your test environment resembles your production environment, it will never be 100% the same. Sometimes bugs will only reveal themselves in the real world, under actual operation. As such, you'll need to test for mobile users in your real-world production environment. While testing in production poses some risk, it can be riskier not to invest in the practice as you'll end up blind to the levels of performance actually experienced by your mobile users.

Don't Skimp on Mobile Performance Testing
People are becoming more dependent on smartphones and other connected mobile devices for quick access to applications and websites. If you aren't testing for mobile users, you will fall short in terms of application performance.

If you complete this mobile performance testing checklist, however, you will not only appease managers and stakeholders, but you will also be able to successfully deliver exceptional application performance to all of your mobile users, regardless of device, network conditions and geographic location.

To learn more about mobile performance testing strategies and tools that will delight your users, check out our recent webinar on the subject here.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

IoT & Smart Cities Stories
If a machine can invent, does this mean the end of the patent system as we know it? The patent system, both in the US and Europe, allows companies to protect their inventions and helps foster innovation. However, Artificial Intelligence (AI) could be set to disrupt the patent system as we know it. This talk will examine how AI may change the patent landscape in the years to come. Furthermore, ways in which companies can best protect their AI related inventions will be examined from both a US and...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
Chris Matthieu is the President & CEO of Computes, inc. He brings 30 years of experience in development and launches of disruptive technologies to create new market opportunities as well as enhance enterprise product portfolios with emerging technologies. His most recent venture was Octoblu, a cross-protocol Internet of Things (IoT) mesh network platform, acquired by Citrix. Prior to co-founding Octoblu, Chris was founder of Nodester, an open-source Node.JS PaaS which was acquired by AppFog and ...
The deluge of IoT sensor data collected from connected devices and the powerful AI required to make that data actionable are giving rise to a hybrid ecosystem in which cloud, on-prem and edge processes become interweaved. Attendees will learn how emerging composable infrastructure solutions deliver the adaptive architecture needed to manage this new data reality. Machine learning algorithms can better anticipate data storms and automate resources to support surges, including fully scalable GPU-c...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...