|By Sellers Smith||
|November 23, 2013 05:00 PM EST||
Tests are an investment in the quality of any given system. There's always a cost to build, run, and maintain each test in terms of time and resources. There's also a great deal of value to be extracted from running the right test at the right time. It's important to remember that for everything you do test, there's something else you're not testing as a result.
Understanding that some tests are more important than others is vital to creating a useful and fluid test plan capable of catering to modern software development techniques. Traditional waterfall development - where everything is delivered for a specific release date in a finished package - has been succeeded by continuous feature roll outs into live systems. This necessitates a different approach from QA departments.
How Do You Build Good Tests?
You can't design or execute the right tests without understanding the intended purpose of the system. Testers need to have an insight into the end user's expectations. Communication between the product people at the business end, the engineers working on the code, and the test department enables you to score tests in terms of their importance and work out where each test cycle should be focusing.
We can break it down into three simple queries: why, what, and how.
"Why" is a higher level overview that really ties into the business side. It's the big-picture thinking that reveals why you're building the software in the first place. What audience need is your product fulfilling? For example, we need to build an e-commerce website to sell our product to the general public.
"What" is really focused on individual features or functions of a system. Using a shopping cart analogy for an e-commerce website, you might say that users must be able to add and remove items from their shopping cart, or that they shouldn't be able to add something that's out of stock.
"How" relates to the practical application of your testing. How exactly is the software going to be tested? How is the success and failure measured?
Good tests are always going to cover our trio, but it can be a useful exercise to break things down.
If you get too caught up in the "what" and the "how," it's possible to miss the "why" completely and it's the most important element because it dictates that some tests are more important than others. The business case for developing your software in the first place has to remain front and center throughout the project. If you begin to lose sight of what the end user needs, then you could be focusing your efforts in the wrong places. Delivering value to your customers is critical. Good testers and good tests always retain and use an awareness of what the intended audience wants and expects.
One technique we can employ is risk-based analysis of tasks. With risk-based analysis, we can arrive at a numerical value for each test, which gives you a sense of its importance. We can assign a score of between 1 and 9 to each test. At the top end, a score of 9 would be a critical test, and at the other end of the spectrum a score of 1 might indicate a test that only needs to be used sparingly. The value is determined by multiplying two factors:
- Impact to the user: What are they trying to accomplish and what would the impact be if they couldn't? How critical is this action?
- Probability of failure: How likely is it that this code will fail? This is heavily influenced by how new it is and how much testing it has already undergone.
If we return to our e-commerce website analogy then we could take the action of being able to buy goods, clearly that's essential, so it would be assigned a 3. However, the functionality for adding goods to the basket and checking out has been there since day one so it has already been tested quite extensively; however, some new features have been added that could impact the code, so that might result in a score of 2. Multiply the two together and you've got a 6. This figure will change over time, because probability of failure will go up if this part of the system is significantly altered, and it will go down over time if it isn't. There's also a discretionary factor that might lead you to bump that 6 up to a 7 if you feel it's merited.
Testers come up with specific scenarios of how an end user might interact with the software and what their expected outcome would be. A typical test might consist of many steps detailed in a script, but this approach can cause problems. What if a new tester comes in to run the test? Is the intent of the test clear to them? What if the implementation of the feature changes? Perhaps the steps no longer result in the expected outcome and the test fails, but that doesn't necessarily mean that the software is not working as intended.
The steps and scripts are delving into the "how," but if the "what" is distinct from the "how" then there's less chance of erroneous failure. Allow the tester some room for an exploratory approach and you're likely to get better results. If something can be tightly scripted and you expect it to be a part of your regression testing, then there's an argument for looking at automating it.
Adopting a technique like Specification by Example or Behavior Driven Design, you're going to lay each test out in this format:
- Given certain preconditions
- When one or more actions happen
- Then you should get this outcome
Regardless of the specifics of the user interface, or the stops along the way between A and Z, the "Given, When, Then" format covers the essential core of the scenario and ensures that the feature does what it's intended to do, without necessarily spelling out exactly how it should do it. It can be used to generate tables of scenarios that describe the actions, variables, and outcomes to test.
Getting down to the nuts and bolts of how testers will create, document, and run tests, we come to the "how." Since projects are more fluid now and requirements or priorities can change on a daily basis, there needs to be some flexibility in the "how" and a willingness to continually reassess the worth of individual tests. Finding the right balance between automated tests, manually scripted tests, and exploratory testing is an ongoing challenge that's driven by the "what" and the "why."
Traditionally, manual tests have been fully documented as a sequence of steps with an expected result for each step, but this is time-consuming and difficult to maintain. It's possible to borrow from automation by plugging in higher-level actions, or keywords, that refer to a detailed script or a common business action that's understood by the tester. There's also been a move toward exploratory testing, where the intent of the test is defined but the steps are dynamic. Finally, there's a place for disposable testing where you might use a recording tool to quickly document each step in a manual test as you work through it. These tests will need to be redone if anything changes, but as it's a relatively quick process and you're actually testing while you create the test, that's not necessarily a problem.
Each individual test should be viewed as an investment. You have to decide whether to run it, whether it needs to be maintained, or if it's time to drop it. You need to continually assess the value and the cost of each test so that you get maximum value from each test cycle. Never lose sight of the business requirements. When a test fails, ask yourself if it's a problem with the system or a problem with the test, and make sure that you always listen to the business people, the software engineers, and most importantly the customer.
The 4th International Internet of @ThingsExpo, co-located with the 17th International Cloud Expo - to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA - announces that its Call for Papers is open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
May. 26, 2015 09:00 AM EDT Reads: 1,532
The Domain Name Service (DNS) is one of the most important components in networking infrastructure, enabling users and services to access applications by translating URLs (names) into IP addresses (numbers). Because every icon and URL and all embedded content on a website requires a DNS lookup loading complex sites necessitates hundreds of DNS queries. In addition, as more internet-enabled ‘Things' get connected, people will rely on DNS to name and find their fridges, toasters and toilets. According to a recent IDG Research Services Survey this rate of traffic will only grow. What's driving t...
May. 26, 2015 09:00 AM EDT Reads: 5,062
Today’s enterprise is being driven by disruptive competitive and human capital requirements to provide enterprise application access through not only desktops, but also mobile devices. To retrofit existing programs across all these devices using traditional programming methods is very costly and time consuming – often prohibitively so. In his session at @ThingsExpo, Jesse Shiah, CEO, President, and Co-Founder of AgilePoint Inc., discussed how you can create applications that run on all mobile devices as well as laptops and desktops using a visual drag-and-drop application – and eForms-buildi...
May. 26, 2015 08:00 AM EDT Reads: 5,367
The Internet of Things promises to transform businesses (and lives), but navigating the business and technical path to success can be difficult to understand. In his session at @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, demonstrated how to approach creating broadly successful connected customer solutions using real world business transformation studies including New England BioLabs and more.
May. 26, 2015 08:00 AM EDT Reads: 5,853
The recent trends like cloud computing, social, mobile and Internet of Things are forcing enterprises to modernize in order to compete in the competitive globalized markets. However, enterprises are approaching newer technologies with a more silo-ed way, gaining only sub optimal benefits. The Modern Enterprise model is presented as a newer way to think of enterprise IT, which takes a more holistic approach to embracing modern technologies.
May. 25, 2015 11:00 PM EDT Reads: 5,935
The true value of the Internet of Things (IoT) lies not just in the data, but through the services that protect the data, perform the analysis and present findings in a usable way. With many IoT elements rooted in traditional IT components, Big Data and IoT isn’t just a play for enterprise. In fact, the IoT presents SMBs with the prospect of launching entirely new activities and exploring innovative areas. CompTIA research identifies several areas where IoT is expected to have the greatest impact.
May. 25, 2015 09:00 PM EDT Reads: 4,776
Every day we read jaw-dropping stats on the explosion of data. We allocate significant resources to harness and better understand it. We build businesses around it. But we’ve only just begun. For big payoffs in Big Data, CIOs are turning to cognitive computing. Cognitive computing’s ability to securely extract insights, understand natural language, and get smarter each time it’s used is the next, logical step for Big Data.
May. 25, 2015 08:00 PM EDT Reads: 1,877
There's no doubt that the Internet of Things is driving the next wave of innovation. Google has spent billions over the past few months vacuuming up companies that specialize in smart appliances and machine learning. Already, Philips light bulbs, Audi automobiles, and Samsung washers and dryers can communicate with and be controlled from mobile devices. To take advantage of the opportunities the Internet of Things brings to your business, you'll want to start preparing now.
May. 25, 2015 07:00 PM EDT Reads: 5,722
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal an...
May. 25, 2015 04:00 PM EDT Reads: 2,441
P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at @ThingsExpo, Robin Raymond, Chief Architect at Hookflash, will walk through the shifting landscape of traditional telephone and voice services ...
May. 25, 2015 03:00 PM EDT Reads: 4,222
The 17th International Cloud Expo has announced that its Call for Papers is open. 17th International Cloud Expo, to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, APM, APIs, Microservices, Security, Big Data, Internet of Things, DevOps and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
May. 25, 2015 02:00 PM EDT Reads: 4,149
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, described how to revolutioniz...
May. 25, 2015 02:00 PM EDT Reads: 4,752
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo, June 9-11, 2015, at the Javits Center in New York City. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be
May. 25, 2015 01:15 PM EDT Reads: 1,832
The security devil is always in the details of the attack: the ones you've endured, the ones you prepare yourself to fend off, and the ones that, you fear, will catch you completely unaware and defenseless. The Internet of Things (IoT) is nothing if not an endless proliferation of details. It's the vision of a world in which continuous Internet connectivity and addressability is embedded into a growing range of human artifacts, into the natural world, and even into our smartphones, appliances, and physical persons. In the IoT vision, every new "thing" - sensor, actuator, data source, data con...
May. 25, 2015 01:00 PM EDT Reads: 6,161
Container frameworks, such as Docker, provide a variety of benefits, including density of deployment across infrastructure, convenience for application developers to push updates with low operational hand-holding, and a fairly well-defined deployment workflow that can be orchestrated. Container frameworks also enable a DevOps approach to application development by cleanly separating concerns between operations and development teams. But running multi-container, multi-server apps with containers is very hard. You have to learn five new and different technologies and best practices (libswarm, sy...
May. 25, 2015 12:00 PM EDT Reads: 1,806
SYS-CON Events announced today that DragonGlass, an enterprise search platform, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. After eleven years of designing and building custom applications, OpenCrowd has launched DragonGlass, a cloud-based platform that enables the development of search-based applications. These are a new breed of applications that utilize a search index as their backbone for data retrieval. They can easily adapt to new data sets and provide access to both structured and unstruc...
May. 25, 2015 12:00 PM EDT Reads: 1,752
There's Big Data, then there's really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at Big Data Expo®, Hannah Smalltree, Director at Treasure Data, discussed how IoT, Big Data and deployments are processing massive data volumes from wearables, utilities and other machines...
May. 25, 2015 10:00 AM EDT Reads: 4,062
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists will peel away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud environment, and we must architect and code accordingly. At the very least, you'll have no problem fil...
May. 25, 2015 10:00 AM EDT Reads: 1,741
SYS-CON Events announced today that MetraTech, now part of Ericsson, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Ericsson is the driving force behind the Networked Society- a world leader in communications infrastructure, software and services. Some 40% of the world’s mobile traffic runs through networks Ericsson has supplied, serving more than 2.5 billion subscribers.
May. 25, 2015 09:45 AM EDT Reads: 1,088
The worldwide cellular network will be the backbone of the future IoT, and the telecom industry is clamoring to get on board as more than just a data pipe. In his session at @ThingsExpo, Evan McGee, CTO of Ring Plus, Inc., discussed what service operators can offer that would benefit IoT entrepreneurs, inventors, and consumers. Evan McGee is the CTO of RingPlus, a leading innovative U.S. MVNO and wireless enabler. His focus is on combining web technologies with traditional telecom to create a new breed of unified communication that is easily accessible to the general consumer. With over a de...
May. 25, 2015 06:00 AM EDT Reads: 4,661