Welcome!

Agile Computing Authors: Nate Vickery, Liz McMillan, Elizabeth White, Pat Romanski, Jeev Trika

Related Topics: Agile Computing, Java IoT, Microservices Expo, Microsoft Cloud

Agile Computing: Article

Testing Basics Might Have Averted Obamacare Health Site Fiasco

Attention to testing best practices could’ve avoided a hellish user experience and bad PR

It made headlines for all the wrong reasons when it launched on October 1, but things could have been so different for the HealthCare.gov website if only it had been tested properly before release. Users trying to enroll encountered all sorts of glitches, including very slow page updates, "page not found" errors and frequent crashes.

Early server outages were blamed on an unexpectedly high volume of traffic as nearly 5 million Americans tried to access the website on day one, but it soon emerged that serious flaws existed in the software, and the security was not properly assessed or signed off.

According to CBS, the security testing was never completed. Fox uncovered a testing bulletin from the day before the launch that revealed the site could only handle 1,100 users "before response time gets too high." The Washington Examiner revealed, via an anonymous source, that the full testing was delayed until just a few days before the launch and instead of the 4 to 6 months of testing that should have been conducted it was only tested for 4 to 6 days.

Amid the apologies, the resignations, and the frantic efforts to fix it up by the end of November, there are serious and important lessons to be learned. A proper testing plan with a realistic schedule would have prevented this catastrophe.

Start with an Estimate
It's incredibly rare for any software to be released with zero defects, but major functional bugs and inadequate security is certainly avoidable if you plan correctly. That starts with a realistic estimate of the scope of the testing that's required. The QA department must be consulted and asked to use their experience to provide a picture of how much testing is needed.

That plan will be based on documentation outlining the requirements of the software and discussion with the developers, as well as the wealth of experience that testing professionals possess. If requirements change significantly, or new requests are introduced, then the plan must be altered to cater for that. This is one major area where things obviously went awry. According to the Washington Examiner's source there were "ever-changing, conflicting and exceedingly late project directions. The actual system requirements for Oct. 1 were changing up until the week before."

This is a clear recipe for disaster.

Agile Testing
Modern software development is typically based on Agile methodology where requirements are built into the system quickly and feedback informs the project going forward. This approach does not mesh with traditional testing where testers would work out a comprehensive test plan based on detailed documentation, and then carry out that testing in a predefined block at the end of the project.

To adapt testing for modern software development it pays to get testers involved earlier in the process. They need to understand the system and really identify with the end user. It's much more cost effective to fix flaws and bugs sooner rather than later.

There's a logistical consideration as well. Each new build means a full regression test, bug fix verification, and a healthy dose of exploratory testing to make sure the new features are working as intended. It's important for the test team to scale up as the amount of work grows, and as much of the regression testing as possible should be automated to reduce the workload.

Exploratory Testing
With a fast-paced development it is absolutely vital to get experienced testers and have them perform some level of exploratory testing. This combines their knowledge about how the system should work with educated guesses about where it might fail. It's also very useful when documentation is lacking because testers can effectively design and execute tests at the same time.

Targeted exploratory testing is the perfect complement to scripted testing. It requires some creative thinking and some freedom for the tester, but it can be a great way of emulating an end user and ensuring that specific features and functions actually deliver what they're supposed to. Properly recorded by good cloud-based testing tools, the data can be used to provide clarity for developers trying to fix problems, and it can serve as the basis of scripted testing or even automated tests in the future.

Test Management
A project such as this, where disparate teams have to work together toward a common goal, can be an integration nightmare. Test management can be a real challenge, so the right tool is invaluable. The full lifecycle of every defect or requirement should be recorded to produce a clear chain from the original feature request, through the test case, to the defect, and on to repeated test cycles. It has to be clear who is responsible for each action every step of the way, so the blame game can be avoided entirely.

The ultimate aim is traceability, usability, and transparency.

If this data is gathered then it becomes easier to apply root cause analysis at a later date and discover where things went wrong. Remember that the earlier you can catch and fix the defect, the cheaper and easier it is to do. Identifying the root causes of the problems with the HealthCare.gov website requires an objective analysis of the original requirements, the documentation, the code implementation and integration, the test planning, and the test cycles. Understanding what went wrong through this process could ensure that the same mistakes are not made again in the future.

Knowing When to Pull the Trigger
Kathleen Sebelius, the health and human services secretary, apologized for her part in the botched website launch, but the real problem, and her cardinal sin, was to tell Obama that the website was ready to be launched in the first place.

QA departments are not the gatekeepers for projects, business decisions are always going to trump everything else, and the pressure to deliver ensures that every project launches with defects in it, but you ignore them at your peril. If the testers had been consulted about the state of the website and the back end before launch, you can bet they would have pointed out that it wasn't ready for prime time. A one- or two-month delay would undoubtedly have been greeted with some alarm and criticism, but it would have caused far less damaging PR than releasing an unfinished and potentially insecure product.

More Stories By Vu Lam

Vu Lam is founder and CEO of QASymphony, developers of defect capture tools that track user interactions with applications. He was previously with First Consulting Group and was an early pioneer in Vietnam’s offshore IT services industry since 1995. He holds an MS degree in electrical engineering from Purdue University. You may reach him at [email protected]

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
In his session at 18th Cloud Expo, Bruce Swann, Senior Product Marketing Manager at Adobe, will discuss how the Adobe Marketing Cloud can help marketers embrace opportunities for personalized, relevant and real-time customer engagement across offline (direct mail, point of sale, call center) and digital (email, website, SMS, mobile apps, social networks, connected objects). Bruce Swann has more than 15 years of experience working with digital marketing disciplines like web analytics, social med...
Designing IoT applications is complex, but deploying them in a scalable fashion is even more complex. A scalable, API first IaaS cloud is a good start, but in order to understand the various components specific to deploying IoT applications, one needs to understand the architecture of these applications and figure out how to scale these components independently. In his session at @ThingsExpo, Nara Rajagopalan is CEO of Accelerite, will discuss the fundamental architecture of IoT applications, ...
The essence of data analysis involves setting up data pipelines that consist of several operations that are chained together – starting from data collection, data quality checks, data integration, data analysis and data visualization (including the setting up of interaction paths in that visualization). In our opinion, the challenges stem from the technology diversity at each stage of the data pipeline as well as the lack of process around the analysis.
As cloud and storage projections continue to rise, the number of organizations moving to the cloud is escalating and it is clear cloud storage is here to stay. However, is it secure? Data is the lifeblood for government entities, countries, cloud service providers and enterprises alike and losing or exposing that data can have disastrous results. There are new concepts for data storage on the horizon that will deliver secure solutions for storing and moving sensitive data around the world. ...
What a difference a year makes. Organizations aren’t just talking about IoT possibilities, it is now baked into their core business strategy. With IoT, billions of devices generating data from different companies on different networks around the globe need to interact. From efficiency to better customer insights to completely new business models, IoT will turn traditional business models upside down. In the new customer-centric age, the key to success is delivering critical services and apps wit...
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York and Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty ...
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit y...
There are several IoTs: the Industrial Internet, Consumer Wearables, Wearables and Healthcare, Supply Chains, and the movement toward Smart Grids, Cities, Regions, and Nations. There are competing communications standards every step of the way, a bewildering array of sensors and devices, and an entire world of competing data analytics platforms. To some this appears to be chaos. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will discuss the vast to...
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.
SYS-CON Events announced today that Enzu, a leading provider of cloud hosting solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to foc...
SYS-CON Events announced today the How to Create Angular 2 Clients for the Cloud Workshop, being held June 7, 2016, in conjunction with 18th Cloud Expo | @ThingsExpo, at the Javits Center in New York, NY. Angular 2 is a complete re-write of the popular framework AngularJS. Programming in Angular 2 is greatly simplified. Now it’s a component-based well-performing framework. The immersive one-day workshop led by Yakov Fain, a Java Champion and a co-founder of the IT consultancy Farata Systems and...
IoT generates lots of temporal data. But how do you unlock its value? How do you coordinate the diverse moving parts that must come together when developing your IoT product? What are the key challenges addressed by Data as a Service? How does cloud computing underlie and connect the notions of Digital and DevOps What is the impact of the API economy? What is the business imperative for Cognitive Computing? Get all these questions and hundreds more like them answered at the 18th Cloud Expo...
Customer experience has become a competitive differentiator for companies, and it’s imperative that brands seamlessly connect the customer journey across all platforms. With the continued explosion of IoT, join us for a look at how to build a winning digital foundation in the connected era – today and in the future. In his session at @ThingsExpo, Chris Nguyen, Group Product Marketing Manager at Adobe, will discuss how to successfully leverage mobile, rapidly deploy content, capture real-time d...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, will provide an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life ...
SYS-CON Events announced today that ContentMX, the marketing technology and services company with a singular mission to increase engagement and drive more conversations for enterprise, channel and SMB technology marketers, has been named “Sponsor & Exhibitor Lounge Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York City, New York. “CloudExpo is a great opportunity to start a conversation with new prospects, but what happens after the...
SYS-CON Events announced today that 24Notion has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. 24Notion is full-service global creative digital marketing, technology and lifestyle agency that combines strategic ideas with customized tactical execution. With a broad understand of the art of traditional marketing, new media, communications and social influence, 24Notion uniquely understands how to con...
The demand for organizations to expand their infrastructure to multiple IT environments like the cloud, on-premise, mobile, bring your own device (BYOD) and the Internet of Things (IoT) continues to grow. As this hybrid infrastructure increases, the challenge to monitor the security of these systems increases in volume and complexity. In his session at 18th Cloud Expo, Stephen Coty, Chief Security Evangelist at Alert Logic, will show how properly configured and managed security architecture can...
When it comes to IoT in the enterprise, namely the commercial building and hospitality markets, a benefit not getting the attention it deserves is energy efficiency, and IoT's direct impact on a cleaner, greener environment when installed in smart buildings. Until now clean technology was offered piecemeal and led with point solutions that require significant systems integration to orchestrate and deploy. There didn't exist a 'top down' approach that can manage and monitor the way a Smart Buildi...
SYS-CON Events announced today BZ Media LLC has been named “Media Sponsor” of SYS-CON's 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. BZ Media LLC is a high-tech media company that produces technical conferences and expositions, and publishes a magazine, newsletters and websites in the software development, SharePoint, mobile development and Commercial Drone markets.
WebRTC is bringing significant change to the communications landscape that will bridge the worlds of web and telephony, making the Internet the new standard for communications. Cloud9 took the road less traveled and used WebRTC to create a downloadable enterprise-grade communications platform that is changing the communication dynamic in the financial sector. In his session at @ThingsExpo, Leo Papadopoulos, CTO of Cloud9, will discuss the importance of WebRTC and how it enables companies to fo...