Click here to close now.

Welcome!

Web 2.0 Authors: Liz McMillan, Carmen Gonzalez, Pat Romanski, Lori MacVittie, Mike Kavis

Related Topics: Big Data Journal, Java, SOA & WOA, Virtualization, Web 2.0, Cloud Expo, Apache

Big Data Journal: Article

Examining the True Cost of Big Data

As you start on your Big Data journey or project, be sure to ask what exactly the business requires

The good news about the Big Data market is that we generally all agree on the definition of Big Data, which has come to be known as data that has volume, velocity and variety where businesses need to collect, store, manage and analyze in order to derive business value or otherwise known as the "4 V's." However, the problem with such a broad definition is that it can mean different things to different people once you start to put some real values next to those V's.

Let's be honest, Volume can be a different thing to different organizations. To some it is anything above 10 terabytes of managed data in their BI environment and to others it is petabyte scale and nothing less. Likewise velocity can be multi-billions of daily records coming into the enterprise from various external and internal networks. When it really comes down to it, each business situation will be quite different not only from a size and speed perspective but also more important from the business use-case or requirement. A large bank's Big Data problem could be very different to that of an online retailer or an airline. If you compare what say a hospital is trying to do collecting and analyzing all the sensor patient data compared to a utilities provider running a smart-grid or a telecommunications operator. True, all could be categorized as machine generated or raw data but the exact type of data might be different not to mention the volume or growth rate. Probably the one unique common denominator across all aforementioned industries is that everyone is keeping the data for longer time-periods. No one is throwing it away - not even the detailed data.

The Many Cost Factors to Consider
Costs will of course vary depending on the individual allocated IT budget but regardless, how the company allocates IT budget dollars to new Big Data initiatives needs consideration. Let's face it, enterprise buyers didn't suddenly come into a bunch of newfound IT assets or line items on their budget and the current world economic situation would certainly not suggest so. More likely existing budgets are being re-allocated and instead of spending more on say existing traditional data warehouses or appliances, monies are being allocated to new projects running on open source projects including Apache Hadoop which promises both low cost, ease of scale not to mention the obvious best approach to managing and analyzing multi-structured data sets. The difficultly then arises how do you integrate or have your Hadoop environment co-exist with the established BI or DW environment that the business has grown to love and rely upon?

Leverage What You Already Have
Let's assume you have a data warehouse or data mart in place today and you already use various ETL or data movement tools and BI dashboard, analytics or reporting tools and you don't want to disrupt business users which could not only impacting performance levels but also training up on a new set of tools. In fact you already likely beholden to strict SLA's around response times for the various business reports and KPI's. However, at the same time the business is demanding access to new data sets in order to glean better insights either directly analyzing this data or co-mingling it with existing customer data. This could take the form of web-logs, click stream data or social media data from various interactive sites the business is now leveraging and tracking. The promise of impacting profit margins and gaining a competitive edge just cannot be avoided.

As we all know, traditional relational or columnar databases can't handle the unstructured data types so IT needs to rollout a different solution to satisfy the business demands. Evaluations can take many forms but typically will start with which Hadoop distribution, which NoSQL or NewSQL database and what query access tools in addition to MapReduce. It is certainly no easy task as there are a large number of technology solutions on the market today that claim to run on or with Hadoop providing MapReduce or SQL-like capabilities which all satisfy the requirement of managing volumes of unstructured data. Some are more mature than others; some proven and not all are low-cost. Open source on the surface looks very low cost but as soon as you require any level of support, which lets face it once it's live and relied upon as a business critical environment, you will need to allocate a line item on your budget. The Big Data line item won't just be one line as it will need to include all components required to properly rollout a Big Data solution to truly satisfy the business demands. Just like any other IT environment the obvious pieces will include: Software licensing and support, hardware, skilled dedicated resources, professional services and training and the dedicated time of business users to provide input on key requirements including specifying types of reports, queries and analysis which will naturally change and evolve over time.

Big Data Costs Can Quickly Creep Up
In terms of the hardware expenditure required to manage the new Big Data set, you may start out with a Hadoop cluster of say 10 nodes and yes that is certainly manageable but if your data velocity is significant, you can quickly reach 100+ nodes and now you will face a number of other expenses including additional headcount and skilled resources to manage the environment proactively in addition to tools for managing the cluster including system management and alerting and potentially add-on software which can vary by business use-case but might cover real-time analytics against streaming data for say fraud detection or detection of unusual patterns. You may also need a business tool to provide a front-end GUI dashboard to track specific KPIs or data visualization tools so business users can quickly understand what is going on. Very quickly the costs become less about the storage and hardware and more around the software that focuses on getting the most value from this newly collected data set.

There is no denying the fact that Big Data presents great new opportunities but reaching the point of a quantifiable ROI in a fast time frame is still a very real challenge. Everyone is talking about Big Data and all the innovative technology approaches to tackling it but it is still difficult to find lots of business success stories within any one-industry sector. It's still fairly immature but the good news is that its moving at a much faster pace than any other IT project today and certainly our data warehouse and BI forefathers have provided lessons learned over the past two decades.

Big Data Is Big Business but It Comes with Strict Requirements
If we want to examine more closely the main areas of expenditure for a Big Data project, it is probably best to look at it through the lens of a specific type of business and use-case. Let's take a large financial institution that has a number of existing traditional data warehouse / BI environments but because the business doesn't want to throw any data away (well let's face it regulations don't allow that for a number of years) and realistically the business wants to retain specific data sets for ongoing trending and analysis. This includes examining questions such as "what constitutes a low-risk client based on spending behavior patterns over a specific time period cross-referenced with customer demographics" which will help the institution better target a particular segment of the market.

Given the IT budget doesn't allow for increased spend that correlates with data growth rates, they need to seriously reduce costs and so decide to go the route of a Hadoop-based environment given its promise for low-cost scale and the fact that it can provide insights into customer patterns by capturing semi- and unstructured data. Front-ending the warehouse with a dedicated Hadoop cluster is the preferred architectural approach but the business users still want access to both the Hadoop environment and the existing traditional data warehouse environment.

Given we are talking about a financial institution, the question of security and availability quickly come to the top of the requirements list. At the same time, if business users want to access that data, SQL query access and using the current BI tool against that new set of data is also a requirement. If you can avoid having to the move large chunks of data on a frequent basis from one to the other, it will not only reduce costs but also latency. In an ideal world, being able to leverage the skill sets you already have and avoiding duplication of work is key.

Below is a quick table outlining the main cost factors to be considered and a set of comments against each of these areas that could reduce costs.

 

Big Data on Hadoop Cost Factors

Key Consideration to drive down cost

 

Storage

Look at databases that provide data compression to yield storage savings (better than GZip or LZO).

 

Hardware (Nodes)

Granular data compression at database level will reduce nodes over time.

 

Data Analytics - Skilled Resources

Examine technology solutions that provide standard SQL or BI tool access in addition to MapReduce (Pig etc.)

 

Cluster management - Skilled Resources

Leverage existing Dev-operations staff if you deploy a SQL-compliant data environment

 

Security

Look for database solutions that provide built-in security permissions and access.

 

Availability / DR

Consider a data management environment that doesn't require additional tools for replication.

 

Training

Consider solutions where you don't need to retrain or hire all new resources. Leverage what you have (standard SQL-skilled DBAs)

Summary: Consider All Factors and Get Business Buy-in Quickly
Big Data is fundamentally a business problem. If you begin with the question of "what is the business trying to achieve by collecting, storing and analyzing this new set of data...", you will start down the right path to realizing business gains. Whether you outsource the initiative or bring in external consultants and vendors to manage the project, the same questions will arise and in order to leverage what you already have which includes both existing IT environments and skills, you will be better able to contain costs.

Furthermore, we all love the promise of new innovative technologies including Hadoop and MapReduce but without leveraging tried and tested standards we have come to love and respect, it doesn't make a whole lot of sense from both a technical or economic sense. As you start on your Big Data journey or project, be sure to ask what exactly the business requires and how can you leverage what you already have today. We all know, getting business user buy-in and success is half the battle to a successful rollout.

More Stories By John Bantleman

John Bantleman, CEO of RainStor, has more than 20 years’ experience in the management of software companies. Prior to overseeing RainStor, he transformed LBMS into a $45 million business prior to its successful NASDAQ flotation in 1997. Today’s LBMS’ technology is now part of CA’s product portfolio. The following year John was instrumental in the launch of Evolve, and drove the company through to a successful IPO on NASDAQ.

Returning to the UK in 2003, John spent 12 months working on the advisory boards of venture capital organizations such as Apax Partners. He joined RainStor Inc. as Chairman in 2004 and became CEO at the start of 2007 and relocated back to the US to head-up worldwide operations in 2009.

Comments (3) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Vikas.Deolaliker 09/21/12 06:49:00 PM EDT

Great article. Another data point, the IT budget is up only 4% in 2013 over 2012, so don't expect everyone to rush into Bigdata.

The fourth "V" is visualization. If you cannot render the analysis in a intuitive way, there is no value in that analysis. In fact, visualization should be the first step in design of a bigdata system - it helps trim down the architectural bloat into something that is within budget and useful.

Elad Israeli 09/19/12 06:07:00 PM EDT

Fascinating post. Still waiting for someone to crack the nut that is Big Data Analytics.

douglaney 08/29/12 03:36:00 PM EDT

Great piece John. Excellent detail. Thought you and your readers might be interested in where the "3Vs" of big data originated--in a Gartner piece I authored over 11 years ago. I recently unearthed a copy so folks to refer to and cite it.

Cheers,
Doug Laney, VP Research, Gartner, @doug_laney

@ThingsExpo Stories
Operational Hadoop and the Lambda Architecture for Streaming Data Apache Hadoop is emerging as a distributed platform for handling large and fast incoming streams of data. Predictive maintenance, supply chain optimization, and Internet-of-Things analysis are examples where Hadoop provides the scalable storage, processing, and analytics platform to gain meaningful insights from granular data that is typically only valuable from a large-scale, aggregate view. One architecture useful for capturing and analyzing streaming data is the Lambda Architecture, representing a model of how to analyze rea...
SYS-CON Events announced today that Vitria Technology, Inc. will exhibit at SYS-CON’s @ThingsExpo, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Vitria will showcase the company’s new IoT Analytics Platform through live demonstrations at booth #330. Vitria’s IoT Analytics Platform, fully integrated and powered by an operational intelligence engine, enables customers to rapidly build and operationalize advanced analytics to deliver timely business outcomes for use cases across the industrial, enterprise, and consumer segments.
The explosion of connected devices / sensors is creating an ever-expanding set of new and valuable data. In parallel the emerging capability of Big Data technologies to store, access, analyze, and react to this data is producing changes in business models under the umbrella of the Internet of Things (IoT). In particular within the Insurance industry, IoT appears positioned to enable deep changes by altering relationships between insurers, distributors, and the insured. In his session at @ThingsExpo, Michael Sick, a Senior Manager and Big Data Architect within Ernst and Young's Financial Servi...
SYS-CON Events announced today that Open Data Centers (ODC), a carrier-neutral colocation provider, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. Open Data Centers is a carrier-neutral data center operator in New Jersey and New York City offering alternative connectivity options for carriers, service providers and enterprise customers.
When it comes to the Internet of Things, hooking up will get you only so far. If you want customers to commit, you need to go beyond simply connecting products. You need to use the devices themselves to transform how you engage with every customer and how you manage the entire product lifecycle. In his session at @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, will show how “product relationship management” can help you leverage your connected devices and the data they generate about customer usage and product performance to deliver extremely compelling and reliabl...
SYS-CON Events announced today that CodeFutures, a leading supplier of database performance tools, has been named a “Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. CodeFutures is an independent software vendor focused on providing tools that deliver database performance tools that increase productivity during database development and increase database performance and scalability during production.
The IoT market is projected to be $1.9 trillion tidal wave that’s bigger than the combined market for smartphones, tablets and PCs. While IoT is widely discussed, what not being talked about are the monetization opportunities that are created from ubiquitous connectivity and the ensuing avalanche of data. While we cannot foresee every service that the IoT will enable, we should future-proof operations by preparing to monetize them with extremely agile systems.
There’s Big Data, then there’s really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. Learn about IoT, Big Data and deployments processing massive data volumes from wearables, utilities and other machines.
The explosion of connected devices / sensors is creating an ever-expanding set of new and valuable data. In parallel the emerging capability of Big Data technologies to store, access, analyze, and react to this data is producing changes in business models under the umbrella of the Internet of Things (IoT). In particular within the Insurance industry, IoT appears positioned to enable deep changes by altering relationships between insurers, distributors, and the insured. In his session at @ThingsExpo, Michael Sick, a Senior Manager and Big Data Architect within Ernst and Young's Financial Servi...
“In the past year we've seen a lot of stabilization of WebRTC. You can now use it in production with a far greater degree of certainty. A lot of the real developments in the past year have been in things like the data channel, which will enable a whole new type of application," explained Peter Dunkley, Technical Director at Acision, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Intelligent Systems Services will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Established in 1994, Intelligent Systems Services Inc. is located near Washington, DC, with representatives and partners nationwide. ISS’s well-established track record is based on the continuous pursuit of excellence in designing, implementing and supporting nationwide clients’ mission-critical systems. ISS has completed many successful projects in Healthcare, Commercial, Manufacturing, ...
PubNub on Monday has announced that it is partnering with IBM to bring its sophisticated real-time data streaming and messaging capabilities to Bluemix, IBM’s cloud development platform. “Today’s app and connected devices require an always-on connection, but building a secure, scalable solution from the ground up is time consuming, resource intensive, and error-prone,” said Todd Greene, CEO of PubNub. “PubNub enables web, mobile and IoT developers building apps on IBM Bluemix to quickly add scalable realtime functionality with minimal effort and cost.”
The major cloud platforms defy a simple, side-by-side analysis. Each of the major IaaS public-cloud platforms offers their own unique strengths and functionality. Options for on-site private cloud are diverse as well, and must be designed and deployed while taking existing legacy architecture and infrastructure into account. Then the reality is that most enterprises are embarking on a hybrid cloud strategy and programs. In this Power Panel at 15th Cloud Expo (http://www.CloudComputingExpo.com), moderated by Ashar Baig, Research Director, Cloud, at Gigaom Research, Nate Gordon, Director of T...
Sensor-enabled things are becoming more commonplace, precursors to a larger and more complex framework that most consider the ultimate promise of the IoT: things connecting, interacting, sharing, storing, and over time perhaps learning and predicting based on habits, behaviors, location, preferences, purchases and more. In his session at @ThingsExpo, Tom Wesselman, Director of Communications Ecosystem Architecture at Plantronics, will examine the still nascent IoT as it is coalescing, including what it is today, what it might ultimately be, the role of wearable tech, and technology gaps stil...
DevOps tends to focus on the relationship between Dev and Ops, putting an emphasis on the ops and application infrastructure. But that’s changing with microservices architectures. In her session at DevOps Summit, Lori MacVittie, Evangelist for F5 Networks, will focus on how microservices are changing the underlying architectures needed to scale, secure and deliver applications based on highly distributed (micro) services and why that means an expansion into “the network” for DevOps.
The Internet of Things (IoT) is causing data centers to become radically decentralized and atomized within a new paradigm known as “fog computing.” To support IoT applications, such as connected cars and smart grids, data centers' core functions will be decentralized out to the network's edges and endpoints (aka “fogs”). As this trend takes hold, Big Data analytics platforms will focus on high-volume log analysis (aka “logs”) and rely heavily on cognitive-computing algorithms (aka “cogs”) to make sense of it all.
The Internet of Everything (IoE) brings together people, process, data and things to make networked connections more relevant and valuable than ever before – transforming information into knowledge and knowledge into wisdom. IoE creates new capabilities, richer experiences, and unprecedented opportunities to improve business and government operations, decision making and mission support capabilities. In his session at @ThingsExpo, Gary Hall, Chief Technology Officer, Federal Defense at Cisco Systems, will break down the core capabilities of IoT in multiple settings and expand upon IoE for bo...
With several hundred implementations of IoT-enabled solutions in the past 12 months alone, this session will focus on experience over the art of the possible. Many can only imagine the most advanced telematics platform ever deployed, supporting millions of customers, producing tens of thousands events or GBs per trip, and hundreds of TBs per month. With the ability to support a billion sensor events per second, over 30PB of warm data for analytics, and hundreds of PBs for an data analytics archive, in his session at @ThingsExpo, Jim Kaskade, Vice President and General Manager, Big Data & Ana...
For years, we’ve relied too heavily on individual network functions or simplistic cloud controllers. However, they are no longer enough for today’s modern cloud data center. Businesses need a comprehensive platform architecture in order to deliver a complete networking suite for IoT environment based on OpenStack. In his session at @ThingsExpo, Dhiraj Sehgal from PLUMgrid will discuss what a holistic networking solution should really entail, and how to build a complete platform that is scalable, secure, agile and automated.
We’re no longer looking to the future for the IoT wave. It’s no longer a distant dream but a reality that has arrived. It’s now time to make sure the industry is in alignment to meet the IoT growing pains – cooperate and collaborate as well as innovate. In his session at @ThingsExpo, Jim Hunter, Chief Scientist & Technology Evangelist at Greenwave Systems, will examine the key ingredients to IoT success and identify solutions to challenges the industry is facing. The deep industry expertise behind this presentation will provide attendees with a leading edge view of rapidly emerging IoT oppor...