Welcome!

Agile Computing Authors: Elizabeth White, Liz McMillan, Pat Romanski, Larry Alton, Astadia CloudGPS

Related Topics: Recurring Revenue, Microsoft Cloud, Containers Expo Blog, @CloudExpo

Recurring Revenue: Article

Big Data on Grids or on Clouds?

Now that we have a new computing paradigm, Cloud Computing, how can Clouds help our data?

Now that we have a new computing paradigm, Cloud Computing, how can Clouds help our data? Replace our internal data vaults as we hoped Grids would? Are Grids dead now that we have Clouds? Despite all the promising developments in the Grid and Cloud computing space, and the avalanche of publications and talks on this subject, many people still seem to be confused about internal data and compute resources, versus Grids versus Clouds, and they are hesitant to take the next step. I think there are a number of issues driving this uncertainty.

Grids didn't keep all their promises
Grids did not evolve (as some of us originally thought) into the next fundamental IT infrastructure for everything and everybody. Because of the diversity of computing and data environments, we had to develop different middleware (department, enterprise, global, compute, data, sensors, scientific instruments, etc.), and had to face different usage models with different benefits. Enterprise Grids were (and are) providing better resource utilization and business flexibility, while global Grids are best suited to complex R&D collaboration with resource sharing. For enterprise usage, setting up and operating Grids was often complicated and did not remove all the (data) bottlenecks. For researchers this characteristic was seen to be a necessary evil. Implementing complex applications on supercomputers has never been easy. So what.

Grid: the way station to the Cloud
After 40 years of dealing with data processing, Grid computing was indeed the next big thing for the grand challenge R&D expert, while for the enterprise CIO, the Grid was a way station on its way to the Cloud model. For the enterprise today, Clouds are providing all the missing pieces: easy to use, economies of scale, business elasticity up and down, and pay-as you go (thus getting rid of some capital expenditure). And in cases where security matters, there is the private Cloud, within the enterprise's firewall. In more complex enterprise environments, with applications running under different policies, private Clouds can easily connect via the Internet to (external) public Clouds -- and vice versa -- forming a hybrid Cloud infrastructure that balances security with efficiency.

Different policies, what does that mean?
No data processing job is alike. Jobs differ by priority, strategic importance, deadline, budget, IP and licenses. In addition, the nature of the code often necessitates a specific computer architecture, operating system, memory, storage, and other resources. These important differentiating factors strongly influence where and when a data processing job is run. For any job, a set of specific requirements decide on the set of specific policies that have to be defined and programmed, such that any of these jobs will run just according to these policies. Ideally, this is guaranteed by a dynamic resource broker that controls submission to Grid or Cloud resources, be they local or global, private or public.

Grids or Clouds?
One important question is still open: how do I find out, and then tell the resource broker, whether my data should run on the Grid or in the Cloud? The answer, among others, depends on the algorithmic structure of the program, which might be intolerant of high latency and low bandwidth. The performance limitations are exhibited mainly by parallel applications with tightly-coupled, data-intensive inter-process communication, running in parallel on hundreds or even thousands of processors or cores.

The good news is, however, that many applications do not require high bandwidth and low latency. Parameter studies often seen in science, engineering, and business intelligence, where a single self-contained application executes with many different parameters, resulting in many independent jobs. The list of examples is extensive -  analyzing the data from a particle physics collider, identifying the solution parameter in optimization, ensemble runs to quantify climate model uncertainties, identifying potential drug targets via screening a database of ligand structures, studying economic model sensitivity to parameters, and analyzing different materials and their resistance in crash tests, to name just a few.

Big Data needs Grids or Clouds, and often both
Obviously, there is no "Grids or Clouds" for the enterprise. There is just "Grids and Clouds", it really depends on the individual scenario. In general, CIOs have to evaluate three different scenarios:

  • (1) the Private Cloud: optimizing and virtualizing the company's internal enterprise IT infrastructure, including the data layer (here is where Momentum can help);
  • (2) the Hybrid Cloud: do (1) and connect to external clouds;
  • or (3) the Public Cloud: do (2) and successively move data (processing) to the external cloud provider.

The choice for the best-suited scenario depends on many aspects: sensitive / competitive data and applications (e.g. medical patient records), individual return on investment, security policies, interoperability between private and public clouds, lose of control when moving data outside the corporation, cloud-enabling data and applications, the current software licensing model, protection of intellectual property, legal issues, and more.

The good news is that CIOs can always start with a hybrid infrastructure in mind: combining private and public cloud resources, balanced according to specific requirements. This provides the best of both worlds, avoiding the worst of each individual world.

More Stories By Pawel Plaszczak

Pawel’s international software engineering experience includes work at CERN, British Telecommunications and Argonne National Laboratory. In 2003, Pawel founded GridwiseTech to lead pioneering work for the early adopters of scalable systems. Under Pawel's leadership the company has won the trust and respect of customers including Turner Broadcasting, Ricoh, and Philips, and led numerous research efforts for international consortia. Pawel is the author of numerous articles and tutorials, the book "Grid Computing: The Savvy Manager's Guide", and a frequent speaker at professional conferences and events.

@ThingsExpo Stories
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It’s clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Tha...
SYS-CON Events announced today that Avere Systems, a leading provider of enterprise storage for the hybrid cloud, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Avere delivers a more modern architectural approach to storage that doesn't require the overprovisioning of storage capacity to achieve performance, overspending on expensive storage media for inactive data or the overbui...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant th...
Digital transformation is changing the face of business. The IDC predicts that enterprises will commit to a massive new scale of digital transformation, to stake out leadership positions in the "digital transformation economy." Accordingly, attendees at the upcoming Cloud Expo | @ThingsExpo at the Santa Clara Convention Center in Santa Clara, CA, Oct 31-Nov 2, will find fresh new content in a new track called Enterprise Cloud & Digital Transformation.
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software. They hope to capture value from emerging technologies such as IoT, SDN, and AI. Ultimately, irrespective of the vertical, it is about deriving value from independent software applications participating in an ecosystem as one comprehensive solution. In his session at @ThingsExpo, Kausik Sridhar, founder and CTO of Pulzze Systems, will discuss how given the magnitude of today's applicati...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...