Welcome!

Agile Computing Authors: Liz McMillan, John Basso, Igor Drobiazko, Pat Romanski, Dana Gardner

Related Topics: @CloudExpo

@CloudExpo: Article

Creating a Common Cloud Computing Reference API - Part One

A lot of what I'm outlining here may be better suited to a reference implementation than a standard

Reuven Cohen's "ElasticVapor" Blog

Over the last few weeks I've been engaged in several conversations about the need for a common, interoperable and open set of cloud computing standards. During these conversations a recurring theme has started to emerge. A need for cloud interoperability or the ability for diverse cloud systems and organizations to work together in a common way.

In my discussion yesterday with Rich Wolski of the Eucalyptus project he described the need for a "CloudVirt" API similar to that of the Libvirt project for virtualization. For those of you that don't know about libvirt, it's an open source toolkit which enables a common API interaction with the virtualization capabilities of recent versions of Linux (and other OSes).

I would like to take this opportunity to share my ideas as well as get some feedback on some of the key points I see for the creation of common cloud computing reference API or standard.

* Cloud Resource Description
The ability to describe resources is (in my opinion) the most important aspect of any standardization effort. One potential avenue might be to use the Resource Description Framework proposed by the W3C. The Resource Description Framework (RDF) is a family of specifications, originally designed as a metadata data model, which has come to be used as a general method of modeling information through a variety of syntax formats. The RDF metadata model is based upon the idea of making statements about Web resources (or Cloud Resources) in the form of subject-predicate-object expressions, called triples in RDF lingo. This standardized approach could be modified as a primary mechanism for describing cloud resources both locally and remotely.

* Cloud Federation (Cloud 2 Cloud)
The holy grail of cloud computing may very well be the ability to seamlessly bridge both private clouds (datacenters) and remote cloud resources such as EC2 in a secure and efficient manor. To accomplish this a federation standard must be enabled. One of the biggest hurdles to over come in federation is the lack of clear definition to what federation is.

So let me take a stab at defining it.

Cloud federation manages consistency and access controls when two or more independent geographically distinct clouds share either authentication, files, computing resources, command and control or access to storage resources. Cloud federations can be classified into three categories: peer-to-peer, replication, and hierarchical. Peer 2 peer seems to be the most logical first step in creating a federation spec. Protocols like XMPP, P4P and Virtual Distributed Ethernet may make for good starting points.

* Distributed Network Management
The need for a distributed and optimized virtual network is an important aspect in any multi-cloud deployment. One potential direction could be to explore the use of VPN or VDE technologies. My preference would be to use VDE, (Virtual Distributed Ethernet). A quick refresher, a VPN is a way to connect one or more remote computers to a protected network, generally tunnelling the traffic through another network. VDE implements a virtual ethernet in all its aspects, virtual switches, virtual cables. A VDE can also be used to create a VPN.

VDE interconnects real computers running (through a tap interface), virtual machines as well as the other networking interfaces through a common open framework. VDE supports heterogeneous virtual machines running on different hosting computers and could be the ideal starting point. Network shaping and optimization may also play an important role in the ability to bridge two or cloud resources.

Some network optimization aspects may include;

  • Compression - Relies on data patterns that can be represented more efficiently.
  • Caching/Proxy - Relies on human behavior , accessing the same data over and over.
  • Protocol Spoofing - Bundles multiple requests from chatty applications into one.
  • Application Shaping - Controls data usage based on spotting specific patterns in the data and allowing or disallowing specific traffic.
  • Equalizing - Makes assumptions on what needs immediate priority based on the data usage.
  • Connection Limits - Prevents access gridlock in routers and access points due to denial of service or peer to peer.
  • Simple Rate Limits - Prevents one user from getting more than a fixed amount of data.

* Memory Management
When looking at the creation of compute cloud memory tends to be a major factor in the performance of a given virtual environment, whether a virtual machine or some other application component. Cloud memory management will need to involve ways to allocate portions of virtual memory to programs at their request, and freeing it for reuse when no longer needed. This is particularly important in "platform as a service" cloud deployments.

Several key memory management aspects may include;

  • Provide memory space to enable several processes to be executed at the same time
  • Provide a satisfactory level of performance for the system users
  • Protect each program's resources
  • Share (if desired) memory space between processes
  • Make the addressing of memory space as transparent as possible for the programmer.

* Distributed Storage
I've been working on creating a cloud abstraction layer called "cloud raid" as part of our ElasticDrive platform and have been looking at different approaches for our implementation. My initial idea is to connect multiple remote cloud storage services (S3, Nirvanix, CloudFS) for a variety of purposes. During my research the XAM specification began to look like the most suitable candidate. XAM addresses storage interoperability, information assurance (security), storage transparency, long-term records retention and automation for Information Lifecycle Management (ILM)-based practices.

XAM looks to solve key cloud storage problem spots including;

  • Interoperability: Applications can work with any XAM conformant storage system; information can be migrated and shared
  • Compliance: Integrated record retention and disposition metadata
  • ILM Practices: Framework for classification, policy, and implementation
  • Migration: Ability to automate migration process to maintain long-term readability
  • Discovery: Application-independent structured discovery avoids application obsolescence

Potential Future Additions to the API

* I/o
The virtualization of I/O resources is a critical part of enabling a set of emerging cloud deployment models. In large scale cloud deployments a recurring issue has the ability to effectively management I/o resources whether on a machine level or network. One of the problems a lot of users are encountering is that of the "nasty neighbor" or a user who has taken all available system I/o resources.

A common I/o API for sharing, security, performance, and scalability will need to be addressed to help resolve these issues. I've been speaking with several hardware vendors on how we might be able to address this problem. This will most like have to be done at a later point after a first draft has been released.

* Monitoring and System Metrics
One of the best aspects of using cloud technology is the ability to scale applications in tandem to the underlying infrastructure and the demands placed on it. Rather then just scaling on system load, users should have the ability to selectively scale on other metrics such as response time, network throughput or other metrics made available. Having a uniform way to interact with system metrics will enable cloud providers and consumers a common way to scale applications.

Security & Auditability.
In my conversations with several wall street CIO's the questions of both security and cloud transparency with regards to external audits has come up frequently.

My list of requirements is by no means a complete list. Cloud computing encompasses a wide variety of technologies, architectures and deployment models. What I am attempting to do is address the initial pain points whether you are deploying a cloud or just using it. A lot of what I've outlined may be better suited to a reference implementation than a standard, but none the less I thought I'd put these out ideas out for discussion.

-- Update --

Looks like I've forgotten an obvious yet important aspect to my cloud standards. Authentication. Maybe something like OAuth or OpenID could form the basis for this as well. I'll need to do some more thinking on this one.

More Stories By Reuven Cohen

An instigator, part time provocateur, bootstrapper, amateur cloud lexicographer, and purveyor of random thoughts, 140 characters at a time.

Reuven is an early innovator in the cloud computing space as the founder of Enomaly in 2004 (Acquired by Virtustream in February 2012). Enomaly was among the first to develop a self service infrastructure as a service (IaaS) platform (ECP) circa 2005. As well as SpotCloud (2011) the first commodity style cloud computing Spot Market.

Reuven is also the co-creator of CloudCamp (100+ Cities around the Globe) CloudCamp is an unconference where early adopters of Cloud Computing technologies exchange ideas and is the largest of the ‘barcamp’ style of events.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
MiamiWebDesigner 08/25/08 06:25:14 AM EDT

Kudos to the Cloud Crowd for Re-Inventing the Wheel!

One thing 30 years in the IT industry has taught me is that the more things change, the more they stay the same. Another is that the only memory we seem to access is short-term. Yet another is that techno-marketeers rely on that, so they can put labels like "revolutionary" and "innovative" on platforms, products and services that are mere re-inventions of the wheel ... and often poor copies at that.

A good example is all the buzz about "Cloud Computing" in general and "SaaS" (software as a service) in particular:

http://tinyurl.com/6let8x

Both terms are bogus. The only true cloud computing takes place in aircraft. What they're actually referring to by "the cloud" is a large-scale and often remotely located and managed computing platform. We have had those since the dawn of electronic IT. IBM calls them "mainframes":

http://tinyurl.com/5kdhcb

The only innovation offered by today's cloud crowd is actually more of a speculation, i.e. that server farms can deliver the same solid performance as Big Iron. And even that's not original. Anyone remember Datapoint's ARCnet, or DEC's VAXclusters? Whatever happened to those guys, anyway...?

And as for SaaS, selling the sizzle while keeping the steak is a marketing ploy most rightfully accredited to society's oldest profession. Its first application in IT was (and for many still is) known as the "service bureau". And I don't mean the contemporary service bureau (mis)conception labelled "Service 2.0" by a Wikipedia contributor whose historical perspective is apparently constrained to four years:

http://tinyurl.com/5fpb8e

Instead, I mean the computer service bureau industry that spawned ADAPSO (the Association of Data Processing Service Organizations) in 1960, and whose chronology comprises a notable portion of the IEEE's "Annals of the History of Computing":

http://tinyurl.com/5lvjdl

So ... for any of you slide rule-toting, pocket-protected keypunch-card cowboys who may be just coming out of a 40-year coma, let me give you a quick IT update:

1. "Mainframe" is now "Cloud" (with concomitant ethereal substance).

2. "Terminal" is now "Web Browser" (with much cooler games, and infinitely more distractions).

3. "Service Bureau" is now "SaaS" (but app upgrades are just as painful, and custom mods equally elusive).

4. Most IT buzzwords boil down to techno-hyped BS (just as they always have).

Bruce Arnold, Web Design Miami Florida
http://www.PervasivePersuasion.com

@ThingsExpo Stories
This is not a small hotel event. It is also not a big vendor party where politicians and entertainers are more important than real content. This is Cloud Expo, the world's longest-running conference and exhibition focused on Cloud Computing and all that it entails. If you want serious presentations and valuable insight about Cloud Computing for three straight days, then register now for Cloud Expo.
IoT device adoption is growing at staggering rates, and with it comes opportunity for developers to meet consumer demand for an ever more connected world. Wireless communication is the key part of the encompassing components of any IoT device. Wireless connectivity enhances the device utility at the expense of ease of use and deployment challenges. Since connectivity is fundamental for IoT device development, engineers must understand how to overcome the hurdles inherent in incorporating multipl...
The increasing popularity of the Internet of Things necessitates that our physical and cognitive relationship with wearable technology will change rapidly in the near future. This advent means logging has become a thing of the past. Before, it was on us to track our own data, but now that data is automatically available. What does this mean for mHealth and the "connected" body? In her session at @ThingsExpo, Lisa Calkins, CEO and co-founder of Amadeus Consulting, will discuss the impact of wea...
SYS-CON Events announced today that Stratoscale, the software company developing the next generation data center operating system, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Stratoscale is revolutionizing the data center with a zero-to-cloud-in-minutes solution. With Stratoscale’s hardware-agnostic, Software Defined Data Center (SDDC) solution to store everything, run anything and scale everywhere...
So, you bought into the current machine learning craze and went on to collect millions/billions of records from this promising new data source. Now, what do you do with them? Too often, the abundance of data quickly turns into an abundance of problems. How do you extract that "magic essence" from your data without falling into the common pitfalls? In her session at @ThingsExpo, Natalia Ponomareva, Software Engineer at Google, will provide tips on how to be successful in large scale machine lear...
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus inter...
Angular 2 is a complete re-write of the popular framework AngularJS. Programming in Angular 2 is greatly simplified – now it's a component-based well-performing framework. This immersive one-day workshop at 18th Cloud Expo, led by Yakov Fain, a Java Champion and a co-founder of the IT consultancy Farata Systems and the product company SuranceBay, will provide you with everything you wanted to know about Angular 2.
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
You deployed your app with the Bluemix PaaS and it's gaining some serious traction, so it's time to make some tweaks. Did you design your application in a way that it can scale in the cloud? Were you even thinking about the cloud when you built the app? If not, chances are your app is going to break. Check out this webcast to learn various techniques for designing applications that will scale successfully in Bluemix, for the confidence you need to take your apps to the next level and beyond.
We’ve worked with dozens of early adopters across numerous industries and will debunk common misperceptions, which starts with understanding that many of the connected products we’ll use over the next 5 years are already products, they’re just not yet connected. With an IoT product, time-in-market provides much more essential feedback than ever before. Innovation comes from what you do with the data that the connected product provides in order to enhance the customer experience and optimize busi...
Increasing IoT connectivity is forcing enterprises to find elegant solutions to organize and visualize all incoming data from these connected devices with re-configurable dashboard widgets to effectively allow rapid decision-making for everything from immediate actions in tactical situations to strategic analysis and reporting. In his session at 18th Cloud Expo, Shikhir Singh, Senior Developer Relations Manager at Sencha, will discuss how to create HTML5 dashboards that interact with IoT devic...
Artificial Intelligence has the potential to massively disrupt IoT. In his session at 18th Cloud Expo, AJ Abdallat, CEO of Beyond AI, will discuss what the five main drivers are in Artificial Intelligence that could shape the future of the Internet of Things. AJ Abdallat is CEO of Beyond AI. He has over 20 years of management experience in the fields of artificial intelligence, sensors, instruments, devices and software for telecommunications, life sciences, environmental monitoring, process...
SYS-CON Events announced today that Ericsson has been named “Gold Sponsor” of SYS-CON's @ThingsExpo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. Ericsson is a world leader in the rapidly changing environment of communications technology – providing equipment, software and services to enable transformation through mobility. Some 40 percent of global mobile traffic runs through networks we have supplied. More than 1 billion subscribers around the world re...
Digital payments using wearable devices such as smart watches, fitness trackers, and payment wristbands are an increasing area of focus for industry participants, and consumer acceptance from early trials and deployments has encouraged some of the biggest names in technology and banking to continue their push to drive growth in this nascent market. Wearable payment systems may utilize near field communication (NFC), radio frequency identification (RFID), or quick response (QR) codes and barcodes...
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry's single source for the cloud. Fusion's advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including cloud...
In his session at @ThingsExpo, Chris Klein, CEO and Co-founder of Rachio, will discuss next generation communities that are using IoT to create more sustainable, intelligent communities. One example is Sterling Ranch, a 10,000 home development that – with the help of Siemens – will integrate IoT technology into the community to provide residents with energy and water savings as well as intelligent security. Everything from stop lights to sprinkler systems to building infrastructures will run ef...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
The IoT has the potential to create a renaissance of manufacturing in the US and elsewhere. In his session at 18th Cloud Expo, Florent Solt, CTO and chief architect of Netvibes, will discuss how the expected exponential increase in the amount of data that will be processed, transported, stored, and accessed means there will be a huge demand for smart technologies to deliver it. Florent Solt is the CTO and chief architect of Netvibes. Prior to joining Netvibes in 2007, he co-founded Rift Technol...
Join IBM June 8 at 18th Cloud Expo at the Javits Center in New York City, NY, and learn how to innovate like a startup and scale for the enterprise. You need to deliver quality applications faster and cheaper, attract and retain customers with an engaging experience across devices, and seamlessly integrate your enterprise systems. And you can't take 12 months to do it.