Agile Computing Authors: Elizabeth White, Carmen Gonzalez, Liz McMillan, JP Morgenthal, Jim Kaskade

Related Topics: @CloudExpo

@CloudExpo: Article

Creating a Common Cloud Computing Reference API - Part One

A lot of what I'm outlining here may be better suited to a reference implementation than a standard

Reuven Cohen's "ElasticVapor" Blog

Over the last few weeks I've been engaged in several conversations about the need for a common, interoperable and open set of cloud computing standards. During these conversations a recurring theme has started to emerge. A need for cloud interoperability or the ability for diverse cloud systems and organizations to work together in a common way.

In my discussion yesterday with Rich Wolski of the Eucalyptus project he described the need for a "CloudVirt" API similar to that of the Libvirt project for virtualization. For those of you that don't know about libvirt, it's an open source toolkit which enables a common API interaction with the virtualization capabilities of recent versions of Linux (and other OSes).

I would like to take this opportunity to share my ideas as well as get some feedback on some of the key points I see for the creation of common cloud computing reference API or standard.

* Cloud Resource Description
The ability to describe resources is (in my opinion) the most important aspect of any standardization effort. One potential avenue might be to use the Resource Description Framework proposed by the W3C. The Resource Description Framework (RDF) is a family of specifications, originally designed as a metadata data model, which has come to be used as a general method of modeling information through a variety of syntax formats. The RDF metadata model is based upon the idea of making statements about Web resources (or Cloud Resources) in the form of subject-predicate-object expressions, called triples in RDF lingo. This standardized approach could be modified as a primary mechanism for describing cloud resources both locally and remotely.

* Cloud Federation (Cloud 2 Cloud)
The holy grail of cloud computing may very well be the ability to seamlessly bridge both private clouds (datacenters) and remote cloud resources such as EC2 in a secure and efficient manor. To accomplish this a federation standard must be enabled. One of the biggest hurdles to over come in federation is the lack of clear definition to what federation is.

So let me take a stab at defining it.

Cloud federation manages consistency and access controls when two or more independent geographically distinct clouds share either authentication, files, computing resources, command and control or access to storage resources. Cloud federations can be classified into three categories: peer-to-peer, replication, and hierarchical. Peer 2 peer seems to be the most logical first step in creating a federation spec. Protocols like XMPP, P4P and Virtual Distributed Ethernet may make for good starting points.

* Distributed Network Management
The need for a distributed and optimized virtual network is an important aspect in any multi-cloud deployment. One potential direction could be to explore the use of VPN or VDE technologies. My preference would be to use VDE, (Virtual Distributed Ethernet). A quick refresher, a VPN is a way to connect one or more remote computers to a protected network, generally tunnelling the traffic through another network. VDE implements a virtual ethernet in all its aspects, virtual switches, virtual cables. A VDE can also be used to create a VPN.

VDE interconnects real computers running (through a tap interface), virtual machines as well as the other networking interfaces through a common open framework. VDE supports heterogeneous virtual machines running on different hosting computers and could be the ideal starting point. Network shaping and optimization may also play an important role in the ability to bridge two or cloud resources.

Some network optimization aspects may include;

  • Compression - Relies on data patterns that can be represented more efficiently.
  • Caching/Proxy - Relies on human behavior , accessing the same data over and over.
  • Protocol Spoofing - Bundles multiple requests from chatty applications into one.
  • Application Shaping - Controls data usage based on spotting specific patterns in the data and allowing or disallowing specific traffic.
  • Equalizing - Makes assumptions on what needs immediate priority based on the data usage.
  • Connection Limits - Prevents access gridlock in routers and access points due to denial of service or peer to peer.
  • Simple Rate Limits - Prevents one user from getting more than a fixed amount of data.

* Memory Management
When looking at the creation of compute cloud memory tends to be a major factor in the performance of a given virtual environment, whether a virtual machine or some other application component. Cloud memory management will need to involve ways to allocate portions of virtual memory to programs at their request, and freeing it for reuse when no longer needed. This is particularly important in "platform as a service" cloud deployments.

Several key memory management aspects may include;

  • Provide memory space to enable several processes to be executed at the same time
  • Provide a satisfactory level of performance for the system users
  • Protect each program's resources
  • Share (if desired) memory space between processes
  • Make the addressing of memory space as transparent as possible for the programmer.

* Distributed Storage
I've been working on creating a cloud abstraction layer called "cloud raid" as part of our ElasticDrive platform and have been looking at different approaches for our implementation. My initial idea is to connect multiple remote cloud storage services (S3, Nirvanix, CloudFS) for a variety of purposes. During my research the XAM specification began to look like the most suitable candidate. XAM addresses storage interoperability, information assurance (security), storage transparency, long-term records retention and automation for Information Lifecycle Management (ILM)-based practices.

XAM looks to solve key cloud storage problem spots including;

  • Interoperability: Applications can work with any XAM conformant storage system; information can be migrated and shared
  • Compliance: Integrated record retention and disposition metadata
  • ILM Practices: Framework for classification, policy, and implementation
  • Migration: Ability to automate migration process to maintain long-term readability
  • Discovery: Application-independent structured discovery avoids application obsolescence

Potential Future Additions to the API

* I/o
The virtualization of I/O resources is a critical part of enabling a set of emerging cloud deployment models. In large scale cloud deployments a recurring issue has the ability to effectively management I/o resources whether on a machine level or network. One of the problems a lot of users are encountering is that of the "nasty neighbor" or a user who has taken all available system I/o resources.

A common I/o API for sharing, security, performance, and scalability will need to be addressed to help resolve these issues. I've been speaking with several hardware vendors on how we might be able to address this problem. This will most like have to be done at a later point after a first draft has been released.

* Monitoring and System Metrics
One of the best aspects of using cloud technology is the ability to scale applications in tandem to the underlying infrastructure and the demands placed on it. Rather then just scaling on system load, users should have the ability to selectively scale on other metrics such as response time, network throughput or other metrics made available. Having a uniform way to interact with system metrics will enable cloud providers and consumers a common way to scale applications.

Security & Auditability.
In my conversations with several wall street CIO's the questions of both security and cloud transparency with regards to external audits has come up frequently.

My list of requirements is by no means a complete list. Cloud computing encompasses a wide variety of technologies, architectures and deployment models. What I am attempting to do is address the initial pain points whether you are deploying a cloud or just using it. A lot of what I've outlined may be better suited to a reference implementation than a standard, but none the less I thought I'd put these out ideas out for discussion.

-- Update --

Looks like I've forgotten an obvious yet important aspect to my cloud standards. Authentication. Maybe something like OAuth or OpenID could form the basis for this as well. I'll need to do some more thinking on this one.

More Stories By Reuven Cohen

An instigator, part time provocateur, bootstrapper, amateur cloud lexicographer, and purveyor of random thoughts, 140 characters at a time.

Reuven is an early innovator in the cloud computing space as the founder of Enomaly in 2004 (Acquired by Virtustream in February 2012). Enomaly was among the first to develop a self service infrastructure as a service (IaaS) platform (ECP) circa 2005. As well as SpotCloud (2011) the first commodity style cloud computing Spot Market.

Reuven is also the co-creator of CloudCamp (100+ Cities around the Globe) CloudCamp is an unconference where early adopters of Cloud Computing technologies exchange ideas and is the largest of the ‘barcamp’ style of events.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

Most Recent Comments
MiamiWebDesigner 08/25/08 06:25:14 AM EDT

Kudos to the Cloud Crowd for Re-Inventing the Wheel!

One thing 30 years in the IT industry has taught me is that the more things change, the more they stay the same. Another is that the only memory we seem to access is short-term. Yet another is that techno-marketeers rely on that, so they can put labels like "revolutionary" and "innovative" on platforms, products and services that are mere re-inventions of the wheel ... and often poor copies at that.

A good example is all the buzz about "Cloud Computing" in general and "SaaS" (software as a service) in particular:


Both terms are bogus. The only true cloud computing takes place in aircraft. What they're actually referring to by "the cloud" is a large-scale and often remotely located and managed computing platform. We have had those since the dawn of electronic IT. IBM calls them "mainframes":


The only innovation offered by today's cloud crowd is actually more of a speculation, i.e. that server farms can deliver the same solid performance as Big Iron. And even that's not original. Anyone remember Datapoint's ARCnet, or DEC's VAXclusters? Whatever happened to those guys, anyway...?

And as for SaaS, selling the sizzle while keeping the steak is a marketing ploy most rightfully accredited to society's oldest profession. Its first application in IT was (and for many still is) known as the "service bureau". And I don't mean the contemporary service bureau (mis)conception labelled "Service 2.0" by a Wikipedia contributor whose historical perspective is apparently constrained to four years:


Instead, I mean the computer service bureau industry that spawned ADAPSO (the Association of Data Processing Service Organizations) in 1960, and whose chronology comprises a notable portion of the IEEE's "Annals of the History of Computing":


So ... for any of you slide rule-toting, pocket-protected keypunch-card cowboys who may be just coming out of a 40-year coma, let me give you a quick IT update:

1. "Mainframe" is now "Cloud" (with concomitant ethereal substance).

2. "Terminal" is now "Web Browser" (with much cooler games, and infinitely more distractions).

3. "Service Bureau" is now "SaaS" (but app upgrades are just as painful, and custom mods equally elusive).

4. Most IT buzzwords boil down to techno-hyped BS (just as they always have).

Bruce Arnold, Web Design Miami Florida

@ThingsExpo Stories
SYS-CON Events announced today that SoftNet Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. SoftNet Solutions specializes in Enterprise Solutions for Hadoop and Big Data. It offers customers the most open, robust, and value-conscious portfolio of solutions, services, and tools for the shortest route to success with Big Data. The unique differentiator is the ability to architect and ...
"Matrix is an ambitious open standard and implementation that's set up to break down the fragmentation problems that exist in IP messaging and VoIP communication," explained John Woolf, Technical Evangelist at Matrix, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
In past @ThingsExpo presentations, Joseph di Paolantonio has explored how various Internet of Things (IoT) and data management and analytics (DMA) solution spaces will come together as sensor analytics ecosystems. This year, in his session at @ThingsExpo, Joseph di Paolantonio from DataArchon, will be adding the numerous Transportation areas, from autonomous vehicles to “Uber for containers.” While IoT data in any one area of Transportation will have a huge impact in that area, combining sensor...
Established in 1998, Calsoft is a leading software product engineering Services Company specializing in Storage, Networking, Virtualization and Cloud business verticals. Calsoft provides End-to-End Product Development, Quality Assurance Sustenance, Solution Engineering and Professional Services expertise to assist customers in achieving their product development and business goals. The company's deep domain knowledge of Storage, Virtualization, Networking and Cloud verticals helps in delivering ...
SYS-CON Events announced today that 910Telecom will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Housed in the classic Denver Gas & Electric Building, 910 15th St., 910Telecom is a carrier-neutral telecom hotel located in the heart of Denver. Adjacent to CenturyLink, AT&T, and Denver Main, 910Telecom offers connectivity to all major carriers, Internet service providers, Internet backbones and ...
In his general session at 18th Cloud Expo, Lee Atchison, Principal Cloud Architect and Advocate at New Relic, discussed cloud as a ‘better data center’ and how it adds new capacity (faster) and improves application availability (redundancy). The cloud is a ‘Dynamic Tool for Dynamic Apps’ and resource allocation is an integral part of your application architecture, so use only the resources you need and allocate /de-allocate resources on the fly.
In the next five to ten years, millions, if not billions of things will become smarter. This smartness goes beyond connected things in our homes like the fridge, thermostat and fancy lighting, and into heavily regulated industries including aerospace, pharmaceutical/medical devices and energy. “Smartness” will embed itself within individual products that are part of our daily lives. We will engage with smart products - learning from them, informing them, and communicating with them. Smart produc...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smar...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, will discuss the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docke...
WebRTC sits at the intersection between VoIP and the Web. As such, it poses some interesting challenges for those developing services on top of it, but also for those who need to test and monitor these services. In his session at WebRTC Summit, Tsahi Levent-Levi, co-founder of testRTC, reviewed the various challenges posed by WebRTC when it comes to testing and monitoring and on ways to overcome them.
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life sett...
SYS-CON Events announced today that Coalfire will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Coalfire is the trusted leader in cybersecurity risk management and compliance services. Coalfire integrates advisory and technical assessments and recommendations to the corporate directors, executives, boards, and IT organizations for global brands and organizations in the technology, cloud, health...
@ThingsExpo has been named the Top 5 Most Influential M2M Brand by Onalytica in the ‘Machine to Machine: Top 100 Influencers and Brands.' Onalytica analyzed the online debate on M2M by looking at over 85,000 tweets to provide the most influential individuals and brands that drive the discussion. According to Onalytica the "analysis showed a very engaged community with a lot of interactive tweets. The M2M discussion seems to be more fragmented and driven by some of the major brands present in the...
The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
SYS-CON Events announced today that Transparent Cloud Computing (T-Cloud) Consortium will exhibit at the 19th International Cloud Expo®, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. The Transparent Cloud Computing Consortium (T-Cloud Consortium) will conduct research activities into changes in the computing model as a result of collaboration between "device" and "cloud" and the creation of new value and markets through organic data proces...
SYS-CON Events announced today that MathFreeOn will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. MathFreeOn is Software as a Service (SaaS) used in Engineering and Math education. Write scripts and solve math problems online. MathFreeOn provides online courses for beginners or amateurs who have difficulties in writing scripts. In accordance with various mathematical topics, there are more tha...
@ThingsExpo has been named the Top 5 Most Influential Internet of Things Brand by Onalytica in the ‘The Internet of Things Landscape 2015: Top 100 Individuals and Brands.' Onalytica analyzed Twitter conversations around the #IoT debate to uncover the most influential brands and individuals driving the conversation. Onalytica captured data from 56,224 users. The PageRank based methodology they use to extract influencers on a particular topic (tweets mentioning #InternetofThings or #IoT in this ...
SYS-CON Events announced today that Niagara Networks will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Niagara Networks offers the highest port-density systems, and the most complete Next-Generation Network Visibility systems including Network Packet Brokers, Bypass Switches, and Network TAPs.
SYS-CON Events announced today that Embotics, the cloud automation company, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Embotics is the cloud automation company for IT organizations and service providers that need to improve provisioning or enable self-service capabilities. With a relentless focus on delivering a premier user experience and unmatched customer support, Embotics is the fas...
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and ...