Agile Computing Authors: Carmen Gonzalez, Liz McMillan, Elizabeth White, William Schmarzo, John Mertic

Related Topics: @CloudExpo

@CloudExpo: Blog Post

Monitoring Applications in the Cloud

Small steps make your environment more resilient

Cloud computing is a rather powerful tool that allows even the smallest of businesses to provide an enterprise class environment for web applications.  In a nutshell, the cloud is nothing more than the ability to rent computer services on demand from a 3rd party provider.  At MioWorks.com we use Amazon Web Services, but there are several other services out there for you to explore.

Mastering the cloud takes a bit of work, a dash of experience and an openness to learn from others.  But once you do master it, the benefits are tremendous.  You’ll never have to order another server or rent a rack in a data center.  You’ll be able to fluidly control your environment by increasing and decreasing the services you need on the fly, saving time and money.

This power, flexibility and potential demands that you pay attention to the details.  You must anticipate that the cloud can have hiccups and that as quickly as a server comes to life, that server can disappear.  In previous blog posts I’ve already talked about the importance of backups and recovery drills, but let’s take a step back.  Today let’s talk about monitoring and how important it is to your survival.

Ok I’ll bite, why is monitoring so important

Let me sum this up in a single sentence: Monitoring can be the difference between “whew that was close” and “holy s$%t we are down”.  I lied -  I need another sentence…  Monitoring can also be the difference between a five minute outage and a five hour outage.

What to monitor

Every web based application environment in the cloud is a jigsaw puzzle of pieces.  At the core you have your virtual hardware followed by your operating system.  Each of your servers is then configured differently depending on its specific duty.  You may have application servers, web servers, search servers, database servers and the list goes on.  Each of these servers needs to be monitored from several points of view - both internally and externally.

Internal Monitoring

The big question isn’t “Is the server running?” it should be “Is the server and all of its pieces running correctly? Each virtual server in your setup is a maze of processes, files, directories and file systems.  At any given time a hiccup can occur within this delicate environment that will eventually disrupt the end user’s ability to use your service.   In our environment we use monit and munin (two open source tools) on the inside to provide us with critical monitoring, recovery & trending capabilities.

Monit provides systems monitoring and error recovery for our Unix systems.  In our environment we have configured monit to watch dozens of potential failure points.   Monit can start a process if it is not running and can kill/restart a process if it takes too many resources. Monit is also configurable as an intrusion detection system by watching for changes in files, directories and file systems.  By spending a little time learning and using Monit your system administrator has a great tool to keep a constant eye on all the pieces of the puzzle.

In addition to the direct monitoring and error recovery system, we also like to see the bigger picture.  We use Munin to aggregate information across our server pool.  Munin provides a graphical view that allows your team to quickly see what’s different from yesterday.   You can quickly determine your resource utlization and plan in ADVANCE any increase of capacity.

From the outside

Keeping track of all the pieces inside the cloud is very important, but you also need to know how your environment in the cloud is performing to the outside world.  There are more external monitoring services out there than I can count.  But I’ll tell you who we use.  Our favorite at the moment is monitis.com.  We like them because starting at just $10/month you get on demand fault & performance monitoring for your environment. This external watchdog system helps to keep everyone informed if/when the cloud is having issues.  It also provides us with important statistics on response time and application performance that we use to determine how to adjust our infrastructure.

Continuous improvement

Your monitoring program must become a living, breathing element of your systems administration.  As new problems arise or potential problems are identified, the monitoring system must be adjusted to be proactive.   The good news is that the more you adjust your monitoring and error recovery system, the less you’ll be surprised in the future.  It takes discipline to post mortem each problem and determine how to proactively detect for it in the future.  And this discipline will distinguish your application in the frenzy of the cloud.

Real world results of a good monitoring program

In the real world your monitoring system can be the difference between keeping your systems alive and thriving OR having unhappy customers and missed SLAs. It can help you pinpoint exactly what went wrong and reduce the time it takes for the first responders to identify and solve the issue.  There are lots of solutions in the marketplace including commerical  and open source alternatives.  It may seem overwhelming at first, but once you start the process and improve little by little, you’ll be amazed at the positive impact your monitoring program will have on your environment stability and your ability to get some sleep.

More Stories By David Abramowski

David Abramowski is a co-founder of guidearama.com. David's background as a technologist and a product marketing manager enables him to look at today's solutions from the perspective of the user. David's career spans early stage startups including Axent Technologies, Vignette and Morph Labs as well as enterprise mainstays such as Symantec. You can also follow David on twitter @dabramowski

@ThingsExpo Stories
@ThingsExpo has been named the Top 5 Most Influential Internet of Things Brand by Onalytica in the ‘The Internet of Things Landscape 2015: Top 100 Individuals and Brands.' Onalytica analyzed Twitter conversations around the #IoT debate to uncover the most influential brands and individuals driving the conversation. Onalytica captured data from 56,224 users. The PageRank based methodology they use to extract influencers on a particular topic (tweets mentioning #InternetofThings or #IoT in this ...
@ThingsExpo has been named the Top 5 Most Influential M2M Brand by Onalytica in the ‘Machine to Machine: Top 100 Influencers and Brands.' Onalytica analyzed the online debate on M2M by looking at over 85,000 tweets to provide the most influential individuals and brands that drive the discussion. According to Onalytica the "analysis showed a very engaged community with a lot of interactive tweets. The M2M discussion seems to be more fragmented and driven by some of the major brands present in the...
In the next forty months – just over three years – businesses will undergo extraordinary changes. The exponential growth of digitization and machine learning will see a step function change in how businesses create value, satisfy customers, and outperform their competition. In the next forty months companies will take the actions that will see them get to the next level of the game called Capitalism. Or they won’t – game over. The winners of today and tomorrow think differently, follow different...
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and ...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smar...
Virgil consists of an open-source encryption library, which implements Cryptographic Message Syntax (CMS) and Elliptic Curve Integrated Encryption Scheme (ECIES) (including RSA schema), a Key Management API, and a cloud-based Key Management Service (Virgil Keys). The Virgil Keys Service consists of a public key service and a private key escrow service. 

The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, will discuss the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
What happens when the different parts of a vehicle become smarter than the vehicle itself? As we move toward the era of smart everything, hundreds of entities in a vehicle that communicate with each other, the vehicle and external systems create a need for identity orchestration so that all entities work as a conglomerate. Much like an orchestra without a conductor, without the ability to secure, control, and connect the link between a vehicle’s head unit, devices, and systems and to manage the ...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning techniques such as Cognitive Fingerprinting, wind project operators can utilize these tools to learn from collected data, detect regular patterns, and optimize their own operations. In his session at 18th Cloud Expo, Stuart Gillen, Director of Business Development at SparkCognition, discussed how research has demonstrated the value of Machine Learning in delivering next generation analytics to impr...
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
Amazon has gradually rolled out parts of its IoT offerings, but these are just the tip of the iceberg. In addition to optimizing their backend AWS offerings, Amazon is laying the ground work to be a major force in IoT - especially in the connected home and office. In his session at @ThingsExpo, Chris Kocher, founder and managing director of Grey Heron, explained how Amazon is extending its reach to become a major force in IoT by building on its dominant cloud IoT platform, its Dash Button strat...
SYS-CON Events announced today that SoftNet Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. SoftNet Solutions specializes in Enterprise Solutions for Hadoop and Big Data. It offers customers the most open, robust, and value-conscious portfolio of solutions, services, and tools for the shortest route to success with Big Data. The unique differentiator is the ability to architect and ...
A critical component of any IoT project is what to do with all the data being generated. This data needs to be captured, processed, structured, and stored in a way to facilitate different kinds of queries. Traditional data warehouse and analytical systems are mature technologies that can be used to handle certain kinds of queries, but they are not always well suited to many problems, particularly when there is a need for real-time insights.
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
One of biggest questions about Big Data is “How do we harness all that information for business use quickly and effectively?” Geographic Information Systems (GIS) or spatial technology is about more than making maps, but adding critical context and meaning to data of all types, coming from all different channels – even sensors. In his session at @ThingsExpo, William (Bill) Meehan, director of utility solutions for Esri, will take a closer look at the current state of spatial technology and ar...
Everyone knows that truly innovative companies learn as they go along, pushing boundaries in response to market changes and demands. What's more of a mystery is how to balance innovation on a fresh platform built from scratch with the legacy tech stack, product suite and customers that continue to serve as the business' foundation. In his General Session at 19th Cloud Expo, Michael Chambliss, Head of Engineering at ReadyTalk, will discuss why and how ReadyTalk diverted from healthy revenue an...
You have great SaaS business app ideas. You want to turn your idea quickly into a functional and engaging proof of concept. You need to be able to modify it to meet customers' needs, and you need to deliver a complete and secure SaaS application. How could you achieve all the above and yet avoid unforeseen IT requirements that add unnecessary cost and complexity? You also want your app to be responsive in any device at any time. In his session at 19th Cloud Expo, Mark Allen, General Manager of...
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here