Welcome!

Agile Computing Authors: Rishi Bhargava, Elizabeth White, Ben Uher, Liz McMillan, Yeshim Deniz

Related Topics: @CloudExpo, Java IoT, Microservices Expo, Linux Containers, Containers Expo Blog, @BigDataExpo

@CloudExpo: Article

Take Control of Your Schemalessness with Dynamic Schemas

Addressing the inflexibility of structured data by enabling schemaless data to be dynamically and logically structured

Static data structures have been at the heart of data processing tools since the dawn of computing, but they have always limited the flexibility of the organization leveraging the data. Recently, the rise of flexible formats like JSON have led to schemaless data as an attempt to increase agility. However, schemaless data have proven difficult to work with, because of hidden rigid structure in the form of implied schemas.

EnterpriseWeb addresses the problems of both the inflexibility of structured data as well as the impracticality of schemaless data, by enabling schemaless data to be dynamically and logically structured.

From the fixed-length fields of the 1950s, to the relational structures of modern database management systems, to the semistructured data formats XML and JSON, the structure of our data has always informed code about how it should be processed. Data are defined by their relationships, and we used to hard-code those relationships into rigid structures. That approach allows only one static view, which is difficult to work with, and even more difficult to change. Nevertheless, such rigid data structures - and the models that represent them - are an integral part of enterprise information management.

Traditional relational database management systems (RDBMSs) exemplify this point with their static entity-relationship models (ERMs) and tightly interconnected data structures. XML improves this situation slightly, allowing semi-structured information, but schemas still constrain flexibility and performance. With both approaches, fixed definitions, views, and reports limit the ability for businesses to freely transform information into insight and become obstacles to systemwide change.

The Rise of Schemalessness
This challenge of inflexible data structures has given rise to schemaless data. With JSON in particular, we can create whatever data structure we like when we author data. We don't have to shoehorn data into rigid data structures, thus allowing every record to have its own structure.

But there is a problem with schemaless data. Consider this simple task: how do you create a query for all the addresses in a particular Zip Code if every record has a different name or format for Zip Code? Schemalessness, after all, isn't magic - even schemaless data require some kind of metadata so the code will know how to process such information, what software development guru Martin Fowler calls an implied schema.

Implied schemas represent the structure inherent in any data record. If each address record has its own format, then that format provides the implied schema for that record. Dealing with implied schemas thus falls to the developer, who must figure out how to code software to process these implied schemas, which are different for each and every record.

In Fowler's tutorial on schemalessness, he explains the pros and cons of implied schemas. Despite acknowledging the power of schemalessness to support more flexible and responsive user experiences, he recommends avoiding it and implied schemas for developer convenience. Good advice with respect to traditional software, but the world of data is changing. Today we live in an increasingly schemaless world, where more often than not, the structure of our data is fluid or nonexistent.

Raising the Discussion to Dynamic Schemas
Fowler makes it clear that in the past it has been impractical from the developer's perspective to work systematically with schemaless data, because implied schemas are difficult to deal with. After all, structure is itself useful, and isn't the problem per se. Rather, how to avoid the limitations of static structure without falling into the trap of unmanageable schemaless data that is the real challenge.

EnterpriseWeb's unique approach to modeling solves this critically important challenge by leveraging dynamic schemas that have flexible, metadata-driven relationships with underlying information. Using metadata this way separates concerns, letting people consider relationships from multiple perspectives, rather than from a single static point of view. In addition, it's now possible to change and extend metadata to meet diverse business needs without disruption.

Instead of settling for complex ERMs with their inflexible, tightly coupled data structures or dealing with the coding complexities of implied schemas, developers can project dynamic schemas from the metadata simply by writing different transformations. As a result, dynamic schemas are developer friendly and dynamic - a welcome change from the difficult problem of schemalessness.

Add an Agent for Performance
So far so good, but how do we build software to process all such data in a general way, freeing ourselves from custom coding for implicit schemas? The solution is an intelligent agent.

EnterpriseWeb's intelligent agent, SmartAlex™, is a distributable transaction manager that resolves dynamic schemas for each interaction. Every human or system client interaction is a request for SmartAlex to interpret dynamic schemas (as well as other models and additional metadata) and translate them to a context-specific set of resources in order to construct a custom response.

This Agent-Oriented approach maximizes performance for such dynamic computing. In the background, SmartAlex handles all run time connection and transformation details, sparing programmers from manually integrating resources for varied and unanticipated uses, greatly improving IT productivity while enabling business agility.

SmartAlex logs all system events, indexes all new and updated resources, and tags all changes in relationships for detailed and navigable audit history. This practice creates a feedback loop as SmartAlex leverages the same indexed logs to guide its execution. Data, code, and user interface components, as well as connectors for federated services, systems, databases, and devices, can be updated or replaced without breaking related apps and processes - as SmartAlex is ‘aware' of the changes. In this way EnterpriseWeb supports real time exception and change management for resilient solutions that can evolve naturally.

The EnterpriseWeb Take
Schemalessness was a reaction to the limitations of structured data. People struggled with the constraints of static structure, and figured that if they simply got rid of structure, then the problem would go away. But this move was merely a shell game, as the limitations of fixed schemas shifted to implied schemas, now without the benefits of structure to inform the code responsible for their processing.

The solution is to raise the level of abstraction, and instead of arguing over fixed vs. implied schemas, to work at the dynamic schema level. Such an approach is model-driven, allowing application designers to build models that capture their data structures, and allowing an intelligent agent to use the metadata each model represents to meet the specific needs of each interaction. The real lesson here is that the solution to resolving the challenge of schemalessness combines both dynamic schemas and the action of the agent. Stay tuned to my next newsletter for more information.

More Stories By Jason Bloomberg

Jason Bloomberg is the leading expert on architecting agility for the enterprise. As president of Intellyx, Mr. Bloomberg brings his years of thought leadership in the areas of Cloud Computing, Enterprise Architecture, and Service-Oriented Architecture to a global clientele of business executives, architects, software vendors, and Cloud service providers looking to achieve technology-enabled business agility across their organizations and for their customers. His latest book, The Agile Architecture Revolution (John Wiley & Sons, 2013), sets the stage for Mr. Bloomberg’s groundbreaking Agile Architecture vision.

Mr. Bloomberg is perhaps best known for his twelve years at ZapThink, where he created and delivered the Licensed ZapThink Architect (LZA) SOA course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, the leading SOA advisory and analysis firm, which was acquired by Dovel Technologies in 2011. He now runs the successor to the LZA program, the Bloomberg Agile Architecture Course, around the world.

Mr. Bloomberg is a frequent conference speaker and prolific writer. He has published over 500 articles, spoken at over 300 conferences, Webinars, and other events, and has been quoted in the press over 1,400 times as the leading expert on agile approaches to architecture in the enterprise.

Mr. Bloomberg’s previous book, Service Orient or Be Doomed! How Service Orientation Will Change Your Business (John Wiley & Sons, 2006, coauthored with Ron Schmelzer), is recognized as the leading business book on Service Orientation. He also co-authored the books XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996).

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting).

@ThingsExpo Stories
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, will provide a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services ...
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), will provide an overview of various initiatives to certifiy the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldw...
SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
"delaPlex is a software development company. We do team-based outsourcing development," explained Mark Rivers, COO and Co-founder of delaPlex Software, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
In his keynote at @ThingsExpo, Chris Matthieu, Director of IoT Engineering at Citrix and co-founder and CTO of Octoblu, focused on building an IoT platform and company. He provided a behind-the-scenes look at Octoblu’s platform, business, and pivots along the way (including the Citrix acquisition of Octoblu).
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
SYS-CON Events announced today that HTBase will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. HTBase (Gartner 2016 Cool Vendor) delivers a Composable IT infrastructure solution architected for agility and increased efficiency. It turns compute, storage, and fabric into fluid pools of resources that are easily composed and re-composed to meet each application’s needs. With HTBase, companies can quickly prov...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...
SYS-CON Events announced today that Cloud Academy will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud computing technologies. Ge...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buyers...
SYS-CON Events announced today that Infranics will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Since 2000, Infranics has developed SysMaster Suite, which is required for the stable and efficient management of ICT infrastructure. The ICT management solution developed and provided by Infranics continues to add intelligence to the ICT infrastructure through the IMC (Infra Management Cycle) based on mathemat...
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 20th Cloud Expo, which will take place on June 6-8, 2017 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 add...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
SYS-CON Events announced today that SD Times | BZ Media has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. BZ Media LLC is a high-tech media company that produces technical conferences and expositions, and publishes a magazine, newsletters and websites in the software development, SharePoint, mobile development and commercial UAV markets.
Now that the world has connected “things,” we need to build these devices as truly intelligent in order to create instantaneous and precise results. This means you have to do as much of the processing at the point of entry as you can: at the edge. The killer use cases for IoT are becoming manifest through AI engines on edge devices. An autonomous car has this dual edge/cloud analytics model, producing precise, real-time results. In his session at @ThingsExpo, John Crupi, Vice President and Eng...
There are 66 million network cameras capturing terabytes of data. How did factories in Japan improve physical security at the facilities and improve employee productivity? Edge Computing reduces possible kilobytes of data collected per second to only a few kilobytes of data transmitted to the public cloud every day. Data is aggregated and analyzed close to sensors so only intelligent results need to be transmitted to the cloud. Non-essential data is recycled to optimize storage.