Agile Computing Authors: Yeshim Deniz, Pat Romanski, Liz McMillan, Elizabeth White, Andy Thurai

Related Topics: Agile Computing, @CloudExpo, @DXWorldExpo

Agile Computing: Blog Feed Post

Reinventing Integration Connectors By @SnapLogic | @CloudExpo [#BigData]

Connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI

SnapLogic: Reinventing Integration Connectors

Connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI back in the 1990s. According to Wikipedia, connectors (also called adapters) “are programs that know how to interact with an underlying business application. The adapter performs two-way communication, performing requests from the hub against the application, and notifying the hub when an event of interest occurs in the application (a new record inserted, a transaction completed, etc.).

“Adapters can be specific to an application (e. g., built against the application vendor’s client libraries) or specific to a class of applications (e. g., can interact with any application through a standard communication protocol).”

The problem with traditional connectors are that they are tightly coupled – any change in the data format or interface requirements for either end of any interaction would require an update of the connector, at the risk of a failed interaction. They can also become a single point of failure, a bottleneck that limits scalability and introduces brittleness to the application environment.

The rise of Service-Oriented Architecture (SOA) was in part intended to resolve the limitations of such traditional connectors. By abstracting interfaces to software as Web Services, the theory went, interactions could now be loosely coupled. It was possible to change the underlying software, either on the consumer or provider side, without breaking the interaction, as long as both endpoints conformed to the Web Services interface contract.

Web Services, however, largely fell short in delivering the loose coupling enterprises required to support truly flexible integration. Furthermore, Web Services were really little more than contracted interfaces, and still required an underlying software implementation.

In the 2000s this implementation typically became part of the Enterprise Service Bus (ESB) – a traditional piece of integration middleware exposing connectors much as the earlier generation of EAI middleware did, only now supporting the XML-based Web Services standards.

Then in the late 2000s, cloud computing came on the scene, and enterprises realized that they needed a more lightweight, decentralized approach to integration that didn’t have the limitations of the older ESB technology. These priorities led to the rise of Representational State Transfer (REST) – an architectural style originally intended to build hypermedia applications that generalized the behavior of the World Wide Web, but which soon became a lightweight, HTTP-based approach for building application programming interfaces (APIs).

However, because REST is based upon HTTP, it doesn’t deal well with data formats. After all, you expect standard Internet Media Types when you navigate the Web – HTML, JavaScript, PDF, and the like. To fill this void, the JavaScript Object Notation (JSON) combines the hierarchical structure of XML with the loose typing of JavaScript into a lightweight, web-friendly data format notation that doesn’t require predefined data schemas.

In fact, it’s possible to change a JSON file’s data structure on the fly – an advantage that adds flexibility to be sure, but throws a wrench into connectors or Web Services or any other traditional integration tool that expects data to conform to a particular schema.

Addressing the Challenges of Loose Coupling

SnapLogic is one of the new crop of vendors who are addressing this rather tangled web of strengths and weaknesses of various integration technologies. SnapLogic has implemented a new generation of connector they call Snaps that deal with the challenge of understanding endpoints – in particular, how to deal with changing APIs and applications while avoiding failed interactions.

Snaps are intelligent software components that provide design-time introspection. In other words, schemas and other metadata relevant to the interaction do not need to be known before the integration is set up. Instead, the Snap automatically gathers such metadata, automating the configuration of the integration. In situations where semantic mapping is required, SnapLogic automatically configures the Mapper Snap to facilitate the mapping process by conducting a fuzzy, “best guess” search across input and output fields.

Developers build even complex, multi-step integrations by assembling Snaps into Pipelines simply by snapping them together in a loosely coupled, iterative fashion, as shown in the figure below. Because SnapLogic is REST-based, each Pipeline is abstracted as a RESTful resource, which makes it addressable with a URL just like any other RESTful resource. REST calls can also trigger or schedule Pipelines, as they naturally support asynchronous behavior.


Pipeline illustrating Mapper Snaps and native JSON manipulation (Source: SnapLogic)

Furthermore, SnapLogic is a JSON-centric platform. SnapLogic’s support for JSON documents allows Snap endpoints to directly consume hierarchical data in native format and send them on to downstream Snaps in a Pipeline. As a result, there is no requirement to flatten data into records or to turn a JSON document into a string or BLOB (binary) type.

Traditional connectors often perform data transformations, but they typically require strict, rigid data mapping between endpoints. SnapLogic, in contrast, treats all data as natively hierarchical documents, an architectural decision that provides greater flexibility than traditional connectors.

In particular, Snaps allow developers to reach into these documents at any point. There is also no need to transform the document to flatten out the data. Snaps can also rearrange the data as needed.

This JSON-based document model is a superset of the relational model, so it easily works with relational data as well. However, in the case of relational data, schema changes can wreak havoc for traditional data integration – even changes as simple as adding a column to a table. Such changes, or even more complex examples, are no problem for SnapLogic.

The world of integration connectors will never be the same. True, Snaps are still essentially connectors, but by combining RESTful APIs, native JSON document centricity, and design time metadata introspection, SnapLogic has resolved the knottiest coupling issues inherent in all earlier generation connector technologies. At this point, there’s no reason to integrate any other way.

SnapLogic is an Intellyx client. At the time of writing, no other organizations mentioned in this article are Intellyx clients. Intellyx retains full editorial control over the content of this article.

Read the original blog entry...

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

IoT & Smart Cities Stories
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Cloud computing delivers on-demand resources that provide businesses with flexibility and cost-savings. The challenge in moving workloads to the cloud has been the cost and complexity of ensuring the initial and ongoing security and regulatory (PCI, HIPAA, FFIEC) compliance across private and public clouds. Manual security compliance is slow, prone to human error, and represents over 50% of the cost of managing cloud applications. Determining how to automate cloud security compliance is critical...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound e...
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
As IoT continues to increase momentum, so does the associated risk. Secure Device Lifecycle Management (DLM) is ranked as one of the most important technology areas of IoT. Driving this trend is the realization that secure support for IoT devices provides companies the ability to deliver high-quality, reliable, secure offerings faster, create new revenue streams, and reduce support costs, all while building a competitive advantage in their markets. In this session, we will use customer use cases...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...