Agile Computing Authors: Liz McMillan, Elizabeth White, Rene Buest, Stackify Blog, Greg Pierce

Related Topics: Agile Computing, @CloudExpo, @DXWorldExpo

Agile Computing: Blog Feed Post

Reinventing Integration Connectors By @SnapLogic | @CloudExpo [#BigData]

Connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI

SnapLogic: Reinventing Integration Connectors

Connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI back in the 1990s. According to Wikipedia, connectors (also called adapters) “are programs that know how to interact with an underlying business application. The adapter performs two-way communication, performing requests from the hub against the application, and notifying the hub when an event of interest occurs in the application (a new record inserted, a transaction completed, etc.).

“Adapters can be specific to an application (e. g., built against the application vendor’s client libraries) or specific to a class of applications (e. g., can interact with any application through a standard communication protocol).”

The problem with traditional connectors are that they are tightly coupled – any change in the data format or interface requirements for either end of any interaction would require an update of the connector, at the risk of a failed interaction. They can also become a single point of failure, a bottleneck that limits scalability and introduces brittleness to the application environment.

The rise of Service-Oriented Architecture (SOA) was in part intended to resolve the limitations of such traditional connectors. By abstracting interfaces to software as Web Services, the theory went, interactions could now be loosely coupled. It was possible to change the underlying software, either on the consumer or provider side, without breaking the interaction, as long as both endpoints conformed to the Web Services interface contract.

Web Services, however, largely fell short in delivering the loose coupling enterprises required to support truly flexible integration. Furthermore, Web Services were really little more than contracted interfaces, and still required an underlying software implementation.

In the 2000s this implementation typically became part of the Enterprise Service Bus (ESB) – a traditional piece of integration middleware exposing connectors much as the earlier generation of EAI middleware did, only now supporting the XML-based Web Services standards.

Then in the late 2000s, cloud computing came on the scene, and enterprises realized that they needed a more lightweight, decentralized approach to integration that didn’t have the limitations of the older ESB technology. These priorities led to the rise of Representational State Transfer (REST) – an architectural style originally intended to build hypermedia applications that generalized the behavior of the World Wide Web, but which soon became a lightweight, HTTP-based approach for building application programming interfaces (APIs).

However, because REST is based upon HTTP, it doesn’t deal well with data formats. After all, you expect standard Internet Media Types when you navigate the Web – HTML, JavaScript, PDF, and the like. To fill this void, the JavaScript Object Notation (JSON) combines the hierarchical structure of XML with the loose typing of JavaScript into a lightweight, web-friendly data format notation that doesn’t require predefined data schemas.

In fact, it’s possible to change a JSON file’s data structure on the fly – an advantage that adds flexibility to be sure, but throws a wrench into connectors or Web Services or any other traditional integration tool that expects data to conform to a particular schema.

Addressing the Challenges of Loose Coupling

SnapLogic is one of the new crop of vendors who are addressing this rather tangled web of strengths and weaknesses of various integration technologies. SnapLogic has implemented a new generation of connector they call Snaps that deal with the challenge of understanding endpoints – in particular, how to deal with changing APIs and applications while avoiding failed interactions.

Snaps are intelligent software components that provide design-time introspection. In other words, schemas and other metadata relevant to the interaction do not need to be known before the integration is set up. Instead, the Snap automatically gathers such metadata, automating the configuration of the integration. In situations where semantic mapping is required, SnapLogic automatically configures the Mapper Snap to facilitate the mapping process by conducting a fuzzy, “best guess” search across input and output fields.

Developers build even complex, multi-step integrations by assembling Snaps into Pipelines simply by snapping them together in a loosely coupled, iterative fashion, as shown in the figure below. Because SnapLogic is REST-based, each Pipeline is abstracted as a RESTful resource, which makes it addressable with a URL just like any other RESTful resource. REST calls can also trigger or schedule Pipelines, as they naturally support asynchronous behavior.


Pipeline illustrating Mapper Snaps and native JSON manipulation (Source: SnapLogic)

Furthermore, SnapLogic is a JSON-centric platform. SnapLogic’s support for JSON documents allows Snap endpoints to directly consume hierarchical data in native format and send them on to downstream Snaps in a Pipeline. As a result, there is no requirement to flatten data into records or to turn a JSON document into a string or BLOB (binary) type.

Traditional connectors often perform data transformations, but they typically require strict, rigid data mapping between endpoints. SnapLogic, in contrast, treats all data as natively hierarchical documents, an architectural decision that provides greater flexibility than traditional connectors.

In particular, Snaps allow developers to reach into these documents at any point. There is also no need to transform the document to flatten out the data. Snaps can also rearrange the data as needed.

This JSON-based document model is a superset of the relational model, so it easily works with relational data as well. However, in the case of relational data, schema changes can wreak havoc for traditional data integration – even changes as simple as adding a column to a table. Such changes, or even more complex examples, are no problem for SnapLogic.

The world of integration connectors will never be the same. True, Snaps are still essentially connectors, but by combining RESTful APIs, native JSON document centricity, and design time metadata introspection, SnapLogic has resolved the knottiest coupling issues inherent in all earlier generation connector technologies. At this point, there’s no reason to integrate any other way.

SnapLogic is an Intellyx client. At the time of writing, no other organizations mentioned in this article are Intellyx clients. Intellyx retains full editorial control over the content of this article.

Read the original blog entry...

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

@ThingsExpo Stories
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
DXWorldEXPO | CloudEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
DXWorldEXPO LLC announced today that ICOHOLDER named "Media Sponsor" of Miami Blockchain Event by FinTechEXPO. ICOHOLDER give you detailed information and help the community to invest in the trusty projects. Miami Blockchain Event by FinTechEXPO has opened its Call for Papers. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Miami Blockchain Event by FinTechEXPO also offers s...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
Predicting the future has never been more challenging - not because of the lack of data but because of the flood of ungoverned and risk laden information. Microsoft states that 2.5 exabytes of data are created every day. Expectations and reliance on data are being pushed to the limits, as demands around hybrid options continue to grow.
As IoT continues to increase momentum, so does the associated risk. Secure Device Lifecycle Management (DLM) is ranked as one of the most important technology areas of IoT. Driving this trend is the realization that secure support for IoT devices provides companies the ability to deliver high-quality, reliable, secure offerings faster, create new revenue streams, and reduce support costs, all while building a competitive advantage in their markets. In this session, we will use customer use cases...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
The IoT Will Grow: In what might be the most obvious prediction of the decade, the IoT will continue to expand next year, with more and more devices coming online every single day. What isn’t so obvious about this prediction: where that growth will occur. The retail, healthcare, and industrial/supply chain industries will likely see the greatest growth. Forrester Research has predicted the IoT will become “the backbone” of customer value as it continues to grow. It is no surprise that retail is ...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
DXWorldEXPO LLC announced today that "Miami Blockchain Event by FinTechEXPO" has announced that its Call for Papers is now open. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expe...