WebSinker–generic event detection for unhooked systems to trigger WebHook

Lucas Jellema 1

Quite a few systems – business applications large and small – do not have support for WebHooks. That is: they do not offer the ability to register a URL as HTTP endpoint (WebHook) that the system will send requests to that inform the endpoint about relevant events. However, there are many applications that would like to receive such WebHook calls.

There are integration platforms that can poll the source systems in various ways and from the polling result send something akin to the webhook call. That works fine – in general. However, on some iPaaS platforms, creating such a WebSinker – a polling process to produce the WebHook call – can be difficult or running it can be expensive (depending on how costs of using the iPaaS platform are calculated).

Yesterday for example I was discussing the Make iPaaS solution that needs to run integration scenarios that retrieve data from AFAS (a Dutch ERP application). AFAS does not support webhooks and the iPaaS scenario needs to poll for any changed data. However, this platform charges by the operation performed – including operations that are performed for polling. This runs up quite a bill.

It seems that a generic WebSinker would be useful. This is a component that:

  • connects to a source system
  • polls specific tables/APIs/file locations/,… for relevant changes (is configured in a way that is specific for the source system for polling; remembers if necessary the last poll result to only send webhook triggers for new events)
  • filters the changes based on situation specific conditions
  • sends relevant details from the remaining changes to the WebHook endpoint on the receiving system, which can be an iPaaS integration scenario

The WebSinker is meant for lightweight integration environments. It should be easy to install and configure and operate. WebSinkers are created for specific source systems – with help from people who these sources systems well and understand how relevant changes should be polled for.

image

One thought on “WebSinker–generic event detection for unhooked systems to trigger WebHook

  1. I used Apache NiFi to pull from multiple systems and its really good, data flows that persist data no matter if apache nifi machine/cluster is down, and with +200 get/put/transform components theres no need to code. I can’t suggest anything else as good and fast learning as Apache NiFi, only thing is it needs manual deployment and management as its still not available as SaaS service, but doing that well using containers, plug in some log agents to copy export logs and you have perfect pull system. its also possible to create http endpoint with httpserver but not recommended for critical 24/7 can’t loose any single transaction scenario.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Next Post

Apache NiFi: Automating tasks using NiPyAPI

Apache NiFi has a powerful web-based interface which provides a seamless experience between design, control, feedback, and monitoring. Sometimes however, you want to automate tasks instead of doing them manually using the UI. This does not only allow you to perform the tasks a lot quicker but it also helps […]
%d bloggers like this: