Quite a few systems – business applications large and small – do not have support for WebHooks. That is: they do not offer the ability to register a URL as HTTP endpoint (WebHook) that the system will send requests to that inform the endpoint about relevant events. However, there are many applications that would like to receive such WebHook calls.
There are integration platforms that can poll the source systems in various ways and from the polling result send something akin to the webhook call. That works fine – in general. However, on some iPaaS platforms, creating such a WebSinker – a polling process to produce the WebHook call – can be difficult or running it can be expensive (depending on how costs of using the iPaaS platform are calculated).
Yesterday for example I was discussing the Make iPaaS solution that needs to run integration scenarios that retrieve data from AFAS (a Dutch ERP application). AFAS does not support webhooks and the iPaaS scenario needs to poll for any changed data. However, this platform charges by the operation performed – including operations that are performed for polling. This runs up quite a bill.
It seems that a generic WebSinker would be useful. This is a component that:
- connects to a source system
- polls specific tables/APIs/file locations/,… for relevant changes (is configured in a way that is specific for the source system for polling; remembers if necessary the last poll result to only send webhook triggers for new events)
- filters the changes based on situation specific conditions
- sends relevant details from the remaining changes to the WebHook endpoint on the receiving system, which can be an iPaaS integration scenario
The WebSinker is meant for lightweight integration environments. It should be easy to install and configure and operate. WebSinkers are created for specific source systems – with help from people who these sources systems well and understand how relevant changes should be polled for.
I used Apache NiFi to pull from multiple systems and its really good, data flows that persist data no matter if apache nifi machine/cluster is down, and with +200 get/put/transform components theres no need to code. I can’t suggest anything else as good and fast learning as Apache NiFi, only thing is it needs manual deployment and management as its still not available as SaaS service, but doing that well using containers, plug in some log agents to copy export logs and you have perfect pull system. its also possible to create http endpoint with httpserver but not recommended for critical 24/7 can’t loose any single transaction scenario.