Nifi is fantastically good at one thing, which is dataflow. Where you've got data coming in at point A, but you need it at point B, and for some reason can't convince either A or B to connect directly.
It's not a message bus, nor is it a data processing framework, nor a scheduler, nor an ETL tool. If you try to use it for one of those you're in for a bad time.
What you're describing sounds like you might need a message bus (think ZeroMQ, Kafka, etc). Assuming you're writing the software yourself and want to connect it together.
could you elaborate on the difference between what op is asking for and what nifi/airflow does? to me the use case of moving data through a couple of different services could be solved by a message bus OR nifi.
Nifi is designed to handle the problems that crop up when two systems can't talk directly to each other. It puts a buffer between them to allow one process to keep sending data when the other isn't quite fast enough, it can do some basic transformation when the data isn't quite in the right format, etc.
However if you just need something to send messages then you're better off using a tool that does just that, you don't need the overhead of a system that can connect arbitrary applications that talk in incompatible protocols you just need a single protocol that allows your applications to send each other data.
In their docs Nifi calls itself a dataflow tool and calls dataflow a necessary evil. It's the band-aid you need when you've got a mismatch between the way data is generated and the way it is consumed. It would be insane to deliberately create such a mismatch just to use Nifi.
Can you disambiguate that further? I think you mean that an ETL tool would handle the external interface work whereas Nifi is dealing with data that is somewhat inside your control. Is that an OK description?
Often in ETL the transformation part becomes a goal of itself, in Nifi there are tools to transform data, but the Extract and Load parts work best, it has interfaces for quite a lot of different systems.
And maybe it's just me but everything I've seen being called ETL handled data in batches, in which case you really want support for scheduling, error handling and retries, which Nifi does not really have (there is a 'retry' function somewhere, but I found it confusing and it only seems to work for a single data item, not a good idea when you've got thousands of them). I much prefer Airflow for those scenarios.
It's not a message bus, nor is it a data processing framework, nor a scheduler, nor an ETL tool. If you try to use it for one of those you're in for a bad time.
What you're describing sounds like you might need a message bus (think ZeroMQ, Kafka, etc). Assuming you're writing the software yourself and want to connect it together.