Not GP or specifically Airflow user; but my approach is to have a fixed job graph, and unnecessary jobs immediately succeed. And indeed, jobs are external executables, with all the skip/no skip logic executed therein.
If nothing else, it makes it easy to understand what actually happened and when - just look at job logs.
I’m working on similar system. My plan is to have multiple terminal states for the tasks:
Closed - Passed
Closed - Failed
Closed - Waived
When you hit that Waived state, it should include a note explaining why it was waived. This could be “parent transaction dropped below threshold amount, so we don’t need this control” or “Executive X signed off on it”.
I’m not sure about the auto-skip thing you propose, just from a UX perspective. I don’t want my task list cluttered up with unnecessary things. Still, I am struggling with precisely where to store the business logic about which tasks are needed when. I’m leaning towards implementing that in a reporting layer. Validation would happen in the background and raise warnings, rather than hard stopping people.
The theory there is that the people doing the work generally know what’s needed better than the system does. Thus the system just provides gentle reminders about the typical case, which users can make the choice to suppress.
I think of jobs rather as of prerequisites. If a prerequisite is somehow automatically satisfied (dunno, only back up on Mondays, and today is Tuesday) then it succeeds immediately. There is no "skipping". Wfm.
I find embedding logic into DSLs usually quite painful and less portable than having a static job graph and all the logic firmly in my own code.
If nothing else, it makes it easy to understand what actually happened and when - just look at job logs.