I’m trying to output a ‘success’ file when a job completes, but I’m having an issue.
My main calls 2 functions, the first one calls a function that makes an API call to get a list of account numbers, and then fires off an API call for each one, process the data and then writes each file to disc. The second function copies an empty txt file called ‘success’ to the folder. What appears to be happening is as soon as the first function is run, the second function is executed and the success file is written and the job ends. I’ve experimented with using delay and calling force on the 2nd function, but this is still being called as soon as the first function is called.
The reason I need the success file is that the files are being written to S3 and our DBA needs to set up an event to automatically load the data into Redshift, but he needs a trigger, and as I don’t know what order the account numbers are returned in, or when new accounts are created, I need a static file that I know will always be the last file written.
As the number of API calls being made and the response times can vary, I don’t want to rely on a timed method. Is there a way of achieving this where the second function isn’t executed until all the dependent functions of the first one have completed?