Jet Scheduler issues emailing when using Office 365 MFA

We're having a weird issue with Jet Scheduler. We use multi-factor authentication with Office 365. When we enter the SMTP settings with an app password the test emails just fine. The issue is that after a few days it just stops emailing out the scheduled reports.