I think found the culprit, it's App Nap. That sorry excuse for a power saving feature has been around since Maverick, so I didn't check this until now, based on observations that this happens only under El Capitan (perhaps App Nap behavior changed subtly? I don't know of major changes but who knows...