Wow, that sounds awful...
So maybe it's not even related to other modules, but some... cache problem somewhere? Be it browser or server-side? A Heisenbug.
It remains a complete mystery. I started enabling other modules on my clean install and boom - off it went again. Of course, removing those modules didn’t make any difference so I couldn’t...
Thanks Wilbur for solving this mystery. The secret is in the settings for the Exposed Form.