Ran into this while working on https://forum.backdropcms.org/forum/modal-dialog-styling.
The dialogClass property does work as @argiepiano suggested, but the documentation at
https://api.jqueryui.com/dialog/
pointed out by @tkcent says that it's now deprecated in favor of the "classes" property. That one (classes property) doesn't work though. Not a big deal as I have something working now but I am just wondering about it.
Good point, @leeksoup! Can you please open an issue in the main Backdrop queue about this? We'll need to fix this in core.