Short answer: it's difficult. You can change the url of the project server in installer.settings.json but you lose the ability to install or update modules served by Backdrop's project server.

You can specify custom URLs for "non-official" contrib/custom modules, by adding the project status url property in the .info file of the module. That should allow you to keep Backdrop core and "official" contrib updated from https://github.com/backdrop-contrib as per usual, while the custom module can get updates from elsewhere. I've never personally done it, so untested. But it's been there since D7 (perhaps even before).

Backdrop CMS contributed projects

Join the group at https://github.com/backdrop-ops/contrib to add your own! - Backdrop CMS contributed projects

 I don't see that property mentioned in https://docs.backdropcms.org/creating-modules#info-files but it is mentioned in https://www.drupal.org/node/542202 (last in the list of .info properties).

klonos:

project status url (Only used for custom modules not submitted to drupal.org)

Allows module maintainers to define a URL to check for updates to their module using the Update status module. No module released on drupal.org should define this parameter. URL should point to an XML feed that accepts requests in the form of http://my.domain.com/projects/{project}/{core}. In that example, project status url should be set to http://my.domain.com/projects.

klonos:  The logic seems to be there. See: