This issue is in response to this comment (also here).
The basic idea is to add a extensible settings page for contextual links (e.g., modifiable via hook_form_FORM_ID_alter()) that provide a place to turn on/off and/or modify different aspects of Contextual Links.
A suggested place would be admin/config/administration/contextual-links, which seems to fit, and would also help provide a reason for admin/config/administration to have a submenu (currently there's only one submenu item, admin/config/administration/admin-bar).
An example use case—which also creates the setting page—is PR https://github.com/backdrop/backdrop/pull/3790, which creates layout and theme-related contextual links, in response to https://github.com/backdrop/backdrop-issues/issues/2626. (At this point, it's looking like a different PR that doesn't create contextual links will solve that issue, but this is an example of the type of setting that would belong there.)
It would also be a good home for the settings in Layout Contextual Links module (which is the contrib version of PR https://github.com/backdrop/backdrop/pull/3790).
Other possible use cases that have been mentioned in discussions include finer-grained controls on which contextual links to display, to allow site-builders to turn on/off contextual links selectively, or to optionally add additional information to contextual links (e.g., the title of the thing being linked to).
At this point (assuming that PR https://github.com/backdrop/backdrop/pull/3790 doesn't go forward), there's not yet a specific proposal for something to include in this settings page, but adding the issue here (per @BWPanda's comment) so that it doesn't get lost, and perhaps future proposals of modifications to contextual links that need their own settings can extend or link to this issue and resurrect the concept.
Recent comments
Amilenkov's suggestions are spot on. As for the greyed out Menu module, that's because there must be another module that depends on Menu. Core will gray out modules that cannot be disabled (e.g...
menu partially working
Welcome to the Backdrop community! My first suspicion would be access to the destination page. Backdrop checks access when building the menu, so a link can be enabled but still be hidden...
menu partially working
here's the view when not logged in (ie with no 'sustainable computing' visible but is visible when i login as admin:
menu partially working