I thought I'd post this here in case it may help someone else. It took a while to figure out.

Backdrop's Form API provides the element managed_file, which allows you to add a widget to your forms to upload new managed files. The documentation for this element is good, except that it doesn't contain much about the property #browser_view, other than saying that it allows developers to indicate a View to use in the file browser dialog. 

So, my use case was: I needed to provide a file browser that allowed the user to pick PDF documents from the managed files (not images). These are the steps:

1. Create your form and include the managed_file element. For example:

  $form['certificate_template_file'] = array(
    '#type' => 'managed_file',
    '#title' => t('Template PDF file'),
    '#description' => t('The template PDF file to be used for this certificate.'),
    '#default_value' => $certificate_template->certificate_template_file,
    '#upload_location' => 'public://certificate_templates',
    '#upload_validators' => array(
      'file_validate_extensions' => array('pdf'),
      'file_validate_size' => array(5 * 1024 * 1024),
    ),
    '#browser_view' => 'pdf_doc_library',
  );

The above indicates that the View "pdf_doc_library" will be used for the file browser.

2. Create your View. Show: Files. I used a Grid display.

3. This is important: in the Grid settings, add the Row class "file-browser-file". Backdrop's file browser will look for that class to select the file ID when clicked.

4. Notice the hidden fields Path and Name (image above). The Path field MUST have the option "Use full URL instead of file storage URI" clicked. 

The other important part: rewrite the File ID fields as follows:

 The important part here is to provide a data-file-url, AND a data-fid with the data from the hidden fields. The rest you can customize as you wish.

5. Then a bit of styling added to a CSS file that gets loaded with the form, such as:

 

.file-browser-file:hover {
  background-color: #e5f3ff;
}

.file-browser-selected {
  background-color: #cce8ff;
}

That's it. Then, in the form, when clicking "Select existing files" you'll get:

A minor annoyance: the dialog has the title "Select from Image Library". That title is, unfortunately, hard-coded in file_managed_file_browser_open(). Perhaps I'll create an issue for this. It'd be easy to pass the title within $form_state.