[DX][UX] Form API: Make it so that required fields that are hidden via #states are only validated when visible.

TL;DR

Consider the case where you use a checkbox to conditionally set another form element to be required or not, depending on the state of the checkbox; then:

• The red "required" asterisk is properly added/removed from the form element, depending on whether the checkbox is ticked/unticked BUT this seems to only be "cosmetic", because in actuality...
• If the form element is set to be required via '#required' => TRUE, then the form validation will kick in, no matter if the required attribute was removed via #states (by unticking the checkbox)
• If the form element is set to NOT be required via '#required' => FALSE, then the form validation will NOT kick in, no matter if the required attribute was added via #states (by ticking the checkbox)

---

Describe your issue or idea

Every time that we need to conditionally present a required field in a form, we are forced to either write specific validation functions for that form, or simply settle with making these fields optional, or both. Reason: the way this currently works is that any required field that is hidden via #states is still validated (although not used).

Bad UX: Take #2731 for example. The user enables a feature by ticking a checkbox, which reveals more fields via #states. There is no indication that any field is required, but when the form is submitted, there is a validation error. The user now needs to go through the form again in order to fill in any required fields.

Bad DX: The way we have coded that part of the form is that we have left a mandatory field as optional. We have then added custom validation upon submit. Had we made the field required, it would go through the validation process and throw errors even when not used (when hidden via #states). While filing a PR for #2731, I discovered a workaround where the field can still be rendered as required using #states, but in order to not throw validation errors, we have to still use the custom validation function.

More bad DX: also while filing a PR for #2731, I discovered that when it comes to machine_name type of fields, the workaround where the field can still be rendered as required by using #states no longer works (validation errors get triggered). In order to not throw validation errors for #states-hidden machine names, you also have to also set '#required' => FALSE. So the code looks like this:

'#states' => array(
  'required' => array(
    'input[name="checkbox"]' => array('checked' => TRUE),
  ),
),
// This line is still required in order to not throw form validation errors
// when the checkbox is not ticked.
'#required' => FALSE,

Even more bad DX: this silly required/not-required code above only works half way. It only handles rendering the field as required (red asterisk). In order to have proper validation (only when the checkbox is ticked), you still need to use a custom validation function.

Steps to reproduce (if reporting a bug)

See #2731 for an actual use case. That issue will most likely be addressed sooner than this one here. In that case, try the following...

1. Create a simple form with a checkbox and a text field.
2. Have the text field be required.
3. Also have the text field hidden via #states based on whether the checkbox is ticked or not.
4. Have the checkbox be ticked off by default (so that the required text field is hidden).
5. Submit the form.

Actual behavior (if reporting a bug)

There is a validation error, telling you that the text field (which is hidden) is required.

Expected behavior (if reporting a bug)

There should be no validation errors if the required field is hidden via #states. Validation errors should only be triggered when the field is visible.