Consider references and unions potentially inhabited during privacy-respecting inhabitedness checks
It isn't settled exactly how references to uninhabited types and unions of uninhabited types should act, but we should be more conservative here, as it's likely it will be permitted to soundly have values of such types.
This will also be more important in light of the changes at https://github.com/rust-lang/rust/pull/54125.
cc @RalfJung