Guarantee behavior of transmuting Option::<T>::None subject to NPO

[joshlf]

In #115333, we added a guarantee that transmuting from [0u8; N] to Option<P> is sound where P is a pointer type subject to the null pointer optimization (NPO). It would be useful to be able to guarantee the inverse - that a None::<P> value can be transmutes to an array and that will yield [0u8; N].

Closes #117591

[rustbot]

r? @tgross35

rustbot has assigned @tgross35.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[jieyouxu]

r? libs-api (since this is making a behavior guaranteed)

[RalfJung]

Cc @rust-lang/opsem @rust-lang/lang

This will need a t-lang FCP. @joshlf would be good to write a summary for t-lang, knowing that they will lack all the context we have here. :)

[RalfJung]

Looking at the PR itself, the change LGTM.

[joshlf]

This will need a t-lang FCP. @joshlf would be good to write a summary for t-lang, knowing that they will lack all the context we have here. :)

Currently, zerocopy has the ability to validate at runtime whether a [u8; N] contains a valid Option<&T>. In particular, the NPO guarantees that, if the [u8; N] contains all 0u8 bytes, then it constitutes a valid Option<&T>. We do the same for other types subject to NPO: see e.g. this set of impls.

This works via our TryFromBytes trait. Under the hood, #[derive(TryFromBytes)] synthesizes a predicate over &[u8] - in particular, the TryFromBytes::is_bit_valid method takes a &[u8] (it's actually a different type, but it has the same bit validity and alignment as &[u8]) and determines whether it contains a valid Self.

Eventually, we'd like to not only support going from &[u8] to &T, but from &T to &U for a broader range of pairs of types. We can already support &T -> &U if we can soundly convert &T -> &[u8], but some cases don't satisfy that requirement. For example, consider:

#[repr(C)]
struct T {
    a: u8,
    b: u16,
}

#[repr(C)]
struct U {
    a: bool,
    b: u16,
}

Since T has inter-field padding, it's unsound to convert a &T to a &[u8]. We'd still like to be able to support the &T -> &U case, though. This requires changing TryFromBytes::is_bit_valid to no longer take a &[u8] argument, and to instead take something that we call AsInitialized. An AsInitialized<T> has the invariant that it has initialized bytes wherever T has initialized bytes, but it may be uninitializated in other byte positions. This will permit us to go from &T to &AsInitialized<U>, and have TryFromBytes::is_bit_valid take this &AsInitialized<U> as an argument.

That gets us to this PR: In order to make this change, for each type that currently implements TryFromBytes, we need a guarantee about which bytes are always initialized. Currently, if P is a type subject to NPO, we have no guarantee that an instance of Option<P> has all its bytes initialized. While getting this guarantee in full is blocked on ptr-to-int semantics, this PR is intended as a first step that will at least handle the None case.

[joshtriplett]

This seems consistent with how we already support using Option<&T> rather than *const T in FFI calls, in both directions. I'm surprised we don't already guarantee this; people's code would be broken if we did otherwise.

[joshtriplett]

Per the above, we already have to guarantee this and there'd be widespread breakage if we ever failed to uphold it. So, let's write it down.

@rfcbot merge

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[traviscross]

@rfcbot reviewed

[scottmcm]

Under the restriction (that's in the docs) of the very specific list of NPO types, 100% agreed.

@rfcbot reviewed

(Just wanted to double-check that we weren't accidentally guaranteeing anything for Option<char> or Option<CustomEnumOneOrTwo> or similar.)

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[traviscross]

We talked about this in the lang call today, were happy to see it go forward, and it's now in FCP.

Let's cc @rust-lang/spec, to think about the interplay between the Reference and the library documentation when making language guarantees like this.