1 //! This is a densely packed error representation which is used on targets with
4 //! (Note that `bitpacked` vs `unpacked` here has no relationship to
5 //! `#[repr(packed)]`, it just refers to attempting to use any available bits in
6 //! a more clever manner than `rustc`'s default layout algorithm would).
8 //! Conceptually, it stores the same data as the "unpacked" equivalent we use on
9 //! other targets. Specifically, you can imagine it as an optimized version of
10 //! the following enum (which is roughly equivalent to what's stored by
11 //! `repr_unpacked::Repr`, e.g. `super::ErrorData<Box<Custom>>`):
13 //! ```ignore (exposition-only)
16 //! Simple(ErrorKind),
17 //! SimpleMessage(&'static SimpleMessage),
18 //! Custom(Box<Custom>),
22 //! However, it packs this data into a 64bit non-zero value.
24 //! This optimization not only allows `io::Error` to occupy a single pointer,
25 //! but improves `io::Result` as well, especially for situations like
26 //! `io::Result<()>` (which is now 64 bits) or `io::Result<u64>` (which is now
27 //! 128 bits), which are quite common.
30 //! Tagged values are 64 bits, with the 2 least significant bits used for the
31 //! tag. This means there are there are 4 "variants":
33 //! - **Tag 0b00**: The first variant is equivalent to
34 //! `ErrorData::SimpleMessage`, and holds a `&'static SimpleMessage` directly.
36 //! `SimpleMessage` has an alignment >= 4 (which is requested with
37 //! `#[repr(align)]` and checked statically at the bottom of this file), which
38 //! means every `&'static SimpleMessage` should have the both tag bits as 0,
39 //! meaning its tagged and untagged representation are equivalent.
41 //! This means we can skip tagging it, which is necessary as this variant can
42 //! be constructed from a `const fn`, which probably cannot tag pointers (or
43 //! at least it would be difficult).
45 //! - **Tag 0b01**: The other pointer variant holds the data for
46 //! `ErrorData::Custom` and the remaining 62 bits are used to store a
47 //! `Box<Custom>`. `Custom` also has alignment >= 4, so the bottom two bits
48 //! are free to use for the tag.
50 //! The only important thing to note is that `ptr::wrapping_add` and
51 //! `ptr::wrapping_sub` are used to tag the pointer, rather than bitwise
52 //! operations. This should preserve the pointer's provenance, which would
53 //! otherwise be lost.
55 //! - **Tag 0b10**: Holds the data for `ErrorData::Os(i32)`. We store the `i32`
56 //! in the pointer's most significant 32 bits, and don't use the bits `2..32`
57 //! for anything. Using the top 32 bits is just to let us easily recover the
58 //! `i32` code with the correct sign.
60 //! - **Tag 0b11**: Holds the data for `ErrorData::Simple(ErrorKind)`. This
61 //! stores the `ErrorKind` in the top 32 bits as well, although it doesn't
62 //! occupy nearly that many. Most of the bits are unused here, but it's not
63 //! like we need them for anything else yet.
65 //! # Use of `NonNull<()>`
67 //! Everything is stored in a `NonNull<()>`, which is odd, but actually serves a
70 //! Conceptually you might think of this more like:
72 //! ```ignore (exposition-only)
74 //! // holds integer (Simple/Os) variants, and
75 //! // provides access to the tag bits.
77 //! // Tag is 0, so this is stored untagged.
78 //! msg: &'static SimpleMessage,
79 //! // Tagged (offset) `Box<Custom>` pointer.
80 //! tagged_custom: NonNull<()>,
84 //! But there are a few problems with this:
86 //! 1. Union access is equivalent to a transmute, so this representation would
87 //! require we transmute between integers and pointers in at least one
88 //! direction, which may be UB (and even if not, it is likely harder for a
89 //! compiler to reason about than explicit ptr->int operations).
91 //! 2. Even if all fields of a union have a niche, the union itself doesn't,
92 //! although this may change in the future. This would make things like
93 //! `io::Result<()>` and `io::Result<usize>` larger, which defeats part of
94 //! the motivation of this bitpacking.
96 //! Storing everything in a `NonZeroUsize` (or some other integer) would be a
97 //! bit more traditional for pointer tagging, but it would lose provenance
98 //! information, couldn't be constructed from a `const fn`, and would probably
99 //! run into other issues as well.
101 //! The `NonNull<()>` seems like the only alternative, even if it's fairly odd
102 //! to use a pointer type to store something that may hold an integer, some of
105 use super::{Custom, ErrorData, ErrorKind, SimpleMessage};
106 use alloc::boxed::Box;
107 use core::mem::{align_of, size_of};
108 use core::ptr::NonNull;
110 // The 2 least-significant bits are used as tag.
111 const TAG_MASK: usize = 0b11;
112 const TAG_SIMPLE_MESSAGE: usize = 0b00;
113 const TAG_CUSTOM: usize = 0b01;
114 const TAG_OS: usize = 0b10;
115 const TAG_SIMPLE: usize = 0b11;
118 pub(super) struct Repr(NonNull<()>);
120 // All the types `Repr` stores internally are Send + Sync, and so is it.
121 unsafe impl Send for Repr {}
122 unsafe impl Sync for Repr {}
125 pub(super) fn new_custom(b: Box<Custom>) -> Self {
126 let p = Box::into_raw(b).cast::<u8>();
127 // Should only be possible if an allocator handed out a pointer with
129 debug_assert_eq!((p as usize & TAG_MASK), 0);
130 // Note: We know `TAG_CUSTOM <= size_of::<Custom>()` (static_assert at
131 // end of file), and both the start and end of the expression must be
132 // valid without address space wraparound due to `Box`'s semantics.
134 // This means it would be correct to implement this using `ptr::add`
135 // (rather than `ptr::wrapping_add`), but it's unclear this would give
136 // any benefit, so we just use `wrapping_add` instead.
137 let tagged = p.wrapping_add(TAG_CUSTOM).cast::<()>();
138 // Safety: `TAG_CUSTOM + p` is the same as `TAG_CUSTOM | p`,
139 // because `p`'s alignment means it isn't allowed to have any of the
140 // `TAG_BITS` set (you can verify that addition and bitwise-or are the
141 // same when the operands have no bits in common using a truth table).
143 // Then, `TAG_CUSTOM | p` is not zero, as that would require
144 // `TAG_CUSTOM` and `p` both be zero, and neither is (as `p` came from a
145 // box, and `TAG_CUSTOM` just... isn't zero -- it's `0b01`). Therefore,
146 // `TAG_CUSTOM + p` isn't zero and so `tagged` can't be, and the
147 // `new_unchecked` is safe.
148 let res = Self(unsafe { NonNull::new_unchecked(tagged) });
149 // quickly smoke-check we encoded the right thing (This generally will
150 // only run in libstd's tests, unless the user uses -Zbuild-std)
151 debug_assert!(matches!(res.data(), ErrorData::Custom(_)), "repr(custom) encoding failed");
156 pub(super) fn new_os(code: i32) -> Self {
157 let utagged = ((code as usize) << 32) | TAG_OS;
158 // Safety: `TAG_OS` is not zero, so the result of the `|` is not 0.
159 let res = Self(unsafe { NonNull::new_unchecked(utagged as *mut ()) });
160 // quickly smoke-check we encoded the right thing (This generally will
161 // only run in libstd's tests, unless the user uses -Zbuild-std)
163 matches!(res.data(), ErrorData::Os(c) if c == code),
164 "repr(os) encoding failed for {}",
171 pub(super) fn new_simple(kind: ErrorKind) -> Self {
172 let utagged = ((kind as usize) << 32) | TAG_SIMPLE;
173 // Safety: `TAG_SIMPLE` is not zero, so the result of the `|` is not 0.
174 let res = Self(unsafe { NonNull::new_unchecked(utagged as *mut ()) });
175 // quickly smoke-check we encoded the right thing (This generally will
176 // only run in libstd's tests, unless the user uses -Zbuild-std)
178 matches!(res.data(), ErrorData::Simple(k) if k == kind),
179 "repr(simple) encoding failed {:?}",
186 pub(super) const fn new_simple_message(m: &'static SimpleMessage) -> Self {
187 // Safety: References are never null.
188 Self(unsafe { NonNull::new_unchecked(m as *const _ as *mut ()) })
192 pub(super) fn data(&self) -> ErrorData<&Custom> {
193 // Safety: We're a Repr, decode_repr is fine.
194 unsafe { decode_repr(self.0, |c| &*c) }
198 pub(super) fn data_mut(&mut self) -> ErrorData<&mut Custom> {
199 // Safety: We're a Repr, decode_repr is fine.
200 unsafe { decode_repr(self.0, |c| &mut *c) }
204 pub(super) fn into_data(self) -> ErrorData<Box<Custom>> {
205 let this = core::mem::ManuallyDrop::new(self);
206 // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
207 // safe because we prevent double-drop using `ManuallyDrop`.
208 unsafe { decode_repr(this.0, |p| Box::from_raw(p)) }
215 // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
216 // safe because we're being dropped.
218 let _ = decode_repr(self.0, |p| Box::<Custom>::from_raw(p));
223 // Shared helper to decode a `Repr`'s internal pointer into an ErrorData.
225 // Safety: `ptr`'s bits should be encoded as described in the document at the
226 // top (it should `some_repr.0`)
228 unsafe fn decode_repr<C, F>(ptr: NonNull<()>, make_custom: F) -> ErrorData<C>
230 F: FnOnce(*mut Custom) -> C,
232 let bits = ptr.as_ptr() as usize;
233 match bits & TAG_MASK {
235 let code = ((bits as i64) >> 32) as i32;
239 let kind_bits = (bits >> 32) as u32;
240 let kind = kind_from_prim(kind_bits).unwrap_or_else(|| {
241 debug_assert!(false, "Invalid io::error::Repr bits: `Repr({:#018x})`", bits);
242 // This means the `ptr` passed in was not valid, which violates
243 // the unsafe contract of `decode_repr`.
245 // Using this rather than unwrap meaningfully improves the code
246 // for callers which only care about one variant (usually
248 core::hint::unreachable_unchecked();
250 ErrorData::Simple(kind)
252 TAG_SIMPLE_MESSAGE => ErrorData::SimpleMessage(&*ptr.cast::<SimpleMessage>().as_ptr()),
254 // It would be correct for us to use `ptr::sub` here (see the
255 // comment above the `wrapping_add` call in `new_custom` for why),
256 // but it isn't clear that it makes a difference, so we don't.
257 let custom = ptr.as_ptr().cast::<u8>().wrapping_sub(TAG_CUSTOM).cast::<Custom>();
258 ErrorData::Custom(make_custom(custom))
261 // Can't happen, and compiler can tell
267 // This compiles to the same code as the check+transmute, but doesn't require
268 // unsafe, or to hard-code max ErrorKind or its size in a way the compiler
271 fn kind_from_prim(ek: u32) -> Option<ErrorKind> {
272 macro_rules! from_prim {
273 ($prim:expr => $Enum:ident { $($Variant:ident),* $(,)? }) => {{
274 // Force a compile error if the list gets out of date.
275 const _: fn(e: $Enum) = |e: $Enum| match e {
276 $($Enum::$Variant => ()),*
279 $(v if v == ($Enum::$Variant as _) => Some($Enum::$Variant),)*
284 from_prim!(ek => ErrorKind {
304 StaleNetworkFileHandle,
311 FilesystemQuotaExceeded,
329 // Some static checking to alert us if a change breaks any of the assumptions
330 // that our encoding relies on for correctness and soundness. (Some of these are
331 // a bit overly thorough/cautious, admittedly)
333 // If any of these are hit on a platform that libstd supports, we should likely
334 // just use `repr_unpacked.rs` there instead (unless the fix is easy).
335 macro_rules! static_assert {
336 ($condition:expr) => {
337 const _: () = assert!($condition);
339 (@usize_eq: $lhs:expr, $rhs:expr) => {
340 const _: [(); $lhs] = [(); $rhs];
344 // The bitpacking we use requires pointers be exactly 64 bits.
345 static_assert!(@usize_eq: size_of::<NonNull<()>>(), 8);
347 // We also require pointers and usize be the same size.
348 static_assert!(@usize_eq: size_of::<NonNull<()>>(), size_of::<usize>());
350 // `Custom` and `SimpleMessage` need to be thin pointers.
351 static_assert!(@usize_eq: size_of::<&'static SimpleMessage>(), 8);
352 static_assert!(@usize_eq: size_of::<Box<Custom>>(), 8);
354 static_assert!((TAG_MASK + 1).is_power_of_two());
355 // And they must have sufficient alignment.
356 static_assert!(align_of::<SimpleMessage>() >= TAG_MASK + 1);
357 static_assert!(align_of::<Custom>() >= TAG_MASK + 1);
359 static_assert!(@usize_eq: (TAG_MASK & TAG_SIMPLE_MESSAGE), TAG_SIMPLE_MESSAGE);
360 static_assert!(@usize_eq: (TAG_MASK & TAG_CUSTOM), TAG_CUSTOM);
361 static_assert!(@usize_eq: (TAG_MASK & TAG_OS), TAG_OS);
362 static_assert!(@usize_eq: (TAG_MASK & TAG_SIMPLE), TAG_SIMPLE);
364 // This is obviously true (`TAG_CUSTOM` is `0b01`), but in `Repr::new_custom` we
365 // offset a pointer by this value, and expect it to both be within the same
366 // object, and to not wrap around the address space. See the comment in that
367 // function for further details.
369 // Actually, at the moment we use `ptr::wrapping_add`, not `ptr::add`, so this
370 // check isn't needed for that one, although the assertion that we don't
371 // actually wrap around in that wrapping_add does simplify the safety reasoning
372 // elsewhere considerably.
373 static_assert!(size_of::<Custom>() >= TAG_CUSTOM);
375 // These two store a payload which is allowed to be zero, so they must be
376 // non-zero to preserve the `NonNull`'s range invariant.
377 static_assert!(TAG_OS != 0);
378 static_assert!(TAG_SIMPLE != 0);
379 // We can't tag `SimpleMessage`s, the tag must be 0.
380 static_assert!(@usize_eq: TAG_SIMPLE_MESSAGE, 0);
382 // Check that the point of all of this still holds.
384 // We'd check against `io::Error`, but *technically* it's allowed to vary,
385 // as it's not `#[repr(transparent)]`/`#[repr(C)]`. We could add that, but
386 // the `#[repr()]` would show up in rustdoc, which might be seen as a stable
388 static_assert!(@usize_eq: size_of::<Repr>(), 8);
389 static_assert!(@usize_eq: size_of::<Option<Repr>>(), 8);
390 static_assert!(@usize_eq: size_of::<Result<(), Repr>>(), 8);
391 static_assert!(@usize_eq: size_of::<Result<usize, Repr>>(), 16);