1 //! This is a densely packed error representation which is used on targets with
4 //! (Note that `bitpacked` vs `unpacked` here has no relationship to
5 //! `#[repr(packed)]`, it just refers to attempting to use any available bits in
6 //! a more clever manner than `rustc`'s default layout algorithm would).
8 //! Conceptually, it stores the same data as the "unpacked" equivalent we use on
9 //! other targets. Specifically, you can imagine it as an optimized version of
10 //! the following enum (which is roughly equivalent to what's stored by
11 //! `repr_unpacked::Repr`, e.g. `super::ErrorData<Box<Custom>>`):
13 //! ```ignore (exposition-only)
16 //! Simple(ErrorKind),
17 //! SimpleMessage(&'static SimpleMessage),
18 //! Custom(Box<Custom>),
22 //! However, it packs this data into a 64bit non-zero value.
24 //! This optimization not only allows `io::Error` to occupy a single pointer,
25 //! but improves `io::Result` as well, especially for situations like
26 //! `io::Result<()>` (which is now 64 bits) or `io::Result<u64>` (which is now
27 //! 128 bits), which are quite common.
30 //! Tagged values are 64 bits, with the 2 least significant bits used for the
31 //! tag. This means there are there are 4 "variants":
33 //! - **Tag 0b00**: The first variant is equivalent to
34 //! `ErrorData::SimpleMessage`, and holds a `&'static SimpleMessage` directly.
36 //! `SimpleMessage` has an alignment >= 4 (which is requested with
37 //! `#[repr(align)]` and checked statically at the bottom of this file), which
38 //! means every `&'static SimpleMessage` should have the both tag bits as 0,
39 //! meaning its tagged and untagged representation are equivalent.
41 //! This means we can skip tagging it, which is necessary as this variant can
42 //! be constructed from a `const fn`, which probably cannot tag pointers (or
43 //! at least it would be difficult).
45 //! - **Tag 0b01**: The other pointer variant holds the data for
46 //! `ErrorData::Custom` and the remaining 62 bits are used to store a
47 //! `Box<Custom>`. `Custom` also has alignment >= 4, so the bottom two bits
48 //! are free to use for the tag.
50 //! The only important thing to note is that `ptr::wrapping_add` and
51 //! `ptr::wrapping_sub` are used to tag the pointer, rather than bitwise
52 //! operations. This should preserve the pointer's provenance, which would
53 //! otherwise be lost.
55 //! - **Tag 0b10**: Holds the data for `ErrorData::Os(i32)`. We store the `i32`
56 //! in the pointer's most significant 32 bits, and don't use the bits `2..32`
57 //! for anything. Using the top 32 bits is just to let us easily recover the
58 //! `i32` code with the correct sign.
60 //! - **Tag 0b11**: Holds the data for `ErrorData::Simple(ErrorKind)`. This
61 //! stores the `ErrorKind` in the top 32 bits as well, although it doesn't
62 //! occupy nearly that many. Most of the bits are unused here, but it's not
63 //! like we need them for anything else yet.
65 //! # Use of `NonNull<()>`
67 //! Everything is stored in a `NonNull<()>`, which is odd, but actually serves a
70 //! Conceptually you might think of this more like:
72 //! ```ignore (exposition-only)
74 //! // holds integer (Simple/Os) variants, and
75 //! // provides access to the tag bits.
77 //! // Tag is 0, so this is stored untagged.
78 //! msg: &'static SimpleMessage,
79 //! // Tagged (offset) `Box<Custom>` pointer.
80 //! tagged_custom: NonNull<()>,
84 //! But there are a few problems with this:
86 //! 1. Union access is equivalent to a transmute, so this representation would
87 //! require we transmute between integers and pointers in at least one
88 //! direction, which may be UB (and even if not, it is likely harder for a
89 //! compiler to reason about than explicit ptr->int operations).
91 //! 2. Even if all fields of a union have a niche, the union itself doesn't,
92 //! although this may change in the future. This would make things like
93 //! `io::Result<()>` and `io::Result<usize>` larger, which defeats part of
94 //! the motivation of this bitpacking.
96 //! Storing everything in a `NonZeroUsize` (or some other integer) would be a
97 //! bit more traditional for pointer tagging, but it would lose provenance
98 //! information, couldn't be constructed from a `const fn`, and would probably
99 //! run into other issues as well.
101 //! The `NonNull<()>` seems like the only alternative, even if it's fairly odd
102 //! to use a pointer type to store something that may hold an integer, some of
105 use super::{Custom, ErrorData, ErrorKind, SimpleMessage};
106 use alloc::boxed::Box;
107 use core::marker::PhantomData;
108 use core::mem::{align_of, size_of};
109 use core::ptr::{self, NonNull};
111 // The 2 least-significant bits are used as tag.
112 const TAG_MASK: usize = 0b11;
113 const TAG_SIMPLE_MESSAGE: usize = 0b00;
114 const TAG_CUSTOM: usize = 0b01;
115 const TAG_OS: usize = 0b10;
116 const TAG_SIMPLE: usize = 0b11;
118 /// The internal representation.
120 /// See the module docs for more, this is just a way to hack in a check that we
121 /// indeed are not unwind-safe.
123 /// ```compile_fail,E0277
124 /// fn is_unwind_safe<T: core::panic::UnwindSafe>() {}
125 /// is_unwind_safe::<std::io::Error>();
128 pub(super) struct Repr(NonNull<()>, PhantomData<ErrorData<Box<Custom>>>);
130 // All the types `Repr` stores internally are Send + Sync, and so is it.
131 unsafe impl Send for Repr {}
132 unsafe impl Sync for Repr {}
135 pub(super) fn new(dat: ErrorData<Box<Custom>>) -> Self {
137 ErrorData::Os(code) => Self::new_os(code),
138 ErrorData::Simple(kind) => Self::new_simple(kind),
139 ErrorData::SimpleMessage(simple_message) => Self::new_simple_message(simple_message),
140 ErrorData::Custom(b) => Self::new_custom(b),
144 pub(super) fn new_custom(b: Box<Custom>) -> Self {
145 let p = Box::into_raw(b).cast::<u8>();
146 // Should only be possible if an allocator handed out a pointer with
148 debug_assert_eq!(p.addr() & TAG_MASK, 0);
149 // Note: We know `TAG_CUSTOM <= size_of::<Custom>()` (static_assert at
150 // end of file), and both the start and end of the expression must be
151 // valid without address space wraparound due to `Box`'s semantics.
153 // This means it would be correct to implement this using `ptr::add`
154 // (rather than `ptr::wrapping_add`), but it's unclear this would give
155 // any benefit, so we just use `wrapping_add` instead.
156 let tagged = p.wrapping_add(TAG_CUSTOM).cast::<()>();
157 // Safety: `TAG_CUSTOM + p` is the same as `TAG_CUSTOM | p`,
158 // because `p`'s alignment means it isn't allowed to have any of the
159 // `TAG_BITS` set (you can verify that addition and bitwise-or are the
160 // same when the operands have no bits in common using a truth table).
162 // Then, `TAG_CUSTOM | p` is not zero, as that would require
163 // `TAG_CUSTOM` and `p` both be zero, and neither is (as `p` came from a
164 // box, and `TAG_CUSTOM` just... isn't zero -- it's `0b01`). Therefore,
165 // `TAG_CUSTOM + p` isn't zero and so `tagged` can't be, and the
166 // `new_unchecked` is safe.
167 let res = Self(unsafe { NonNull::new_unchecked(tagged) }, PhantomData);
168 // quickly smoke-check we encoded the right thing (This generally will
169 // only run in std's tests, unless the user uses -Zbuild-std)
170 debug_assert!(matches!(res.data(), ErrorData::Custom(_)), "repr(custom) encoding failed");
175 pub(super) fn new_os(code: i32) -> Self {
176 let utagged = ((code as usize) << 32) | TAG_OS;
177 // Safety: `TAG_OS` is not zero, so the result of the `|` is not 0.
178 let res = Self(unsafe { NonNull::new_unchecked(ptr::invalid_mut(utagged)) }, PhantomData);
179 // quickly smoke-check we encoded the right thing (This generally will
180 // only run in std's tests, unless the user uses -Zbuild-std)
182 matches!(res.data(), ErrorData::Os(c) if c == code),
183 "repr(os) encoding failed for {code}"
189 pub(super) fn new_simple(kind: ErrorKind) -> Self {
190 let utagged = ((kind as usize) << 32) | TAG_SIMPLE;
191 // Safety: `TAG_SIMPLE` is not zero, so the result of the `|` is not 0.
192 let res = Self(unsafe { NonNull::new_unchecked(ptr::invalid_mut(utagged)) }, PhantomData);
193 // quickly smoke-check we encoded the right thing (This generally will
194 // only run in std's tests, unless the user uses -Zbuild-std)
196 matches!(res.data(), ErrorData::Simple(k) if k == kind),
197 "repr(simple) encoding failed {:?}",
204 pub(super) const fn new_simple_message(m: &'static SimpleMessage) -> Self {
205 // Safety: References are never null.
206 Self(unsafe { NonNull::new_unchecked(m as *const _ as *mut ()) }, PhantomData)
210 pub(super) fn data(&self) -> ErrorData<&Custom> {
211 // Safety: We're a Repr, decode_repr is fine.
212 unsafe { decode_repr(self.0, |c| &*c) }
216 pub(super) fn data_mut(&mut self) -> ErrorData<&mut Custom> {
217 // Safety: We're a Repr, decode_repr is fine.
218 unsafe { decode_repr(self.0, |c| &mut *c) }
222 pub(super) fn into_data(self) -> ErrorData<Box<Custom>> {
223 let this = core::mem::ManuallyDrop::new(self);
224 // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
225 // safe because we prevent double-drop using `ManuallyDrop`.
226 unsafe { decode_repr(this.0, |p| Box::from_raw(p)) }
233 // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
234 // safe because we're being dropped.
236 let _ = decode_repr(self.0, |p| Box::<Custom>::from_raw(p));
241 // Shared helper to decode a `Repr`'s internal pointer into an ErrorData.
243 // Safety: `ptr`'s bits should be encoded as described in the document at the
244 // top (it should `some_repr.0`)
246 unsafe fn decode_repr<C, F>(ptr: NonNull<()>, make_custom: F) -> ErrorData<C>
248 F: FnOnce(*mut Custom) -> C,
250 let bits = ptr.as_ptr().addr();
251 match bits & TAG_MASK {
253 let code = ((bits as i64) >> 32) as i32;
257 let kind_bits = (bits >> 32) as u32;
258 let kind = kind_from_prim(kind_bits).unwrap_or_else(|| {
259 debug_assert!(false, "Invalid io::error::Repr bits: `Repr({:#018x})`", bits);
260 // This means the `ptr` passed in was not valid, which violates
261 // the unsafe contract of `decode_repr`.
263 // Using this rather than unwrap meaningfully improves the code
264 // for callers which only care about one variant (usually
266 core::hint::unreachable_unchecked();
268 ErrorData::Simple(kind)
270 TAG_SIMPLE_MESSAGE => ErrorData::SimpleMessage(&*ptr.cast::<SimpleMessage>().as_ptr()),
272 // It would be correct for us to use `ptr::byte_sub` here (see the
273 // comment above the `wrapping_add` call in `new_custom` for why),
274 // but it isn't clear that it makes a difference, so we don't.
275 let custom = ptr.as_ptr().wrapping_byte_sub(TAG_CUSTOM).cast::<Custom>();
276 ErrorData::Custom(make_custom(custom))
279 // Can't happen, and compiler can tell
285 // This compiles to the same code as the check+transmute, but doesn't require
286 // unsafe, or to hard-code max ErrorKind or its size in a way the compiler
289 fn kind_from_prim(ek: u32) -> Option<ErrorKind> {
290 macro_rules! from_prim {
291 ($prim:expr => $Enum:ident { $($Variant:ident),* $(,)? }) => {{
292 // Force a compile error if the list gets out of date.
293 const _: fn(e: $Enum) = |e: $Enum| match e {
294 $($Enum::$Variant => ()),*
297 $(v if v == ($Enum::$Variant as _) => Some($Enum::$Variant),)*
302 from_prim!(ek => ErrorKind {
322 StaleNetworkFileHandle,
329 FilesystemQuotaExceeded,
347 // Some static checking to alert us if a change breaks any of the assumptions
348 // that our encoding relies on for correctness and soundness. (Some of these are
349 // a bit overly thorough/cautious, admittedly)
351 // If any of these are hit on a platform that std supports, we should likely
352 // just use `repr_unpacked.rs` there instead (unless the fix is easy).
353 macro_rules! static_assert {
354 ($condition:expr) => {
355 const _: () = assert!($condition);
357 (@usize_eq: $lhs:expr, $rhs:expr) => {
358 const _: [(); $lhs] = [(); $rhs];
362 // The bitpacking we use requires pointers be exactly 64 bits.
363 static_assert!(@usize_eq: size_of::<NonNull<()>>(), 8);
365 // We also require pointers and usize be the same size.
366 static_assert!(@usize_eq: size_of::<NonNull<()>>(), size_of::<usize>());
368 // `Custom` and `SimpleMessage` need to be thin pointers.
369 static_assert!(@usize_eq: size_of::<&'static SimpleMessage>(), 8);
370 static_assert!(@usize_eq: size_of::<Box<Custom>>(), 8);
372 static_assert!((TAG_MASK + 1).is_power_of_two());
373 // And they must have sufficient alignment.
374 static_assert!(align_of::<SimpleMessage>() >= TAG_MASK + 1);
375 static_assert!(align_of::<Custom>() >= TAG_MASK + 1);
377 static_assert!(@usize_eq: TAG_MASK & TAG_SIMPLE_MESSAGE, TAG_SIMPLE_MESSAGE);
378 static_assert!(@usize_eq: TAG_MASK & TAG_CUSTOM, TAG_CUSTOM);
379 static_assert!(@usize_eq: TAG_MASK & TAG_OS, TAG_OS);
380 static_assert!(@usize_eq: TAG_MASK & TAG_SIMPLE, TAG_SIMPLE);
382 // This is obviously true (`TAG_CUSTOM` is `0b01`), but in `Repr::new_custom` we
383 // offset a pointer by this value, and expect it to both be within the same
384 // object, and to not wrap around the address space. See the comment in that
385 // function for further details.
387 // Actually, at the moment we use `ptr::wrapping_add`, not `ptr::add`, so this
388 // check isn't needed for that one, although the assertion that we don't
389 // actually wrap around in that wrapping_add does simplify the safety reasoning
390 // elsewhere considerably.
391 static_assert!(size_of::<Custom>() >= TAG_CUSTOM);
393 // These two store a payload which is allowed to be zero, so they must be
394 // non-zero to preserve the `NonNull`'s range invariant.
395 static_assert!(TAG_OS != 0);
396 static_assert!(TAG_SIMPLE != 0);
397 // We can't tag `SimpleMessage`s, the tag must be 0.
398 static_assert!(@usize_eq: TAG_SIMPLE_MESSAGE, 0);
400 // Check that the point of all of this still holds.
402 // We'd check against `io::Error`, but *technically* it's allowed to vary,
403 // as it's not `#[repr(transparent)]`/`#[repr(C)]`. We could add that, but
404 // the `#[repr()]` would show up in rustdoc, which might be seen as a stable
406 static_assert!(@usize_eq: size_of::<Repr>(), 8);
407 static_assert!(@usize_eq: size_of::<Option<Repr>>(), 8);
408 static_assert!(@usize_eq: size_of::<Result<(), Repr>>(), 8);
409 static_assert!(@usize_eq: size_of::<Result<usize, Repr>>(), 16);