--- /dev/null
+
+# Beginner's Guide To SIMD
+
+Hello and welcome to our SIMD basics guide!
+
+Because SIMD is a subject that many programmers haven't worked with before, we thought that it's best to outline some terms and other basics for you to get started with.
+
+## Quick Background
+
+**SIMD** stands for *Single Instruction, Multiple Data*. In other words, SIMD is when the CPU performs a single action on more than one logical piece of data at the same time. Instead of adding two registers that each contain one `f32` value and getting an `f32` as the result, you might add two registers that each contain `f32x4` (128 bits of data) and then you get an `f32x4` as the output.
+
+This might seem a tiny bit weird at first, but there's a good reason for it. Back in the day, as CPUs got faster and faster, eventually they got so fast that the CPU would just melt itself. The heat management (heat sinks, fans, etc) simply couldn't keep up with how much electricity was going through the metal. Two main strategies were developed to help get around the limits of physics.
+* One of them you're probably familiar with: Multi-core processors. By giving a processor more than one core, each core can do its own work, and because they're physically distant (at least on the CPU's scale) the heat can still be managed. Unfortunately, not all tasks can just be split up across cores in an efficient way.
+* The second strategy is SIMD. If you can't make the register go any faster, you can still make the register *wider*. This lets you process more data at a time, which is *almost* as good as just having a faster CPU. As with multi-core programming, SIMD doesn't fit every kind of task, so you have to know when it will improve your program.
+
+## Terms
+
+SIMD has a few special vocabulary terms you should know:
+
+* **Vector:** A SIMD value is called a vector. This shouldn't be confused with the `Vec<T>` type. A SIMD vector has a fixed size, known at compile time. All of the elements within the vector are of the same type. This makes vectors *similar to* arrays. One difference is that a vector is generally aligned to its *entire* size (eg: 16 bytes, 32 bytes, etc), not just the size of an individual element. Sometimes vector data is called "packed" data.
+
+* **Vectorize**: An operation that uses SIMD instructions to operate over a vector is often referred to as "vectorized".
+
+* **Autovectorization**: Also known as _implicit vectorization_. This is when a compiler can automatically recognize a situation where scalar instructions may be replaced with SIMD instructions, and use those instead.
+
+* **Scalar:** "Scalar" in mathematical contexts refers to values that can be represented as a single element, mostly numbers like 6, 3.14, or -2. It can also be used to describe "scalar operations" that use strictly scalar values, like addition. This term is mostly used to differentiate between vectorized operations that use SIMD instructions and scalar operations that don't.
+
+* **Lane:** A single element position within a vector is called a lane. If you have `N` lanes available then they're numbered from `0` to `N-1` when referring to them, again like an array. The biggest difference between an array element and a vector lane is that in general is *relatively costly* to access an individual lane value. On most architectures, the vector has to be pushed out of the SIMD register onto the stack, then an individual lane is accessed while it's on the stack (and possibly the stack value is read back into a register). For this reason, when working with SIMD you should avoid reading or writing the value of an individual lane during hot loops.
+
+* **Bit Widths:** When talking about SIMD, the bit widths used are the bit size of the vectors involved, *not* the individual elements. So "128-bit SIMD" has 128-bit vectors, and that might be `f32x4`, `i32x4`, `i16x8`, or other variations. While 128-bit SIMD is the most common, there's also 64-bit, 256-bit, and even 512-bit on the newest CPUs.
+
+* **Vector Register:** The extra-wide registers that are used for SIMD operations are commonly called vector registers, though you may also see "SIMD registers", vendor names for specific features, or even "floating-point register" as it is common for the same registers to be used with both scalar and vectorized floating-point operations.
+
+* **Vertical:** When an operation is "vertical", each lane processes individually without regard to the other lanes in the same vector. For example, a "vertical add" between two vectors would add lane 0 in `a` with lane 0 in `b`, with the total in lane 0 of `out`, and then the same thing for lanes 1, 2, etc. Most SIMD operations are vertical operations, so if your problem is a vertical problem then you can probably solve it with SIMD.
+
+* **Reducing/Reduce:** When an operation is "reducing" (functions named `reduce_*`), the lanes within a single vector are merged using some operation such as addition, returning the merged value as a scalar. For instance, a reducing add would return the sum of all the lanes' values.
+
+* **Target Feature:** Rust calls a CPU architecture extension a `target_feature`. Proper SIMD requires various CPU extensions to be enabled (details below). Don't confuse this with `feature`, which is a Cargo crate concept.
+
+## Target Features
+
+When using SIMD, you should be familiar with the CPU feature set that you're targeting.
+
+On `arm` and `aarch64` it's fairly simple. There's just one CPU feature that controls if SIMD is available: `neon` (or "NEON", all caps, as the ARM docs often put it). Neon registers can be used as 64-bit or 128-bit. When doing 128-bit operations it just uses two 64-bit registers as a single 128-bit register.
+
+> By default, the `aarch64`, `arm`, and `thumb` Rust targets generally do not enable `neon` unless it's in the target string.
+
+On `x86` and `x86_64` it's slightly more complicated. The SIMD support is split into many levels:
+* 128-bit: `sse`, `sse2`, `sse3`, `ssse3` (not a typo!), `sse4.1`, `sse4.2`, `sse4a` (AMD only)
+* 256-bit (mostly): `avx`, `avx2`, `fma`
+* 512-bit (mostly): a *wide* range of `avx512` variations
+
+The list notes the bit widths available at each feature level, though the operations of the more advanced features can generally be used with the smaller register sizes as well. For example, new operations introduced in `avx` generally have a 128-bit form as well as a 256-bit form. This means that even if you only do 128-bit work you can still benefit from the later feature levels.
+
+> By default, the `i686` and `x86_64` Rust targets enable `sse` and `sse2`.
+
+### Selecting Additional Target Features
+
+If you want to enable support for a target feature within your build, generally you should use a [target-feature](https://rust-lang.github.io/packed_simd/perf-guide/target-feature/rustflags.html#target-feature) setting within you `RUSTFLAGS` setting.
+
+If you know that you're targeting a specific CPU you can instead use the [target-cpu](https://rust-lang.github.io/packed_simd/perf-guide/target-feature/rustflags.html#target-cpu) flag and the compiler will enable the correct set of features for that CPU.
+
+The [Steam Hardware Survey](https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam) is one of the few places with data on how common various CPU features are. The dataset is limited to "the kinds of computers owned by people who play computer games", so the info only covers `x86`/`x86_64`, and it also probably skews to slightly higher quality computers than average. Still, we can see that the `sse` levels have very high support, `avx` and `avx2` are quite common as well, and the `avx-512` family is still so early in adoption you can barely find it in consumer grade stuff.
+
+## Running a program compiled for a CPU feature level that the CPU doesn't support is automatic undefined behavior.
+
+This means that if you build your program with `avx` support enabled and run it on a CPU without `avx` support, it's **instantly** undefined behavior.
+
+Even without an `unsafe` block in sight.
+
+This is no bug in Rust, or soundness hole in the type system. You just plain can't make a CPU do what it doesn't know how to do.
+
+This is why the various Rust targets *don't* enable many CPU feature flags by default: requiring a more advanced CPU makes the final binary *less* portable.
+
+So please select an appropriate CPU feature level when building your programs.
+
+## Size, Alignment, and Unsafe Code
+
+Most of the portable SIMD API is designed to allow the user to gloss over the details of different architectures and avoid using unsafe code. However, there are plenty of reasons to want to use unsafe code with these SIMD types, such as using an intrinsic function from `core::arch` to further accelerate particularly specialized SIMD operations on a given platform, while still using the portable API elsewhere. For these cases, there are some rules to keep in mind.
+
+Fortunately, most SIMD types have a fairly predictable size. `i32x4` is bit-equivalent to `[i32; 4]` and so can be bitcast to it, e.g. using [`mem::transmute`], though the API usually offers a safe cast you can use instead.
+
+However, this is not the same as alignment. Computer architectures generally prefer aligned accesses, especially when moving data between memory and vector registers, and while some support specialized operations that can bend the rules to help with this, unaligned access is still typically slow, or even undefined behavior. In addition, different architectures can require different alignments when interacting with their native SIMD types. For this reason, any `#[repr(simd)]` type has a non-portable alignment. If it is necessary to directly interact with the alignment of these types, it should be via [`mem::align_of`].
+
++When working with slices, data correctly aligned for SIMD can be acquired using the [`as_simd`] and [`as_simd_mut`] methods of the slice primitive.
++
+[`mem::transmute`]: https://doc.rust-lang.org/core/mem/fn.transmute.html
+[`mem::align_of`]: https://doc.rust-lang.org/core/mem/fn.align_of.html
++[`as_simd`]: https://doc.rust-lang.org/nightly/std/primitive.slice.html#method.as_simd
++[`as_simd_mut`]: https://doc.rust-lang.org/nightly/std/primitive.slice.html#method.as_simd_mut
++
--- /dev/null
- default = []
+[package]
+name = "core_simd"
+version = "0.1.0"
+edition = "2021"
+homepage = "https://github.com/rust-lang/portable-simd"
+repository = "https://github.com/rust-lang/portable-simd"
+keywords = ["core", "simd", "intrinsics"]
+categories = ["hardware-support", "no-std"]
+license = "MIT OR Apache-2.0"
+
+[features]
++default = ["as_crate"]
++as_crate = []
+std = []
+generic_const_exprs = []
+
+[target.'cfg(target_arch = "wasm32")'.dev-dependencies.wasm-bindgen]
+version = "0.2"
+
+[dev-dependencies.wasm-bindgen-test]
+version = "0.3"
+
+[dev-dependencies.proptest]
+version = "0.10"
+default-features = false
+features = ["alloc"]
+
+[dev-dependencies.test_helpers]
+path = "../test_helpers"
+
+[dev-dependencies]
+std_float = { path = "../std_float/", features = ["as_crate"] }
--- /dev/null
--- /dev/null
++mod float;
++mod int;
++mod uint;
++
++mod sealed {
++ pub trait Sealed {}
++}
++
++pub use float::*;
++pub use int::*;
++pub use uint::*;
--- /dev/null
--- /dev/null
++use super::sealed::Sealed;
++use crate::simd::{
++ intrinsics, LaneCount, Mask, Simd, SimdElement, SimdPartialEq, SimdPartialOrd,
++ SupportedLaneCount,
++};
++
++/// Operations on SIMD vectors of floats.
++pub trait SimdFloat: Copy + Sealed {
++ /// Mask type used for manipulating this SIMD vector type.
++ type Mask;
++
++ /// Scalar type contained by this SIMD vector type.
++ type Scalar;
++
++ /// Bit representation of this SIMD vector type.
++ type Bits;
++
++ /// Raw transmutation to an unsigned integer vector type with the
++ /// same size and number of lanes.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn to_bits(self) -> Self::Bits;
++
++ /// Raw transmutation from an unsigned integer vector type with the
++ /// same size and number of lanes.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn from_bits(bits: Self::Bits) -> Self;
++
++ /// Produces a vector where every lane has the absolute value of the
++ /// equivalently-indexed lane in `self`.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn abs(self) -> Self;
++
++ /// Takes the reciprocal (inverse) of each lane, `1/x`.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn recip(self) -> Self;
++
++ /// Converts each lane from radians to degrees.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn to_degrees(self) -> Self;
++
++ /// Converts each lane from degrees to radians.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn to_radians(self) -> Self;
++
++ /// Returns true for each lane if it has a positive sign, including
++ /// `+0.0`, `NaN`s with positive sign bit and positive infinity.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_sign_positive(self) -> Self::Mask;
++
++ /// Returns true for each lane if it has a negative sign, including
++ /// `-0.0`, `NaN`s with negative sign bit and negative infinity.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_sign_negative(self) -> Self::Mask;
++
++ /// Returns true for each lane if its value is `NaN`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_nan(self) -> Self::Mask;
++
++ /// Returns true for each lane if its value is positive infinity or negative infinity.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_infinite(self) -> Self::Mask;
++
++ /// Returns true for each lane if its value is neither infinite nor `NaN`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_finite(self) -> Self::Mask;
++
++ /// Returns true for each lane if its value is subnormal.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_subnormal(self) -> Self::Mask;
++
++ /// Returns true for each lane if its value is neither zero, infinite,
++ /// subnormal, nor `NaN`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_normal(self) -> Self::Mask;
++
++ /// Replaces each lane with a number that represents its sign.
++ ///
++ /// * `1.0` if the number is positive, `+0.0`, or `INFINITY`
++ /// * `-1.0` if the number is negative, `-0.0`, or `NEG_INFINITY`
++ /// * `NAN` if the number is `NAN`
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn signum(self) -> Self;
++
++ /// Returns each lane with the magnitude of `self` and the sign of `sign`.
++ ///
++ /// For any lane containing a `NAN`, a `NAN` with the sign of `sign` is returned.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn copysign(self, sign: Self) -> Self;
++
++ /// Returns the minimum of each lane.
++ ///
++ /// If one of the values is `NAN`, then the other value is returned.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn simd_min(self, other: Self) -> Self;
++
++ /// Returns the maximum of each lane.
++ ///
++ /// If one of the values is `NAN`, then the other value is returned.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn simd_max(self, other: Self) -> Self;
++
++ /// Restrict each lane to a certain interval unless it is NaN.
++ ///
++ /// For each lane in `self`, returns the corresponding lane in `max` if the lane is
++ /// greater than `max`, and the corresponding lane in `min` if the lane is less
++ /// than `min`. Otherwise returns the lane in `self`.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn simd_clamp(self, min: Self, max: Self) -> Self;
++
++ /// Returns the sum of the lanes of the vector.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{f32x2, SimdFloat};
++ /// let v = f32x2::from_array([1., 2.]);
++ /// assert_eq!(v.reduce_sum(), 3.);
++ /// ```
++ fn reduce_sum(self) -> Self::Scalar;
++
++ /// Reducing multiply. Returns the product of the lanes of the vector.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{f32x2, SimdFloat};
++ /// let v = f32x2::from_array([3., 4.]);
++ /// assert_eq!(v.reduce_product(), 12.);
++ /// ```
++ fn reduce_product(self) -> Self::Scalar;
++
++ /// Returns the maximum lane in the vector.
++ ///
++ /// Returns values based on equality, so a vector containing both `0.` and `-0.` may
++ /// return either.
++ ///
++ /// This function will not return `NaN` unless all lanes are `NaN`.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{f32x2, SimdFloat};
++ /// let v = f32x2::from_array([1., 2.]);
++ /// assert_eq!(v.reduce_max(), 2.);
++ ///
++ /// // NaN values are skipped...
++ /// let v = f32x2::from_array([1., f32::NAN]);
++ /// assert_eq!(v.reduce_max(), 1.);
++ ///
++ /// // ...unless all values are NaN
++ /// let v = f32x2::from_array([f32::NAN, f32::NAN]);
++ /// assert!(v.reduce_max().is_nan());
++ /// ```
++ fn reduce_max(self) -> Self::Scalar;
++
++ /// Returns the minimum lane in the vector.
++ ///
++ /// Returns values based on equality, so a vector containing both `0.` and `-0.` may
++ /// return either.
++ ///
++ /// This function will not return `NaN` unless all lanes are `NaN`.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{f32x2, SimdFloat};
++ /// let v = f32x2::from_array([3., 7.]);
++ /// assert_eq!(v.reduce_min(), 3.);
++ ///
++ /// // NaN values are skipped...
++ /// let v = f32x2::from_array([1., f32::NAN]);
++ /// assert_eq!(v.reduce_min(), 1.);
++ ///
++ /// // ...unless all values are NaN
++ /// let v = f32x2::from_array([f32::NAN, f32::NAN]);
++ /// assert!(v.reduce_min().is_nan());
++ /// ```
++ fn reduce_min(self) -> Self::Scalar;
++}
++
++macro_rules! impl_trait {
++ { $($ty:ty { bits: $bits_ty:ty, mask: $mask_ty:ty }),* } => {
++ $(
++ impl<const LANES: usize> Sealed for Simd<$ty, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ }
++
++ impl<const LANES: usize> SimdFloat for Simd<$ty, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ type Mask = Mask<<$mask_ty as SimdElement>::Mask, LANES>;
++ type Scalar = $ty;
++ type Bits = Simd<$bits_ty, LANES>;
++
++ #[inline]
++ fn to_bits(self) -> Simd<$bits_ty, LANES> {
++ assert_eq!(core::mem::size_of::<Self>(), core::mem::size_of::<Self::Bits>());
++ // Safety: transmuting between vector types is safe
++ unsafe { core::mem::transmute_copy(&self) }
++ }
++
++ #[inline]
++ fn from_bits(bits: Simd<$bits_ty, LANES>) -> Self {
++ assert_eq!(core::mem::size_of::<Self>(), core::mem::size_of::<Self::Bits>());
++ // Safety: transmuting between vector types is safe
++ unsafe { core::mem::transmute_copy(&bits) }
++ }
++
++ #[inline]
++ fn abs(self) -> Self {
++ // Safety: `self` is a float vector
++ unsafe { intrinsics::simd_fabs(self) }
++ }
++
++ #[inline]
++ fn recip(self) -> Self {
++ Self::splat(1.0) / self
++ }
++
++ #[inline]
++ fn to_degrees(self) -> Self {
++ // to_degrees uses a special constant for better precision, so extract that constant
++ self * Self::splat(Self::Scalar::to_degrees(1.))
++ }
++
++ #[inline]
++ fn to_radians(self) -> Self {
++ self * Self::splat(Self::Scalar::to_radians(1.))
++ }
++
++ #[inline]
++ fn is_sign_positive(self) -> Self::Mask {
++ !self.is_sign_negative()
++ }
++
++ #[inline]
++ fn is_sign_negative(self) -> Self::Mask {
++ let sign_bits = self.to_bits() & Simd::splat((!0 >> 1) + 1);
++ sign_bits.simd_gt(Simd::splat(0))
++ }
++
++ #[inline]
++ fn is_nan(self) -> Self::Mask {
++ self.simd_ne(self)
++ }
++
++ #[inline]
++ fn is_infinite(self) -> Self::Mask {
++ self.abs().simd_eq(Self::splat(Self::Scalar::INFINITY))
++ }
++
++ #[inline]
++ fn is_finite(self) -> Self::Mask {
++ self.abs().simd_lt(Self::splat(Self::Scalar::INFINITY))
++ }
++
++ #[inline]
++ fn is_subnormal(self) -> Self::Mask {
++ self.abs().simd_ne(Self::splat(0.0)) & (self.to_bits() & Self::splat(Self::Scalar::INFINITY).to_bits()).simd_eq(Simd::splat(0))
++ }
++
++ #[inline]
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn is_normal(self) -> Self::Mask {
++ !(self.abs().simd_eq(Self::splat(0.0)) | self.is_nan() | self.is_subnormal() | self.is_infinite())
++ }
++
++ #[inline]
++ fn signum(self) -> Self {
++ self.is_nan().select(Self::splat(Self::Scalar::NAN), Self::splat(1.0).copysign(self))
++ }
++
++ #[inline]
++ fn copysign(self, sign: Self) -> Self {
++ let sign_bit = sign.to_bits() & Self::splat(-0.).to_bits();
++ let magnitude = self.to_bits() & !Self::splat(-0.).to_bits();
++ Self::from_bits(sign_bit | magnitude)
++ }
++
++ #[inline]
++ fn simd_min(self, other: Self) -> Self {
++ // Safety: `self` and `other` are float vectors
++ unsafe { intrinsics::simd_fmin(self, other) }
++ }
++
++ #[inline]
++ fn simd_max(self, other: Self) -> Self {
++ // Safety: `self` and `other` are floating point vectors
++ unsafe { intrinsics::simd_fmax(self, other) }
++ }
++
++ #[inline]
++ fn simd_clamp(self, min: Self, max: Self) -> Self {
++ assert!(
++ min.simd_le(max).all(),
++ "each lane in `min` must be less than or equal to the corresponding lane in `max`",
++ );
++ let mut x = self;
++ x = x.simd_lt(min).select(min, x);
++ x = x.simd_gt(max).select(max, x);
++ x
++ }
++
++ #[inline]
++ fn reduce_sum(self) -> Self::Scalar {
++ // LLVM sum is inaccurate on i586
++ if cfg!(all(target_arch = "x86", not(target_feature = "sse2"))) {
++ self.as_array().iter().sum()
++ } else {
++ // Safety: `self` is a float vector
++ unsafe { intrinsics::simd_reduce_add_ordered(self, 0.) }
++ }
++ }
++
++ #[inline]
++ fn reduce_product(self) -> Self::Scalar {
++ // LLVM product is inaccurate on i586
++ if cfg!(all(target_arch = "x86", not(target_feature = "sse2"))) {
++ self.as_array().iter().product()
++ } else {
++ // Safety: `self` is a float vector
++ unsafe { intrinsics::simd_reduce_mul_ordered(self, 1.) }
++ }
++ }
++
++ #[inline]
++ fn reduce_max(self) -> Self::Scalar {
++ // Safety: `self` is a float vector
++ unsafe { intrinsics::simd_reduce_max(self) }
++ }
++
++ #[inline]
++ fn reduce_min(self) -> Self::Scalar {
++ // Safety: `self` is a float vector
++ unsafe { intrinsics::simd_reduce_min(self) }
++ }
++ }
++ )*
++ }
++}
++
++impl_trait! { f32 { bits: u32, mask: i32 }, f64 { bits: u64, mask: i64 } }
--- /dev/null
--- /dev/null
++use super::sealed::Sealed;
++use crate::simd::{
++ intrinsics, LaneCount, Mask, Simd, SimdElement, SimdPartialOrd, SupportedLaneCount,
++};
++
++/// Operations on SIMD vectors of signed integers.
++pub trait SimdInt: Copy + Sealed {
++ /// Mask type used for manipulating this SIMD vector type.
++ type Mask;
++
++ /// Scalar type contained by this SIMD vector type.
++ type Scalar;
++
++ /// Lanewise saturating add.
++ ///
++ /// # Examples
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdInt};
++ /// use core::i32::{MIN, MAX};
++ /// let x = Simd::from_array([MIN, 0, 1, MAX]);
++ /// let max = Simd::splat(MAX);
++ /// let unsat = x + max;
++ /// let sat = x.saturating_add(max);
++ /// assert_eq!(unsat, Simd::from_array([-1, MAX, MIN, -2]));
++ /// assert_eq!(sat, Simd::from_array([-1, MAX, MAX, MAX]));
++ /// ```
++ fn saturating_add(self, second: Self) -> Self;
++
++ /// Lanewise saturating subtract.
++ ///
++ /// # Examples
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdInt};
++ /// use core::i32::{MIN, MAX};
++ /// let x = Simd::from_array([MIN, -2, -1, MAX]);
++ /// let max = Simd::splat(MAX);
++ /// let unsat = x - max;
++ /// let sat = x.saturating_sub(max);
++ /// assert_eq!(unsat, Simd::from_array([1, MAX, MIN, 0]));
++ /// assert_eq!(sat, Simd::from_array([MIN, MIN, MIN, 0]));
++ fn saturating_sub(self, second: Self) -> Self;
++
++ /// Lanewise absolute value, implemented in Rust.
++ /// Every lane becomes its absolute value.
++ ///
++ /// # Examples
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdInt};
++ /// use core::i32::{MIN, MAX};
++ /// let xs = Simd::from_array([MIN, MIN +1, -5, 0]);
++ /// assert_eq!(xs.abs(), Simd::from_array([MIN, MAX, 5, 0]));
++ /// ```
++ fn abs(self) -> Self;
++
++ /// Lanewise saturating absolute value, implemented in Rust.
++ /// As abs(), except the MIN value becomes MAX instead of itself.
++ ///
++ /// # Examples
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdInt};
++ /// use core::i32::{MIN, MAX};
++ /// let xs = Simd::from_array([MIN, -2, 0, 3]);
++ /// let unsat = xs.abs();
++ /// let sat = xs.saturating_abs();
++ /// assert_eq!(unsat, Simd::from_array([MIN, 2, 0, 3]));
++ /// assert_eq!(sat, Simd::from_array([MAX, 2, 0, 3]));
++ /// ```
++ fn saturating_abs(self) -> Self;
++
++ /// Lanewise saturating negation, implemented in Rust.
++ /// As neg(), except the MIN value becomes MAX instead of itself.
++ ///
++ /// # Examples
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdInt};
++ /// use core::i32::{MIN, MAX};
++ /// let x = Simd::from_array([MIN, -2, 3, MAX]);
++ /// let unsat = -x;
++ /// let sat = x.saturating_neg();
++ /// assert_eq!(unsat, Simd::from_array([MIN, 2, -3, MIN + 1]));
++ /// assert_eq!(sat, Simd::from_array([MAX, 2, -3, MIN + 1]));
++ /// ```
++ fn saturating_neg(self) -> Self;
++
++ /// Returns true for each positive lane and false if it is zero or negative.
++ fn is_positive(self) -> Self::Mask;
++
++ /// Returns true for each negative lane and false if it is zero or positive.
++ fn is_negative(self) -> Self::Mask;
++
++ /// Returns numbers representing the sign of each lane.
++ /// * `0` if the number is zero
++ /// * `1` if the number is positive
++ /// * `-1` if the number is negative
++ fn signum(self) -> Self;
++
++ /// Returns the sum of the lanes of the vector, with wrapping addition.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{i32x4, SimdInt};
++ /// let v = i32x4::from_array([1, 2, 3, 4]);
++ /// assert_eq!(v.reduce_sum(), 10);
++ ///
++ /// // SIMD integer addition is always wrapping
++ /// let v = i32x4::from_array([i32::MAX, 1, 0, 0]);
++ /// assert_eq!(v.reduce_sum(), i32::MIN);
++ /// ```
++ fn reduce_sum(self) -> Self::Scalar;
++
++ /// Returns the product of the lanes of the vector, with wrapping multiplication.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{i32x4, SimdInt};
++ /// let v = i32x4::from_array([1, 2, 3, 4]);
++ /// assert_eq!(v.reduce_product(), 24);
++ ///
++ /// // SIMD integer multiplication is always wrapping
++ /// let v = i32x4::from_array([i32::MAX, 2, 1, 1]);
++ /// assert!(v.reduce_product() < i32::MAX);
++ /// ```
++ fn reduce_product(self) -> Self::Scalar;
++
++ /// Returns the maximum lane in the vector.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{i32x4, SimdInt};
++ /// let v = i32x4::from_array([1, 2, 3, 4]);
++ /// assert_eq!(v.reduce_max(), 4);
++ /// ```
++ fn reduce_max(self) -> Self::Scalar;
++
++ /// Returns the minimum lane in the vector.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{i32x4, SimdInt};
++ /// let v = i32x4::from_array([1, 2, 3, 4]);
++ /// assert_eq!(v.reduce_min(), 1);
++ /// ```
++ fn reduce_min(self) -> Self::Scalar;
++
++ /// Returns the cumulative bitwise "and" across the lanes of the vector.
++ fn reduce_and(self) -> Self::Scalar;
++
++ /// Returns the cumulative bitwise "or" across the lanes of the vector.
++ fn reduce_or(self) -> Self::Scalar;
++
++ /// Returns the cumulative bitwise "xor" across the lanes of the vector.
++ fn reduce_xor(self) -> Self::Scalar;
++}
++
++macro_rules! impl_trait {
++ { $($ty:ty),* } => {
++ $(
++ impl<const LANES: usize> Sealed for Simd<$ty, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ }
++
++ impl<const LANES: usize> SimdInt for Simd<$ty, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ type Mask = Mask<<$ty as SimdElement>::Mask, LANES>;
++ type Scalar = $ty;
++
++ #[inline]
++ fn saturating_add(self, second: Self) -> Self {
++ // Safety: `self` is a vector
++ unsafe { intrinsics::simd_saturating_add(self, second) }
++ }
++
++ #[inline]
++ fn saturating_sub(self, second: Self) -> Self {
++ // Safety: `self` is a vector
++ unsafe { intrinsics::simd_saturating_sub(self, second) }
++ }
++
++ #[inline]
++ fn abs(self) -> Self {
++ const SHR: $ty = <$ty>::BITS as $ty - 1;
++ let m = self >> Simd::splat(SHR);
++ (self^m) - m
++ }
++
++ #[inline]
++ fn saturating_abs(self) -> Self {
++ // arith shift for -1 or 0 mask based on sign bit, giving 2s complement
++ const SHR: $ty = <$ty>::BITS as $ty - 1;
++ let m = self >> Simd::splat(SHR);
++ (self^m).saturating_sub(m)
++ }
++
++ #[inline]
++ fn saturating_neg(self) -> Self {
++ Self::splat(0).saturating_sub(self)
++ }
++
++ #[inline]
++ fn is_positive(self) -> Self::Mask {
++ self.simd_gt(Self::splat(0))
++ }
++
++ #[inline]
++ fn is_negative(self) -> Self::Mask {
++ self.simd_lt(Self::splat(0))
++ }
++
++ #[inline]
++ fn signum(self) -> Self {
++ self.is_positive().select(
++ Self::splat(1),
++ self.is_negative().select(Self::splat(-1), Self::splat(0))
++ )
++ }
++
++ #[inline]
++ fn reduce_sum(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_add_ordered(self, 0) }
++ }
++
++ #[inline]
++ fn reduce_product(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_mul_ordered(self, 1) }
++ }
++
++ #[inline]
++ fn reduce_max(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_max(self) }
++ }
++
++ #[inline]
++ fn reduce_min(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_min(self) }
++ }
++
++ #[inline]
++ fn reduce_and(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_and(self) }
++ }
++
++ #[inline]
++ fn reduce_or(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_or(self) }
++ }
++
++ #[inline]
++ fn reduce_xor(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_xor(self) }
++ }
++ }
++ )*
++ }
++}
++
++impl_trait! { i8, i16, i32, i64, isize }
--- /dev/null
--- /dev/null
++use super::sealed::Sealed;
++use crate::simd::{intrinsics, LaneCount, Simd, SupportedLaneCount};
++
++/// Operations on SIMD vectors of unsigned integers.
++pub trait SimdUint: Copy + Sealed {
++ /// Scalar type contained by this SIMD vector type.
++ type Scalar;
++
++ /// Lanewise saturating add.
++ ///
++ /// # Examples
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdUint};
++ /// use core::u32::MAX;
++ /// let x = Simd::from_array([2, 1, 0, MAX]);
++ /// let max = Simd::splat(MAX);
++ /// let unsat = x + max;
++ /// let sat = x.saturating_add(max);
++ /// assert_eq!(unsat, Simd::from_array([1, 0, MAX, MAX - 1]));
++ /// assert_eq!(sat, max);
++ /// ```
++ fn saturating_add(self, second: Self) -> Self;
++
++ /// Lanewise saturating subtract.
++ ///
++ /// # Examples
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdUint};
++ /// use core::u32::MAX;
++ /// let x = Simd::from_array([2, 1, 0, MAX]);
++ /// let max = Simd::splat(MAX);
++ /// let unsat = x - max;
++ /// let sat = x.saturating_sub(max);
++ /// assert_eq!(unsat, Simd::from_array([3, 2, 1, 0]));
++ /// assert_eq!(sat, Simd::splat(0));
++ fn saturating_sub(self, second: Self) -> Self;
++
++ /// Returns the sum of the lanes of the vector, with wrapping addition.
++ fn reduce_sum(self) -> Self::Scalar;
++
++ /// Returns the product of the lanes of the vector, with wrapping multiplication.
++ fn reduce_product(self) -> Self::Scalar;
++
++ /// Returns the maximum lane in the vector.
++ fn reduce_max(self) -> Self::Scalar;
++
++ /// Returns the minimum lane in the vector.
++ fn reduce_min(self) -> Self::Scalar;
++
++ /// Returns the cumulative bitwise "and" across the lanes of the vector.
++ fn reduce_and(self) -> Self::Scalar;
++
++ /// Returns the cumulative bitwise "or" across the lanes of the vector.
++ fn reduce_or(self) -> Self::Scalar;
++
++ /// Returns the cumulative bitwise "xor" across the lanes of the vector.
++ fn reduce_xor(self) -> Self::Scalar;
++}
++
++macro_rules! impl_trait {
++ { $($ty:ty),* } => {
++ $(
++ impl<const LANES: usize> Sealed for Simd<$ty, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ }
++
++ impl<const LANES: usize> SimdUint for Simd<$ty, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ type Scalar = $ty;
++
++ #[inline]
++ fn saturating_add(self, second: Self) -> Self {
++ // Safety: `self` is a vector
++ unsafe { intrinsics::simd_saturating_add(self, second) }
++ }
++
++ #[inline]
++ fn saturating_sub(self, second: Self) -> Self {
++ // Safety: `self` is a vector
++ unsafe { intrinsics::simd_saturating_sub(self, second) }
++ }
++
++ #[inline]
++ fn reduce_sum(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_add_ordered(self, 0) }
++ }
++
++ #[inline]
++ fn reduce_product(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_mul_ordered(self, 1) }
++ }
++
++ #[inline]
++ fn reduce_max(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_max(self) }
++ }
++
++ #[inline]
++ fn reduce_min(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_min(self) }
++ }
++
++ #[inline]
++ fn reduce_and(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_and(self) }
++ }
++
++ #[inline]
++ fn reduce_or(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_or(self) }
++ }
++
++ #[inline]
++ fn reduce_xor(self) -> Self::Scalar {
++ // Safety: `self` is an integer vector
++ unsafe { intrinsics::simd_reduce_xor(self) }
++ }
++ }
++ )*
++ }
++}
++
++impl_trait! { u8, u16, u32, u64, usize }
--- /dev/null
--- /dev/null
++use crate::simd::{intrinsics, LaneCount, Mask, Simd, SimdElement, SupportedLaneCount};
++
++/// Parallel `PartialEq`.
++pub trait SimdPartialEq {
++ /// The mask type returned by each comparison.
++ type Mask;
++
++ /// Test if each lane is equal to the corresponding lane in `other`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn simd_eq(self, other: Self) -> Self::Mask;
++
++ /// Test if each lane is equal to the corresponding lane in `other`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn simd_ne(self, other: Self) -> Self::Mask;
++}
++
++macro_rules! impl_number {
++ { $($number:ty),* } => {
++ $(
++ impl<const LANES: usize> SimdPartialEq for Simd<$number, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ type Mask = Mask<<$number as SimdElement>::Mask, LANES>;
++
++ #[inline]
++ fn simd_eq(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_eq(self, other)) }
++ }
++
++ #[inline]
++ fn simd_ne(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_ne(self, other)) }
++ }
++ }
++ )*
++ }
++}
++
++impl_number! { f32, f64, u8, u16, u32, u64, usize, i8, i16, i32, i64, isize }
++
++macro_rules! impl_mask {
++ { $($integer:ty),* } => {
++ $(
++ impl<const LANES: usize> SimdPartialEq for Mask<$integer, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ type Mask = Self;
++
++ #[inline]
++ fn simd_eq(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Self::from_int_unchecked(intrinsics::simd_eq(self.to_int(), other.to_int())) }
++ }
++
++ #[inline]
++ fn simd_ne(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Self::from_int_unchecked(intrinsics::simd_ne(self.to_int(), other.to_int())) }
++ }
++ }
++ )*
++ }
++}
++
++impl_mask! { i8, i16, i32, i64, isize }
--- /dev/null
- /// A type representing a vector lane count.
+mod sealed {
+ pub trait Sealed {}
+}
+use sealed::Sealed;
+
- /// Helper trait for vector lane counts.
++/// Specifies the number of lanes in a SIMD vector as a type.
+pub struct LaneCount<const LANES: usize>;
+
+impl<const LANES: usize> LaneCount<LANES> {
+ /// The number of bytes in a bitmask with this many lanes.
+ pub const BITMASK_LEN: usize = (LANES + 7) / 8;
+}
+
++/// Statically guarantees that a lane count is marked as supported.
++///
++/// This trait is *sealed*: the list of implementors below is total.
++/// Users do not have the ability to mark additional `LaneCount<N>` values as supported.
++/// Only SIMD vectors with supported lane counts are constructable.
+pub trait SupportedLaneCount: Sealed {
+ #[doc(hidden)]
+ type BitMask: Copy + Default + AsRef<[u8]> + AsMut<[u8]>;
+}
+
+impl<const LANES: usize> Sealed for LaneCount<LANES> {}
+
+impl SupportedLaneCount for LaneCount<1> {
+ type BitMask = [u8; 1];
+}
+impl SupportedLaneCount for LaneCount<2> {
+ type BitMask = [u8; 1];
+}
+impl SupportedLaneCount for LaneCount<4> {
+ type BitMask = [u8; 1];
+}
+impl SupportedLaneCount for LaneCount<8> {
+ type BitMask = [u8; 1];
+}
+impl SupportedLaneCount for LaneCount<16> {
+ type BitMask = [u8; 2];
+}
+impl SupportedLaneCount for LaneCount<32> {
+ type BitMask = [u8; 4];
+}
+impl SupportedLaneCount for LaneCount<64> {
+ type BitMask = [u8; 8];
+}
--- /dev/null
- #![deny(unsafe_op_in_unsafe_fn)]
+#![no_std]
+#![feature(
+ convert_float_to_int,
+ decl_macro,
+ intra_doc_pointers,
+ platform_intrinsics,
+ repr_simd,
+ simd_ffi,
+ staged_api,
+ stdsimd
+)]
+#![cfg_attr(feature = "generic_const_exprs", feature(generic_const_exprs))]
+#![cfg_attr(feature = "generic_const_exprs", allow(incomplete_features))]
+#![warn(missing_docs)]
++#![deny(unsafe_op_in_unsafe_fn, clippy::undocumented_unsafe_blocks)]
+#![unstable(feature = "portable_simd", issue = "86656")]
+//! Portable SIMD module.
+
+#[path = "mod.rs"]
+mod core_simd;
+pub use self::core_simd::simd;
+pub use simd::*;
--- /dev/null
- use crate::simd::{intrinsics, LaneCount, Simd, SimdElement, SupportedLaneCount};
+//! Types and traits associated with masking lanes of vectors.
+//! Types representing
+#![allow(non_camel_case_types)]
+
+#[cfg_attr(
+ not(all(target_arch = "x86_64", target_feature = "avx512f")),
+ path = "masks/full_masks.rs"
+)]
+#[cfg_attr(
+ all(target_arch = "x86_64", target_feature = "avx512f"),
+ path = "masks/bitmask.rs"
+)]
+mod mask_impl;
+
+mod to_bitmask;
+pub use to_bitmask::ToBitMask;
+
- (value.lanes_eq(Simd::splat(0)) | value.lanes_eq(Simd::splat(-1))).all()
++#[cfg(feature = "generic_const_exprs")]
++pub use to_bitmask::{bitmask_len, ToBitMaskArray};
++
++use crate::simd::{intrinsics, LaneCount, Simd, SimdElement, SimdPartialEq, SupportedLaneCount};
+use core::cmp::Ordering;
+use core::{fmt, mem};
+
+mod sealed {
+ use super::*;
+
+ /// Not only does this seal the `MaskElement` trait, but these functions prevent other traits
+ /// from bleeding into the parent bounds.
+ ///
+ /// For example, `eq` could be provided by requiring `MaskElement: PartialEq`, but that would
+ /// prevent us from ever removing that bound, or from implementing `MaskElement` on
+ /// non-`PartialEq` types in the future.
+ pub trait Sealed {
+ fn valid<const LANES: usize>(values: Simd<Self, LANES>) -> bool
+ where
+ LaneCount<LANES>: SupportedLaneCount,
+ Self: SimdElement;
+
+ fn eq(self, other: Self) -> bool;
+
+ const TRUE: Self;
+
+ const FALSE: Self;
+ }
+}
+use sealed::Sealed;
+
+/// Marker trait for types that may be used as SIMD mask elements.
+///
+/// # Safety
+/// Type must be a signed integer.
+pub unsafe trait MaskElement: SimdElement + Sealed {}
+
+macro_rules! impl_element {
+ { $ty:ty } => {
+ impl Sealed for $ty {
+ fn valid<const LANES: usize>(value: Simd<Self, LANES>) -> bool
+ where
+ LaneCount<LANES>: SupportedLaneCount,
+ {
- /// Vector of eight 8-bit masks
++ (value.simd_eq(Simd::splat(0 as _)) | value.simd_eq(Simd::splat(-1 as _))).all()
+ }
+
+ fn eq(self, other: Self) -> bool { self == other }
+
+ const TRUE: Self = -1;
+ const FALSE: Self = 0;
+ }
+
++ // Safety: this is a valid mask element type
+ unsafe impl MaskElement for $ty {}
+ }
+}
+
+impl_element! { i8 }
+impl_element! { i16 }
+impl_element! { i32 }
+impl_element! { i64 }
+impl_element! { isize }
+
+/// A SIMD vector mask for `LANES` elements of width specified by `Element`.
+///
++/// Masks represent boolean inclusion/exclusion on a per-lane basis.
++///
+/// The layout of this type is unspecified.
+#[repr(transparent)]
+pub struct Mask<T, const LANES: usize>(mask_impl::Mask<T, LANES>)
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount;
+
+impl<T, const LANES: usize> Copy for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+impl<T, const LANES: usize> Clone for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<T, const LANES: usize> Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ /// Construct a mask by setting all lanes to the given value.
+ pub fn splat(value: bool) -> Self {
+ Self(mask_impl::Mask::splat(value))
+ }
+
+ /// Converts an array of bools to a SIMD mask.
+ pub fn from_array(array: [bool; LANES]) -> Self {
+ // SAFETY: Rust's bool has a layout of 1 byte (u8) with a value of
+ // true: 0b_0000_0001
+ // false: 0b_0000_0000
+ // Thus, an array of bools is also a valid array of bytes: [u8; N]
+ // This would be hypothetically valid as an "in-place" transmute,
+ // but these are "dependently-sized" types, so copy elision it is!
+ unsafe {
+ let bytes: [u8; LANES] = mem::transmute_copy(&array);
+ let bools: Simd<i8, LANES> =
+ intrinsics::simd_ne(Simd::from_array(bytes), Simd::splat(0u8));
+ Mask::from_int_unchecked(intrinsics::simd_cast(bools))
+ }
+ }
+
+ /// Converts a SIMD mask to an array of bools.
+ pub fn to_array(self) -> [bool; LANES] {
+ // This follows mostly the same logic as from_array.
+ // SAFETY: Rust's bool has a layout of 1 byte (u8) with a value of
+ // true: 0b_0000_0001
+ // false: 0b_0000_0000
+ // Thus, an array of bools is also a valid array of bytes: [u8; N]
+ // Since our masks are equal to integers where all bits are set,
+ // we can simply convert them to i8s, and then bitand them by the
+ // bitpattern for Rust's "true" bool.
+ // This would be hypothetically valid as an "in-place" transmute,
+ // but these are "dependently-sized" types, so copy elision it is!
+ unsafe {
+ let mut bytes: Simd<i8, LANES> = intrinsics::simd_cast(self.to_int());
+ bytes &= Simd::splat(1i8);
+ mem::transmute_copy(&bytes)
+ }
+ }
+
+ /// Converts a vector of integers to a mask, where 0 represents `false` and -1
+ /// represents `true`.
+ ///
+ /// # Safety
+ /// All lanes must be either 0 or -1.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub unsafe fn from_int_unchecked(value: Simd<T, LANES>) -> Self {
+ // Safety: the caller must confirm this invariant
+ unsafe { Self(mask_impl::Mask::from_int_unchecked(value)) }
+ }
+
+ /// Converts a vector of integers to a mask, where 0 represents `false` and -1
+ /// represents `true`.
+ ///
+ /// # Panics
+ /// Panics if any lane is not 0 or -1.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn from_int(value: Simd<T, LANES>) -> Self {
+ assert!(T::valid(value), "all values must be either 0 or -1",);
+ // Safety: the validity has been checked
+ unsafe { Self::from_int_unchecked(value) }
+ }
+
+ /// Converts the mask to a vector of integers, where 0 represents `false` and -1
+ /// represents `true`.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original value"]
+ pub fn to_int(self) -> Simd<T, LANES> {
+ self.0.to_int()
+ }
+
++ /// Converts the mask to a mask of any other lane size.
++ #[inline]
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ pub fn cast<U: MaskElement>(self) -> Mask<U, LANES> {
++ Mask(self.0.convert())
++ }
++
+ /// Tests the value of the specified lane.
+ ///
+ /// # Safety
+ /// `lane` must be less than `LANES`.
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub unsafe fn test_unchecked(&self, lane: usize) -> bool {
+ // Safety: the caller must confirm this invariant
+ unsafe { self.0.test_unchecked(lane) }
+ }
+
+ /// Tests the value of the specified lane.
+ ///
+ /// # Panics
+ /// Panics if `lane` is greater than or equal to the number of lanes in the vector.
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub fn test(&self, lane: usize) -> bool {
+ assert!(lane < LANES, "lane index out of range");
+ // Safety: the lane index has been checked
+ unsafe { self.test_unchecked(lane) }
+ }
+
+ /// Sets the value of the specified lane.
+ ///
+ /// # Safety
+ /// `lane` must be less than `LANES`.
+ #[inline]
+ pub unsafe fn set_unchecked(&mut self, lane: usize, value: bool) {
+ // Safety: the caller must confirm this invariant
+ unsafe {
+ self.0.set_unchecked(lane, value);
+ }
+ }
+
+ /// Sets the value of the specified lane.
+ ///
+ /// # Panics
+ /// Panics if `lane` is greater than or equal to the number of lanes in the vector.
+ #[inline]
+ pub fn set(&mut self, lane: usize, value: bool) {
+ assert!(lane < LANES, "lane index out of range");
+ // Safety: the lane index has been checked
+ unsafe {
+ self.set_unchecked(lane, value);
+ }
+ }
+
+ /// Returns true if any lane is set, or false otherwise.
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub fn any(self) -> bool {
+ self.0.any()
+ }
+
+ /// Returns true if all lanes are set, or false otherwise.
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub fn all(self) -> bool {
+ self.0.all()
+ }
+}
+
+// vector/array conversion
+impl<T, const LANES: usize> From<[bool; LANES]> for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn from(array: [bool; LANES]) -> Self {
+ Self::from_array(array)
+ }
+}
+
+impl<T, const LANES: usize> From<Mask<T, LANES>> for [bool; LANES]
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn from(vector: Mask<T, LANES>) -> Self {
+ vector.to_array()
+ }
+}
+
+impl<T, const LANES: usize> Default for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ #[must_use = "method returns a defaulted mask with all lanes set to false (0)"]
+ fn default() -> Self {
+ Self::splat(false)
+ }
+}
+
+impl<T, const LANES: usize> PartialEq for Mask<T, LANES>
+where
+ T: MaskElement + PartialEq,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ fn eq(&self, other: &Self) -> bool {
+ self.0 == other.0
+ }
+}
+
+impl<T, const LANES: usize> PartialOrd for Mask<T, LANES>
+where
+ T: MaskElement + PartialOrd,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ #[must_use = "method returns a new Ordering and does not mutate the original value"]
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ self.0.partial_cmp(&other.0)
+ }
+}
+
+impl<T, const LANES: usize> fmt::Debug for Mask<T, LANES>
+where
+ T: MaskElement + fmt::Debug,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_list()
+ .entries((0..LANES).map(|lane| self.test(lane)))
+ .finish()
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitAnd for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitand(self, rhs: Self) -> Self {
+ Self(self.0 & rhs.0)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitAnd<bool> for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitand(self, rhs: bool) -> Self {
+ self & Self::splat(rhs)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitAnd<Mask<T, LANES>> for bool
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Mask<T, LANES>;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitand(self, rhs: Mask<T, LANES>) -> Mask<T, LANES> {
+ Mask::splat(self) & rhs
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitOr for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitor(self, rhs: Self) -> Self {
+ Self(self.0 | rhs.0)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitOr<bool> for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitor(self, rhs: bool) -> Self {
+ self | Self::splat(rhs)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitOr<Mask<T, LANES>> for bool
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Mask<T, LANES>;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitor(self, rhs: Mask<T, LANES>) -> Mask<T, LANES> {
+ Mask::splat(self) | rhs
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitXor for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitxor(self, rhs: Self) -> Self::Output {
+ Self(self.0 ^ rhs.0)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitXor<bool> for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitxor(self, rhs: bool) -> Self::Output {
+ self ^ Self::splat(rhs)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitXor<Mask<T, LANES>> for bool
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Mask<T, LANES>;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitxor(self, rhs: Mask<T, LANES>) -> Self::Output {
+ Mask::splat(self) ^ rhs
+ }
+}
+
+impl<T, const LANES: usize> core::ops::Not for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Mask<T, LANES>;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn not(self) -> Self::Output {
+ Self(!self.0)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitAndAssign for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ fn bitand_assign(&mut self, rhs: Self) {
+ self.0 = self.0 & rhs.0;
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitAndAssign<bool> for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ fn bitand_assign(&mut self, rhs: bool) {
+ *self &= Self::splat(rhs);
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitOrAssign for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ fn bitor_assign(&mut self, rhs: Self) {
+ self.0 = self.0 | rhs.0;
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitOrAssign<bool> for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ fn bitor_assign(&mut self, rhs: bool) {
+ *self |= Self::splat(rhs);
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitXorAssign for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ fn bitxor_assign(&mut self, rhs: Self) {
+ self.0 = self.0 ^ rhs.0;
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitXorAssign<bool> for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ fn bitxor_assign(&mut self, rhs: bool) {
+ *self ^= Self::splat(rhs);
+ }
+}
+
- /// Vector of 16 8-bit masks
++/// A mask for SIMD vectors with eight elements of 8 bits.
+pub type mask8x8 = Mask<i8, 8>;
+
- /// Vector of 32 8-bit masks
++/// A mask for SIMD vectors with 16 elements of 8 bits.
+pub type mask8x16 = Mask<i8, 16>;
+
- /// Vector of 16 8-bit masks
++/// A mask for SIMD vectors with 32 elements of 8 bits.
+pub type mask8x32 = Mask<i8, 32>;
+
- /// Vector of four 16-bit masks
++/// A mask for SIMD vectors with 64 elements of 8 bits.
+pub type mask8x64 = Mask<i8, 64>;
+
- /// Vector of eight 16-bit masks
++/// A mask for SIMD vectors with four elements of 16 bits.
+pub type mask16x4 = Mask<i16, 4>;
+
- /// Vector of 16 16-bit masks
++/// A mask for SIMD vectors with eight elements of 16 bits.
+pub type mask16x8 = Mask<i16, 8>;
+
- /// Vector of 32 16-bit masks
++/// A mask for SIMD vectors with 16 elements of 16 bits.
+pub type mask16x16 = Mask<i16, 16>;
+
- /// Vector of two 32-bit masks
++/// A mask for SIMD vectors with 32 elements of 16 bits.
+pub type mask16x32 = Mask<i16, 32>;
+
- /// Vector of four 32-bit masks
++/// A mask for SIMD vectors with two elements of 32 bits.
+pub type mask32x2 = Mask<i32, 2>;
+
- /// Vector of eight 32-bit masks
++/// A mask for SIMD vectors with four elements of 32 bits.
+pub type mask32x4 = Mask<i32, 4>;
+
- /// Vector of 16 32-bit masks
++/// A mask for SIMD vectors with eight elements of 32 bits.
+pub type mask32x8 = Mask<i32, 8>;
+
- /// Vector of two 64-bit masks
++/// A mask for SIMD vectors with 16 elements of 32 bits.
+pub type mask32x16 = Mask<i32, 16>;
+
- /// Vector of four 64-bit masks
++/// A mask for SIMD vectors with two elements of 64 bits.
+pub type mask64x2 = Mask<i64, 2>;
+
- /// Vector of eight 64-bit masks
++/// A mask for SIMD vectors with four elements of 64 bits.
+pub type mask64x4 = Mask<i64, 4>;
+
- /// Vector of two pointer-width masks
++/// A mask for SIMD vectors with eight elements of 64 bits.
+pub type mask64x8 = Mask<i64, 8>;
+
- /// Vector of four pointer-width masks
++/// A mask for SIMD vectors with two elements of pointer width.
+pub type masksizex2 = Mask<isize, 2>;
+
- /// Vector of eight pointer-width masks
++/// A mask for SIMD vectors with four elements of pointer width.
+pub type masksizex4 = Mask<isize, 4>;
+
- Self(value.0.convert())
++/// A mask for SIMD vectors with eight elements of pointer width.
+pub type masksizex8 = Mask<isize, 8>;
+
+macro_rules! impl_from {
+ { $from:ty => $($to:ty),* } => {
+ $(
+ impl<const LANES: usize> From<Mask<$from, LANES>> for Mask<$to, LANES>
+ where
+ LaneCount<LANES>: SupportedLaneCount,
+ {
+ fn from(value: Mask<$from, LANES>) -> Self {
++ value.cast()
+ }
+ }
+ )*
+ }
+}
+impl_from! { i8 => i16, i32, i64, isize }
+impl_from! { i16 => i32, i64, isize, i8 }
+impl_from! { i32 => i64, isize, i8, i16 }
+impl_from! { i64 => isize, i8, i16, i32 }
+impl_from! { isize => i8, i16, i32, i64 }
--- /dev/null
+#![allow(unused_imports)]
+use super::MaskElement;
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Simd, SupportedLaneCount, ToBitMask};
+use core::marker::PhantomData;
+
+/// A mask where each lane is represented by a single bit.
+#[repr(transparent)]
+pub struct Mask<T, const LANES: usize>(
+ <LaneCount<LANES> as SupportedLaneCount>::BitMask,
+ PhantomData<T>,
+)
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount;
+
+impl<T, const LANES: usize> Copy for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+impl<T, const LANES: usize> Clone for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<T, const LANES: usize> PartialEq for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn eq(&self, other: &Self) -> bool {
+ self.0.as_ref() == other.0.as_ref()
+ }
+}
+
+impl<T, const LANES: usize> PartialOrd for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn partial_cmp(&self, other: &Self) -> Option<core::cmp::Ordering> {
+ self.0.as_ref().partial_cmp(other.0.as_ref())
+ }
+}
+
+impl<T, const LANES: usize> Eq for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+impl<T, const LANES: usize> Ord for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn cmp(&self, other: &Self) -> core::cmp::Ordering {
+ self.0.as_ref().cmp(other.0.as_ref())
+ }
+}
+
+impl<T, const LANES: usize> Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn splat(value: bool) -> Self {
+ let mut mask = <LaneCount<LANES> as SupportedLaneCount>::BitMask::default();
+ if value {
+ mask.as_mut().fill(u8::MAX)
+ } else {
+ mask.as_mut().fill(u8::MIN)
+ }
+ if LANES % 8 > 0 {
+ *mask.as_mut().last_mut().unwrap() &= u8::MAX >> (8 - LANES % 8);
+ }
+ Self(mask, PhantomData)
+ }
+
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub unsafe fn test_unchecked(&self, lane: usize) -> bool {
+ (self.0.as_ref()[lane / 8] >> (lane % 8)) & 0x1 > 0
+ }
+
+ #[inline]
+ pub unsafe fn set_unchecked(&mut self, lane: usize, value: bool) {
+ unsafe {
+ self.0.as_mut()[lane / 8] ^= ((value ^ self.test_unchecked(lane)) as u8) << (lane % 8)
+ }
+ }
+
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original value"]
+ pub fn to_int(self) -> Simd<T, LANES> {
+ unsafe {
+ intrinsics::simd_select_bitmask(self.0, Simd::splat(T::TRUE), Simd::splat(T::FALSE))
+ }
+ }
+
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub unsafe fn from_int_unchecked(value: Simd<T, LANES>) -> Self {
+ unsafe { Self(intrinsics::simd_bitmask(value), PhantomData) }
+ }
+
++ #[cfg(feature = "generic_const_exprs")]
++ #[inline]
++ #[must_use = "method returns a new array and does not mutate the original value"]
++ pub fn to_bitmask_array<const N: usize>(self) -> [u8; N] {
++ assert!(core::mem::size_of::<Self>() == N);
++
++ // Safety: converting an integer to an array of bytes of the same size is safe
++ unsafe { core::mem::transmute_copy(&self.0) }
++ }
++
++ #[cfg(feature = "generic_const_exprs")]
++ #[inline]
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ pub fn from_bitmask_array<const N: usize>(bitmask: [u8; N]) -> Self {
++ assert!(core::mem::size_of::<Self>() == N);
++
++ // Safety: converting an array of bytes to an integer of the same size is safe
++ Self(unsafe { core::mem::transmute_copy(&bitmask) }, PhantomData)
++ }
++
+ #[inline]
+ pub fn to_bitmask_integer<U>(self) -> U
+ where
+ super::Mask<T, LANES>: ToBitMask<BitMask = U>,
+ {
+ // Safety: these are the same types
+ unsafe { core::mem::transmute_copy(&self.0) }
+ }
+
+ #[inline]
+ pub fn from_bitmask_integer<U>(bitmask: U) -> Self
+ where
+ super::Mask<T, LANES>: ToBitMask<BitMask = U>,
+ {
+ // Safety: these are the same types
+ unsafe { Self(core::mem::transmute_copy(&bitmask), PhantomData) }
+ }
+
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn convert<U>(self) -> Mask<U, LANES>
+ where
+ U: MaskElement,
+ {
+ // Safety: bitmask layout does not depend on the element width
+ unsafe { core::mem::transmute_copy(&self) }
+ }
+
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub fn any(self) -> bool {
+ self != Self::splat(false)
+ }
+
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub fn all(self) -> bool {
+ self == Self::splat(true)
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitAnd for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+ <LaneCount<LANES> as SupportedLaneCount>::BitMask: AsRef<[u8]> + AsMut<[u8]>,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitand(mut self, rhs: Self) -> Self {
+ for (l, r) in self.0.as_mut().iter_mut().zip(rhs.0.as_ref().iter()) {
+ *l &= r;
+ }
+ self
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitOr for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+ <LaneCount<LANES> as SupportedLaneCount>::BitMask: AsRef<[u8]> + AsMut<[u8]>,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitor(mut self, rhs: Self) -> Self {
+ for (l, r) in self.0.as_mut().iter_mut().zip(rhs.0.as_ref().iter()) {
+ *l |= r;
+ }
+ self
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitXor for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitxor(mut self, rhs: Self) -> Self::Output {
+ for (l, r) in self.0.as_mut().iter_mut().zip(rhs.0.as_ref().iter()) {
+ *l ^= r;
+ }
+ self
+ }
+}
+
+impl<T, const LANES: usize> core::ops::Not for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn not(mut self) -> Self::Output {
+ for x in self.0.as_mut() {
+ *x = !*x;
+ }
+ if LANES % 8 > 0 {
+ *self.0.as_mut().last_mut().unwrap() &= u8::MAX >> (8 - LANES % 8);
+ }
+ self
+ }
+}
--- /dev/null
- fn reverse_bits(self) -> Self;
+//! Masks that take up full SIMD vector registers.
+
+use super::MaskElement;
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Simd, SupportedLaneCount, ToBitMask};
+
++#[cfg(feature = "generic_const_exprs")]
++use crate::simd::ToBitMaskArray;
++
+#[repr(transparent)]
+pub struct Mask<T, const LANES: usize>(Simd<T, LANES>)
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount;
+
+impl<T, const LANES: usize> Copy for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+impl<T, const LANES: usize> Clone for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<T, const LANES: usize> PartialEq for Mask<T, LANES>
+where
+ T: MaskElement + PartialEq,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn eq(&self, other: &Self) -> bool {
+ self.0.eq(&other.0)
+ }
+}
+
+impl<T, const LANES: usize> PartialOrd for Mask<T, LANES>
+where
+ T: MaskElement + PartialOrd,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn partial_cmp(&self, other: &Self) -> Option<core::cmp::Ordering> {
+ self.0.partial_cmp(&other.0)
+ }
+}
+
+impl<T, const LANES: usize> Eq for Mask<T, LANES>
+where
+ T: MaskElement + Eq,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+impl<T, const LANES: usize> Ord for Mask<T, LANES>
+where
+ T: MaskElement + Ord,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn cmp(&self, other: &Self) -> core::cmp::Ordering {
+ self.0.cmp(&other.0)
+ }
+}
+
+// Used for bitmask bit order workaround
+pub(crate) trait ReverseBits {
- fn reverse_bits(self) -> Self { <$int>::reverse_bits(self) }
++ // Reverse the least significant `n` bits of `self`.
++ // (Remaining bits must be 0.)
++ fn reverse_bits(self, n: usize) -> Self;
+}
+
+macro_rules! impl_reverse_bits {
+ { $($int:ty),* } => {
+ $(
+ impl ReverseBits for $int {
- bitmask.reverse_bits()
++ #[inline(always)]
++ fn reverse_bits(self, n: usize) -> Self {
++ let rev = <$int>::reverse_bits(self);
++ let bitsize = core::mem::size_of::<$int>() * 8;
++ if n < bitsize {
++ // Shift things back to the right
++ rev >> (bitsize - n)
++ } else {
++ rev
++ }
++ }
+ }
+ )*
+ }
+}
+
+impl_reverse_bits! { u8, u16, u32, u64 }
+
+impl<T, const LANES: usize> Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn splat(value: bool) -> Self {
+ Self(Simd::splat(if value { T::TRUE } else { T::FALSE }))
+ }
+
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub unsafe fn test_unchecked(&self, lane: usize) -> bool {
+ T::eq(self.0[lane], T::TRUE)
+ }
+
+ #[inline]
+ pub unsafe fn set_unchecked(&mut self, lane: usize, value: bool) {
+ self.0[lane] = if value { T::TRUE } else { T::FALSE }
+ }
+
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original value"]
+ pub fn to_int(self) -> Simd<T, LANES> {
+ self.0
+ }
+
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub unsafe fn from_int_unchecked(value: Simd<T, LANES>) -> Self {
+ Self(value)
+ }
+
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn convert<U>(self) -> Mask<U, LANES>
+ where
+ U: MaskElement,
+ {
+ // Safety: masks are simply integer vectors of 0 and -1, and we can cast the element type.
+ unsafe { Mask(intrinsics::simd_cast(self.0)) }
+ }
+
++ #[cfg(feature = "generic_const_exprs")]
++ #[inline]
++ #[must_use = "method returns a new array and does not mutate the original value"]
++ pub fn to_bitmask_array<const N: usize>(self) -> [u8; N]
++ where
++ super::Mask<T, LANES>: ToBitMaskArray,
++ [(); <super::Mask<T, LANES> as ToBitMaskArray>::BYTES]: Sized,
++ {
++ assert_eq!(<super::Mask<T, LANES> as ToBitMaskArray>::BYTES, N);
++
++ // Safety: N is the correct bitmask size
++ unsafe {
++ // Compute the bitmask
++ let bitmask: [u8; <super::Mask<T, LANES> as ToBitMaskArray>::BYTES] =
++ intrinsics::simd_bitmask(self.0);
++
++ // Transmute to the return type, previously asserted to be the same size
++ let mut bitmask: [u8; N] = core::mem::transmute_copy(&bitmask);
++
++ // LLVM assumes bit order should match endianness
++ if cfg!(target_endian = "big") {
++ for x in bitmask.as_mut() {
++ *x = x.reverse_bits();
++ }
++ };
++
++ bitmask
++ }
++ }
++
++ #[cfg(feature = "generic_const_exprs")]
++ #[inline]
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ pub fn from_bitmask_array<const N: usize>(mut bitmask: [u8; N]) -> Self
++ where
++ super::Mask<T, LANES>: ToBitMaskArray,
++ [(); <super::Mask<T, LANES> as ToBitMaskArray>::BYTES]: Sized,
++ {
++ assert_eq!(<super::Mask<T, LANES> as ToBitMaskArray>::BYTES, N);
++
++ // Safety: N is the correct bitmask size
++ unsafe {
++ // LLVM assumes bit order should match endianness
++ if cfg!(target_endian = "big") {
++ for x in bitmask.as_mut() {
++ *x = x.reverse_bits();
++ }
++ }
++
++ // Transmute to the bitmask type, previously asserted to be the same size
++ let bitmask: [u8; <super::Mask<T, LANES> as ToBitMaskArray>::BYTES] =
++ core::mem::transmute_copy(&bitmask);
++
++ // Compute the regular mask
++ Self::from_int_unchecked(intrinsics::simd_select_bitmask(
++ bitmask,
++ Self::splat(true).to_int(),
++ Self::splat(false).to_int(),
++ ))
++ }
++ }
++
+ #[inline]
+ pub(crate) fn to_bitmask_integer<U: ReverseBits>(self) -> U
+ where
+ super::Mask<T, LANES>: ToBitMask<BitMask = U>,
+ {
+ // Safety: U is required to be the appropriate bitmask type
+ let bitmask: U = unsafe { intrinsics::simd_bitmask(self.0) };
+
+ // LLVM assumes bit order should match endianness
+ if cfg!(target_endian = "big") {
- bitmask.reverse_bits()
++ bitmask.reverse_bits(LANES)
+ } else {
+ bitmask
+ }
+ }
+
+ #[inline]
+ pub(crate) fn from_bitmask_integer<U: ReverseBits>(bitmask: U) -> Self
+ where
+ super::Mask<T, LANES>: ToBitMask<BitMask = U>,
+ {
+ // LLVM assumes bit order should match endianness
+ let bitmask = if cfg!(target_endian = "big") {
++ bitmask.reverse_bits(LANES)
+ } else {
+ bitmask
+ };
+
+ // Safety: U is required to be the appropriate bitmask type
+ unsafe {
+ Self::from_int_unchecked(intrinsics::simd_select_bitmask(
+ bitmask,
+ Self::splat(true).to_int(),
+ Self::splat(false).to_int(),
+ ))
+ }
+ }
+
+ #[inline]
+ #[must_use = "method returns a new bool and does not mutate the original value"]
+ pub fn any(self) -> bool {
+ // Safety: use `self` as an integer vector
+ unsafe { intrinsics::simd_reduce_any(self.to_int()) }
+ }
+
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original value"]
+ pub fn all(self) -> bool {
+ // Safety: use `self` as an integer vector
+ unsafe { intrinsics::simd_reduce_all(self.to_int()) }
+ }
+}
+
+impl<T, const LANES: usize> core::convert::From<Mask<T, LANES>> for Simd<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn from(value: Mask<T, LANES>) -> Self {
+ value.0
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitAnd for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitand(self, rhs: Self) -> Self {
+ // Safety: `self` is an integer vector
+ unsafe { Self(intrinsics::simd_and(self.0, rhs.0)) }
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitOr for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitor(self, rhs: Self) -> Self {
+ // Safety: `self` is an integer vector
+ unsafe { Self(intrinsics::simd_or(self.0, rhs.0)) }
+ }
+}
+
+impl<T, const LANES: usize> core::ops::BitXor for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn bitxor(self, rhs: Self) -> Self {
+ // Safety: `self` is an integer vector
+ unsafe { Self(intrinsics::simd_xor(self.0, rhs.0)) }
+ }
+}
+
+impl<T, const LANES: usize> core::ops::Not for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ type Output = Self;
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ fn not(self) -> Self::Output {
+ Self::splat(true) ^ self
+ }
+}
--- /dev/null
- ///
- /// # Safety
- /// This trait is `unsafe` and sealed, since the `BitMask` type must match the number of lanes in
- /// the mask.
- pub unsafe trait ToBitMask: Sealed {
+use super::{mask_impl, Mask, MaskElement};
+use crate::simd::{LaneCount, SupportedLaneCount};
+
+mod sealed {
+ pub trait Sealed {}
+}
+pub use sealed::Sealed;
+
+impl<T, const LANES: usize> Sealed for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+/// Converts masks to and from integer bitmasks.
+///
+/// Each bit of the bitmask corresponds to a mask lane, starting with the LSB.
- { $(unsafe impl ToBitMask<BitMask=$int:ty> for Mask<_, $lanes:literal>)* } => {
++pub trait ToBitMask: Sealed {
+ /// The integer bitmask type.
+ type BitMask;
+
+ /// Converts a mask to a bitmask.
+ fn to_bitmask(self) -> Self::BitMask;
+
+ /// Converts a bitmask to a mask.
+ fn from_bitmask(bitmask: Self::BitMask) -> Self;
+}
+
++/// Converts masks to and from byte array bitmasks.
++///
++/// Each bit of the bitmask corresponds to a mask lane, starting with the LSB of the first byte.
++#[cfg(feature = "generic_const_exprs")]
++pub trait ToBitMaskArray: Sealed {
++ /// The length of the bitmask array.
++ const BYTES: usize;
++
++ /// Converts a mask to a bitmask.
++ fn to_bitmask_array(self) -> [u8; Self::BYTES];
++
++ /// Converts a bitmask to a mask.
++ fn from_bitmask_array(bitmask: [u8; Self::BYTES]) -> Self;
++}
++
+macro_rules! impl_integer_intrinsic {
- unsafe impl<T: MaskElement> ToBitMask for Mask<T, $lanes> {
++ { $(impl ToBitMask<BitMask=$int:ty> for Mask<_, $lanes:literal>)* } => {
+ $(
- unsafe impl ToBitMask<BitMask=u8> for Mask<_, 1>
- unsafe impl ToBitMask<BitMask=u8> for Mask<_, 2>
- unsafe impl ToBitMask<BitMask=u8> for Mask<_, 4>
- unsafe impl ToBitMask<BitMask=u8> for Mask<_, 8>
- unsafe impl ToBitMask<BitMask=u16> for Mask<_, 16>
- unsafe impl ToBitMask<BitMask=u32> for Mask<_, 32>
- unsafe impl ToBitMask<BitMask=u64> for Mask<_, 64>
++ impl<T: MaskElement> ToBitMask for Mask<T, $lanes> {
+ type BitMask = $int;
+
+ fn to_bitmask(self) -> $int {
+ self.0.to_bitmask_integer()
+ }
+
+ fn from_bitmask(bitmask: $int) -> Self {
+ Self(mask_impl::Mask::from_bitmask_integer(bitmask))
+ }
+ }
+ )*
+ }
+}
+
+impl_integer_intrinsic! {
++ impl ToBitMask<BitMask=u8> for Mask<_, 1>
++ impl ToBitMask<BitMask=u8> for Mask<_, 2>
++ impl ToBitMask<BitMask=u8> for Mask<_, 4>
++ impl ToBitMask<BitMask=u8> for Mask<_, 8>
++ impl ToBitMask<BitMask=u16> for Mask<_, 16>
++ impl ToBitMask<BitMask=u32> for Mask<_, 32>
++ impl ToBitMask<BitMask=u64> for Mask<_, 64>
++}
++
++/// Returns the minimum numnber of bytes in a bitmask with `lanes` lanes.
++#[cfg(feature = "generic_const_exprs")]
++pub const fn bitmask_len(lanes: usize) -> usize {
++ (lanes + 7) / 8
++}
++
++#[cfg(feature = "generic_const_exprs")]
++impl<T: MaskElement, const LANES: usize> ToBitMaskArray for Mask<T, LANES>
++where
++ LaneCount<LANES>: SupportedLaneCount,
++{
++ const BYTES: usize = bitmask_len(LANES);
++
++ fn to_bitmask_array(self) -> [u8; Self::BYTES] {
++ self.0.to_bitmask_array()
++ }
++
++ fn from_bitmask_array(bitmask: [u8; Self::BYTES]) -> Self {
++ Mask(mask_impl::Mask::from_bitmask_array(bitmask))
++ }
+}
--- /dev/null
- #[macro_use]
- mod reduction;
-
+#[macro_use]
+mod swizzle;
+
+pub(crate) mod intrinsics;
+
+#[cfg(feature = "generic_const_exprs")]
+mod to_bytes;
+
- mod comparisons;
++mod elements;
++mod eq;
+mod fmt;
+mod iter;
+mod lane_count;
+mod masks;
- mod math;
+mod ops;
- mod round;
++mod ord;
+mod select;
+mod vector;
+mod vendor;
+
+#[doc = include_str!("core_simd_docs.md")]
+pub mod simd {
+ pub(crate) use crate::core_simd::intrinsics;
+
++ pub use crate::core_simd::elements::*;
++ pub use crate::core_simd::eq::*;
+ pub use crate::core_simd::lane_count::{LaneCount, SupportedLaneCount};
+ pub use crate::core_simd::masks::*;
++ pub use crate::core_simd::ord::*;
+ pub use crate::core_simd::swizzle::*;
+ pub use crate::core_simd::vector::*;
+}
--- /dev/null
- use crate::simd::{LaneCount, Simd, SimdElement, SupportedLaneCount};
++use crate::simd::{LaneCount, Simd, SimdElement, SimdPartialEq, SupportedLaneCount};
+use core::ops::{Add, Mul};
+use core::ops::{BitAnd, BitOr, BitXor};
+use core::ops::{Div, Rem, Sub};
+use core::ops::{Shl, Shr};
+
+mod assign;
+mod deref;
+mod unary;
+
+impl<I, T, const LANES: usize> core::ops::Index<I> for Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+ I: core::slice::SliceIndex<[T]>,
+{
+ type Output = I::Output;
+ fn index(&self, index: I) -> &Self::Output {
+ &self.as_array()[index]
+ }
+}
+
+impl<I, T, const LANES: usize> core::ops::IndexMut<I> for Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+ I: core::slice::SliceIndex<[T]>,
+{
+ fn index_mut(&mut self, index: I) -> &mut Self::Output {
+ &mut self.as_mut_array()[index]
+ }
+}
+
+macro_rules! unsafe_base {
+ ($lhs:ident, $rhs:ident, {$simd_call:ident}, $($_:tt)*) => {
++ // Safety: $lhs and $rhs are vectors
+ unsafe { $crate::simd::intrinsics::$simd_call($lhs, $rhs) }
+ };
+}
+
+/// SAFETY: This macro should not be used for anything except Shl or Shr, and passed the appropriate shift intrinsic.
+/// It handles performing a bitand in addition to calling the shift operator, so that the result
+/// is well-defined: LLVM can return a poison value if you shl, lshr, or ashr if rhs >= <Int>::BITS
+/// At worst, this will maybe add another instruction and cycle,
+/// at best, it may open up more optimization opportunities,
+/// or simply be elided entirely, especially for SIMD ISAs which default to this.
+///
+// FIXME: Consider implementing this in cg_llvm instead?
+// cg_clif defaults to this, and scalar MIR shifts also default to wrapping
+macro_rules! wrap_bitshift {
+ ($lhs:ident, $rhs:ident, {$simd_call:ident}, $int:ident) => {
++ #[allow(clippy::suspicious_arithmetic_impl)]
++ // Safety: $lhs and the bitand result are vectors
+ unsafe {
+ $crate::simd::intrinsics::$simd_call(
+ $lhs,
+ $rhs.bitand(Simd::splat(<$int>::BITS as $int - 1)),
+ )
+ }
+ };
+}
+
+/// SAFETY: This macro must only be used to impl Div or Rem and given the matching intrinsic.
+/// It guards against LLVM's UB conditions for integer div or rem using masks and selects,
+/// thus guaranteeing a Rust value returns instead.
+///
+/// | | LLVM | Rust
+/// | :--------------: | :--- | :----------
+/// | N {/,%} 0 | UB | panic!()
+/// | <$int>::MIN / -1 | UB | <$int>::MIN
+/// | <$int>::MIN % -1 | UB | 0
+///
+macro_rules! int_divrem_guard {
+ ( $lhs:ident,
+ $rhs:ident,
+ { const PANIC_ZERO: &'static str = $zero:literal;
+ $simd_call:ident
+ },
+ $int:ident ) => {
- if $rhs.lanes_eq(Simd::splat(0)).any() {
++ if $rhs.simd_eq(Simd::splat(0 as _)).any() {
+ panic!($zero);
+ } else {
+ // Prevent otherwise-UB overflow on the MIN / -1 case.
+ let rhs = if <$int>::MIN != 0 {
+ // This should, at worst, optimize to a few branchless logical ops
+ // Ideally, this entire conditional should evaporate
+ // Fire LLVM and implement those manually if it doesn't get the hint
- ($lhs.lanes_eq(Simd::splat(<$int>::MIN))
++ ($lhs.simd_eq(Simd::splat(<$int>::MIN))
+ // type inference can break here, so cut an SInt to size
- & $rhs.lanes_eq(Simd::splat(-1i64 as _)))
- .select(Simd::splat(1), $rhs)
++ & $rhs.simd_eq(Simd::splat(-1i64 as _)))
++ .select(Simd::splat(1 as _), $rhs)
+ } else {
+ // Nice base case to make it easy to const-fold away the other branch.
+ $rhs
+ };
++ // Safety: $lhs and rhs are vectors
+ unsafe { $crate::simd::intrinsics::$simd_call($lhs, rhs) }
+ }
+ };
+}
+
+macro_rules! for_base_types {
+ ( T = ($($scalar:ident),*);
+ type Lhs = Simd<T, N>;
+ type Rhs = Simd<T, N>;
+ type Output = $out:ty;
+
+ impl $op:ident::$call:ident {
+ $macro_impl:ident $inner:tt
+ }) => {
+ $(
+ impl<const N: usize> $op<Self> for Simd<$scalar, N>
+ where
+ $scalar: SimdElement,
+ LaneCount<N>: SupportedLaneCount,
+ {
+ type Output = $out;
+
+ #[inline]
+ #[must_use = "operator returns a new vector without mutating the inputs"]
+ fn $call(self, rhs: Self) -> Self::Output {
+ $macro_impl!(self, rhs, $inner, $scalar)
+ }
+ })*
+ }
+}
+
+// A "TokenTree muncher": takes a set of scalar types `T = {};`
+// type parameters for the ops it implements, `Op::fn` names,
+// and a macro that expands into an expr, substituting in an intrinsic.
+// It passes that to for_base_types, which expands an impl for the types,
+// using the expanded expr in the function, and recurses with itself.
+//
+// tl;dr impls a set of ops::{Traits} for a set of types
+macro_rules! for_base_ops {
+ (
+ T = $types:tt;
+ type Lhs = Simd<T, N>;
+ type Rhs = Simd<T, N>;
+ type Output = $out:ident;
+ impl $op:ident::$call:ident
+ $inner:tt
+ $($rest:tt)*
+ ) => {
+ for_base_types! {
+ T = $types;
+ type Lhs = Simd<T, N>;
+ type Rhs = Simd<T, N>;
+ type Output = $out;
+ impl $op::$call
+ $inner
+ }
+ for_base_ops! {
+ T = $types;
+ type Lhs = Simd<T, N>;
+ type Rhs = Simd<T, N>;
+ type Output = $out;
+ $($rest)*
+ }
+ };
+ ($($done:tt)*) => {
+ // Done.
+ }
+}
+
+// Integers can always accept add, mul, sub, bitand, bitor, and bitxor.
+// For all of these operations, simd_* intrinsics apply wrapping logic.
+for_base_ops! {
+ T = (i8, i16, i32, i64, isize, u8, u16, u32, u64, usize);
+ type Lhs = Simd<T, N>;
+ type Rhs = Simd<T, N>;
+ type Output = Self;
+
+ impl Add::add {
+ unsafe_base { simd_add }
+ }
+
+ impl Mul::mul {
+ unsafe_base { simd_mul }
+ }
+
+ impl Sub::sub {
+ unsafe_base { simd_sub }
+ }
+
+ impl BitAnd::bitand {
+ unsafe_base { simd_and }
+ }
+
+ impl BitOr::bitor {
+ unsafe_base { simd_or }
+ }
+
+ impl BitXor::bitxor {
+ unsafe_base { simd_xor }
+ }
+
+ impl Div::div {
+ int_divrem_guard {
+ const PANIC_ZERO: &'static str = "attempt to divide by zero";
+ simd_div
+ }
+ }
+
+ impl Rem::rem {
+ int_divrem_guard {
+ const PANIC_ZERO: &'static str = "attempt to calculate the remainder with a divisor of zero";
+ simd_rem
+ }
+ }
+
+ // The only question is how to handle shifts >= <Int>::BITS?
+ // Our current solution uses wrapping logic.
+ impl Shl::shl {
+ wrap_bitshift { simd_shl }
+ }
+
+ impl Shr::shr {
+ wrap_bitshift {
+ // This automatically monomorphizes to lshr or ashr, depending,
+ // so it's fine to use it for both UInts and SInts.
+ simd_shr
+ }
+ }
+}
+
+// We don't need any special precautions here:
+// Floats always accept arithmetic ops, but may become NaN.
+for_base_ops! {
+ T = (f32, f64);
+ type Lhs = Simd<T, N>;
+ type Rhs = Simd<T, N>;
+ type Output = Self;
+
+ impl Add::add {
+ unsafe_base { simd_add }
+ }
+
+ impl Mul::mul {
+ unsafe_base { simd_mul }
+ }
+
+ impl Sub::sub {
+ unsafe_base { simd_sub }
+ }
+
+ impl Div::div {
+ unsafe_base { simd_div }
+ }
+
+ impl Rem::rem {
+ unsafe_base { simd_rem }
+ }
+}
--- /dev/null
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Simd, SimdElement, SupportedLaneCount};
+use core::ops::{Neg, Not}; // unary ops
+
+macro_rules! neg {
+ ($(impl<const LANES: usize> Neg for Simd<$scalar:ty, LANES>)*) => {
+ $(impl<const LANES: usize> Neg for Simd<$scalar, LANES>
+ where
+ $scalar: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+ {
+ type Output = Self;
+
+ #[inline]
+ #[must_use = "operator returns a new vector without mutating the input"]
+ fn neg(self) -> Self::Output {
++ // Safety: `self` is a signed vector
+ unsafe { intrinsics::simd_neg(self) }
+ }
+ })*
+ }
+}
+
+neg! {
+ impl<const LANES: usize> Neg for Simd<f32, LANES>
+
+ impl<const LANES: usize> Neg for Simd<f64, LANES>
+
+ impl<const LANES: usize> Neg for Simd<i8, LANES>
+
+ impl<const LANES: usize> Neg for Simd<i16, LANES>
+
+ impl<const LANES: usize> Neg for Simd<i32, LANES>
+
+ impl<const LANES: usize> Neg for Simd<i64, LANES>
+
+ impl<const LANES: usize> Neg for Simd<isize, LANES>
+}
+
+macro_rules! not {
+ ($(impl<const LANES: usize> Not for Simd<$scalar:ty, LANES>)*) => {
+ $(impl<const LANES: usize> Not for Simd<$scalar, LANES>
+ where
+ $scalar: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+ {
+ type Output = Self;
+
+ #[inline]
+ #[must_use = "operator returns a new vector without mutating the input"]
+ fn not(self) -> Self::Output {
+ self ^ (Simd::splat(!(0 as $scalar)))
+ }
+ })*
+ }
+}
+
+not! {
+ impl<const LANES: usize> Not for Simd<i8, LANES>
+
+ impl<const LANES: usize> Not for Simd<i16, LANES>
+
+ impl<const LANES: usize> Not for Simd<i32, LANES>
+
+ impl<const LANES: usize> Not for Simd<i64, LANES>
+
+ impl<const LANES: usize> Not for Simd<isize, LANES>
+
+ impl<const LANES: usize> Not for Simd<u8, LANES>
+
+ impl<const LANES: usize> Not for Simd<u16, LANES>
+
+ impl<const LANES: usize> Not for Simd<u32, LANES>
+
+ impl<const LANES: usize> Not for Simd<u64, LANES>
+
+ impl<const LANES: usize> Not for Simd<usize, LANES>
+}
--- /dev/null
--- /dev/null
++use crate::simd::{intrinsics, LaneCount, Mask, Simd, SimdPartialEq, SupportedLaneCount};
++
++/// Parallel `PartialOrd`.
++pub trait SimdPartialOrd: SimdPartialEq {
++ /// Test if each lane is less than the corresponding lane in `other`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn simd_lt(self, other: Self) -> Self::Mask;
++
++ /// Test if each lane is less than or equal to the corresponding lane in `other`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn simd_le(self, other: Self) -> Self::Mask;
++
++ /// Test if each lane is greater than the corresponding lane in `other`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn simd_gt(self, other: Self) -> Self::Mask;
++
++ /// Test if each lane is greater than or equal to the corresponding lane in `other`.
++ #[must_use = "method returns a new mask and does not mutate the original value"]
++ fn simd_ge(self, other: Self) -> Self::Mask;
++}
++
++/// Parallel `Ord`.
++pub trait SimdOrd: SimdPartialOrd {
++ /// Returns the lane-wise maximum with `other`.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn simd_max(self, other: Self) -> Self;
++
++ /// Returns the lane-wise minimum with `other`.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn simd_min(self, other: Self) -> Self;
++
++ /// Restrict each lane to a certain interval.
++ ///
++ /// For each lane, returns `max` if `self` is greater than `max`, and `min` if `self` is
++ /// less than `min`. Otherwise returns `self`.
++ ///
++ /// # Panics
++ ///
++ /// Panics if `min > max` on any lane.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ fn simd_clamp(self, min: Self, max: Self) -> Self;
++}
++
++macro_rules! impl_integer {
++ { $($integer:ty),* } => {
++ $(
++ impl<const LANES: usize> SimdPartialOrd for Simd<$integer, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ #[inline]
++ fn simd_lt(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_lt(self, other)) }
++ }
++
++ #[inline]
++ fn simd_le(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_le(self, other)) }
++ }
++
++ #[inline]
++ fn simd_gt(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_gt(self, other)) }
++ }
++
++ #[inline]
++ fn simd_ge(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_ge(self, other)) }
++ }
++ }
++
++ impl<const LANES: usize> SimdOrd for Simd<$integer, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ #[inline]
++ fn simd_max(self, other: Self) -> Self {
++ self.simd_lt(other).select(other, self)
++ }
++
++ #[inline]
++ fn simd_min(self, other: Self) -> Self {
++ self.simd_gt(other).select(other, self)
++ }
++
++ #[inline]
++ fn simd_clamp(self, min: Self, max: Self) -> Self {
++ assert!(
++ min.simd_le(max).all(),
++ "each lane in `min` must be less than or equal to the corresponding lane in `max`",
++ );
++ self.simd_max(min).simd_min(max)
++ }
++ }
++ )*
++ }
++}
++
++impl_integer! { u8, u16, u32, u64, usize, i8, i16, i32, i64, isize }
++
++macro_rules! impl_float {
++ { $($float:ty),* } => {
++ $(
++ impl<const LANES: usize> SimdPartialOrd for Simd<$float, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ #[inline]
++ fn simd_lt(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_lt(self, other)) }
++ }
++
++ #[inline]
++ fn simd_le(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_le(self, other)) }
++ }
++
++ #[inline]
++ fn simd_gt(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_gt(self, other)) }
++ }
++
++ #[inline]
++ fn simd_ge(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Mask::from_int_unchecked(intrinsics::simd_ge(self, other)) }
++ }
++ }
++ )*
++ }
++}
++
++impl_float! { f32, f64 }
++
++macro_rules! impl_mask {
++ { $($integer:ty),* } => {
++ $(
++ impl<const LANES: usize> SimdPartialOrd for Mask<$integer, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ #[inline]
++ fn simd_lt(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Self::from_int_unchecked(intrinsics::simd_lt(self.to_int(), other.to_int())) }
++ }
++
++ #[inline]
++ fn simd_le(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Self::from_int_unchecked(intrinsics::simd_le(self.to_int(), other.to_int())) }
++ }
++
++ #[inline]
++ fn simd_gt(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Self::from_int_unchecked(intrinsics::simd_gt(self.to_int(), other.to_int())) }
++ }
++
++ #[inline]
++ fn simd_ge(self, other: Self) -> Self::Mask {
++ // Safety: `self` is a vector, and the result of the comparison
++ // is always a valid mask.
++ unsafe { Self::from_int_unchecked(intrinsics::simd_ge(self.to_int(), other.to_int())) }
++ }
++ }
++
++ impl<const LANES: usize> SimdOrd for Mask<$integer, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ #[inline]
++ fn simd_max(self, other: Self) -> Self {
++ self.simd_gt(other).select_mask(other, self)
++ }
++
++ #[inline]
++ fn simd_min(self, other: Self) -> Self {
++ self.simd_lt(other).select_mask(other, self)
++ }
++
++ #[inline]
++ fn simd_clamp(self, min: Self, max: Self) -> Self {
++ assert!(
++ min.simd_le(max).all(),
++ "each lane in `min` must be less than or equal to the corresponding lane in `max`",
++ );
++ self.simd_max(min).simd_min(max)
++ }
++ }
++ )*
++ }
++}
++
++impl_mask! { i8, i16, i32, i64, isize }
--- /dev/null
- /// Constructs a new vector by selecting values from the lanes of the source vector or vectors to use.
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Simd, SimdElement, SupportedLaneCount};
+
- /// When swizzling one vector, the indices of the result vector are indicated by a `const` array
- /// of `usize`, like [`Swizzle`].
- /// When swizzling two vectors, the indices are indicated by a `const` array of [`Which`], like
- /// [`Swizzle2`].
++/// Constructs a new SIMD vector by copying elements from selected lanes in other vectors.
+///
- /// ## One source vector
++/// When swizzling one vector, lanes are selected by a `const` array of `usize`,
++/// like [`Swizzle`].
++///
++/// When swizzling two vectors, lanes are selected by a `const` array of [`Which`],
++/// like [`Swizzle2`].
+///
+/// # Examples
- /// # use core::simd::{Simd, simd_swizzle};
- /// let v = Simd::<f32, 4>::from_array([0., 1., 2., 3.]);
++///
++/// With a single SIMD vector, the const array specifies lane indices in that vector:
+/// ```
+/// # #![feature(portable_simd)]
- /// let r = simd_swizzle!(v, [3, 0, 1, 2]);
- /// assert_eq!(r.to_array(), [3., 0., 1., 2.]);
++/// # use core::simd::{u32x2, u32x4, simd_swizzle};
++/// let v = u32x4::from_array([10, 11, 12, 13]);
+///
+/// // Keeping the same size
- /// let r = simd_swizzle!(v, [3, 1]);
- /// assert_eq!(r.to_array(), [3., 1.]);
++/// let r: u32x4 = simd_swizzle!(v, [3, 0, 1, 2]);
++/// assert_eq!(r.to_array(), [13, 10, 11, 12]);
+///
+/// // Changing the number of lanes
- /// ## Two source vectors
++/// let r: u32x2 = simd_swizzle!(v, [3, 1]);
++/// assert_eq!(r.to_array(), [13, 11]);
+/// ```
+///
- /// # use core::simd::{Simd, simd_swizzle, Which};
- /// use Which::*;
- /// let a = Simd::<f32, 4>::from_array([0., 1., 2., 3.]);
- /// let b = Simd::<f32, 4>::from_array([4., 5., 6., 7.]);
++/// With two input SIMD vectors, the const array uses `Which` to specify the source of each index:
+/// ```
+/// # #![feature(portable_simd)]
- /// let r = simd_swizzle!(a, b, [First(0), First(1), Second(2), Second(3)]);
- /// assert_eq!(r.to_array(), [0., 1., 6., 7.]);
++/// # use core::simd::{u32x2, u32x4, simd_swizzle, Which};
++/// use Which::{First, Second};
++/// let a = u32x4::from_array([0, 1, 2, 3]);
++/// let b = u32x4::from_array([4, 5, 6, 7]);
+///
+/// // Keeping the same size
- /// let r = simd_swizzle!(a, b, [First(0), Second(0)]);
- /// assert_eq!(r.to_array(), [0., 4.]);
++/// let r: u32x4 = simd_swizzle!(a, b, [First(0), First(1), Second(2), Second(3)]);
++/// assert_eq!(r.to_array(), [0, 1, 6, 7]);
+///
+/// // Changing the number of lanes
- /// An index into one of two vectors.
++/// let r: u32x2 = simd_swizzle!(a, b, [First(0), Second(0)]);
++/// assert_eq!(r.to_array(), [0, 4]);
+/// ```
+#[allow(unused_macros)]
+pub macro simd_swizzle {
+ (
+ $vector:expr, $index:expr $(,)?
+ ) => {
+ {
+ use $crate::simd::Swizzle;
+ struct Impl;
+ impl<const LANES: usize> Swizzle<LANES, {$index.len()}> for Impl {
+ const INDEX: [usize; {$index.len()}] = $index;
+ }
+ Impl::swizzle($vector)
+ }
+ },
+ (
+ $first:expr, $second:expr, $index:expr $(,)?
+ ) => {
+ {
+ use $crate::simd::{Which, Swizzle2};
+ struct Impl;
+ impl<const LANES: usize> Swizzle2<LANES, {$index.len()}> for Impl {
+ const INDEX: [Which; {$index.len()}] = $index;
+ }
+ Impl::swizzle2($first, $second)
+ }
+ }
+}
+
- /// Indexes the first vector.
++/// Specifies a lane index into one of two SIMD vectors.
++///
++/// This is an input type for [Swizzle2] and helper macros like [simd_swizzle].
+#[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub enum Which {
- /// Indexes the second vector.
++ /// Index of a lane in the first input SIMD vector.
+ First(usize),
++ /// Index of a lane in the second input SIMD vector.
+ Second(usize),
+}
+
+/// Create a vector from the elements of another vector.
+pub trait Swizzle<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ /// Map from the lanes of the input vector to the output vector.
+ const INDEX: [usize; OUTPUT_LANES];
+
+ /// Create a new vector from the lanes of `vector`.
+ ///
+ /// Lane `i` of the output is `vector[Self::INDEX[i]]`.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ fn swizzle<T>(vector: Simd<T, INPUT_LANES>) -> Simd<T, OUTPUT_LANES>
+ where
+ T: SimdElement,
+ LaneCount<INPUT_LANES>: SupportedLaneCount,
+ LaneCount<OUTPUT_LANES>: SupportedLaneCount,
+ {
+ // Safety: `vector` is a vector, and `INDEX_IMPL` is a const array of u32.
+ unsafe { intrinsics::simd_shuffle(vector, vector, Self::INDEX_IMPL) }
+ }
+}
+
+/// Create a vector from the elements of two other vectors.
+pub trait Swizzle2<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ /// Map from the lanes of the input vectors to the output vector
+ const INDEX: [Which; OUTPUT_LANES];
+
+ /// Create a new vector from the lanes of `first` and `second`.
+ ///
+ /// Lane `i` is `first[j]` when `Self::INDEX[i]` is `First(j)`, or `second[j]` when it is
+ /// `Second(j)`.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ fn swizzle2<T>(
+ first: Simd<T, INPUT_LANES>,
+ second: Simd<T, INPUT_LANES>,
+ ) -> Simd<T, OUTPUT_LANES>
+ where
+ T: SimdElement,
+ LaneCount<INPUT_LANES>: SupportedLaneCount,
+ LaneCount<OUTPUT_LANES>: SupportedLaneCount,
+ {
+ // Safety: `first` and `second` are vectors, and `INDEX_IMPL` is a const array of u32.
+ unsafe { intrinsics::simd_shuffle(first, second, Self::INDEX_IMPL) }
+ }
+}
+
+/// The `simd_shuffle` intrinsic expects `u32`, so do error checking and conversion here.
+/// This trait hides `INDEX_IMPL` from the public API.
+trait SwizzleImpl<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ const INDEX_IMPL: [u32; OUTPUT_LANES];
+}
+
+impl<T, const INPUT_LANES: usize, const OUTPUT_LANES: usize> SwizzleImpl<INPUT_LANES, OUTPUT_LANES>
+ for T
+where
+ T: Swizzle<INPUT_LANES, OUTPUT_LANES> + ?Sized,
+{
+ const INDEX_IMPL: [u32; OUTPUT_LANES] = {
+ let mut output = [0; OUTPUT_LANES];
+ let mut i = 0;
+ while i < OUTPUT_LANES {
+ let index = Self::INDEX[i];
+ assert!(index as u32 as usize == index);
+ assert!(index < INPUT_LANES, "source lane exceeds input lane count",);
+ output[i] = index as u32;
+ i += 1;
+ }
+ output
+ };
+}
+
+/// The `simd_shuffle` intrinsic expects `u32`, so do error checking and conversion here.
+/// This trait hides `INDEX_IMPL` from the public API.
+trait Swizzle2Impl<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ const INDEX_IMPL: [u32; OUTPUT_LANES];
+}
+
+impl<T, const INPUT_LANES: usize, const OUTPUT_LANES: usize> Swizzle2Impl<INPUT_LANES, OUTPUT_LANES>
+ for T
+where
+ T: Swizzle2<INPUT_LANES, OUTPUT_LANES> + ?Sized,
+{
+ const INDEX_IMPL: [u32; OUTPUT_LANES] = {
+ let mut output = [0; OUTPUT_LANES];
+ let mut i = 0;
+ while i < OUTPUT_LANES {
+ let (offset, index) = match Self::INDEX[i] {
+ Which::First(index) => (false, index),
+ Which::Second(index) => (true, index),
+ };
+ assert!(index < INPUT_LANES, "source lane exceeds input lane count",);
+
+ // lanes are indexed by the first vector, then second vector
+ let index = if offset { index + INPUT_LANES } else { index };
+ assert!(index as u32 as usize == index);
+ output[i] = index as u32;
+ i += 1;
+ }
+ output
+ };
+}
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ /// Reverse the order of the lanes in the vector.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn reverse(self) -> Self {
+ const fn reverse_index<const LANES: usize>() -> [usize; LANES] {
+ let mut index = [0; LANES];
+ let mut i = 0;
+ while i < LANES {
+ index[i] = LANES - i - 1;
+ i += 1;
+ }
+ index
+ }
+
+ struct Reverse;
+
+ impl<const LANES: usize> Swizzle<LANES, LANES> for Reverse {
+ const INDEX: [usize; LANES] = reverse_index::<LANES>();
+ }
+
+ Reverse::swizzle(self)
+ }
+
+ /// Rotates the vector such that the first `OFFSET` elements of the slice move to the end
+ /// while the last `LANES - OFFSET` elements move to the front. After calling `rotate_lanes_left`,
+ /// the element previously in lane `OFFSET` will become the first element in the slice.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn rotate_lanes_left<const OFFSET: usize>(self) -> Self {
+ const fn rotate_index<const OFFSET: usize, const LANES: usize>() -> [usize; LANES] {
+ let offset = OFFSET % LANES;
+ let mut index = [0; LANES];
+ let mut i = 0;
+ while i < LANES {
+ index[i] = (i + offset) % LANES;
+ i += 1;
+ }
+ index
+ }
+
+ struct Rotate<const OFFSET: usize>;
+
+ impl<const OFFSET: usize, const LANES: usize> Swizzle<LANES, LANES> for Rotate<OFFSET> {
+ const INDEX: [usize; LANES] = rotate_index::<OFFSET, LANES>();
+ }
+
+ Rotate::<OFFSET>::swizzle(self)
+ }
+
+ /// Rotates the vector such that the first `LANES - OFFSET` elements of the vector move to
+ /// the end while the last `OFFSET` elements move to the front. After calling `rotate_lanes_right`,
+ /// the element previously at index `LANES - OFFSET` will become the first element in the slice.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn rotate_lanes_right<const OFFSET: usize>(self) -> Self {
+ const fn rotate_index<const OFFSET: usize, const LANES: usize>() -> [usize; LANES] {
+ let offset = LANES - OFFSET % LANES;
+ let mut index = [0; LANES];
+ let mut i = 0;
+ while i < LANES {
+ index[i] = (i + offset) % LANES;
+ i += 1;
+ }
+ index
+ }
+
+ struct Rotate<const OFFSET: usize>;
+
+ impl<const OFFSET: usize, const LANES: usize> Swizzle<LANES, LANES> for Rotate<OFFSET> {
+ const INDEX: [usize; LANES] = rotate_index::<OFFSET, LANES>();
+ }
+
+ Rotate::<OFFSET>::swizzle(self)
+ }
+
+ /// Interleave two vectors.
+ ///
+ /// Produces two vectors with lanes taken alternately from `self` and `other`.
+ ///
+ /// The first result contains the first `LANES / 2` lanes from `self` and `other`,
+ /// alternating, starting with the first lane of `self`.
+ ///
+ /// The second result contains the last `LANES / 2` lanes from `self` and `other`,
+ /// alternating, starting with the lane `LANES / 2` from the start of `self`.
+ ///
+ /// ```
+ /// #![feature(portable_simd)]
+ /// # use core::simd::Simd;
+ /// let a = Simd::from_array([0, 1, 2, 3]);
+ /// let b = Simd::from_array([4, 5, 6, 7]);
+ /// let (x, y) = a.interleave(b);
+ /// assert_eq!(x.to_array(), [0, 4, 1, 5]);
+ /// assert_eq!(y.to_array(), [2, 6, 3, 7]);
+ /// ```
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn interleave(self, other: Self) -> (Self, Self) {
+ const fn lo<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES {
+ let offset = i / 2;
+ idx[i] = if i % 2 == 0 {
+ Which::First(offset)
+ } else {
+ Which::Second(offset)
+ };
+ i += 1;
+ }
+ idx
+ }
+ const fn hi<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES {
+ let offset = (LANES + i) / 2;
+ idx[i] = if i % 2 == 0 {
+ Which::First(offset)
+ } else {
+ Which::Second(offset)
+ };
+ i += 1;
+ }
+ idx
+ }
+
+ struct Lo;
+ struct Hi;
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Lo {
+ const INDEX: [Which; LANES] = lo::<LANES>();
+ }
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Hi {
+ const INDEX: [Which; LANES] = hi::<LANES>();
+ }
+
+ (Lo::swizzle2(self, other), Hi::swizzle2(self, other))
+ }
+
+ /// Deinterleave two vectors.
+ ///
+ /// The first result takes every other lane of `self` and then `other`, starting with
+ /// the first lane.
+ ///
+ /// The second result takes every other lane of `self` and then `other`, starting with
+ /// the second lane.
+ ///
+ /// ```
+ /// #![feature(portable_simd)]
+ /// # use core::simd::Simd;
+ /// let a = Simd::from_array([0, 4, 1, 5]);
+ /// let b = Simd::from_array([2, 6, 3, 7]);
+ /// let (x, y) = a.deinterleave(b);
+ /// assert_eq!(x.to_array(), [0, 1, 2, 3]);
+ /// assert_eq!(y.to_array(), [4, 5, 6, 7]);
+ /// ```
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn deinterleave(self, other: Self) -> (Self, Self) {
+ const fn even<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES / 2 {
+ idx[i] = Which::First(2 * i);
+ idx[i + LANES / 2] = Which::Second(2 * i);
+ i += 1;
+ }
+ idx
+ }
+ const fn odd<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES / 2 {
+ idx[i] = Which::First(2 * i + 1);
+ idx[i + LANES / 2] = Which::Second(2 * i + 1);
+ i += 1;
+ }
+ idx
+ }
+
+ struct Even;
+ struct Odd;
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Even {
+ const INDEX: [Which; LANES] = even::<LANES>();
+ }
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Odd {
+ const INDEX: [Which; LANES] = odd::<LANES>();
+ }
+
+ (Even::swizzle2(self, other), Odd::swizzle2(self, other))
+ }
+}
--- /dev/null
- use crate::simd::intrinsics;
- use crate::simd::{LaneCount, Mask, MaskElement, SupportedLaneCount};
+mod float;
+mod int;
+mod uint;
+
+pub use float::*;
+pub use int::*;
+pub use uint::*;
+
+// Vectors of pointers are not for public use at the current time.
+pub(crate) mod ptr;
+
- /// Get the number of lanes in this vector.
++use crate::simd::{
++ intrinsics, LaneCount, Mask, MaskElement, SimdPartialOrd, SupportedLaneCount, Swizzle,
++};
+
+/// A SIMD vector of `LANES` elements of type `T`. `Simd<T, N>` has the same shape as [`[T; N]`](array), but operates like `T`.
+///
+/// Two vectors of the same type and length will, by convention, support the operators (+, *, etc.) that `T` does.
+/// These take the lanes at each index on the left-hand side and right-hand side, perform the operation,
+/// and return the result in the same lane in a vector of equal size. For a given operator, this is equivalent to zipping
+/// the two arrays together and mapping the operator over each lane.
+///
+/// ```rust
+/// # #![feature(array_zip, portable_simd)]
+/// # use core::simd::{Simd};
+/// let a0: [i32; 4] = [-2, 0, 2, 4];
+/// let a1 = [10, 9, 8, 7];
+/// let zm_add = a0.zip(a1).map(|(lhs, rhs)| lhs + rhs);
+/// let zm_mul = a0.zip(a1).map(|(lhs, rhs)| lhs * rhs);
+///
+/// // `Simd<T, N>` implements `From<[T; N]>
+/// let (v0, v1) = (Simd::from(a0), Simd::from(a1));
+/// // Which means arrays implement `Into<Simd<T, N>>`.
+/// assert_eq!(v0 + v1, zm_add.into());
+/// assert_eq!(v0 * v1, zm_mul.into());
+/// ```
+///
+/// `Simd` with integers has the quirk that these operations are also inherently wrapping, as if `T` was [`Wrapping<T>`].
+/// Thus, `Simd` does not implement `wrapping_add`, because that is the default behavior.
+/// This means there is no warning on overflows, even in "debug" builds.
+/// For most applications where `Simd` is appropriate, it is "not a bug" to wrap,
+/// and even "debug builds" are unlikely to tolerate the loss of performance.
+/// You may want to consider using explicitly checked arithmetic if such is required.
+/// Division by zero still causes a panic, so you may want to consider using floating point numbers if that is unacceptable.
+///
+/// [`Wrapping<T>`]: core::num::Wrapping
+///
+/// # Layout
+/// `Simd<T, N>` has a layout similar to `[T; N]` (identical "shapes"), but with a greater alignment.
+/// `[T; N]` is aligned to `T`, but `Simd<T, N>` will have an alignment based on both `T` and `N`.
+/// It is thus sound to [`transmute`] `Simd<T, N>` to `[T; N]`, and will typically optimize to zero cost,
+/// but the reverse transmutation is more likely to require a copy the compiler cannot simply elide.
+///
+/// # ABI "Features"
+/// Due to Rust's safety guarantees, `Simd<T, N>` is currently passed to and from functions via memory, not SIMD registers,
+/// except as an optimization. `#[inline]` hints are recommended on functions that accept `Simd<T, N>` or return it.
+/// The need for this may be corrected in the future.
+///
+/// # Safe SIMD with Unsafe Rust
+///
+/// Operations with `Simd` are typically safe, but there are many reasons to want to combine SIMD with `unsafe` code.
+/// Care must be taken to respect differences between `Simd` and other types it may be transformed into or derived from.
+/// In particular, the layout of `Simd<T, N>` may be similar to `[T; N]`, and may allow some transmutations,
+/// but references to `[T; N]` are not interchangeable with those to `Simd<T, N>`.
+/// Thus, when using `unsafe` Rust to read and write `Simd<T, N>` through [raw pointers], it is a good idea to first try with
+/// [`read_unaligned`] and [`write_unaligned`]. This is because:
+/// - [`read`] and [`write`] require full alignment (in this case, `Simd<T, N>`'s alignment)
+/// - the likely source for reading or destination for writing `Simd<T, N>` is [`[T]`](slice) and similar types, aligned to `T`
+/// - combining these actions would violate the `unsafe` contract and explode the program into a puff of **undefined behavior**
+/// - the compiler can implicitly adjust layouts to make unaligned reads or writes fully aligned if it sees the optimization
+/// - most contemporary processors suffer no performance penalty for "unaligned" reads and writes that are aligned at runtime
+///
+/// By imposing less obligations, unaligned functions are less likely to make the program unsound,
+/// and may be just as fast as stricter alternatives.
+/// When trying to guarantee alignment, [`[T]::as_simd`][as_simd] is an option for converting `[T]` to `[Simd<T, N>]`,
+/// and allows soundly operating on an aligned SIMD body, but it may cost more time when handling the scalar head and tail.
+/// If these are not sufficient, then it is most ideal to design data structures to be already aligned
+/// to the `Simd<T, N>` you wish to use before using `unsafe` Rust to read or write.
+/// More conventional ways to compensate for these facts, like materializing `Simd` to or from an array first,
+/// are handled by safe methods like [`Simd::from_array`] and [`Simd::from_slice`].
+///
+/// [`transmute`]: core::mem::transmute
+/// [raw pointers]: pointer
+/// [`read_unaligned`]: pointer::read_unaligned
+/// [`write_unaligned`]: pointer::write_unaligned
+/// [`read`]: pointer::read
+/// [`write`]: pointer::write
+/// [as_simd]: slice::as_simd
+#[repr(simd)]
+pub struct Simd<T, const LANES: usize>([T; LANES])
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount;
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ /// Number of lanes in this vector.
+ pub const LANES: usize = LANES;
+
- /// Construct a SIMD vector by setting all lanes to the given value.
- pub const fn splat(value: T) -> Self {
- Self([value; LANES])
++ /// Returns the number of lanes in this SIMD vector.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # use core::simd::u32x4;
++ /// let v = u32x4::splat(0);
++ /// assert_eq!(v.lanes(), 4);
++ /// ```
+ pub const fn lanes(&self) -> usize {
+ LANES
+ }
+
- /// Converts a slice to a SIMD vector containing `slice[..LANES]`
++ /// Constructs a new SIMD vector with all lanes set to the given value.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # use core::simd::u32x4;
++ /// let v = u32x4::splat(8);
++ /// assert_eq!(v.as_array(), &[8, 8, 8, 8]);
++ /// ```
++ pub fn splat(value: T) -> Self {
++ // This is preferred over `[value; LANES]`, since it's explicitly a splat:
++ // https://github.com/rust-lang/rust/issues/97804
++ struct Splat;
++ impl<const LANES: usize> Swizzle<1, LANES> for Splat {
++ const INDEX: [usize; LANES] = [0; LANES];
++ }
++ Splat::swizzle(Simd::<T, 1>::from([value]))
+ }
+
+ /// Returns an array reference containing the entire SIMD vector.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # use core::simd::{Simd, u64x4};
++ /// let v: u64x4 = Simd::from_array([0, 1, 2, 3]);
++ /// assert_eq!(v.as_array(), &[0, 1, 2, 3]);
++ /// ```
+ pub const fn as_array(&self) -> &[T; LANES] {
+ &self.0
+ }
+
+ /// Returns a mutable array reference containing the entire SIMD vector.
+ pub fn as_mut_array(&mut self) -> &mut [T; LANES] {
+ &mut self.0
+ }
+
+ /// Converts an array to a SIMD vector.
+ pub const fn from_array(array: [T; LANES]) -> Self {
+ Self(array)
+ }
+
+ /// Converts a SIMD vector to an array.
+ pub const fn to_array(self) -> [T; LANES] {
+ self.0
+ }
+
- /// `from_slice` will panic if the slice's `len` is less than the vector's `Simd::LANES`.
++ /// Converts a slice to a SIMD vector containing `slice[..LANES]`.
++ ///
+ /// # Panics
- // Safety: The input argument is a vector of a known SIMD type.
++ ///
++ /// Panics if the slice's length is less than the vector's `Simd::LANES`.
++ ///
++ /// # Examples
++ ///
++ /// ```
++ /// # #![feature(portable_simd)]
++ /// # use core::simd::u32x4;
++ /// let source = vec![1, 2, 3, 4, 5, 6];
++ /// let v = u32x4::from_slice(&source);
++ /// assert_eq!(v.as_array(), &[1, 2, 3, 4]);
++ /// ```
+ #[must_use]
+ pub const fn from_slice(slice: &[T]) -> Self {
+ assert!(slice.len() >= LANES, "slice length must be at least the number of lanes");
+ let mut array = [slice[0]; LANES];
+ let mut i = 0;
+ while i < LANES {
+ array[i] = slice[i];
+ i += 1;
+ }
+ Self(array)
+ }
+
+ /// Performs lanewise conversion of a SIMD vector's elements to another SIMD-valid type.
++ ///
+ /// This follows the semantics of Rust's `as` conversion for casting
+ /// integers to unsigned integers (interpreting as the other type, so `-1` to `MAX`),
+ /// and from floats to integers (truncating, or saturating at the limits) for each lane,
+ /// or vice versa.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
+ /// # use core::simd::Simd;
+ /// let floats: Simd<f32, 4> = Simd::from_array([1.9, -4.5, f32::INFINITY, f32::NAN]);
+ /// let ints = floats.cast::<i32>();
+ /// assert_eq!(ints, Simd::from_array([1, -4, i32::MAX, 0]));
+ ///
+ /// // Formally equivalent, but `Simd::cast` can optimize better.
+ /// assert_eq!(ints, Simd::from_array(floats.to_array().map(|x| x as i32)));
+ ///
+ /// // The float conversion does not round-trip.
+ /// let floats_again = ints.cast();
+ /// assert_ne!(floats, floats_again);
+ /// assert_eq!(floats_again, Simd::from_array([1.0, -4.0, 2147483647.0, 0.0]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn cast<U: SimdElement>(self) -> Simd<U, LANES> {
- let enable: Mask<isize, LANES> = enable & idxs.lanes_lt(Simd::splat(slice.len()));
++ // Safety: The input argument is a vector of a valid SIMD element type.
+ unsafe { intrinsics::simd_as(self) }
+ }
+
++ /// Rounds toward zero and converts to the same-width integer type, assuming that
++ /// the value is finite and fits in that type.
++ ///
++ /// # Safety
++ /// The value must:
++ ///
++ /// * Not be NaN
++ /// * Not be infinite
++ /// * Be representable in the return type, after truncating off its fractional part
++ ///
++ /// If these requirements are infeasible or costly, consider using the safe function [cast],
++ /// which saturates on conversion.
++ ///
++ /// [cast]: Simd::cast
++ #[inline]
++ pub unsafe fn to_int_unchecked<I>(self) -> Simd<I, LANES>
++ where
++ T: core::convert::FloatToInt<I>,
++ I: SimdElement,
++ {
++ // Safety: `self` is a vector, and `FloatToInt` ensures the type can be casted to
++ // an integer.
++ unsafe { intrinsics::simd_cast(self) }
++ }
++
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// If an index is out-of-bounds, the lane is instead selected from the `or` vector.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
+ /// # use core::simd::Simd;
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ /// let alt = Simd::from_array([-5, -4, -3, -2]);
+ ///
+ /// let result = Simd::gather_or(&vec, idxs, alt); // Note the lane that is out-of-bounds.
+ /// assert_eq!(result, Simd::from_array([-5, 13, 10, 15]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn gather_or(slice: &[T], idxs: Simd<usize, LANES>, or: Self) -> Self {
+ Self::gather_select(slice, Mask::splat(true), idxs, or)
+ }
+
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// If an index is out-of-bounds, the lane is set to the default value for the type.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
+ /// # use core::simd::Simd;
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ ///
+ /// let result = Simd::gather_or_default(&vec, idxs); // Note the lane that is out-of-bounds.
+ /// assert_eq!(result, Simd::from_array([0, 13, 10, 15]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn gather_or_default(slice: &[T], idxs: Simd<usize, LANES>) -> Self
+ where
+ T: Default,
+ {
+ Self::gather_or(slice, idxs, Self::splat(T::default()))
+ }
+
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If an index is disabled or is out-of-bounds, the lane is selected from the `or` vector.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
+ /// # use core::simd::{Simd, Mask};
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ /// let alt = Simd::from_array([-5, -4, -3, -2]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the mask of the last lane.
+ ///
+ /// let result = Simd::gather_select(&vec, enable, idxs, alt); // Note the lane that is out-of-bounds.
+ /// assert_eq!(result, Simd::from_array([-5, 13, 10, -2]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn gather_select(
+ slice: &[T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ or: Self,
+ ) -> Self {
- /// # use core::simd::{Simd, Mask};
++ let enable: Mask<isize, LANES> = enable & idxs.simd_lt(Simd::splat(slice.len()));
+ // Safety: We have masked-off out-of-bounds lanes.
+ unsafe { Self::gather_select_unchecked(slice, enable, idxs, or) }
+ }
+
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If an index is disabled, the lane is selected from the `or` vector.
+ ///
+ /// # Safety
+ ///
+ /// Calling this function with an `enable`d out-of-bounds index is *[undefined behavior]*
+ /// even if the resulting value is not used.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// let enable = enable & idxs.lanes_lt(Simd::splat(vec.len()));
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdPartialOrd, Mask};
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ /// let alt = Simd::from_array([-5, -4, -3, -2]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the final mask lane.
+ /// // If this mask was used to gather, it would be unsound. Let's fix that.
- /// # use core::simd::{Simd, Mask};
++ /// let enable = enable & idxs.simd_lt(Simd::splat(vec.len()));
+ ///
+ /// // We have masked the OOB lane, so it's safe to gather now.
+ /// let result = unsafe { Simd::gather_select_unchecked(&vec, enable, idxs, alt) };
+ /// assert_eq!(result, Simd::from_array([-5, 13, 10, -2]));
+ /// ```
+ /// [undefined behavior]: https://doc.rust-lang.org/reference/behavior-considered-undefined.html
+ #[must_use]
+ #[inline]
+ pub unsafe fn gather_select_unchecked(
+ slice: &[T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ or: Self,
+ ) -> Self {
+ let base_ptr = crate::simd::ptr::SimdConstPtr::splat(slice.as_ptr());
+ // Ferris forgive me, I have done pointer arithmetic here.
+ let ptrs = base_ptr.wrapping_add(idxs);
+ // Safety: The ptrs have been bounds-masked to prevent memory-unsafe reads insha'allah
+ unsafe { intrinsics::simd_gather(or, ptrs, enable.to_int()) }
+ }
+
+ /// Writes the values in a SIMD vector to potentially discontiguous indices in `slice`.
+ /// If two lanes in the scattered vector would write to the same index
+ /// only the last lane is guaranteed to actually be written.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
+ /// # use core::simd::Simd;
+ /// let mut vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 0]);
+ /// let vals = Simd::from_array([-27, 82, -41, 124]);
+ ///
+ /// vals.scatter(&mut vec, idxs); // index 0 receives two writes.
+ /// assert_eq!(vec, vec![124, 11, 12, 82, 14, 15, 16, 17, 18]);
+ /// ```
+ #[inline]
+ pub fn scatter(self, slice: &mut [T], idxs: Simd<usize, LANES>) {
+ self.scatter_select(slice, Mask::splat(true), idxs)
+ }
+
+ /// Writes the values in a SIMD vector to multiple potentially discontiguous indices in `slice`.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If an enabled index is out-of-bounds, the lane is not written.
+ /// If two enabled lanes in the scattered vector would write to the same index,
+ /// only the last lane is guaranteed to actually be written.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- let enable: Mask<isize, LANES> = enable & idxs.lanes_lt(Simd::splat(slice.len()));
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, Mask};
+ /// let mut vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 0]);
+ /// let vals = Simd::from_array([-27, 82, -41, 124]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the mask of the last lane.
+ ///
+ /// vals.scatter_select(&mut vec, enable, idxs); // index 0's second write is masked, thus omitted.
+ /// assert_eq!(vec, vec![-41, 11, 12, 82, 14, 15, 16, 17, 18]);
+ /// ```
+ #[inline]
+ pub fn scatter_select(
+ self,
+ slice: &mut [T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ ) {
- /// # use core::simd::{Simd, Mask};
++ let enable: Mask<isize, LANES> = enable & idxs.simd_lt(Simd::splat(slice.len()));
+ // Safety: We have masked-off out-of-bounds lanes.
+ unsafe { self.scatter_select_unchecked(slice, enable, idxs) }
+ }
+
+ /// Writes the values in a SIMD vector to multiple potentially discontiguous indices in `slice`.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If two enabled lanes in the scattered vector would write to the same index,
+ /// only the last lane is guaranteed to actually be written.
+ ///
+ /// # Safety
+ ///
+ /// Calling this function with an enabled out-of-bounds index is *[undefined behavior]*,
+ /// and may lead to memory corruption.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// let enable = enable & idxs.lanes_lt(Simd::splat(vec.len()));
++ /// # #[cfg(feature = "as_crate")] use core_simd::simd;
++ /// # #[cfg(not(feature = "as_crate"))] use core::simd;
++ /// # use simd::{Simd, SimdPartialOrd, Mask};
+ /// let mut vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 0]);
+ /// let vals = Simd::from_array([-27, 82, -41, 124]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the mask of the last lane.
+ /// // If this mask was used to scatter, it would be unsound. Let's fix that.
- // TODO use SIMD equality
- self.to_array() == other.to_array()
++ /// let enable = enable & idxs.simd_lt(Simd::splat(vec.len()));
+ ///
+ /// // We have masked the OOB lane, so it's safe to scatter now.
+ /// unsafe { vals.scatter_select_unchecked(&mut vec, enable, idxs); }
+ /// // index 0's second write is masked, thus was omitted.
+ /// assert_eq!(vec, vec![-41, 11, 12, 82, 14, 15, 16, 17, 18]);
+ /// ```
+ /// [undefined behavior]: https://doc.rust-lang.org/reference/behavior-considered-undefined.html
+ #[inline]
+ pub unsafe fn scatter_select_unchecked(
+ self,
+ slice: &mut [T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ ) {
+ // Safety: This block works with *mut T derived from &mut 'a [T],
+ // which means it is delicate in Rust's borrowing model, circa 2021:
+ // &mut 'a [T] asserts uniqueness, so deriving &'a [T] invalidates live *mut Ts!
+ // Even though this block is largely safe methods, it must be exactly this way
+ // to prevent invalidating the raw ptrs while they're live.
+ // Thus, entering this block requires all values to use being already ready:
+ // 0. idxs we want to write to, which are used to construct the mask.
+ // 1. enable, which depends on an initial &'a [T] and the idxs.
+ // 2. actual values to scatter (self).
+ // 3. &mut [T] which will become our base ptr.
+ unsafe {
+ // Now Entering ☢️ *mut T Zone
+ let base_ptr = crate::simd::ptr::SimdMutPtr::splat(slice.as_mut_ptr());
+ // Ferris forgive me, I have done pointer arithmetic here.
+ let ptrs = base_ptr.wrapping_add(idxs);
+ // The ptrs have been bounds-masked to prevent memory-unsafe writes insha'allah
+ intrinsics::simd_scatter(self, ptrs, enable.to_int())
+ // Cleared ☢️ *mut T Zone
+ }
+ }
+}
+
+impl<T, const LANES: usize> Copy for Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+impl<T, const LANES: usize> Clone for Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<T, const LANES: usize> Default for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + Default,
+{
+ #[inline]
+ fn default() -> Self {
+ Self::splat(T::default())
+ }
+}
+
+impl<T, const LANES: usize> PartialEq for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + PartialEq,
+{
+ #[inline]
+ fn eq(&self, other: &Self) -> bool {
++ // Safety: All SIMD vectors are SimdPartialEq, and the comparison produces a valid mask.
++ let mask = unsafe {
++ let tfvec: Simd<<T as SimdElement>::Mask, LANES> = intrinsics::simd_eq(*self, *other);
++ Mask::from_int_unchecked(tfvec)
++ };
++
++ // Two vectors are equal if all lanes tested true for vertical equality.
++ mask.all()
++ }
++
++ #[allow(clippy::partialeq_ne_impl)]
++ #[inline]
++ fn ne(&self, other: &Self) -> bool {
++ // Safety: All SIMD vectors are SimdPartialEq, and the comparison produces a valid mask.
++ let mask = unsafe {
++ let tfvec: Simd<<T as SimdElement>::Mask, LANES> = intrinsics::simd_ne(*self, *other);
++ Mask::from_int_unchecked(tfvec)
++ };
++
++ // Two vectors are non-equal if any lane tested true for vertical non-equality.
++ mask.any()
+ }
+}
+
+impl<T, const LANES: usize> PartialOrd for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + PartialOrd,
+{
+ #[inline]
+ fn partial_cmp(&self, other: &Self) -> Option<core::cmp::Ordering> {
+ // TODO use SIMD equality
+ self.to_array().partial_cmp(other.as_ref())
+ }
+}
+
+impl<T, const LANES: usize> Eq for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + Eq,
+{
+}
+
+impl<T, const LANES: usize> Ord for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + Ord,
+{
+ #[inline]
+ fn cmp(&self, other: &Self) -> core::cmp::Ordering {
+ // TODO use SIMD equality
+ self.to_array().cmp(other.as_ref())
+ }
+}
+
+impl<T, const LANES: usize> core::hash::Hash for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + core::hash::Hash,
+{
+ #[inline]
+ fn hash<H>(&self, state: &mut H)
+ where
+ H: core::hash::Hasher,
+ {
+ self.as_array().hash(state)
+ }
+}
+
+// array references
+impl<T, const LANES: usize> AsRef<[T; LANES]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_ref(&self) -> &[T; LANES] {
+ &self.0
+ }
+}
+
+impl<T, const LANES: usize> AsMut<[T; LANES]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_mut(&mut self) -> &mut [T; LANES] {
+ &mut self.0
+ }
+}
+
+// slice references
+impl<T, const LANES: usize> AsRef<[T]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_ref(&self) -> &[T] {
+ &self.0
+ }
+}
+
+impl<T, const LANES: usize> AsMut<[T]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_mut(&mut self) -> &mut [T] {
+ &mut self.0
+ }
+}
+
+// vector/array conversion
+impl<T, const LANES: usize> From<[T; LANES]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ fn from(array: [T; LANES]) -> Self {
+ Self(array)
+ }
+}
+
+impl<T, const LANES: usize> From<Simd<T, LANES>> for [T; LANES]
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ fn from(vector: Simd<T, LANES>) -> Self {
+ vector.to_array()
+ }
+}
+
+mod sealed {
+ pub trait Sealed {}
+}
+use sealed::Sealed;
+
+/// Marker trait for types that may be used as SIMD vector elements.
+///
+/// # Safety
+/// This trait, when implemented, asserts the compiler can monomorphize
+/// `#[repr(simd)]` structs with the marked type as an element.
+/// Strictly, it is valid to impl if the vector will not be miscompiled.
+/// Practically, it is user-unfriendly to impl it if the vector won't compile,
+/// even when no soundness guarantees are broken by allowing the user to try.
+pub unsafe trait SimdElement: Sealed + Copy {
+ /// The mask element type corresponding to this element type.
+ type Mask: MaskElement;
+}
+
+impl Sealed for u8 {}
++
++// Safety: u8 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for u8 {
+ type Mask = i8;
+}
+
+impl Sealed for u16 {}
++
++// Safety: u16 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for u16 {
+ type Mask = i16;
+}
+
+impl Sealed for u32 {}
++
++// Safety: u32 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for u32 {
+ type Mask = i32;
+}
+
+impl Sealed for u64 {}
++
++// Safety: u64 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for u64 {
+ type Mask = i64;
+}
+
+impl Sealed for usize {}
++
++// Safety: usize is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for usize {
+ type Mask = isize;
+}
+
+impl Sealed for i8 {}
++
++// Safety: i8 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for i8 {
+ type Mask = i8;
+}
+
+impl Sealed for i16 {}
++
++// Safety: i16 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for i16 {
+ type Mask = i16;
+}
+
+impl Sealed for i32 {}
++
++// Safety: i32 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for i32 {
+ type Mask = i32;
+}
+
+impl Sealed for i64 {}
++
++// Safety: i64 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for i64 {
+ type Mask = i64;
+}
+
+impl Sealed for isize {}
++
++// Safety: isize is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for isize {
+ type Mask = isize;
+}
+
+impl Sealed for f32 {}
++
++// Safety: f32 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for f32 {
+ type Mask = i32;
+}
+
+impl Sealed for f64 {}
++
++// Safety: f64 is a valid SIMD element type, and is supported by this API
+unsafe impl SimdElement for f64 {
+ type Mask = i64;
+}
--- /dev/null
- use crate::simd::intrinsics;
- use crate::simd::{LaneCount, Mask, Simd, SupportedLaneCount};
+#![allow(non_camel_case_types)]
+
- /// Implements inherent methods for a float vector containing multiple
- /// `$lanes` of float `$type`, which uses `$bits_ty` as its binary
- /// representation.
- macro_rules! impl_float_vector {
- { $type:ty, $bits_ty:ty, $mask_ty:ty } => {
- impl<const LANES: usize> Simd<$type, LANES>
- where
- LaneCount<LANES>: SupportedLaneCount,
- {
- /// Raw transmutation to an unsigned integer vector type with the
- /// same size and number of lanes.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn to_bits(self) -> Simd<$bits_ty, LANES> {
- assert_eq!(core::mem::size_of::<Self>(), core::mem::size_of::<Simd<$bits_ty, LANES>>());
- unsafe { core::mem::transmute_copy(&self) }
- }
-
- /// Raw transmutation from an unsigned integer vector type with the
- /// same size and number of lanes.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn from_bits(bits: Simd<$bits_ty, LANES>) -> Self {
- assert_eq!(core::mem::size_of::<Self>(), core::mem::size_of::<Simd<$bits_ty, LANES>>());
- unsafe { core::mem::transmute_copy(&bits) }
- }
-
- /// Produces a vector where every lane has the absolute value of the
- /// equivalently-indexed lane in `self`.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn abs(self) -> Self {
- unsafe { intrinsics::simd_fabs(self) }
- }
-
- /// Takes the reciprocal (inverse) of each lane, `1/x`.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn recip(self) -> Self {
- Self::splat(1.0) / self
- }
-
- /// Converts each lane from radians to degrees.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn to_degrees(self) -> Self {
- // to_degrees uses a special constant for better precision, so extract that constant
- self * Self::splat(<$type>::to_degrees(1.))
- }
-
- /// Converts each lane from degrees to radians.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn to_radians(self) -> Self {
- self * Self::splat(<$type>::to_radians(1.))
- }
-
- /// Returns true for each lane if it has a positive sign, including
- /// `+0.0`, `NaN`s with positive sign bit and positive infinity.
- #[inline]
- #[must_use = "method returns a new mask and does not mutate the original value"]
- pub fn is_sign_positive(self) -> Mask<$mask_ty, LANES> {
- !self.is_sign_negative()
- }
-
- /// Returns true for each lane if it has a negative sign, including
- /// `-0.0`, `NaN`s with negative sign bit and negative infinity.
- #[inline]
- #[must_use = "method returns a new mask and does not mutate the original value"]
- pub fn is_sign_negative(self) -> Mask<$mask_ty, LANES> {
- let sign_bits = self.to_bits() & Simd::splat((!0 >> 1) + 1);
- sign_bits.lanes_gt(Simd::splat(0))
- }
-
- /// Returns true for each lane if its value is `NaN`.
- #[inline]
- #[must_use = "method returns a new mask and does not mutate the original value"]
- pub fn is_nan(self) -> Mask<$mask_ty, LANES> {
- self.lanes_ne(self)
- }
-
- /// Returns true for each lane if its value is positive infinity or negative infinity.
- #[inline]
- #[must_use = "method returns a new mask and does not mutate the original value"]
- pub fn is_infinite(self) -> Mask<$mask_ty, LANES> {
- self.abs().lanes_eq(Self::splat(<$type>::INFINITY))
- }
-
- /// Returns true for each lane if its value is neither infinite nor `NaN`.
- #[inline]
- #[must_use = "method returns a new mask and does not mutate the original value"]
- pub fn is_finite(self) -> Mask<$mask_ty, LANES> {
- self.abs().lanes_lt(Self::splat(<$type>::INFINITY))
- }
-
- /// Returns true for each lane if its value is subnormal.
- #[inline]
- #[must_use = "method returns a new mask and does not mutate the original value"]
- pub fn is_subnormal(self) -> Mask<$mask_ty, LANES> {
- self.abs().lanes_ne(Self::splat(0.0)) & (self.to_bits() & Self::splat(<$type>::INFINITY).to_bits()).lanes_eq(Simd::splat(0))
- }
-
- /// Returns true for each lane if its value is neither zero, infinite,
- /// subnormal, nor `NaN`.
- #[inline]
- #[must_use = "method returns a new mask and does not mutate the original value"]
- pub fn is_normal(self) -> Mask<$mask_ty, LANES> {
- !(self.abs().lanes_eq(Self::splat(0.0)) | self.is_nan() | self.is_subnormal() | self.is_infinite())
- }
-
- /// Replaces each lane with a number that represents its sign.
- ///
- /// * `1.0` if the number is positive, `+0.0`, or `INFINITY`
- /// * `-1.0` if the number is negative, `-0.0`, or `NEG_INFINITY`
- /// * `NAN` if the number is `NAN`
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn signum(self) -> Self {
- self.is_nan().select(Self::splat(<$type>::NAN), Self::splat(1.0).copysign(self))
- }
-
- /// Returns each lane with the magnitude of `self` and the sign of `sign`.
- ///
- /// If any lane is a `NAN`, then a `NAN` with the sign of `sign` is returned.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn copysign(self, sign: Self) -> Self {
- let sign_bit = sign.to_bits() & Self::splat(-0.).to_bits();
- let magnitude = self.to_bits() & !Self::splat(-0.).to_bits();
- Self::from_bits(sign_bit | magnitude)
- }
-
- /// Returns the minimum of each lane.
- ///
- /// If one of the values is `NAN`, then the other value is returned.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn min(self, other: Self) -> Self {
- unsafe { intrinsics::simd_fmin(self, other) }
- }
-
- /// Returns the maximum of each lane.
- ///
- /// If one of the values is `NAN`, then the other value is returned.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn max(self, other: Self) -> Self {
- unsafe { intrinsics::simd_fmax(self, other) }
- }
-
- /// Restrict each lane to a certain interval unless it is NaN.
- ///
- /// For each lane in `self`, returns the corresponding lane in `max` if the lane is
- /// greater than `max`, and the corresponding lane in `min` if the lane is less
- /// than `min`. Otherwise returns the lane in `self`.
- #[inline]
- #[must_use = "method returns a new vector and does not mutate the original value"]
- pub fn clamp(self, min: Self, max: Self) -> Self {
- assert!(
- min.lanes_le(max).all(),
- "each lane in `min` must be less than or equal to the corresponding lane in `max`",
- );
- let mut x = self;
- x = x.lanes_lt(min).select(min, x);
- x = x.lanes_gt(max).select(max, x);
- x
- }
- }
- };
- }
-
- impl_float_vector! { f32, u32, i32 }
- impl_float_vector! { f64, u64, i64 }
-
- /// Vector of two `f32` values
++use crate::simd::Simd;
+
- /// Vector of four `f32` values
++/// A 64-bit SIMD vector with two elements of type `f32`.
+pub type f32x2 = Simd<f32, 2>;
+
- /// Vector of eight `f32` values
++/// A 128-bit SIMD vector with four elements of type `f32`.
+pub type f32x4 = Simd<f32, 4>;
+
- /// Vector of 16 `f32` values
++/// A 256-bit SIMD vector with eight elements of type `f32`.
+pub type f32x8 = Simd<f32, 8>;
+
- /// Vector of two `f64` values
++/// A 512-bit SIMD vector with 16 elements of type `f32`.
+pub type f32x16 = Simd<f32, 16>;
+
- /// Vector of four `f64` values
++/// A 128-bit SIMD vector with two elements of type `f64`.
+pub type f64x2 = Simd<f64, 2>;
+
- /// Vector of eight `f64` values
++/// A 256-bit SIMD vector with four elements of type `f64`.
+pub type f64x4 = Simd<f64, 4>;
+
++/// A 512-bit SIMD vector with eight elements of type `f64`.
+pub type f64x8 = Simd<f64, 8>;
--- /dev/null
- use crate::simd::{LaneCount, Mask, Simd, SupportedLaneCount};
-
- /// Implements additional integer traits (Eq, Ord, Hash) on the specified vector `$name`, holding multiple `$lanes` of `$type`.
- macro_rules! impl_integer_vector {
- { $type:ty } => {
- impl<const LANES: usize> Simd<$type, LANES>
- where
- LaneCount<LANES>: SupportedLaneCount,
- {
- /// Returns true for each positive lane and false if it is zero or negative.
- #[inline]
- pub fn is_positive(self) -> Mask<$type, LANES> {
- self.lanes_gt(Self::splat(0))
- }
-
- /// Returns true for each negative lane and false if it is zero or positive.
- #[inline]
- pub fn is_negative(self) -> Mask<$type, LANES> {
- self.lanes_lt(Self::splat(0))
- }
-
- /// Returns numbers representing the sign of each lane.
- /// * `0` if the number is zero
- /// * `1` if the number is positive
- /// * `-1` if the number is negative
- #[inline]
- pub fn signum(self) -> Self {
- self.is_positive().select(
- Self::splat(1),
- self.is_negative().select(Self::splat(-1), Self::splat(0))
- )
- }
- }
- }
- }
-
- impl_integer_vector! { isize }
- impl_integer_vector! { i16 }
- impl_integer_vector! { i32 }
- impl_integer_vector! { i64 }
- impl_integer_vector! { i8 }
-
- /// Vector of two `isize` values
+#![allow(non_camel_case_types)]
+
- /// Vector of four `isize` values
++use crate::simd::Simd;
++
++/// A SIMD vector with two elements of type `isize`.
+pub type isizex2 = Simd<isize, 2>;
+
- /// Vector of eight `isize` values
++/// A SIMD vector with four elements of type `isize`.
+pub type isizex4 = Simd<isize, 4>;
+
- /// Vector of two `i16` values
++/// A SIMD vector with eight elements of type `isize`.
+pub type isizex8 = Simd<isize, 8>;
+
- /// Vector of four `i16` values
++/// A 32-bit SIMD vector with two elements of type `i16`.
+pub type i16x2 = Simd<i16, 2>;
+
- /// Vector of eight `i16` values
++/// A 64-bit SIMD vector with four elements of type `i16`.
+pub type i16x4 = Simd<i16, 4>;
+
- /// Vector of 16 `i16` values
++/// A 128-bit SIMD vector with eight elements of type `i16`.
+pub type i16x8 = Simd<i16, 8>;
+
- /// Vector of 32 `i16` values
++/// A 256-bit SIMD vector with 16 elements of type `i16`.
+pub type i16x16 = Simd<i16, 16>;
+
- /// Vector of two `i32` values
++/// A 512-bit SIMD vector with 32 elements of type `i16`.
+pub type i16x32 = Simd<i16, 32>;
+
- /// Vector of four `i32` values
++/// A 64-bit SIMD vector with two elements of type `i32`.
+pub type i32x2 = Simd<i32, 2>;
+
- /// Vector of eight `i32` values
++/// A 128-bit SIMD vector with four elements of type `i32`.
+pub type i32x4 = Simd<i32, 4>;
+
- /// Vector of 16 `i32` values
++/// A 256-bit SIMD vector with eight elements of type `i32`.
+pub type i32x8 = Simd<i32, 8>;
+
- /// Vector of two `i64` values
++/// A 512-bit SIMD vector with 16 elements of type `i32`.
+pub type i32x16 = Simd<i32, 16>;
+
- /// Vector of four `i64` values
++/// A 128-bit SIMD vector with two elements of type `i64`.
+pub type i64x2 = Simd<i64, 2>;
+
- /// Vector of eight `i64` values
++/// A 256-bit SIMD vector with four elements of type `i64`.
+pub type i64x4 = Simd<i64, 4>;
+
- /// Vector of four `i8` values
++/// A 512-bit SIMD vector with eight elements of type `i64`.
+pub type i64x8 = Simd<i64, 8>;
+
- /// Vector of eight `i8` values
++/// A 32-bit SIMD vector with four elements of type `i8`.
+pub type i8x4 = Simd<i8, 4>;
+
- /// Vector of 16 `i8` values
++/// A 64-bit SIMD vector with eight elements of type `i8`.
+pub type i8x8 = Simd<i8, 8>;
+
- /// Vector of 32 `i8` values
++/// A 128-bit SIMD vector with 16 elements of type `i8`.
+pub type i8x16 = Simd<i8, 16>;
+
- /// Vector of 64 `i8` values
++/// A 256-bit SIMD vector with 32 elements of type `i8`.
+pub type i8x32 = Simd<i8, 32>;
+
++/// A 512-bit SIMD vector with 64 elements of type `i8`.
+pub type i8x64 = Simd<i8, 64>;
--- /dev/null
- /// Vector of two `usize` values
+#![allow(non_camel_case_types)]
+
+use crate::simd::Simd;
+
- /// Vector of four `usize` values
++/// A SIMD vector with two elements of type `usize`.
+pub type usizex2 = Simd<usize, 2>;
+
- /// Vector of eight `usize` values
++/// A SIMD vector with four elements of type `usize`.
+pub type usizex4 = Simd<usize, 4>;
+
- /// Vector of two `u16` values
++/// A SIMD vector with eight elements of type `usize`.
+pub type usizex8 = Simd<usize, 8>;
+
- /// Vector of four `u16` values
++/// A 32-bit SIMD vector with two elements of type `u16`.
+pub type u16x2 = Simd<u16, 2>;
+
- /// Vector of eight `u16` values
++/// A 64-bit SIMD vector with four elements of type `u16`.
+pub type u16x4 = Simd<u16, 4>;
+
- /// Vector of 16 `u16` values
++/// A 128-bit SIMD vector with eight elements of type `u16`.
+pub type u16x8 = Simd<u16, 8>;
+
- /// Vector of 32 `u16` values
++/// A 256-bit SIMD vector with 16 elements of type `u16`.
+pub type u16x16 = Simd<u16, 16>;
+
- /// Vector of two `u32` values
++/// A 512-bit SIMD vector with 32 elements of type `u16`.
+pub type u16x32 = Simd<u16, 32>;
+
- /// Vector of four `u32` values
++/// A 64-bit SIMD vector with two elements of type `u32`.
+pub type u32x2 = Simd<u32, 2>;
+
- /// Vector of eight `u32` values
++/// A 128-bit SIMD vector with four elements of type `u32`.
+pub type u32x4 = Simd<u32, 4>;
+
- /// Vector of 16 `u32` values
++/// A 256-bit SIMD vector with eight elements of type `u32`.
+pub type u32x8 = Simd<u32, 8>;
+
- /// Vector of two `u64` values
++/// A 512-bit SIMD vector with 16 elements of type `u32`.
+pub type u32x16 = Simd<u32, 16>;
+
- /// Vector of four `u64` values
++/// A 128-bit SIMD vector with two elements of type `u64`.
+pub type u64x2 = Simd<u64, 2>;
+
- /// Vector of eight `u64` values
++/// A 256-bit SIMD vector with four elements of type `u64`.
+pub type u64x4 = Simd<u64, 4>;
+
- /// Vector of four `u8` values
++/// A 512-bit SIMD vector with eight elements of type `u64`.
+pub type u64x8 = Simd<u64, 8>;
+
- /// Vector of eight `u8` values
++/// A 32-bit SIMD vector with four elements of type `u8`.
+pub type u8x4 = Simd<u8, 4>;
+
- /// Vector of 16 `u8` values
++/// A 64-bit SIMD vector with eight elements of type `u8`.
+pub type u8x8 = Simd<u8, 8>;
+
- /// Vector of 32 `u8` values
++/// A 128-bit SIMD vector with 16 elements of type `u8`.
+pub type u8x16 = Simd<u8, 16>;
+
- /// Vector of 64 `u8` values
++/// A 256-bit SIMD vector with 32 elements of type `u8`.
+pub type u8x32 = Simd<u8, 32>;
+
++/// A 512-bit SIMD vector with 64 elements of type `u8`.
+pub type u8x64 = Simd<u8, 64>;
--- /dev/null
- use core_simd::i16x2;
+#![feature(portable_simd)]
-
- #[test]
- fn max_is_not_lexicographic() {
- let a = i16x2::splat(10);
- let b = i16x2::from_array([-4, 12]);
- assert_eq!(a.max(b), i16x2::from_array([10, 12]));
- }
-
- #[test]
- fn min_is_not_lexicographic() {
- let a = i16x2::splat(10);
- let b = i16x2::from_array([12, -4]);
- assert_eq!(a.min(b), i16x2::from_array([10, -4]));
- }
-
- #[test]
- fn clamp_is_not_lexicographic() {
- let a = i16x2::splat(10);
- let lo = i16x2::from_array([-12, -4]);
- let up = i16x2::from_array([-4, 12]);
- assert_eq!(a.clamp(lo, up), i16x2::from_array([-4, 10]));
-
- let x = i16x2::from_array([1, 10]);
- let y = x.clamp(i16x2::splat(0), i16x2::splat(9));
- assert_eq!(y, i16x2::from_array([1, 9]));
- }
+
+#[macro_use]
+mod ops_macros;
+impl_signed_tests! { i16 }
--- /dev/null
+#![feature(portable_simd)]
+
+#[cfg(target_arch = "wasm32")]
+use wasm_bindgen_test::*;
+
+#[cfg(target_arch = "wasm32")]
+wasm_bindgen_test_configure!(run_in_browser);
+
+macro_rules! test_mask_api {
+ { $type:ident } => {
+ #[allow(non_snake_case)]
+ mod $type {
+ #[cfg(target_arch = "wasm32")]
+ use wasm_bindgen_test::*;
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test)]
+ fn set_and_test() {
+ let values = [true, false, false, true, false, false, true, false];
+ let mut mask = core_simd::Mask::<$type, 8>::splat(false);
+ for (lane, value) in values.iter().copied().enumerate() {
+ mask.set(lane, value);
+ }
+ for (lane, value) in values.iter().copied().enumerate() {
+ assert_eq!(mask.test(lane), value);
+ }
+ }
+
+ #[test]
+ #[should_panic]
+ fn set_invalid_lane() {
+ let mut mask = core_simd::Mask::<$type, 8>::splat(false);
+ mask.set(8, true);
+ let _ = mask;
+ }
+
+ #[test]
+ #[should_panic]
+ fn test_invalid_lane() {
+ let mask = core_simd::Mask::<$type, 8>::splat(false);
+ let _ = mask.test(8);
+ }
+
+ #[test]
+ fn any() {
+ assert!(!core_simd::Mask::<$type, 8>::splat(false).any());
+ assert!(core_simd::Mask::<$type, 8>::splat(true).any());
+ let mut v = core_simd::Mask::<$type, 8>::splat(false);
+ v.set(2, true);
+ assert!(v.any());
+ }
+
+ #[test]
+ fn all() {
+ assert!(!core_simd::Mask::<$type, 8>::splat(false).all());
+ assert!(core_simd::Mask::<$type, 8>::splat(true).all());
+ let mut v = core_simd::Mask::<$type, 8>::splat(false);
+ v.set(2, true);
+ assert!(!v.all());
+ }
+
+ #[test]
+ fn roundtrip_int_conversion() {
+ let values = [true, false, false, true, false, false, true, false];
+ let mask = core_simd::Mask::<$type, 8>::from_array(values);
+ let int = mask.to_int();
+ assert_eq!(int.to_array(), [-1, 0, 0, -1, 0, 0, -1, 0]);
+ assert_eq!(core_simd::Mask::<$type, 8>::from_int(int), mask);
+ }
+
+ #[test]
+ fn roundtrip_bitmask_conversion() {
+ use core_simd::ToBitMask;
+ let values = [
+ true, false, false, true, false, false, true, false,
+ true, true, false, false, false, false, false, true,
+ ];
+ let mask = core_simd::Mask::<$type, 16>::from_array(values);
+ let bitmask = mask.to_bitmask();
+ assert_eq!(bitmask, 0b1000001101001001);
+ assert_eq!(core_simd::Mask::<$type, 16>::from_bitmask(bitmask), mask);
+ }
++
++ #[test]
++ fn roundtrip_bitmask_conversion_short() {
++ use core_simd::ToBitMask;
++
++ let values = [
++ false, false, false, true,
++ ];
++ let mask = core_simd::Mask::<$type, 4>::from_array(values);
++ let bitmask = mask.to_bitmask();
++ assert_eq!(bitmask, 0b1000);
++ assert_eq!(core_simd::Mask::<$type, 4>::from_bitmask(bitmask), mask);
++
++ let values = [true, false];
++ let mask = core_simd::Mask::<$type, 2>::from_array(values);
++ let bitmask = mask.to_bitmask();
++ assert_eq!(bitmask, 0b01);
++ assert_eq!(core_simd::Mask::<$type, 2>::from_bitmask(bitmask), mask);
++ }
++
++ #[test]
++ fn cast() {
++ fn cast_impl<T: core_simd::MaskElement>()
++ where
++ core_simd::Mask<$type, 8>: Into<core_simd::Mask<T, 8>>,
++ {
++ let values = [true, false, false, true, false, false, true, false];
++ let mask = core_simd::Mask::<$type, 8>::from_array(values);
++
++ let cast_mask = mask.cast::<T>();
++ assert_eq!(values, cast_mask.to_array());
++
++ let into_mask: core_simd::Mask<T, 8> = mask.into();
++ assert_eq!(values, into_mask.to_array());
++ }
++
++ cast_impl::<i8>();
++ cast_impl::<i16>();
++ cast_impl::<i32>();
++ cast_impl::<i64>();
++ cast_impl::<isize>();
++ }
++
++ #[cfg(feature = "generic_const_exprs")]
++ #[test]
++ fn roundtrip_bitmask_array_conversion() {
++ use core_simd::ToBitMaskArray;
++ let values = [
++ true, false, false, true, false, false, true, false,
++ true, true, false, false, false, false, false, true,
++ ];
++ let mask = core_simd::Mask::<$type, 16>::from_array(values);
++ let bitmask = mask.to_bitmask_array();
++ assert_eq!(bitmask, [0b01001001, 0b10000011]);
++ assert_eq!(core_simd::Mask::<$type, 16>::from_bitmask_array(bitmask), mask);
++ }
+ }
+ }
+}
+
+mod mask_api {
+ test_mask_api! { i8 }
+ test_mask_api! { i16 }
+ test_mask_api! { i32 }
+ test_mask_api! { i64 }
+ test_mask_api! { isize }
+}
+
+#[test]
+fn convert() {
+ let values = [true, false, false, true, false, false, true, false];
+ assert_eq!(
+ core_simd::Mask::<i8, 8>::from_array(values),
+ core_simd::Mask::<i32, 8>::from_array(values).into()
+ );
+}
--- /dev/null
- fn min<const LANES: usize>() {
+/// Implements a test on a unary operation using proptest.
+///
+/// Compares the vector operation to the equivalent scalar operation.
+#[macro_export]
+macro_rules! impl_unary_op_test {
+ { $scalar:ty, $trait:ident :: $fn:ident, $scalar_fn:expr } => {
+ test_helpers::test_lanes! {
+ fn $fn<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &<core_simd::Simd<$scalar, LANES> as core::ops::$trait>::$fn,
+ &$scalar_fn,
+ &|_| true,
+ );
+ }
+ }
+ };
+ { $scalar:ty, $trait:ident :: $fn:ident } => {
+ impl_unary_op_test! { $scalar, $trait::$fn, <$scalar as core::ops::$trait>::$fn }
+ };
+}
+
+/// Implements a test on a binary operation using proptest.
+///
+/// Compares the vector operation to the equivalent scalar operation.
+#[macro_export]
+macro_rules! impl_binary_op_test {
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident, $scalar_fn:expr } => {
+ mod $fn {
+ use super::*;
+ use core_simd::Simd;
+
+ test_helpers::test_lanes! {
+ fn normal<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &<Simd<$scalar, LANES> as core::ops::$trait>::$fn,
+ &$scalar_fn,
+ &|_, _| true,
+ );
+ }
+
+ fn assign<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &|mut a, b| { <Simd<$scalar, LANES> as core::ops::$trait_assign>::$fn_assign(&mut a, b); a },
+ &$scalar_fn,
+ &|_, _| true,
+ );
+ }
+ }
+ }
+ };
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident } => {
+ impl_binary_op_test! { $scalar, $trait::$fn, $trait_assign::$fn_assign, <$scalar as core::ops::$trait>::$fn }
+ };
+}
+
+/// Implements a test on a binary operation using proptest.
+///
+/// Like `impl_binary_op_test`, but allows providing a function for rejecting particular inputs
+/// (like the `proptest_assume` macro).
+///
+/// Compares the vector operation to the equivalent scalar operation.
+#[macro_export]
+macro_rules! impl_binary_checked_op_test {
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident, $scalar_fn:expr, $check_fn:expr } => {
+ mod $fn {
+ use super::*;
+ use core_simd::Simd;
+
+ test_helpers::test_lanes! {
+ fn normal<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &<Simd<$scalar, LANES> as core::ops::$trait>::$fn,
+ &$scalar_fn,
+ &|x, y| x.iter().zip(y.iter()).all(|(x, y)| $check_fn(*x, *y)),
+ );
+ }
+
+ fn assign<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &|mut a, b| { <Simd<$scalar, LANES> as core::ops::$trait_assign>::$fn_assign(&mut a, b); a },
+ &$scalar_fn,
+ &|x, y| x.iter().zip(y.iter()).all(|(x, y)| $check_fn(*x, *y)),
+ )
+ }
+ }
+ }
+ };
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident, $check_fn:expr } => {
+ impl_binary_checked_op_test! { $scalar, $trait::$fn, $trait_assign::$fn_assign, <$scalar as core::ops::$trait>::$fn, $check_fn }
+ };
+}
+
+#[macro_export]
+macro_rules! impl_common_integer_tests {
+ { $vector:ident, $scalar:ident } => {
+ test_helpers::test_lanes! {
+ fn reduce_sum<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ $vector::<LANES>::from_array(x).reduce_sum(),
+ x.iter().copied().fold(0 as $scalar, $scalar::wrapping_add),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_product<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ $vector::<LANES>::from_array(x).reduce_product(),
+ x.iter().copied().fold(1 as $scalar, $scalar::wrapping_mul),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_and<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ $vector::<LANES>::from_array(x).reduce_and(),
+ x.iter().copied().fold(-1i8 as $scalar, <$scalar as core::ops::BitAnd>::bitand),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_or<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ $vector::<LANES>::from_array(x).reduce_or(),
+ x.iter().copied().fold(0 as $scalar, <$scalar as core::ops::BitOr>::bitor),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_xor<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ $vector::<LANES>::from_array(x).reduce_xor(),
+ x.iter().copied().fold(0 as $scalar, <$scalar as core::ops::BitXor>::bitxor),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_max<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ $vector::<LANES>::from_array(x).reduce_max(),
+ x.iter().copied().max().unwrap(),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_min<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ $vector::<LANES>::from_array(x).reduce_min(),
+ x.iter().copied().min().unwrap(),
+ );
+ Ok(())
+ });
+ }
+ }
+ }
+}
+
+/// Implement tests for signed integers.
+#[macro_export]
+macro_rules! impl_signed_tests {
+ { $scalar:tt } => {
+ mod $scalar {
++ use core_simd::simd::SimdInt;
+ type Vector<const LANES: usize> = core_simd::Simd<Scalar, LANES>;
+ type Scalar = $scalar;
+
+ impl_common_integer_tests! { Vector, Scalar }
+
+ test_helpers::test_lanes! {
+ fn neg<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &<Vector::<LANES> as core::ops::Neg>::neg,
+ &<Scalar as core::ops::Neg>::neg,
+ &|x| !x.contains(&Scalar::MIN),
+ );
+ }
+
+ fn is_positive<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_positive,
+ &Scalar::is_positive,
+ &|_| true,
+ );
+ }
+
+ fn is_negative<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_negative,
+ &Scalar::is_negative,
+ &|_| true,
+ );
+ }
+
+ fn signum<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::signum,
+ &Scalar::signum,
+ &|_| true,
+ )
+ }
+
+ fn div_min_may_overflow<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(Scalar::MIN);
+ let b = Vector::<LANES>::splat(-1);
+ assert_eq!(a / b, a);
+ }
+
+ fn rem_min_may_overflow<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(Scalar::MIN);
+ let b = Vector::<LANES>::splat(-1);
+ assert_eq!(a % b, Vector::<LANES>::splat(0));
+ }
+
- assert_eq!(a.min(b), a);
++ fn simd_min<const LANES: usize>() {
++ use core_simd::simd::SimdOrd;
+ let a = Vector::<LANES>::splat(Scalar::MIN);
+ let b = Vector::<LANES>::splat(0);
- assert_eq!(a.min(b), b);
++ assert_eq!(a.simd_min(b), a);
+ let a = Vector::<LANES>::splat(Scalar::MAX);
+ let b = Vector::<LANES>::splat(0);
- fn max<const LANES: usize>() {
++ assert_eq!(a.simd_min(b), b);
+ }
+
- assert_eq!(a.max(b), b);
++ fn simd_max<const LANES: usize>() {
++ use core_simd::simd::SimdOrd;
+ let a = Vector::<LANES>::splat(Scalar::MIN);
+ let b = Vector::<LANES>::splat(0);
- assert_eq!(a.max(b), a);
++ assert_eq!(a.simd_max(b), b);
+ let a = Vector::<LANES>::splat(Scalar::MAX);
+ let b = Vector::<LANES>::splat(0);
- fn clamp<const LANES: usize>() {
++ assert_eq!(a.simd_max(b), a);
+ }
+
- assert_eq!(zero.clamp(min, max), zero);
- assert_eq!(zero.clamp(min, one), zero);
- assert_eq!(zero.clamp(one, max), one);
- assert_eq!(zero.clamp(min, negone), negone);
++ fn simd_clamp<const LANES: usize>() {
++ use core_simd::simd::SimdOrd;
+ let min = Vector::<LANES>::splat(Scalar::MIN);
+ let max = Vector::<LANES>::splat(Scalar::MAX);
+ let zero = Vector::<LANES>::splat(0);
+ let one = Vector::<LANES>::splat(1);
+ let negone = Vector::<LANES>::splat(-1);
- fn min<const LANES: usize>() {
++ assert_eq!(zero.simd_clamp(min, max), zero);
++ assert_eq!(zero.simd_clamp(min, one), zero);
++ assert_eq!(zero.simd_clamp(one, max), one);
++ assert_eq!(zero.simd_clamp(min, negone), negone);
+ }
+ }
+
+ test_helpers::test_lanes_panic! {
+ fn div_by_all_zeros_panics<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(0);
+ let _ = a / b;
+ }
+
+ fn div_by_one_zero_panics<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let mut b = Vector::<LANES>::splat(21);
+ b[0] = 0 as _;
+ let _ = a / b;
+ }
+
+ fn rem_zero_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(0);
+ let _ = a % b;
+ }
+ }
+
+ test_helpers::test_lanes! {
+ fn div_neg_one_no_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(-1);
+ let _ = a / b;
+ }
+
+ fn rem_neg_one_no_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(-1);
+ let _ = a % b;
+ }
+ }
+
+ impl_binary_op_test!(Scalar, Add::add, AddAssign::add_assign, Scalar::wrapping_add);
+ impl_binary_op_test!(Scalar, Sub::sub, SubAssign::sub_assign, Scalar::wrapping_sub);
+ impl_binary_op_test!(Scalar, Mul::mul, MulAssign::mul_assign, Scalar::wrapping_mul);
+
+ // Exclude Div and Rem panicking cases
+ impl_binary_checked_op_test!(Scalar, Div::div, DivAssign::div_assign, Scalar::wrapping_div, |x, y| y != 0 && !(x == Scalar::MIN && y == -1));
+ impl_binary_checked_op_test!(Scalar, Rem::rem, RemAssign::rem_assign, Scalar::wrapping_rem, |x, y| y != 0 && !(x == Scalar::MIN && y == -1));
+
+ impl_unary_op_test!(Scalar, Not::not);
+ impl_binary_op_test!(Scalar, BitAnd::bitand, BitAndAssign::bitand_assign);
+ impl_binary_op_test!(Scalar, BitOr::bitor, BitOrAssign::bitor_assign);
+ impl_binary_op_test!(Scalar, BitXor::bitxor, BitXorAssign::bitxor_assign);
+ }
+ }
+}
+
+/// Implement tests for unsigned integers.
+#[macro_export]
+macro_rules! impl_unsigned_tests {
+ { $scalar:tt } => {
+ mod $scalar {
++ use core_simd::simd::SimdUint;
+ type Vector<const LANES: usize> = core_simd::Simd<Scalar, LANES>;
+ type Scalar = $scalar;
+
+ impl_common_integer_tests! { Vector, Scalar }
+
+ test_helpers::test_lanes_panic! {
+ fn rem_zero_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(0);
+ let _ = a % b;
+ }
+ }
+
+ impl_binary_op_test!(Scalar, Add::add, AddAssign::add_assign, Scalar::wrapping_add);
+ impl_binary_op_test!(Scalar, Sub::sub, SubAssign::sub_assign, Scalar::wrapping_sub);
+ impl_binary_op_test!(Scalar, Mul::mul, MulAssign::mul_assign, Scalar::wrapping_mul);
+
+ // Exclude Div and Rem panicking cases
+ impl_binary_checked_op_test!(Scalar, Div::div, DivAssign::div_assign, Scalar::wrapping_div, |_, y| y != 0);
+ impl_binary_checked_op_test!(Scalar, Rem::rem, RemAssign::rem_assign, Scalar::wrapping_rem, |_, y| y != 0);
+
+ impl_unary_op_test!(Scalar, Not::not);
+ impl_binary_op_test!(Scalar, BitAnd::bitand, BitAndAssign::bitand_assign);
+ impl_binary_op_test!(Scalar, BitOr::bitor, BitOrAssign::bitor_assign);
+ impl_binary_op_test!(Scalar, BitXor::bitxor, BitXorAssign::bitxor_assign);
+ }
+ }
+}
+
+/// Implement tests for floating point numbers.
+#[macro_export]
+macro_rules! impl_float_tests {
+ { $scalar:tt, $int_scalar:tt } => {
+ mod $scalar {
++ use core_simd::SimdFloat;
+ type Vector<const LANES: usize> = core_simd::Simd<Scalar, LANES>;
+ type Scalar = $scalar;
+
+ impl_unary_op_test!(Scalar, Neg::neg);
+ impl_binary_op_test!(Scalar, Add::add, AddAssign::add_assign);
+ impl_binary_op_test!(Scalar, Sub::sub, SubAssign::sub_assign);
+ impl_binary_op_test!(Scalar, Mul::mul, MulAssign::mul_assign);
+ impl_binary_op_test!(Scalar, Div::div, DivAssign::div_assign);
+ impl_binary_op_test!(Scalar, Rem::rem, RemAssign::rem_assign);
+
+ test_helpers::test_lanes! {
+ fn is_sign_positive<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_sign_positive,
+ &Scalar::is_sign_positive,
+ &|_| true,
+ );
+ }
+
+ fn is_sign_negative<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_sign_negative,
+ &Scalar::is_sign_negative,
+ &|_| true,
+ );
+ }
+
+ fn is_finite<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_finite,
+ &Scalar::is_finite,
+ &|_| true,
+ );
+ }
+
+ fn is_infinite<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_infinite,
+ &Scalar::is_infinite,
+ &|_| true,
+ );
+ }
+
+ fn is_nan<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_nan,
+ &Scalar::is_nan,
+ &|_| true,
+ );
+ }
+
+ fn is_normal<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_normal,
+ &Scalar::is_normal,
+ &|_| true,
+ );
+ }
+
+ fn is_subnormal<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_subnormal,
+ &Scalar::is_subnormal,
+ &|_| true,
+ );
+ }
+
+ fn abs<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::abs,
+ &Scalar::abs,
+ &|_| true,
+ )
+ }
+
+ fn recip<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::recip,
+ &Scalar::recip,
+ &|_| true,
+ )
+ }
+
+ fn to_degrees<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::to_degrees,
+ &Scalar::to_degrees,
+ &|_| true,
+ )
+ }
+
+ fn to_radians<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::to_radians,
+ &Scalar::to_radians,
+ &|_| true,
+ )
+ }
+
+ fn signum<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::signum,
+ &Scalar::signum,
+ &|_| true,
+ )
+ }
+
+ fn copysign<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &Vector::<LANES>::copysign,
+ &Scalar::copysign,
+ &|_, _| true,
+ )
+ }
+
- &Vector::<LANES>::min,
++ fn simd_min<const LANES: usize>() {
+ // Regular conditions (both values aren't zero)
+ test_helpers::test_binary_elementwise(
- assert!(p_zero.min(n_zero).to_array().iter().all(|x| *x == 0.));
- assert!(n_zero.min(p_zero).to_array().iter().all(|x| *x == 0.));
++ &Vector::<LANES>::simd_min,
+ &Scalar::min,
+ // Reject the case where both values are zero with different signs
+ &|a, b| {
+ for (a, b) in a.iter().zip(b.iter()) {
+ if *a == 0. && *b == 0. && a.signum() != b.signum() {
+ return false;
+ }
+ }
+ true
+ }
+ );
+
+ // Special case where both values are zero
+ let p_zero = Vector::<LANES>::splat(0.);
+ let n_zero = Vector::<LANES>::splat(-0.);
- fn max<const LANES: usize>() {
++ assert!(p_zero.simd_min(n_zero).to_array().iter().all(|x| *x == 0.));
++ assert!(n_zero.simd_min(p_zero).to_array().iter().all(|x| *x == 0.));
+ }
+
- &Vector::<LANES>::max,
++ fn simd_max<const LANES: usize>() {
+ // Regular conditions (both values aren't zero)
+ test_helpers::test_binary_elementwise(
- assert!(p_zero.max(n_zero).to_array().iter().all(|x| *x == 0.));
- assert!(n_zero.max(p_zero).to_array().iter().all(|x| *x == 0.));
++ &Vector::<LANES>::simd_max,
+ &Scalar::max,
+ // Reject the case where both values are zero with different signs
+ &|a, b| {
+ for (a, b) in a.iter().zip(b.iter()) {
+ if *a == 0. && *b == 0. && a.signum() != b.signum() {
+ return false;
+ }
+ }
+ true
+ }
+ );
+
+ // Special case where both values are zero
+ let p_zero = Vector::<LANES>::splat(0.);
+ let n_zero = Vector::<LANES>::splat(-0.);
- fn clamp<const LANES: usize>() {
++ assert!(p_zero.simd_max(n_zero).to_array().iter().all(|x| *x == 0.));
++ assert!(n_zero.simd_max(p_zero).to_array().iter().all(|x| *x == 0.));
+ }
+
- let result_vector = Vector::from_array(value).clamp(min.into(), max.into()).to_array();
++ fn simd_clamp<const LANES: usize>() {
+ test_helpers::test_3(&|value: [Scalar; LANES], mut min: [Scalar; LANES], mut max: [Scalar; LANES]| {
+ for (min, max) in min.iter_mut().zip(max.iter_mut()) {
+ if max < min {
+ core::mem::swap(min, max);
+ }
+ if min.is_nan() {
+ *min = Scalar::NEG_INFINITY;
+ }
+ if max.is_nan() {
+ *max = Scalar::INFINITY;
+ }
+ }
+
+ let mut result_scalar = [Scalar::default(); LANES];
+ for i in 0..LANES {
+ result_scalar[i] = value[i].clamp(min[i], max[i]);
+ }
++ let result_vector = Vector::from_array(value).simd_clamp(min.into(), max.into()).to_array();
+ test_helpers::prop_assert_biteq!(result_scalar, result_vector);
+ Ok(())
+ })
+ }
+
+ fn reduce_sum<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ Vector::<LANES>::from_array(x).reduce_sum(),
+ x.iter().sum(),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_product<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
+ Vector::<LANES>::from_array(x).reduce_product(),
+ x.iter().product(),
+ );
+ Ok(())
+ });
+ }
+
+ fn reduce_max<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ let vmax = Vector::<LANES>::from_array(x).reduce_max();
+ let smax = x.iter().copied().fold(Scalar::NAN, Scalar::max);
+ // 0 and -0 are treated the same
+ if !(x.contains(&0.) && x.contains(&-0.) && vmax.abs() == 0. && smax.abs() == 0.) {
+ test_helpers::prop_assert_biteq!(vmax, smax);
+ }
+ Ok(())
+ });
+ }
+
+ fn reduce_min<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ let vmax = Vector::<LANES>::from_array(x).reduce_min();
+ let smax = x.iter().copied().fold(Scalar::NAN, Scalar::min);
+ // 0 and -0 are treated the same
+ if !(x.contains(&0.) && x.contains(&-0.) && vmax.abs() == 0. && smax.abs() == 0.) {
+ test_helpers::prop_assert_biteq!(vmax, smax);
+ }
+ Ok(())
+ });
+ }
+ }
+
+ #[cfg(feature = "std")]
+ mod std {
+ use std_float::StdFloat;
+
+ use super::*;
+ test_helpers::test_lanes! {
+ fn sqrt<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::sqrt,
+ &Scalar::sqrt,
+ &|_| true,
+ )
+ }
+
+ fn mul_add<const LANES: usize>() {
+ test_helpers::test_ternary_elementwise(
+ &Vector::<LANES>::mul_add,
+ &Scalar::mul_add,
+ &|_, _, _| true,
+ )
+ }
+ }
+ }
+ }
+ }
+}
--- /dev/null
- let mut runner = proptest::test_runner::TestRunner::default();
+#![feature(portable_simd)]
+
+macro_rules! float_rounding_test {
+ { $scalar:tt, $int_scalar:tt } => {
+ mod $scalar {
+ use std_float::StdFloat;
+
+ type Vector<const LANES: usize> = core_simd::Simd<$scalar, LANES>;
+ type Scalar = $scalar;
+ type IntScalar = $int_scalar;
+
+ test_helpers::test_lanes! {
+ fn ceil<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::ceil,
+ &Scalar::ceil,
+ &|_| true,
+ )
+ }
+
+ fn floor<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::floor,
+ &Scalar::floor,
+ &|_| true,
+ )
+ }
+
+ fn round<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::round,
+ &Scalar::round,
+ &|_| true,
+ )
+ }
+
+ fn trunc<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::trunc,
+ &Scalar::trunc,
+ &|_| true,
+ )
+ }
+
+ fn fract<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::fract,
+ &Scalar::fract,
+ &|_| true,
+ )
+ }
+ }
+
+ test_helpers::test_lanes! {
+ fn to_int_unchecked<const LANES: usize>() {
+ // The maximum integer that can be represented by the equivalently sized float has
+ // all of the mantissa digits set to 1, pushed up to the MSB.
+ const ALL_MANTISSA_BITS: IntScalar = ((1 << <Scalar>::MANTISSA_DIGITS) - 1);
+ const MAX_REPRESENTABLE_VALUE: Scalar =
+ (ALL_MANTISSA_BITS << (core::mem::size_of::<Scalar>() * 8 - <Scalar>::MANTISSA_DIGITS as usize - 1)) as Scalar;
+
++ let mut runner = test_helpers::make_runner();
+ runner.run(
+ &test_helpers::array::UniformArrayStrategy::new(-MAX_REPRESENTABLE_VALUE..MAX_REPRESENTABLE_VALUE),
+ |x| {
+ let result_1 = unsafe { Vector::from_array(x).to_int_unchecked::<IntScalar>().to_array() };
+ let result_2 = {
+ let mut result: [IntScalar; LANES] = [0; LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = unsafe { i.to_int_unchecked::<IntScalar>() };
+ }
+ result
+ };
+ test_helpers::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ },
+ ).unwrap();
+ }
+ }
+ }
+ }
+}
+
+float_rounding_test! { f32, i32 }
+float_rounding_test! { f64, i64 }
--- /dev/null
- fn make_runner() -> proptest::test_runner::TestRunner {
+pub mod array;
+
+#[cfg(target_arch = "wasm32")]
+pub mod wasm;
+
+#[macro_use]
+pub mod biteq;
+
+/// Specifies the default strategy for testing a type.
+///
+/// This strategy should be what "makes sense" to test.
+pub trait DefaultStrategy {
+ type Strategy: proptest::strategy::Strategy<Value = Self>;
+ fn default_strategy() -> Self::Strategy;
+}
+
+macro_rules! impl_num {
+ { $type:tt } => {
+ impl DefaultStrategy for $type {
+ type Strategy = proptest::num::$type::Any;
+ fn default_strategy() -> Self::Strategy {
+ proptest::num::$type::ANY
+ }
+ }
+ }
+}
+
+impl_num! { i8 }
+impl_num! { i16 }
+impl_num! { i32 }
+impl_num! { i64 }
+impl_num! { isize }
+impl_num! { u8 }
+impl_num! { u16 }
+impl_num! { u32 }
+impl_num! { u64 }
+impl_num! { usize }
+impl_num! { f32 }
+impl_num! { f64 }
+
+#[cfg(not(target_arch = "wasm32"))]
+impl DefaultStrategy for u128 {
+ type Strategy = proptest::num::u128::Any;
+ fn default_strategy() -> Self::Strategy {
+ proptest::num::u128::ANY
+ }
+}
+
+#[cfg(not(target_arch = "wasm32"))]
+impl DefaultStrategy for i128 {
+ type Strategy = proptest::num::i128::Any;
+ fn default_strategy() -> Self::Strategy {
+ proptest::num::i128::ANY
+ }
+}
+
+#[cfg(target_arch = "wasm32")]
+impl DefaultStrategy for u128 {
+ type Strategy = crate::wasm::u128::Any;
+ fn default_strategy() -> Self::Strategy {
+ crate::wasm::u128::ANY
+ }
+}
+
+#[cfg(target_arch = "wasm32")]
+impl DefaultStrategy for i128 {
+ type Strategy = crate::wasm::i128::Any;
+ fn default_strategy() -> Self::Strategy {
+ crate::wasm::i128::ANY
+ }
+}
+
+impl<T: core::fmt::Debug + DefaultStrategy, const LANES: usize> DefaultStrategy for [T; LANES] {
+ type Strategy = crate::array::UniformArrayStrategy<T::Strategy, Self>;
+ fn default_strategy() -> Self::Strategy {
+ Self::Strategy::new(T::default_strategy())
+ }
+}
+
+#[cfg(not(miri))]
- fn make_runner() -> proptest::test_runner::TestRunner {
++pub fn make_runner() -> proptest::test_runner::TestRunner {
+ Default::default()
+}
+#[cfg(miri)]
++pub fn make_runner() -> proptest::test_runner::TestRunner {
+ // Only run a few tests on Miri
+ proptest::test_runner::TestRunner::new(proptest::test_runner::Config::with_cases(4))
+}
+
+/// Test a function that takes a single value.
+pub fn test_1<A: core::fmt::Debug + DefaultStrategy>(
+ f: &dyn Fn(A) -> proptest::test_runner::TestCaseResult,
+) {
+ let mut runner = make_runner();
+ runner.run(&A::default_strategy(), f).unwrap();
+}
+
+/// Test a function that takes two values.
+pub fn test_2<A: core::fmt::Debug + DefaultStrategy, B: core::fmt::Debug + DefaultStrategy>(
+ f: &dyn Fn(A, B) -> proptest::test_runner::TestCaseResult,
+) {
+ let mut runner = make_runner();
+ runner
+ .run(&(A::default_strategy(), B::default_strategy()), |(a, b)| {
+ f(a, b)
+ })
+ .unwrap();
+}
+
+/// Test a function that takes two values.
+pub fn test_3<
+ A: core::fmt::Debug + DefaultStrategy,
+ B: core::fmt::Debug + DefaultStrategy,
+ C: core::fmt::Debug + DefaultStrategy,
+>(
+ f: &dyn Fn(A, B, C) -> proptest::test_runner::TestCaseResult,
+) {
+ let mut runner = make_runner();
+ runner
+ .run(
+ &(
+ A::default_strategy(),
+ B::default_strategy(),
+ C::default_strategy(),
+ ),
+ |(a, b, c)| f(a, b, c),
+ )
+ .unwrap();
+}
+
+/// Test a unary vector function against a unary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_unary_elementwise<Scalar, ScalarResult, Vector, VectorResult, const LANES: usize>(
+ fv: &dyn Fn(Vector) -> VectorResult,
+ fs: &dyn Fn(Scalar) -> ScalarResult,
+ check: &dyn Fn([Scalar; LANES]) -> bool,
+) where
+ Scalar: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar; LANES]> + From<[Scalar; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_1(&|x: [Scalar; LANES]| {
+ proptest::prop_assume!(check(x));
+ let result_1: [ScalarResult; LANES] = fv(x.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = fs(*i);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a unary vector function against a unary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_unary_mask_elementwise<Scalar, Vector, Mask, const LANES: usize>(
+ fv: &dyn Fn(Vector) -> Mask,
+ fs: &dyn Fn(Scalar) -> bool,
+ check: &dyn Fn([Scalar; LANES]) -> bool,
+) where
+ Scalar: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar; LANES]> + From<[Scalar; LANES]> + Copy,
+ Mask: Into<[bool; LANES]> + From<[bool; LANES]> + Copy,
+{
+ test_1(&|x: [Scalar; LANES]| {
+ proptest::prop_assume!(check(x));
+ let result_1: [bool; LANES] = fv(x.into()).into();
+ let result_2: [bool; LANES] = {
+ let mut result = [false; LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = fs(*i);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a binary vector function against a binary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_binary_elementwise<
+ Scalar1,
+ Scalar2,
+ ScalarResult,
+ Vector1,
+ Vector2,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Vector1, Vector2) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2) -> ScalarResult,
+ check: &dyn Fn([Scalar1; LANES], [Scalar2; LANES]) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector1: Into<[Scalar1; LANES]> + From<[Scalar1; LANES]> + Copy,
+ Vector2: Into<[Scalar2; LANES]> + From<[Scalar2; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_2(&|x: [Scalar1; LANES], y: [Scalar2; LANES]| {
+ proptest::prop_assume!(check(x, y));
+ let result_1: [ScalarResult; LANES] = fv(x.into(), y.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for ((i1, i2), o) in x.iter().zip(y.iter()).zip(result.iter_mut()) {
+ *o = fs(*i1, *i2);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a binary vector-scalar function against a binary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_binary_scalar_rhs_elementwise<
+ Scalar1,
+ Scalar2,
+ ScalarResult,
+ Vector,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Vector, Scalar2) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2) -> ScalarResult,
+ check: &dyn Fn([Scalar1; LANES], Scalar2) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar1; LANES]> + From<[Scalar1; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_2(&|x: [Scalar1; LANES], y: Scalar2| {
+ proptest::prop_assume!(check(x, y));
+ let result_1: [ScalarResult; LANES] = fv(x.into(), y).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = fs(*i, y);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a binary vector-scalar function against a binary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_binary_scalar_lhs_elementwise<
+ Scalar1,
+ Scalar2,
+ ScalarResult,
+ Vector,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Scalar1, Vector) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2) -> ScalarResult,
+ check: &dyn Fn(Scalar1, [Scalar2; LANES]) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar2; LANES]> + From<[Scalar2; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_2(&|x: Scalar1, y: [Scalar2; LANES]| {
+ proptest::prop_assume!(check(x, y));
+ let result_1: [ScalarResult; LANES] = fv(x, y.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for (i, o) in y.iter().zip(result.iter_mut()) {
+ *o = fs(x, *i);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a ternary vector function against a ternary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_ternary_elementwise<
+ Scalar1,
+ Scalar2,
+ Scalar3,
+ ScalarResult,
+ Vector1,
+ Vector2,
+ Vector3,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Vector1, Vector2, Vector3) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2, Scalar3) -> ScalarResult,
+ check: &dyn Fn([Scalar1; LANES], [Scalar2; LANES], [Scalar3; LANES]) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar3: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector1: Into<[Scalar1; LANES]> + From<[Scalar1; LANES]> + Copy,
+ Vector2: Into<[Scalar2; LANES]> + From<[Scalar2; LANES]> + Copy,
+ Vector3: Into<[Scalar3; LANES]> + From<[Scalar3; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_3(
+ &|x: [Scalar1; LANES], y: [Scalar2; LANES], z: [Scalar3; LANES]| {
+ proptest::prop_assume!(check(x, y, z));
+ let result_1: [ScalarResult; LANES] = fv(x.into(), y.into(), z.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for ((i1, (i2, i3)), o) in
+ x.iter().zip(y.iter().zip(z.iter())).zip(result.iter_mut())
+ {
+ *o = fs(*i1, *i2, *i3);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ },
+ );
+}
+
+/// Expand a const-generic test into separate tests for each possible lane count.
+#[macro_export]
+macro_rules! test_lanes {
+ {
+ $(fn $test:ident<const $lanes:ident: usize>() $body:tt)*
+ } => {
+ $(
+ mod $test {
+ use super::*;
+
+ fn implementation<const $lanes: usize>()
+ where
+ core_simd::LaneCount<$lanes>: core_simd::SupportedLaneCount,
+ $body
+
+ #[cfg(target_arch = "wasm32")]
+ wasm_bindgen_test::wasm_bindgen_test_configure!(run_in_browser);
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ fn lanes_1() {
+ implementation::<1>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ fn lanes_2() {
+ implementation::<2>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ fn lanes_4() {
+ implementation::<4>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_8() {
+ implementation::<8>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_16() {
+ implementation::<16>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_32() {
+ implementation::<32>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_64() {
+ implementation::<64>();
+ }
+ }
+ )*
+ }
+}
+
+/// Expand a const-generic `#[should_panic]` test into separate tests for each possible lane count.
+#[macro_export]
+macro_rules! test_lanes_panic {
+ {
+ $(fn $test:ident<const $lanes:ident: usize>() $body:tt)*
+ } => {
+ $(
+ mod $test {
+ use super::*;
+
+ fn implementation<const $lanes: usize>()
+ where
+ core_simd::LaneCount<$lanes>: core_simd::SupportedLaneCount,
+ $body
+
+ #[test]
+ #[should_panic]
+ fn lanes_1() {
+ implementation::<1>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_2() {
+ implementation::<2>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_4() {
+ implementation::<4>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_8() {
+ implementation::<8>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_16() {
+ implementation::<16>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_32() {
+ implementation::<32>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_64() {
+ implementation::<64>();
+ }
+ }
+ )*
+ }
+}
projects / rust.git / commitdiff