--- /dev/null
- * **Horizontal:** When an operation is "horizontal", the lanes within a single vector interact in some way. A "horizontal add" might add up lane 0 of `a` with lane 1 of `a`, with the total in lane 0 of `out`.
+
+# Beginner's Guide To SIMD
+
+Hello and welcome to our SIMD basics guide!
+
+Because SIMD is a subject that many programmers haven't worked with before, we thought that it's best to outline some terms and other basics for you to get started with.
+
+## Quick Background
+
+**SIMD** stands for *Single Instruction, Multiple Data*. In other words, SIMD is when the CPU performs a single action on more than one logical piece of data at the same time. Instead of adding two registers that each contain one `f32` value and getting an `f32` as the result, you might add two registers that each contain `f32x4` (128 bits of data) and then you get an `f32x4` as the output.
+
+This might seem a tiny bit weird at first, but there's a good reason for it. Back in the day, as CPUs got faster and faster, eventually they got so fast that the CPU would just melt itself. The heat management (heat sinks, fans, etc) simply couldn't keep up with how much electricity was going through the metal. Two main strategies were developed to help get around the limits of physics.
+* One of them you're probably familiar with: Multi-core processors. By giving a processor more than one core, each core can do its own work, and because they're physically distant (at least on the CPU's scale) the heat can still be managed. Unfortunately, not all tasks can just be split up across cores in an efficient way.
+* The second strategy is SIMD. If you can't make the register go any faster, you can still make the register *wider*. This lets you process more data at a time, which is *almost* as good as just having a faster CPU. As with multi-core programming, SIMD doesn't fit every kind of task, so you have to know when it will improve your program.
+
+## Terms
+
+SIMD has a few special vocabulary terms you should know:
+
+* **Vector:** A SIMD value is called a vector. This shouldn't be confused with the `Vec<T>` type. A SIMD vector has a fixed size, known at compile time. All of the elements within the vector are of the same type. This makes vectors *similar to* arrays. One difference is that a vector is generally aligned to its *entire* size (eg: 16 bytes, 32 bytes, etc), not just the size of an individual element. Sometimes vector data is called "packed" data.
+
+* **Vectorize**: An operation that uses SIMD instructions to operate over a vector is often referred to as "vectorized".
+
+* **Autovectorization**: Also known as _implicit vectorization_. This is when a compiler can automatically recognize a situation where scalar instructions may be replaced with SIMD instructions, and use those instead.
+
+* **Scalar:** "Scalar" in mathematical contexts refers to values that can be represented as a single element, mostly numbers like 6, 3.14, or -2. It can also be used to describe "scalar operations" that use strictly scalar values, like addition. This term is mostly used to differentiate between vectorized operations that use SIMD instructions and scalar operations that don't.
+
+* **Lane:** A single element position within a vector is called a lane. If you have `N` lanes available then they're numbered from `0` to `N-1` when referring to them, again like an array. The biggest difference between an array element and a vector lane is that in general is *relatively costly* to access an individual lane value. On most architectures, the vector has to be pushed out of the SIMD register onto the stack, then an individual lane is accessed while it's on the stack (and possibly the stack value is read back into a register). For this reason, when working with SIMD you should avoid reading or writing the value of an individual lane during hot loops.
+
+* **Bit Widths:** When talking about SIMD, the bit widths used are the bit size of the vectors involved, *not* the individual elements. So "128-bit SIMD" has 128-bit vectors, and that might be `f32x4`, `i32x4`, `i16x8`, or other variations. While 128-bit SIMD is the most common, there's also 64-bit, 256-bit, and even 512-bit on the newest CPUs.
+
+* **Vector Register:** The extra-wide registers that are used for SIMD operations are commonly called vector registers, though you may also see "SIMD registers", vendor names for specific features, or even "floating-point register" as it is common for the same registers to be used with both scalar and vectorized floating-point operations.
+
+* **Vertical:** When an operation is "vertical", each lane processes individually without regard to the other lanes in the same vector. For example, a "vertical add" between two vectors would add lane 0 in `a` with lane 0 in `b`, with the total in lane 0 of `out`, and then the same thing for lanes 1, 2, etc. Most SIMD operations are vertical operations, so if your problem is a vertical problem then you can probably solve it with SIMD.
+
- [`mem::align_of`]: https://doc.rust-lang.org/core/mem/fn.align_of.html
++* **Reducing/Reduce:** When an operation is "reducing" (functions named `reduce_*`), the lanes within a single vector are merged using some operation such as addition, returning the merged value as a scalar. For instance, a reducing add would return the sum of all the lanes' values.
+
+* **Target Feature:** Rust calls a CPU architecture extension a `target_feature`. Proper SIMD requires various CPU extensions to be enabled (details below). Don't confuse this with `feature`, which is a Cargo crate concept.
+
+## Target Features
+
+When using SIMD, you should be familiar with the CPU feature set that you're targeting.
+
+On `arm` and `aarch64` it's fairly simple. There's just one CPU feature that controls if SIMD is available: `neon` (or "NEON", all caps, as the ARM docs often put it). Neon registers can be used as 64-bit or 128-bit. When doing 128-bit operations it just uses two 64-bit registers as a single 128-bit register.
+
+> By default, the `aarch64`, `arm`, and `thumb` Rust targets generally do not enable `neon` unless it's in the target string.
+
+On `x86` and `x86_64` it's slightly more complicated. The SIMD support is split into many levels:
+* 128-bit: `sse`, `sse2`, `sse3`, `ssse3` (not a typo!), `sse4.1`, `sse4.2`, `sse4a` (AMD only)
+* 256-bit (mostly): `avx`, `avx2`, `fma`
+* 512-bit (mostly): a *wide* range of `avx512` variations
+
+The list notes the bit widths available at each feature level, though the operations of the more advanced features can generally be used with the smaller register sizes as well. For example, new operations introduced in `avx` generally have a 128-bit form as well as a 256-bit form. This means that even if you only do 128-bit work you can still benefit from the later feature levels.
+
+> By default, the `i686` and `x86_64` Rust targets enable `sse` and `sse2`.
+
+### Selecting Additional Target Features
+
+If you want to enable support for a target feature within your build, generally you should use a [target-feature](https://rust-lang.github.io/packed_simd/perf-guide/target-feature/rustflags.html#target-feature) setting within you `RUSTFLAGS` setting.
+
+If you know that you're targeting a specific CPU you can instead use the [target-cpu](https://rust-lang.github.io/packed_simd/perf-guide/target-feature/rustflags.html#target-cpu) flag and the compiler will enable the correct set of features for that CPU.
+
+The [Steam Hardware Survey](https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam) is one of the few places with data on how common various CPU features are. The dataset is limited to "the kinds of computers owned by people who play computer games", so the info only covers `x86`/`x86_64`, and it also probably skews to slightly higher quality computers than average. Still, we can see that the `sse` levels have very high support, `avx` and `avx2` are quite common as well, and the `avx-512` family is still so early in adoption you can barely find it in consumer grade stuff.
+
+## Running a program compiled for a CPU feature level that the CPU doesn't support is automatic undefined behavior.
+
+This means that if you build your program with `avx` support enabled and run it on a CPU without `avx` support, it's **instantly** undefined behavior.
+
+Even without an `unsafe` block in sight.
+
+This is no bug in Rust, or soundness hole in the type system. You just plain can't make a CPU do what it doesn't know how to do.
+
+This is why the various Rust targets *don't* enable many CPU feature flags by default: requiring a more advanced CPU makes the final binary *less* portable.
+
+So please select an appropriate CPU feature level when building your programs.
+
+## Size, Alignment, and Unsafe Code
+
+Most of the portable SIMD API is designed to allow the user to gloss over the details of different architectures and avoid using unsafe code. However, there are plenty of reasons to want to use unsafe code with these SIMD types, such as using an intrinsic function from `core::arch` to further accelerate particularly specialized SIMD operations on a given platform, while still using the portable API elsewhere. For these cases, there are some rules to keep in mind.
+
+Fortunately, most SIMD types have a fairly predictable size. `i32x4` is bit-equivalent to `[i32; 4]` and so can be bitcast to it, e.g. using [`mem::transmute`], though the API usually offers a safe cast you can use instead.
+
+However, this is not the same as alignment. Computer architectures generally prefer aligned accesses, especially when moving data between memory and vector registers, and while some support specialized operations that can bend the rules to help with this, unaligned access is still typically slow, or even undefined behavior. In addition, different architectures can require different alignments when interacting with their native SIMD types. For this reason, any `#[repr(simd)]` type has a non-portable alignment. If it is necessary to directly interact with the alignment of these types, it should be via [`mem::align_of`].
+
+[`mem::transmute`]: https://doc.rust-lang.org/core/mem/fn.transmute.html
++[`mem::align_of`]: https://doc.rust-lang.org/core/mem/fn.align_of.html
--- /dev/null
- default = ["std", "generic_const_exprs"]
+[package]
+name = "core_simd"
+version = "0.1.0"
+edition = "2021"
+homepage = "https://github.com/rust-lang/portable-simd"
+repository = "https://github.com/rust-lang/portable-simd"
+keywords = ["core", "simd", "intrinsics"]
+categories = ["hardware-support", "no-std"]
+license = "MIT OR Apache-2.0"
+
+[features]
++default = []
+std = []
+generic_const_exprs = []
+
+[target.'cfg(target_arch = "wasm32")'.dev-dependencies.wasm-bindgen]
+version = "0.2"
+
+[dev-dependencies.wasm-bindgen-test]
+version = "0.3"
+
+[dev-dependencies.proptest]
+version = "0.10"
+default-features = false
+features = ["alloc"]
+
+[dev-dependencies.test_helpers]
+path = "../test_helpers"
+
+[dev-dependencies]
+std_float = { path = "../std_float/", features = ["as_crate"] }
--- /dev/null
- if det.horizontal_sum() == 0. {
+//! 4x4 matrix inverse
+// Code ported from the `packed_simd` crate
+// Run this code with `cargo test --example matrix_inversion`
+#![feature(array_chunks, portable_simd)]
+use core_simd::simd::*;
+use Which::*;
+
+// Gotta define our own 4x4 matrix since Rust doesn't ship multidim arrays yet :^)
+#[derive(Copy, Clone, Debug, PartialEq, PartialOrd)]
+pub struct Matrix4x4([[f32; 4]; 4]);
+
+#[allow(clippy::too_many_lines)]
+pub fn scalar_inv4x4(m: Matrix4x4) -> Option<Matrix4x4> {
+ let m = m.0;
+
+ #[rustfmt::skip]
+ let mut inv = [
+ // row 0:
+ [
+ // 0,0:
+ m[1][1] * m[2][2] * m[3][3] -
+ m[1][1] * m[2][3] * m[3][2] -
+ m[2][1] * m[1][2] * m[3][3] +
+ m[2][1] * m[1][3] * m[3][2] +
+ m[3][1] * m[1][2] * m[2][3] -
+ m[3][1] * m[1][3] * m[2][2],
+ // 0,1:
+ -m[0][1] * m[2][2] * m[3][3] +
+ m[0][1] * m[2][3] * m[3][2] +
+ m[2][1] * m[0][2] * m[3][3] -
+ m[2][1] * m[0][3] * m[3][2] -
+ m[3][1] * m[0][2] * m[2][3] +
+ m[3][1] * m[0][3] * m[2][2],
+ // 0,2:
+ m[0][1] * m[1][2] * m[3][3] -
+ m[0][1] * m[1][3] * m[3][2] -
+ m[1][1] * m[0][2] * m[3][3] +
+ m[1][1] * m[0][3] * m[3][2] +
+ m[3][1] * m[0][2] * m[1][3] -
+ m[3][1] * m[0][3] * m[1][2],
+ // 0,3:
+ -m[0][1] * m[1][2] * m[2][3] +
+ m[0][1] * m[1][3] * m[2][2] +
+ m[1][1] * m[0][2] * m[2][3] -
+ m[1][1] * m[0][3] * m[2][2] -
+ m[2][1] * m[0][2] * m[1][3] +
+ m[2][1] * m[0][3] * m[1][2],
+ ],
+ // row 1
+ [
+ // 1,0:
+ -m[1][0] * m[2][2] * m[3][3] +
+ m[1][0] * m[2][3] * m[3][2] +
+ m[2][0] * m[1][2] * m[3][3] -
+ m[2][0] * m[1][3] * m[3][2] -
+ m[3][0] * m[1][2] * m[2][3] +
+ m[3][0] * m[1][3] * m[2][2],
+ // 1,1:
+ m[0][0] * m[2][2] * m[3][3] -
+ m[0][0] * m[2][3] * m[3][2] -
+ m[2][0] * m[0][2] * m[3][3] +
+ m[2][0] * m[0][3] * m[3][2] +
+ m[3][0] * m[0][2] * m[2][3] -
+ m[3][0] * m[0][3] * m[2][2],
+ // 1,2:
+ -m[0][0] * m[1][2] * m[3][3] +
+ m[0][0] * m[1][3] * m[3][2] +
+ m[1][0] * m[0][2] * m[3][3] -
+ m[1][0] * m[0][3] * m[3][2] -
+ m[3][0] * m[0][2] * m[1][3] +
+ m[3][0] * m[0][3] * m[1][2],
+ // 1,3:
+ m[0][0] * m[1][2] * m[2][3] -
+ m[0][0] * m[1][3] * m[2][2] -
+ m[1][0] * m[0][2] * m[2][3] +
+ m[1][0] * m[0][3] * m[2][2] +
+ m[2][0] * m[0][2] * m[1][3] -
+ m[2][0] * m[0][3] * m[1][2],
+ ],
+ // row 2
+ [
+ // 2,0:
+ m[1][0] * m[2][1] * m[3][3] -
+ m[1][0] * m[2][3] * m[3][1] -
+ m[2][0] * m[1][1] * m[3][3] +
+ m[2][0] * m[1][3] * m[3][1] +
+ m[3][0] * m[1][1] * m[2][3] -
+ m[3][0] * m[1][3] * m[2][1],
+ // 2,1:
+ -m[0][0] * m[2][1] * m[3][3] +
+ m[0][0] * m[2][3] * m[3][1] +
+ m[2][0] * m[0][1] * m[3][3] -
+ m[2][0] * m[0][3] * m[3][1] -
+ m[3][0] * m[0][1] * m[2][3] +
+ m[3][0] * m[0][3] * m[2][1],
+ // 2,2:
+ m[0][0] * m[1][1] * m[3][3] -
+ m[0][0] * m[1][3] * m[3][1] -
+ m[1][0] * m[0][1] * m[3][3] +
+ m[1][0] * m[0][3] * m[3][1] +
+ m[3][0] * m[0][1] * m[1][3] -
+ m[3][0] * m[0][3] * m[1][1],
+ // 2,3:
+ -m[0][0] * m[1][1] * m[2][3] +
+ m[0][0] * m[1][3] * m[2][1] +
+ m[1][0] * m[0][1] * m[2][3] -
+ m[1][0] * m[0][3] * m[2][1] -
+ m[2][0] * m[0][1] * m[1][3] +
+ m[2][0] * m[0][3] * m[1][1],
+ ],
+ // row 3
+ [
+ // 3,0:
+ -m[1][0] * m[2][1] * m[3][2] +
+ m[1][0] * m[2][2] * m[3][1] +
+ m[2][0] * m[1][1] * m[3][2] -
+ m[2][0] * m[1][2] * m[3][1] -
+ m[3][0] * m[1][1] * m[2][2] +
+ m[3][0] * m[1][2] * m[2][1],
+ // 3,1:
+ m[0][0] * m[2][1] * m[3][2] -
+ m[0][0] * m[2][2] * m[3][1] -
+ m[2][0] * m[0][1] * m[3][2] +
+ m[2][0] * m[0][2] * m[3][1] +
+ m[3][0] * m[0][1] * m[2][2] -
+ m[3][0] * m[0][2] * m[2][1],
+ // 3,2:
+ -m[0][0] * m[1][1] * m[3][2] +
+ m[0][0] * m[1][2] * m[3][1] +
+ m[1][0] * m[0][1] * m[3][2] -
+ m[1][0] * m[0][2] * m[3][1] -
+ m[3][0] * m[0][1] * m[1][2] +
+ m[3][0] * m[0][2] * m[1][1],
+ // 3,3:
+ m[0][0] * m[1][1] * m[2][2] -
+ m[0][0] * m[1][2] * m[2][1] -
+ m[1][0] * m[0][1] * m[2][2] +
+ m[1][0] * m[0][2] * m[2][1] +
+ m[2][0] * m[0][1] * m[1][2] -
+ m[2][0] * m[0][2] * m[1][1],
+ ],
+ ];
+
+ let det = m[0][0] * inv[0][0] + m[0][1] * inv[1][0] + m[0][2] * inv[2][0] + m[0][3] * inv[3][0];
+ if det == 0. {
+ return None;
+ }
+
+ let det_inv = 1. / det;
+
+ for row in &mut inv {
+ for elem in row.iter_mut() {
+ *elem *= det_inv;
+ }
+ }
+
+ Some(Matrix4x4(inv))
+}
+
+pub fn simd_inv4x4(m: Matrix4x4) -> Option<Matrix4x4> {
+ let m = m.0;
+ let m_0 = f32x4::from_array(m[0]);
+ let m_1 = f32x4::from_array(m[1]);
+ let m_2 = f32x4::from_array(m[2]);
+ let m_3 = f32x4::from_array(m[3]);
+
+ const SHUFFLE01: [Which; 4] = [First(0), First(1), Second(0), Second(1)];
+ const SHUFFLE02: [Which; 4] = [First(0), First(2), Second(0), Second(2)];
+ const SHUFFLE13: [Which; 4] = [First(1), First(3), Second(1), Second(3)];
+ const SHUFFLE23: [Which; 4] = [First(2), First(3), Second(2), Second(3)];
+
+ let tmp = simd_swizzle!(m_0, m_1, SHUFFLE01);
+ let row1 = simd_swizzle!(m_2, m_3, SHUFFLE01);
+
+ let row0 = simd_swizzle!(tmp, row1, SHUFFLE02);
+ let row1 = simd_swizzle!(row1, tmp, SHUFFLE13);
+
+ let tmp = simd_swizzle!(m_0, m_1, SHUFFLE23);
+ let row3 = simd_swizzle!(m_2, m_3, SHUFFLE23);
+ let row2 = simd_swizzle!(tmp, row3, SHUFFLE02);
+ let row3 = simd_swizzle!(row3, tmp, SHUFFLE13);
+
+ let tmp = (row2 * row3).reverse().rotate_lanes_right::<2>();
+ let minor0 = row1 * tmp;
+ let minor1 = row0 * tmp;
+ let tmp = tmp.rotate_lanes_right::<2>();
+ let minor0 = (row1 * tmp) - minor0;
+ let minor1 = (row0 * tmp) - minor1;
+ let minor1 = minor1.rotate_lanes_right::<2>();
+
+ let tmp = (row1 * row2).reverse().rotate_lanes_right::<2>();
+ let minor0 = (row3 * tmp) + minor0;
+ let minor3 = row0 * tmp;
+ let tmp = tmp.rotate_lanes_right::<2>();
+
+ let minor0 = minor0 - row3 * tmp;
+ let minor3 = row0 * tmp - minor3;
+ let minor3 = minor3.rotate_lanes_right::<2>();
+
+ let tmp = (row3 * row1.rotate_lanes_right::<2>())
+ .reverse()
+ .rotate_lanes_right::<2>();
+ let row2 = row2.rotate_lanes_right::<2>();
+ let minor0 = row2 * tmp + minor0;
+ let minor2 = row0 * tmp;
+ let tmp = tmp.rotate_lanes_right::<2>();
+ let minor0 = minor0 - row2 * tmp;
+ let minor2 = row0 * tmp - minor2;
+ let minor2 = minor2.rotate_lanes_right::<2>();
+
+ let tmp = (row0 * row1).reverse().rotate_lanes_right::<2>();
+ let minor2 = minor2 + row3 * tmp;
+ let minor3 = row2 * tmp - minor3;
+ let tmp = tmp.rotate_lanes_right::<2>();
+ let minor2 = row3 * tmp - minor2;
+ let minor3 = minor3 - row2 * tmp;
+
+ let tmp = (row0 * row3).reverse().rotate_lanes_right::<2>();
+ let minor1 = minor1 - row2 * tmp;
+ let minor2 = row1 * tmp + minor2;
+ let tmp = tmp.rotate_lanes_right::<2>();
+ let minor1 = row2 * tmp + minor1;
+ let minor2 = minor2 - row1 * tmp;
+
+ let tmp = (row0 * row2).reverse().rotate_lanes_right::<2>();
+ let minor1 = row3 * tmp + minor1;
+ let minor3 = minor3 - row1 * tmp;
+ let tmp = tmp.rotate_lanes_right::<2>();
+ let minor1 = minor1 - row3 * tmp;
+ let minor3 = row1 * tmp + minor3;
+
+ let det = row0 * minor0;
+ let det = det.rotate_lanes_right::<2>() + det;
+ let det = det.reverse().rotate_lanes_right::<2>() + det;
+
++ if det.reduce_sum() == 0. {
+ return None;
+ }
+ // calculate the reciprocal
+ let tmp = f32x4::splat(1.0) / det;
+ let det = tmp + tmp - det * tmp * tmp;
+
+ let res0 = minor0 * det;
+ let res1 = minor1 * det;
+ let res2 = minor2 * det;
+ let res3 = minor3 * det;
+
+ let mut m = m;
+
+ m[0] = res0.to_array();
+ m[1] = res1.to_array();
+ m[2] = res2.to_array();
+ m[3] = res3.to_array();
+
+ Some(Matrix4x4(m))
+}
+
+#[cfg(test)]
+#[rustfmt::skip]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test() {
+ let tests: &[(Matrix4x4, Option<Matrix4x4>)] = &[
+ // Identity:
+ (Matrix4x4([
+ [1., 0., 0., 0.],
+ [0., 1., 0., 0.],
+ [0., 0., 1., 0.],
+ [0., 0., 0., 1.],
+ ]),
+ Some(Matrix4x4([
+ [1., 0., 0., 0.],
+ [0., 1., 0., 0.],
+ [0., 0., 1., 0.],
+ [0., 0., 0., 1.],
+ ]))
+ ),
+ // None:
+ (Matrix4x4([
+ [1., 2., 3., 4.],
+ [12., 11., 10., 9.],
+ [5., 6., 7., 8.],
+ [16., 15., 14., 13.],
+ ]),
+ None
+ ),
+ // Other:
+ (Matrix4x4([
+ [1., 1., 1., 0.],
+ [0., 3., 1., 2.],
+ [2., 3., 1., 0.],
+ [1., 0., 2., 1.],
+ ]),
+ Some(Matrix4x4([
+ [-3., -0.5, 1.5, 1.0],
+ [ 1., 0.25, -0.25, -0.5],
+ [ 3., 0.25, -1.25, -0.5],
+ [-3., 0.0, 1.0, 1.0],
+ ]))
+ ),
+
+
+ ];
+
+ for &(input, output) in tests {
+ assert_eq!(scalar_inv4x4(input), output);
+ assert_eq!(simd_inv4x4(input), output);
+ }
+ }
+}
+
+fn main() {
+ // Empty main to make cargo happy
+}
--- /dev/null
- e += bi.mass * (bi.v * bi.v).horizontal_sum() * 0.5;
+#![feature(portable_simd)]
+extern crate std_float;
+
+/// Benchmarks game nbody code
+/// Taken from the `packed_simd` crate
+/// Run this benchmark with `cargo test --example nbody`
+mod nbody {
+ use core_simd::simd::*;
+ #[allow(unused)] // False positive?
+ use std_float::StdFloat;
+
+ use std::f64::consts::PI;
+ const SOLAR_MASS: f64 = 4.0 * PI * PI;
+ const DAYS_PER_YEAR: f64 = 365.24;
+
+ #[derive(Debug, Clone, Copy)]
+ struct Body {
+ pub x: f64x4,
+ pub v: f64x4,
+ pub mass: f64,
+ }
+
+ const N_BODIES: usize = 5;
+ const BODIES: [Body; N_BODIES] = [
+ // sun:
+ Body {
+ x: f64x4::from_array([0., 0., 0., 0.]),
+ v: f64x4::from_array([0., 0., 0., 0.]),
+ mass: SOLAR_MASS,
+ },
+ // jupiter:
+ Body {
+ x: f64x4::from_array([
+ 4.84143144246472090e+00,
+ -1.16032004402742839e+00,
+ -1.03622044471123109e-01,
+ 0.,
+ ]),
+ v: f64x4::from_array([
+ 1.66007664274403694e-03 * DAYS_PER_YEAR,
+ 7.69901118419740425e-03 * DAYS_PER_YEAR,
+ -6.90460016972063023e-05 * DAYS_PER_YEAR,
+ 0.,
+ ]),
+ mass: 9.54791938424326609e-04 * SOLAR_MASS,
+ },
+ // saturn:
+ Body {
+ x: f64x4::from_array([
+ 8.34336671824457987e+00,
+ 4.12479856412430479e+00,
+ -4.03523417114321381e-01,
+ 0.,
+ ]),
+ v: f64x4::from_array([
+ -2.76742510726862411e-03 * DAYS_PER_YEAR,
+ 4.99852801234917238e-03 * DAYS_PER_YEAR,
+ 2.30417297573763929e-05 * DAYS_PER_YEAR,
+ 0.,
+ ]),
+ mass: 2.85885980666130812e-04 * SOLAR_MASS,
+ },
+ // uranus:
+ Body {
+ x: f64x4::from_array([
+ 1.28943695621391310e+01,
+ -1.51111514016986312e+01,
+ -2.23307578892655734e-01,
+ 0.,
+ ]),
+ v: f64x4::from_array([
+ 2.96460137564761618e-03 * DAYS_PER_YEAR,
+ 2.37847173959480950e-03 * DAYS_PER_YEAR,
+ -2.96589568540237556e-05 * DAYS_PER_YEAR,
+ 0.,
+ ]),
+ mass: 4.36624404335156298e-05 * SOLAR_MASS,
+ },
+ // neptune:
+ Body {
+ x: f64x4::from_array([
+ 1.53796971148509165e+01,
+ -2.59193146099879641e+01,
+ 1.79258772950371181e-01,
+ 0.,
+ ]),
+ v: f64x4::from_array([
+ 2.68067772490389322e-03 * DAYS_PER_YEAR,
+ 1.62824170038242295e-03 * DAYS_PER_YEAR,
+ -9.51592254519715870e-05 * DAYS_PER_YEAR,
+ 0.,
+ ]),
+ mass: 5.15138902046611451e-05 * SOLAR_MASS,
+ },
+ ];
+
+ fn offset_momentum(bodies: &mut [Body; N_BODIES]) {
+ let (sun, rest) = bodies.split_at_mut(1);
+ let sun = &mut sun[0];
+ for body in rest {
+ let m_ratio = body.mass / SOLAR_MASS;
+ sun.v -= body.v * Simd::splat(m_ratio);
+ }
+ }
+
+ fn energy(bodies: &[Body; N_BODIES]) -> f64 {
+ let mut e = 0.;
+ for i in 0..N_BODIES {
+ let bi = &bodies[i];
- e -= bi.mass * bj.mass / (dx * dx).horizontal_sum().sqrt()
++ e += bi.mass * (bi.v * bi.v).reduce_sum() * 0.5;
+ for bj in bodies.iter().take(N_BODIES).skip(i + 1) {
+ let dx = bi.x - bj.x;
- (r[i] * r[i]).horizontal_sum(),
- (r[i + 1] * r[i + 1]).horizontal_sum(),
++ e -= bi.mass * bj.mass / (dx * dx).reduce_sum().sqrt()
+ }
+ }
+ e
+ }
+
+ fn advance(bodies: &mut [Body; N_BODIES], dt: f64) {
+ const N: usize = N_BODIES * (N_BODIES - 1) / 2;
+
+ // compute distance between bodies:
+ let mut r = [f64x4::splat(0.); N];
+ {
+ let mut i = 0;
+ for j in 0..N_BODIES {
+ for k in j + 1..N_BODIES {
+ r[i] = bodies[j].x - bodies[k].x;
+ i += 1;
+ }
+ }
+ }
+
+ let mut mag = [0.0; N];
+ for i in (0..N).step_by(2) {
+ let d2s = f64x2::from_array([
++ (r[i] * r[i]).reduce_sum(),
++ (r[i + 1] * r[i + 1]).reduce_sum(),
+ ]);
+ let dmags = f64x2::splat(dt) / (d2s * d2s.sqrt());
+ mag[i] = dmags[0];
+ mag[i + 1] = dmags[1];
+ }
+
+ let mut i = 0;
+ for j in 0..N_BODIES {
+ for k in j + 1..N_BODIES {
+ let f = r[i] * Simd::splat(mag[i]);
+ bodies[j].v -= f * Simd::splat(bodies[k].mass);
+ bodies[k].v += f * Simd::splat(bodies[j].mass);
+ i += 1
+ }
+ }
+ for body in bodies {
+ body.x += Simd::splat(dt) * body.v
+ }
+ }
+
+ pub fn run(n: usize) -> (f64, f64) {
+ let mut bodies = BODIES;
+ offset_momentum(&mut bodies);
+ let energy_before = energy(&bodies);
+ for _ in 0..n {
+ advance(&mut bodies, 0.01);
+ }
+ let energy_after = energy(&bodies);
+
+ (energy_before, energy_after)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ // Good enough for demonstration purposes, not going for strictness here.
+ fn approx_eq_f64(a: f64, b: f64) -> bool {
+ (a - b).abs() < 0.00001
+ }
+ #[test]
+ fn test() {
+ const OUTPUT: [f64; 2] = [-0.169075164, -0.169087605];
+ let (energy_before, energy_after) = super::nbody::run(1000);
+ assert!(approx_eq_f64(energy_before, OUTPUT[0]));
+ assert!(approx_eq_f64(energy_after, OUTPUT[1]));
+ }
+}
+
+fn main() {
+ {
+ let (energy_before, energy_after) = nbody::run(1000);
+ println!("Energy before: {energy_before}");
+ println!("Energy after: {energy_after}");
+ }
+}
--- /dev/null
- *out = sum.horizontal_sum();
+#![feature(portable_simd)]
+
+use core_simd::simd::*;
+
+fn a(i: usize, j: usize) -> f64 {
+ ((i + j) * (i + j + 1) / 2 + i + 1) as f64
+}
+
+fn mult_av(v: &[f64], out: &mut [f64]) {
+ assert!(v.len() == out.len());
+ assert!(v.len() % 2 == 0);
+
+ for (i, out) in out.iter_mut().enumerate() {
+ let mut sum = f64x2::splat(0.0);
+
+ let mut j = 0;
+ while j < v.len() {
+ let b = f64x2::from_slice(&v[j..]);
+ let a = f64x2::from_array([a(i, j), a(i, j + 1)]);
+ sum += b / a;
+ j += 2
+ }
- *out = sum.horizontal_sum();
++ *out = sum.reduce_sum();
+ }
+}
+
+fn mult_atv(v: &[f64], out: &mut [f64]) {
+ assert!(v.len() == out.len());
+ assert!(v.len() % 2 == 0);
+
+ for (i, out) in out.iter_mut().enumerate() {
+ let mut sum = f64x2::splat(0.0);
+
+ let mut j = 0;
+ while j < v.len() {
+ let b = f64x2::from_slice(&v[j..]);
+ let a = f64x2::from_array([a(j, i), a(j + 1, i)]);
+ sum += b / a;
+ j += 2
+ }
++ *out = sum.reduce_sum();
+ }
+}
+
+fn mult_atav(v: &[f64], out: &mut [f64], tmp: &mut [f64]) {
+ mult_av(v, tmp);
+ mult_atv(tmp, out);
+}
+
+pub fn spectral_norm(n: usize) -> f64 {
+ assert!(n % 2 == 0, "only even lengths are accepted");
+
+ let mut u = vec![1.0; n];
+ let mut v = u.clone();
+ let mut tmp = u.clone();
+
+ for _ in 0..10 {
+ mult_atav(&u, &mut v, &mut tmp);
+ mult_atav(&v, &mut u, &mut tmp);
+ }
+ (dot(&u, &v) / dot(&v, &v)).sqrt()
+}
+
+fn dot(x: &[f64], y: &[f64]) -> f64 {
+ // This is auto-vectorized:
+ x.iter().zip(y).map(|(&x, &y)| x * y).sum()
+}
+
+#[cfg(test)]
+#[test]
+fn test() {
+ assert_eq!(&format!("{:.9}", spectral_norm(100)), "1.274219991");
+}
+
+fn main() {
+ // Empty main to make cargo happy
+}
--- /dev/null
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Mask, Simd, SimdElement, SupportedLaneCount};
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ T: SimdElement + PartialEq,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ /// Test if each lane is equal to the corresponding lane in `other`.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn lanes_eq(self, other: Self) -> Mask<T::Mask, LANES> {
+ // Safety: `self` is a vector, and the result of the comparison
+ // is always a valid mask.
+ unsafe { Mask::from_int_unchecked(intrinsics::simd_eq(self, other)) }
+ }
+
+ /// Test if each lane is not equal to the corresponding lane in `other`.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn lanes_ne(self, other: Self) -> Mask<T::Mask, LANES> {
+ // Safety: `self` is a vector, and the result of the comparison
+ // is always a valid mask.
+ unsafe { Mask::from_int_unchecked(intrinsics::simd_ne(self, other)) }
+ }
+}
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ T: SimdElement + PartialOrd,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ /// Test if each lane is less than the corresponding lane in `other`.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn lanes_lt(self, other: Self) -> Mask<T::Mask, LANES> {
+ // Safety: `self` is a vector, and the result of the comparison
+ // is always a valid mask.
+ unsafe { Mask::from_int_unchecked(intrinsics::simd_lt(self, other)) }
+ }
+
+ /// Test if each lane is greater than the corresponding lane in `other`.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn lanes_gt(self, other: Self) -> Mask<T::Mask, LANES> {
+ // Safety: `self` is a vector, and the result of the comparison
+ // is always a valid mask.
+ unsafe { Mask::from_int_unchecked(intrinsics::simd_gt(self, other)) }
+ }
+
+ /// Test if each lane is less than or equal to the corresponding lane in `other`.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn lanes_le(self, other: Self) -> Mask<T::Mask, LANES> {
+ // Safety: `self` is a vector, and the result of the comparison
+ // is always a valid mask.
+ unsafe { Mask::from_int_unchecked(intrinsics::simd_le(self, other)) }
+ }
+
+ /// Test if each lane is greater than or equal to the corresponding lane in `other`.
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original value"]
+ pub fn lanes_ge(self, other: Self) -> Mask<T::Mask, LANES> {
+ // Safety: `self` is a vector, and the result of the comparison
+ // is always a valid mask.
+ unsafe { Mask::from_int_unchecked(intrinsics::simd_ge(self, other)) }
+ }
+}
++
++macro_rules! impl_ord_methods_vector {
++ { $type:ty } => {
++ impl<const LANES: usize> Simd<$type, LANES>
++ where
++ LaneCount<LANES>: SupportedLaneCount,
++ {
++ /// Returns the lane-wise minimum with `other`.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ #[inline]
++ pub fn min(self, other: Self) -> Self {
++ self.lanes_gt(other).select(other, self)
++ }
++
++ /// Returns the lane-wise maximum with `other`.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ #[inline]
++ pub fn max(self, other: Self) -> Self {
++ self.lanes_lt(other).select(other, self)
++ }
++
++ /// Restrict each lane to a certain interval.
++ ///
++ /// For each lane, returns `max` if `self` is greater than `max`, and `min` if `self` is
++ /// less than `min`. Otherwise returns `self`.
++ ///
++ /// # Panics
++ ///
++ /// Panics if `min > max` on any lane.
++ #[must_use = "method returns a new vector and does not mutate the original value"]
++ #[inline]
++ pub fn clamp(self, min: Self, max: Self) -> Self {
++ assert!(
++ min.lanes_le(max).all(),
++ "each lane in `min` must be less than or equal to the corresponding lane in `max`",
++ );
++ self.max(min).min(max)
++ }
++ }
++ }
++}
++
++impl_ord_methods_vector!(i8);
++impl_ord_methods_vector!(i16);
++impl_ord_methods_vector!(i32);
++impl_ord_methods_vector!(i64);
++impl_ord_methods_vector!(isize);
++impl_ord_methods_vector!(u8);
++impl_ord_methods_vector!(u16);
++impl_ord_methods_vector!(u32);
++impl_ord_methods_vector!(u64);
++impl_ord_methods_vector!(usize);
--- /dev/null
-
+//! This module contains the LLVM intrinsics bindings that provide the functionality for this
+//! crate.
+//!
+//! The LLVM assembly language is documented here: <https://llvm.org/docs/LangRef.html>
+//!
+//! A quick glossary of jargon that may appear in this module, mostly paraphrasing LLVM's LangRef:
+//! - poison: "undefined behavior as a value". specifically, it is like uninit memory (such as padding bytes). it is "safe" to create poison, BUT
+//! poison MUST NOT be observed from safe code, as operations on poison return poison, like NaN. unlike NaN, which has defined comparisons,
+//! poison is neither true nor false, and LLVM may also convert it to undef (at which point it is both). so, it can't be conditioned on, either.
+//! - undef: "a value that is every value". functionally like poison, insofar as Rust is concerned. poison may become this. note:
+//! this means that division by poison or undef is like division by zero, which means it inflicts...
+//! - "UB": poison and undef cover most of what people call "UB". "UB" means this operation immediately invalidates the program:
+//! LLVM is allowed to lower it to `ud2` or other opcodes that may cause an illegal instruction exception, and this is the "good end".
+//! The "bad end" is that LLVM may reverse time to the moment control flow diverged on a path towards undefined behavior,
+//! and destroy the other branch, potentially deleting safe code and violating Rust's `unsafe` contract.
+//!
+//! Note that according to LLVM, vectors are not arrays, but they are equivalent when stored to and loaded from memory.
+//!
+//! Unless stated otherwise, all intrinsics for binary operations require SIMD vectors of equal types and lengths.
+
+// These intrinsics aren't linked directly from LLVM and are mostly undocumented, however they are
+// mostly lowered to the matching LLVM instructions by the compiler in a fairly straightforward manner.
+// The associated LLVM instruction or intrinsic is documented alongside each Rust intrinsic function.
+extern "platform-intrinsic" {
+ /// add/fadd
+ pub(crate) fn simd_add<T>(x: T, y: T) -> T;
+
+ /// sub/fsub
+ pub(crate) fn simd_sub<T>(lhs: T, rhs: T) -> T;
+
+ /// mul/fmul
+ pub(crate) fn simd_mul<T>(x: T, y: T) -> T;
+
+ /// udiv/sdiv/fdiv
+ /// ints and uints: {s,u}div incur UB if division by zero occurs.
+ /// ints: sdiv is UB for int::MIN / -1.
+ /// floats: fdiv is never UB, but may create NaNs or infinities.
+ pub(crate) fn simd_div<T>(lhs: T, rhs: T) -> T;
+
+ /// urem/srem/frem
+ /// ints and uints: {s,u}rem incur UB if division by zero occurs.
+ /// ints: srem is UB for int::MIN / -1.
+ /// floats: frem is equivalent to libm::fmod in the "default" floating point environment, sans errno.
+ pub(crate) fn simd_rem<T>(lhs: T, rhs: T) -> T;
+
+ /// shl
+ /// for (u)ints. poison if rhs >= lhs::BITS
+ pub(crate) fn simd_shl<T>(lhs: T, rhs: T) -> T;
+
+ /// ints: ashr
+ /// uints: lshr
+ /// poison if rhs >= lhs::BITS
+ pub(crate) fn simd_shr<T>(lhs: T, rhs: T) -> T;
+
+ /// and
+ pub(crate) fn simd_and<T>(x: T, y: T) -> T;
+
+ /// or
+ pub(crate) fn simd_or<T>(x: T, y: T) -> T;
+
+ /// xor
+ pub(crate) fn simd_xor<T>(x: T, y: T) -> T;
+
+ /// fptoui/fptosi/uitofp/sitofp
+ /// casting floats to integers is truncating, so it is safe to convert values like e.g. 1.5
+ /// but the truncated value must fit in the target type or the result is poison.
+ /// use `simd_as` instead for a cast that performs a saturating conversion.
+ pub(crate) fn simd_cast<T, U>(x: T) -> U;
+ /// follows Rust's `T as U` semantics, including saturating float casts
+ /// which amounts to the same as `simd_cast` for many cases
+ pub(crate) fn simd_as<T, U>(x: T) -> U;
+
+ /// neg/fneg
+ /// ints: ultimately becomes a call to cg_ssa's BuilderMethods::neg. cg_llvm equates this to `simd_sub(Simd::splat(0), x)`.
+ /// floats: LLVM's fneg, which changes the floating point sign bit. Some arches have instructions for it.
+ /// Rust panics for Neg::neg(int::MIN) due to overflow, but it is not UB in LLVM without `nsw`.
+ pub(crate) fn simd_neg<T>(x: T) -> T;
+
+ /// fabs
+ pub(crate) fn simd_fabs<T>(x: T) -> T;
+
+ // minnum/maxnum
+ pub(crate) fn simd_fmin<T>(x: T, y: T) -> T;
+ pub(crate) fn simd_fmax<T>(x: T, y: T) -> T;
+
+ // these return Simd<int, N> with the same BITS size as the inputs
+ pub(crate) fn simd_eq<T, U>(x: T, y: T) -> U;
+ pub(crate) fn simd_ne<T, U>(x: T, y: T) -> U;
+ pub(crate) fn simd_lt<T, U>(x: T, y: T) -> U;
+ pub(crate) fn simd_le<T, U>(x: T, y: T) -> U;
+ pub(crate) fn simd_gt<T, U>(x: T, y: T) -> U;
+ pub(crate) fn simd_ge<T, U>(x: T, y: T) -> U;
+
+ // shufflevector
+ // idx: LLVM calls it a "shuffle mask vector constant", a vector of i32s
+ pub(crate) fn simd_shuffle<T, U, V>(x: T, y: T, idx: U) -> V;
+
+ /// llvm.masked.gather
+ /// like a loop of pointer reads
+ /// val: vector of values to select if a lane is masked
+ /// ptr: vector of pointers to read from
+ /// mask: a "wide" mask of integers, selects as if simd_select(mask, read(ptr), val)
+ /// note, the LLVM intrinsic accepts a mask vector of <N x i1>
+ /// FIXME: review this if/when we fix up our mask story in general?
+ pub(crate) fn simd_gather<T, U, V>(val: T, ptr: U, mask: V) -> T;
+ /// llvm.masked.scatter
+ /// like gather, but more spicy, as it writes instead of reads
+ pub(crate) fn simd_scatter<T, U, V>(val: T, ptr: U, mask: V);
+
+ // {s,u}add.sat
+ pub(crate) fn simd_saturating_add<T>(x: T, y: T) -> T;
+
+ // {s,u}sub.sat
+ pub(crate) fn simd_saturating_sub<T>(lhs: T, rhs: T) -> T;
+
+ // reductions
+ // llvm.vector.reduce.{add,fadd}
+ pub(crate) fn simd_reduce_add_ordered<T, U>(x: T, y: U) -> U;
+ // llvm.vector.reduce.{mul,fmul}
+ pub(crate) fn simd_reduce_mul_ordered<T, U>(x: T, y: U) -> U;
+ #[allow(unused)]
+ pub(crate) fn simd_reduce_all<T>(x: T) -> bool;
+ #[allow(unused)]
+ pub(crate) fn simd_reduce_any<T>(x: T) -> bool;
+ pub(crate) fn simd_reduce_max<T, U>(x: T) -> U;
+ pub(crate) fn simd_reduce_min<T, U>(x: T) -> U;
+ pub(crate) fn simd_reduce_and<T, U>(x: T) -> U;
+ pub(crate) fn simd_reduce_or<T, U>(x: T) -> U;
+ pub(crate) fn simd_reduce_xor<T, U>(x: T) -> U;
+
+ // truncate integer vector to bitmask
++ // `fn simd_bitmask(vector) -> unsigned integer` takes a vector of integers and
++ // returns either an unsigned integer or array of `u8`.
++ // Every element in the vector becomes a single bit in the returned bitmask.
++ // If the vector has less than 8 lanes, a u8 is returned with zeroed trailing bits.
++ // The bit order of the result depends on the byte endianness. LSB-first for little
++ // endian and MSB-first for big endian.
++ //
++ // UB if called on a vector with values other than 0 and -1.
+ #[allow(unused)]
+ pub(crate) fn simd_bitmask<T, U>(x: T) -> U;
+
+ // select
+ // first argument is a vector of integers, -1 (all bits 1) is "true"
+ // logically equivalent to (yes & m) | (no & (m^-1),
+ // but you can use it on floats.
+ pub(crate) fn simd_select<M, T>(m: M, yes: T, no: T) -> T;
+ #[allow(unused)]
+ pub(crate) fn simd_select_bitmask<M, T>(m: M, yes: T, no: T) -> T;
+}
--- /dev/null
- #![cfg_attr(not(feature = "std"), no_std)]
++#![no_std]
+#![feature(
- const_fn_trait_bound,
+ convert_float_to_int,
+ decl_macro,
+ intra_doc_pointers,
+ platform_intrinsics,
+ repr_simd,
+ simd_ffi,
+ staged_api,
+ stdsimd
+)]
+#![cfg_attr(feature = "generic_const_exprs", feature(generic_const_exprs))]
+#![cfg_attr(feature = "generic_const_exprs", allow(incomplete_features))]
+#![warn(missing_docs)]
+#![deny(unsafe_op_in_unsafe_fn)]
+#![unstable(feature = "portable_simd", issue = "86656")]
+//! Portable SIMD module.
+
+#[path = "mod.rs"]
+mod core_simd;
+pub use self::core_simd::simd;
+pub use simd::*;
--- /dev/null
+use super::{mask_impl, Mask, MaskElement};
+use crate::simd::{LaneCount, SupportedLaneCount};
+
+mod sealed {
+ pub trait Sealed {}
+}
+pub use sealed::Sealed;
+
+impl<T, const LANES: usize> Sealed for Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+/// Converts masks to and from integer bitmasks.
+///
+/// Each bit of the bitmask corresponds to a mask lane, starting with the LSB.
+///
+/// # Safety
+/// This trait is `unsafe` and sealed, since the `BitMask` type must match the number of lanes in
+/// the mask.
+pub unsafe trait ToBitMask: Sealed {
+ /// The integer bitmask type.
+ type BitMask;
+
+ /// Converts a mask to a bitmask.
+ fn to_bitmask(self) -> Self::BitMask;
+
+ /// Converts a bitmask to a mask.
+ fn from_bitmask(bitmask: Self::BitMask) -> Self;
+}
+
+macro_rules! impl_integer_intrinsic {
+ { $(unsafe impl ToBitMask<BitMask=$int:ty> for Mask<_, $lanes:literal>)* } => {
+ $(
+ unsafe impl<T: MaskElement> ToBitMask for Mask<T, $lanes> {
+ type BitMask = $int;
+
+ fn to_bitmask(self) -> $int {
+ self.0.to_bitmask_integer()
+ }
+
+ fn from_bitmask(bitmask: $int) -> Self {
+ Self(mask_impl::Mask::from_bitmask_integer(bitmask))
+ }
+ }
+ )*
+ }
+}
+
+impl_integer_intrinsic! {
++ unsafe impl ToBitMask<BitMask=u8> for Mask<_, 1>
++ unsafe impl ToBitMask<BitMask=u8> for Mask<_, 2>
++ unsafe impl ToBitMask<BitMask=u8> for Mask<_, 4>
+ unsafe impl ToBitMask<BitMask=u8> for Mask<_, 8>
+ unsafe impl ToBitMask<BitMask=u16> for Mask<_, 16>
+ unsafe impl ToBitMask<BitMask=u32> for Mask<_, 32>
+ unsafe impl ToBitMask<BitMask=u64> for Mask<_, 64>
+}
--- /dev/null
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
+use crate::simd::intrinsics::{simd_saturating_add, simd_saturating_sub};
+use crate::simd::{LaneCount, Simd, SupportedLaneCount};
+
+macro_rules! impl_uint_arith {
+ ($($ty:ty),+) => {
+ $( impl<const LANES: usize> Simd<$ty, LANES> where LaneCount<LANES>: SupportedLaneCount {
+
+ /// Lanewise saturating add.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ #[doc = concat!("# use core::", stringify!($ty), "::MAX;")]
+ /// let x = Simd::from_array([2, 1, 0, MAX]);
+ /// let max = Simd::splat(MAX);
+ /// let unsat = x + max;
+ /// let sat = x.saturating_add(max);
+ /// assert_eq!(unsat, Simd::from_array([1, 0, MAX, MAX - 1]));
+ /// assert_eq!(sat, max);
+ /// ```
+ #[inline]
+ pub fn saturating_add(self, second: Self) -> Self {
+ // Safety: `self` is a vector
+ unsafe { simd_saturating_add(self, second) }
+ }
+
+ /// Lanewise saturating subtract.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ #[doc = concat!("# use core::", stringify!($ty), "::MAX;")]
+ /// let x = Simd::from_array([2, 1, 0, MAX]);
+ /// let max = Simd::splat(MAX);
+ /// let unsat = x - max;
+ /// let sat = x.saturating_sub(max);
+ /// assert_eq!(unsat, Simd::from_array([3, 2, 1, 0]));
+ /// assert_eq!(sat, Simd::splat(0));
+ #[inline]
+ pub fn saturating_sub(self, second: Self) -> Self {
+ // Safety: `self` is a vector
+ unsafe { simd_saturating_sub(self, second) }
+ }
+ })+
+ }
+}
+
+macro_rules! impl_int_arith {
+ ($($ty:ty),+) => {
+ $( impl<const LANES: usize> Simd<$ty, LANES> where LaneCount<LANES>: SupportedLaneCount {
+
+ /// Lanewise saturating add.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ #[doc = concat!("# use core::", stringify!($ty), "::{MIN, MAX};")]
+ /// let x = Simd::from_array([MIN, 0, 1, MAX]);
+ /// let max = Simd::splat(MAX);
+ /// let unsat = x + max;
+ /// let sat = x.saturating_add(max);
+ /// assert_eq!(unsat, Simd::from_array([-1, MAX, MIN, -2]));
+ /// assert_eq!(sat, Simd::from_array([-1, MAX, MAX, MAX]));
+ /// ```
+ #[inline]
+ pub fn saturating_add(self, second: Self) -> Self {
+ // Safety: `self` is a vector
+ unsafe { simd_saturating_add(self, second) }
+ }
+
+ /// Lanewise saturating subtract.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ #[doc = concat!("# use core::", stringify!($ty), "::{MIN, MAX};")]
+ /// let x = Simd::from_array([MIN, -2, -1, MAX]);
+ /// let max = Simd::splat(MAX);
+ /// let unsat = x - max;
+ /// let sat = x.saturating_sub(max);
+ /// assert_eq!(unsat, Simd::from_array([1, MAX, MIN, 0]));
+ /// assert_eq!(sat, Simd::from_array([MIN, MIN, MIN, 0]));
+ #[inline]
+ pub fn saturating_sub(self, second: Self) -> Self {
+ // Safety: `self` is a vector
+ unsafe { simd_saturating_sub(self, second) }
+ }
+
+ /// Lanewise absolute value, implemented in Rust.
+ /// Every lane becomes its absolute value.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ #[doc = concat!("# use core::", stringify!($ty), "::{MIN, MAX};")]
+ /// let xs = Simd::from_array([MIN, MIN +1, -5, 0]);
+ /// assert_eq!(xs.abs(), Simd::from_array([MIN, MAX, 5, 0]));
+ /// ```
+ #[inline]
+ pub fn abs(self) -> Self {
+ const SHR: $ty = <$ty>::BITS as $ty - 1;
+ let m = self >> Simd::splat(SHR);
+ (self^m) - m
+ }
+
+ /// Lanewise saturating absolute value, implemented in Rust.
+ /// As abs(), except the MIN value becomes MAX instead of itself.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ #[doc = concat!("# use core::", stringify!($ty), "::{MIN, MAX};")]
+ /// let xs = Simd::from_array([MIN, -2, 0, 3]);
+ /// let unsat = xs.abs();
+ /// let sat = xs.saturating_abs();
+ /// assert_eq!(unsat, Simd::from_array([MIN, 2, 0, 3]));
+ /// assert_eq!(sat, Simd::from_array([MAX, 2, 0, 3]));
+ /// ```
+ #[inline]
+ pub fn saturating_abs(self) -> Self {
+ // arith shift for -1 or 0 mask based on sign bit, giving 2s complement
+ const SHR: $ty = <$ty>::BITS as $ty - 1;
+ let m = self >> Simd::splat(SHR);
+ (self^m).saturating_sub(m)
+ }
+
+ /// Lanewise saturating negation, implemented in Rust.
+ /// As neg(), except the MIN value becomes MAX instead of itself.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
++ /// # use core::simd::Simd;
+ #[doc = concat!("# use core::", stringify!($ty), "::{MIN, MAX};")]
+ /// let x = Simd::from_array([MIN, -2, 3, MAX]);
+ /// let unsat = -x;
+ /// let sat = x.saturating_neg();
+ /// assert_eq!(unsat, Simd::from_array([MIN, 2, -3, MIN + 1]));
+ /// assert_eq!(sat, Simd::from_array([MAX, 2, -3, MIN + 1]));
+ /// ```
+ #[inline]
+ pub fn saturating_neg(self) -> Self {
+ Self::splat(0).saturating_sub(self)
+ }
+ })+
+ }
+}
+
+impl_uint_arith! { u8, u16, u32, u64, usize }
+impl_int_arith! { i8, i16, i32, i64, isize }
--- /dev/null
- /// Horizontal wrapping add. Returns the sum of the lanes of the vector, with wrapping addition.
+use crate::simd::intrinsics::{
+ simd_reduce_add_ordered, simd_reduce_and, simd_reduce_max, simd_reduce_min,
+ simd_reduce_mul_ordered, simd_reduce_or, simd_reduce_xor,
+};
+use crate::simd::{LaneCount, Simd, SimdElement, SupportedLaneCount};
+use core::ops::{BitAnd, BitOr, BitXor};
+
+macro_rules! impl_integer_reductions {
+ { $scalar:ty } => {
+ impl<const LANES: usize> Simd<$scalar, LANES>
+ where
+ LaneCount<LANES>: SupportedLaneCount,
+ {
- pub fn horizontal_sum(self) -> $scalar {
++ /// Reducing wrapping add. Returns the sum of the lanes of the vector, with wrapping addition.
+ #[inline]
- /// Horizontal wrapping multiply. Returns the product of the lanes of the vector, with wrapping multiplication.
++ pub fn reduce_sum(self) -> $scalar {
+ // Safety: `self` is an integer vector
+ unsafe { simd_reduce_add_ordered(self, 0) }
+ }
+
- pub fn horizontal_product(self) -> $scalar {
++ /// Reducing wrapping multiply. Returns the product of the lanes of the vector, with wrapping multiplication.
+ #[inline]
- /// Horizontal maximum. Returns the maximum lane in the vector.
++ pub fn reduce_product(self) -> $scalar {
+ // Safety: `self` is an integer vector
+ unsafe { simd_reduce_mul_ordered(self, 1) }
+ }
+
- pub fn horizontal_max(self) -> $scalar {
++ /// Reducing maximum. Returns the maximum lane in the vector.
+ #[inline]
- /// Horizontal minimum. Returns the minimum lane in the vector.
++ pub fn reduce_max(self) -> $scalar {
+ // Safety: `self` is an integer vector
+ unsafe { simd_reduce_max(self) }
+ }
+
- pub fn horizontal_min(self) -> $scalar {
++ /// Reducing minimum. Returns the minimum lane in the vector.
+ #[inline]
- /// Horizontal add. Returns the sum of the lanes of the vector.
++ pub fn reduce_min(self) -> $scalar {
+ // Safety: `self` is an integer vector
+ unsafe { simd_reduce_min(self) }
+ }
+ }
+ }
+}
+
+impl_integer_reductions! { i8 }
+impl_integer_reductions! { i16 }
+impl_integer_reductions! { i32 }
+impl_integer_reductions! { i64 }
+impl_integer_reductions! { isize }
+impl_integer_reductions! { u8 }
+impl_integer_reductions! { u16 }
+impl_integer_reductions! { u32 }
+impl_integer_reductions! { u64 }
+impl_integer_reductions! { usize }
+
+macro_rules! impl_float_reductions {
+ { $scalar:ty } => {
+ impl<const LANES: usize> Simd<$scalar, LANES>
+ where
+ LaneCount<LANES>: SupportedLaneCount,
+ {
+
- pub fn horizontal_sum(self) -> $scalar {
++ /// Reducing add. Returns the sum of the lanes of the vector.
+ #[inline]
- /// Horizontal multiply. Returns the product of the lanes of the vector.
++ pub fn reduce_sum(self) -> $scalar {
+ // LLVM sum is inaccurate on i586
+ if cfg!(all(target_arch = "x86", not(target_feature = "sse2"))) {
+ self.as_array().iter().sum()
+ } else {
+ // Safety: `self` is a float vector
+ unsafe { simd_reduce_add_ordered(self, 0.) }
+ }
+ }
+
- pub fn horizontal_product(self) -> $scalar {
++ /// Reducing multiply. Returns the product of the lanes of the vector.
+ #[inline]
- /// Horizontal maximum. Returns the maximum lane in the vector.
++ pub fn reduce_product(self) -> $scalar {
+ // LLVM product is inaccurate on i586
+ if cfg!(all(target_arch = "x86", not(target_feature = "sse2"))) {
+ self.as_array().iter().product()
+ } else {
+ // Safety: `self` is a float vector
+ unsafe { simd_reduce_mul_ordered(self, 1.) }
+ }
+ }
+
- pub fn horizontal_max(self) -> $scalar {
++ /// Reducing maximum. Returns the maximum lane in the vector.
+ ///
+ /// Returns values based on equality, so a vector containing both `0.` and `-0.` may
+ /// return either. This function will not return `NaN` unless all lanes are `NaN`.
+ #[inline]
- /// Horizontal minimum. Returns the minimum lane in the vector.
++ pub fn reduce_max(self) -> $scalar {
+ // Safety: `self` is a float vector
+ unsafe { simd_reduce_max(self) }
+ }
+
- pub fn horizontal_min(self) -> $scalar {
++ /// Reducing minimum. Returns the minimum lane in the vector.
+ ///
+ /// Returns values based on equality, so a vector containing both `0.` and `-0.` may
+ /// return either. This function will not return `NaN` unless all lanes are `NaN`.
+ #[inline]
- /// Horizontal bitwise "and". Returns the cumulative bitwise "and" across the lanes of
++ pub fn reduce_min(self) -> $scalar {
+ // Safety: `self` is a float vector
+ unsafe { simd_reduce_min(self) }
+ }
+ }
+ }
+}
+
+impl_float_reductions! { f32 }
+impl_float_reductions! { f64 }
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ Self: BitAnd<Self, Output = Self>,
+ T: SimdElement + BitAnd<T, Output = T>,
+ LaneCount<LANES>: SupportedLaneCount,
+{
- pub fn horizontal_and(self) -> T {
++ /// Reducing bitwise "and". Returns the cumulative bitwise "and" across the lanes of
+ /// the vector.
+ #[inline]
- /// Horizontal bitwise "or". Returns the cumulative bitwise "or" across the lanes of
++ pub fn reduce_and(self) -> T {
+ unsafe { simd_reduce_and(self) }
+ }
+}
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ Self: BitOr<Self, Output = Self>,
+ T: SimdElement + BitOr<T, Output = T>,
+ LaneCount<LANES>: SupportedLaneCount,
+{
- pub fn horizontal_or(self) -> T {
++ /// Reducing bitwise "or". Returns the cumulative bitwise "or" across the lanes of
+ /// the vector.
+ #[inline]
- /// Horizontal bitwise "xor". Returns the cumulative bitwise "xor" across the lanes of
++ pub fn reduce_or(self) -> T {
+ unsafe { simd_reduce_or(self) }
+ }
+}
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ Self: BitXor<Self, Output = Self>,
+ T: SimdElement + BitXor<T, Output = T>,
+ LaneCount<LANES>: SupportedLaneCount,
+{
- pub fn horizontal_xor(self) -> T {
++ /// Reducing bitwise "xor". Returns the cumulative bitwise "xor" across the lanes of
+ /// the vector.
+ #[inline]
++ pub fn reduce_xor(self) -> T {
+ unsafe { simd_reduce_xor(self) }
+ }
+}
--- /dev/null
- /// # #[cfg(feature = "std")] use core_simd::{Simd, Mask};
- /// # #[cfg(not(feature = "std"))] use core::simd::{Simd, Mask};
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Mask, MaskElement, Simd, SimdElement, SupportedLaneCount};
+
+impl<T, const LANES: usize> Mask<T, LANES>
+where
+ T: MaskElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ /// Choose lanes from two vectors.
+ ///
+ /// For each lane in the mask, choose the corresponding lane from `true_values` if
+ /// that lane mask is true, and `false_values` if that lane mask is false.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Mask;
- /// # #[cfg(not(feature = "std"))] use core::simd::Mask;
++ /// # use core::simd::{Simd, Mask};
+ /// let a = Simd::from_array([0, 1, 2, 3]);
+ /// let b = Simd::from_array([4, 5, 6, 7]);
+ /// let mask = Mask::from_array([true, false, false, true]);
+ /// let c = mask.select(a, b);
+ /// assert_eq!(c.to_array(), [0, 5, 6, 3]);
+ /// ```
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn select<U>(
+ self,
+ true_values: Simd<U, LANES>,
+ false_values: Simd<U, LANES>,
+ ) -> Simd<U, LANES>
+ where
+ U: SimdElement<Mask = T>,
+ {
+ // Safety: The mask has been cast to a vector of integers,
+ // and the operands to select between are vectors of the same type and length.
+ unsafe { intrinsics::simd_select(self.to_int(), true_values, false_values) }
+ }
+
+ /// Choose lanes from two masks.
+ ///
+ /// For each lane in the mask, choose the corresponding lane from `true_values` if
+ /// that lane mask is true, and `false_values` if that lane mask is false.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
++ /// # use core::simd::Mask;
+ /// let a = Mask::<i32, 4>::from_array([true, true, false, false]);
+ /// let b = Mask::<i32, 4>::from_array([false, false, true, true]);
+ /// let mask = Mask::<i32, 4>::from_array([true, false, false, true]);
+ /// let c = mask.select_mask(a, b);
+ /// assert_eq!(c.to_array(), [true, false, true, false]);
+ /// ```
+ #[inline]
+ #[must_use = "method returns a new mask and does not mutate the original inputs"]
+ pub fn select_mask(self, true_values: Self, false_values: Self) -> Self {
+ self & true_values | !self & false_values
+ }
+}
--- /dev/null
- /// # #[cfg(feature = "std")] use core_simd::{Simd, simd_swizzle};
- /// # #[cfg(not(feature = "std"))] use core::simd::{Simd, simd_swizzle};
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Simd, SimdElement, SupportedLaneCount};
+
+/// Constructs a new vector by selecting values from the lanes of the source vector or vectors to use.
+///
+/// When swizzling one vector, the indices of the result vector are indicated by a `const` array
+/// of `usize`, like [`Swizzle`].
+/// When swizzling two vectors, the indices are indicated by a `const` array of [`Which`], like
+/// [`Swizzle2`].
+///
+/// # Examples
+/// ## One source vector
+/// ```
+/// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::{Simd, simd_swizzle, Which};
- /// # #[cfg(not(feature = "std"))] use core::simd::{Simd, simd_swizzle, Which};
++/// # use core::simd::{Simd, simd_swizzle};
+/// let v = Simd::<f32, 4>::from_array([0., 1., 2., 3.]);
+///
+/// // Keeping the same size
+/// let r = simd_swizzle!(v, [3, 0, 1, 2]);
+/// assert_eq!(r.to_array(), [3., 0., 1., 2.]);
+///
+/// // Changing the number of lanes
+/// let r = simd_swizzle!(v, [3, 1]);
+/// assert_eq!(r.to_array(), [3., 1.]);
+/// ```
+///
+/// ## Two source vectors
+/// ```
+/// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++/// # use core::simd::{Simd, simd_swizzle, Which};
+/// use Which::*;
+/// let a = Simd::<f32, 4>::from_array([0., 1., 2., 3.]);
+/// let b = Simd::<f32, 4>::from_array([4., 5., 6., 7.]);
+///
+/// // Keeping the same size
+/// let r = simd_swizzle!(a, b, [First(0), First(1), Second(2), Second(3)]);
+/// assert_eq!(r.to_array(), [0., 1., 6., 7.]);
+///
+/// // Changing the number of lanes
+/// let r = simd_swizzle!(a, b, [First(0), Second(0)]);
+/// assert_eq!(r.to_array(), [0., 4.]);
+/// ```
+#[allow(unused_macros)]
+pub macro simd_swizzle {
+ (
+ $vector:expr, $index:expr $(,)?
+ ) => {
+ {
+ use $crate::simd::Swizzle;
+ struct Impl;
+ impl<const LANES: usize> Swizzle<LANES, {$index.len()}> for Impl {
+ const INDEX: [usize; {$index.len()}] = $index;
+ }
+ Impl::swizzle($vector)
+ }
+ },
+ (
+ $first:expr, $second:expr, $index:expr $(,)?
+ ) => {
+ {
+ use $crate::simd::{Which, Swizzle2};
+ struct Impl;
+ impl<const LANES: usize> Swizzle2<LANES, {$index.len()}> for Impl {
+ const INDEX: [Which; {$index.len()}] = $index;
+ }
+ Impl::swizzle2($first, $second)
+ }
+ }
+}
+
+/// An index into one of two vectors.
+#[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub enum Which {
+ /// Indexes the first vector.
+ First(usize),
+ /// Indexes the second vector.
+ Second(usize),
+}
+
+/// Create a vector from the elements of another vector.
+pub trait Swizzle<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ /// Map from the lanes of the input vector to the output vector.
+ const INDEX: [usize; OUTPUT_LANES];
+
+ /// Create a new vector from the lanes of `vector`.
+ ///
+ /// Lane `i` of the output is `vector[Self::INDEX[i]]`.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ fn swizzle<T>(vector: Simd<T, INPUT_LANES>) -> Simd<T, OUTPUT_LANES>
+ where
+ T: SimdElement,
+ LaneCount<INPUT_LANES>: SupportedLaneCount,
+ LaneCount<OUTPUT_LANES>: SupportedLaneCount,
+ {
+ // Safety: `vector` is a vector, and `INDEX_IMPL` is a const array of u32.
+ unsafe { intrinsics::simd_shuffle(vector, vector, Self::INDEX_IMPL) }
+ }
+}
+
+/// Create a vector from the elements of two other vectors.
+pub trait Swizzle2<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ /// Map from the lanes of the input vectors to the output vector
+ const INDEX: [Which; OUTPUT_LANES];
+
+ /// Create a new vector from the lanes of `first` and `second`.
+ ///
+ /// Lane `i` is `first[j]` when `Self::INDEX[i]` is `First(j)`, or `second[j]` when it is
+ /// `Second(j)`.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ fn swizzle2<T>(
+ first: Simd<T, INPUT_LANES>,
+ second: Simd<T, INPUT_LANES>,
+ ) -> Simd<T, OUTPUT_LANES>
+ where
+ T: SimdElement,
+ LaneCount<INPUT_LANES>: SupportedLaneCount,
+ LaneCount<OUTPUT_LANES>: SupportedLaneCount,
+ {
+ // Safety: `first` and `second` are vectors, and `INDEX_IMPL` is a const array of u32.
+ unsafe { intrinsics::simd_shuffle(first, second, Self::INDEX_IMPL) }
+ }
+}
+
+/// The `simd_shuffle` intrinsic expects `u32`, so do error checking and conversion here.
+/// This trait hides `INDEX_IMPL` from the public API.
+trait SwizzleImpl<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ const INDEX_IMPL: [u32; OUTPUT_LANES];
+}
+
+impl<T, const INPUT_LANES: usize, const OUTPUT_LANES: usize> SwizzleImpl<INPUT_LANES, OUTPUT_LANES>
+ for T
+where
+ T: Swizzle<INPUT_LANES, OUTPUT_LANES> + ?Sized,
+{
+ const INDEX_IMPL: [u32; OUTPUT_LANES] = {
+ let mut output = [0; OUTPUT_LANES];
+ let mut i = 0;
+ while i < OUTPUT_LANES {
+ let index = Self::INDEX[i];
+ assert!(index as u32 as usize == index);
+ assert!(index < INPUT_LANES, "source lane exceeds input lane count",);
+ output[i] = index as u32;
+ i += 1;
+ }
+ output
+ };
+}
+
+/// The `simd_shuffle` intrinsic expects `u32`, so do error checking and conversion here.
+/// This trait hides `INDEX_IMPL` from the public API.
+trait Swizzle2Impl<const INPUT_LANES: usize, const OUTPUT_LANES: usize> {
+ const INDEX_IMPL: [u32; OUTPUT_LANES];
+}
+
+impl<T, const INPUT_LANES: usize, const OUTPUT_LANES: usize> Swizzle2Impl<INPUT_LANES, OUTPUT_LANES>
+ for T
+where
+ T: Swizzle2<INPUT_LANES, OUTPUT_LANES> + ?Sized,
+{
+ const INDEX_IMPL: [u32; OUTPUT_LANES] = {
+ let mut output = [0; OUTPUT_LANES];
+ let mut i = 0;
+ while i < OUTPUT_LANES {
+ let (offset, index) = match Self::INDEX[i] {
+ Which::First(index) => (false, index),
+ Which::Second(index) => (true, index),
+ };
+ assert!(index < INPUT_LANES, "source lane exceeds input lane count",);
+
+ // lanes are indexed by the first vector, then second vector
+ let index = if offset { index + INPUT_LANES } else { index };
+ assert!(index as u32 as usize == index);
+ output[i] = index as u32;
+ i += 1;
+ }
+ output
+ };
+}
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ /// Reverse the order of the lanes in the vector.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn reverse(self) -> Self {
+ const fn reverse_index<const LANES: usize>() -> [usize; LANES] {
+ let mut index = [0; LANES];
+ let mut i = 0;
+ while i < LANES {
+ index[i] = LANES - i - 1;
+ i += 1;
+ }
+ index
+ }
+
+ struct Reverse;
+
+ impl<const LANES: usize> Swizzle<LANES, LANES> for Reverse {
+ const INDEX: [usize; LANES] = reverse_index::<LANES>();
+ }
+
+ Reverse::swizzle(self)
+ }
+
+ /// Rotates the vector such that the first `OFFSET` elements of the slice move to the end
+ /// while the last `LANES - OFFSET` elements move to the front. After calling `rotate_lanes_left`,
+ /// the element previously in lane `OFFSET` will become the first element in the slice.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn rotate_lanes_left<const OFFSET: usize>(self) -> Self {
+ const fn rotate_index<const OFFSET: usize, const LANES: usize>() -> [usize; LANES] {
+ let offset = OFFSET % LANES;
+ let mut index = [0; LANES];
+ let mut i = 0;
+ while i < LANES {
+ index[i] = (i + offset) % LANES;
+ i += 1;
+ }
+ index
+ }
+
+ struct Rotate<const OFFSET: usize>;
+
+ impl<const OFFSET: usize, const LANES: usize> Swizzle<LANES, LANES> for Rotate<OFFSET> {
+ const INDEX: [usize; LANES] = rotate_index::<OFFSET, LANES>();
+ }
+
+ Rotate::<OFFSET>::swizzle(self)
+ }
+
+ /// Rotates the vector such that the first `LANES - OFFSET` elements of the vector move to
+ /// the end while the last `OFFSET` elements move to the front. After calling `rotate_lanes_right`,
+ /// the element previously at index `LANES - OFFSET` will become the first element in the slice.
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn rotate_lanes_right<const OFFSET: usize>(self) -> Self {
+ const fn rotate_index<const OFFSET: usize, const LANES: usize>() -> [usize; LANES] {
+ let offset = LANES - OFFSET % LANES;
+ let mut index = [0; LANES];
+ let mut i = 0;
+ while i < LANES {
+ index[i] = (i + offset) % LANES;
+ i += 1;
+ }
+ index
+ }
+
+ struct Rotate<const OFFSET: usize>;
+
+ impl<const OFFSET: usize, const LANES: usize> Swizzle<LANES, LANES> for Rotate<OFFSET> {
+ const INDEX: [usize; LANES] = rotate_index::<OFFSET, LANES>();
+ }
+
+ Rotate::<OFFSET>::swizzle(self)
+ }
+
+ /// Interleave two vectors.
+ ///
+ /// Produces two vectors with lanes taken alternately from `self` and `other`.
+ ///
+ /// The first result contains the first `LANES / 2` lanes from `self` and `other`,
+ /// alternating, starting with the first lane of `self`.
+ ///
+ /// The second result contains the last `LANES / 2` lanes from `self` and `other`,
+ /// alternating, starting with the lane `LANES / 2` from the start of `self`.
+ ///
+ /// ```
+ /// #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ /// let a = Simd::from_array([0, 1, 2, 3]);
+ /// let b = Simd::from_array([4, 5, 6, 7]);
+ /// let (x, y) = a.interleave(b);
+ /// assert_eq!(x.to_array(), [0, 4, 1, 5]);
+ /// assert_eq!(y.to_array(), [2, 6, 3, 7]);
+ /// ```
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn interleave(self, other: Self) -> (Self, Self) {
+ const fn lo<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES {
+ let offset = i / 2;
+ idx[i] = if i % 2 == 0 {
+ Which::First(offset)
+ } else {
+ Which::Second(offset)
+ };
+ i += 1;
+ }
+ idx
+ }
+ const fn hi<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES {
+ let offset = (LANES + i) / 2;
+ idx[i] = if i % 2 == 0 {
+ Which::First(offset)
+ } else {
+ Which::Second(offset)
+ };
+ i += 1;
+ }
+ idx
+ }
+
+ struct Lo;
+ struct Hi;
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Lo {
+ const INDEX: [Which; LANES] = lo::<LANES>();
+ }
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Hi {
+ const INDEX: [Which; LANES] = hi::<LANES>();
+ }
+
+ (Lo::swizzle2(self, other), Hi::swizzle2(self, other))
+ }
+
+ /// Deinterleave two vectors.
+ ///
+ /// The first result takes every other lane of `self` and then `other`, starting with
+ /// the first lane.
+ ///
+ /// The second result takes every other lane of `self` and then `other`, starting with
+ /// the second lane.
+ ///
+ /// ```
+ /// #![feature(portable_simd)]
++ /// # use core::simd::Simd;
+ /// let a = Simd::from_array([0, 4, 1, 5]);
+ /// let b = Simd::from_array([2, 6, 3, 7]);
+ /// let (x, y) = a.deinterleave(b);
+ /// assert_eq!(x.to_array(), [0, 1, 2, 3]);
+ /// assert_eq!(y.to_array(), [4, 5, 6, 7]);
+ /// ```
+ #[inline]
+ #[must_use = "method returns a new vector and does not mutate the original inputs"]
+ pub fn deinterleave(self, other: Self) -> (Self, Self) {
+ const fn even<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES / 2 {
+ idx[i] = Which::First(2 * i);
+ idx[i + LANES / 2] = Which::Second(2 * i);
+ i += 1;
+ }
+ idx
+ }
+ const fn odd<const LANES: usize>() -> [Which; LANES] {
+ let mut idx = [Which::First(0); LANES];
+ let mut i = 0;
+ while i < LANES / 2 {
+ idx[i] = Which::First(2 * i + 1);
+ idx[i + LANES / 2] = Which::Second(2 * i + 1);
+ i += 1;
+ }
+ idx
+ }
+
+ struct Even;
+ struct Odd;
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Even {
+ const INDEX: [Which; LANES] = even::<LANES>();
+ }
+
+ impl<const LANES: usize> Swizzle2<LANES, LANES> for Odd {
+ const INDEX: [Which; LANES] = odd::<LANES>();
+ }
+
+ (Even::swizzle2(self, other), Odd::swizzle2(self, other))
+ }
+}
--- /dev/null
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
+mod float;
+mod int;
+mod uint;
+
+pub use float::*;
+pub use int::*;
+pub use uint::*;
+
+// Vectors of pointers are not for public use at the current time.
+pub(crate) mod ptr;
+
+use crate::simd::intrinsics;
+use crate::simd::{LaneCount, Mask, MaskElement, SupportedLaneCount};
+
+/// A SIMD vector of `LANES` elements of type `T`. `Simd<T, N>` has the same shape as [`[T; N]`](array), but operates like `T`.
+///
+/// Two vectors of the same type and length will, by convention, support the operators (+, *, etc.) that `T` does.
+/// These take the lanes at each index on the left-hand side and right-hand side, perform the operation,
+/// and return the result in the same lane in a vector of equal size. For a given operator, this is equivalent to zipping
+/// the two arrays together and mapping the operator over each lane.
+///
+/// ```rust
+/// # #![feature(array_zip, portable_simd)]
+/// # use core::simd::{Simd};
+/// let a0: [i32; 4] = [-2, 0, 2, 4];
+/// let a1 = [10, 9, 8, 7];
+/// let zm_add = a0.zip(a1).map(|(lhs, rhs)| lhs + rhs);
+/// let zm_mul = a0.zip(a1).map(|(lhs, rhs)| lhs * rhs);
+///
+/// // `Simd<T, N>` implements `From<[T; N]>
+/// let (v0, v1) = (Simd::from(a0), Simd::from(a1));
+/// // Which means arrays implement `Into<Simd<T, N>>`.
+/// assert_eq!(v0 + v1, zm_add.into());
+/// assert_eq!(v0 * v1, zm_mul.into());
+/// ```
+///
+/// `Simd` with integers has the quirk that these operations are also inherently wrapping, as if `T` was [`Wrapping<T>`].
+/// Thus, `Simd` does not implement `wrapping_add`, because that is the default behavior.
+/// This means there is no warning on overflows, even in "debug" builds.
+/// For most applications where `Simd` is appropriate, it is "not a bug" to wrap,
+/// and even "debug builds" are unlikely to tolerate the loss of performance.
+/// You may want to consider using explicitly checked arithmetic if such is required.
+/// Division by zero still causes a panic, so you may want to consider using floating point numbers if that is unacceptable.
+///
+/// [`Wrapping<T>`]: core::num::Wrapping
+///
+/// # Layout
+/// `Simd<T, N>` has a layout similar to `[T; N]` (identical "shapes"), but with a greater alignment.
+/// `[T; N]` is aligned to `T`, but `Simd<T, N>` will have an alignment based on both `T` and `N`.
+/// It is thus sound to [`transmute`] `Simd<T, N>` to `[T; N]`, and will typically optimize to zero cost,
+/// but the reverse transmutation is more likely to require a copy the compiler cannot simply elide.
+///
+/// # ABI "Features"
+/// Due to Rust's safety guarantees, `Simd<T, N>` is currently passed to and from functions via memory, not SIMD registers,
+/// except as an optimization. `#[inline]` hints are recommended on functions that accept `Simd<T, N>` or return it.
+/// The need for this may be corrected in the future.
+///
+/// # Safe SIMD with Unsafe Rust
+///
+/// Operations with `Simd` are typically safe, but there are many reasons to want to combine SIMD with `unsafe` code.
+/// Care must be taken to respect differences between `Simd` and other types it may be transformed into or derived from.
+/// In particular, the layout of `Simd<T, N>` may be similar to `[T; N]`, and may allow some transmutations,
+/// but references to `[T; N]` are not interchangeable with those to `Simd<T, N>`.
+/// Thus, when using `unsafe` Rust to read and write `Simd<T, N>` through [raw pointers], it is a good idea to first try with
+/// [`read_unaligned`] and [`write_unaligned`]. This is because:
+/// - [`read`] and [`write`] require full alignment (in this case, `Simd<T, N>`'s alignment)
+/// - the likely source for reading or destination for writing `Simd<T, N>` is [`[T]`](slice) and similar types, aligned to `T`
+/// - combining these actions would violate the `unsafe` contract and explode the program into a puff of **undefined behavior**
+/// - the compiler can implicitly adjust layouts to make unaligned reads or writes fully aligned if it sees the optimization
+/// - most contemporary processors suffer no performance penalty for "unaligned" reads and writes that are aligned at runtime
+///
+/// By imposing less obligations, unaligned functions are less likely to make the program unsound,
+/// and may be just as fast as stricter alternatives.
+/// When trying to guarantee alignment, [`[T]::as_simd`][as_simd] is an option for converting `[T]` to `[Simd<T, N>]`,
+/// and allows soundly operating on an aligned SIMD body, but it may cost more time when handling the scalar head and tail.
+/// If these are not sufficient, then it is most ideal to design data structures to be already aligned
+/// to the `Simd<T, N>` you wish to use before using `unsafe` Rust to read or write.
+/// More conventional ways to compensate for these facts, like materializing `Simd` to or from an array first,
+/// are handled by safe methods like [`Simd::from_array`] and [`Simd::from_slice`].
+///
+/// [`transmute`]: core::mem::transmute
+/// [raw pointers]: pointer
+/// [`read_unaligned`]: pointer::read_unaligned
+/// [`write_unaligned`]: pointer::write_unaligned
+/// [`read`]: pointer::read
+/// [`write`]: pointer::write
+/// [as_simd]: slice::as_simd
+#[repr(simd)]
+pub struct Simd<T, const LANES: usize>([T; LANES])
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount;
+
+impl<T, const LANES: usize> Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ /// Number of lanes in this vector.
+ pub const LANES: usize = LANES;
+
+ /// Get the number of lanes in this vector.
+ pub const fn lanes(&self) -> usize {
+ LANES
+ }
+
+ /// Construct a SIMD vector by setting all lanes to the given value.
+ pub const fn splat(value: T) -> Self {
+ Self([value; LANES])
+ }
+
+ /// Returns an array reference containing the entire SIMD vector.
+ pub const fn as_array(&self) -> &[T; LANES] {
+ &self.0
+ }
+
+ /// Returns a mutable array reference containing the entire SIMD vector.
+ pub fn as_mut_array(&mut self) -> &mut [T; LANES] {
+ &mut self.0
+ }
+
+ /// Converts an array to a SIMD vector.
+ pub const fn from_array(array: [T; LANES]) -> Self {
+ Self(array)
+ }
+
+ /// Converts a SIMD vector to an array.
+ pub const fn to_array(self) -> [T; LANES] {
+ self.0
+ }
+
+ /// Converts a slice to a SIMD vector containing `slice[..LANES]`
+ /// # Panics
+ /// `from_slice` will panic if the slice's `len` is less than the vector's `Simd::LANES`.
+ #[must_use]
+ pub const fn from_slice(slice: &[T]) -> Self {
+ assert!(slice.len() >= LANES, "slice length must be at least the number of lanes");
+ let mut array = [slice[0]; LANES];
+ let mut i = 0;
+ while i < LANES {
+ array[i] = slice[i];
+ i += 1;
+ }
+ Self(array)
+ }
+
+ /// Performs lanewise conversion of a SIMD vector's elements to another SIMD-valid type.
+ /// This follows the semantics of Rust's `as` conversion for casting
+ /// integers to unsigned integers (interpreting as the other type, so `-1` to `MAX`),
+ /// and from floats to integers (truncating, or saturating at the limits) for each lane,
+ /// or vice versa.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ /// let floats: Simd<f32, 4> = Simd::from_array([1.9, -4.5, f32::INFINITY, f32::NAN]);
+ /// let ints = floats.cast::<i32>();
+ /// assert_eq!(ints, Simd::from_array([1, -4, i32::MAX, 0]));
+ ///
+ /// // Formally equivalent, but `Simd::cast` can optimize better.
+ /// assert_eq!(ints, Simd::from_array(floats.to_array().map(|x| x as i32)));
+ ///
+ /// // The float conversion does not round-trip.
+ /// let floats_again = ints.cast();
+ /// assert_ne!(floats, floats_again);
+ /// assert_eq!(floats_again, Simd::from_array([1.0, -4.0, 2147483647.0, 0.0]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn cast<U: SimdElement>(self) -> Simd<U, LANES> {
+ // Safety: The input argument is a vector of a known SIMD type.
+ unsafe { intrinsics::simd_as(self) }
+ }
+
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// If an index is out-of-bounds, the lane is instead selected from the `or` vector.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::Simd;
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ /// let alt = Simd::from_array([-5, -4, -3, -2]);
+ ///
+ /// let result = Simd::gather_or(&vec, idxs, alt); // Note the lane that is out-of-bounds.
+ /// assert_eq!(result, Simd::from_array([-5, 13, 10, 15]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn gather_or(slice: &[T], idxs: Simd<usize, LANES>, or: Self) -> Self {
+ Self::gather_select(slice, Mask::splat(true), idxs, or)
+ }
+
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// If an index is out-of-bounds, the lane is set to the default value for the type.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::{Simd, Mask};
- /// # #[cfg(not(feature = "std"))] use core::simd::{Simd, Mask};
++ /// # use core::simd::Simd;
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ ///
+ /// let result = Simd::gather_or_default(&vec, idxs); // Note the lane that is out-of-bounds.
+ /// assert_eq!(result, Simd::from_array([0, 13, 10, 15]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn gather_or_default(slice: &[T], idxs: Simd<usize, LANES>) -> Self
+ where
+ T: Default,
+ {
+ Self::gather_or(slice, idxs, Self::splat(T::default()))
+ }
+
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If an index is disabled or is out-of-bounds, the lane is selected from the `or` vector.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::{Simd, Mask};
- /// # #[cfg(not(feature = "std"))] use core::simd::{Simd, Mask};
++ /// # use core::simd::{Simd, Mask};
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ /// let alt = Simd::from_array([-5, -4, -3, -2]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the mask of the last lane.
+ ///
+ /// let result = Simd::gather_select(&vec, enable, idxs, alt); // Note the lane that is out-of-bounds.
+ /// assert_eq!(result, Simd::from_array([-5, 13, 10, -2]));
+ /// ```
+ #[must_use]
+ #[inline]
+ pub fn gather_select(
+ slice: &[T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ or: Self,
+ ) -> Self {
+ let enable: Mask<isize, LANES> = enable & idxs.lanes_lt(Simd::splat(slice.len()));
+ // Safety: We have masked-off out-of-bounds lanes.
+ unsafe { Self::gather_select_unchecked(slice, enable, idxs, or) }
+ }
+
+ /// Reads from potentially discontiguous indices in `slice` to construct a SIMD vector.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If an index is disabled, the lane is selected from the `or` vector.
+ ///
+ /// # Safety
+ ///
+ /// Calling this function with an `enable`d out-of-bounds index is *[undefined behavior]*
+ /// even if the resulting value is not used.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::Simd;
- /// # #[cfg(not(feature = "std"))] use core::simd::Simd;
++ /// # use core::simd::{Simd, Mask};
+ /// let vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 5]);
+ /// let alt = Simd::from_array([-5, -4, -3, -2]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the final mask lane.
+ /// // If this mask was used to gather, it would be unsound. Let's fix that.
+ /// let enable = enable & idxs.lanes_lt(Simd::splat(vec.len()));
+ ///
+ /// // We have masked the OOB lane, so it's safe to gather now.
+ /// let result = unsafe { Simd::gather_select_unchecked(&vec, enable, idxs, alt) };
+ /// assert_eq!(result, Simd::from_array([-5, 13, 10, -2]));
+ /// ```
+ /// [undefined behavior]: https://doc.rust-lang.org/reference/behavior-considered-undefined.html
+ #[must_use]
+ #[inline]
+ pub unsafe fn gather_select_unchecked(
+ slice: &[T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ or: Self,
+ ) -> Self {
+ let base_ptr = crate::simd::ptr::SimdConstPtr::splat(slice.as_ptr());
+ // Ferris forgive me, I have done pointer arithmetic here.
+ let ptrs = base_ptr.wrapping_add(idxs);
+ // Safety: The ptrs have been bounds-masked to prevent memory-unsafe reads insha'allah
+ unsafe { intrinsics::simd_gather(or, ptrs, enable.to_int()) }
+ }
+
+ /// Writes the values in a SIMD vector to potentially discontiguous indices in `slice`.
+ /// If two lanes in the scattered vector would write to the same index
+ /// only the last lane is guaranteed to actually be written.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::{Simd, Mask};
- /// # #[cfg(not(feature = "std"))] use core::simd::{Simd, Mask};
++ /// # use core::simd::Simd;
+ /// let mut vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 0]);
+ /// let vals = Simd::from_array([-27, 82, -41, 124]);
+ ///
+ /// vals.scatter(&mut vec, idxs); // index 0 receives two writes.
+ /// assert_eq!(vec, vec![124, 11, 12, 82, 14, 15, 16, 17, 18]);
+ /// ```
+ #[inline]
+ pub fn scatter(self, slice: &mut [T], idxs: Simd<usize, LANES>) {
+ self.scatter_select(slice, Mask::splat(true), idxs)
+ }
+
+ /// Writes the values in a SIMD vector to multiple potentially discontiguous indices in `slice`.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If an enabled index is out-of-bounds, the lane is not written.
+ /// If two enabled lanes in the scattered vector would write to the same index,
+ /// only the last lane is guaranteed to actually be written.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
- /// # #[cfg(feature = "std")] use core_simd::{Simd, Mask};
- /// # #[cfg(not(feature = "std"))] use core::simd::{Simd, Mask};
++ /// # use core::simd::{Simd, Mask};
+ /// let mut vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 0]);
+ /// let vals = Simd::from_array([-27, 82, -41, 124]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the mask of the last lane.
+ ///
+ /// vals.scatter_select(&mut vec, enable, idxs); // index 0's second write is masked, thus omitted.
+ /// assert_eq!(vec, vec![-41, 11, 12, 82, 14, 15, 16, 17, 18]);
+ /// ```
+ #[inline]
+ pub fn scatter_select(
+ self,
+ slice: &mut [T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ ) {
+ let enable: Mask<isize, LANES> = enable & idxs.lanes_lt(Simd::splat(slice.len()));
+ // Safety: We have masked-off out-of-bounds lanes.
+ unsafe { self.scatter_select_unchecked(slice, enable, idxs) }
+ }
+
+ /// Writes the values in a SIMD vector to multiple potentially discontiguous indices in `slice`.
+ /// The mask `enable`s all `true` lanes and disables all `false` lanes.
+ /// If two enabled lanes in the scattered vector would write to the same index,
+ /// only the last lane is guaranteed to actually be written.
+ ///
+ /// # Safety
+ ///
+ /// Calling this function with an enabled out-of-bounds index is *[undefined behavior]*,
+ /// and may lead to memory corruption.
+ ///
+ /// # Examples
+ /// ```
+ /// # #![feature(portable_simd)]
++ /// # use core::simd::{Simd, Mask};
+ /// let mut vec: Vec<i32> = vec![10, 11, 12, 13, 14, 15, 16, 17, 18];
+ /// let idxs = Simd::from_array([9, 3, 0, 0]);
+ /// let vals = Simd::from_array([-27, 82, -41, 124]);
+ /// let enable = Mask::from_array([true, true, true, false]); // Note the mask of the last lane.
+ /// // If this mask was used to scatter, it would be unsound. Let's fix that.
+ /// let enable = enable & idxs.lanes_lt(Simd::splat(vec.len()));
+ ///
+ /// // We have masked the OOB lane, so it's safe to scatter now.
+ /// unsafe { vals.scatter_select_unchecked(&mut vec, enable, idxs); }
+ /// // index 0's second write is masked, thus was omitted.
+ /// assert_eq!(vec, vec![-41, 11, 12, 82, 14, 15, 16, 17, 18]);
+ /// ```
+ /// [undefined behavior]: https://doc.rust-lang.org/reference/behavior-considered-undefined.html
+ #[inline]
+ pub unsafe fn scatter_select_unchecked(
+ self,
+ slice: &mut [T],
+ enable: Mask<isize, LANES>,
+ idxs: Simd<usize, LANES>,
+ ) {
+ // Safety: This block works with *mut T derived from &mut 'a [T],
+ // which means it is delicate in Rust's borrowing model, circa 2021:
+ // &mut 'a [T] asserts uniqueness, so deriving &'a [T] invalidates live *mut Ts!
+ // Even though this block is largely safe methods, it must be exactly this way
+ // to prevent invalidating the raw ptrs while they're live.
+ // Thus, entering this block requires all values to use being already ready:
+ // 0. idxs we want to write to, which are used to construct the mask.
+ // 1. enable, which depends on an initial &'a [T] and the idxs.
+ // 2. actual values to scatter (self).
+ // 3. &mut [T] which will become our base ptr.
+ unsafe {
+ // Now Entering ☢️ *mut T Zone
+ let base_ptr = crate::simd::ptr::SimdMutPtr::splat(slice.as_mut_ptr());
+ // Ferris forgive me, I have done pointer arithmetic here.
+ let ptrs = base_ptr.wrapping_add(idxs);
+ // The ptrs have been bounds-masked to prevent memory-unsafe writes insha'allah
+ intrinsics::simd_scatter(self, ptrs, enable.to_int())
+ // Cleared ☢️ *mut T Zone
+ }
+ }
+}
+
+impl<T, const LANES: usize> Copy for Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+}
+
+impl<T, const LANES: usize> Clone for Simd<T, LANES>
+where
+ T: SimdElement,
+ LaneCount<LANES>: SupportedLaneCount,
+{
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<T, const LANES: usize> Default for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + Default,
+{
+ #[inline]
+ fn default() -> Self {
+ Self::splat(T::default())
+ }
+}
+
+impl<T, const LANES: usize> PartialEq for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + PartialEq,
+{
+ #[inline]
+ fn eq(&self, other: &Self) -> bool {
+ // TODO use SIMD equality
+ self.to_array() == other.to_array()
+ }
+}
+
+impl<T, const LANES: usize> PartialOrd for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + PartialOrd,
+{
+ #[inline]
+ fn partial_cmp(&self, other: &Self) -> Option<core::cmp::Ordering> {
+ // TODO use SIMD equality
+ self.to_array().partial_cmp(other.as_ref())
+ }
+}
+
+impl<T, const LANES: usize> Eq for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + Eq,
+{
+}
+
+impl<T, const LANES: usize> Ord for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + Ord,
+{
+ #[inline]
+ fn cmp(&self, other: &Self) -> core::cmp::Ordering {
+ // TODO use SIMD equality
+ self.to_array().cmp(other.as_ref())
+ }
+}
+
+impl<T, const LANES: usize> core::hash::Hash for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement + core::hash::Hash,
+{
+ #[inline]
+ fn hash<H>(&self, state: &mut H)
+ where
+ H: core::hash::Hasher,
+ {
+ self.as_array().hash(state)
+ }
+}
+
+// array references
+impl<T, const LANES: usize> AsRef<[T; LANES]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_ref(&self) -> &[T; LANES] {
+ &self.0
+ }
+}
+
+impl<T, const LANES: usize> AsMut<[T; LANES]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_mut(&mut self) -> &mut [T; LANES] {
+ &mut self.0
+ }
+}
+
+// slice references
+impl<T, const LANES: usize> AsRef<[T]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_ref(&self) -> &[T] {
+ &self.0
+ }
+}
+
+impl<T, const LANES: usize> AsMut<[T]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ #[inline]
+ fn as_mut(&mut self) -> &mut [T] {
+ &mut self.0
+ }
+}
+
+// vector/array conversion
+impl<T, const LANES: usize> From<[T; LANES]> for Simd<T, LANES>
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ fn from(array: [T; LANES]) -> Self {
+ Self(array)
+ }
+}
+
+impl<T, const LANES: usize> From<Simd<T, LANES>> for [T; LANES]
+where
+ LaneCount<LANES>: SupportedLaneCount,
+ T: SimdElement,
+{
+ fn from(vector: Simd<T, LANES>) -> Self {
+ vector.to_array()
+ }
+}
+
+mod sealed {
+ pub trait Sealed {}
+}
+use sealed::Sealed;
+
+/// Marker trait for types that may be used as SIMD vector elements.
+///
+/// # Safety
+/// This trait, when implemented, asserts the compiler can monomorphize
+/// `#[repr(simd)]` structs with the marked type as an element.
+/// Strictly, it is valid to impl if the vector will not be miscompiled.
+/// Practically, it is user-unfriendly to impl it if the vector won't compile,
+/// even when no soundness guarantees are broken by allowing the user to try.
+pub unsafe trait SimdElement: Sealed + Copy {
+ /// The mask element type corresponding to this element type.
+ type Mask: MaskElement;
+}
+
+impl Sealed for u8 {}
+unsafe impl SimdElement for u8 {
+ type Mask = i8;
+}
+
+impl Sealed for u16 {}
+unsafe impl SimdElement for u16 {
+ type Mask = i16;
+}
+
+impl Sealed for u32 {}
+unsafe impl SimdElement for u32 {
+ type Mask = i32;
+}
+
+impl Sealed for u64 {}
+unsafe impl SimdElement for u64 {
+ type Mask = i64;
+}
+
+impl Sealed for usize {}
+unsafe impl SimdElement for usize {
+ type Mask = isize;
+}
+
+impl Sealed for i8 {}
+unsafe impl SimdElement for i8 {
+ type Mask = i8;
+}
+
+impl Sealed for i16 {}
+unsafe impl SimdElement for i16 {
+ type Mask = i16;
+}
+
+impl Sealed for i32 {}
+unsafe impl SimdElement for i32 {
+ type Mask = i32;
+}
+
+impl Sealed for i64 {}
+unsafe impl SimdElement for i64 {
+ type Mask = i64;
+}
+
+impl Sealed for isize {}
+unsafe impl SimdElement for isize {
+ type Mask = isize;
+}
+
+impl Sealed for f32 {}
+unsafe impl SimdElement for f32 {
+ type Mask = i32;
+}
+
+impl Sealed for f64 {}
+unsafe impl SimdElement for f64 {
+ type Mask = i64;
+}
--- /dev/null
+#![feature(portable_simd)]
++use core_simd::i16x2;
+
+#[macro_use]
+mod ops_macros;
+impl_signed_tests! { i16 }
++
++#[test]
++fn max_is_not_lexicographic() {
++ let a = i16x2::splat(10);
++ let b = i16x2::from_array([-4, 12]);
++ assert_eq!(a.max(b), i16x2::from_array([10, 12]));
++}
++
++#[test]
++fn min_is_not_lexicographic() {
++ let a = i16x2::splat(10);
++ let b = i16x2::from_array([12, -4]);
++ assert_eq!(a.min(b), i16x2::from_array([10, -4]));
++}
++
++#[test]
++fn clamp_is_not_lexicographic() {
++ let a = i16x2::splat(10);
++ let lo = i16x2::from_array([-12, -4]);
++ let up = i16x2::from_array([-4, 12]);
++ assert_eq!(a.clamp(lo, up), i16x2::from_array([-4, 10]));
++
++ let x = i16x2::from_array([1, 10]);
++ let y = x.clamp(i16x2::splat(0), i16x2::splat(9));
++ assert_eq!(y, i16x2::from_array([1, 9]));
++}
--- /dev/null
- fn horizontal_sum<const LANES: usize>() {
+/// Implements a test on a unary operation using proptest.
+///
+/// Compares the vector operation to the equivalent scalar operation.
+#[macro_export]
+macro_rules! impl_unary_op_test {
+ { $scalar:ty, $trait:ident :: $fn:ident, $scalar_fn:expr } => {
+ test_helpers::test_lanes! {
+ fn $fn<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &<core_simd::Simd<$scalar, LANES> as core::ops::$trait>::$fn,
+ &$scalar_fn,
+ &|_| true,
+ );
+ }
+ }
+ };
+ { $scalar:ty, $trait:ident :: $fn:ident } => {
+ impl_unary_op_test! { $scalar, $trait::$fn, <$scalar as core::ops::$trait>::$fn }
+ };
+}
+
+/// Implements a test on a binary operation using proptest.
+///
+/// Compares the vector operation to the equivalent scalar operation.
+#[macro_export]
+macro_rules! impl_binary_op_test {
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident, $scalar_fn:expr } => {
+ mod $fn {
+ use super::*;
+ use core_simd::Simd;
+
+ test_helpers::test_lanes! {
+ fn normal<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &<Simd<$scalar, LANES> as core::ops::$trait>::$fn,
+ &$scalar_fn,
+ &|_, _| true,
+ );
+ }
+
+ fn assign<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &|mut a, b| { <Simd<$scalar, LANES> as core::ops::$trait_assign>::$fn_assign(&mut a, b); a },
+ &$scalar_fn,
+ &|_, _| true,
+ );
+ }
+ }
+ }
+ };
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident } => {
+ impl_binary_op_test! { $scalar, $trait::$fn, $trait_assign::$fn_assign, <$scalar as core::ops::$trait>::$fn }
+ };
+}
+
+/// Implements a test on a binary operation using proptest.
+///
+/// Like `impl_binary_op_test`, but allows providing a function for rejecting particular inputs
+/// (like the `proptest_assume` macro).
+///
+/// Compares the vector operation to the equivalent scalar operation.
+#[macro_export]
+macro_rules! impl_binary_checked_op_test {
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident, $scalar_fn:expr, $check_fn:expr } => {
+ mod $fn {
+ use super::*;
+ use core_simd::Simd;
+
+ test_helpers::test_lanes! {
+ fn normal<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &<Simd<$scalar, LANES> as core::ops::$trait>::$fn,
+ &$scalar_fn,
+ &|x, y| x.iter().zip(y.iter()).all(|(x, y)| $check_fn(*x, *y)),
+ );
+ }
+
+ fn assign<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &|mut a, b| { <Simd<$scalar, LANES> as core::ops::$trait_assign>::$fn_assign(&mut a, b); a },
+ &$scalar_fn,
+ &|x, y| x.iter().zip(y.iter()).all(|(x, y)| $check_fn(*x, *y)),
+ )
+ }
+ }
+ }
+ };
+ { $scalar:ty, $trait:ident :: $fn:ident, $trait_assign:ident :: $fn_assign:ident, $check_fn:expr } => {
+ impl_binary_checked_op_test! { $scalar, $trait::$fn, $trait_assign::$fn_assign, <$scalar as core::ops::$trait>::$fn, $check_fn }
+ };
+}
+
+#[macro_export]
+macro_rules! impl_common_integer_tests {
+ { $vector:ident, $scalar:ident } => {
+ test_helpers::test_lanes! {
- $vector::<LANES>::from_array(x).horizontal_sum(),
++ fn reduce_sum<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_product<const LANES: usize>() {
++ $vector::<LANES>::from_array(x).reduce_sum(),
+ x.iter().copied().fold(0 as $scalar, $scalar::wrapping_add),
+ );
+ Ok(())
+ });
+ }
+
- $vector::<LANES>::from_array(x).horizontal_product(),
++ fn reduce_product<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_and<const LANES: usize>() {
++ $vector::<LANES>::from_array(x).reduce_product(),
+ x.iter().copied().fold(1 as $scalar, $scalar::wrapping_mul),
+ );
+ Ok(())
+ });
+ }
+
- $vector::<LANES>::from_array(x).horizontal_and(),
++ fn reduce_and<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_or<const LANES: usize>() {
++ $vector::<LANES>::from_array(x).reduce_and(),
+ x.iter().copied().fold(-1i8 as $scalar, <$scalar as core::ops::BitAnd>::bitand),
+ );
+ Ok(())
+ });
+ }
+
- $vector::<LANES>::from_array(x).horizontal_or(),
++ fn reduce_or<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_xor<const LANES: usize>() {
++ $vector::<LANES>::from_array(x).reduce_or(),
+ x.iter().copied().fold(0 as $scalar, <$scalar as core::ops::BitOr>::bitor),
+ );
+ Ok(())
+ });
+ }
+
- $vector::<LANES>::from_array(x).horizontal_xor(),
++ fn reduce_xor<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_max<const LANES: usize>() {
++ $vector::<LANES>::from_array(x).reduce_xor(),
+ x.iter().copied().fold(0 as $scalar, <$scalar as core::ops::BitXor>::bitxor),
+ );
+ Ok(())
+ });
+ }
+
- $vector::<LANES>::from_array(x).horizontal_max(),
++ fn reduce_max<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_min<const LANES: usize>() {
++ $vector::<LANES>::from_array(x).reduce_max(),
+ x.iter().copied().max().unwrap(),
+ );
+ Ok(())
+ });
+ }
+
- $vector::<LANES>::from_array(x).horizontal_min(),
++ fn reduce_min<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_sum<const LANES: usize>() {
++ $vector::<LANES>::from_array(x).reduce_min(),
+ x.iter().copied().min().unwrap(),
+ );
+ Ok(())
+ });
+ }
+ }
+ }
+}
+
+/// Implement tests for signed integers.
+#[macro_export]
+macro_rules! impl_signed_tests {
+ { $scalar:tt } => {
+ mod $scalar {
+ type Vector<const LANES: usize> = core_simd::Simd<Scalar, LANES>;
+ type Scalar = $scalar;
+
+ impl_common_integer_tests! { Vector, Scalar }
+
+ test_helpers::test_lanes! {
+ fn neg<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &<Vector::<LANES> as core::ops::Neg>::neg,
+ &<Scalar as core::ops::Neg>::neg,
+ &|x| !x.contains(&Scalar::MIN),
+ );
+ }
+
+ fn is_positive<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_positive,
+ &Scalar::is_positive,
+ &|_| true,
+ );
+ }
+
+ fn is_negative<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_negative,
+ &Scalar::is_negative,
+ &|_| true,
+ );
+ }
+
+ fn signum<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::signum,
+ &Scalar::signum,
+ &|_| true,
+ )
+ }
+
+ fn div_min_may_overflow<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(Scalar::MIN);
+ let b = Vector::<LANES>::splat(-1);
+ assert_eq!(a / b, a);
+ }
+
+ fn rem_min_may_overflow<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(Scalar::MIN);
+ let b = Vector::<LANES>::splat(-1);
+ assert_eq!(a % b, Vector::<LANES>::splat(0));
+ }
+
++ fn min<const LANES: usize>() {
++ let a = Vector::<LANES>::splat(Scalar::MIN);
++ let b = Vector::<LANES>::splat(0);
++ assert_eq!(a.min(b), a);
++ let a = Vector::<LANES>::splat(Scalar::MAX);
++ let b = Vector::<LANES>::splat(0);
++ assert_eq!(a.min(b), b);
++ }
++
++ fn max<const LANES: usize>() {
++ let a = Vector::<LANES>::splat(Scalar::MIN);
++ let b = Vector::<LANES>::splat(0);
++ assert_eq!(a.max(b), b);
++ let a = Vector::<LANES>::splat(Scalar::MAX);
++ let b = Vector::<LANES>::splat(0);
++ assert_eq!(a.max(b), a);
++ }
++
++ fn clamp<const LANES: usize>() {
++ let min = Vector::<LANES>::splat(Scalar::MIN);
++ let max = Vector::<LANES>::splat(Scalar::MAX);
++ let zero = Vector::<LANES>::splat(0);
++ let one = Vector::<LANES>::splat(1);
++ let negone = Vector::<LANES>::splat(-1);
++ assert_eq!(zero.clamp(min, max), zero);
++ assert_eq!(zero.clamp(min, one), zero);
++ assert_eq!(zero.clamp(one, max), one);
++ assert_eq!(zero.clamp(min, negone), negone);
++ }
+ }
+
+ test_helpers::test_lanes_panic! {
+ fn div_by_all_zeros_panics<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(0);
+ let _ = a / b;
+ }
+
+ fn div_by_one_zero_panics<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let mut b = Vector::<LANES>::splat(21);
+ b[0] = 0 as _;
+ let _ = a / b;
+ }
+
+ fn rem_zero_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(0);
+ let _ = a % b;
+ }
+ }
+
+ test_helpers::test_lanes! {
+ fn div_neg_one_no_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(-1);
+ let _ = a / b;
+ }
+
+ fn rem_neg_one_no_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(-1);
+ let _ = a % b;
+ }
+ }
+
+ impl_binary_op_test!(Scalar, Add::add, AddAssign::add_assign, Scalar::wrapping_add);
+ impl_binary_op_test!(Scalar, Sub::sub, SubAssign::sub_assign, Scalar::wrapping_sub);
+ impl_binary_op_test!(Scalar, Mul::mul, MulAssign::mul_assign, Scalar::wrapping_mul);
+
+ // Exclude Div and Rem panicking cases
+ impl_binary_checked_op_test!(Scalar, Div::div, DivAssign::div_assign, Scalar::wrapping_div, |x, y| y != 0 && !(x == Scalar::MIN && y == -1));
+ impl_binary_checked_op_test!(Scalar, Rem::rem, RemAssign::rem_assign, Scalar::wrapping_rem, |x, y| y != 0 && !(x == Scalar::MIN && y == -1));
+
+ impl_unary_op_test!(Scalar, Not::not);
+ impl_binary_op_test!(Scalar, BitAnd::bitand, BitAndAssign::bitand_assign);
+ impl_binary_op_test!(Scalar, BitOr::bitor, BitOrAssign::bitor_assign);
+ impl_binary_op_test!(Scalar, BitXor::bitxor, BitXorAssign::bitxor_assign);
+ }
+ }
+}
+
+/// Implement tests for unsigned integers.
+#[macro_export]
+macro_rules! impl_unsigned_tests {
+ { $scalar:tt } => {
+ mod $scalar {
+ type Vector<const LANES: usize> = core_simd::Simd<Scalar, LANES>;
+ type Scalar = $scalar;
+
+ impl_common_integer_tests! { Vector, Scalar }
+
+ test_helpers::test_lanes_panic! {
+ fn rem_zero_panic<const LANES: usize>() {
+ let a = Vector::<LANES>::splat(42);
+ let b = Vector::<LANES>::splat(0);
+ let _ = a % b;
+ }
+ }
+
+ impl_binary_op_test!(Scalar, Add::add, AddAssign::add_assign, Scalar::wrapping_add);
+ impl_binary_op_test!(Scalar, Sub::sub, SubAssign::sub_assign, Scalar::wrapping_sub);
+ impl_binary_op_test!(Scalar, Mul::mul, MulAssign::mul_assign, Scalar::wrapping_mul);
+
+ // Exclude Div and Rem panicking cases
+ impl_binary_checked_op_test!(Scalar, Div::div, DivAssign::div_assign, Scalar::wrapping_div, |_, y| y != 0);
+ impl_binary_checked_op_test!(Scalar, Rem::rem, RemAssign::rem_assign, Scalar::wrapping_rem, |_, y| y != 0);
+
+ impl_unary_op_test!(Scalar, Not::not);
+ impl_binary_op_test!(Scalar, BitAnd::bitand, BitAndAssign::bitand_assign);
+ impl_binary_op_test!(Scalar, BitOr::bitor, BitOrAssign::bitor_assign);
+ impl_binary_op_test!(Scalar, BitXor::bitxor, BitXorAssign::bitxor_assign);
+ }
+ }
+}
+
+/// Implement tests for floating point numbers.
+#[macro_export]
+macro_rules! impl_float_tests {
+ { $scalar:tt, $int_scalar:tt } => {
+ mod $scalar {
+ type Vector<const LANES: usize> = core_simd::Simd<Scalar, LANES>;
+ type Scalar = $scalar;
+
+ impl_unary_op_test!(Scalar, Neg::neg);
+ impl_binary_op_test!(Scalar, Add::add, AddAssign::add_assign);
+ impl_binary_op_test!(Scalar, Sub::sub, SubAssign::sub_assign);
+ impl_binary_op_test!(Scalar, Mul::mul, MulAssign::mul_assign);
+ impl_binary_op_test!(Scalar, Div::div, DivAssign::div_assign);
+ impl_binary_op_test!(Scalar, Rem::rem, RemAssign::rem_assign);
+
+ test_helpers::test_lanes! {
+ fn is_sign_positive<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_sign_positive,
+ &Scalar::is_sign_positive,
+ &|_| true,
+ );
+ }
+
+ fn is_sign_negative<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_sign_negative,
+ &Scalar::is_sign_negative,
+ &|_| true,
+ );
+ }
+
+ fn is_finite<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_finite,
+ &Scalar::is_finite,
+ &|_| true,
+ );
+ }
+
+ fn is_infinite<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_infinite,
+ &Scalar::is_infinite,
+ &|_| true,
+ );
+ }
+
+ fn is_nan<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_nan,
+ &Scalar::is_nan,
+ &|_| true,
+ );
+ }
+
+ fn is_normal<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_normal,
+ &Scalar::is_normal,
+ &|_| true,
+ );
+ }
+
+ fn is_subnormal<const LANES: usize>() {
+ test_helpers::test_unary_mask_elementwise(
+ &Vector::<LANES>::is_subnormal,
+ &Scalar::is_subnormal,
+ &|_| true,
+ );
+ }
+
+ fn abs<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::abs,
+ &Scalar::abs,
+ &|_| true,
+ )
+ }
+
+ fn recip<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::recip,
+ &Scalar::recip,
+ &|_| true,
+ )
+ }
+
+ fn to_degrees<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::to_degrees,
+ &Scalar::to_degrees,
+ &|_| true,
+ )
+ }
+
+ fn to_radians<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::to_radians,
+ &Scalar::to_radians,
+ &|_| true,
+ )
+ }
+
+ fn signum<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::signum,
+ &Scalar::signum,
+ &|_| true,
+ )
+ }
+
+ fn copysign<const LANES: usize>() {
+ test_helpers::test_binary_elementwise(
+ &Vector::<LANES>::copysign,
+ &Scalar::copysign,
+ &|_, _| true,
+ )
+ }
+
+ fn min<const LANES: usize>() {
+ // Regular conditions (both values aren't zero)
+ test_helpers::test_binary_elementwise(
+ &Vector::<LANES>::min,
+ &Scalar::min,
+ // Reject the case where both values are zero with different signs
+ &|a, b| {
+ for (a, b) in a.iter().zip(b.iter()) {
+ if *a == 0. && *b == 0. && a.signum() != b.signum() {
+ return false;
+ }
+ }
+ true
+ }
+ );
+
+ // Special case where both values are zero
+ let p_zero = Vector::<LANES>::splat(0.);
+ let n_zero = Vector::<LANES>::splat(-0.);
+ assert!(p_zero.min(n_zero).to_array().iter().all(|x| *x == 0.));
+ assert!(n_zero.min(p_zero).to_array().iter().all(|x| *x == 0.));
+ }
+
+ fn max<const LANES: usize>() {
+ // Regular conditions (both values aren't zero)
+ test_helpers::test_binary_elementwise(
+ &Vector::<LANES>::max,
+ &Scalar::max,
+ // Reject the case where both values are zero with different signs
+ &|a, b| {
+ for (a, b) in a.iter().zip(b.iter()) {
+ if *a == 0. && *b == 0. && a.signum() != b.signum() {
+ return false;
+ }
+ }
+ true
+ }
+ );
+
+ // Special case where both values are zero
+ let p_zero = Vector::<LANES>::splat(0.);
+ let n_zero = Vector::<LANES>::splat(-0.);
+ assert!(p_zero.max(n_zero).to_array().iter().all(|x| *x == 0.));
+ assert!(n_zero.max(p_zero).to_array().iter().all(|x| *x == 0.));
+ }
+
+ fn clamp<const LANES: usize>() {
+ test_helpers::test_3(&|value: [Scalar; LANES], mut min: [Scalar; LANES], mut max: [Scalar; LANES]| {
+ for (min, max) in min.iter_mut().zip(max.iter_mut()) {
+ if max < min {
+ core::mem::swap(min, max);
+ }
+ if min.is_nan() {
+ *min = Scalar::NEG_INFINITY;
+ }
+ if max.is_nan() {
+ *max = Scalar::INFINITY;
+ }
+ }
+
+ let mut result_scalar = [Scalar::default(); LANES];
+ for i in 0..LANES {
+ result_scalar[i] = value[i].clamp(min[i], max[i]);
+ }
+ let result_vector = Vector::from_array(value).clamp(min.into(), max.into()).to_array();
+ test_helpers::prop_assert_biteq!(result_scalar, result_vector);
+ Ok(())
+ })
+ }
+
- Vector::<LANES>::from_array(x).horizontal_sum(),
++ fn reduce_sum<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_product<const LANES: usize>() {
++ Vector::<LANES>::from_array(x).reduce_sum(),
+ x.iter().sum(),
+ );
+ Ok(())
+ });
+ }
+
- Vector::<LANES>::from_array(x).horizontal_product(),
++ fn reduce_product<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
+ test_helpers::prop_assert_biteq! (
- fn horizontal_max<const LANES: usize>() {
++ Vector::<LANES>::from_array(x).reduce_product(),
+ x.iter().product(),
+ );
+ Ok(())
+ });
+ }
+
- let vmax = Vector::<LANES>::from_array(x).horizontal_max();
++ fn reduce_max<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
- fn horizontal_min<const LANES: usize>() {
++ let vmax = Vector::<LANES>::from_array(x).reduce_max();
+ let smax = x.iter().copied().fold(Scalar::NAN, Scalar::max);
+ // 0 and -0 are treated the same
+ if !(x.contains(&0.) && x.contains(&-0.) && vmax.abs() == 0. && smax.abs() == 0.) {
+ test_helpers::prop_assert_biteq!(vmax, smax);
+ }
+ Ok(())
+ });
+ }
+
- let vmax = Vector::<LANES>::from_array(x).horizontal_min();
++ fn reduce_min<const LANES: usize>() {
+ test_helpers::test_1(&|x| {
++ let vmax = Vector::<LANES>::from_array(x).reduce_min();
+ let smax = x.iter().copied().fold(Scalar::NAN, Scalar::min);
+ // 0 and -0 are treated the same
+ if !(x.contains(&0.) && x.contains(&-0.) && vmax.abs() == 0. && smax.abs() == 0.) {
+ test_helpers::prop_assert_biteq!(vmax, smax);
+ }
+ Ok(())
+ });
+ }
+ }
+
+ #[cfg(feature = "std")]
+ mod std {
+ use std_float::StdFloat;
+
+ use super::*;
+ test_helpers::test_lanes! {
+ fn sqrt<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::sqrt,
+ &Scalar::sqrt,
+ &|_| true,
+ )
+ }
+
+ fn mul_add<const LANES: usize>() {
+ test_helpers::test_ternary_elementwise(
+ &Vector::<LANES>::mul_add,
+ &Scalar::mul_add,
+ &|_, _, _| true,
+ )
+ }
+ }
+ }
+ }
+ }
+}
--- /dev/null
- #[cfg(feature = "std")]
+#![feature(portable_simd)]
+
+macro_rules! float_rounding_test {
+ { $scalar:tt, $int_scalar:tt } => {
+ mod $scalar {
+ use std_float::StdFloat;
+
+ type Vector<const LANES: usize> = core_simd::Simd<$scalar, LANES>;
+ type Scalar = $scalar;
+ type IntScalar = $int_scalar;
+
+ test_helpers::test_lanes! {
+ fn ceil<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::ceil,
+ &Scalar::ceil,
+ &|_| true,
+ )
+ }
+
+ fn floor<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::floor,
+ &Scalar::floor,
+ &|_| true,
+ )
+ }
+
+ fn round<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::round,
+ &Scalar::round,
+ &|_| true,
+ )
+ }
+
+ fn trunc<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::trunc,
+ &Scalar::trunc,
+ &|_| true,
+ )
+ }
+
+ fn fract<const LANES: usize>() {
+ test_helpers::test_unary_elementwise(
+ &Vector::<LANES>::fract,
+ &Scalar::fract,
+ &|_| true,
+ )
+ }
+ }
+
+ test_helpers::test_lanes! {
+ fn to_int_unchecked<const LANES: usize>() {
+ // The maximum integer that can be represented by the equivalently sized float has
+ // all of the mantissa digits set to 1, pushed up to the MSB.
+ const ALL_MANTISSA_BITS: IntScalar = ((1 << <Scalar>::MANTISSA_DIGITS) - 1);
+ const MAX_REPRESENTABLE_VALUE: Scalar =
+ (ALL_MANTISSA_BITS << (core::mem::size_of::<Scalar>() * 8 - <Scalar>::MANTISSA_DIGITS as usize - 1)) as Scalar;
+
+ let mut runner = proptest::test_runner::TestRunner::default();
+ runner.run(
+ &test_helpers::array::UniformArrayStrategy::new(-MAX_REPRESENTABLE_VALUE..MAX_REPRESENTABLE_VALUE),
+ |x| {
+ let result_1 = unsafe { Vector::from_array(x).to_int_unchecked::<IntScalar>().to_array() };
+ let result_2 = {
+ let mut result: [IntScalar; LANES] = [0; LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = unsafe { i.to_int_unchecked::<IntScalar>() };
+ }
+ result
+ };
+ test_helpers::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ },
+ ).unwrap();
+ }
+ }
+ }
+ }
+}
+
+float_rounding_test! { f32, i32 }
+float_rounding_test! { f64, i64 }
--- /dev/null
- core_simd = { path = "../core_simd" }
+[package]
+name = "std_float"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
++core_simd = { path = "../core_simd", default-features = false }
+
+[features]
+default = ["as_crate"]
+as_crate = []
--- /dev/null
- let mut runner = proptest::test_runner::TestRunner::default();
+pub mod array;
+
+#[cfg(target_arch = "wasm32")]
+pub mod wasm;
+
+#[macro_use]
+pub mod biteq;
+
+/// Specifies the default strategy for testing a type.
+///
+/// This strategy should be what "makes sense" to test.
+pub trait DefaultStrategy {
+ type Strategy: proptest::strategy::Strategy<Value = Self>;
+ fn default_strategy() -> Self::Strategy;
+}
+
+macro_rules! impl_num {
+ { $type:tt } => {
+ impl DefaultStrategy for $type {
+ type Strategy = proptest::num::$type::Any;
+ fn default_strategy() -> Self::Strategy {
+ proptest::num::$type::ANY
+ }
+ }
+ }
+}
+
+impl_num! { i8 }
+impl_num! { i16 }
+impl_num! { i32 }
+impl_num! { i64 }
+impl_num! { isize }
+impl_num! { u8 }
+impl_num! { u16 }
+impl_num! { u32 }
+impl_num! { u64 }
+impl_num! { usize }
+impl_num! { f32 }
+impl_num! { f64 }
+
+#[cfg(not(target_arch = "wasm32"))]
+impl DefaultStrategy for u128 {
+ type Strategy = proptest::num::u128::Any;
+ fn default_strategy() -> Self::Strategy {
+ proptest::num::u128::ANY
+ }
+}
+
+#[cfg(not(target_arch = "wasm32"))]
+impl DefaultStrategy for i128 {
+ type Strategy = proptest::num::i128::Any;
+ fn default_strategy() -> Self::Strategy {
+ proptest::num::i128::ANY
+ }
+}
+
+#[cfg(target_arch = "wasm32")]
+impl DefaultStrategy for u128 {
+ type Strategy = crate::wasm::u128::Any;
+ fn default_strategy() -> Self::Strategy {
+ crate::wasm::u128::ANY
+ }
+}
+
+#[cfg(target_arch = "wasm32")]
+impl DefaultStrategy for i128 {
+ type Strategy = crate::wasm::i128::Any;
+ fn default_strategy() -> Self::Strategy {
+ crate::wasm::i128::ANY
+ }
+}
+
+impl<T: core::fmt::Debug + DefaultStrategy, const LANES: usize> DefaultStrategy for [T; LANES] {
+ type Strategy = crate::array::UniformArrayStrategy<T::Strategy, Self>;
+ fn default_strategy() -> Self::Strategy {
+ Self::Strategy::new(T::default_strategy())
+ }
+}
+
++#[cfg(not(miri))]
++fn make_runner() -> proptest::test_runner::TestRunner {
++ Default::default()
++}
++#[cfg(miri)]
++fn make_runner() -> proptest::test_runner::TestRunner {
++ // Only run a few tests on Miri
++ proptest::test_runner::TestRunner::new(proptest::test_runner::Config::with_cases(4))
++}
++
+/// Test a function that takes a single value.
+pub fn test_1<A: core::fmt::Debug + DefaultStrategy>(
+ f: &dyn Fn(A) -> proptest::test_runner::TestCaseResult,
+) {
- let mut runner = proptest::test_runner::TestRunner::default();
++ let mut runner = make_runner();
+ runner.run(&A::default_strategy(), f).unwrap();
+}
+
+/// Test a function that takes two values.
+pub fn test_2<A: core::fmt::Debug + DefaultStrategy, B: core::fmt::Debug + DefaultStrategy>(
+ f: &dyn Fn(A, B) -> proptest::test_runner::TestCaseResult,
+) {
- let mut runner = proptest::test_runner::TestRunner::default();
++ let mut runner = make_runner();
+ runner
+ .run(&(A::default_strategy(), B::default_strategy()), |(a, b)| {
+ f(a, b)
+ })
+ .unwrap();
+}
+
+/// Test a function that takes two values.
+pub fn test_3<
+ A: core::fmt::Debug + DefaultStrategy,
+ B: core::fmt::Debug + DefaultStrategy,
+ C: core::fmt::Debug + DefaultStrategy,
+>(
+ f: &dyn Fn(A, B, C) -> proptest::test_runner::TestCaseResult,
+) {
++ let mut runner = make_runner();
+ runner
+ .run(
+ &(
+ A::default_strategy(),
+ B::default_strategy(),
+ C::default_strategy(),
+ ),
+ |(a, b, c)| f(a, b, c),
+ )
+ .unwrap();
+}
+
+/// Test a unary vector function against a unary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_unary_elementwise<Scalar, ScalarResult, Vector, VectorResult, const LANES: usize>(
+ fv: &dyn Fn(Vector) -> VectorResult,
+ fs: &dyn Fn(Scalar) -> ScalarResult,
+ check: &dyn Fn([Scalar; LANES]) -> bool,
+) where
+ Scalar: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar; LANES]> + From<[Scalar; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_1(&|x: [Scalar; LANES]| {
+ proptest::prop_assume!(check(x));
+ let result_1: [ScalarResult; LANES] = fv(x.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = fs(*i);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a unary vector function against a unary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_unary_mask_elementwise<Scalar, Vector, Mask, const LANES: usize>(
+ fv: &dyn Fn(Vector) -> Mask,
+ fs: &dyn Fn(Scalar) -> bool,
+ check: &dyn Fn([Scalar; LANES]) -> bool,
+) where
+ Scalar: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar; LANES]> + From<[Scalar; LANES]> + Copy,
+ Mask: Into<[bool; LANES]> + From<[bool; LANES]> + Copy,
+{
+ test_1(&|x: [Scalar; LANES]| {
+ proptest::prop_assume!(check(x));
+ let result_1: [bool; LANES] = fv(x.into()).into();
+ let result_2: [bool; LANES] = {
+ let mut result = [false; LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = fs(*i);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a binary vector function against a binary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_binary_elementwise<
+ Scalar1,
+ Scalar2,
+ ScalarResult,
+ Vector1,
+ Vector2,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Vector1, Vector2) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2) -> ScalarResult,
+ check: &dyn Fn([Scalar1; LANES], [Scalar2; LANES]) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector1: Into<[Scalar1; LANES]> + From<[Scalar1; LANES]> + Copy,
+ Vector2: Into<[Scalar2; LANES]> + From<[Scalar2; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_2(&|x: [Scalar1; LANES], y: [Scalar2; LANES]| {
+ proptest::prop_assume!(check(x, y));
+ let result_1: [ScalarResult; LANES] = fv(x.into(), y.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for ((i1, i2), o) in x.iter().zip(y.iter()).zip(result.iter_mut()) {
+ *o = fs(*i1, *i2);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a binary vector-scalar function against a binary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_binary_scalar_rhs_elementwise<
+ Scalar1,
+ Scalar2,
+ ScalarResult,
+ Vector,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Vector, Scalar2) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2) -> ScalarResult,
+ check: &dyn Fn([Scalar1; LANES], Scalar2) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar1; LANES]> + From<[Scalar1; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_2(&|x: [Scalar1; LANES], y: Scalar2| {
+ proptest::prop_assume!(check(x, y));
+ let result_1: [ScalarResult; LANES] = fv(x.into(), y).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for (i, o) in x.iter().zip(result.iter_mut()) {
+ *o = fs(*i, y);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a binary vector-scalar function against a binary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_binary_scalar_lhs_elementwise<
+ Scalar1,
+ Scalar2,
+ ScalarResult,
+ Vector,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Scalar1, Vector) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2) -> ScalarResult,
+ check: &dyn Fn(Scalar1, [Scalar2; LANES]) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector: Into<[Scalar2; LANES]> + From<[Scalar2; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_2(&|x: Scalar1, y: [Scalar2; LANES]| {
+ proptest::prop_assume!(check(x, y));
+ let result_1: [ScalarResult; LANES] = fv(x, y.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for (i, o) in y.iter().zip(result.iter_mut()) {
+ *o = fs(x, *i);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ });
+}
+
+/// Test a ternary vector function against a ternary scalar function, applied elementwise.
+#[inline(never)]
+pub fn test_ternary_elementwise<
+ Scalar1,
+ Scalar2,
+ Scalar3,
+ ScalarResult,
+ Vector1,
+ Vector2,
+ Vector3,
+ VectorResult,
+ const LANES: usize,
+>(
+ fv: &dyn Fn(Vector1, Vector2, Vector3) -> VectorResult,
+ fs: &dyn Fn(Scalar1, Scalar2, Scalar3) -> ScalarResult,
+ check: &dyn Fn([Scalar1; LANES], [Scalar2; LANES], [Scalar3; LANES]) -> bool,
+) where
+ Scalar1: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar2: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ Scalar3: Copy + Default + core::fmt::Debug + DefaultStrategy,
+ ScalarResult: Copy + Default + biteq::BitEq + core::fmt::Debug + DefaultStrategy,
+ Vector1: Into<[Scalar1; LANES]> + From<[Scalar1; LANES]> + Copy,
+ Vector2: Into<[Scalar2; LANES]> + From<[Scalar2; LANES]> + Copy,
+ Vector3: Into<[Scalar3; LANES]> + From<[Scalar3; LANES]> + Copy,
+ VectorResult: Into<[ScalarResult; LANES]> + From<[ScalarResult; LANES]> + Copy,
+{
+ test_3(
+ &|x: [Scalar1; LANES], y: [Scalar2; LANES], z: [Scalar3; LANES]| {
+ proptest::prop_assume!(check(x, y, z));
+ let result_1: [ScalarResult; LANES] = fv(x.into(), y.into(), z.into()).into();
+ let result_2: [ScalarResult; LANES] = {
+ let mut result = [ScalarResult::default(); LANES];
+ for ((i1, (i2, i3)), o) in
+ x.iter().zip(y.iter().zip(z.iter())).zip(result.iter_mut())
+ {
+ *o = fs(*i1, *i2, *i3);
+ }
+ result
+ };
+ crate::prop_assert_biteq!(result_1, result_2);
+ Ok(())
+ },
+ );
+}
+
+/// Expand a const-generic test into separate tests for each possible lane count.
+#[macro_export]
+macro_rules! test_lanes {
+ {
+ $(fn $test:ident<const $lanes:ident: usize>() $body:tt)*
+ } => {
+ $(
+ mod $test {
+ use super::*;
+
+ fn implementation<const $lanes: usize>()
+ where
+ core_simd::LaneCount<$lanes>: core_simd::SupportedLaneCount,
+ $body
+
+ #[cfg(target_arch = "wasm32")]
+ wasm_bindgen_test::wasm_bindgen_test_configure!(run_in_browser);
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ fn lanes_1() {
+ implementation::<1>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ fn lanes_2() {
+ implementation::<2>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
+ fn lanes_4() {
+ implementation::<4>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
++ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_8() {
+ implementation::<8>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
++ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_16() {
+ implementation::<16>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
++ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_32() {
+ implementation::<32>();
+ }
+
+ #[test]
+ #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
++ #[cfg(not(miri))] // Miri intrinsic implementations are uniform and larger tests are sloooow
+ fn lanes_64() {
+ implementation::<64>();
+ }
+ }
+ )*
+ }
+}
+
+/// Expand a const-generic `#[should_panic]` test into separate tests for each possible lane count.
+#[macro_export]
+macro_rules! test_lanes_panic {
+ {
+ $(fn $test:ident<const $lanes:ident: usize>() $body:tt)*
+ } => {
+ $(
+ mod $test {
+ use super::*;
+
+ fn implementation<const $lanes: usize>()
+ where
+ core_simd::LaneCount<$lanes>: core_simd::SupportedLaneCount,
+ $body
+
+ #[test]
+ #[should_panic]
+ fn lanes_1() {
+ implementation::<1>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_2() {
+ implementation::<2>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_4() {
+ implementation::<4>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_8() {
+ implementation::<8>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_16() {
+ implementation::<16>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_32() {
+ implementation::<32>();
+ }
+
+ #[test]
+ #[should_panic]
+ fn lanes_64() {
+ implementation::<64>();
+ }
+ }
+ )*
+ }
+}
projects / rust.git / commitdiff