]> git.lizzy.rs Git - rust.git/commit - src/tools/rust-analyzer
Auto merge of #89165 - jkugelman:read-to-end-overallocation, r=joshtriplett
authorbors <bors@rust-lang.org>
Mon, 4 Oct 2021 04:44:56 +0000 (04:44 +0000)
committerbors <bors@rust-lang.org>
Mon, 4 Oct 2021 04:44:56 +0000 (04:44 +0000)
commitd25de31a0eeb14ab0c8c4613496fe2d3d9a085dd
tree777a374c55c4ffea0abfe1204289fe34ee19babd
parente737694a4d66b01308b73d4559a35b43e414faf9
parent9b9c24ec7f7c850d0c92babeb45a5ccff940a2c1
Auto merge of #89165 - jkugelman:read-to-end-overallocation, r=joshtriplett

Fix read_to_end to not grow an exact size buffer

If you know how much data to expect and use `Vec::with_capacity` to pre-allocate a buffer of that capacity, `Read::read_to_end` will still double its capacity. It needs some space to perform a read, even though that read ends up returning `0`.

It's a bummer to carefully pre-allocate 1GB to read a 1GB file into memory and end up using 2GB.

This fixes that behavior by special casing a full buffer and reading into a small "probe" buffer instead. If that read returns `0` then it's confirmed that the buffer was the perfect size. If it doesn't, the probe buffer is appended to the normal buffer and the read loop continues.

Fixing this allows several workarounds in the standard library to be removed:

- `Take` no longer needs to override `Read::read_to_end`.
- The `reservation_size` callback that allowed `Take` to inhibit the previous over-allocation behavior isn't needed.
- `fs::read` doesn't need to reserve an extra byte in `initial_buffer_size`.

Curiously, there was a unit test that specifically checked that `Read::read_to_end` *does* over-allocate. I removed that test, too.
library/std/src/fs.rs
library/std/src/io/mod.rs