| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
We can create the expected error manually, rather than trying to produce
a real one, so the error conversion test can run on all targets. Before,
it was only running on 64-bit and not miri.
In Fedora, we also found that s390x was not getting the expected error,
"successfully" allocating the huge size because it was optimizing the
real `malloc` call away. It's possible to counter that by looking at the
pointer in any way, like a debug print, but it's more robust to just
deal with errors directly, since this test is only about conversion.
|
|
|
|
They are either outdated as emscripten now supports i128 or they are
subsumed by #[cfg_attr(not(panic = "unwind"), ignore]
|
|
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
|
|
|
|
Co-authored-by: Mike Pedersen <mike@mikepedersen.dk>
Co-authored-by: Nathan West <Lucretiel@gmail.com>
|
|
|
|
|
|
|
|
Replace `io::Cursor::{remaining_slice, is_empty}`
This is a late follow up to the concerns raised in https://github.com/rust-lang/rust/issues/86369.
https://github.com/rust-lang/rust/issues/86369#issuecomment-953096691
> This API seems focussed on the `Read` side of things. When `Seek`ing around and `Write`ing data, `is_empty` becomes confusing and `remaining_slice` is not very useful. When writing, the part of the slice before the cursor is much more interesting. Maybe we should have functions for both? Or a single function that returns both slices? (If we also have a `mut` version, a single function would be useful to allow mutable access to both sides at once.)
New feature name: `cursor_remaining` > `cursor_split`.
Added functions:
```rust
fn split(&self) -> (&[u8], &[u8]);
// fn before(&self) -> &[u8];
// fn after(&self) -> &[u8];
fn split_mut(&mut self) -> (&mut [u8], &mut [u8]);
// fn before_mut(&mut self) -> &mut [u8];
// fn after_mut(&mut self) -> &mut [u8];
```
A question was raised in https://github.com/rust-lang/rust/issues/86369#issuecomment-927124211 about whether to return a lifetime that would reflect the lifetime of the underlying bytes (`impl Cursor<&'a [u8]> { fn after(&self) -> &'a [u8] }`). The downside of doing this would be that it would not be possible to implement these functions generically over `T: AsRef<[u8]>`.
## Update
Based on the review, before* and after* methods where removed.
|
|
The previous commit updated `rustfmt.toml` appropriately. This commit is
the outcome of running `x fmt --all` with the new formatting options.
|
|
split_mut}`
|
|
Many tiny changes to stdlib doc comments to make them consistent (for example
"Returns foo", rather than "Return foo", per RFC1574), adding missing periods, paragraph
breaks, backticks for monospace style, and other minor nits.
https://github.com/rust-lang/rfcs/blob/master/text/1574-more-api-documentation-conventions.md#appendix-a-full-conventions-text
|
|
|
|
|
|
|
|
|
|
Specialize some methods of `io::Chain`
This PR specializes the implementation of some methods of `io::Chain`, which could bring performance improvements when using it.
|
|
This also keeps the old `advance` method under `advance_unchecked` name.
This makes pattern like `std::io::default_read_buf` safe to write.
|
|
|
|
|
|
Add `BufRead::skip_until`
Alternative version of `BufRead::read_until` that simply discards data, rather than copying it into a buffer.
Useful for situations like skipping irrelevant data in a binary file format that is NUL-terminated.
<details>
<summary>Benchmark</summary>
```
running 2 tests
test bench_read_until ... bench: 123 ns/iter (+/- 6)
test bench_skip_until ... bench: 66 ns/iter (+/- 3)
```
```rs
#![feature(test)]
extern crate test;
use test::Bencher;
use std::io::{ErrorKind, BufRead};
fn skip_until<R: BufRead + ?Sized>(r: &mut R, delim: u8) -> Result<usize, std::io::Error> {
let mut read = 0;
loop {
let (done, used) = {
let available = match r.fill_buf() {
Ok(n) => n,
Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
Err(e) => return Err(e),
};
match memchr::memchr(delim, available) {
Some(i) => (true, i + 1),
None => (false, available.len()),
}
};
r.consume(used);
read += used;
if done || used == 0 {
return Ok(read);
}
}
}
const STR: &[u8] = b"Ferris\0Hello, world!\0";
#[bench]
fn bench_skip_until(b: &mut Bencher) {
b.iter(|| {
let mut io = std::io::Cursor::new(test::black_box(STR));
skip_until(&mut io, b'\0').unwrap();
let mut hello = Vec::with_capacity(b"Hello, world!\0".len());
let num_bytes = io.read_until(b'\0', &mut hello).unwrap();
assert_eq!(num_bytes, b"Hello, world!\0".len());
assert_eq!(hello, b"Hello, world!\0");
});
}
#[bench]
fn bench_read_until(b: &mut Bencher) {
b.iter(|| {
let mut io = std::io::Cursor::new(test::black_box(STR));
io.read_until(b'\0', &mut Vec::new()).unwrap();
let mut hello = Vec::with_capacity(b"Hello, world!\0".len());
let num_bytes = io.read_until(b'\0', &mut hello).unwrap();
assert_eq!(num_bytes, b"Hello, world!\0".len());
assert_eq!(hello, b"Hello, world!\0");
});
}
```
</details>
|
|
The extra `\0` in this commit is needed because the assertion on line 49 will fail otherwise (as `skip_until` stops reading on EOF and therefore does not read a trailing `\0`, returning 6 read bytes rather than the expected 7)
|
|
|
|
|
|
This works around performance issues on Windows by limiting reads the size of reads when the expected size is known.
|
|
std::io: migrate ReadBuf to BorrowBuf/BorrowCursor
This PR replaces `ReadBuf` (used by the `Read::read_buf` family of methods) with `BorrowBuf` and `BorrowCursor`.
The general idea is to split `ReadBuf` because its API is large and confusing. `BorrowBuf` represents a borrowed buffer which is mostly read-only and (other than for construction) deals only with filled vs unfilled segments. a `BorrowCursor` is a mostly write-only view of the unfilled part of a `BorrowBuf` which distinguishes between initialized and uninitialized segments. For `Read::read_buf`, the caller would create a `BorrowBuf`, then pass a `BorrowCursor` to `read_buf`.
In addition to the major API split, I've made the following smaller changes:
* Removed some methods entirely from the API (mostly the functionality can be replicated with two calls rather than a single one)
* Unified naming, e.g., by replacing initialized with init and assume_init with set_init
* Added an easy way to get the number of bytes written to a cursor (`written` method)
As well as simplifying the API (IMO), this approach has the following advantages:
* Since we pass the cursor by value, we remove the 'unsoundness footgun' where a malicious `read_buf` could swap out the `ReadBuf`.
* Since `read_buf` cannot write into the filled part of the buffer, we prevent the filled part shrinking or changing which could cause underflow for the caller or unexpected behaviour.
## Outline
```rust
pub struct BorrowBuf<'a>
impl Debug for BorrowBuf<'_>
impl<'a> From<&'a mut [u8]> for BorrowBuf<'a>
impl<'a> From<&'a mut [MaybeUninit<u8>]> for BorrowBuf<'a>
impl<'a> BorrowBuf<'a> {
pub fn capacity(&self) -> usize
pub fn len(&self) -> usize
pub fn init_len(&self) -> usize
pub fn filled(&self) -> &[u8]
pub fn unfilled<'this>(&'this mut self) -> BorrowCursor<'this, 'a>
pub fn clear(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
}
pub struct BorrowCursor<'buf, 'data>
impl<'buf, 'data> BorrowCursor<'buf, 'data> {
pub fn clone<'this>(&'this mut self) -> BorrowCursor<'this, 'data>
pub fn capacity(&self) -> usize
pub fn written(&self) -> usize
pub fn init_ref(&self) -> &[u8]
pub fn init_mut(&mut self) -> &mut [u8]
pub fn uninit_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn as_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn advance(&mut self, n: usize) -> &mut Self
pub fn ensure_init(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
pub fn append(&mut self, buf: &[u8])
}
```
## TODO
* ~~Migrate non-unix libs and tests~~
* ~~Naming~~
* ~~`BorrowBuf` or `BorrowedBuf` or `SliceBuf`? (We might want an owned equivalent for the async IO traits)~~
* ~~Should we rename the `readbuf` module? We might keep the name indicate it includes both the buf and cursor variations and someday the owned version too. Or we could change it. It is not publicly exposed, so it is not that important~~.
* ~~`read_buf` method: we read into the cursor now, so the `_buf` suffix is a bit weird.~~
* ~~Documentation~~
* Tests are incomplete (I adjusted existing tests, but did not add new ones).
cc https://github.com/rust-lang/rust/issues/78485, https://github.com/rust-lang/rust/issues/94741
supersedes: https://github.com/rust-lang/rust/pull/95770, https://github.com/rust-lang/rust/pull/93359
fixes #93305
|
|
|
|
Signed-off-by: Nick Cameron <nrc@ncameron.org>
|
|
Signed-off-by: Nick Cameron <nrc@ncameron.org>
|
|
protect `std::io::Take::limit` from overflow in `read`
Resolves #94981
|
|
fixs #94981
|
|
|
|
|
|
|
|
Reading a file into an empty vector or string buffer can incur
unnecessary `read` syscalls and memory re-allocations as the buffer
"warms up" and grows to its final size. This is perhaps a necessary evil
with generic readers, but files can be read in smarter by checking the
file size and reserving that much capacity.
`std::fs::read` and `read_to_string` already perform this optimization:
they open the file, reads its metadata, and call `with_capacity` with
the file size. This ensures that the buffer does not need to be resized
and an initial string of small `read` syscalls.
However, if a user opens the `File` themselves and calls
`file.read_to_end` or `file.read_to_string` they do not get this
optimization.
```rust
let mut buf = Vec::new();
file.read_to_end(&mut buf)?;
```
I searched through this project's codebase and even here are a *lot* of
examples of this. They're found all over in unit tests, which isn't a
big deal, but there are also several real instances in the compiler and
in Cargo. I've documented the ones I found in a comment here:
https://github.com/rust-lang/rust/issues/89516#issuecomment-934423999
Most telling, the `Read` trait and the `read_to_end` method both show
this exact pattern as examples of how to use readers. What this says to
me is that this shouldn't be solved by simply fixing the instances of it
in this codebase. If it's here it's certain to be prevalent in the wider
Rust ecosystem.
To that end, this commit adds specializations of `read_to_end` and
`read_to_string` directly on `File`. This way it's no longer a minor
footgun to start with an empty buffer when reading a file in.
A nice side effect of this change is that code that accesses a `File` as
a bare `Read` constraint or via a `dyn Read` trait object will benefit.
For example, this code from `compiler/rustc_serialize/src/json.rs`:
```rust
pub fn from_reader(rdr: &mut dyn Read) -> Result<Json, BuilderError> {
let mut contents = Vec::new();
match rdr.read_to_end(&mut contents) {
```
Related changes:
- I also added specializations to `BufReader` to delegate to
`self.inner`'s methods. That way it can call `File`'s optimized
implementations if the inner reader is a file.
- The private `std::io::append_to_string` function is now marked
`unsafe`.
- `File::read_to_string` being more efficient means that the performance
note for `io::read_to_string` can be softened. I've added @camelid's
suggested wording from:
https://github.com/rust-lang/rust/issues/80218#issuecomment-936806502
|
|
If you know how much data to expect and use `Vec::with_capacity` to
pre-allocate a buffer of that capacity, `Read::read_to_end` will still
double its capacity. It needs some space to perform a read, even though
that read ends up returning `0`.
It's a bummer to carefully pre-allocate 1GB to read a 1GB file into
memory and end up using 2GB.
This fixes that behavior by special casing a full buffer and reading
into a small "probe" buffer instead. If that read returns `0` then it's
confirmed that the buffer was the perfect size. If it doesn't, the probe
buffer is appended to the normal buffer and the read loop continues.
Fixing this allows several workarounds in the standard library to be
removed:
- `Take` no longer needs to override `Read::read_to_end`.
- The `reservation_size` callback that allowed `Take` to inhibit the
previous over-allocation behavior isn't needed.
- `fs::read` doesn't need to reserve an extra byte in
`initial_buffer_size`.
Curiously, there was a unit test that specifically checked that
`Read::read_to_end` *does* over-allocate. I removed that test, too.
|
|
|
|
Add has_data_left() to BufRead
This is a continuation of #40747 and also addresses #40745. The problem with the previous PR was that it had "eof" in its method name. This PR uses a more descriptive method name, but I'm open to changing it.
|
|
Specialize `io::Bytes::size_hint` for more types
Improve the result of `<io::Bytes as Iterator>::size_hint` for some readers. I did not manage to specialize `SizeHint` for `io::Cursor`
Side question: would it be interesting for `io::Read` to have an optional `size_hint` method ?
|
|
|
|
|
|
|
|
To make way for a new IoSlice(Mut)::advance function that advances a
single slice.
Also changes the signature to accept a `&mut &mut [IoSlice]`, not
returning anything. This will better match the future IoSlice::advance
function.
|
|
|
|
|
|
|
|
|
|
|
