about summary refs log tree commit diff
path: root/library/std/src/io
AgeCommit message (Collapse)AuthorLines
2020-12-08Fix outdated comment about not needing to flush stderr.Mara Bos-7/+3
2020-12-04IntoInnerError: Provide into_errorIan Jackson-0/+21
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
2020-12-04IntoInnerError: Provide into_partsIan Jackson-0/+24
In particular, IntoIneerError only currently provides .error() which returns a reference, not an owned value. This is not helpful and means that a caller of BufWriter::into_inner cannot acquire an owned io::Error which seems quite wrong. Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
2020-12-04std: impl of `Write` for `&mut [u8]`: document the buffer full errorIan Jackson-0/+4
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
2020-12-04BufWriter: Provide into_raw_partsIan Jackson-0/+73
If something goes wrong, one might want to unpeel the layers of nested Writers to perform recovery actions on the underlying writer, or reuse its resources. `into_inner` can be used for this when the inner writer is still working. But when the inner writer is broken, and returning errors, `into_inner` simply gives you the error from flush, and the same `Bufwriter` back again. Here I provide the necessary function, which I have chosen to call `into_raw_parts`. I had to do something with `panicked`. Returning it to the caller as a boolean seemed rather bare. Throwing the buffered data away in this situation also seems unfriendly: maybe the programmer knows something about the underlying writer and can recover somehow. So I went for a custom Error. This may be overkill, but it does have the nice property that a caller who actually wants to look at the buffered data, rather than simply extracting the inner writer, will be told by the type system if they forget to handle the panicked case. If a caller doesn't need the buffer, it can just be discarded. That WriterPanicked is a newtype around Vec<u8> means that hopefully the layouts of the Ok and Err variants can be very similar, with just a boolean discriminant. So this custom error type should compile down to nearly no code. Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
2020-11-22Reduce branching in write_vectored for BufWriterMikhail Zabaluev-14/+9
Do what write does and optimize for the most likely case: slices are much smaller than the buffer. If a slice does not fit completely in the remaining capacity of the buffer, it is left out rather than buffered partially. Special treatment is only left for oversized slices that are written directly to the underlying writer.
2020-11-22Fix is_write_vectored in LineWriterShimMikhail Zabaluev-1/+7
Now that BufWriter always claims to support vectored writes, look through it at the wrapped writer to decide whether to use vectored writes for LineWriter.
2020-11-22Make is_write_vectored return true for BufWriterMikhail Zabaluev-1/+1
BufWriter provides an efficient implementation of write_vectored also when the underlying writer does not support vectored writes.
2020-11-22Optimize write_vectored for BufWriterMikhail Zabaluev-12/+50
If the underlying writer does not support efficient vectored output, do it differently: always try to coalesce the slices in the buffer until one comes that does not fit entirely. Flush the buffer before the first slice if needed.
2020-11-17Fix typo in `std::io::Write` docsWilliam Chargin-2/+2
These referred to a “`Write`er”—extra *e*. Presumably a copy-paste holdover from “`Read`er”. Test Plan: Running ``git grep '`\?[Ww]rite`\?er'`` no longer finds any results. wchargin-branch: io-write-docs
2020-11-16Rollup merge of #78714 - m-ou-se:simplify-local-streams, r=KodrAusMara Bos-152/+47
Simplify output capturing This is a sequence of incremental improvements to the unstable/internal `set_panic` and `set_print` mechanism used by the `test` crate: 1. Remove the `LocalOutput` trait and use `Arc<Mutex<dyn Write>>` instead of `Box<dyn LocalOutput>`. In practice, all implementations of `LocalOutput` were just `Arc<Mutex<..>>`. This simplifies some logic and removes all custom `Sink` implementations such as `library/test/src/helpers/sink.rs`. Also removes a layer of indirection, as the outermost `Box` is now gone. It also means that locking now happens per `write_fmt`, not per individual `write` within. (So `"{} {}\n"` now results in one `lock()`, not four or more.) 2. Since in all cases the `dyn Write`s were just `Vec<u8>`s, replace the type with `Arc<Mutex<Vec<u8>>>`. This simplifies things more, as error handling and flushing can be removed now. This also removes the hack needed in the default panic handler to make this work with `::realstd`, as (unlike `Write`) `Vec<u8>` is from `alloc`, not `std`. 3. Replace the `RefCell`s by regular `Cell`s. The `RefCell`s were mostly used as `mem::replace(&mut *cell.borrow_mut(), something)`, which is just `Cell::replace`. This removes an unecessary bookkeeping and makes the code a bit easier to read. 4. Merge `set_panic` and `set_print` into a single `set_output_capture`. Neither the test crate nor rustc (the only users of this feature) have a use for using these separately. Merging them simplifies things even more. This uses a new function name and feature name, to make it clearer this is internal and not supposed to be used by other crates. Might be easier to review per commit.
2020-11-14Auto merge of #75272 - the8472:spec-copy, r=KodrAusbors-74/+102
specialize io::copy to use copy_file_range, splice or sendfile Fixes #74426. Also covers #60689 but only as an optimization instead of an official API. The specialization only covers std-owned structs so it should avoid the problems with #71091 Currently linux-only but it should be generalizable to other unix systems that have sendfile/sosplice and similar. There is a bit of optimization potential around the syscall count. Right now it may end up doing more syscalls than the naive copy loop when doing short (<8KiB) copies between file descriptors. The test case executes the following: ``` [pid 103776] statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=17, ...}) = 0 [pid 103776] write(4, "wxyz", 4) = 4 [pid 103776] write(4, "iklmn", 5) = 5 [pid 103776] copy_file_range(3, NULL, 4, NULL, 5, 0) = 5 ``` 0-1 `stat` calls to identify the source file type. 0 if the type can be inferred from the struct from which the FD was extracted 𝖬 `write` to drain the `BufReader`/`BufWriter` wrappers. only happen when buffers are present. 𝖬 ≾ number of wrappers present. If there is a write buffer it may absorb the read buffer contents first so only result in a single write. Vectored writes would also be an option but that would require more invasive changes to `BufWriter`. 𝖭 `copy_file_range`/`splice`/`sendfile` until file size, EOF or the byte limit from `Take` is reached. This should generally be *much* more efficient than the read-write loop and also have other benefits such as DMA offload or extent sharing. ## Benchmarks ``` OLD test io::tests::bench_file_to_file_copy ... bench: 21,002 ns/iter (+/- 750) = 6240 MB/s [ext4] test io::tests::bench_file_to_file_copy ... bench: 35,704 ns/iter (+/- 1,108) = 3671 MB/s [btrfs] test io::tests::bench_file_to_socket_copy ... bench: 57,002 ns/iter (+/- 4,205) = 2299 MB/s test io::tests::bench_socket_pipe_socket_copy ... bench: 142,640 ns/iter (+/- 77,851) = 918 MB/s NEW test io::tests::bench_file_to_file_copy ... bench: 14,745 ns/iter (+/- 519) = 8889 MB/s [ext4] test io::tests::bench_file_to_file_copy ... bench: 6,128 ns/iter (+/- 227) = 21389 MB/s [btrfs] test io::tests::bench_file_to_socket_copy ... bench: 13,767 ns/iter (+/- 3,767) = 9520 MB/s test io::tests::bench_socket_pipe_socket_copy ... bench: 26,471 ns/iter (+/- 6,412) = 4951 MB/s ```
2020-11-13limit visibility of copy offload helpers to sys::unix moduleThe8472-182/+0
2020-11-13move copy specialization tests to their own moduleThe8472-181/+182
2020-11-13move copy specialization into sys::unix moduleThe8472-377/+8
2020-11-13add benchmarksThe8472-1/+131
2020-11-13reduce syscalls by inferring FD types based on source struct instead of ↵The8472-66/+102
calling stat() also adds handling for edge-cases involving large sparse files where sendfile could fail with EOVERFLOW
2020-11-13add forwarding specializations for &mut variantsThe8472-0/+21
`impl Write for &mut T where T: Write`, thus the same should apply to the specialization traits
2020-11-13prioritize sendfile over splice since it results in fewer context switches ↵The8472-16/+16
when sending to pipes splice returns to userspace when the pipe is full, sendfile just blocks until it's done, this can achieve much higher throughput
2020-11-13move tests module into separate fileThe8472-57/+52
2020-11-13hide unused exports on other platformsThe8472-3/+3
2020-11-13specialize io::copy to use copy_file_range, splice or sendfileThe8472-73/+469
Currently it only applies to linux systems. It can be extended to make use of similar syscalls on other unix systems.
2020-11-10Merge set_panic and set_print into set_output_capture.Mara Bos-90/+30
There were no use cases for setting them separately. Merging them simplifies some things.
2020-11-10Use Cell instead of RefCell for LOCAL_{STDOUT,STDERR}.Mara Bos-15/+16
2020-11-10Use Vec<u8> for LOCAL_STD{OUT,ERR} instead of dyn Write.Mara Bos-38/+23
It was only ever used with Vec<u8> anyway. This simplifies some things. - It no longer needs to be flushed, because that's a no-op anyway for a Vec<u8>. - Writing to a Vec<u8> never fails. - No #[cfg(test)] code is needed anymore to use `realstd` instead of `std`, because Vec comes from alloc, not std (like Write).
2020-11-10Remove io::LocalOutput and use Arc<Mutex<dyn>> for local streams.Mara Bos-52/+21
2020-11-08Rollup merge of #78811 - a1phyr:const_io_structs, r=dtolnayMara Bos-7/+27
Make some std::io functions `const` Tracking issue: #78812 Make the following functions `const`: - `io::Cursor::new` - `io::Cursor::get_ref` - `io::Cursor::position` - `io::empty` - `io::repeat` - `io::sink` r? `````@dtolnay`````
2020-11-06Add tracking issueBenoît du Garreau-6/+6
2020-11-06Make some std::io functions `const`Benoît du Garreau-7/+27
Includes: - io::Cursor::new - io::Cursor::get_ref - io::Cursor::position - io::empty - io::repeat - io::sink
2020-11-05document HACKsPeter Jaszkowiak-0/+2
2020-11-05Intra-doc links for std::io::bufferedPeter Jaszkowiak-3/+3
2020-10-27Auto merge of #78227 - SergioBenitez:test-stdout-threading, r=m-ou-sebors-9/+39
Capture output from threads spawned in tests This is revival of #75172. Original text: > Fixes #42474. > > r? `@​dtolnay` since you expressed interest in this, but feel free to redirect if you aren't the right person anymore. --- Closes #75172.
2020-10-26fix(docs): typo in BufWriter documentationMichele Lacchia-1/+1
2020-10-22Only load LOCAL_STREAMS if they are being usedSergio Benitez-0/+5
2020-10-22Capture output from threads spawned in testsTyler Mandry-9/+34
Fixes #42474.
2020-10-16Rollup merge of #76084 - Lucretiel:split-buffered, r=dtolnayDylan DPC-1438/+1463
Refactor io/buffered.rs into submodules This pull request splits `BufWriter`, `BufReader`, `LineWriter`, and `LineWriterShim` (along with their associated tests) into separate submodules. It contains no functional changes. This change is being made in anticipation of adding another type of buffered writer which can be switched between line- and block-buffering mode. Part of a series of pull requests resolving #60673.
2020-09-27Optimize set_{panic,print}(None).Mara Bos-0/+8
2020-09-27Relax memory ordering of LOCAL_STREAMS and document it.Mara Bos-5/+17
2020-09-27Only use LOCAL_{STDOUT,STDERR} when set_{print/panic} is used.Mara Bos-23/+40
The thread local LOCAL_STDOUT and LOCAL_STDERR are only used by the test crate to capture output from tests when running them in the same process in differen threads. However, every program will check these variables on every print, even outside of testing. This involves allocating a thread local key, and registering a thread local destructor. This can be somewhat expensive. This change keeps a global flag (LOCAL_STREAMS) which will be set to true when either of these local streams is used. (So, effectively only in test and benchmark runs.) When this flag is off, these thread locals are not even looked at and therefore will not be initialized on the first output on every thread, which also means no thread local destructors will be registered.
2020-09-27Auto merge of #77154 - fusion-engineering-forks:lazy-stdio, r=dtolnaybors-100/+40
Remove std::io::lazy::Lazy in favour of SyncOnceCell The (internal) std::io::lazy::Lazy was used to lazily initialize the stdout and stdin buffers (and mutexes). It uses atexit() to register a destructor to flush the streams on exit, and mark the streams as 'closed'. Using the stream afterwards would result in a panic. Stdout uses a LineWriter which contains a BufWriter that will flush the buffer on drop. This one is important to be executed during shutdown, to make sure no buffered output is lost. It also forbids access to stdout afterwards, since the buffer is already flushed and gone. Stdin uses a BufReader, which does not implement Drop. It simply forgets any previously read data that was not read from the buffer yet. This means that in the case of stdin, the atexit() function's only effect is making stdin inaccessible to the program, such that later accesses result in a panic. This is uncessary, as it'd have been safe to access stdin during shutdown of the program. --- This change removes the entire io::lazy module in favour of SyncOnceCell. SyncOnceCell's fast path is much faster (a single atomic operation) than locking a sys_common::Mutex on every access like Lazy did. However, SyncOnceCell does not use atexit() to drop the contained object during shutdown. As noted above, this is not a problem for stdin. It simply means stdin is now usable during shutdown. The atexit() call for stdout is moved to the stdio module. Unlike the now-removed Lazy struct, SyncOnceCell does not have a 'gone and unusable' state that panics. Instead of adding this again, this simply replaces the buffer with one with zero capacity. This effectively flushes the old buffer *and* makes any writes afterwards pass through directly without touching a buffer, making print!() available during shutdown without panicking. --- In addition, because the contents of the SyncOnceCell are no longer dropped, we can now use `&'static` instead of `Arc` in `Stdout` and `Stdin`. This also saves two levels of indirection in `stdin()` and `stdout()`, since Lazy effectively stored a `Box<Arc<T>>`, and SyncOnceCell stores the `T` directly.
2020-09-24Call ReentrantMutex::init() in stdout().Mara Bos-1/+3
2020-09-24Drop use of Arc from Stdin and Stdout.Mara Bos-27/+23
2020-09-24Remove std::io::lazy::Lazy in favour of SyncOnceCellMara Bos-98/+40
The (internal) std::io::lazy::Lazy was used to lazily initialize the stdout and stdin buffers (and mutexes). It uses atexit() to register a destructor to flush the streams on exit, and mark the streams as 'closed'. Using the stream afterwards would result in a panic. Stdout uses a LineWriter which contains a BufWriter that will flush the buffer on drop. This one is important to be executed during shutdown, to make sure no buffered output is lost. It also forbids access to stdout afterwards, since the buffer is already flushed and gone. Stdin uses a BufReader, which does not implement Drop. It simply forgets any previously read data that was not read from the buffer yet. This means that in the case of stdin, the atexit() function's only effect is making stdin inaccessible to the program, such that later accesses result in a panic. This is uncessary, as it'd have been safe to access stdin during shutdown of the program. --- This change removes the entire io::lazy module in favour of SyncOnceCell. SyncOnceCell's fast path is much faster (a single atomic operation) than locking a sys_common::Mutex on every access like Lazy did. However, SyncOnceCell does not use atexit() to drop the contained object during shutdown. As noted above, this is not a problem for stdin. It simply means stdin is now usable during shutdown. The atexit() call for stdout is moved to the stdio module. Unlike the now-removed Lazy struct, SyncOnceCell does not have a 'gone and unusable' state that panics. Instead of adding this again, this simply replaces the buffer with one with zero capacity. This effectively flushes the old buffer *and* makes any writes afterwards pass through directly without touching a buffer, making print!() available during shutdown without panicking.
2020-09-21Rollup merge of #76275 - FedericoPonzi:immutable-write-impl-73836, r=dtolnayecstatic-morse-0/+78
Implementation of Write for some immutable ref structs Fixes #73836
2020-09-21Updates stability attributes to the current nightly versionFederico Ponzi-3/+3
2020-09-11Deduplicates io::Write implementationsFederico Ponzi-14/+14
2020-09-10move buffered.rs to mod.rsNathan West-1/+1
2020-09-10Refactor io/buffered.rs into submodulesNathan West-1306/+1331
2020-09-07Auto merge of #74366 - t-rapp:tr-bufreader-pos, r=LukasKalbertodtbors-0/+87
Implement Seek::stream_position() for BufReader Optimization over `BufReader::seek()` for getting the current position without flushing the internal buffer. Related to #31100. Based on the code in #70577.
2020-09-07Implement Seek::stream_position() for BufReaderTobias Rapp-0/+87
Optimization over BufReader::seek() for getting the current position without flushing the internal buffer. Related to #31100. Based on code in #70577.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-07-13 02:01:51 +0000