| Age | Commit message (Collapse) | Author | Lines |
|---|
|
In addition to adding this method, a handful of passes are updated to use it.
|
|
Co-authored-by: Joe ST <joe@fbstj.net>
|
|
CTFE interning: don't walk allocations that don't need it
The interning of const allocations visits the mplace looking for references to intern. Walking big aggregates like big static arrays can be costly, so we only do it if the allocation we're interning contains references or interior mutability.
Walking ZSTs was avoided before, and this optimization is now applied to cases where there are no references/relocations either.
---
While initially looking at this in the context of #93215, I've been testing with smaller allocations than the 16GB one in that issue, and with different init/uninit patterns (esp. via padding).
In that example, by default, `eval_to_allocation_raw` is the heaviest query followed by `incr_comp_serialize_result_cache`. So I'll show numbers when incremental compilation is disabled, to focus on the const allocations themselves at 95% of the compilation time, at bigger array sizes on these minimal examples like `static ARRAY: [u64; LEN] = [0; LEN];`.
That is a close construction to parts of the `ctfe-stress-test-5` benchmark, which has const allocations in the megabytes, while most crates usually have way smaller ones. This PR will have the most impact in these situations, as the walk during the interning starts to dominate the runtime.
Unicode crates (some of which are present in our benchmarks) like `ucd`, `encoding_rs`, etc come to mind as having bigger than usual allocations as well, because of big tables of code points (in the hundreds of KB, so still an order of magnitude or 2 less than the stress test).
In a check build, for a single static array shown above, from 100 to 10^9 u64s (for lengths in powers of ten), the constant factors are lowered:
(log scales for easier comparisons)
(linear scale for absolute diff at higher Ns)
For one of the alternatives of that issue
```rust
const ROWS: usize = 100_000;
const COLS: usize = 10_000;
static TWODARRAY: [[u128; COLS]; ROWS] = [[0; COLS]; ROWS];
```
we can see a similar reduction of around 3x (from 38s to 12s or so).
For the same size, the slowest case IIRC is when there are uninitialized bytes e.g. via padding
```rust
const ROWS: usize = 100_000;
const COLS: usize = 10_000;
static TWODARRAY: [[(u64, u8); COLS]; ROWS] = [[(0, 0); COLS]; ROWS];
```
then interning/walking does not dominate anymore (but means there is likely still some interesting work left to do here).
Compile times in this case rise up quite a bit, and avoiding interning walks has less impact: around 23%, from 730s on master to 568s with this PR.
|
|
|
|
|
|
move MIR syntax into a dedicated file and ping some people whenever it changes
Adding or changing MIR operations/statements/whatever should be under significant scrutiny wrt their wider impact, specified semantics, and so on. So let's start by putting all that into a dedicated file and pinging some people whenever that file changes.
This PR only moves definitions around, and then fiddles with imports until it all works again.
|
|
|
|
|
|
interpret: adjust error from constructing an invalid value
|
|
|
|
|
|
|
|
This limits access to the relocations data a bit (instead of increasing it just
for the purposes of interning).
|
|
this makes the api more flexible. the original function now calls the PlaceRef
version to avoid duplicating the code.
|
|
interpret: convert_tag_add_extra: allow tagger to raise errors
Needed for https://github.com/rust-lang/miri/issues/2234
r? `@oli-obk`
|
|
Remove dereferencing of Box from codegen
Through #94043, #94414, #94873, and #95328, I've been fixing issues caused by Box being treated like a pointer when it is not a pointer. However, these PRs just introduced special cases for Box. This PR removes those special cases and instead transforms a deref of Box into a deref of the pointer it contains.
Hopefully, this is the end of the Box<T, A> ICEs.
|
|
|
|
|
|
ctfe: limit hashing of big const allocations when interning
Const allocations are only hashed for interning. However, they can be large, making the hashing expensive especially since it uses `FxHash`: it's better suited to short keys, not potentially big buffers like the actual bytes of allocation and the associated 1/8th sized `InitMask`.
We can partially hash these fields when they're large, hashing the length, and head and tail of these buffers, to
limit possible collisions while avoiding most of the hashing work.
r? `@ghost`
|
|
|
|
Big const allocations hash a large amount of data for interning:
the whole bytes buffer, and the 1/8th sized initmask, with FxHash.
This hash function is made for shorter keys.
This only hashes the length, and head and tail of these buffers, to
limit possible collisions while avoiding most of the hashing work.
|
|
raise errors
|
|
|
|
|
|
Rename rustc_serialize::opaque::Encoder as MemEncoder.
This avoids the name clash with `rustc_serialize::Encoder` (a trait),
and allows lots qualifiers to be removed and imports to be simplified
(e.g. fewer `as` imports).
(This was previously merged as commit 5 in #94732 and then was reverted
in #97905 because of a perf regression caused by commit 4 in #94732.)
r? ```@bjorn3```
|
|
|
|
|
|
|
|
|
|
This avoids the name clash with `rustc_serialize::Encoder` (a trait),
and allows lots qualifiers to be removed and imports to be simplified
(e.g. fewer `as` imports).
(This was previously merged as commit 5 in #94732 and then was reverted
in #97905 because of a perf regression caused by commit 4 in #94732.)
|
|
And likewise for the `Const::val` method.
Because its type is called `ConstKind`. Also `val` is a confusing name
because `ConstKind` is an enum with seven variants, one of which is
called `Value`. Also, this gives consistency with `TyS` and `PredicateS`
which have `kind` fields.
The commit also renames a few `Const` variables from `val` to `c`, to
avoid confusion with the `ConstKind::Value` variant.
|
|
|
|
Revert part of #94372 to improve performance
#94732 was supposed to give small but widespread performance improvements, as judged from three per-merge performance runs. But the performance run that occurred after merging included a roughly equal number of improvements and regressions, for unclear reasons.
This PR is for a test run reverting those changes, to see what happens.
r? `@ghost`
|
|
|
|
use precise spans for recursive const evaluation
This fixes https://github.com/rust-lang/rust/issues/73283 by using a `TyCtxtAt` with a more precise span when the interpreter recursively calls itself. Hopefully such calls are sufficiently rare that this does not cost us too much performance.
(In theory, cycles can also arise through layout computation, as layout can depend on consts -- but layout computation happens all the time so we'd have to do something to not make this terrible for performance.)
|
|
Make `Encodable` and `Encoder` infallible.
A follow-up to #93066.
r? `@ghost`
|
|
Rollup of 5 pull requests
Successful merges:
- #97595 (Remove unwrap from get_vtable)
- #97597 (Preserve unused pointer to address casts)
- #97819 (Recover `import` instead of `use` in item)
- #97823 (Recover missing comma after match arm)
- #97851 (Use repr(C) when depending on struct layout in ptr tests)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
|
|
This avoids the name clash with `rustc_serialize::Encoder` (a trait),
and allows lots qualifiers to be removed and imports to be simplified
(e.g. fewer `as` imports).
|
|
This commit makes type folding more like the way chalk does it.
Currently, `TypeFoldable` has `fold_with` and `super_fold_with` methods.
- `fold_with` is the standard entry point, and defaults to calling
`super_fold_with`.
- `super_fold_with` does the actual work of traversing a type.
- For a few types of interest (`Ty`, `Region`, etc.) `fold_with` instead
calls into a `TypeFolder`, which can then call back into
`super_fold_with`.
With the new approach, `TypeFoldable` has `fold_with` and
`TypeSuperFoldable` has `super_fold_with`.
- `fold_with` is still the standard entry point, *and* it does the
actual work of traversing a type, for all types except types of
interest.
- `super_fold_with` is only implemented for the types of interest.
Benefits of the new model.
- I find it easier to understand. The distinction between types of
interest and other types is clearer, and `super_fold_with` doesn't
exist for most types.
- With the current model is easy to get confused and implement a
`super_fold_with` method that should be left defaulted. (Some of the
precursor commits fixed such cases.)
- With the current model it's easy to call `super_fold_with` within
`TypeFolder` impls where `fold_with` should be called. The new
approach makes this mistake impossible, and this commit fixes a number
of such cases.
- It's potentially faster, because it avoids the `fold_with` ->
`super_fold_with` call in all cases except types of interest. A lot of
the time the compile would inline those away, but not necessarily
always.
|
|
Because `TypeFoldable::try_fold_mir_const` exists, and even though
`visit_mir_const` isn't needed right now, the consistency makes the code
easier to understand.
|
|
There are two impls of the `Encoder` trait: `opaque::Encoder` and
`opaque::FileEncoder`. The former encodes into memory and is infallible, the
latter writes to file and is fallible.
Currently, standard `Result`/`?`/`unwrap` error handling is used, but this is a
bit verbose and has non-trivial cost, which is annoying given how rare failures
are (especially in the infallible `opaque::Encoder` case).
This commit changes how `Encoder` fallibility is handled. All the `emit_*`
methods are now infallible. `opaque::Encoder` requires no great changes for
this. `opaque::FileEncoder` now implements a delayed error handling strategy.
If a failure occurs, it records this via the `res` field, and all subsequent
encoding operations are skipped if `res` indicates an error has occurred. Once
encoding is complete, the new `finish` method is called, which returns a
`Result`. In other words, there is now a single `Result`-producing method
instead of many of them.
This has very little effect on how any file errors are reported if
`opaque::FileEncoder` has any failures.
Much of this commit is boring mechanical changes, removing `Result` return
values and `?` or `unwrap` from expressions. The more interesting parts are as
follows.
- serialize.rs: The `Encoder` trait gains an `Ok` associated type. The
`into_inner` method is changed into `finish`, which returns
`Result<Vec<u8>, !>`.
- opaque.rs: The `FileEncoder` adopts the delayed error handling
strategy. Its `Ok` type is a `usize`, returning the number of bytes
written, replacing previous uses of `FileEncoder::position`.
- Various methods that take an encoder now consume it, rather than being
passed a mutable reference, e.g. `serialize_query_result_cache`.
|
|
|
|
Various refactors to the incr comp workproduct handling
This is the result of me looking into adding support for having multiple object files for a single codegen unit to incr comp. This is necessary to support inline assembly in cg_clif without requiring partial linking which is not supported on Windows and seems to fail on macOS for some reason. Cg_clif uses an external assembler to handle inline asm and thus produces one object file with regular functions and one object file containing compiled inline asm for each codegen unit which uses inline asm. Current incr comp can't handle this. This PR doesn't yet add support for this, but it makes it easier to do so.
|
|
interpret: better control over whether we read data with provenance
The resolution in https://github.com/rust-lang/unsafe-code-guidelines/issues/286 seems to be that when we load data at integer type, we implicitly strip provenance. So let's implement that in Miri at least for scalar loads. This makes use of the fact that `Scalar` layouts distinguish pointer-sized integers and pointers -- so I was expecting some wild bugs where layouts set this incorrectly, but so far that does not seem to happen.
This does not entirely implement the solution to https://github.com/rust-lang/unsafe-code-guidelines/issues/286; we still do the wrong thing for integers in larger types: we will `copy_op` them and then do validation, and validation will complain about the provenance. To fix that we need mutating validation; validation needs to strip the provenance rather than complaining about it. This is a larger undertaking (but will also help resolve https://github.com/rust-lang/miri/issues/845 since we can reset padding to `Uninit`).
The reason this is useful is that we can now implement `addr` as a `transmute` from a pointer to an integer, and actually get the desired behavior of stripping provenance without exposing it!
|
|
It returns the previous work product or panics if there is none. This rename
makes the purpose of this method clearer.
|
|
implicit provenance stripping where possible
|
|
Replace `&Vec<_>`s with `&[_]`s
It's generally preferable to use `&[_]` since it's one less indirection and it can be created from types other that `Vec`.
I've left `&Vec` in some locals where it doesn't really matter, in cases where `TypeFoldable` is expected (`TypeFoldable: Clone` so slice can't implement it) and in cases where it's `&TypeAliasThatIsActiallyVec`. Nothing important, really, I was just a little annoyed by `visit_generic_param_vec` :D
r? `@compiler-errors`
|
|
|
|
It doesn't do anything for all encoders
|
|
|
