Add documentation for generators

2 files changed, 320 insertions, 4 deletions

diff --git a/src/doc/unstable-book/src/language-features/generators.md b/src/doc/unstable-book/src/language-features/generators.md
new file mode 100644
index 00000000000..8d9d9846a23
--- /dev/null
+++ b/src/doc/unstable-book/src/language-features/generators.md
@@ -0,0 +1,232 @@
+# `generators`
+
+The tracking issue for this feature is: [#43122]
+
+[#34511]: https://github.com/rust-lang/rust/issues/43122
+
+------------------------
+
+The `generators` feature gate in Rust allows you to define generator or
+coroutine literals. A generator is a "resumable function" that syntactically
+resembles a closure but compiles to much different semantics in the compiler
+itself.
+
+Generators are an extra-unstable feature in the compiler right now. Added in
+[RFC 2033] they're mostly intended right now as a information/constraint
+gathering phase. The intent is that experimentation can happen on the nightly
+compiler before actual stabilization. A further RFC will be required to
+stabilize generators/coroutines and will likely contain at least a few small
+tweaks to the overall design.
+
+[RFC 2033]: https://github.com/rust-lang/rfcs/pull/2033
+
+A syntactical example of a generator is:
+
+```rust
+#![feature(generators, generator_trait)]
+
+use std::ops::{Generator, State};
+
+fn main() {
+    let mut generator = || {
+        yield 1;
+        return "foo"
+    };
+
+    match generator.resume(()) {
+        State::Yielded(1) => {}
+        _ => panic!("unexpected return from resume"),
+    }
+    match generator.resume(()) {
+        State::Complete("foo") => {}
+        _ => panic!("unexpected return from resume"),
+    }
+}
+```
+
+Generators are closure-like literals which can contain a `yield` statement. The
+`yield` statement takes an optional expression of a value to yield out of the
+generator. All generator literals implement the `Generator` trait in the
+`std::ops` module. The `Generator` trait has one main method, `resume`, which
+resumes execution of the closure at the previous suspension point.
+
+An example of the control flow of generators is that the following example
+prints all numbers in order:
+
+```rust
+#![feature(generators, generator_trait)]
+
+use std::ops::Generator;
+
+fn main() {
+    let mut generator = || {
+        println!("2");
+        yield;
+        println!("4");
+    };
+
+    println!("1");
+    drop(generator.resume(()));
+    println!("3");
+    drop(generator.resume(()));
+    println!("5");
+}
+```
+
+At this time the main intended use case of generators is an implementation
+primitive for async/await syntax, but generators will likely be extended to
+ergonomic implementations of iterators and other primitives in the future.
+Feedback on the design and usage is always appreciated!
+
+### The `Generator` trait
+
+The `Generator` trait in `std::ops` currently looks like:
+
+```
+pub trait Generator<Arg = ()> {
+    type Yield;
+    type Return;
+    fn resume(&mut self, arg: Arg) -> State<Self::Yield, Self::Return>;
+}
+```
+
+The `Generator::Yield` type is the type of values that can be yielded with the
+`yield` statement. The `Generator::Return` type is the returned type of the
+generator. This is typically the last expression in a generator's definition or
+any value passed to `return` in a generator. The `resume` function is the entry
+point for executing the `Generator` itself.
+
+The return value of `resume`, `State`, looks like:
+
+```
+pub enum State<Y, R> {
+    Yielded(Y),
+    Complete(R),
+}
+```
+
+The `Yielded` variant indicates that the generator can later be resumed. This
+corresponds to a `yield` point in a generator. The `Complete` variant indicates
+that the generator is complete and cannot be resumed again. Calling `resume`
+after a generator has returned `Complete` will likely result in a panic of the
+program.
+
+### Closure-like semantics
+
+The closure-like syntax for generators alludes to the fact that they also have
+closure-like semantics. Namely:
+
+* When created, a generator executes no code. A closure literal does not
+  actually execute any of the closure's code on construction, and similarly a
+  generator literal does not execute any code inside the generator when
+  constructed.
+
+* Generators can capture outer variables by reference or by move, and this can
+  be tweaked with the `move` keyword at the beginning of the closure. Like
+  closures all generators will have an implicit environment which is inferred by
+  the compiler. Outer variables can be moved into a generator for use as the
+  generator progresses.
+
+* Generator literals produce a value with a unique type which implements the
+  `std::ops::Generator` trait. This allows actual execution of the genrator
+  through the `Generator::resume` method as well as also naming it in return
+  types and such.
+
+* Traits like `Send` and `Sync` are automatically implemented for a `Generator`
+  depending on the captured variables of the environment. Note, though, that
+  generators, like closures, do not implement traits like `Copy` or `Clone`
+  automatically.
+
+* Whenever a generator is dropped it will drop all captured environment
+  variables.
+
+### Generators as state machines
+
+In the compiler generators are currently compiled as state machines. Each
+`yield` expression will correspond to a different state that stores all live
+variables over that suspension point. Resumption of a generator will dispatch on
+the current state and then execute internally until a `yield` is reached, at
+which point all state is saved off in the generator and a value is returned.
+
+Let's take a look at an example to see what's going on here:
+
+```rust
+#![feature(generators, generator_trait)]
+
+use std::ops::Generator;
+
+fn main() {
+    let ret = "foo";
+    let mut generator = move || {
+        yield 1;
+        return ret
+    };
+
+    drop(generator.resume(()))
+    drop(generator.resume(()))
+}
+```
+
+This generator literal will compile down to something similar to:
+
+```rust
+#![feature(generators, generator_trait)]
+
+use std::ops::{Generator, State};
+
+fn main() {
+    let ret = "foo";
+    let mut generator = {
+        enum __Generator {
+            Start(&'static str),
+            Yield1(&'static str),
+            Done,
+        }
+
+        impl Generator for __Generator {
+            type Yield = i32;
+            type Return = &'static str;
+
+            fn resume(&mut self, arg: ()) -> State<i32, &'static str> {
+                use std::mem;
+                match mem::replace(self, __Generator::Done) {
+                    __Generator::Start(s) => {
+                        *self = __Generator::Yield1(s);
+                        State::Yielded(1)
+                    }
+
+                    __Generator::Yield1(s) => {
+                        *self = __Generator::Done;
+                        State::Complete(s)
+                    }
+
+                    __Generator::Done => {
+                        panic!("generator resumed after completion")
+                    }
+                }
+            }
+        }
+
+        __Generator::Start(ret) : impl Generator<Yield = i32, Return = &'static str>
+    };
+
+    drop(generator.resume(()))
+    drop(generator.resume(()))
+}
+```
+
+Notably here we can see that the compiler is generating a fresh type,
+`__Generator` in this case. This type has a number of states (represented here
+as an `enum`) corresponding to each of the conceptual states of the generator.
+At the beginning we're closing over our outer variable `foo` and then that
+variable is also live over the `yield` point, so it's stored in both states.
+
+When the generator starts it'll immediately yield 1, but it saves off its state
+just before it does so indicating that it has reached the yield point. Upon
+resuming again we'll execute the `return ret` which returns the `Complete`
+state.
+
+Here we can also note that the `Done` state, if resumed, panics immediately as
+it's invalid to resume a completed generator. It's also worth noting that this
+is just a rough desugaring, not a normative specification for what the compiler
+does.
diff --git a/src/libcore/ops/generator.rs b/src/libcore/ops/generator.rs
index 1e54eaaaa28..4d125573afc 100644
--- a/src/libcore/ops/generator.rs
+++ b/src/libcore/ops/generator.rs
@@ -9,33 +9,117 @@
 // except according to those terms.
 
 /// The result of a generator resumption.
+///
+/// This enum is returned from the `Generator::resume` method and indicates the
+/// possible return values of a generator. Currently this corresponds to either
+/// a suspension point (`Yielded`) or a termination point (`Complete`).
 #[derive(Debug)]
 #[cfg_attr(not(stage0), lang = "generator_state")]
-#[unstable(feature = "generator_trait", issue = "0")]
+#[unstable(feature = "generator_trait", issue = "43122")]
 pub enum State<Y, R> {
     /// The generator suspended with a value.
+    ///
+    /// This state indicates that a generator has been suspended, and typically
+    /// corresponds to a `yield` statement. The value provided in this variant
+    /// corresponds to the expression passed to `yield` and allows generators to
+    /// provide a value each time they yield.
     Yielded(Y),
 
     /// The generator completed with a return value.
+    ///
+    /// This state indicates that a generator has finished execution with the
+    /// provided value. Once a generator has returned `Complete` it is
+    /// considered a programmer error to call `resume` again.
     Complete(R),
 }
 
 /// The trait implemented by builtin generator types.
+///
+/// Generators, also commonly referred to as coroutines, are currently an
+/// experimental language feature in Rust. Added in [RFC 2033] generators are
+/// currently intended to primarily provide a building block for async/await
+/// syntax but will likely extend to also providing an ergonomic definition for
+/// iterators and other primitives.
+///
+/// The syntax and semantics for generators is unstable and will require a
+/// further RFC for stabilization. At this time, though, the syntax is
+/// closure-like:
+///
+/// ```rust
+/// #![feature(generators, generator_trait)]
+///
+/// use std::ops::{Generator, State};
+///
+/// fn main() {
+///     let mut generator = || {
+///         yield 1;
+///         return "foo"
+///     };
+///
+///     match generator.resume(()) {
+///         State::Yielded(1) => {}
+///         _ => panic!("unexpected return from resume"),
+///     }
+///     match generator.resume(()) {
+///         State::Complete("foo") => {}
+///         _ => panic!("unexpected return from resume"),
+///     }
+/// }
+/// ```
+///
+/// More documentation of generators can be found in the unstable book.
+///
+/// [RFC 2033]: https://github.com/rust-lang/rfcs/pull/2033
 #[cfg_attr(not(stage0), lang = "generator")]
-#[unstable(feature = "generator_trait", issue = "0")]
+#[unstable(feature = "generator_trait", issue = "43122")]
 #[fundamental]
 pub trait Generator<Arg = ()> {
     /// The type of value this generator yields.
+    ///
+    /// This associated type corresponds to the `yield` expression and the
+    /// values which are allowed to be returned each time a generator yields.
+    /// For example an iterator-as-a-generator would likely have this type as
+    /// `T`, the type being iterated over.
     type Yield;
 
     /// The type of value this generator returns.
+    ///
+    /// This corresponds to the type returned from a generator either with a
+    /// `return` statement or implicitly as the last expression of a generator
+    /// literal. For example futures would use this as `Result<T, E>` as it
+    /// represents a completed future.
     type Return;
 
-    /// This resumes the execution of the generator.
+    /// Resumes the execution of this generator.
+    ///
+    /// This function will resume execution of the generator or start execution
+    /// if it hasn't already. This call will return back into the generator's
+    /// last suspension point, resuming execution from the latest `yield`. The
+    /// generator will continue executing until it either yields or returns, at
+    /// which point this function will return.
+    ///
+    /// # Return value
+    ///
+    /// The `State` enum returned from this function indicates what state the
+    /// generator is in upon returning. If the `Yielded` variant is returned
+    /// then the generator has reached a suspension point and a value has been
+    /// yielded out. Generators in this state are available for resumption at a
+    /// later point.
+    ///
+    /// If `Complete` is returned then the generator has completely finished
+    /// with the value provided. It is invalid for the generator to be resumed
+    /// again.
+    ///
+    /// # Panics
+    ///
+    /// This function may panic if it is called after the `Complete` variant has
+    /// been returned previously. While generator literals in the language are
+    /// guaranteed to panic on resuming after `Complete`, this is not guaranteed
+    /// for all implementations of the `Generator` trait.
     fn resume(&mut self, arg: Arg) -> State<Self::Yield, Self::Return>;
 }
 
-#[unstable(feature = "generator_trait", issue = "0")]
+#[unstable(feature = "generator_trait", issue = "43122")]
 impl<'a, T, U> Generator<U> for &'a mut T
     where T: Generator<U> + ?Sized
 {