Logo
Logo

Atharva Pandey/Lesson 17: Pin in Async Context — Why futures must be pinned

Created Thu, 06 Feb 2025 08:27:44 +0000 Modified Thu, 06 Feb 2025 08:27:44 +0000

I’ve been hand-waving around Pin for 16 lessons. Every time we saw Pin<&mut Self> in a poll method, I said “don’t worry about it.” But now it’s time to worry about it, because without Pin, async Rust’s entire safety model falls apart.

The good news: the mental model is simpler than it looks. The bad news: the syntax is still ugly. Let’s deal with both.

The Problem Pin Solves

When the compiler turns your async fn into a state machine, it needs to store local variables across .await points. Sometimes those variables reference each other:

async fn problem() {
    let data = vec![1, 2, 3];
    let reference = &data[0]; // Points to data's buffer

    some_async_op().await; // <-- State machine pauses here

    println!("{reference}"); // Uses the reference after resuming
}

async fn some_async_op() {
    tokio::time::sleep(tokio::time::Duration::from_millis(1)).await;
}

The compiler generates a struct like:

struct ProblemFuture {
    data: Vec<i32>,
    reference: *const i32, // Points into data!
    state: State,
}

This is a self-referential structreference points into data, which is in the same struct. If you move this struct in memory (like pushing it into a Vec or returning it from a function), reference becomes a dangling pointer.

Pin prevents the struct from being moved after it’s been polled. That’s the entire purpose.

What Pin Actually Is

Pin<P> is a wrapper around a pointer P that prevents the pointed-to value from being moved:

// Pin<&mut T> means: you have a mutable reference to T,
// but you promise not to move the T out of this reference.

// Pin<Box<T>> means: T is on the heap, and you promise
// not to move it to a different memory location.

The key insight: Pin doesn’t prevent access to the value. You can still read and modify it. It just prevents moving it.

Unpin: The Opt-Out

Most types in Rust are Unpin — they don’t care about being moved. Numbers, strings, vecs, hashmaps — all Unpin. For these types, Pin is a no-op. You can freely move them even when pinned.

use std::pin::Pin;

fn demo_unpin() {
    let mut x = 42;
    let pinned = Pin::new(&mut x); // This works because i32: Unpin

    // We can get a regular &mut back because i32 is Unpin
    let regular_ref = Pin::into_inner(pinned);
    *regular_ref = 43;
}

Async futures generated by the compiler are !Unpin (not Unpin) when they contain self-references across .await points. That’s when Pin becomes meaningful.

Pin in Practice

Pinning on the heap (most common)

use std::pin::Pin;
use std::future::Future;

async fn example() -> i32 {
    tokio::time::sleep(tokio::time::Duration::from_millis(1)).await;
    42
}

#[tokio::main]
async fn main() {
    // Box::pin puts the future on the heap and pins it there
    let future: Pin<Box<dyn Future<Output = i32>>> = Box::pin(example());

    let result = future.await;
    println!("{result}");
}

Box::pin is the most common way to pin a future. It heap-allocates the future and returns a Pin<Box<...>>. The future can’t move because it’s on the heap and the Pin prevents you from taking it out.

Pinning on the stack

use tokio::time::{sleep, Duration};

#[tokio::main]
async fn main() {
    let future = sleep(Duration::from_millis(100));

    // pin! macro pins the future on the stack
    tokio::pin!(future);

    // Now we can poll it or use it in select!
    future.await;
    println!("Done!");
}

tokio::pin! (or std::pin::pin! in Rust 1.68+) shadows the variable with a pinned version. The original future can’t be accessed anymore — only the pinned reference.

When you need Pin: storing futures in collections

use std::future::Future;
use std::pin::Pin;
use tokio::time::{sleep, Duration};

#[tokio::main]
async fn main() {
    // You can't put `impl Future` in a Vec — you need trait objects
    // And trait object futures need to be pinned
    let mut futures: Vec<Pin<Box<dyn Future<Output = String> + Send>>> = vec![];

    for i in 0..5 {
        futures.push(Box::pin(async move {
            sleep(Duration::from_millis(100 * i)).await;
            format!("future-{i}")
        }));
    }

    // Process them with futures::future::join_all
    let results = futures::future::join_all(futures).await;
    println!("{results:?}");
}

When you need Pin: returning different futures

use std::future::Future;
use std::pin::Pin;
use tokio::time::{sleep, Duration};

fn make_future(fast: bool) -> Pin<Box<dyn Future<Output = String> + Send>> {
    if fast {
        Box::pin(async {
            sleep(Duration::from_millis(10)).await;
            "fast".to_string()
        })
    } else {
        Box::pin(async {
            sleep(Duration::from_millis(1000)).await;
            "slow".to_string()
        })
    }
}

#[tokio::main]
async fn main() {
    let result = make_future(true).await;
    println!("{result}");
}

Without Pin<Box<dyn Future>>, you can’t return different future types from the same function because they have different concrete types.

Implementing Future with Pin

When you implement Future manually, you need to handle Pin:

use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::time::{Sleep, sleep, Duration};

struct RetryFuture<F, Fut>
where
    F: Fn() -> Fut,
    Fut: Future<Output = Result<String, String>>,
{
    factory: F,
    current: Option<Pin<Box<Fut>>>,
    retries_left: u32,
}

impl<F, Fut> RetryFuture<F, Fut>
where
    F: Fn() -> Fut,
    Fut: Future<Output = Result<String, String>>,
{
    fn new(factory: F, max_retries: u32) -> Self {
        RetryFuture {
            factory,
            current: None,
            retries_left: max_retries,
        }
    }
}

impl<F, Fut> Future for RetryFuture<F, Fut>
where
    F: Fn() -> Fut + Unpin,
    Fut: Future<Output = Result<String, String>>,
{
    type Output = Result<String, String>;

    fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        loop {
            // Start a new attempt if needed
            if self.current.is_none() {
                let fut = (self.factory)();
                self.current = Some(Box::pin(fut));
            }

            // Poll the current attempt
            let current = self.current.as_mut().unwrap();
            match current.as_mut().poll(cx) {
                Poll::Ready(Ok(val)) => return Poll::Ready(Ok(val)),
                Poll::Ready(Err(e)) => {
                    if self.retries_left > 0 {
                        self.retries_left -= 1;
                        self.current = None; // Will create new attempt next loop
                        println!("Retry ({} left): {e}", self.retries_left);
                        continue;
                    }
                    return Poll::Ready(Err(e));
                }
                Poll::Pending => return Poll::Pending,
            }
        }
    }
}

use std::sync::atomic::{AtomicU32, Ordering};

#[tokio::main]
async fn main() {
    let attempt = AtomicU32::new(0);

    let result = RetryFuture::new(
        || {
            let n = attempt.fetch_add(1, Ordering::Relaxed);
            async move {
                sleep(Duration::from_millis(50)).await;
                if n < 2 {
                    Err(format!("attempt {n} failed"))
                } else {
                    Ok(format!("attempt {n} succeeded"))
                }
            }
        },
        5,
    ).await;

    println!("Result: {result:?}");
}

Notice how the inner future is stored as Pin<Box<Fut>>. We can’t store it as just Fut because we need to poll it through a pinned reference.

pin_project: The Ergonomic Helper

The pin-project crate makes working with pinned fields much less painful:

[dependencies]
pin-project = "1"
use pin_project::pin_project;
use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::time::{Sleep, sleep, Duration};

#[pin_project]
struct Timed<F> {
    #[pin]
    inner: F,
    start: Option<std::time::Instant>,
    label: String,
}

impl<F: Future> Future for Timed<F> {
    type Output = F::Output;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<F::Output> {
        let this = self.project(); // Gives us pinned access to fields

        if this.start.is_none() {
            *this.start = Some(std::time::Instant::now());
        }

        match this.inner.poll(cx) {
            Poll::Ready(val) => {
                let elapsed = this.start.unwrap().elapsed();
                println!("[{}] completed in {:?}", this.label, elapsed);
                Poll::Ready(val)
            }
            Poll::Pending => Poll::Pending,
        }
    }
}

fn timed<F: Future>(label: &str, future: F) -> Timed<F> {
    Timed {
        inner: future,
        start: None,
        label: label.to_string(),
    }
}

#[tokio::main]
async fn main() {
    let result = timed("my-operation", async {
        sleep(Duration::from_millis(150)).await;
        42
    }).await;

    println!("Got: {result}");
}

The #[pin] attribute on inner means it’ll be projected as Pin<&mut F>. Fields without #[pin] are projected as &mut T. The project() method gives you safe access to both.

Common Pin Mistakes

Mistake 1: Trying to move a pinned value

// This won't compile:
// let pinned = Box::pin(42);
// let moved = *pinned; // Error: can't move out of Pin

Mistake 2: Forgetting to pin before polling

use tokio::time::{sleep, Duration};

#[tokio::main]
async fn main() {
    let fut = sleep(Duration::from_millis(100));

    // To use in select!, the future needs to be pinned
    tokio::pin!(fut);

    tokio::select! {
        _ = &mut fut => println!("Timer done"),
        _ = sleep(Duration::from_secs(1)) => println!("Fallback"),
    }
}

Mistake 3: Not understanding that most types are Unpin

// For Unpin types, Pin is essentially a no-op.
// Don't over-complicate things when you don't need to.

// This is fine — String is Unpin:
let mut s = String::from("hello");
let _pinned = Pin::new(&mut s);

// You only need to worry about Pin with:
// 1. Compiler-generated futures (from async fn)
// 2. Manual Future implementations with self-references
// 3. Types that explicitly opt out of Unpin

The Mental Model

Think of it this way:

  1. async fn generates a state machine struct that might contain self-references
  2. Self-referential structs can’t be moved without invalidating the references
  3. Pin is a contract that says “I won’t move this value”
  4. Unpin is an escape hatch that says “moving me is fine, Pin doesn’t matter”
  5. Box::pin() and tokio::pin!() are how you create pinned values in practice

In day-to-day async Rust, you’ll encounter Pin in three situations:

  • Returning Pin<Box<dyn Future>> from functions
  • Using tokio::pin!() in select! loops
  • Implementing Future manually

For everything else, the compiler handles it for you. async fn and .await hide all the pinning mechanics. That’s the whole point of the syntax sugar.

Next lesson: tracing async code — because when you have thousands of concurrent tasks, println! debugging doesn’t scale.