Note: In the middle of writing this question, I realized the specific example I give below doesn't actually work perfectly with the title, but I couldn't think of another way of putting it, and I think it is still a good description for my general issue, so I've left it as is for now.
I'm currently working on an internal component for a project of mine, and have run into a fairly opaque compiler error I've been trying to solve for the last day or so without much success. The original code was quite long, so I've spent a little while trying to create a minimal example that shows the issue. This example is quite divorced from my actual use-case, so please do not comment on its specific structure; it is meant to be purely illustrative.
My original code centered around iterators. Trait, GetAssoc, and Ref are meant to capture the essence of that API. Namely, they correspond roughly to Iterator, IntoIterator, and slice::Iter, respectively, while Struct is meant to represent something like a Vec. Try not to pay much attention to these; they exist purely to make the overall span of code being investigated smaller.
#![allow(unused)]
use std::marker::PhantomData;
trait Trait {} // Iterator
trait GetAssoc { // IntoIterator
type Assoc: Trait;
fn get(self) -> Self::Assoc;
}
struct Struct {} // Vec
struct Ref<'a> { _phantom: PhantomData<&'a ()> } // slice::Iter
impl<'a> Trait for Ref<'a> {}
impl<'a> GetAssoc for &'a Struct {
type Assoc = Ref<'a>;
fn get(self) -> Self::Assoc { Ref { _phantom: PhantomData } }
}
// *ACTUAL CODE STARTS BELOW HERE*
struct Wrapper<'a, T: 'a, F: Fn(&'a T) -> Ret, Ret: Trait> {
value: T,
map: F,
_phantom: PhantomData<(&'a (), Ret)>
}
impl<'a, T: 'a, F: Fn(&'a T) -> Ret, Ret: Trait> Wrapper<'a, T, F, Ret> {
fn transform(&'a self) -> Ret {
(self.map)(&self.value)
}
}
fn main() {
let obj = Struct {};
let wrapper = Wrapper { value: obj, map: |x| x.get(), _phantom: PhantomData };
{ wrapper.transform(); }
let _a = wrapper;
}
Roughly: I am wrapping a value and a closure together (Wrapper), then invoking that closure in a method (Wrapper::transform). The example exactly as given here results in cannot move out of `wrapper` because it is borrowed on rustc 1.86, at let _a = wrapper; (this line exists just to demonstrate the issue). In earlier, more faithful versions of this example, I could replace the transform call with { (wrapper.map)(&wrapper.value); } (just expanding it out directly) to make this compile. However this unfortunately doesn't work here.
If you replace the closure passed to wrapper.map with a function fn map(x: &Struct) -> Ref { x.get() }, it works fine. If you try to replicate that signature on the closure (|x: &Struct| -> Ref { x.get() }), it fails (in some versions of the test this also worked). I cannot use impl Trait return types in the closure since it is currently not allowed, and using it in the function causes it to fail again.
My problem here is that while writing out the full map signature in a function here is trivial, it is anything but in my actual use-case. The real return type is a fairly large chain of iterator adaptors (some of which exist purely to work around other issues in the type-system), and I end up having to write it out in full, as attempting to leave any to be inferred runs into trait solver limitations (this example cannot reproduce that).
If I remove the explicit lifetimes on Wrapper and its impl, I run into
map: |x| x.get(),
-- ^^^^^^^ returning this value requires that `'1` must outlive `'2`
Using the function map replacement then gives
{ wrapper.transform(); }
^^^^^^^^^ method not found in `Wrapper<Struct, fn(&Struct) -> Ref<'a> {map},
even if I explicitly annotate it, which feels a bit insane considering the impl has the exact same signature as the type itself. Expanding the call to transform gives this wonderful little error :)
expected struct `Ref<'_>`
found struct `Ref<'_>`
There are other possible solutions to my original issue, like using a trait on the wrapped type instead of a closure (except for me this also runs into trait solver limitations, and is fairly un-ergonomic to use), or just throwing away the generics entirely and creating a type only for my specific case, but it just seems ridiculous that I can't get this to work given how simple I feel it should be.
Is there any way I can write either Wrapper or the map closure to make this compile without needing to write out the return types (if you can get an impl Trait to work that's fine) in full? I'd prefer to keep the closure and a completely inferred return type as the actual closure code is fairly small, but would accept needing to use functions. Moreover, is this somehow intended behavior, or just a limitation in rustc?
I am unwilling to accept a solution that uses unstable features (though of course you are welcome to mention them), unless it is evident that stabilization is immanent (ie could be reasonably expected by rustc 1.88-1.89).
The basic problem here is that you're trying to do two different things with the 'a lifetime on Wrapper:
map; andtransform.(You have an 'a in both places: F: Fn(&'a T) -> Ret hardcodes 'a allowing Ret to also have it hardcoded; and fn transform(&'a self) has a &'a on the self parameter.)
As such, you're attempting to call a method whose receiver type is &'a Wrapper<'a, _, _, _> – you have a reference to a type that lasts as long as the type itself does (because Wrapper has to live at least as long as 'a due to containing an &'a (), but 'a has to live at least as long as the referenced Wrapper so that the &'a Wrapper remains valid). In other words, this means that as long as your code contains a call to transform, Wrapper is permanently borrowed – the borrow can't possibly end because (by definition) it lasts as long as the object being borrowed does (they have the same lifetime!).
Rust does allow you to create a permanent borrow, but once you do, you can never again create a conflicting borrow of the object, nor move it. So you're getting an error because you're attempting to move a permanently borrowed object. (It would be nice if Rust's error messages were clearer in this situation: this sort of thing is easy to do by mistake and it can be quite hard to work out what's going on.) The closure doesn't directly have anything to do with the issue; the type of transform is enough to cause the problem. (That said, the closure is indirectly causing the issue because you wouldn't have had to write the type of transform like that otherwise.)
The basic issue behind what's going on is that your code is attempting to implement F for one particular lifetime of the argument to the called closure – it has a hardcoded &'a T argument type, so it only allows the use of arguments with lifetime 'a, so the argument always has to have the same lifetime. That implies that any given Wrapper object is only usable once (which is why you ended up writing it with the permanent borrow – because, given the type of F, transform fundamentally can't be written to work more than once, the code naturally ended up being written in a state where each Wrapper could only be used once).
Instead, F is, in this case, inherently a lifetime-generic function; you want it to be callable with a reference with any lifetime 'b, and to return a return value with the same lifetime 'b. Syntactically, that would look like this:
F: for<'b> Fn(&'b T) -> Ret<'b>
…which would be fine if Ret were a concrete type constructor. But it isn't – it's a type parameter of Wrapper, so you've basically ended up designing a generic wrapper type where one of its generic parameters is not a type, but rather a generic type constructor. These are normally known as higher-kinded types, and although they exist in some programming languages, they don't exist in Rust. (As seen here, there are situations where they're potentially useful; however, they can also be quite confusing to work with and would likely be difficult to implement in the type-checker. So it's unlikely they'll be implemented in Rust any time soon, and the Rust developers are more likely to work on alternatives than on implementing higher-kinded types directly.)
If you're OK with writing out the complicated type in question once and it's always the same, the simplest approach is to use a type alias:
type Ret<'b> = Ref<'b>; // big complicated right hand side
struct Wrapper<T, F: for<'b> Fn(&'b T) -> Ret<'b>> {
value: T,
map: F,
}
impl<T, F: for<'b> Fn(&'b T) -> Ret<'b>> Wrapper<T, F> {
fn transform(&self) -> Ret {
(self.map)(&self.value)
}
}
fn main() {
let obj = Struct {};
let wrapper = Wrapper { value: obj, map: |x| x.get() };
{ wrapper.transform(); }
let _a = wrapper;
}
This way, Ret<'b> is always the same type apart from the lifetime 'b, so Wrapper doesn't have to be generic over it – and the lifetime 'a has basically disappeared altogether, so you don't get any issues with Wrapper becoming permanently borrowed. (The lifetime that was 'a is still present in transform as an elided lifetime, with the signature being sugar for fn transform<'a>(&'a self) -> Ret<'a>, but is no longer in Wrapper.) This solution is easy to understand and works on stable Rust, but isn't quite as general as the original code. I would recommend using it if it works in your case, because it makes it clear what's happening to both the user and compiler.
Generic associated types are a language feature which is able to replace most uses of higher-kinded types. However, it's generally not very usable, and generally works better in the internals of libraries than it does as something for an end-user to write directly (it exists primarily because some programs can't be written at all without it).
It's not too bad to write Wrapper in terms of a generic associated type:
trait FnReturningTrait<T> {
type Ret<'b>: Trait where Self: 'b, T: 'b;
fn call<'b>(&'b self, arg: &'b T) -> Self::Ret<'b>;
}
struct Wrapper<T, F: FnReturningTrait<T>> {
value: T,
map: F,
}
impl<T, F: FnReturningTrait<T>> Wrapper<T, F> {
fn transform(&self) -> <F as FnReturningTrait<T>>::Ret<'_> {
self.map.call(&self.value)
}
}
The idea here is that F of Wrapper is a trait that supports the operation we want: taking a *'b T as argument, and returning a Trait whose lifetime depends on 'b as its return value. Implementing transform here is easy; the trait FnReturningTrait is unfortunately a bit complex, though, because there are a lot of lifetime annotations.
The problem comes when it comes to actually using Wrapper, because closures don't implement FnReturningTrait and I couldn't find a way to blanket-implement it even using unstable features. So you have to define the closure by hand in main:
struct GetAssocClosure;
impl<T> FnReturningTrait<T> for GetAssocClosure
where for<'b> &'b T: GetAssoc {
type Ret<'b> = <&'b T as GetAssoc>::Assoc
where Self: 'b, T: 'b;
fn call<'b>(&'b self, arg: &'b T) -> Self::Ret<'b> {
arg.get()
}
}
let wrapper = Wrapper { value: obj, map: GetAssocClosure };
This isn't actually difficult. But it's also very verbose and not very easy to read, because you can't use the normal closure syntax. As such, I'd reserve this technique for situations where there's no other way to implement what you want.
Although the technique in the next section is generally more usable, this one works on the most versions of Rust (back to Rust 1.65) and is fully general.
impl Trait in trait (Rust 1.75+)The previous solution is somewhat unwieldy when it comes to defining the traits. When you don't need to be able to name the return type, it recently became possible to sugar the generic associated type into a return-position impl Trait. That makes it possible to write what is effectively the previous example, but in a somewhat less ugly way:
trait FnReturningTrait<T> {
fn call<'b>(&'b self, arg: &'b T) -> impl Trait;
}
struct Wrapper<T, F: FnReturningTrait<T>> {
value: T,
map: F,
}
impl<T, F: FnReturningTrait<T>> Wrapper<T, F> {
fn transform(&self) -> impl Trait { // Rust 2024 edition
self.map.call(&self.value)
}
}
fn main() {
let obj = Struct {};
struct GetAssocClosure;
impl<T> FnReturningTrait<T> for GetAssocClosure
where for<'b> &'b T: GetAssoc {
fn call<'b>(&'b self, arg: &'b T) -> impl Trait {
arg.get()
}
}
let wrapper = Wrapper { value: obj, map: GetAssocClosure };
{ wrapper.transform(); }
let _a = wrapper;
}
The main difference is that because Ret is no longer named – it's just used as an opaque return value – if you're using Rust 2024 edition, an explicit listing of all the lifetime constraints on it is no longer needed.
This technique can be made to work back as far as Rust 1.75 and in previous editions, but the syntax for declaring and defining transform varies:
fn transform(&self) -> impl Trait; // Rust 2024 edition
fn transform(&self) -> impl Trait + use<'_, F, T> // Rust 1.82+, any edition
fn transform(&self) -> impl Trait + '_; // Rust 1.75+, somewhat hacky
The first two possibilities say that the lifetime of the return value may be based on the lifetime of self, which is the correct behaviour for the function. This was made the default in Rust 2024 edition (and has always been the default for a return-position impl Trait in a trait, as opposed to an associated function like transform is).
The third possibility says that the return value lasts at least as long as self, which isn't actually what you want, but is close enough to work in this case (it's known as the "outlives trick") – it works by interacting with the way the lifetime erasure algorithm in Rust 2021 and earlier chooses default lifetimes, but adds an extra constraint (which fortunately happens to be true in this case, but isn't always true in cases where multiple lifetimes are involved).
The main disadvantage of this is that closures only implement Fn traits – they don't implement custom traits that use the impl Trait syntax – so you have to implement the closure by hand rather than using closure syntax.
I couldn't find an exact solution using unstable features, but I got very close using unboxed_closures:
#![feature(unboxed_closures)]
struct Wrapper<T, F> {
value: T,
map: F,
}
impl<T, F: for<'b> Fn<(&'b T,), Output: Trait>> Wrapper<T, F> {
fn transform(&self) -> impl Trait {
(self.map)(&self.value)
}
}
fn main() {
let obj = Struct {};
fn get_assoc_closure<'b>(s: &'b Struct) -> impl Trait {
s.get()
}
let wrapper = Wrapper { value: obj, map: get_assoc_closure };
// let wrapper = Wrapper { value: obj, map: |x: &Struct| x.get() };
{ wrapper.transform(); }
let _a = wrapper;
}
The idea is to simulate impl Trait in a Fn by placing a requirement on the output of the Fn, which is the behaviour you'd intuitively expect for impl Trait on the return position of a Fn. This actually does work in the case where the Fn is a function, and has its argument and return type listed. The unboxed_closures feature is required in order to be able to name the parts of the Fn trait indivdually (which isn't possible in stable Rust).
Unfortunately, the closure version (with |x: &Struct| x.get()) doesn't work and I couldn't find a way to make it work – that looks like it may actually be a closure inference bug, given that it works when the closure is converted into a function.