The Borrow Checker

Introduction

So I've been playing around with Rust for maybe 12 months now on and off and one of the things I've learned is I love Rust. Whether I can even justify Rust as something I would happily use for production stuff in my day to day job I'm not entirely convinced yet (I think it's real close though). But I'd really love to one day, I mean really I would love to.

So why is Rust great and worth your time to learn it. Well, recently MicroSoft said they're interested in it (see here), it's used behind the scenes in places in both AWS and Azure (see this link for AWS and this link for Azure), it can do green threading (https://tokio.rs/) and the performance is incredible (comparable to C/C++, from various internet sources and experience). Essentially it's worthy of several articles in itself, google "why do developers love ruse".

But why do I say I can't justify moving away from higher level languages like Java, JavaScript or Python. A few reasons:

• No pulling punches, it's hard to get started. One problem is can we actually find developers to develop and maintain existing Rust without paying a fortune because they're scarce. I'm hoping in time there will be more.
• Unit testing isn't great in my view. They probably haven't concentrated on this much as the philosophy seems to be "if it compiles, it probably works". Honestly, even if that's true (which it isn't, even for Rust) I still want to unit test properly. It's not dreadful, it can do it, but when I compare it to working with JavaScript or Python... nope, not up to scratch yet. Mocking for example, I'd love to see a decent mocking crate come out (ping me if I missed a really obvious one) that will mock out whatever I want it to or maybe something in the compiler itself that works for tests only.
• For cloud technologies we're in infancy here really. I haven't played with this much but a quick google suggests this to be the case. Whilst trying to get Java/Node/Python stuff running on AWS or Azure there's tonnes of stuff around (often created by the vendors), but Rust, it's just not on peoples radar enough yet. Again I hope and think it will be more so in the next few years.
• Green threading needs work still. Comparing this with Go here which, whilst I strongly prefer Rust overall, gives me super nice green threading, Rust is harder to achieve the same thing with. I kinda prefer that Rust doesn't have this kind of thing coded into its runtime and Rust can do this with the Tokio crate which is great, but it's hard and it's in its infancy. I'd still recommend using Tokio and there is some serious stuff written using it, but for the faint hearted it is not. I believe this will improve dramatically in time, not even necessarily that long from now and it's not really a fair complaint as the core team are just getting futures and async/await sorted out properly now, still I want it. Normal threading however is totally there so this is not the end of the world, I could live with this.

So I was thinking to myself what can I do to help with the above problems. I mean I wanna use this thing, it's great. For any low level systems programming stuff I'd use it in a heartbeat but I wanna use it day to day. Well for some of these problems maybe I can help with by writing a series of articles as tutorials on the challenging parts of learning Rust. Now there are loads of articles about this kind of thing and I'll reference this truly excellent series here but this references Haskell similarities too much for my liking. I don't promise mine is really any better but I'm trying to aim it at a slightly lower level, ideally I'd like to aim this at those without programming experience but reasonably it's unlikely without a bit of experience, however I'd like to think non-experts can gain something from these articles. Knowing roughly what a function, module, string, integer and memory is in some languages (not necessarily Rust) will be assumed, but just a basic grasp should suffice for this article to make sense (reading the Rust book afterwards may be a different story). I'm hoping because I'm reasonably new I can remember the things I struggled with and how I got over them (I think anyway). Please, please give feedback and let me know if anything is still confusing, I'll try to keep improving where I can.

So what I think makes Rust initially really hard is the borrow checker and memory management so that will be the subject of this first article. This is super unfortunate in a lot of ways as the borrow checker is the best thing about Rust, my advice to those new to Rust is don't be afraid of it. It loves you, it wants you to write great bug free code, it wants to help you. Whilst Rust has a difficult and steep learning curve it gives super helpful compile time errors (as I'll try and persuade in this article). I've spoken to people who've given up on Rust because of the borrow checker, if that's true then this article is for you, please give it another go after reading this and if you still don't understand something please ask me afterwards.

I have also noticed that reading up about Rust on the web as well, it's way better than it used to be. It does have this reputation of being super hard to use still but the Rust community is great, they listen to problems and they try and make it better and they have, see this article on non-lexical lifetimes for example. Whilst green threading isn't quite as good as I'd like yet, I know they're working super hard on it to improve things and before long I'm pretty sure it will be.

This will be a series of articles about the difficult parts of Rust but it is not meant to be exhaustive, I'm not trying to replace the Rust book and I never will, the Rust book is great but it's a better reference than it is a tutorial. I'd strongly recommend you to try and get to grips with the Rust book too, you'll need it and you'll want to use it as a reference.

If you're scared of Rust now after reading this intro and I scared you off, I'm really sorry. Please do give it a go still, I'm trying to prepare you for a challenge rather than scare you off. I will try my best to hold your hand (metaphorically of course) and guide you through the initial stages as best as I can. There is nothing in this article that is rocket science or even that difficult, I think it's more the amount you need to know that causes problems.

At the end of the day though, if you're after a nice easy programming language, Rust just isn't it. Rust I believe is a strong competitor for the lower level C/C++'s of the world and for complex programs in higher level languages like Python or Java. If you want a quick simple program and/or rapid prototyping Rust isn't what you're after, at least it's not worth learning for that reason.

It's recommended to try and run the programs below and experiment with them yourself. If you don't want to install Rust try the Rust playground. There will also be compiler warnings like warning: unused variable: i, this I would not allow in production stuff I'm writing, in tutorials I don't want to get bogged down with this but basically if I used it this would go away. It's telling me if I take away this variable the program is unchanged, it's trying to help me again.

Finally, I make no pretence of being a Rust expert but I've probably solved enough basics problems in it to be a bit above beginner now. I'd love to get some feedback from any Rustaceans out there as to what they think, please ping me.

Ownership

OK, so with all that let's get started. First topic is ownership and always will be. It's important, learn and digest how this works. Once you do you're well on you're way to doing cool stuff with the Rust, I promise.

So memory management is the theme here basically. If you've programmed in C or C++ before you likely know about the pains memory management can cause. If you've programmed in higher level languages you may know less as it deals with it for you, but it does so with the expense of a garbage collector. You may never have even cared about the garbage collector, it will be slowing things down for you and it only solves the freeing problem, we still need to check for null pointers. But really garbage collectors are great, I much prefer that to C/C++ where memory management just gets in the way of what you're trying to do. Rust offers something in between essentially but solves other problems in the meantime (like checking for null pointers).

So if you've not programmed in C/C++ before, well if you want to reserve memory in a dynamic way (now I need a 10 element list but soon I'll need 12, that kind of thing) you need to reserve memory and free it when done. For example, to reserve memory for an integer on the heap (the dynamic memory store basically, you don't really need to worry about this for now) I need to do:

int* i = new int;*i = 1;

and then when I'm done I need to do:

delete i;

If I forgot the latter or do it twice then badness ensures, particularly in the latter case. It's as simple as that, the standards say if I free twice basically anything can happen. I tried freeing twice on Linux and nothing really happened, on a Mac I got the following (your results may vary):

  ./a.outa.out(67865,0x1047a05c0) malloc: *** error for object 0x7fa4e8c02ac0: pointer being freed was not allocateda.out(67865,0x1047a05c0) malloc: *** set a breakpoint in malloc_error_break to debug[1]    67865 abort      ./a.out

In C++ you need to think about the different types of memory here (stack and heap to those familiar), in Rust less so. The stack and heap still exist but honestly to begin with you don't need to worry so much. If you get deeper into Rust maybe but I've honestly not had to care yet. Nor is there GC (garbage collection) but it's as safe as when there is GC. I can't get the above error from Rust, I've tried (even in unsafe Rust which we're not touching here, stay away for now, there be dragons). I get compiler errors but no runtime errors and of course compiler errors are better because I get them sooner and I know I must have broken something.

The reason has to do with ownership, in Rust everything has to be "owned" by something. Once it's no longer owned by something the memory will be cleared for us at that point. It's as if the compiler went and added the delete statement in C++ for me, cool, I don't need to remember it. But what does it mean to be "owned" by something.

Let take a very simple example here:

fn main() {    let i: u32 = 32;}

That's a full, bug free but ultimately pointless Rust program. This declares a variable i to be of type u32 (unsigned 32 bit integer, Rust's type system is way more sane than C/C++) and sets it to be 32. Rust's compiler notices that when that function returns, i is no longer needed so it does any cleanup necessary (technically this would work in C/C++ too because it's on the stack, but this would also work on the heap in Rust). Essentially you don't need to worry about memory management, it has taken care of for you.

Similarly, we can consider the following program

fn main() {                        // Scope of `main` starts here    let i: u32 = 32;    {                              // A new inner scope starts here        let j: u32 = i + 5;        println!("j is: {}", j);    }                              // The inner scope ends here and `j` is dropped}                                  // The `main` scope ends here and `i` is dropped

Running this program prints the line j is: 37. We won't explain the println statement thoroughly but essentially it prints the string given and every time we have a {} inside that string we can substitute a variable value. Here the {} is replaced with the value of j.

Here we have created another scope in the middle with the braces. If you're not familiar with a scope like this from other C-family languages, basically a scope is created by these curly braces, anything between a left curly brace { and a right curly brace } is a scope. In this case I have a scope for the main function created between lines 1 and 7 an inner scope created after defining i in line 3 that ends on line 6. Everything in this inner scope has access to the variables in the scope above it but the main scope itself doesn't have access to stuff in the middle. So for example the inner scope has i and j but the main scope only has i. Having inner scopes like this is not uncommon in Rust, a bit like JavaScript in this sense but even more so. We can limit the time we need variables for with scopes like this. You can also shadow variables by calling them the same name in inner scopes, see here for example.

Now returning to the code example, we have i as before that goes out of scope once main returns. However, now we have another scope by the inner pair of braces. In this scope we create a new u32 variable j, we also have access to i so we can set j equal to i + 5, or indeed 37. Now once we get to the end of the inner scope, i.e. once the program goes past the right curly brace on line 6, then we can no longer access j. For example consider the following program:

fn main() {    let i: u32 = 32;    {        let j: u32 = i + 5;    }    println!("j is: {}", j);}

When we compile this we get:

 rustc simple_rust_program.rserror[E0425]: cannot find value `j` in this scope --> simple_rust_program.rs:6:26  |6 |     println!("j is: {}", j);  |                          ^ help: a local variable with a similar name exists: `i`error: aborting due to previous errorFor more information about this error, try `rustc --explain E0425`.

Let's take a moment now and look at how brilliant this error output is here. This tells me exactly what I need to know, j doesn't exist at line 6. It even tells me a command to explain the error more fully and it thinks maybe I meant i. I know exactly what it's complaining about and it really tries to help me fix it. I really love the borrow checker for this, it's really trying to help you (I recommend trying to remember this).

So we've covered what ownership means, it's pretty straight forward right, what's the big deal? Well let's move onto the next level, transferring this ownership.

Transferring ownership

Sometimes of course we want to pass variables around, in functions calls for example we may want to pass in a variable as a parameter. Now we have 3 ways (these 3 ways come up a lot in Rust) to do this, transfer ownership, pass by reference or pass by mutable reference. We cover transferring ownership now. The other two will be covered under the Borrowing section below. We can also think about smart pointers as I mention, this technically falls under these 3 things too though, I'll cover this in a later article.

Transferring ownership is really easy syntactically and just what you'd expect. The following program shows something being passed by ownership:

fn take_ownership(a: String) {    println!("I have a: {}", a);                 // and a is now owned by take_ownership}fn main() {    let a: String = "Test String".to_string();   // a is owned by main    take_ownership(a);                           // Now we pass a to take_ownership    //println!("I try to print a: {}", a);    // If I uncomment this line above then I get an error because this scope no longer owns a}

So if you pass ownership like this, it's gone from the original function, only one thing can own a variable at any one time. Unless you pass it back of course. I can alter the above as follows:

fn take_ownership(a: String) -> String {    println!("I have a: {}", a);    return a;}fn main() {    let a: String = "Test String".to_string();    let a = take_ownership(a);    println!("a: {}", a);            // We've passed a back so we're OK now}

As an aside, the above program isn't really very Rust like in the way I wrote the function. Equivalently this function can be written:

fn take_ownership(a: String) -> String {    println!("I have a: {}", a);    a}

This is exactly the same thing. Because there's no semi-colon on the last line Rust knows this is a return. There's an explanation involving statements and expressions but I don't want to go into this here, just be aware if you see this, this is just a return.

So simple enough right. Well no, not quite. There's a complication that we'll talk about shortly but first we cover mutability now.

Mutability

Next on the agenda is to cover mutability, or you may prefer to think of non-constant variables or variables you can alter (or just variables maybe, surely I can alter a variable or it doesn't vary, just go with it). Essentially, by default, everything in Rust is immutable, that is it's constant and can't have its value altered. If I try I get something like the following:

error[E0384]: cannot assign twice to immutable variable `i` --> immutable_variables.rs:3:5  |2 |     let i: u32 = 1;  |         -  |         |  |         first assignment to `i`  |         help: make this binding mutable: `mut i`3 |     i = 2;  |     ^^^^^ cannot assign twice to immutable variableerror: aborting due to previous errorFor more information about this error, try `rustc --explain E0384`.

Pretty clear where the problem is, right? But how do I do that? Quite easy actually, we can create a mutable variable (one that can be changed) by simply adding the keyword mut as per the code below.

fn main() {    let mut i: u32 = 1;        // Added mut, now I can change it    i = 2;    println!("i is: {}", i);}

What's interesting about this is that I can pass ownership of immutable variable around to be a mutable variable and that's absolutely fine. For example:

fn take_ownership_and_pass_back(mut a: String) -> String {    a = a + "2";    a}fn main() {    let a: String = "TEST".to_string();    let a = take_ownership_and_pass_back(a);    println!("a: {}", a);}

So I change a from immutable to mutable, and that's fine. If I run this I get a line a: TEST2. But a was immutable? Well Rust makes no guarantees you won't purposefully change it to mutable but you've got to try to change it. It's not trying to tell you what to do it's trying to get you to stop changing things you never meant to. Generally, immutable variables are a great thing for this reason and that's why they're the default. The point really is that I can limit the amount of time something needs to be mutable for and thus limit the places where it's possible to alter values (and create bugs according to functional programming, we wouldn't want that).

Copying and Cloning

So I said earlier that you can transfer ownership of a variable and I showed you how to do it. I also told you there's a minor complication, well here it is. Let's have a look at the following example:

fn main() {    let i: u64 = 123;    {        let mut j = i;     // <-- If i didn't satisfy copy this would be a move        j = 124;        println!("Not finished and j is: {}", j);    }    println!("Finished and i is: {}", i);    // ^^^ And if i didn't satisfy copy and hence moved above I wouldn't be allowed to use it here}

By the logic I mentioned previously, here i should be moved into the inner block when j is created, as a mutable variable j which I can alter, but then I can no longer access i as it's gone. Yet I run it, it compiles successfully and I get:

Not finished and j is: 124Finished and i is: 123

So I can see I've copied the value of i into j, not moved anything as I thought, and I can happily carry on using an unchanged i. Rust has helped me because it knows u64's satisfy the Copy trait. See here for more details about the Copy trait but I'll explain what this means briefly. If you've come across interfaces from other languages, traits are kinda like interfaces, read "interface" for now instead of trait and it's close enough. Essentially a trait is something that can be satisfied and then things can act accordingly based on this. In this case anything that satisfies the Copy trait, the compiler knows its value gets copied rather than moved.

On the other hand if I change i to be a String s as in the following code:

fn main() {    let s: String = "123".to_string();    {        let mut t = s;        t = "124".to_string();        println!("Not finished and t is: {}", t);    }    println!("Finished and s is: {}", s);}

Then I get an error as follows:

error[E0382]: borrow of moved value: `s`  --> src/main.rs:12:39   |2  |     let s: String = "123".to_string();   |         - move occurs because `s` has type `std::string::String`, which does not implement the `Copy` trait...5  |         let mut t = s;   |                     - value moved here...12 |     println!("Finished and s is: {}", s);   |                                       ^ value borrowed here after moveerror: aborting due to previous errorFor more information about this error, try `rustc --explain E0382`.

Again, look at the amazing output, gcc or clang eat your heart out. This tells me exactly what I need to know, s was created here on line 2, moved here on line 5, but now I've tried to use it on line 12 and I'm not allowed.

So the error is great but what do I do here now? Well there's a few things you can do. If you're happy with the copy you can simply add a .clone() to the line changing the value for t, as in:

        t = "124".to_string().clone();

Now Rust copies it cause I've told it to. Whatever's being cloned needs to satisfy the Clone trait, many things do but not always. Cloning means the whole memory of the value of that variable is copied into memory elsewhere and if something is likely to be particularly big, you may not want it to satisfy this trait so you can't accidentally do this.

But basically what this really means is that whenever we pass a u32 like this it copies the data rather than moving it. So by copying it I get a whole new variable and it's totally unrelated to the original one. Rust is not alone here, I've seen a few languages that do this, Python for example. It's usually small fixed size things this is true of by default, booleans and numbers mostly. If you ever find something like this worked when it looked like it shouldn't then probably just Rust copied it and is trying to help you. Of course if this happens and you change the second variable the first is unchanged, one to be careful of.

So now I've covered most of what you need to know to get going with ownership. But we won't get very far unless it's a pretty simple program (and I told you not to bother using Rust in that case). Sometimes we just want to borrow something which we'll take a look at now.

Borrowing

So we know how to move and copy stuff, but what if I don't want to move or copy it? I want access to it I guess to see what's in there but it's not mine, or even I want to change some stuff but it's not mine. Well you've a few choices as follows:

• If I want to only be able to read from it I can create a "reference" to it, I can have as many of these as I like (as long as no "mutable references" from the next exist).
• If I want to be able to update it too I can create a "mutable reference" to it, but I can only have one of these, period. If I have a "mutable reference" the compiler will stop me from creating any other "references" mutable or not ("references" are immutable by default, as indeed everything is in Rust).
• I can create "smart pointers" to it. I'll discuss this in a future article, this is kinda the last ditch effort but sometimes is needed.

So what are references and immutable references? Well instead of passing ownership I pass a reference, it's kinda like a pointer in C if you're familiar with this, if not I'll try and explain. A reference holds some kind of value, a memory address or something like that points to where the original is. When you take a reference you leave ownership alone but you say to something else, you can point over here and take a look, or even you can point over here and alter it if it's a mutable reference, but it's still mine. We'll look at an example now.

For the code example above I really wanted to be able to change it in another scope without taking ownership, maybe in another function or something. Well I could do this and this works:

fn main() {    let mut s: String = "123".to_string();    {        let t = &mut s;        *t = "124".to_string();        println!("Not finished and t is: {}", t);    }    println!("Finished and s is: {}", s);}

If I run this I get what I want:

Not finished and t is: 124Finished and s is: 124

So what have I done here. First, I've had to change the original string to be mutable because I'm changing it, I didn't need to before because I never wanted to change the original. I also created a mutable reference t by doing let t = &mut s;. This & is the syntax for me taking a reference and mut implies I want a mutable reference, something I can change. If I'd just written let t = &s; then t would just be a reference, here I wanted a mutable reference. When I want to read or write (if it's mutable) to the original I need to dereference it really and I do so by writing *t, the * here means the value this address is pointing to. If I written t = "124".to_string() here I'd get the following error:

 --> src/main.rs:7:13  |7 |         t = "124".to_string();  |             ^^^^^^^^^^^^^^^^^ expected mutable reference, found struct `std::string::String`  |  = note: expected type `&mut std::string::String`             found type `std::string::String`help: consider dereferencing here to assign to the mutable borrowed piece of memory  |7 |         *t = "124".to_string();  |         ^^error: aborting due to previous errorFor more information about this error, try `rustc --explain E0308`.

I'll stop gushing about Rust's error reporting after this one but look, told me exactly how to fix it.

However Rust is smart, in non-ambiguous cases the Rust compiler knows places where I clearly meant to be dereferencing this automatically. So I put the * to dereference the t in the line *t = "124".to_string(); because, as we see above, Rust doesn't work this out for me. But in the next line where I print the variable I can see that I don't need to dereference, there's no harm in doing so, but if I don't the compiler knows "well he obviously meant to". In fact most Rust devs from what I can tell wouldn't bother putting the * on this line either.

Now let's try and create another reference after creating the immutable reference. As you can see if you try, this is not allowed:

error[E0502]: cannot borrow `s` as immutable because it is also borrowed as mutable --> src/main.rs:6:17  |5 |         let t = &mut s;  |                 ------ mutable borrow occurs here6 |         let u = &s;  |                 ^^ immutable borrow occurs here7 |8 |         *t = "124".to_string();  |         -- mutable borrow later used hereerror: aborting due to previous errorFor more information about this error, try `rustc --explain E0502`.

This is because as I hinted earlier, if I try and create any other reference when a mutable reference exists then Rust will block me. The reasons are essentially (as far as I'm aware) for creating safe multi-threading. So if one thread had a mutable reference that could be altering stuff away, I might get data race conditions if I try and read it in another thread. For this reason limiting the scope as I do here is a common thing to see in Rust code. Once the scope changes things get cleared up and it's a great way out of some problems you'll encounter, create a really small scope for changing stuff.

Similarly if I try and create a mutable reference after a reference exists (I have to use it though otherwise Rust is smart enough to know there's no problem):

error[E0502]: cannot borrow `s` as mutable because it is also borrowed as immutable  --> src/main.rs:6:17   |5  |         let u = &s;   |                 -- immutable borrow occurs here6  |         let t = &mut s;   |                 ^^^^^^ mutable borrow occurs here...11 |         println!("Not finished and u is: {}", u);   |                                               - immutable borrow later used hereerror: aborting due to previous errorFor more information about this error, try `rustc --explain E0502`.

That just about covers the basics of the borrow checker. See, easy right. Well there's more, of course and I'll cover that off in future entries in this series, I think that's enough for now. I can't promise you can go ahead and write useful Rust programs now but I hope that at least Chapter 4 of the book will make sense now. In the next few articles in this series I'll try to cover structs, options, threads, matching, enums, smart pointers and how to work with the borrow checker with these things. Stay tuned.