tiny shell · Splitting the command line

Before we can run ls -la, we need to see it as a command and its arguments. The obvious tokenizer is split_whitespace, and for milestone two the obvious tokenizer is the right one:

fn tokenize(line: &str) -> Vec<String> {
    line.split_whitespace().map(str::to_string).collect()
}

Wire it into the loop from milestone one and print the pieces. The first word is the program; the rest are its arguments.

challenge: handle quoted arguments

split_whitespace breaks echo "hello world" into three tokens, but a real shell yields two. Extend tokenize so double-quoted spans keep their spaces. Keep the signature; you will want to walk the characters yourself.

solution

Walk the characters with a flag for whether you are inside quotes:

fn tokenize(line: &str) -> Vec<String> {
    let mut tokens = Vec::new();
    let mut current = String::new();
    let mut in_quotes = false;

    for c in line.chars() {
        match c {
            '"' => in_quotes = !in_quotes,
            c if c.is_whitespace() && !in_quotes => {
                if !current.is_empty() {
                    tokens.push(std::mem::take(&mut current));
                }
            }
            c => current.push(c),
        }
    }
    if !current.is_empty() {
        tokens.push(current);
    }
    tokens
}

Unterminated quotes silently close at end of line — a real shell would keep reading. That wart becomes a milestone of its own later.