If I do not set a passphrase for the SSH Key generation what risks do I run?
From what I understand the only risk of someone gaining access would be if the laptop was stolen and therefor have the SSH Key on it.
Is this correct?
That is correct. With an un-protected private key, you'd need to remove the public key from github and wherever else it's present ASAP to revoke access.
As a good practice, though, I always recommend encrypting the private key with a passphrase. With ssh-agent (on Linux/OSX) or pageant (windows), there's very little pain involved in using an encrypted key.