/dev/xvdb is indeed mounted, you need to check to see if anything is stored on there that you want to keep, although keeping important stuff on an ephemeral drive is a REALLY bad idea.
You will need to unmount /dev/xvdb before you do anything with it.
While you can
it will make a swap space of the whole ephemeral drive, which you almost certainly don't need. Also, if you partition your swap, you can use the rest of the ephemeral drive for things like the tmp folder, or storing sessions (if your host is a webserver). Ephemeral drives are very quick, but sadly not very persisent.
Anyway, back to swap partitions!
Better to either sfdisk as Abhishek mentions, or manually create a swap partition using fdisk:
Press N to create a new partition
P for primary
1 for the first partition
Press Enter to accept the first location
Enter +xG where x is the size of the swapspace you want. I typically use twice the amount of RAM, but this is not a hard and fast rule
Enter T to change the type
Enter 82 for Linux Swap
Enter W to write the changes
Enter q to quit
You can now create your swap space with
And then enable it with
One word of warning however, and I apologise If Im "Teaching granny to suck eggs" But as the name implies, an Ephemeral drive is... well, Ephemeral. If you ever shutdown your instance, you will have to recreate your swap partition and enable it. For this reason, dont add your newly created swap space to your fstab.
Rebooting should be fine however.