Yes and No. It all depends on the way you are using your USB drive, and what operating system you use.
As a general rule. No, but it isn't a bad idea, just in case if you are not sure of what your USB drive is doing (IE: being used as a RAM expander - or used to cache memory).
If you are unsure - or feel safer doing so - YES :Eject USB drive.
