That icon called "favicon"
You can simply generate it by online tools like these:
http://tools.dynamicdrive.com/favicon/
http://www.favicon.cc/
All you need to upload the logo or image you want to use as icon and generate it there, it will not take a minute.
yes rename your logo icon to favicon.ico. and also you can make a different icon for each directory.
you may also write this definition tags inside element: