Communiquez avec les autres et partagez vos connaissances professionnelles

Inscrivez-vous ou connectez-vous pour rejoindre votre communauté professionnelle.

Suivre

Which is better Robots.txt or Meta Robots Tag?

user-image
Question ajoutée par Rameez Bin Abdul Majeed , Digital Marketing Manager , Muscat Higher Education Centre
Date de publication: 2016/12/13
Benita Johnson
par Benita Johnson , MARKETING CONSULTANT , DIGITAL

Both are supported by all crawlers which respect webmasters wishes. Not all do, but against them neither technique is sufficient.

You can use robots.txt rules for general things, like disallow whole sections of your site. If you say Disallow: /family then all links starting with /family are not indexed by a crawler.

Meta tag can be used to disallow a single page. Pages disallowed by meta tags do not affect sub pages in the page hierarchy. If you have meta disallow tag on /work, it does not prevent a crawler from accessing /work/my-publications if there is a link to it on an allowed page.

Davis Parker
par Davis Parker , DIGITAL PROJECT MANAGER , Employers name

On the most basic level, neither the meta robots tag or the Robots.txt has authority over the other – but rather the “noindex” request has authority over the “index” request.

What I’ve suggested to development teams I’ve worked with in the past is to set the Robots.txt file to allow all crawlers, then use the meta robots tag to request that a page not be indexed on a page by page basis. This is easy to remember, easy to update, and makes it easy to avoid conflicts and confusion.

Meta Robots tag is much better as it helps in forcing the search engine crawlers not to index and display the hidden pages in your server.

Winona James
par Winona James , Marketing Manager , ABC Media Marketing Partners

Personally I think the meta tag is even more useful than the robots.txt file.

 

The robots.txt file is intended to PREVENT bots from spidering files in the first place.

 

The robots meta tag can in some ways be used to get indexed files deindexed. And it

is more versatile as a tool to 'guide' bots through your website without having them

index all the files.

 

I always put up a robots.txt file BEFORE I upload any of the other files.

Then I fine-tune using the meta tag.

Suvik Krishnan
par Suvik Krishnan , Marketing Analyst , AFT Marketing House, Inc

There is no real "MAJOR" difference between the files. They both do the same work however the META tag is not as reliable as the .txt This is because I think (could be wrong) that the Robot.txt is a seperate file. The intention of the file,tag is to inform the search engines ROBOT not to index certain pages. IE Contact us ETC ETC... Hope that helps...

Farooq Ahmed
par Farooq Ahmed , Senior SEO Manager , Data Ladder

Robots.txt is the file to block certain pages/content from search engines. If you want to block folders or any section of website robots.txt file is the better option.

 

If you want to block some specific pages you can use meta robot tags. In some situation you really need to use these tags for eg. listing pages.

 

There is a very huge difference between meta robot and robots.txt.

In robots.txt, we ask crawlers which page you have to crawl and which one you have to exclude but we don't ask crawler to not to index those excluded pages from crawling.

But if we use meta robots tag, we can ask search engine crawlers not to index this page.The tag to be used for this is:

<#meta name = "robot name", content = "noindex"> (remove #)

OR

<#meta name = "robot name", content = "follow, noindex"> (remove #)

In the second meta tag, I have asked robot to follow that URL but not to index in search engine.

chris adams
par chris adams , DIGITAL MARKETING MANAGER

You can have any one but if your website has plenty of web pages then robots.txt is easy and reduces time complexity

 

Tofique Shaikh
par Tofique Shaikh , Full time Freelancer , Self

You should use 'noindex,follow' in a robots meta tag, rather than robots.txt, because it will allow the link juice to pass through. It is better from a SEO perspective.

Owais Jaffar
par Owais Jaffar , Head of SEO and Social Media , Knowledge Path

While there is little difference between the two, the implementation of either could affect your SERPs. So it is best to be aware of what is end goal of the activity that you want to implement. Robots.txt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages. You could choose to use both a meta robots tag and a robots.txt file as neither has authority over the other, but “noindex” always has authority over “index” requests.

Eijaz Ibrahim Pardhan
par Eijaz Ibrahim Pardhan , Senior SEO Executive , JBS

Robots.txt is better and is more common.

More Questions Like This