Outdated or blocked web pages.
bu seçenegi seçtim 60 kadar sayfa ekledim şimdi bekliyorum ayrıca birşey yapmaya gerek varmı eski sayfalar duruyorsa silermi yoksa sayfaları bulamadığı zamanmı siliyor bu seçenek sayfalara User-Agent: *
Disallow: / kodu koymaya gerek varmı
Alıntı
şöyle birşey buldum bunlardan hangisini robots dosyasına eklememiz gerekiyor
Alıntı
Block or remove pages using a robots.txt file
You can use a robots.txt file to block Googlebot from crawling pages on your site.
For example, to block Googlebot from crawling all pages under a particular directory (for example, lemurs), you'd use the following robots.txt entry:
User-agent: Googlebot
Disallow: /lemurs
To block Googlebot from crawling all files of a specific file type (for example, .gif), you'd use the following robots.txt entry:
User-agent: Googlebot
Disallow: /*.gif$
To block Googlebot from crawling dynamically generated pages, you'd use this robots.txt entry:
User-agent: Googlebot
Disallow: /*?
Pages you block in this way may still be added to the Google index if other sites link to them. As a result, the URL of the page and, potentially, other publicly available information can appear in Google search results. However, no content from your pages will be crawled, indexed, or displayed.
To entirely prevent a page from being added to the Google index even if other sites link to it, use a noindex meta tag, and ensure that the page does not appear in robots.txt. When Googlebot crawls the page, it will recognize the noindex meta tag and drop the URL from the index.