Selamlar,

Robots.txt hakkında Pek çok konu var ama hepsi birbirinden farklı yorumlara rastladım.

Kullandığım robots.txt dosyası hakkında yorumlarınız var ise çok sevinirim.
Eklemem gerekenler/çıkarmam gerekenler gibi...
Wordpress kullanıyorum.

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-login/
Disallow: /cgi-bin/

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: YandexNews
Allow: /

Sitemap: https://xxx.com/sitemap.xml.gz
Sitemap: https://xxx.com/sitemap.xml
Sitemap: https://xxx.com/sitemap-image.xml.gz
Sitemap: https://xxx.com/sitemap-image.xml
Sitemap: https://xxx.com/sitemap-news.xml
Sitemap: https://xxx.com/sitemap-mobile.xml.gz
Sitemap: https://xxx.com/sitemap-mobile.xml
Sitemap: https://xxx.com/sitemap_addl.xml.gz
Sitemap: https://xxx.com/sitemap_post.xml.gz
Sitemap: https://xxx.com/sitemap_page.xml.gz
Sitemap: https://xxx.com/sitemap_attachment_1.xml.gz
Sitemap: https://xxx.com/sitemap_attachment_2.xml.gz
Sitemap: https://xxx.com/sitemap_attachment_3.xml.gz
Sitemap: https://xxx.com/sitemap_cep-telefonlari.xml.gz
Sitemap: https://xxx.com/sitemap_category.xml.gz
Sitemap: https://xxx.com/sitemap_post_tag.xml.gz
Sitemap: https://xxx.com/sitemap_post_format.xml.gz