Robots.txt dosyanızı, Public_Html içine, yani root bölümüne atmanız gerek. www.siteadi.com/robots.txt şeklinde ulaşılabilir olması gerek. Archive.org ve diğer botlar engellenmiş haliyle aşağıda. Site haritanızı ekleyerek direkt kullanabilirsiniz.

User-agent: Googlebot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot-Image
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot-News
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot-Video
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot-Mobile
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Mediapartners-Google
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: AdsBot-Google
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: AdsBot-Google-Mobile
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Applebot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Bingbot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: msnbot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Twitterbot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Yandex
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: DuckDuckBot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: ia_archiver
Disallow: /

User-agent: *
Disallow: /

Sitemap: https://www.siteadi.com/sitemap.xml