merhaba aşağıdaki şekilde robots.txt oluşturup denermisin

User-agent: Googlebot
Disallow:
User-agent: googlebot-image
Disallow:
User-agent: googlebot-mobile
Disallow:
User-agent: *
Disallow:
Disallow: /cgi-bin/