Bunun için robots.txt yi fazla kurcalamanızı tavsiye etmem. Google webmaster tools aracılığı ile Google dizini bölümünden URL'leri kaldır seçeneğini kullanmanız daha mantıklıdır. Kırık link 404 hatası verdiği veya ulaşılmadığı müddetçe zaten bir zaman sonra arama motorlarından kalkacaktır. Engelleme ayrı, dizinden kaldırmak ayrı. İkisi farklı şeyler...
Dünyanın neresine gidersen git ençok kullanılan 404 yöntemi robots.txt tir
sırf bunun için webmaster tools ta robots.txt tester mevcuttur bilip bilmeden teşvik etmeyelim lütfen
--R10.NET; Flood Engellendi -->-> Yeni yazılan mesaj 08:36:19 -->-> Daha önceki mesaj 08:33:43 --
User-agent: *
Disallow:
Disallow: /*ref=*
Disallow: /*?ref=
Disallow: /*?ref=ornek1.com
Disallow: /*?ref=ornek2.net
++++++++++++++++++++++++++++++++++
Robots.txt dosyanızda www kullanmayın.
hepsinde ref çekmesine gerek yok
Örnek robots.txtimden bir örnek vereyim
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Disallow: /konu-the-mg-okul-not-yonetimi-scripti.html?action=newpost
Disallow: http://scriptburada.net/konu-forum-kurallari.html
Disallow: http://scriptburada.net/forum-e-ticaret
Disallow: http://scriptburada.net/konu-stwity-takipci-script-v1-3.html
Disallow: http://scriptburada.net/konu-wordpress%E2%80%99te-eklenti-nasil-kurulur.html
Disallow: http://scriptburada.net/konu-wordpress-nedir.html
Disallow: http://scriptburada.net/konu-wordpress-vip-eklenti-chat-sistemi.html
Disallow: http://scriptburada.net/konu-wordpress-haber-blog-sitesi-temasi.html
Disallow: http://scriptburada.net/konu-tema-logo-satma-scripti-full.html
Disallow: http://scriptburada.net/konu-psd-magazine-dergi-kapaklari.html
Disallow: http://scriptburada.net/konu-wordpress-puan-durumu-fikstur-eklentisi.html
Disallow: http://scriptburada.net/konu-hesabiniza-her-gun-yuzlerce-havale-gelsin-hem-del-5-er-10-lira.html
Disallow: http://scriptburada.net/whmcs
Disallow: http://scriptburada.net/archive/index.php?forum-8.html
Disallow: http://scriptburada.net/sohbet.php
Disallow: http://scriptburada.net/forumdisplay.php?fid=133
Disallow: http://scriptburada.net/member.php?action=profile&uid=3
Disallow: http://scriptburada.net/konu-wordpress-haber-magazin-teknoloji-sitesi-temasi.html
Disallow: http://scriptburada.net/forumdisplay.php?fid=129
Disallow: http://scriptburada.net/uye-stwity.html
Disallow: http://scriptburada.net/forumdisplay.php?fid=136
Disallow: http://scriptburada.net/archive/index.php?forum-9.html
Disallow: http://scriptburada.net/tagcloud.php?tag=guppy
Disallow: http://scriptburada.net/tagcloud.php?tag=kasma
Disallow: http://scriptburada.net/archive/index.php?thread-39.html
Disallow: http://scriptburada.net/uye-saimyildiz.html
Disallow: http://scriptburada.net/forumdisplay.php?fid=116
Disallow: http://scriptburada.net/archive/index.php?thread-724.html
Disallow: http://scriptburada.net/forumdisplay.php?fid=117
Disallow: http://scriptburada.net/konu-sunucumuz-tasinmistir.html
Disallow: http://scriptburada.net/uye-bykasparow.html
Disallow: http://scriptburada.net/uye-kutay4343.html
Disallow: http://scriptburada.net/konu-ucretsiz-wordpress-haber-temasi-%E2%80%93-snapwire.html
Disallow: http://scriptburada.net/tagcloud.php?tag=platinum
Disallow: http://scriptburada.net/archive/index.php?thread-212.html
Disallow: http://scriptburada.net/konu-canli-destek-sistemi-coklu-konusma-live-zilla-tarzi-bedava-full.html
Disallow: http://scriptburada.net/uye-olderstyle.html
Disallow: http://scriptburada.net/uye-yakuza.html
Disallow: http://scriptburada.net/uye-s3s.html
Disallow: http://scriptburada.net/tagcloud.php?tag=linkex-
Disallow: http://scriptburada.net/uye-umut54848.html
Disallow: http://scriptburada.net/archive/index.php?forum-1.html
Disallow: http://scriptburada.net/konu-mt2dosyalar-cloud-index-tasarimi.html
Disallow: http://scriptburada.net/archive/index.php?thread-734.html
Disallow: http://scriptburada.net/tagcloud.php?tag=king
Disallow: http://scriptburada.net/tagcloud.php?tag=kisaca
Disallow: http://scriptburada.net/tagcloud.php?tag=sayac
Disallow: http://scriptburada.net/tagcloud.php?tag=links
Disallow: http://scriptburada.net/tagcloud.php?tag=mailchimp
Disallow: http://scriptburada.net/tagcloud.php?tag=code
Disallow: http://scriptburada.net/forumdisplay.php?fid=113
Disallow: http://scriptburada.net/tagcloud.php?tag=yukleme
Disallow: http://scriptburada.net/showthread.php?tid=745
Disallow: http://scriptburada.net/uye-a2012.html
Disallow: http://scriptburada.net/tagcloud.php?tag=tags
Disallow: http://scriptburada.net/tagcloud.php?tag=siyah
Disallow: http://scriptburada.net/tagcloud.php?tag=qr-code
Disallow: http://scriptburada.net/tagcloud.php?tag=smart
Disallow: http://scriptburada.net/konu-yoast-wordpress-seo-ucretsiz-wp-seo-eklentisi.html?action=lastpost
Disallow: http://scriptburada.net/tagcloud.php?tag=zcumbeton