禁止搜索引擎收录的方法（forbidden search engine included method）

《禁止搜索引擎收录的方法（forbidden search engine included method）》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、禁止搜索引擎收录的方法（Forbiddensearchengineincludedmethod）Whatisanrobots.txtfile?SearchenginesusespiderprogramstoautomaticallyaccesswebpagesontheInternetandaccesswebpages.Whenaccessingawebsite,spiderwillfirstcheckwhetherthereisaplaintextfilecalledrobots.txtintheroo

2、tfieldofthewebsite,whichspecifiesthecrawlrangeofspideronyourwebsite.Youcancreatearobots.txtonyourwebsiteanddeclareinthefilethatthesitedoesnotwanttobepartofthesearchengine,orspecifythesearchenginetoonlyincludespecificparts.Pleasenotethatyouonlyneedtousethe

3、robots.txtfilewhenyourwebsitecontainscontentthatyoudon'twanttobeincludedinthesearchengine.Ifyouwantthesearchenginetoincludeeverythingonthesite,donotbuildrobots.txtfiles.ReturntotopWhereistherobots.txtfile?Therobots.txtfileshouldbeplacedunderthewebrootdire

4、ctory.Forexample,whenthespidertoaccessawebsite(suchashttp://www.abc.com),itwillfirstcheckthehttp://www.abc.com/robots.txtfileexistsinthesite,ifSpiderfindsthefile,itwillbebasedonthecontentsofthisdocument,todeterminethescopeoftherighttolimitaccesstoit.SiteU

5、RLcorrespondingrobots.txtURLHttp://www.w3.org/http://www.w3.org/robots.txtHttp://www.w3.org:80/http://www.w3.org:80/robots.txtHttp://www.w3.org:1234/http://www.w3.org:1234/robots.txtHttp://w3.org/http://w3.org/robots.txtReturntotopIsetupinrobots.txtbanned

6、Baiduincludedmywebsitecontent,whystillappearinBaidusearchresults?Iftheothersitelinkstoprohibitincludedwebsettingsinyourrobots.txtfile,thenthesepagesmaystillappearinBaidusearchresults,butthecontentonyourpagewillnotbebuiltintotheindex,captureanddisplay,disp

7、layintheBaidusearchresultsistheonlyotherwebsiteonyourwebdescription.ReturntotopSearchenginesarenotallowedtotracklinkstowebpages,butonlytowebpagesIfyoudon'twantthesearchenginetotrackthelinksonthispageanddon'tpasstheweightofthelink,putthemetatagintothe

8、>sectionofthepage:Ifyoudon'twantBaidutotrackaparticularlink,Baidualsosupportsmoreprecisecontrols.Pleasewritethistagdirectlyonalink:s