Kayan fayil na robots.txt da aka adana a tushen shafin yanar gizonku zai gaya wa 'yan fashin yanar gizon kamar bincike na bincike wanda ke kan hanyoyi da fayilolin da aka ba su izuwa. Yana da sauƙi don amfani da fayil robots.txt, amma akwai wasu abubuwa da ya kamata ka tuna:
- Ƙungiyoyin yanar gizon Black hat za su yi watsi da fayil din robots.txt. Mafi yawan iri iri iri ne burbushin malware da robots neman adiresoshin imel don girbi.
- Wasu sabon masu shirye-shirye za su rubuta jigilar injuna da basu watsi da fayil na robots.txt. Ana yin wannan ta hanyar kuskure.
- Duk wanda zai iya ganin fayil din robots.txt. Ana kiran su da robots.txt kullum kuma ana adana su a kowane shafin yanar gizon.
- A ƙarshe, idan wani ya haɗa zuwa fayil ko shugabanci da aka cire ta hanyar fayil na robots.txt daga shafin da ba a cire ta hanyar robots.txt ba, injunan bincike zasu iya samun shi.
Kada ku yi amfani da fayilolin robots.txt don boye duk wani abu mai muhimmanci. Maimakon haka, ya kamata ka sanya bayanai mai mahimmanci bayan kalmomin sirri masu asiri ko barin shi a kan yanar gizo gaba ɗaya.
Yadda za a yi amfani da waɗannan samfurin Samfurin
Kwafi rubutu daga samfurin da ke kusa da abin da kake son yi, da kuma manna shi cikin fayil ɗin robots.txt. Canja robot, shugabanci, da sunayen fayiloli don daidaita tsarin da kuka fi so.
Fassara guda biyu na Robots.txt
Mai amfani mai amfani: *
Disallow: /
Wannan fayil ya ce duk wani robot (Mai amfani-wakili: *) wanda ya isa ya kamata ya watsar da kowane shafi a shafin (Disallow: /).
Mai amfani mai amfani: *
Disallow:
Wannan fayil ya ce duk wani robot (Mai amfani-wakili: *) wanda yake samun dama ya yarda a duba kowane shafin a kan shafin (Disallow:).
Hakanan zaka iya yin wannan ta hanyar barin fayil din robots.txt ɗinka ko ba tare da ɗaya a kan shafin ba.
Kare Tsararren Bayanai Daga Kamfanin Robots
Mai amfani mai amfani: *
Disallow: / cgi-bin /
Disallow: / temp /
Wannan fayil ya ce duk wani robot (Mai amfani-wakili: *) wanda ya isa ya kamata ya watsar da kundayen adireshi / cgi-bin / da / temp / (Disallow: / cgi-bin / Disallow: / temp /).
Kare Shafuka masu Musamman Daga Robots
Mai amfani mai amfani: *
Disallow: /jenns-stuff.htm
Disallow: /private.php
Wannan fayil ya ce duk wani robot (Mai amfani-wakili: *) wanda ya isa ya kamata ya watsar da fayilolin /jenns-stuff.htm da /private.php (Disallow: /jenns-stuff.htm Disallow: /private.php).
Tsayar da Wuta Mai Mahimmanci daga Samun dama ga Yanar Gizo
Mai amfani mai amfani: Lycos / xx
Disallow: /
Wannan fayil yana cewa Lycos bot (Mai amfani-wakili: Lycos / xx) ba a yarda izinin shiga ko'ina a shafin ba (Disallow: /).
Bada izinin Daya Na Musamman Musamman
Mai amfani mai amfani: *
Disallow: /
Mai amfani: Googlebot
Disallow:
Wannan fayil ɗin na farko ya watsar da dukkanin fashi kamar yadda muka yi a sama, sannan a bayyane ya sa Googlebot (Mai amfani-Google: Googlebot) ta sami dama ga duk abin da (Disallow:).
Haɗa Hanyoyi masu yawa don samo asirin da kuke so
Duk da yake yana da kyau a yi amfani da layi mai amfani mai amfani, kamar mai amfani: *, zaka iya zama kamar yadda kake so. Ka tuna cewa rukunin baro sun karanta fayil din. To, idan layi na farko an ce an katange dukkan 'yan fashi daga komai, sannan daga bisani a cikin fayil yana cewa duk' yan fashi suna ba da izini ga kowane abu, fashi zai sami dama ga komai.
Idan ba ka tabbatar ko ka rubuta fayil ɗin robots.txt daidai ba, zaka iya amfani da kayan yanar gizon yanar gizon Google don bincika fayil ɗin robots.txt ko rubuta sabon abu.