{"id":165835,"date":"2024-12-16T11:56:38","date_gmt":"2024-12-16T10:56:38","guid":{"rendered":"https:\/\/de.wordpress.org\/support\/topic\/was-bedeuten-befehle-in-meiner-robots-txt-datei\/"},"modified":"2024-12-16T11:56:38","modified_gmt":"2024-12-16T10:56:38","slug":"was-bedeuten-befehle-in-meiner-robots-txt-datei","status":"publish","type":"topic","link":"https:\/\/de.wordpress.org\/support\/topic\/was-bedeuten-befehle-in-meiner-robots-txt-datei\/","title":{"rendered":"Was bedeuten Befehle in meiner robots.txt Datei?"},"content":{"rendered":"\n<pre class=\"wp-block-code\"><code>Hallo an Alle,<br \/><br \/>K\u00f6nnte mir bitte einer erl\u00e4utern, was die Befehle in meiner robot.txt Datei bedeuten?<br \/><br \/>Die Befehle bedeuten doch was gecrawelt werden darf und was nicht?<br \/><br \/>Wenn ja, welche Seiten sind von dem Craweln ausgeschlossen?<br \/><br \/>Lg.<br \/><br \/>User-agent: *<br \/>Disallow: \/widerruf<br \/>Disallow: \/widerrufsbelehrung<br \/>Disallow: \/agb<br \/>Disallow: \/datenschutz<br \/>Disallow: \/impressum<br \/>Disallow: \/meine Seite-impressum<br \/>Disallow: \/meine Seite-brillenreiniger-datenschutz<br \/>Disallow: \/wp-content\/plugins\/<br \/>Disallow: \/readme.html<br \/>Disallow: \/refer\/<br \/>Disallow: \/cgi-bin\/<br \/>Disallow: \/wp-admin\/<br \/>Disallow: \/trackback\/<br \/>Disallow: \/feed\/<br \/>Disallow: \/comments\/<br \/>Disallow: *\/trackback\/<br \/>Disallow: *\/feed\/<br \/>Disallow: *\/comments\/<br \/>Disallow: \/wp-content\/uploads\/wc-logs\/<br \/>Disallow: \/wp-content\/uploads\/woocommerce_transient_files\/<br \/>Disallow: \/wp-content\/uploads\/woocommerce_uploads\/<br \/>Disallow: \/wp-admin\/<br \/>Allow: \/wp-admin\/admin-ajax.php START YOAST BLOCK ---------------------------<br \/><br \/>User-agent: *<br \/>Disallow: \/?s=<br \/>Disallow: \/page\/*\/?s=<br \/>Disallow: \/search\/<br \/>Disallow: \/wp-json\/<br \/>Disallow: \/?rest_route=<br \/><br \/>User-agent: AdsBot<br \/>Disallow: \/<br \/><br \/>Sitemap: https:\/\/meine Seite.de\/sitemap_index.xml --------------------------- END YOAST BLOCK<\/code><\/pre>\n","protected":false},"template":"","class_list":["post-165835","topic","type-topic","status-publish","hentry"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/de.wordpress.org\/support\/wp-json\/wp\/v2\/topic\/165835","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/de.wordpress.org\/support\/wp-json\/wp\/v2\/topic"}],"about":[{"href":"https:\/\/de.wordpress.org\/support\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":0,"href":"https:\/\/de.wordpress.org\/support\/wp-json\/wp\/v2\/topic\/165835\/revisions"}],"wp:attachment":[{"href":"https:\/\/de.wordpress.org\/support\/wp-json\/wp\/v2\/media?parent=165835"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}