
06/11/2024
Hereβs how to set up a robots.txt.
Key Directives in robots.txt
β’ π€ User-agent: Specifies which bot the rules apply to (e.g., User-agent: Googlebot or User-agent: * for all bots).
β’ π« Disallow: Blocks bots from accessing certain files or directories (e.g., Disallow: /private/).
β’ β
Allow: Grants access to specific files or folders within a disallowed directory (only in Google) (e.g., Allow: /public/file.html).
β’ πΊοΈ Sitemap: Points bots to your XML sitemap, helping them navigate your siteβs structure (e.g., Sitemap: https://example.com/sitemap.xml).
Example of a Basic robots.txt
π€ User-agent: *
π« Disallow: /private/
β
Allow: /public/
πΊοΈ Sitemap: https://example.com/sitemap.xml
This file helps search engines know what to crawl and index on your website! π
π€ User-agent: *
π« Disallow: /private/
β
Allow: /public/
πΊοΈ Sitemap: https://example.com/sitemap.xml