• 0 Posts
  • 44 Comments
Joined 1 month ago
cake
Cake day: August 13th, 2025

help-circle

  • If you are using Hugo use this robots.txt template that automatically updates every build:

    {{- $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.txt" -}}
    {{- $resource := resources.GetRemote $url -}}
    {{- with try $resource -}}
      {{ with .Err }}
        {{ errorf "%s" . }}
      {{ else with .Value }}
    	{{- .Content -}}
      {{ else }}
        {{ errorf "Unable to get remote resource %q" $url }}
      {{ end }}
    {{ end -}}
    
    Sitemap: {{ "sitemap.xml" | absURL }}
    

    Optionally lead rouge bots to poisoned pages:

    {{- $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.txt" -}}
    {{- $resource := resources.GetRemote $url -}}
    {{- with try $resource -}}
      {{ with .Err }}
        {{ errorf "%s" . }}
      {{ else with .Value }}
        {{- printf "%s\n%s\n\n" "User-Agent: *" "Disallow: /train-me" }}
        {{- .Content -}}
      {{ else }}
        {{ errorf "Unable to get remote resource %q" $url }}
      {{ end }}
    {{ end -}}
    
    Sitemap: {{ "sitemap.xml" | absURL }}
    

    Check out how to poison your pages for rouge bots in this article

    Repo was deleted and the internet archive was excluded.

    I use Quixotic and a Python script to poison the pages and I included those in my site update script.

    Its all cobbled together in amateur fashion from the deleted article but its honest work.


















  • Not mad, the comment was not adding anything of value in my opinion because it was not clear what it meant. So the down vote is for “this comment is not useful and perhaps lazy”. I also upvote things which I do not agree with but they are proposed in a civil manner, and when I’m proven wrong I say " thanks for letting me know this". So… This is not personal.

    And I never claimed that anything (post, comment or whatever) needed words to have meaning. Not sure what your point is though…