SuperminarenandClaude Sonnet 5.5
81edf08d8e
- Page.noindex: marked pages get a robots meta tag and a Disallow line in the generated robots.txt (single source of truth in src/pages.py) - templates/robots.txt: public pages indexable, noindex pages and data files not - Footer shows an optional operator line from contents/snippets/footer_operator.md - Tests for robots.txt, noindex meta and footer Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
12 lines
333 B
Plaintext
12 lines
333 B
Plaintext
# robots.txt – generated by the site generator, do not edit by hand.
|
||
# The course pages may be crawled and indexed. Pages with contact details may not.
|
||
|
||
User-agent: *
|
||
Allow: /
|
||
{% for page in noindex_pages -%}
|
||
Disallow: /{{ page }}
|
||
{% endfor -%}
|
||
# Data files (search indexes etc.) are not pages
|
||
Disallow: /*.json$
|
||
Disallow: /*.zip$
|