htmlnetlifyrobots.txteleventy

Eleventy generated robots.txt is not valid in Lighthouse report

I'm using Eleventy with Netlify, and when running Lighthouse on my website, I get the error "robots.txt is not valid" / "Unknown directive". I've tried verifying in Google's robots.txt tester and get "Error - Syntax not understood":

google search console syntax not understood error

I have both robots.html and sitemap.html static files in my /src folder:

robots.html

---
permalink: /robots.txt
layout: null
eleventyExcludeFromCollections: true
---

User-agent: * Disallow: /admin/ Allow: / Sitemap:
{{ client.domain }}/sitemap.xml

sitemap.html

---
permalink: /sitemap.xml
layout: null
eleventyExcludeFromCollections: true
---
{% sitemap collections.sitemap %}

These are then used to generate a robots.txt and sitemap.xml file in /public when I commit a change. I know this line in my .eleventy.js file is responsible for the automatic sitemap generation:

eleventyConfig.addPlugin(pluginSitemap, configSitemap);

I am not very familiar with how this part works to get more specific, but I get the resultant files in /public:

robots.txt

User-agent: * Disallow: /admin/ Allow: / Sitemap:
https://rklogginglumber.com/sitemap.xml

Note that there is always an extra line \n generated above the User-agent line (you can see this in the image I attached above of Google's robots.txt tester). Not sure why this is, but just in case that space could be causing the error, I wanted to note that.

sitemap.xml

<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"><url><loc>https://rklogginglumber.com/</loc><lastmod>2025-01-26T23:52:06.325Z</lastmod></url><url><loc>https://rklogginglumber.com/about/</loc><lastmod>2025-01-26T23:55:47.684Z</lastmod></url><url><loc>https://rklogginglumber.com/contact/</loc><lastmod>2025-01-26T23:55:47.685Z</lastmod></url><url><loc>https://rklogginglumber.com/other-services/</loc><lastmod>2025-01-27T00:03:07.573Z</lastmod></url></urlset>

I originally built this website off of a website kit, CodeStitchOfficial/Intermediate-Website-Kit-SASS, and the error persists even on a fresh clone of the kit.

Solution 
  • User-agent: * Disallow: /admin/ Allow: / Sitemap:
https://rklogginglumber.com/sitemap.xml

    The directives User-agent, Disallow, Allow and Sitemap should all be in an individual line of their own.

    User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://rklogginglumber.com/sitemap.xml