reactjsnext.jsrobots.txt

Generating a dynamic /robots.txt file in a Next.js app


I need a way to answer dynamically to the /robots.txt request.

And that's why I've decided to go with getServerSideProps

https://nextjs.org/docs/basic-features/data-fetching#getserversideprops-server-side-rendering

If you export an async function called getServerSideProps from a page, Next.js will pre-render this page on each request using the data returned by getServerSideProps.

export async function getServerSideProps(context) {
  return {
    props: {}, // will be passed to the page component as props
  }
}

Inside the context parameter we have the req and res objects.

enter image description here

The response for the robots.txt will depend on the req.headers.host value.

For example:

This is my current code:

pages/robots.txt.tsx

import React from "react";
import { GetServerSideProps } from "next";

interface Robots {
  ENV: "TEST" | "PROD"
}

export const getServerSideProps : GetServerSideProps<Robots> = async (context) => {
  const { req, res } = context;
  const { host } = req.headers;

  res.write("XXX");
  res.end();

  return({                // This is unnecessary (but Next.js requires it to be here)
    props: {
      ENV: "TEST"
    }
  });
};

const Robots: React.FC<Robots> = (props) => {  // This is also unnecessary (but Next.js requires it to be here)
  console.log("Rendering Robots...");

  return(
    <div>
      I am Robots 
    </div>
  );
};

export default Robots;  // This is also unnecessary (but Next.js requires it to be here).

It seems to be working:

enter image description here

But the weird part is that Next.js demands me to export a component from that page. And also returns a props: {} object from getServerSideProps is also required.

What is the way to go here? I'm basically using req,res from getServerSideProps to returning something that is not a page. Is this an anti-pattern?

UPDATE

Yes, this is an anti-pattern. You should use rewrites. See the selected answer.


Solution

  • You can use an API route instead for the logic and have a rewrite map /robots.txt requests to /api/robots in your Next.js config file.

    // next.config.js
    module.exports = {
        // ...
        async rewrites() {
            return [
                {
                    source: '/robots.txt',
                    destination: '/api/robots'
                }
            ];
        }
    }
    
    // /pages/api/robots
    export default function handler(req, res) {
        res.send('XXX'); // Send your `robots.txt content here
    }