Your Web News in One Place

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

Sign up for GreenGeeks web hosting
December 18, 2020 03:01 pm GMT

Create a Dynamic Sitemap with Next.js

One of the best ways to drive traffic to your website is to have strong Search Engine Optimization (SEO). You can provide search engines with all the URLs for your website using a Sitemap. This allows for easier indexing and more efficient crawling by the search engines.

Maintaining a static sitemap can be tedious and more work if your website is always changing. The best solution is to dynamically create one.

Let's check out a couple of ways we can accomplish this.

Create a sitemap using a script at build time

If all of your content and pages is local in your project, you can easily use a script at build time to create a sitemap.xml.

My blog uses MDX files instead of a CMS, so I do not have to worry about my content changing after build time.

My script uses globby to traverse the file system and return all my routes. First thing we need to do is install it as a dev dependency.

npm i -D globby
Enter fullscreen mode Exit fullscreen mode

Then we can create the script:

scripts/generate-sitemap.js

const fs = require('fs');const globby = require('globby');const generateSitemap = async () => {  // Fetch all routes based on patterns  // Your folder structure might be different so change bellow to match your needs  const pages = await globby([    'pages/**/*.{ts,tsx,mdx}', // All routes inside /pages    '_content/**/*.mdx', // All MDX files inside my /_content    '!pages/**/[*.{ts,tsx}', // Ignore my dynamic route index Example /pages/blog/[slug].tsx    '!pages/_*.{ts,tsx}', // Ignore next.js files    '!pages/api', // Ignore API routes    '!pages/admin.tsx' // Ignore pages not meant to be indexed  ]);  urlSet = pages    .map((page) => {      // Remove none route related parts of filename.      const path = page        .replace('pages', '')        .replace('_content', '')        .replace(/(.tsx|.ts)/, '')        .replace('.mdx', '');      // Remove the word index from route      const route = path === '/index' ? '' : path;      // Build url portion of sitemap.xml      return `<url><loc>https://codebycorey.com${route}</loc></url>`;    })    .join('');  // Add urlSet to entire sitemap string  const sitemap = `<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">${urlSet}</urlset>`;  // Create sitemap file  fs.writeFileSync('public/sitemap.xml', sitemap);};module.exports = generateSitemap;
Enter fullscreen mode Exit fullscreen mode

To run the script at build time, you can create a next.config.js file. This will modify Next.js build process.

const generateSitemap = require('./scripts/generate-sitemap');const generateRSS = require('./scripts/generate-rss');module.exports = {  webpack: (config, { isServer }) => {    if (isServer) {      generateSitemap();    }    return config;  }};
Enter fullscreen mode Exit fullscreen mode

Now when you build your website, you should see a freshly created public/sitemap.xml.

Lastly, I recommend adding public/sitemap.xml to your .gitignore since it is a generated file.

Create a sitemap using a route

You cannot create a sitemap at build time When you are using a content management system (CMS). It might work when you first build your project, but if you push out new content after the build, your sitemap will be outdated.

What we could do is create an API route to fetch the data, and we rewrite the sitemap request to use the API route.

First create the API route:

public/api/sitemap

export default async (req, res) => {  // Fetch data from a CMS.  const resp = await fetch('MOCK_URL');  const externalPosts = await resp.json();  const routes = externalPosts.map((post) => `/posts/${posts.slug}`);  const localRoutes = await globby([    'pages/**/*.{ts,tsx,mdx}', // All routes inside /pages    '!pages/**/[*.{ts,tsx}', // Ignore my dynamic route index Example /pages/blog/[slug].tsx    '!pages/_*.{ts,tsx}', // Ignore next.js files    '!pages/api', // Ignore API routes    '!pages/admin.tsx' // Ignore pages not meant to be indexed  ]);  const pages = routes.concat(localRoutes);  urlSet = pages    .map((page) => {      // Remove none route related parts of filename.      const path = page        .replace('pages', '')        .replace('_content', '')        .replace(/(.tsx|.ts)/, '')        .replace('.mdx', '');      // Remove the word index from route      const route = path === '/index' ? '' : path;      // Build url portion of sitemap.xml      return `<url><loc>https://codebycorey.com${route}</loc></url>`;    })    .join('');  // Add urlSet to entire sitemap string  const sitemap = `<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">${urlSet}</urlset>`;  // set response content header to xml  res.setHeader('Content-Type', 'text/xml');  // write the sitemap  res.write(sitemap);  res.end();};
Enter fullscreen mode Exit fullscreen mode

Now we can create a route rewrite to make /sitemap.xml actually call /api/sitemap.

Create next.config.js and add:

module.exports = {  async rewrites() {    return [      {        source: '/sitemap.xml',        destination: '/api/sitemap'      }    ];  }};
Enter fullscreen mode Exit fullscreen mode

Now when you navigate to http://localhost:3000/sitemap.xml, you should see your sitemap based on local files and your CMS.

Bonus: Robots.txt

One additional thing you can add to your website to improve SEO is a robots.txt (AKA robots exclusion standard). This basically tells search engines which routes they are not allowed to index.

Create public/robots.txt and add

User-agent: *Disallow:Sitemap: https://your-url.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

This will tell search engines they are welcome to crawl your entire website.

If you would like to block any pages from being indexed, add it as disallow field.

User-agent: *Disallow: /adminDisallow: /secret-pageSitemap: https://your-url.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode
  • Follow me on Twitter for random posts about tech and programming. I am also documenting my journey learning design.
  • Nest.js Docs

Original Link: https://dev.to/codebycorey/create-a-dynamic-sitemap-with-next-js-14do

Share this article:    Share on Facebook
View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To