The simplest setup robots.txt in Drupal 8

User-agent: *Disallow: /

## robots.txt## This file is to prevent the crawling and indexing of certain parts# of your site by web crawlers and spiders run by sites like Yahoo!# and Google. By telling these "robots" where not to go on your site,# you save bandwidth and server resources.User-agent: *# CSS, JS, ImagesAllow: /misc/*.css$Allow: /misc/*.css?Allow: /misc/*.js$Allow: /misc/*.js?Allow: /misc/*.gifAllow: /misc/*.jpgAllow: /misc/*.jpegAllow: /misc/*.pngAllow: /modules/*.css$Allow: /modules/*.css?Allow: /modules/*.js$Allow: /modules/*.js?Allow: /modules/*.gifAllow: /modules/*.jpgAllow: /modules/*.jpegAllow: /modules/*.pngAllow: /profiles/*.css$Allow: /profiles/*.css?Allow: /profiles/*.js$Allow: /profiles/*.js?Allow: /profiles/*.gifAllow: /profiles/*.jpgAllow: /profiles/*.jpegAllow: /profiles/*.pngAllow: /themes/*.css$Allow: /themes/*.css?Allow: /themes/*.js$Allow: /themes/*.js?Allow: /themes/*.gifAllow: /themes/*.jpgAllow: /themes/*.jpegAllow: /themes/*.png# DirectoriesDisallow: /includes/Disallow: /misc/Disallow: /modules/Disallow: /profiles/Disallow: /scripts/Disallow: /themes/# FilesDisallow: /CHANGELOG.txtDisallow: /cron.phpDisallow: /INSTALL.mysql.txtDisallow: /INSTALL.pgsql.txtDisallow: /INSTALL.sqlite.txtDisallow: /install.phpDisallow: /INSTALL.txtDisallow: /LICENSE.txtDisallow: /MAINTAINERS.txtDisallow: /update.phpDisallow: /UPGRADE.txtDisallow: /xmlrpc.php# Paths (clean URLs)Disallow: /admin/Disallow: /comment/reply/Disallow: /filter/tips/Disallow: /node/add/Disallow: /search/Disallow: /user/register/Disallow: /user/password/Disallow: /user/login/Disallow: /user/logout/# Paths (no clean URLs)Disallow: /?q=admin/Disallow: /?q=comment/reply/Disallow: /?q=filter/tips/Disallow: /?q=node/add/Disallow: /?q=search/Disallow: /?q=user/password/Disallow: /?q=user/register/Disallow: /?q=user/login/Disallow: /?q=user/logout/Sitemap: https://site.com/sitemap.xml