SitewalkBot
SitewalkBot is the crawler behind Sitewalk. Someone asked it to look at your site. This page says what it does and how to stop it.
What it is
A person typed your site address into Sitewalk and asked for a technical SEO audit. The bot fetches pages to build that report.
- User agent
SitewalkBot/1.0 (+https://sitewalkseo.com/bot)
How it behaves
- It reads your robots.txt first and obeys it for pages, images, links, and the stylesheet and script checks. The site-wide requests below are the exception.
- It uses ports 80 and 443 only.
- It identifies itself with the user agent above, including when it opens a page in a browser. It never uses another crawler's name.
- It sends one request at a time to your site. Page and image requests are at least 1.5 seconds apart. The one exception: when the owner of a site has verified it and runs a Pro audit of it, they are at least 1 second apart. The eight site-wide requests below are not held to that: three of them are 1 second apart and the rest follow one another.
- It stops at the first 429 (too many requests) or 403 (forbidden) answer from your server.
- There is a daily limit on how many audits can target one site, across all users.
- It will not follow links to private or internal network addresses.
What it requests
Your robots.txt, your XML sitemap (the addresses your robots.txt lists, and the usual places such as /sitemap.xml), and the HTML pages it reaches by following links from the address it was given. It reads the HTML of each page the way a search engine does. It does not submit forms, sign in, or fetch anything behind a login.
When someone asks for a site's pages to be read after their scripts run, and the site has been verified, SitewalkBot also opens some pages in a browser: the home page, the pages that look empty without scripts, and a sample of the rest, in audits of up to 100 pages. These pages were already crawled, so your robots.txt had already allowed them. For each page it loads the page itself and the scripts, stylesheets and data requests the page makes to your own site, the way a visitor's browser would. It loads nothing from other websites, and does not load images, fonts or video. It does not scroll, click, type or sign in. A live connection a script opens on the page, such as a WebSocket, is blocked and never reaches your site. Pages are opened one at a time, at least 1.5 seconds apart, with up to 15 seconds to load and 1.5 seconds more for scripts to finish, and the whole browser step for an audit stops at 10 minutes.
It also sends a HEAD request (a request for a file's headers only) for each image on your site that those pages use, so it can read the image's size. No image is downloaded, and the requests keep the same 1.5-second pace. Images on other sites are not requested. Like pages, images are checked against your robots.txt first, and an image it disallows is left out.
Beyond pages and images, each audit makes up to 8 site-wide requests, one at a time: your robots.txt, your XML sitemap, your llms.txt, your home page and its other http, https, www and non-www versions, and one secure connection to read your certificate. None of these 8 requests checks your robots.txt first. The three extra home page requests are 1 second apart. All of them use the SitewalkBot user agent and do not follow redirects.
Separately, an audit sends up to 40 HEAD requests (headers only, nothing downloaded) for the stylesheets and scripts your pages use, to check that each one still loads. These go to your own site only, 1.5 to 3 seconds apart, and each file is checked against your robots.txt first. A file it disallows is skipped. If your site answers one with 429 or 403, Sitewalk stops checking files for that audit.
Each audit has a page cap: 50 pages without a confirmed email, and up to 500 for anyone with one. With the pause between requests and the image checks, 50 pages usually take a few minutes and 500 pages 20 to 60 minutes.
Checking links to other sites
Pages often link to other sites. For each audit, Sitewalk checks a small sample of those links to see whether they still work. If your site is linked from a page someone audited, SitewalkBot may send it a few requests. The check follows these limits:
- Up to 25 addresses on other sites per audit, and at most 3 per site.
- At least 1.5 seconds between requests to the same site (1 second when the site's owner has verified it and runs a Pro audit).
- It reads at most 1 KiB of each response.
- It identifies itself as SitewalkBot and does not follow redirects.
- It reads the other site's robots.txt first. It skips a site whose robots.txt disallows SitewalkBot, or gives a server error, a redirect, no answer or 429 (too many requests).
Why your site was crawled
Someone asked Sitewalk to audit your address. Sitewalk crawls a site only when a person asks for an audit of it. It does not crawl the web on its own schedule and it does not build a search index. The report goes to the person who asked.
If Sitewalk is refused (HTTP 401, 403 or 429)
Some hosts and firewalls turn away traffic from data centre addresses, whatever the user agent says. If an audit of your site fails with a refusal, ask your host or firewall to allow Sitewalk's server and the SitewalkBot/1.0 user agent. Allowing the user agent alone is often not enough. A block like this usually covers every subdomain of the domain, so retrying www or another subdomain will not help.
Opt out
Add this to your robots.txt. The bot will not crawl your site after it next reads the file. The exceptions are the address an audit starts from and the site-wide requests listed above (robots.txt, the XML sitemap, llms.txt, the home page and its other versions, and one secure connection), which do not check robots.txt first.
User-agent: SitewalkBot Disallow: /
To block only part of the site, replace / with the path. The same lines also stop SitewalkBot from checking links to your site from other people's audits.
Report a problem
If the bot is misbehaving, or you want it stopped and cannot edit robots.txt, write to the abuse contact. Include your domain and the time of the requests.
- Abuse contact
- contact@
sitewalkseo.com - Response time
- within 2 business days