Manual Shopify Crawler Settings
SEMRush is useless with my Shopify site because I cannot control the robots.txt. Since Shopify is growing, something should be done. Could the crawl-delay setting be something I can set manually, so more value could be derived from SEMrush? Or, could there be a setting for eCommerce - Platform / if Shopify is selected, then have the bot operate differently.
Thanks for your time and consideration.
Joseph Terzieva commented
I'm excited to see that this is planned. This issue ultimately caused me to look for other options for SEO, but I came back because I couldn't find anything better. This is the single piece of SEMRush for Shopify users that hurts our ability to maximize site quality.
This. Or add user agent choice that emulates Majestic so it will use the Majestic crawl rate delay directive (appears to be 12) in the default Shopify robots.txt
Same issue here I have tried all of the crawler delay options, none of them gave an improvement. As we have no control over the robots file or the security systems shopify have in place, can you (SEMRush) add an option for maybe 1 URL per 30 seconds or per minute for example so we can test?
I would rather the audit took a whole day to complete than not working
Lothar Bongartz, 1der1.com commented
It seems semrush still doesn't get along with Shopify pages.
There should be a communication between semrush and Shopify...
Chris Eckman commented
I get a lot of 430 errors from scanning shopify sites.
It looks like if you slow down the crawler to pause every 30 seconds you can fix some of these issues. Is there anyway to do that now, and if not, can you add that? I know that it has to respect the robots, but
Is this still under review a year later? I'm a Shopify expert and would like to use SEMrush to help my clients... the error's stop me from doing that.
This is keeping me from using SEMRush
Niall O'Meara commented
I have a Shopify site and can't get a decent site audit because I can't control the rate limit of the crawl.
Please would you allow me to control the crawl rate within SEMRush?
I know hundreds of other people have been asking for this for well over a year now.
Please please get this sorted. I don't want to switch to another SEO tool really but fear I may have to.
Please get back to me.
We are using SEMrush on a Shopify site and because of Shopify's API limits, we're getting a bunch of 430 HTTP errors. We could exclude all internal broken links, or 4xx errors to avoid seeing these, but then we might miss "legitimate" errors.
I would like to suggest that exclusion be done on a per HTTP code basis. This may help situation like the one described here: https://semrush.uservoice.com/forums/214396-general/suggestions/8735614-ability-to-sort-by-http-code-in-broken-external-l