Several weeks ago, I received this e-mail from Google about our Google Products search listings…
Hello,
In order to create the best experience for our users, Google verifies all product listings that are submitted to the Google Merchant Center. For example, we make sure that listings contain all required attributes, such as ‘price’ and ‘link.’ We also crawl these links to ensure that the pages you’ve specified exist and represent the correct products. When attempting to access some of the links that you provided, we found that a robots.txt file is preventing us from crawling them.
In order to resolve this, please update your robots.txt file to allow us to crawl your product pages by February 28, 2010. After this date, we will suspend any listings that we’re not able to crawl.
The below scenarios are the most common reasons why we may be unable to crawl your links:
1. You’re submitting tracking URLs which you do not want indexed. To resolve this issue, you may include the ‘rel=canonical’ tag in the pages you submit. Please visit http://googlemerchantblog.blogspot.com/2009/12/make-your-results-look-better-on.html for additional details. Please also note that in addition to including the ‘rel=canonical’ tag, you’ll also need to update your robots.txt file (see point 2).
2. All of the content you’re submitting has been roboted. Please be sure that the user-agent ‘Googlebot’ is not being blocked from crawling your website or product pages. To ensure the ‘Googlebot’ is not being blocked, please add the following two lines of text to the end of your robots.txt file:
User-agent: Googlebot
Disallow:
For more information on robots.txt files, please visit http://www.robotstxt.org. If you have any questions, please contact your webmaster directly.
3. Some of the items you’re submitting no longer exist and are redirecting to a roboted error page. Please remove any unavailable content from your feed and then resubmit it. Additionally, please be sure to update and submit your data feed as often as your items’ information changes — up to once per day.
After you’ve made the changes, please upload your items again. If you’re using the scheduling feature to submit your feed, we recommend that you upload the corrected feed manually before your next scheduled upload. Additionally, please note that your feed is automatically re-evaluated and therefore there’s no need to contact us confirming the resubmission of your items. If you have any questions, please visit our Help Center at http://www.google.com/support/merchants to find answers to frequently asked questions.
Sincerely,
The Google Team
Hmm not cool. It is very important to me to be able to track my traffic from Google Products…without being able to track it I wouldn’t have known about it’s crazy growth.
It looked like we had two options. The first was to remove Disallow: /*? from our robots.txt. The second option was to remove the tracking code from the feed. I felt like editing our robots.txt would open the flood gates…I don’t want the search engines to crawl my URLs with tracking codes or URLs with sorting and/or filtering options.
So I asked our IT guy to remove the tracking code from the feed…I figured Google Analytics would just treat it as a referral. Not the case…looking at the stats from this past weekend, it appears Google is treating traffic from Google Products as organic Google traffic.
I don’t understand why Google can’t separate the two, especially because both are THEIR products.
I have until the end of the month to troubleshoot the issue…unfortunately it looks like my only option is to edit the robots.txt.
Related posts:
- Google Products, a Necessity for Ecommerce Companies What kind of SEM doesn’t love a good quality source...
- Adding Your Sitemap to the Robots.txt From time to time I spent days doing nothing but...
- Indexed Pages Jump Since we’ve launched the new AS website we had a...
- Google Webmaster Tools Parameter Handling Has anyone used this yet? I’m really interested in it...
Related posts brought to you by Yet Another Related Posts Plugin.