Google Guy's Warnings About "Automated Search Engine Queries"
Following exchanges in the Google section of the webmaster forum at seochat.com about automated software queries to search engines such as Google, I thought of inviting WebPositionGold for their comments. Here is what David Elliot of FirstPlace Software, Inc. had to say:
To address the main issue discussed in the forum, Google states that they do not like automated queries to be run on their service. They say accessing their service with a browser is fine, but all other tools that automate searching are not. We updated WebPosition a year or more ago to emulate a browser as closely as possible to avoid unfair discrimination toward WebPosition users. There is no other tool on the market that does a better job of emulating a browser than WebPosition.
You may hear of a Web site being banned by Google, but normally it takes some form of spamming on the person's Web site for that to happen, not simply running WebPosition's Reporter. That's because WebPosition Gold 1.60 as well as WebPosition Gold 2 do NOT pass the user's URL to Google when running keyword reports or verifying URLs. In the case of URL verification, it does a query on a keyword phrase and then scans the results for the user's domain name offline.
Therefore, Google's server cannot look at rank checking queries from your IP address and say they belong to a particular Web site which is on another IP and ban it. Although they can certainly launch scare tactics to make you believe that if you check your rankings your Web site will be banned, that has not been the case as we've observed it. If it were, we'd be hearing a flood of complaints from our customers about being banned and that is just not happening.
Earlier versions of WebPosition Gold 1.x, mainly at the end of 2000 and first month or so of 2001 did pass the domain on URL searches but this has long since been changed. Most people saying that using WebPosition will get your IP blocked are basing their belief on those incidences. The browser emulation problems that led to a small number of people being banned have long since been corrected. Even at that time, less than 1% of WebPosition customers were even affected. However, the rumours continue to circulate anytime someone is dropped from a search engine (which happens all the time whether you use WebPosition or not) that WebPosition must be responsible. WebPosition's Page Critic will actually tell you how to avoid accidentally spamming a search engine, which is by far the easiest way to get banned by a search engine.
As a side note, FirstPlace Software has made numerous communication attempts to Google's management in response to their complaint at the end of 2000. For a period of six months, those attempts were ignored and put off. After extraordinary efforts, we did eventually receive a single response from a Google VP. We sent a detailed proposal of how we might work together to encourage the reduction of queries on their service, discourage abuse, promote their Web marketing services such as AdWords, and address other issues they might have if they could clearly spell them out. Some of these things required cooperation on their end to accomplish.
Unfortunately, our proposal was not responded to by their VP. So considering the six-month history of calls, faxes, and letters being ignored, we deemed that the matter was in reality not that important to them and thereby dropped the matter. Since we updated the product long ago, we've had no substantiated reports of WebPosition customers being blocked or banned simply for running rank checks or for submitting, so it's been a non-issue.
If you do use an automated tool of any kind, you should try to be sensitive to the needs of a search engine and not abuse their service. More specifically:
a) Avoid excessive numbers of queries if you choose to monitor your rankings on Google. Most people do not have time to improve their rankings on 100's of keywords. Therefore, don't rank check on 100's of keywords if you don't have the time to do anything about all those rankings anyway.
b) If you choose to run queries, run your queries at night and during off-peak periods, which is something Google has suggested in the past. This is when many of their servers are presumably standing idle, waiting to handle the increased volume during peak periods. Our scheduler makes this easy to do. The scheduler ease of use and flexibility have been
improved in WebPosition Gold 2 to further this goal.
c) Do not run your queries more often than is really necessary. Since Google normally doesn't update their entire index more than once a month, you should not have a strong reason to check your rankings more often than that.
d) I have heard that some people who are concerned about Google's statements choose to monitor their Google positions via the Yahoo Web Matches option in the Reporter. Although these rankings can vary a bit from Google.com, normally because the index is not always as up to date, it can be a reasonable alternative for some people.
e) Avoid spamming on your Web site (same color text as background, excessive keyword use, etc.) since this is by far the quickest and easiest way to get yourself red-flagged whether you use WebPosition or not. WebPosition's Page Critic gives extensive advice in this regard and is updated monthly so you can achieve top rankings through "search engine friendly" web pages.
f) We admit that we do have some concerns that Google's Toolbar program that some people use could be used to set cookies or other things to help identify people conducting certain activities. We have seen a forum posting earlier this year that indicated that Google may have used the tool to track down the author of a product that reported PageRank outside of the official toolbar program and demand he stop publishing such a tool.
Therefore, although WebPosition does not record cookies or things that could easily associate your domain name to queries you do, we cannot vouch for what information the Google toolbar could communicate to Google's server. If you want to error on the side of caution, consider using the toolbar on a separate machine and IP from all your other SEO work, whether using WPG or otherwise, or not using it at all. (If you never downloaded their toolbar and installed it in your browser, then you don't need to be concerned about this).
In the case of WebPosition Gold 2, you can use the "Be courteous to the search engines" feature on the Options tab of the Reporter so you do not query their service so quickly. This gives you an added level of safety you will not find elsewhere if you don't mind the missions taking longer to run. The submitter has a similar feature to randomly submit at various intervals to more precisely emulate manual submissions in your browser so that you're not discriminated against simply for using WebPosition rather than your browser to submit.
We recognize that online businesses have a fundamental need to measure their search engine rankings and automate submissions to save time and increase accuracy. This can be particularly true if you are paying a search engine to be included in their index. You can't invest time or money into a marketing effort without having a tool to measure your results.
Part of that measurement comes from measuring traffic, but another part comes from measuring your rankings and whether they improve or decline over time. Whether you choose to use a tool like WebPosition to assist you in that, another tool, your browser, or not at all is ultimately up to you as the marketing manager. Ensuring your Web site is visible on the search engines is an effective, and for some businesses, essential marketing method. The desire to be found on the search engines is not one that is likely to ever change.
Keep in mind that if you do nothing, then your Web site is not likely to be found by anyone on the search engines, so in affect, you are already "banned" for all intensive purposes. WebPosition is the very best rated tool out there for creating a search engine friendly Web site that will rank highly. You can find many reviews and testimonials at the following links:
In conclusion, we are not receiving reports to my knowledge of customer's IP's or Web sites being blocked or banned. I also want to say Google has an outstanding service that should truly be applauded. The accuracy of their search results are some of the best in the business! Therefore, we by no means wish to promote the abuse of Google or any other service.
We do hope Google realizes that the same people that financially support Google by buying advertising via AdWords and other services also have a need to measure their rankings. If they don't do that with the tool we offer, they will simply use some other tool, so this is not simply a "WebPosition" issue. With that in mind, at least in the case of FirstPlace Software, it is our desire to promote the responsible use of our product, to discourage search engine spamming, and to work together with search engines and the Web marketing community whenever possible. Most search engines realize this, but in the case of Google, they have a different viewpoint.
I hope this helps clarify things a bit. I apologize for being long-winded, but I wanted to give you as much information as I could so you're better informed and can make intelligent decisions on promoting your Web site. Let me know if you have any other questions or concerns.
FirstPlace Software, Inc.
For the fastest answers to commonly asked questions about WebPositionGold see:
For more information, check out this article by Matt Paolini: "Dispelling the Myths - Will WebPostion Get My Site Banned from Google?"