broken links due to cut-off URLs

[Catfish]

Some of my websites have problems because tons of broken links to subpages keep popping up on various sites like gotrendy.org, elgoog.ch, de.ventro.org etc. I have no idea how my links got there. Some of those sites look like search engines, which seem to kind of scrape search results from Google, but show links with incomplete URLs.

That would not be a problem if it wasn't for the fact that they cut off the URLs, hence the broken links.
It seems that my sites get penalized because of this. Google visits site on day xx, webmaster tools lists new 50 broken links next day, visitor numbers drop from 50 to 5 next day. No other changes made on the sites, so I can only assume that it is because of those links.

I can't remove those links from the cache - the removal tool allows you only to remove URLs from your own domain, for obvious reasons. All I can do is keep redirecting those new URLs, but even that is a problem.
If the broken link is for example showing a URL like xxx.mydomain.com/po.. , a redirect would have an effect on all pages with a URL starting with /po . Plus, of course the broken links keep popping up so it is a huge task to stay on top of this.

I do not understand how Google can punish you for links on pages that I do not control???

Anyway, what options do I have here?

[Michael Martinez]

This seems to be a growing, widespread problem. I have seen these broken links popping up in multiple Webmaster Tools reports. I haven't yet discerned a pattern for it but I'm sure it has something to do with RSS-feed scraping.

[GFPC]

This seems to be a growing, widespread problem. I have seen these broken links popping up in multiple Webmaster Tools reports. I haven't yet discerned a pattern for it but I'm sure it has something to do with RSS-feed scraping.

Does it affect ones site? I mean its another site linking to his site incorrectly.

How could that hurt his site?

[Michael Martinez]

The thinking is that if your site has a large number or high percentage of broken URLs when Google and Bing crawl it that they may decide it's a low-quality site.

I have no gut feeling for how many people actually believe this...it's more an idea that someone tossed out maybe last year and has been making its rounds in the SEO community.

I have been advising people to do SOMETHING about significant 404-crawling but only as a pre-emptive measure. I have no data that suggests it actually harms a site in any way to have a large number of 404s. It might be that Matt Cutts or some other Googler said something that has been magnified by repetition and vagueness.

It just seems prudent to me to follow the crawlers and clean up the mess because if crawlers find those links people may find them, too.

[Catfish]

created a similar thread in the Google Webmasters Central, and someone gave me a link Official Google Webmaster Central Blog: Do 404s hurt my site?

But according to what I have seen in the past, that info can't be 100% correct. The ups and downs on my websites are just too obvious.
Every time I fix the crawl errors with redirects, visitor numbers go up for a week, then G finds new 404s, numbers go down again.

[Peter Willis]

It's not ideal, but you could set something to catch all of the 404 errors and just 301 them all to your homepage. It would at least prevent Google seeing tons of 404 errors, and potentially resolve your up and down spikes.

[Catfish]

It's not ideal, but you could set something to catch all of the 404 errors and just 301 them all to your homepage. It would at least prevent Google seeing tons of 404 errors, and potentially resolve your up and down spikes.

Sounds good, checking on how to do this on a Joomla site. Looks like normally you can do that by inserting "ErrorDocument 404 index.php" in the htaccess file. Just with Joomla this doesnt work, but I am sure I can find a way.

Is this kind of redirect going to create any problems SEO-wise?

[Catfish]

Found something for Joomla here. Not sure whether that is a solution for Wordpress as well.

Actually I never thought about this as a general solution to the problem (redirecting all 404s to homepage). Unless there are any negative consequences SEO-wise, that would save a lot of work fixing broken URLs of any kind.
Of course, if I changed the URL of an existing article, I'd want to properly redirect it to the new URL, but for broken URLs that have no "value" (no incoming links pointing to that URL), a general redirect would be great - or am I missing something here?

[Peter Willis]

I mean the main negative is that you are potentially sending dodgy, perhaps irrelevant, links to your homepage. Also it obviously isn't an "ideal" solution in that users will likely be confused if they click a link and get taken straight back to the homepage. It also will make it harder for you to spot any potential issues.

Another option would be to redirect everything a new 404 page (yourdomain.com/404 or yourdomain.com/error - whatever you like as long as it returns a 404 header). That way you wouldn't have loads of bad pages getting crawled and potentially indexed, only the one. And you wouldn't be risking pointing the potentially "bad juice" at the homepage.

I'm afraid I can't help you with the technicals, just the theory