Portal Home > Knowledgebase > Articles Database > HTTP/1.1 403 Forbidden [googlebot]
Posted by markgelo, 04-14-2014, 11:39 PM i really need your help.. my websites ranking is now loosing because of this error, and i don't know how to fix this.. even my web hosting provider can't fix this. when i fetch as google there's an error and also when i view my website via googlebot 2.1 using user agent switcher i got that error too, and when i test my sitemaps on webmaster tool, i got and hope you guys help me.. this is my .htaccess Bthis i my website: http://www.naruto-tv.info all subdomain and other domain that i'm using right now had a same problem and hosted on same server. but i have one subdomain that hosted on other server and working fine. ERROR LOGS: BTW!! i'm using shared hosting.
Posted by fabin, 04-15-2014, 12:08 AM Is the begin WordPress and end WordPress lines properly commented in .htaccess? Btw, the website works for me. Too many ads.
Posted by markgelo, 04-15-2014, 01:14 AM SORRY FOR THE ADS, yeah website is running but googlebot view my website as 403 error forbidden. try to view my webiste via google bot 2.1 agent. and you will know what i mean.
Posted by Kailash12, 04-15-2014, 01:20 AM Did you block any IP address using .htaccess or did you install any security plugin which can block the IP addresses? From the error, it is clear that Googlebots IP addresses are either blocked by your WordPress plugin, .htaccess or at server firewall.
Posted by markgelo, 04-15-2014, 01:36 AM i didn't blocked google bot, and my hosting provider said that they didn't blocked googlebot too. i already disabled all of my plugins, but still the same., i tried to transfer my other sub domain to other web hosting provider and everything is workiing fine. so what do you think? if you are not busy can u help me? i will pm my cpanel account to you to check?
Posted by Kailash12, 04-16-2014, 03:19 AM If you or your hosting provider are unable to find the block for Googlebots IPs, you are left with no other option than to change your hosting provider.
Posted by fabin, 04-16-2014, 03:30 AM Is there only those three lines in your .htaccess file? Is there any other .htaccess files in parent folders. This has to be some issue with .htaccess itself.
Posted by W3USR, 04-16-2014, 04:54 AM Your error is not related to Google bot IP. Your Wordpress/Server is blocking the Google bot user agent. MSN bot/Baidu too in your block list, Yahoo slurp works fine. Last edited by W3USR; 04-16-2014 at 05:00 AM.
Posted by zoid, 04-16-2014, 05:11 AM That is correct.
Posted by markgelo, 04-16-2014, 11:47 PM but my server said that they didn't block anything, and also i tried to disable all plugins that i have but still the same.. but when i point my domian to other server everything is working fine, but i can't host my naruto-tv.info to my old hosting provider , if i do that i will violate their TOS. and besides they can't refund my money if i move other offshore web hosting.
Posted by markgelo, 04-17-2014, 12:04 AM this is whole .htaccess RewriteOptions inherit RedirectMatch 301 ^/([0-9]{4})/([0-9]{2})/([^/]+).html$ http://www.naruto-tv.info/$3 # BEGIN WPSuperCache RewriteEngine On RewriteBase / #If you serve pages from behind a proxy you may want to change 'RewriteCond %{HTTPS} on' to something more sensible AddDefaultCharset UTF-8 RewriteCond %{REQUEST_URI} !^.*[^/]$ RewriteCond %{REQUEST_URI} !^.*//.*$ RewriteCond %{REQUEST_METHOD} !POST RewriteCond %{QUERY_STRING} !.*=.* RewriteCond %{HTTP:Cookie} !^.*(comment_author_|wordpress_logged_in|wp-postpass_).*$ RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTProfile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTP_USER_AGENT} !^.*(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800).* [NC] RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC] RewriteCond %{HTTP:Accept-Encoding} gzip RewriteCond %{HTTPS} on RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html.gz -f RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html.gz" [L] RewriteCond %{REQUEST_URI} !^.*[^/]$ RewriteCond %{REQUEST_URI} !^.*//.*$ RewriteCond %{REQUEST_METHOD} !POST RewriteCond %{QUERY_STRING} !.*=.* RewriteCond %{HTTP:Cookie} !^.*(comment_author_|wordpress_logged_in|wp-postpass_).*$ RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTProfile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTP_USER_AGENT} !^.*(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800).* [NC] RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC] RewriteCond %{HTTP:Accept-Encoding} gzip RewriteCond %{HTTPS} !on RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html.gz -f RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html.gz" [L] RewriteCond %{REQUEST_URI} !^.*[^/]$ RewriteCond %{REQUEST_URI} !^.*//.*$ RewriteCond %{REQUEST_METHOD} !POST RewriteCond %{QUERY_STRING} !.*=.* RewriteCond %{HTTP:Cookie} !^.*(comment_author_|wordpress_logged_in|wp-postpass_).*$ RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTProfile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTP_USER_AGENT} !^.*(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800).* [NC] RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC] RewriteCond %{HTTPS} on RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html -f RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html" [L] RewriteCond %{REQUEST_URI} !^.*[^/]$ RewriteCond %{REQUEST_URI} !^.*//.*$ RewriteCond %{REQUEST_METHOD} !POST RewriteCond %{QUERY_STRING} !.*=.* RewriteCond %{HTTP:Cookie} !^.*(comment_author_|wordpress_logged_in|wp-postpass_).*$ RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTProfile} !^[a-z0-9\"]+ [NC] RewriteCond %{HTTP_USER_AGENT} !^.*(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800).* [NC] RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC] RewriteCond %{HTTPS} !on RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html -f RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html" [L] # END WPSuperCache # BEGIN WordPressRewriteEngine On RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L] # END WordPress RewriteCond %{HTTP_HOST} ^.*$ RewriteRule ^\(\[0\-9\]\{4\}\)\/\(\[0\-9\]\{2\}\)\/\(\[\^\/\]\+\)\.html$ "http\:\/\/markgiver\.naruto\-tv\.info\/\$3" [R=301,L] RewriteCond %{HTTP_HOST} ^.*$ RewriteRule ^\(\[0\-9\]\{4\}\)\/\(\[0\-9\]\{2\}\)\/\(\[\^\/\]\+\)\.html$ "http\:\/\/www\.naruto\-tv\.info\/\$3" [R=301,L]
Posted by markgelo, 04-17-2014, 01:15 AM QUOTE=markgelo;9085075]this is whole .htaccess
Posted by W3USR, 04-17-2014, 04:28 AM From wp-super cache settings turn caching off, click the 'Delete Cache' option, then disable the plugin. If that didn't work, create a backup of your .htaccess file, and try the following rules. ################ RewriteOptions inherit RedirectMatch 301 ^/([0-9]{4})/([0-9]{2})/([^/]+).html$ http://www.naruto-tv.info/$3 # BEGIN WordPress RewriteEngine On RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L] # END WordPress RewriteCond %{HTTP_HOST} ^.*$ RewriteRule ^\(\[0\-9\]\{4\}\)\/\(\[0\-9\]\{2\}\)\/\(\[\^\/\]\+\)\.html$ "http\:\/\/markgiver\.naruto\-tv\.info\/\$3" [R=301,L] RewriteCond %{HTTP_HOST} ^.*$ RewriteRule ^\(\[0\-9\]\{4\}\)\/\(\[0\-9\]\{2\}\)\/\(\[\^\/\]\+\)\.html$ "http\:\/\/www\.naruto\-tv\.info\/\$3" [R=301,L]
Posted by markgelo, 04-17-2014, 04:47 AM still the same
Posted by W3USR, 04-17-2014, 03:26 PM Create a backup and try re-installing wordpress, test it without any plugins, or you can move all of your wordpress files to another folder, create an index.html file and test again as Googlebot.
Posted by markgelo, 04-17-2014, 07:16 PM hi.. still the same.
Posted by JacobN, 04-17-2014, 11:54 PM Hey markgelo, here's how I'd test it out to see where the problem might be: If you're checking your .htaccess file in your /public_html directory, go up one level to your home and check for one at /home/markgelo/.htaccess which could be overriding the one WordPress is using. At the top of your WordPress .htaccess file place this code: Try to access your site with a GoogleBot user-agent, do you see My Block text, or still the General HTTP error: HTTP 403 error (Forbidden) HTTP Error: 403 one? Based off your error logs, it would seem the 403 is getting caused at the Apache level instead of your .htaccess file, here's one of yours: Where typically on my servers I'll see something like: Apache usually points out the .htaccess file that is handling the server configuration, unless the config is set server-wide at the Apache level, or if a ModSecurity rule is getting triggered (but this usually results in a 406 error when properly configured). You also might want to look at your site in Google Chrome developer tools (open a new tab, hit F12, then visit your site). It seems like your scripts are going through some errors, and especially this part about iframes could possibly be affecting bots trying to crawl your site properly: Hopefully that will help you confirm if it is indeed coming from your .htaccess file at all. Let me know what you find out!
Posted by markgelo, 04-18-2014, 12:23 AM hello.. i add this code on .htaccess ErrorDocument 403 "My Block" and My Block appear.
Posted by markgelo, 04-18-2014, 12:26 AM i don't see any 403 error forbidden when i add that code. and you are right! when i view my website via googlebot "My Block" appear, what does it mean?
Posted by markgelo, 04-18-2014, 12:41 AM one more thing!!! i only added that code to my main domain .htaccess but, all subdomian are affected even without adding that code into their htaccess
Posted by JacobN, 04-18-2014, 02:23 AM Did you find a /home/markgelo/.htaccess file? Being that you are seeing the My Block message now means if you check your error log you will probably see that .htaccess file mentioned as doing the client denying. If you do see that, it verifies what I said in my first reply that the 403 is being caused at the server level and not in your .htaccess file. Either that or you have a WordPress plug in that is sending the 403 error. You can try simply renaming your /wp-content/plugins directory to /wp-content/plugins-bak then hit your site again with GoogleBot. Then just rename the /plugins folder back again, and if it did work with all plugins disabled try disabling them one at a time.
Posted by chiragpatel, 04-18-2014, 02:28 AM Can you share the content of /robots.txt ????
Posted by markgelo, 04-18-2014, 02:48 AM actually i didn't see any .htaccess on /home/markgelo/ i added that code on /home/markgelo/public_html/.htaccess i already did what so said about plugins. but still the same
Posted by markgelo, 04-18-2014, 02:53 AM User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/
Posted by JacobN, 04-18-2014, 03:14 AM Have you tried temporarily renaming your .htaccess file yet? If it's not that or a WordPress plugin then I would assume it's a server wide block. Can you setup a separate addon domain on the same server with just a plain HTML index file and see if that is also having issues?
Posted by chiragpatel, 04-18-2014, 03:19 AM It seems your robots.txt is perfect... Now in Wordpress can you check following thing??? In Dashboard > settings > Reading > Discourage search engines from indexing this site. Please make sure you have unmarked this option
Posted by The Dude, 04-18-2014, 03:24 AM I was just gonna say that..... IT MIGHT BE THE HOST BLOCKING GOOGLE FROM INDEXING SITES THEY HOST!!
Posted by chiragpatel, 04-18-2014, 03:35 AM Also please check your sever CSF firewall (if installed) there you will need to modify mod security. Google for "CSF google bot blocking" and you will get so many articles.
Posted by markgelo, 04-18-2014, 03:45 AM when i set new subdomain, everything is fine, googlebot can crawl it. damn!! what is the real caused of this. tsk!!
Posted by markgelo, 04-18-2014, 03:48 AM i don't have anything like that on my cpanel.
Posted by markgelo, 04-18-2014, 03:51 AM it's already unmarked when i view this option.
Posted by markgelo, 04-18-2014, 03:55 AM yeah, i want that too, but i have 3 months subscription on them and they can't refund my money.
Posted by zoid, 04-18-2014, 09:25 AM The provider isnt blocking anything.
Posted by W3USR, 04-19-2014, 01:03 AM Create a blank subdomain like http://test.naruto-tv.info and test it with google bot useragent. If you are getting the same error, show it to your hosting provider.
Posted by markgelo, 04-19-2014, 09:18 AM no error, and they already know that. and still they can't fix it.
Posted by zoid, 04-19-2014, 09:31 AM I am afraid you need to fix it, not they.
Posted by Steven, 04-19-2014, 12:55 PM Did some testing here: As figured out earlier, it does not like the google useragents: So I tested against an image on your site: This was also denied. So I think we can rule out it being denied in wordpress/php since the image is not being served via PHP. So it either has to be a block at your host or in your htaccess (which it does not appear to be).
Posted by zoid, 04-19-2014, 01:28 PM Precisely, however the host is most unlikely the reason as other sites on the same machine serve such requests without issues.
Posted by markgelo, 04-28-2014, 07:53 AM i really don't know how to fix this, and they can't fix this too.
Add to Favourites 
Print this Article
