{"id":22206,"date":"2021-07-17T09:32:54","date_gmt":"2021-07-17T00:32:54","guid":{"rendered":"https:\/\/www.souichi.club\/?p=22206"},"modified":"2022-03-19T17:22:55","modified_gmt":"2022-03-19T08:22:55","slug":"htaccess","status":"publish","type":"post","link":"https:\/\/www.souichi.club\/en\/seo\/htaccess\/","title":{"rendered":"Added a setting to .htaccess to prohibit bot traffic from accessing the site."},"content":{"rendered":"<ul class=\"bogo-language-switcher list-view\"><li class=\"en-US en current first\"><span class=\"bogoflags bogoflags-us\"><\/span> <span class=\"bogo-language-name\"><a rel=\"alternate\" hreflang=\"en-US\" href=\"https:\/\/www.souichi.club\/en\/wp-json\/wp\/v2\/posts\/22206\/\" title=\"English\" class=\"current\" aria-current=\"page\">English<\/a><\/span><\/li>\n<li class=\"ja last\"><span class=\"bogoflags bogoflags-jp\"><\/span> <span class=\"bogo-language-name\"><a rel=\"alternate\" hreflang=\"ja\" href=\"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/posts\/22206\/\" title=\"Japanese\">\u65e5\u672c\u8a9e<\/a><\/span><\/li>\n<\/ul>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_83 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a2e993f249ea\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #454e5c;color:#454e5c\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #454e5c;color:#454e5c\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a2e993f249ea\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.souichi.club\/en\/seo\/htaccess\/#bot_traffic\" >bot traffic<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.souichi.club\/en\/seo\/htaccess\/#How_to_check_the_referrer_referrer_source\" >How to check the referrer (referrer source)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.souichi.club\/en\/seo\/htaccess\/#Main_referrers\" >Main referrers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.souichi.club\/en\/seo\/htaccess\/#How_to_specify_exclusion\" >How to specify exclusion<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.souichi.club\/en\/seo\/htaccess\/#Restarting_apache\" >Restarting apache<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"bot_traffic\"><\/span>bot traffic<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In a <a href=\"https:\/\/www.souichi.club\/en\/technology\/bot-traffic\/\" target=\"_blank\" rel=\"noreferrer noopener\">previous article<\/a>, I wrote about a time when I received a large amount of access from bot traffic in one day.<\/p>\n\n\n\n<p>I didn&#8217;t take immediate action at that time, but after that I had several accesses from the same site, so I added a setting to my .htaccss to prohibit access from bot traffic.<\/p>\n\n\n\n<p>The reason why I didn&#8217;t take immediate action at that time was because I had already received information that people were accessing the site by changing the referrer address.<\/p>\n\n\n\n<p>I thought that prohibiting access by specifying the referrer would be a weasel word, so I left it alone for a while.<\/p>\n\n\n\n<p>However, now that I know the tendency and characteristics of referrers to some extent, I decided to exclude them by using regular expressions in &#8220;.htaccess&#8221;.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_check_the_referrer_referrer_source\"><\/span>How to check the referrer (referrer source)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>I checked it with Google Analytics.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Select behavior-&gt; Site Content &gt; All Pages from the menu.<\/li><li>Refine the dates in the target range<\/li><li>Add &#8220;Source\/Medium&#8221; in the Secondary dimension<\/li><\/ul>\n\n\n\n<p>It turns out that &#8220;bot-traffic.icu&#8221; is the referrer.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><a class=\"r_image\" href=\"https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?ssl=1\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?resize=768%2C361&#038;ssl=1\" alt=\"Check referal\" class=\"wp-image-22268\" width=\"768\" height=\"361\" srcset=\"https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?resize=1024%2C481&amp;ssl=1 1024w, https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?resize=300%2C141&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?resize=768%2C360&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?resize=1536%2C721&amp;ssl=1 1536w, https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?resize=520%2C245&amp;ssl=1 520w, https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-en-01.png?w=1920&amp;ssl=1 1920w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/a><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Main_referrers\"><\/span>Main referrers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The referrers that I am aware of are as follows.<\/p>\n\n\n\n<p>They all lead to the same site (I dare not post a screen shot).<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>bot-traffic.icu<\/li><li>bottraffic999.xyz<\/li><li>bottraffic143.xyz<\/li><\/ul>\n\n\n\n<p>Based on the above trend, there is a possibility that they will continue to change the number part (999, 143) to access the site.<\/p>\n\n\n\n<p>They also seem to change the top level of the domain (the icu and xyz parts).<\/p>\n\n\n\n<p>In some cases, &#8220;-&#8221; (hyphen) is inserted between &#8220;bot&#8221; and &#8220;traffic&#8221;, and in other cases, it is not.<\/p>\n\n\n\n<p>After checking the referrers every time, I decided to use regular expressions to specify access-prohibited referrers in &#8220;.htaccess&#8221; because it is troublesome to add them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_specify_exclusion\"><\/span>How to specify exclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The following description was added to the &#8220;.htaccess&#8221; file.<\/p>\n\n\n<pre><code class=\"lang-shell\">RewriteEngine on\nRewriteCond %{HTTP_REFERER} bot(|-)traffic(|[0-9]{3}).... [NC]\nRewriteRule .* - [F,L]<\/code><\/pre>\n\n\n<p>The RewriteEngine on line is already there, so only RewriteCond and RewriteRule are added.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><a class=\"r_image\" href=\"https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-01.png?ssl=1\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-01.png?resize=448%2C103&#038;ssl=1\" alt=\"RewriteCond\u3068RewrireRule\u306e\u8ffd\u52a0\" class=\"wp-image-22141\" width=\"448\" height=\"103\" srcset=\"https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-01.png?w=597&amp;ssl=1 597w, https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-01.png?resize=300%2C69&amp;ssl=1 300w\" sizes=\"auto, (max-width: 448px) 100vw, 448px\" \/><\/a><\/figure>\n\n\n\n<p>Parameter Meaning.<\/p>\n\n\n<figure class=\"wp-block-table is-style-regular\">\n<table class=\"r_table01\">\n<tbody>\n<tr>\n<td class=\"r_td30\">\n<p>RewriteCond<\/p>\n<\/td>\n<td>\n<p>Specify the match condition<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td class=\"r_td30\">\n<p>%{HTTP_REFERER}<\/p>\n<\/td>\n<td>\n<p>Indicates a referrer<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>(|-)<\/p>\n<\/td>\n<td>\n<p>A regular expression, representing none or &#8220;-&#8221; (hyphen).<\/p>\n<p>&#8220;|&#8221; (pipe) represents an or condition.<\/p>\n<p>&#8220;()&#8221; (parentheses) represents a group.<\/p>\n<p>I chose this specification because there were both patterns, &#8220;bot-traffic&#8221; and &#8220;bottraffic&#8221;.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>(|[0-9]{3})<\/p>\n<\/td>\n<td>\n<p>A regular expression representing a none or three-digit number.<\/p>\n<p>[0-9]: number<br>{3}: 3 digits<br>This is the case when there are no digits specified, 999, and 143, but if the number of digits changes, it needs to be adjusted.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>\\.<\/p>\n<\/td>\n<td>\n<p>&#8220;\\&#8221; (backslash) specifies an escape sequence (the subsequent character is not determined as a regular expression character).<\/p>\n<p>&#8220;.&#8221; (dot) has the meaning of any single character in a regular expression.<\/p>\n<p>In this case, &#8220;.&#8221; (dot), I want it to be recognized as a regular expression character, so I specify an escape sequence.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>&#8230;<\/p>\n<\/td>\n<td>\n<p>This is a regular expression representing any three characters.<\/p>\n<p>I used this specification because there are cases where the top level of the domain is icu and cases where it is xyz.<\/p>\n<p>If the number of patterns does not increase in the future, (icu|xyz) may be acceptable.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>[NC]<\/p>\n<\/td>\n<td>\n<p>Not case-sensitive<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>RewriteRule<\/p>\n<\/td>\n<td>\n<p>Specifying URL conversion rules<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>.*<\/p>\n<\/td>\n<td>\n<p>Above, I want to target the referrers of all bot traffic matched by RewriteCond, so I specify an arbitrary string (0 to n characters) that matches all of them.<\/p>\n<ul>\n<li>&#8220;.&#8221; (dot): any single character<\/li>\n<li>&#8220;*&#8221; (asterisk): zero or more repetitions of the previous character<\/li>\n<\/ul>\n<p>^(. *)$, but it basically means the same thing.<\/p>\n<ul>\n<li>&#8220;^&#8221; (caret): the character immediately following is the beginning of the line<\/li>\n<li>&#8220;$&#8221; (dollar): the character immediately before is the end of the line<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>&#8211;<\/p>\n<\/td>\n<td>\n<p>Specify that no rewrite (URL conversion) should be done.<\/p>\n<p>Since this is bot traffic, there is no need to rewrite the URL, so &#8220;-&#8221; (hyphen) is specified.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>[F]<\/p>\n<\/td>\n<td>\n<p>Specify the access forbidden (403-Forbidden).<\/p>\n<p>Disable access to bot traffic.<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p>[L]<\/p>\n<\/td>\n<td>\n<p>Ignore all rules after that.<\/p>\n<p>I specify this because I don&#8217;t need to apply this rule to bot traffic even if I add RewriteCond under this rule in the future.<\/p>\n<p>If you specify [F], it will say &#8220;Ignore subsequent rules&#8221;, so there is no need to specify L, but I specified it so that you can see it explicitly.<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Restarting_apache\"><\/span>Restarting apache<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>After modifying and saving the .htaccess, restart apache with the following command.<\/p>\n\n\n<pre><code class=\"lang-shell\">sudo service apache2 restart<\/code><\/pre>\n\n\n<p>That concludes this article.<\/p>\n\n\n<div class=\"postscript_balloon\">Finally.<\/div>\n\n\n<p>I hope this article will be useful to someone somewhere.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There was a large amount of access from bot traffic in one day. <\/p>\n<p>Added a setting to .htaccss to disallow access from bot traffic.<\/p>\n","protected":false},"author":5,"featured_media":22143,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_locale":"en_US","_original_post":"https:\/\/www.souichi.club\/?p=22137","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[45],"tags":[],"class_list":["post-22206","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo","en-US"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.souichi.club\/wp-content\/uploads\/htaccess-02.png?fit=1859%2C900&ssl=1","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/posts\/22206","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/comments?post=22206"}],"version-history":[{"count":1,"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/posts\/22206\/revisions"}],"predecessor-version":[{"id":24531,"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/posts\/22206\/revisions\/24531"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/media\/22143"}],"wp:attachment":[{"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/media?parent=22206"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/categories?post=22206"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.souichi.club\/wp-json\/wp\/v2\/tags?post=22206"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}