How to deal with Spam Issues and Gmail?

[americanninja]

I'm currently having issues with gmail limiting my server from passing email through due to it thinking my server is sending a lot of spammy (unsolicited) emails.

I see this showing up in the syslog:

May 2 19:13:19 panel postfix/smtp[27422]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400e:c03::1a]:25: Network is unreachable
May 2 19:13:20 panel postfix/smtp[27422]: 438B5142BE7: host gmail-smtp-in.l.google.com[74.125.25.27] said: 421-4.7.0 [104.xxx.xxx.xxx 15] Our system has detected an unusual rate of 421-4.7.0 unsolicited mail originating from your IP address. To protect our 421-4.7.0 users from spam, mail sent from your IP address has been temporarily 421-4.7.0 rate limited. Please visit 421-4.7.0 http://www.google.com/mail/help/bulk_mail.html to review our Bulk 421 4.7.0 Email Senders Guidelines. ht8si11619298pdb.99 - gsmtp (in reply to end of DATA command)

So I think this is happening due to spam emails being sent to my email addresses on my server which are forwarded to my gmail account. So gmail thinks my server is sending these spam emails right?

For example, I have a several emails addresses setup like email1@mydomain.com, email2@mydomain.com, and email3@mydomain.com. I have all these email addresses setup to simply forward to my gmail email of myaddress@gmail.com. This is so I have all email delivered to one common email box. So when spam emails get sent to email1@mydomain.com, they are forwarded along to my gmail address and delivered through to gmail. I'm assuming that gmail is now seeing my server IP as spammy, because it sees me passing along these spammy email addresses. I had the same setup before with cpanel, and I don't know if it was a problem, but I don't think it was. Is there a way to inform gmail that this email is simply being forwarded along, so that it doesn't think my server IP address is spammer or something?

Here is what I see when I run mailq

root@panel:~# mailq
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
438B5142BE7 971 Sat May 2 15:06:57 HansBanta@abehide.com
(host alt1.gmail-smtp-in.l.google.com[64.233.191.26] said: 421-4.7.0 [104.xxx.xxx.xxx 15] Our system has detected an unusual rate of 421-4.7.0 unsolicited mail originating from your IP address. To protect our 421-4.7.0 users from spam, mail sent from your IP address has been temporarily 421-4.7.0 rate limited. Please visit 421-4.7.0 http://www.google.com/mail/help/bulk_mail.html to review our Bulk 421 4.7.0 Email Senders Guidelines. a20si605283igr.2 - gsmtp (in reply to end of DATA command))
-- 1 Kbytes in 1 Request.


And here is this email in the system:

root@panel:~# postcat -qv 438B5142BE7
postcat: name_mask: all
postcat: inet_addr_local: configured 2 IPv4 addresses
postcat: inet_addr_local: configured 2 IPv6 addresses
*** ENVELOPE RECORDS deferred/4/438B5142BE7 ***
message_size: 971 735 1 0 971
message_arrival_time: Sat May 2 15:06:57 2015
create_time: Sat May 2 15:06:57 2015
named_attribute: log_ident=438B5142BE7
named_attribute: rewrite_context=remote
sender: HansBanta@abehide.com
named_attribute: log_client_name=142-0-69-56.static.avestadns.com
named_attribute: log_client_address=142.0.69.56
named_attribute: log_client_port=44907
named_attribute: log_message_origin=142-0-69-56.static.avestadns.com[142.0.69.56]
named_attribute: log_helo_name=abehide.com
named_attribute: log_protocol_name=ESMTP
named_attribute: client_name=142-0-69-56.static.avestadns.com
named_attribute: reverse_client_name=142-0-69-56.static.avestadns.com
named_attribute: client_address=142.0.69.56
named_attribute: client_port=44907
named_attribute: helo_name=abehide.com
named_attribute: protocol_name=ESMTP
named_attribute: client_address_type=2
warning_message_time: Sat May 2 19:06:57 2015
named_attribute: dsn_orig_rcpt=rfc822;myemail@mydomain.com
original_recipient: myemail@mydomain.com
recipient: myemail@gmail.com
*** MESSAGE CONTENTS deferred/4/438B5142BE7 ***
regular_text: Received: from abehide.com (142-0-69-56.static.avestadns.com [142.0.69.56])
regular_text: by panel.mydomain.com (Postfix) with ESMTP id 438B5142BE7
regular_text: for <myemail@mydomain.com>; Sat, 2 May 2015 15:06:57 +0900 (JST)
regular_text: Message-ID: <399a01d0845b$23f5b520$6cb63faa@HansBanta>
regular_text: From: "Matthew" <HansBanta@abehide.com>
regular_text: To: "Matthew" <myemail@mydomain.com>
regular_text: Subject: What is your most trusted way to improve your sexual performance? Believe me, it can be improved!
regular_text: Date: Fri, 01 May 2015 22:06:57 -0700
regular_text: MIME-Version: 1.0
regular_text: Content-Type: text/plain;
regular_text: charset="us-ascii"
regular_text: Content-Transfer-Encoding: 7bit
regular_text: X-MSMail-Priority: Normal
regular_text: X-Mailer: Microsoft Outlook, Build 10.0.2627
regular_text: X-MimeOLE: Produced By Microsoft MimeOLE V10.0.2627
regular_text:
regular_text: These amazing medicines will bring you back to life
regular_text: You won?t be able to keep your boner down with these pills
regular_text: Check out our store and find your favorite!
regular_text:
regular_text: Get FREE tablets right now!
regular_text:
regular_text: http://zzb.bz/xF8qz
regular_text:
regular_text: Only World licensed remedies
regular_text:
regular_text:
*** HEADER EXTRACTED deferred/4/438B5142BE7 ***
*** MESSAGE FILE END deferred/4/438B5142BE7 ***

[apinto]

Greetings, americanninja and welcome to Sentora Forums.
It seems to me you have to setup a more strict Spam Filtering (see "Ubuntu 14.04 | Install and Configure Postfix + Amavisd-new + ClamAV + Spamassassin" ).

From what I can tell Gmail is greylisting you due to the amount of spam you sent to Gmail, probably on your previsous Cpanel Setup you had strict spam rules (honestly Sentora defaults are not optimal to filter spam).

On the most basic level you can try to enable the RBL filtering (I do not recommend using this without testing because it can cause a lot of false positives) you can see a list of some RBL's on Wikipedia, remember that using RBL's you need to keep checking if they are being updated or not, the default "SpamHaus" is a safe one to use (still with false positives).

After you reduce the amount of Spam sent to Gmail servers you should be removed from the greylisting with time...


Note: I hope I understood your problem correctly, if not please tell me

[americanninja]

Thank you apinto! I will have to give that a try. I was worried about adding on software to the server after I got everything working with Sentora (i.e. in case it messes things up with Sentora and brings down the web server). Is it risky to do this? I'm assuming anything you install that's connected to the web server functions could cause trouble for Sentora, as Sentora is managing the server. Please let me know.

As for the gmail greylisting, perhaps it's only the spammy emails that it's doing this? For example I just tested a few features on my drupal site which sends out emails to users based on things they do on the site. I have a test user on my Drupal site with one of my emails setup on my server. myemail@mydomain.com. I ran through and did a few things on the website that I know would trigger emails and those immediately came through to my gmail account (from my server via Drupal -> myemail@mydomain.com -> myemail@gmail.com).

These came through instantaneously, so no issues receiving them to gmail. And if I look at "mailq" now, I see nothing in the queue except that spam email which I posted above. So it seems gmail may just be delaying receiving this email? And it will keep doing this and eventually fall off of mailq's queue once it runs out of retries? Is this how these email servers work??

I'm a bit new to this. Obviously I think it will be a good idea to add the spam filter stuff to my server, but honestly I rather not mess around with the server unless I have to. I funnel all my email through to my gmail account and gmail does a great job of filtering the spam for me. So spam is no problem for me in terms of my inbox (which is why I funnel all my email to gmail in the first place).

Appreciate any advice you can give. Thank you for the quick response!!

[apinto]

americanninja let's go by parts    (this is a long post, get ready!)

Thank you apinto! I will have to give that a try. I was worried about adding on software to the server after I got everything working with Sentora (i.e. in case it messes things up with Sentora and brings down the web server). Is it risky to do this? I'm assuming anything you install that's connected to the web server functions could cause trouble for Sentora, as Sentora is managing the server. Please let me know. 

Yes, every single change you make to the server you have some degree of risk of messing up the sentora environment.
The rule is always do a backup, I mean ALWAYS BACKUP.
You can never be too sure it will not break something.

As for the gmail greylisting, perhaps it's only the spammy emails that it's doing this? For example I just tested a few features on my drupal site which sends out emails to users based on things they do on the site. I have a test user on my Drupal site with one of my emails setup on my server. myemail@mydomain.com. I ran through and did a few things on the website that I know would trigger emails and those immediately came through to my gmail account (from my server via Drupal -> myemail@mydomain.com -> myemail@gmail.com). 

Probably yes (gmail spam filtering is a shady zone, it's hard to find facts about it  ).
My favorite Spam Checking tool is https://www.mail-tester.com/ (its easy to use and read the results, but probably there are better tools for more extensive usage), use this to check how your emails are being checked against SpamAssassin, SPF rules, DKIM, ReverseDNS etc. Consider a score above 8.5 to be good, anything above 9.5 is excelent;
However read all the diagnostics because even if SpamAssassin is not considering something critical, Gmail might!

These came through instantaneously, so no issues receiving them to gmail. And if I look at "mailq" now, I see nothing in the queue except that spam email which I posted above. So it seems gmail may just be delaying receiving this email?

Use the same tool to check your emails for spam, if they pass, you are ok to go  .

And it will keep doing this and eventually fall off of mailq's queue once it runs out of retries? Is this how these email servers work??

Postfix will retry multiple times or until he gets a permanent error, please read at least the first paragraph: http://jrs-s.net/2013/04/17/configuring-...n-postfix/ 

I'm a bit new to this. Obviously I think it will be a good idea to add the spam filter stuff to my server, but honestly I rather not mess around with the server unless I have to. I funnel all my email through to my gmail account and gmail does a great job of filtering the spam for me. So spam is no problem for me in terms of my inbox (which is why I funnel all my email to gmail in the first place). 

Appreciate any advice you can give. Thank you for the quick response!!

Never be afraid of experimenting (just don't experiment on production servers  ).
Everyone was new to everything at some point in their life, its only our persistence that makes us knowledgeable at any given topic; be persistent!


Regarding what you do (funneling all email through Gmail)  I also do the same (bye bye Outlook/Thunderbird ) and I've never got any issues, I even encourage my clients to do it.
You can even setup Gmail to check other servers using SMTP/POP or to send email using your own SMTP server instead of an alias (I guess oyu already knew this).

Going back to the "experimenting" with the server, well I STRONGLY encourage you to try Vagrant, it is possibly some learning curve to get your head around all the concepts of visualization but the documentations is really good and easy to follow if you are persistent.
Vagrant is a tool that allows you to "Create and configure lightweight, reproducible, and portable development environments." This means you can test what you wish on a virtual server at your local machine and "reset" in case something goes wrong, after you are comfortable you can update the production server safely knowing it won't break anything (still keep doing backup... we never know for sure...  )

You can check my topic about Sentora Vagrant Box (http://forums.sentora.org/showthread.php?tid=1476) that is a Ubuntu 14.04 x64 with Sentora default install.


If you still have issues with your email check the following thread about a user who had his email sent to spam on Gmail, Hotmail and Yahoo, but I believe it does not apply to your case however you can still check your IP reputation.

Make sure your ip is not blacklisted or with bad reputation.

Checking Senderbase (https://www.senderbase.org/lookup/?searc...38.182.138) does not look bad at all.

Are you sending to Gmail?
What is the reason gmail tells you for it going to spam? (please refer to: http://www.whatcounts.com/2012/03/gmail-...marketers/)

Does it only happens on Gmail?
One tool I found usefull is https://www.mail-tester.com/ , do a test and post back here the results url so we can check it better.

Try DKIM... might help...

Sometimes Gmail and Hotmail mark some IP ranges for unknown reasons (most related to previous owners spam reputation), when this happens there is little you can do, however I know of reports that sending email to multiple gmail address and marking the emails as "Not Spam" does help bringing your reputation back up (do this carefully or you might end up in a worse situation).

[americanninja]

@Apinto...thank you for the detailed response and write up. Sorry for taking so long to reply. This is great stuff!!

1.) Yes, I completely agree. I just need to learn more about digitalocean and how to make duplicates of the server so I can try on a non production instance of it. I haven't done this yet with Digitalocean, as I am new to their service as well. Does Sentora have any kind of backup feature to dump all the current configuration settings so that if I had to I can rebuild a new server with a default Sentora install and then just import the saved config file...i.e. it would setup my domains, email addresses and forwards, etc? Of course I would still need to move my http files and mysql databases, but at least the other stuff could be imported? Or are you saying to simply clone the entire server? What do you do?

2.) The mail-tester tool is awesome...love it! Great recommendation...thank you! I got a score of 9, so I guess I am pretty good...I just got dinged on the plaintext or ssl pieces as I don't have those setup. So I guess I am good there.

3.) Another great link...thank you! I will test making these changes this weekend. I think this will solve my probably of the mailq getting large with emails stuck in there.

Thank you for all the great advice. Right now it seems all the mail in my mailq deferred is because of spammers sending emails to my myname@mydomain.com email address. They then get stuck in the queue because gmail is refusing to deliver them or says it will delay it due to it being spammy. My legitimate emails are getting through immediately, so it doesn't seem to affect anything. Basically it's just a bunch of spam junk that gmail is doing a good job of rejecting. The only problem is my server has it sitting in the queue to retry to send through, and gmail is just gonna block it again.

I guess I could write a script to simply remove any mail in the deferred list which has the "blocked" word in it. Or just periodically just clear the mailq with a cron job script. But I think the best way will be to modify the postfix settings so that it simply just discards emails after it attempts twice within an hour or something. I'm guessing most of the emails stuck in deferred queues these days are just spam emails that will never get through anyways.

For example here are two in my queue right now:

4D1F1148538 208 Thu May 7 11:49:50 Mayer_Enrique30@autoplates.com
(host alt1.gmail-smtp-in.l.google.com[64.233.191.26] said: 550-5.7.1 [104.xx.xx.xx 12] Our system has detected that this message is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1 http://support.google.com/mail/bin/answe...wer=188131 for 550 5.7.1 more information. j12si1743109icc.29 - gsmtp (in reply to end of DATA command))
myemail@gmail.com

E8D99148560 32637 Thu May 7 19:26:56 reitwi@blr.pin.philips.com
(host alt1.gmail-smtp-in.l.google.com[64.233.191.27] said: 552-5.7.0 This message was blocked because its content presents a potential 552-5.7.0 security issue. Please visit 552-5.7.0 http://support.google.com/mail/bin/answe...nswer=6590 to review our 552 5.7.0 message content and attachment content guidelines. fb6si1804369icb.68 - gsmtp (in reply to end of DATA command))
myemail@gmail.com

So these will never be passed through and these are basically the only emails I have stuck in my queue. So I guess the question is, do I need to bother with installing a spam protection software on my server or just change the postfix settings to get rid of these emails faster? Does gmail penalize me if my server keeps trying to send these? I think its better to leave the spam software off my server to reduce load on server and let gmail handle the filtering for me. It does a good of enough job already. Thoughts?

Thank you!