Ubuntu - Email - Blacklist Removal - Gmail

These are the steps needed to remove your IP from Gmail’s email blacklist:

• Gmail Blacklist Criteria
• Why Was I Blacklisted?
• Blacklist Investigation Process
• Gmail SMTP Errors
• Gmail Blacklist Removal Instructions
• Getting Help

Gmail does not disclose the details of their filtering process. If they did, spammers would quickly find a way around their filters.

However, some common reasons why Gmail may reject email from your server include:

• Sending large volumes of email.
• Sudden changes in email volume.
• Sending email to “spam trap” addresses.
• Sending email to unknown users.
• Inclusion of your server’s IP on public blacklist.
• Gmail users marking your messages as spam.
• Using a new IP address to send email.
• Incorrect DNS Settings

If your server is doing any of these things, then you appear to be sending spam. As a result, Gmail may block your server’s IP address.

Google may use signals from these public blacklists:

• pbl.spamhaus.org – This blacklist includes dynamic and non mail server IP ranges. For a server to be listed in here would be unusual.
• sbl.spamhaus.org – This blacklist includes emails that Spamhaus has identified as spam.
• xbl.spamhaus.org – This blacklist includes bots and exploit agents.
• cbl.abuseat.org – This blacklist includes emails sent to spam traps or reported by their users as spam.

You can use the Multi-RBL lookup tool to check these and other lists. Inclusion on these lists is a good indication that you have some type of spam issue on your server.

Usually one of these three causes:

• Spammers exploiting web applications (>90%).
• Customer’s password or computer compromised (~5%).
• Poor email practices such as blindly forwarding email to Gmail (~5%).

In over 90% of cases, hackers use insecure web applications to send spam.

When they do this, spam volume, user complaints and other issues trigger Gmail’s blacklist filters. They start blocking your server to protect their users from spam. Spammers often send email using SSH tunnels.

Even in the absence of security issues, your server may still look like a spamming system.

If you have users forwarding email from your server to Gmail and they forward spam, then your server looks like it is sending the spam. As a result, Gmail may block your sever.

If you want to get off and stay off the blacklist, you must dig into your server and understand why your server was blacklisted. If you do not, then your removal effort will be wasted.

This is the process to investigate the blacklisting:

• Check email server logs for 500 errors.
• Check email logs for blocks to other ESPs and public blacklists.
• Look for Excessive SMTP Authentications, especially from varying IPs for the same user.
• If you have PHP scripts, configure PHP to log mail using the mail.log ini entry.
• Lookup your IP in your favorite blacklist lookup tool.
• Check your server’s sending reputation at SenderScore.org.
• Check for users bulk forwarding email to Gmail.com and related domains.
• Check for any newsletters or newsgroups that originate from the server.
• Identify any bulk marketing campaigns that may be on the server.
• Verify that DNS related entries (PTR, DKIM, SPF) are correct.
• Look at historical logs and determine if the email volume to Gmail has increased.

This process can be time consuming, especially on a busy server. It is recommended that you start by checking for user compromises, while these account for relatively few cases, they are much easier to diagnose than web application issues.

For example, on Plesk/Postfix setups, you can string together shell commands like:

grep sasl_username /var/log/maillog|awk {'print $NF'} |sort |uniq -c |sort -n

This quickly returns a list of user authentications by user name. If you see high values, that user may be worth a more detailed look.

You can use similar commands to pull out all sorts of email summary information on your server.

While digging into the server’s email history, keep a watch on:

• New 550 and 421 errors from other email providers
• IP listings in public blacklists.
• Changes in the Gmail Blacklist response code
• Your SenderScore.

Usually, this investigation turns up a compromised web script or email user’s password. You can then fix this issue by updating or removing the script or simply resetting the user’s password.

Once you fix the underlying issue, monitor the server’s email volume and response codes from Gmail. If things do not clear up, then you can submit a removal request to Google.

In most cases a removal request is never needed to be submitted. Cleaning up the issue and fixing any DNS problems will usually resolve the listing in 3-5 days.

Blacklists block email but they do not route it to the spam folder. (See our post on why email is going to the spam folder).

If you are blacklisted, your email will be rejected with a 421 or 550 SMTP error.

You can spot this in your email server’s logs:

Example of a 550 Error:

Remote_host_said:_550-5.7.1 Our_system_has_detected_an_unusual_rate_of
 unsolicited_mail_originating_from_your_IP_address.
 _To_protect_our users_from_spam,_mail_sent_from_your_IP_address_has_been_blocked.
 Please_visit_http://www.google.com/mail/help/bulk_mail.html
 _to_review_our_Bulk_Email_Senders_Guidelines

Example of a 412 Error:

421-4.7.0 unsolicited mail originating from your IP address.
To protect ourn421-4.7.0users from spam, mail sent from your IP address has been temporarilyn4
21-4.7.0 rate limited.  Please visit http://www.google.com/mail/help/bulk_mail.n421 4.7.0 html
to review our Bulk Email Senders Guidelines. l41si55243084eef.158 - gsmtp

If you see either of these errors, then you are blacklisted and you can work on getting off the list.

Here’s the full list of Gmail error codes:

You must stop the spam-like behavior before submitting a request to Gmail. If you do not, your efforts and their time will be wasted.

If you have stopped the spam coming from your server, Gmail will usually remove your IP automatically in 3-5 days.

If not, then you may need to contact them for assistance.

To do so, you need to use this form. Be sure to be logged into your Gmail/Google account before you start the process.

It is highly recommend all areas are completed although they are not all required. You want to give the blacklist removal team as much information as possible to decide the you are not a spammer.

Keep it brief and to the point. For example, I commonly use this text:

The server had a compromised web application that was used to send spam to Gmail. We have removed this application from the server. Since removing the application, we no longer see unauthorized email being sent to Gmail.

Make sure your headers are complete and in text format. You only need to include one example. In general, try to find an example that is simplistic. Such as a message going directly from your server to Gmail. If the message was relayed through a third party, the headers can be obscured.

Try to use a text (.txt) file if possible. Avoid Windows or Mac specific formats.

Only copy the relevant portion of the server logs. Just 2-3 entries will suffice. They should look like the examples 550 and 421 examples above.

While not required, this is a key step to show that your server’s DNS is working. Successful results will look similar to:

<br data-mce-bogus="1">
[jeffh@office ~]$ host -t mx gmail.com
gmail.com mail is handled by 40 alt4.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 5 gmail-smtp-in.l.google.com.
gmail.com mail is handled by 10 alt1.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 20 alt2.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 30 alt3.gmail-smtp-in.l.google.com.

Make sure you do this from the impacted server using one of the records from your DNS lookup. Successful results will look similar to:

<br data-mce-bogus="1">
[jeffh@office ~]$ telnet alt4.gmail-smtp-in.l.google.com 25
Trying 2800:3f0:4003:c01::1a...
Connected to alt4.gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP c68si3349613vkd.85 - gsmtp

Example of a ping test. Note that if you have firewalls blocking ICMP traffic, this test may fail. If it fails, just do not include it in the removal request.

<br data-mce-bogus="1">
[jeffh@office ~]$ ping -c5 alt4.gmail-smtp-in.l.google.com
PING alt4.gmail-smtp-in.l.google.com (64.233.190.26) 56(84) bytes of data.
64 bytes from ce-in-f26.1e100.net (64.233.190.26): icmp_seq=1 ttl=43 time=169 ms
64 bytes from ce-in-f26.1e100.net (64.233.190.26): icmp_seq=2 ttl=43 time=169 ms
64 bytes from ce-in-f26.1e100.net (64.233.190.26): icmp_seq=3 ttl=43 time=169 ms
64 bytes from ce-in-f26.1e100.net (64.233.190.26): icmp_seq=4 ttl=43 time=169 ms
64 bytes from ce-in-f26.1e100.net (64.233.190.26): icmp_seq=5 ttl=43 time=169 ms
--- alt4.gmail-smtp-in.l.google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4175ms
rtt min/avg/max/mdev = 169.448/169.487/169.600/0.523 ms

This is an open-ended field but keep it short. I usually use this to note any unexpected issues or if a customer had previously sent a removal request but did not clean up the server.

Once you have all of the data complete, you can submit the form. You should see a message informing that the request will be investigated.

Usually, this is resolved within 5 business days.

Just be warned that there are no quick fixes. If you rush off to the removal page without fixing the issue, you will likely just be listed again.

In July 2015, Google launched Gmail Postmaster Tools. This is similar to webmaster tools but for email. If you managed email for your domain or your customers, you may want to sign up.