About to go public-live with phase1 of the user-integration side of this site I've been hammering at for the last 6 months; but with the testing of the user registration/login system I am getting a huge failure rate to send anything to gmail.com addresses. System messages and DBLOG reckon that all is well and has been transported. Nothing arrives even into spam for at least 80% of gmail attempts.

My own domains, other administrative domains, and even Hotmail work.

I have installed Maillog in order to try to troubleshoot this, but unfortunately had to add an issue in it's queue because it's not logging anything.

Any other ideas?

 

Accepted answer

Righto, so this is the fix ( although I may be counting my chickens!! );

because we are using Sendgrid, due to the fact we have a database of 4k users already, the Sendgrid system was being blocked by gmail before entering the gmail servers:

Site -> SendGrid ->Gmail-> User

                            X-----breakage happens here

Sendgrid doesn't report on Gmail blocking, but a few others do, so I was able to understand the mechanics involved; SG use numerous SHARED servers to handle the distribution of the load of emails going through their system from your site and all their other clients. 'Other clients' is the clincher; it takes very little before one of the many shared servers gets marked as a spam source, which takes a really long time to clear. And so because your neighbour was naughty you get tarred with the same brush.
The solution is paying for the rather expensive "dedicated IP" of the Pro account, so that you are only ever using one, dedicated IP address to send your traffic, rather than the many that would otherwise occur in Shared space.

eg Shared traffic is split over Shared Servers:

  • 159.183.77.210
  • 159.183.77.211
  • 159.183.77.212
  • 159.183.77.213

and Gmail blocks ....212 because some spammy stuff went across it last week, thereby reducing your deliverability to Gmail by 25%.

Change to Dedicated IP:

  • 159.183.77.215

And Gmail lets everything through, because you're not sending spam, right? ;-)

Mailchimp does the same thing, apparently, but at a higher cost.

Comments

Righto, so this is the fix ( although I may be counting my chickens!! );

because we are using Sendgrid, due to the fact we have a database of 4k users already, the Sendgrid system was being blocked by gmail before entering the gmail servers:

Site -> SendGrid ->Gmail-> User

                            X-----breakage happens here

Sendgrid doesn't report on Gmail blocking, but a few others do, so I was able to understand the mechanics involved; SG use numerous SHARED servers to handle the distribution of the load of emails going through their system from your site and all their other clients. 'Other clients' is the clincher; it takes very little before one of the many shared servers gets marked as a spam source, which takes a really long time to clear. And so because your neighbour was naughty you get tarred with the same brush.
The solution is paying for the rather expensive "dedicated IP" of the Pro account, so that you are only ever using one, dedicated IP address to send your traffic, rather than the many that would otherwise occur in Shared space.

eg Shared traffic is split over Shared Servers:

  • 159.183.77.210
  • 159.183.77.211
  • 159.183.77.212
  • 159.183.77.213

and Gmail blocks ....212 because some spammy stuff went across it last week, thereby reducing your deliverability to Gmail by 25%.

Change to Dedicated IP:

  • 159.183.77.215

And Gmail lets everything through, because you're not sending spam, right? ;-)

Mailchimp does the same thing, apparently, but at a higher cost.

Have you tried Google Workspace? I had similar problems with my shared hosting (which uses sendgrid). So I decided to sign up for Google Workspace, and use the SMTP module with my credentials to send emails through their server. This fixed the problem for me - before it was a nightmare, outlook, hotmail, aol, and even yahoo would block all the server emails. With Workspace the problem is mostly gone (with the exception of a few private universities in the US).

That's interesting. No, haven't tried yet, have signed up for large 3 years of VPS running over in Sydney, and need to deal with volume of 10k/month emails minimum. I can't see GW details of the volume send-able.