Testing with emails

Testing a product like Campaign Monitor means dealing with a lot of emails as test data. Here are some things I’ve learned about that.

Unique email addresses
In order to check the contents of emails that you have sent, you need to have an accessible email account. But trying to send all emails to one email address has the following limitations:
– If your system does not accept sending to duplicate email addresses, this may not be possible.
– It limits the diversity of the test data.

Luckily, Gmail has a nifty feature that can help work around these problems. So, if your email address is johnsmith@gmail.com, you can easily create more varied test data using the + symbol. For example:


Doing large sends
Doing extremely large sends to one Gmail account (1000+ emails) is generally not a great idea, as I found out the hard way. I once tried to send myself about 3 million emails. Gmail reacted by receiving and storing a few hundred emails, then it stopped for a few hours and wouldn’t let me have any, then it let me have a few hundred more, stopped again, started again…repeat. This went on for about four days before I had to stop it by setting up a filter to immediately delete all the test emails on sight. In addition to this behaviour, it grouped the emails in groups of about 79 at a time because the emails were all from the same address. Why 79? I have no idea, but it made it really difficult to check how many of each email type I had received, when I was dealing with thousands.

Usually when doing sends of this size, I don’t want to check the contents of every single email anyway because I am usually testing the system’s capability of handling that much volume. I’m not interested in testing Gmail’s mail handling capabilities in this context either. So we have a local catch-all domain name set up that just discards any emails sent to it. I’ll send the majority of the emails to this domain, and just a couple of emails will also go to a checkable account.

Checking plain text and HTML versions of emails
If the application sends separate HTML and plain text versions of emails, there’s something important to know about checking both of these versions. When selecting “view in plain text” for an HTML email, some email clients won’t show the plain text version sent by the application. Instead it will create a plain text version from the HTML email currently being viewed. Outlook is guilty of this. I recommend using Mozilla Thunderbird, which will display the “true” plain text version.

Using GUI automation to check email content
Web-based email clients offer a way for GUI automation tools to check email content. But using GUI automation to check these email clients means that the test suite can run into problems if any GUI updates are made to the web-based email client website. For simplicity, we have found that it’s better to set up a web-based email client locally. Here, we have been using MailBee and it’s worked out fine for us.

Generating subscriber lists
I wrote up a little script in Ruby to generate large subscriber lists, taking advantage of Gmail’s + symbol suffix feature. Here’s the code for it:

dict = File.open('names')
lines = dict.readlines

number_of_subscribers= ARGV[0]
fileName = "SubscriberList" + number_of_subscribers + ".csv"
subscriber_file = File.open(fileName, 'w')

for i in 1..number_of_subscribers.to_i
while "" == (line = ic.iconv(lines[rand(lines.size)].rstrip))
subscriber_file.puts("yournamehere+" + i.to_s + "@gmail.com, " + line + "\n")

puts "Created: " + fileName

It reads from a file called “names”, which I populated with a list of names, each on a new line. I tried to find a name generator that would give me a realistic range of names, but they were all a bit too clean. So I ended up yanking the names from one of our own mailing lists, which had some absolutely gorgeous test data in it. Just to give you an idea of the sort of things people will realistically enter into a free-text name input field, here are a few sanitized examples:

John Smith
john smith
J.C. Smith
John C. Smith
John Smith-Jones
John-Paul Smith
John O’Smith
Jon van der Smith
Jühn Smøth
J . C . Smith
John Smith & Co
J?ohn Smi?th
“John, Smith.”
John Smith I
John Smith Incorporated
john’s product name