Email Testing

A blog post is lackluster without working code you can play around with in your own environment. All the code samples in this post can be found here.

Email testing is critical for business success and boosts email performance. It is not something we want to forego while testing our web applications because modern email services allow painless automated email testing. The core goal is to enable the last mile of end to end testing, to enable a typical web app to be tested from start to finish.

For example imagine a scenario where a user starts having received an email invite from an organization, through company proprietary services or third party such as LinkedIn invitations. Then, the user verifies email content, accepts the invite, and joins the organization.
Later, the user can leave the organization - or get removed by an administrator - then receives another email notification.
Using an email service, the entirety of this requirement is possible to automate and execute within seconds.

That being stated, email testing is a fundamental enabler for SaaS test architectures by permitting stateless tests that can scale; tests that independently handle their state and can be executed by n number of entities at the same time.

• Email Testing
  • What to test in an email
  • Why you do not want to use the "+" trick
  • Speed reduces the cost
  • What do we need from an email service?
  • How to test emails
  • Achieving Stateless tests with unique emails
  • Setup for this case study
  • Environment variables
  • Older, difficult approaches
  • The overhead is abstracted away with Cypress Mailosaur plugin
  • Data focused assertions with cy-spok

What to test in an email

Typically email testing involves :

• validating email fields; from, to, cc, bcc, subject, attachments.

• HTML content and links in the email.

• Spam checks and maybe visual checks.

Why you do not want to use the "+" trick

Stateless tests that can scale are a necessity in any modern web application testing. We want tests that independently handle their state and tests that can be executed by n number of entities at the same time.

While testing SaaS applications, which generically have Subscriptions, Users, Organizations (ex: Slack, Cypress Dashboard Service, etc.) a lot of the end to end workflows can rely on having unique users. Otherwise only one test execution can happen at a time and they clash with other simultaneous test executions. This constraint reduces test automation to cron jobs or semaphores; testing hardware is a classic example.

Some ways to address unique users is by utilizing Gmail tricks or AWS Simple Email Service. It is possible that you do not have to check actual email content (from, to, cc, bcc, subject, attachments etc.) and only want to have unique users, then you are on the right path with stateless tests and this is enough for your current needs.

However, these approaches are generally problematic for email testing:

• Non-existing emails can prompt bouncing emails to your cloud service and that can be a headache.
• Too many emails can flag the email for spam; think load testing or many e2e tests running per day.
• With the + trick, if there is even a way to check the email content, there is no way to differentiate between emails being received. We end up in a situation where every spec has a unique email, which defeats statelessness; we're locked into cron jobs or semaphores.
• Unreliable email speeds, which add up in CI costs, and more importantly engineer feedback time.

If you want to avoid such issues and check real email content in automation, email services provide value.

Speed reduces the cost

Email services can also provide cost savings in test execution time by receiving emails faster, tests running quicker in the pipeline with less CI resources being consumed and less time waiting for tests to finish. If you are running 1000 pipelines a year, and save 3-4 seconds per pipeline execution, the email service can be already paying for its annual subscription just by providing the extra speed.

What do we need from an email service?

We need a few things from an email service for test automation:

• Unique email servers per service and app so that there is a predictable inbox.
• Being able to create (unlimited) users on the fly and have emails sent to them.
• Receiving the emails fast.
• Being able to verify the content of those emails effortlessly.

How to test emails

There are plenty of email testing solutions available, and combinations of test frameworks that integrate with them.
For the code snippets and working examples, we will be using Cypress and Mailosaur, but the ideas should generally apply to any tuple of email services and test automation frameworks.

When using Cypress with Mailosaur, there are 3 test-development approaches:

1. Implement Mailosaur API using Cypress API testing capabilities using cy.request() or cy.api(). Utilize plugins and helper utilities to construct test suites. This was the original approach a long time ago. The test spec can be found here.

2. Utilize Mailosaur's Node package and implement them using cy.task() which allows running node within Cypress. This is a easier and cleaner approach than (1), and it became available as soon as the npm package was released. The test spec can be found here.

3. Use the Cypress Mailosaur plugin and abstract away all the complexity! The test spec can be found here.

If using the sample repo 3 weeks after the date of this post, note that you will have to start a new Mailosaur trial account and replace environment variables in cypress.json for yourself.

Achieving Stateless tests with unique emails

If in every test execution a new, unique user was used and the emails to this unique user could be verified in isolation, it would be possible to achieve a stateless test. The only side effect would be to the email service inbox, but if the test only checked emails by reference and cleaned up after itself, the email service mailbox would not be impacted.

This is easy to achieve with Mailosaur, here are two approaches to this: Mailosaur's Node package or our own util with faker.js.

Note that these emails are unlimited; the term Users at Mailosaur Pricing Page refers to the number of employees at your company accessing the Mailosaur web interface.

// at cypress/plugins/mailosaur-tasks.js// generates a random email address// sample output:   ojh788.<serverId>@mailosaur.ioconst createEmail = () => mailosaurClient  .servers  .generateEmailAddress(envVars.MAILOSAUR_SERVERID););// our custom function at a helper file or commands file. // The only difference is the defined prefixed name.// sample output:  fakerJsName.<serverId>@mailosaur.ioconst createMailosaurEmail = randomName =>  `${randomName}.${Cypress.env('MAILOSAUR_SERVERID')}@mailosaur.io`;

Setup for this case study

The only setup we need is a way for emails to be sent to our server. For this, the sample repo is using sendmail npm package . Any tool can be used here for the case study. Note that this one needs node 12, therefore an .nvmrc file has been included.

Environment variables

We recommend these values as environment variables in the cypress.json file "env" property. You can grab them from Mailosaur web application by creating a free trial account using any email. The trial account lasts for two weeks.

If you are using the referenced repo within three weeks of this post, no need to do anything here.

  "MAILOSAUR_SERVERID": "******",  "MAILOSAUR_PASSWORD": "******",  "MAILOSAUR_API_KEY": "*******",  "MAILOSAUR_API": "https://mailosaur.com/api",  "MAILOSAUR_SERVERNAME": "user-configurable-server-name"

Older, difficult approaches

We will cover the latest and greatest way with Cypress Mailosaur test plugin in this guide. For the older approaches, the evolution over time, the code has been shared and you can also check out an older version of this post for a read through.

The overhead is abstracted away with Cypress Mailosaur plugin

Mailosaur team released a Cypress plugin in mid 2020. With it, we do not have to replicate any complex API utities or use cy.task via Mailosaur npm package. There is no need to create cy.task utilities or even hybridize them. With the Cypress Mailosaur plugin, you can just use the custom Cypress commands Mailosaur team created for us.

All we need is to install the package npm install cypress-mailosaur --save-dev and add the following line to cypress/support/index.js:
import 'cypress-mailosaur'. The new kid on the block is cy-spok, and that will make our assertions a bliss. npm il cy-spok -D that too, and we only need it imported at the top spec files with import spok from 'cy-spok'.

Mailosaur plugin has a few handy functions which help you abstract complex needs. Some of our favorites are bolded.

• mailosaurListServers()
• mailosaurCreateServer({ name })
• mailosaurGetServer(serverId)
• mailosaurUpdateServer(serverId, server)
• mailosaurDeleteServer(serverId)
• mailosaurListMessages(serverId)
• mailosaurCreateMessage(serverId)
• mailosaurGetMessage(serverId, criteria)
• mailosaurGetMessageById(messageId)
• mailosaurSearchMessages(serverId, criteria, options)
• mailosaurGetMessagesBySubject(serverId, subjectSearchText)
• mailosaurGetMessagesByBody(serverId, bodySearchText)
• mailosaurGetMessagesBySentTo(serverId, emailAddress)
• mailosaurDeleteMessage(messageId)
• mailosaurDeleteAllMessages(serverId)
• mailosaurDownloadAttachment(attachmentId)
• mailosaurDownloadMessage(messageId)
• mailosaurGetSpamAnalysis(messageId)

With the Cypress Mailosaur plugin, we do not have to implement any cy.task() utilities, custom helper functions or hybrid helpers.

You can find a working version of this code here. Here are a few approaches with various commands, and basic Cypress assertions.

// an email has been sent in the before hook it('uses the mailosaur cypress plugin to check the email content', () => {  cy.mailosaurListMessages(Cypress.env('MAILOSAUR_SERVERID'))    .its('items')    .its('length')    .should('not.eq', 0)  cy.log('get message')  cy.mailosaurGetMessage(    Cypress.env('MAILOSAUR_SERVERID'),    { sentTo: userEmail }    // note from Jon at Mailosaur:    // The get method looks for messages received within the last hour    // if looking for emails existing before that, you have to add this. Optional otherwise    // { receivedAfter: new Date('2000-01-01') }  ).then((emailContent) => {    cy.wrap(emailContent)      .its('from')      .its(0)      .its('email')      .should('contain', '[email protected]')    cy.wrap(emailContent)      .its('to')      .its(0)      .its('email')      .should('eq', userEmail)    cy.wrap(emailContent)      .its('subject')      .should('contain', 'MailComposer sendmail')  })  cy.log('alternate approach to getting message by sent to')  cy.mailosaurGetMessagesBySentTo(    Cypress.env('MAILOSAUR_SERVERID'),    userEmail  ).then((emailItem) => {    // the response is slightly different, but you can modify it to serve the same purpose    const emailContent = emailItem.items[0]    cy.wrap(emailContent)      .its('from')      .its(0)      .its('email')      .should('contain', '[email protected]')    cy.wrap(emailContent)      .its('to')      .its(0)      .its('email')      .should('eq', userEmail)    cy.wrap(emailContent)      .its('subject')      .should('contain', 'MailComposer sendmail')  })  cy.mailosaurGetMessagesBySentTo(    Cypress.env('MAILOSAUR_SERVERID'),    userEmail  )    .its('items')    .its(0)    .its('id')    .then((messageId) => {    cy.log('does convenient spam analysis')    cy.mailosaurGetSpamAnalysis(messageId).its('score').should('be.gt', 0)  })})

Data focused assertions with cy-spok

Let's take a look at the Mailosaur interface. We can add multiple servers here, limited by the subscription plan. We can to have a unique server per app or service, and we can prefix random user names by the environment to make diagnostics easier.

We can look at our Messages / Inbox. The email is from the test we executed via node mailer.

There isn't too much to look at in our email, just some html and text in it.

On the Spam tab, our score is looking pretty bad; the spam analytics must be working correctly.

The JSON tab is where the magic starts happening. It is a JSON representation of our email. Look how nicely it matches what we see in DevTools.

It would be so good if our test matched that data 1:1. This is where cy-spok comes in.

We will let the magic do the talking.

it('uses cy-spok, matches email payload and verifies the data', () => {  cy.mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVERID'), {    sentTo: userEmail  })    .should(    spok({      from: [        {          email: '[email protected]'        }      ],      to: [        {          email: userEmail        }      ],      subject: 'MailComposer sendmail',      html: {        body: (b) => expect(b).to.include('here is some text')      }    })  )    .its('id')    .then((messageId) => {      cy.mailosaurGetSpamAnalysis(messageId)        .its('score').should('be.gt', 0)      // commented out for testing      // we should use it in CI so that the test is stateless      // cy.mailosaurDeleteMessage(messageId)  })})

Email testing used to be an arduous task. Over the few years, with contributions from the community, it became effortless. This blog post and the referenced repository introduced you to email testing, and told a story of how it evolved. Equipped with this knowledge, now you can confidently implement modern, effortless, confident email testing solutions for your teams. Cheers!