Introduction

Cypress is the up and coming framework for end to end testing. It has grown a lot in the last year or so and seems to be on track to become a feature-rich testing framework with an intuitive API that will replace Selenium based frameworks in many places. It has plenty of industry support and is used by many large organizations. It offers the possiblity to rapidly write e2e tests which interact with your web app. It is easy to extend, to write custom commands and to mock requests and responses.

The only browser supported at the moment is Chrome (or the chromium based electron), which isn’t necessarily a problem for us, as we recommend our users to use Chrome anyway.

In this post I will tell you a little bit about our codebase and the challenges we encountered implementing Cypress into it. I will also reflect on how it allows us to shorten the feedback loop between developing and deploying. We still have some way to go, but are already seeing the benefits.

Example of a test running in Cypress

The olden days

Our app grew organically over the years and has become quite complex with lots of moving parts. Especially the graphical editor, where our clients can design detailed solar panel systems on top of an aerial image of a building or field, has lots of features that are highly integrated with each other. Before 2018, we used to test these features manually with every biweekly release, to check if everything was still functioning. We had 8 release test scenarios which an employee had to go through step by step to make sure everything was still working properly, along with someone spending some time as a general bughunter, who tried breaking the app. This had several advantages:

• Employees became familiar with new features and changes in the app
• Bugs could be described properly by experienced users, making them easy to find

But there were also many disadvantages:

• It cost a lot of time, where many people worked on it for 20-60 minutes.
• Test coverage was limited, new features weren’t added to the scenarios
• Even monkeys make mistakes and miss bugs that end up in production
• It tied us to a biweekly release schedule, slowing down the overall development process
• We had to maintain multiple branches, sometimes causing conflicts

Transitioning to automated end to end testing

We decided to look for ways to start automating our release testing. At first, we spiked a branch where we used Selenium. This quickly became very tedious and inspired us to start looking at other solutions. Ultimately, we settled on Cypress, which was easy to set up and get running.

We started with implementing a test for a simple scenario. This helped us get acquainted with the way Cypress works, learning about the proper way to select elements on the page, build commands that handle repetitive tasks, wait for responses from our backend api, etc.

Challenges we’ve encountered

We’ve come across several challenges while implementing Cypress, first a couple that are specific to our app:

• Logging in in different environments
• Interacting with our editor

Then some that are more general:

• Thinking Async
• Integrating Cypress in our CI (Codeship) - Work in progress
• Changing the way we think about the way our app works

We’ll go over these one by one.

Logging in in different environments

We have a Python/Django backend and use sessions rather than tokens. Locally, we started with just adding an admin cypress user to our development database. But when we deploy a release candidate to our beta server for testing, we don’t want to have such a user there with admin privileges. So for now we still have to settle for getting a session id manually and passing this in as an environment variable to test things on beta.

We wrote a custom command, which visits the login page, and logs in when working on localhost, but when testing against a web server we require a session id to be passed in:

cypress open --config baseUrl={url-of-server} --env sessionid={session-id}

We would like to switch to using tokens, and perhaps to expose an endpoint on beta which can generate temporary access tokens for use during tests, but haven’t implemented this yet.

Interacting with our editor

This was the most complex problem we faced. Accessing objects' locations that get drawn on an html canvas, some of which are only visible when a user’s mouse is in a specific location posed some challenges.

Finding the objects was solved by exposing the fabric canvas on which they were drawn on the global window object when we are not in production. This allowed us to find grids, panels, and the buttons that are used to manipulate those in our editor. We wrote several helper functions to make the tests themselves look more natural, but it was still quite complex and in need of commenting:

cy.window().then((window) => {
    const fabricCanvas = window.fabricCanvas

    // Move one grid so it definitely won't cover the other
    // getGrids and getPosition are helper functions that
    // grab the grids on the canvas and gets the position of
    // an element respectively.
    const grid = getGrids(fabricCanvas)[0]
    const { x, y } = getPosition(grid)

    cy.get('#editor-map')
        .trigger('mousedown', x, y)
        .trigger('mousemove', x + 100, y + 100)
        .trigger('mouseup')
})

You see that it is still quite nuts-and-bolts when it comes to clicking on the editor (in this case we click on a grid, move it 100px right and down and release the mouse). A next step we did, therefore, was to generalize this idea and move some of this code to a command:

// Add a command where we grab the fabricCanvas
Cypress.Commands.add('withFabricCanvas', () => {
    cy.window().should('have.property', 'fabricCanvas')
    cy.window().then(window => window.fabricCanvas)
})

// Add a command to move a grid, which can be chained onto the previous one
Cypress.Commands.add('moveGrid', { prevSubject: true }, (grid, offset) => {
    const { x, y } = getPosition(grid)
    cy.get('#editor-map')
        .trigger('mousedown', x, y)
        .trigger('mousemove', x + offset.x, y + offset.y)
        .trigger('mouseup')
})

// Now the code in the test becomes
cy.withFabricCanvas()
    .getGrid(1) // gets grid with id 1
    .moveGrid({ x: 100, y: 100 })

Thinking Async

One of the biggest gotchas when writing Cypress tests is that eventhough the code runs asynchronously, the structure of the code is synchronous, as if we are using async/await without the keywords:

cy.visit('/hello/world') // blocks until the document is loaded at that url
cy.get('#button').click() // gets the button and clicks it
cy.get('.message').should('contain', 'Hello World!') // do assertion

Normally one would expect asynchronous actions to complete after the next line of javascript, but with cypress, all these actions are blocking, with a timeout determined in the configuration. This allows us to write code in a structured way without having to create Christmas tree after Christmas tree or .then() after .then(). But it did require some getting used to. One has to reason differently about the code.

Integrating Cypress into our CI (Codeship) - Work in progress

We took some initial steps to get Cypress to work in our CI. The first step consisted in making the tests run inside a docker container instead of on our local machine. It should be possible to replicate this on the CI and run the tests there with the appropriate alterations to our configuration files.

At the moment we have our tests working properly in a local docker container. For this we built upon the example on GitHub by Gleb Bahmutov, one of the lead developers at Cypress. Integrating it into our CI seems not too difficult, although we didn’t push forward on this just yet. Some attention needs to be put into minimizing the build times on Codeship, so a direct feedback loop for us developers remains available.

Changing the way we think about the way our app works

This has probably been the biggest challenge, and the most abstract. We learned how to think about testability in a different way. Before, the end to end tests were more separated from developing time, but now we are reaching the point where we can start getting more frequent feedback on how a code change influences the way our app works. This will improve the developer experience as well as increase the knowledge the dev team holds. We can start working in a more directed way because of this.

We can now write tests for new features immediately and make sure they behave as intended. We haven’t applied a complete TDD approach to developing new features just yet, as often the way things behave in our editor aren’t completely specified when development starts. In the best case these tests can be written at the same time and in the worst case afterwards. This gives us the confidence that our app will keep working for our users. We need to instill these new habits as developers.

Another thing is that we are taking a full stock of our features and aim for a high testing coverage for these. Things that have slipped through the cracks before can now be tested.

Conclusion

So there you have it. In our opinion, Cypress saves us a lot of work. It helps us to catch regressions and maintain functionality across our app. It also forces us to think ahead of time about how we will implement certain new features, which will result in smarter decisions over time.

The biggest hurdles have been overcome. In the future, we would like to do the following (in no particular order):

1. We are already working pixel perfect testing into our app, with the Cypress Visual Regression package by Michael Herman, which enables us to more confidently refactor some of the code of our editor that no-one dared to touch before.
2. Create a test database against which our tests can run, so we can be sure that we are always working with identical solar systems, and won’t have to create solar systems from scratch for every test.
3. Mocking requests/responses for some external integrations, so we can start testing the outer limits of our app as well.
4. Making sure the tests run automatically in our CI software, so we can start deploying automatically once a branch has been merged, thereby speeding up development and delivery of features and bug fixes.

We’re very enthusiastic about Cypress and the possibilities it offers. Our software at Solar Monkey will be better because of it.