Case study: 64% increase in release frequency

How to Use Preview Environments Effectively (this is how we do it at Uffizzi)

‍
I’m writing this because after talking with several of our users and potential users I realized that it’s often not readily apparent how Preview Environments should or can be used effectively in your Software Development Life-cycle. The concept of ephemeral environments per pull request is not something completely new and it’s not something that we invented, but it was new to us and odds are if you’re reading this it’s new to you as well!  

Historically-speaking, having a Preview Environment capability meant rolling your own solution which has also meant that both the capability and the knowledge have been pretty siloed. The teams who have built their own solutions are a fraction of the total software producing industry, so in the same way we’re making Preview Environments readily available to any team, we also want to share our lessons learned to help other teams get a head start on improving their development velocity.

We built Uffizzi, and we still had to learn through some trial and error how to best use it for maximizing our own development efficiency (and we’re still learning!).

How much of a difference do Preview Environments make?

The potential of how ephemeral preview environments would increase development velocity for software teams was, of course, why we built Uffizzi, but I admit I was nervous to dig into the metrics—our process feels faster, but is the hype true? And how much does it really help?

For my assessment I went back a total of 8 months in our repo history so that I could evaluate our performance the 4 months prior to using Preview Environments and then contrast that with the 4 months after.

After we implemented our own Uffizzi on Uffizzi solution our Deployment/Release frequency went up dramatically—and it’s still trending up because we’re still getting better at our own process.

In the before Preview Environments period, from Mar to July 2022 we had a total of 23 releases. We averaged 1 release every 5.6 days or roughly about once a week.

In the after Preview Environments period, between July and Nov we had a total of 56 releases—that means we averaged 1 release every 2.2 days—a 2.4x improvement!

But that’s not the whole story. Let’s make sure we’re being as scientific as we reasonably can be. These numbers need to be normalized because throughout that time we also nearly doubled our engineering team. So it would make sense that we had a lot more output.  

When I normalized the data for the number of engineers over the given time periods we see a 64% improvement in our release frequency. We went from an average of 5.26 releases per engineer in the before period to an average of 8.62 releases per engineer in the after period.

In summary we were releasing around once a week in the before period and now several months into using Preview Environments we are averaging one release for every working day. Across October and November we released 42 times which averages exactly once every workday!

What else does the data tell us?

36% improvement in total issue throughput‍

In the four months before Preview Environments we pushed 59 issues to production or 13.5 per engineer. In the four months after implementing Preview Environments we pushed 119 issues to production or 18.3 per engineer.

‍20% reduction in issues per release‍

This means we’re making smaller changes and at a faster rate. Prior to having a preview environment for every feature branch we necessarily batched our releases, now it’s much more common to release single issues as they become ready.  ‍

What about qualitative improvements?

‍This is a list of benefits that our team came up with comparing how we used to operate to how we operate now:

• No time spent tracking down “who created this bug?”
• No time spent resolving merge conflicts ahead of testing
• Less Context Switching.  Feedback loop is fast so Developers can address issues before they’ve moved on to another feature.
• It’s easier to resolve merge conflicts at the feature branch level
• Fewer returned tickets
• No test environment data inconsistencies to manage
• No fratricide by another developer’s commit(s)
• No “code freezes” to maintain stability of a branch/environment for Testing or a Release
• We don’t spend time managing environments. If we break a Preview Environment we just throw it away and start over.‍

How does what we’re doing now differ from what we were doing before?

‍Previously we used a Persistent, Shared Test Environment Strategy. We had two persistent test environments: a QA environment and a UAT/Staging environment. I would describe this as a traditional develop and test workflow. Developers would check out a feature branch from develop, they would write code and test on their local machine using docker-compose to spin up their local develop environment.  

When a feature was ready to move “right” in the testing process they would open a Pull Request (PR) to our QA branch. At this point the developer would resolve any merge conflicts and complete the merge. The merge kicked off our CI/CD and the updated QA branch was built and deployed to our static environment at qa.app.uffizzi.com.  

The problems we experienced here are common to all shared environment strategies and are the exact reason we built Uffizzi. We referred to our QA branch and the QA environment as “dirty” and “polluted” because anyone could merge to it, it was shared amongst our entire team, and bug introductions were fairly commonplace.

Once features were deployed to QA the tester on our team would test for the desired functionality. Often this process was interrupted by bugs that had been introduced by any one of a number of recent commits from multiple feature branches.  

As a rule our product team would not review features in QA because we considered it unreliable. Why waste time when there was a good chance that what we were testing wasn’t stable enough to provide reliable results. This meant that the product team was heavily delayed in their ability to provide feedback and subsequently our iterations were quite slow.  

When functionality failed or bugs were found in QA the first question we would ask is “who created this?”. After some time determining responsibility, the responsible developer would pull the QA branch to their local environment and debug. For any fixes they would again have to open a PR to the QA branch and resolve any conflicts before merging. A lot of time is wasted in this deconfliction step because of how often the QA branch was changing given that the whole team was using it.

When features would pass testing they would then go into code review. If there was an issue in code review the responsible developer would fix the issue, merge to QA again, and the testing process started over. When features passed code review they would then be merged into the staging branch. At this point either QA or product or both would test the features again and those that passed would be merged into develop and then develop into main which would release the feature(s) to production.    

In summary we used two shared environments and had an overly burdensome multi-step process that delayed our development progress significantly. We were only releasing about once a week.

Now we have eliminated both of our persistent, shared environments and we only use ephemeral Preview Environments for every Pull Request (there are a few exceptions which I’ll note).‍

The major differences in our overall flow‍

• Testing happens pre-merge, in a clean environment for every PR. When we open a PR an environment is automatically created in the background without any manual steps. This means we don’t have to deal with merge conflicts to initiate testing or code review.
• Merge conflicts are handled in feature branches. With every new change pushed into production all of the feature branches are re-based from develop. This makes it easier to manage conflicts because these branches are much less busy or “polluted” than a shared QA branch.
• Individual Developers take a more active role in the testing. Each developer has an environment that is isolated to their PR contribution and they can confirm functionality without the impacts of other contributions.
• Code Review occurs after a Developer confirms functionality but before any merging. There’s no need to repeat testing by QA when a code review fails.
• Testing flow has become much more sophisticated and responsive due to a risk-based tagging system.
  

By the Numbers

1. A developer checks out a feature branch from develop.

2. They develop and test locally using `docker-compose` to manage their local Development environment

3. They open a pull request to merge the feature branch into develop.

4. The CI pipeline is automatically triggered. That includes:
- Image build
- Linter/Test
- Deploy a Preview Environment
- Post a comment with the deployment URL to the PR

5. Developer confirms functionality based on the ticket in the Preview Environment

6. Issue moves into “Code Review” where a Team Lead or an assigned peer does the review

7. After the issue passes “Code Review”, the feature enters “QA” and testing follows an appropriate testing pathway in accordance with feature risk where the speed of the pathway is directionally proportional to associated risk.  
‍
The flow is assigned with an issue tag by the Product Manager (PM) or Team Lead (TL). If no tag is assigned the Default is QA Test.
‍
Test pathways:
- Fast Track - Developer Test (low risk features)
- Routine Track - QA Test (medium-high risk features)
- Deliberate Track - Product Manager Test (high risk - major change to UX features)
‍
For features deemed low risk—unlikely to impact other functionality—the feature is tested by the Dev who created it. After they test, it immediately moves to “Ready for Release”. This makes for a very fast track because the Developer can write it, test it, and ship it.
‍
For features deemed medium-high risk—potential to impact other functionality—QA will test.
‍
For features deemed high risk a Product Manager will confirm functionality before release.
‍
Responsibilities:
- Developer’s core function is confirming the functionality that they added works as intended.
- QA’s core function is testing edge cases and to mitigate the risk of regressions.
‍

8. Automated tests are set to run as part of the CI pipeline against every Preview Environment.

9. After the automated tests pass, the Dev/QA/PM opens the application and tests the new feature manually (or uses the environment URL in other services to test the feature).

10. After tests are passed and the issue becomes “Ready for Release” the feature branch is merged to develop

11. After the develop CI pipeline passes we merge develop into main. This process deploys main to production.

12. At this point all open feature branches must be re-based off of develop and the process starts over with individual feature branches being released as they are ready.

Other Relevant Information

For Epics—where several feature branches need to be released as a set—we create a Release branch and open a PR against develop. This kicks off the same process and we get a Preview Environment of the Release branch to test against.  

If a Preview Environment has expired (time-out), whoever is testing restarts the “Deploy Uffizzi Environment” step in the pipeline and a new environment is created.

In order to use OAuth (GitHub sign-in) and other GitHub integrations with our preview environments, the tester manually updates the callback URL in a Github App we created for our Preview Environments (in the future this will be automated when we roll out a dynamic OAuth solution for Preview Enviroments as part of our platform (Sign up for our newsletter to get updates like this).

Test Data for Preview Environments - We don’t seed our Preview Environments with Test Data but we have several customers who do and there are several ways to do this. You can find more information on this under our Preview Environments guide.

Occasionally we use a traditional QA environment for specific edge cases i.e. if we need to test SSO or data migrations against a larger or more persistent data set or if we want to test something related to the infrastructure itself. The QA environment is not used often (1-2x a month) and is put in a dormant state to save $$.

Conclusion

‍The switch from persistent, shared test environments to ephemeral Preview Environments has been an absolute game-changer for us. A 64% increase in release frequency and a 36% increase in issue throughput blew away our expectations for just how much better our new process is. What’s exciting to us is that we’re still getting better and we’ve had a major cultural shift. We think iteratively and we think fast: how quickly can I get this issue merged to production? Oh, and no one wants to be on the receiving end of a returned ticket. It wasn’t someone else’s commit that broke it!  

I hope this has been helpful. If you’d like to try Uffizzi please check out our quickstart repo where you can create your first preview environment in just a few minutes just by forking and opening a PR to your fork—you don’t even have to touch the Uffizzi UI!