Best Practices for Your Ephemeral Environment
Learn how to use ephemeral environments to improve code delivery and efficiency with best practices and free open-source tools.
In software development, ephemeral environments improve code delivery and efficiency by removing the resource contention and maintenance imposed by persistent test environments.
This article will describe the best practices for creating ephemeral environments and introduce a free, open-source project and a hosted service that can help you implement them faster and at a lower cost.
Summary of best practices for creating ephemeral environments
Some historical context for ephemeral environments
Ephemeral environments are a familiar concept to modern users of static website hosting platforms such as Netlify and Vercel. Website designers use ephemeral environments to preview changes before deploying their code to the live website.
Creating previews for static code is relatively straightforward.
However, web-based applications and e-commerce sites are too complex to preview with similar technology. These applications rely on complex backend code, third-party APIs, microservices, and databases, which means that the ephemeral environment must include the full application stack, not only the frontend code.
Large software as a service (SaaS) vendors have been creating full-stack ephemeral environments for many years to test new features and bug fixes before merging the code branches. However, setting up and maintaining such platforms is expensive - most often requiring a dedicated team.
New free or open-source projects and hosted environment as a service (EaaS) platforms have emerged recently that bring the technology for creating ephemeral environments within the reach of everyday Dev teams of all sizes.
Explanations of ephemeral environment best practices
In the following sections, we’ll elaborate on each best practice summarized above. As part of the last best practice, we’ll introduce a free, open-source tool and a hosted service designed to create and manage ephemeral environments integrated into a software delivery pipeline.
Containerize your code
The average developer associates containers with microservices and the Kubernetes container orchestration platform. However, taking advantage of the basic benefit of a container doesn’t require refactoring an application into microservices or orchestrating containers.
When it comes to helping create ephemeral environments, containerization ensures that the application environment includes all the required binaries, scripts, libraries, configuration files, environment variables, and OS patches and can run on a standard container runtime.
Popular container runtime environments include containerd, cri–o, and Docker. The last of these provides simple steps for creating an application image and containerizing code. Using these instructions, you would create container images for the various software components of your application and place them in a container registry.
Store your code on a hosted, Git-based version control system
It may seem redundant to mention the need for checking your code into a Git repository since it seems so commonplace these days. Still, we include it as a best practice since Git repositories have gone beyond version control and offer continuous integration (CI) functionality that simplifies integration with third-party tools and collaboration across teams. Checking your application code into a service like GitHub is a great starting point for planning ephemeral environments.
For example, GitHub Actions can streamline the workflows for creating new ephemeral environments upon issuing new pull requests and deleting those environments upon merging the code branch.
Using a hosted service such as GitHub can also significantly shorten the time it takes to integrate with a third-party environment as a service provider if you choose to pursue that route in the future.
Leverage Docker Compose or equivalent
Docker Compose takes the logistical complexity out of running a multi-container application, which is key when spinning up and tearing down complex ephemeral environments. Compose files are widely used and can be found in 800k+ Github repos.
With Compose, you define the application environment in a YAML configuration file and take advantage of commands to start and stop each service, view the status of the running services, and organize log files. The Compose getting started page is well organized and includes a simple example to familiarize newcomers with the technology.
Manage secrets safely
If you operate your application with a small team, managing secrets may not be a commonplace discussion topic. As you prepare to create your first ephemeral environment by integrating multiple tools and expanding the scope of users, it’s a good time to take inventory of your secrets.
Secrets include passwords, keys (used for encryption, SSH, and third-party API access), database credentials, tokens, and certificates such as the ones used with TLS and SSL.
Secrets can be stored in secret management services provided by CI/CD vendors, such as GitHub Actions, and injected as environment variables.
In the Docker Compose template example below, the Postgres credentials “POSTGRES_USER” and “POSTGRES_PASSWORD” are referenced using variable substitution through Docker Compose commands.
Integrate ephemeral environments into CI/CD workflows
Ephemeral environments are most valuable when created automatically each time a developer submits a pull request on a new feature branch. That’s when the process is so efficient that it feels almost magical. Once created, the ephemeral environment will be accessible via an assigned URL that will remain active until the pull request is approved and merged with the main branch, at which time the environment automatically disappears.
If you are using GitHub Actions, CircleCI, or GitLab, you can create a final step in your continuous integration pipeline by triggering a workflow designed to deploy an ephemeral environment. Using variables in your Docker compose file allows building and deploying ephemeral environments automatically as part of your CI workflow. For example, GitHub Actions supports the concept of “outputs,” which you can use to pass tags associated with the appropriate container images and use them during the build process with Docker compose. This page of instructions from Uffizzi’s documentation explains the steps in detail.
Seed your application databases
Full-stack ephemeral environments are complex because the applications are stateful and require server-side resources. The engineers engaged in functional application testing rely on these backend components to populate user interface elements, such as dashboards and reports, with real data.
Seeding the application database with data exported from a production environment, sometimes referred to as a SQL dump file, is a common technique to make ephemeral testing environments as realistic as possible for functional testing.
Services like GitHub limit the size of the files stored in the code repositories but offer services for storing large files that are ideal for storing a SQL dump file. These files can also be stored in cloud storage services like AWS S3 and downloaded on database startup.
Database vendors like Postgres, MariaDB, and MySQL provide Docker container images, including functionality for populating a database upon startup. Essentially, the database vendors include scripts with the Docker container images to automatically execute *.sql, *sql.gz, and *.sh files stored in the /docker-entrypoint-initdb.d directory. As an example, you can see the Postgres scripts here.
Collaborate across teams using ephemeral environments
The best time to inform a developer that a new feature needs improvements or contains a bug is while the code is fresh on that developer’s mind, not days or weeks after the fact. This rapid feedback cycle is one of the key benefits provided by ephemeral environments.
The feedback loop should involve the developer, coding peer reviewer, product manager, QA engineer, and, if available, friendly end-users who can provide timely feedback.
Those implementing ephemeral environments must invest time in planning for the cross-departmental collaboration that will unleash their value. Some of the ingredients for successful collaboration across teams include the following:
- Automatically inserting the ephemeral environment URL link (via API) in the feature or bug ticket related to the code branch for easy access
- Creating a temporary chat room (e.g., a Slack channel) for each environment where collaborators can directly exchange messages
- Enlisting the help of product managers to coordinate the feedback involving test engineers and friendly end-users
- Taking advantage of the developer having the code top of mind to make small refinements before a major context switch
Keep in mind that the value of these environments is highly dependent on the quality of the data bootstrapped into them. It’s difficult for peers, QA, and users to comment on functionality if there’s no data, or only extremely simple data, in the environment.
Remove ephemeral environments once the feature branch is merged
What makes an environment ephemeral is that it goes away once the feature or bug-fix branch associated with it has been merged. This action should be automated as much as the creation of the environment to avoid wasting computing resources.
A continuous integration event triggers the workflow that creates the environment upon a developer submitting a pull request. A second event triggers a workflow that removes the environment upon merging the code branch.
Use an EaaS service or open-source project to streamline the process
Designing and implementing an ephemeral environment and integrating it with an existing continuous integration pipeline requires two or three engineers and takes a few months to complete. Traditionally, only larger engineering organizations can afford to make such an investment of resources.
Open-source projects like Uffizzi significantly reduce the necessary time and resources by providing the code, know-how, and blueprint for execution.
For those with even less time, Uffizzi also offers a hosted version to get teams started with the help of customer support. As the first step, you can try the live demo, which doesn’t even require filling out a form or creating a free account. It showcases the concepts covered in this article using a sample project hosted on GitHub to give you a taste of creating live ephemeral environments in only a few minutes.
Last thoughts on ephemeral environments
Ephemeral environments increase engineering efficiency by creating a temporary, isolated, full-stack test environment for every feature branch and shortening the time it takes to release new features and fix bugs. An engineering practice that was once only affordable for large engineering teams is now available to smaller teams with the help of open-source tooling and hosted services specializing in ephemeral environments.
Like this article?
Subscribe to our LinkedIn Newsletter to receive more educational content.
