Simulated Ephemeral Cloud Environments for Embedded Systems

Why Ephemeral Environments for Embedded Systems

During the last few Open Source Summit & Embedded Systems conferences, our team at Uffizzi met with several embedded system teams. We heard from them about the challenges they face in testing and verifying their designs in a way that is fast, saves costs, and allows fast releases/feedback. 

‍

We realized as the world becomes more and more dependent upon the creative applications of embedded systems, the need to be innovative, incorporate feedback, move fast, and do all that at a reduced total cost is essential. However, optimizing innovation with speed and cost widens the room for error. Very often, embedded systems require very high reliability, particularly for regulated industries where malfunctioning products can have serious consequences. To ship features with speed and minimal — ideally zero — errors requires feedback, integration, testing, and delivery to be continuous.

‍

But, during our research, we learned that achieving this state of scalable and efficient continuous integration, testing, and delivery is challenging for embedded systems due to several reasons, most of which were persistent across projects. Listing some of the most notable ones: 

‍

• Setting up development environments to test the hardware, the software, and the interaction between them takes time. Plus, the process of installing the tool is not consistent across developers. 
• Dependency hell can be more prominent in embedded systems — continuous updates to dependencies might not be easily translated to local development environments.
• Device simulation, in the absence of real hardware, is limited by memory and space, which can quickly become expensive. Due to this, testing becomes costly and often, embedded technologies are able to test their product only during later stages.
• Developing/testing on shared environments to save cost reduces visibility into impact of individual SW/HW modules and makes debugging hard. 
• Lack of proper tooling tailored for testing embedded systems
  → most tools do not integrate into their GitOps, CI/CD practices
  → tools are limited by computing resources, which can make running dynamic and load tests hard

These engineering challenges introduce the below operational challenges:

• Time to Market increases.
• Takes longer to get feedback.
• Takes longer to iterate and ship.

What Our Solution Offers and How it Can Help Embedded Technologies

To alleviate the above-mentioned challenges faced by embedded systems, we ventured into designing a cheap and operative ephemeral environment solution for OpenTitan, an open-source, silicon, Root of Trust project, administered by the lowRISC foundation. The OpenTitan project develops both hardware and software/firmware, and so our approach to designing this solution was to create fully-configured, ephemeral environments, where each feature change — software or hardware  change— would have its isolated environment with the SW built and HW simulators configured to simulate the OpenTitan device, and where that feature can be tested.

With these goals in mind, we built an ephemeral, simulated-hardware environment prototype solution for OpenTitan. We ran the same build and test mechanisms locally on our systems and compared the results with lifting the workload to an ephemeral test environment running on Cloud via Uffizzi. We are listing notable features of these ephemeral environments and the value they added to embedded systems development:

‍

• Integrated into OpenTitan’s CI/CD pipelines on GitHub: a new ephemeral environment is produced for each PR and is updated with every change to the PR, allows agile development,
• Containerized environments: do the work once and it can be easily scaled, built, and maintained; made for speed, consistency, and dodging dependency hell,
• Fully configured with tools needed to build the OpenTitan SW and HW,
• Ready-for-use simulated HW, simulated by Verilator,
• Interface to the HW provided through a terminal shared over the web,
• Discrete, purpose-oriented, isolated environments: ephemeral environments ensure features are tested in isolation and add more visibility into the impact of a feature,
• Optimized based on actual resource utilization. Simulating the HW requires a significant allocation of resources. The containerized, ephemeral environments are built with dedicated memory and CPU to OpenTitan tests without overprovisioning. We found the optimal performance for OpenTitan ephemeral environments at 16 GiB available RAM and 4 vCPUs. Our environments took at least 15 minutes less time than the time it’d typically take to run on-device tests as mentioned by OpenTitan.
• Zero to minimal setup required to test against the benchmark. Get feedback from technical and non-technical stakeholders early on. Shortened release cycles.

‍

Summarizing these features, here is what our ephemeral environments built for embedded systems offer: integrated into GitOps practices, self-contained, cheap and fast, consistent, fully-configured, purpose-oriented, isolated, resource-optimized, scalable, ready-to-use, and shortens release cycles.

How We Built Ephemeral Test Environments for Embedded Systems and How Do They Work

Let’s dive deeper into understanding how we built an ephemeral test environment for OpenTitan and how the environments work. Similar steps can be followed for configuring ephemeral test environments for other embedded system projects. 

Step 1: Forking OpenTitan

‍

We started with forking OpenTitan’s GitHub repo. You can find the fork here to follow along. As mentioned previously, ephemeral test environments for OpenTitan are added into OpenTitan’s CI/CD pipelines on GitHub so the process of spinning up a new environment for feature-testing and delivery is done continuously per PR.

‍

Step 2: Containerizing OpenTitan 

‍

We containerized OpenTitan so it is self-contained and can easily be orchestrated. Since OpenTitan builds both SW and HW, several dependencies must be installed to compile the SW written in C and simulate the HW using Verilator, a SystemVerilog simulator. Containerizing the application ensures that all the needed dependencies are installed and that there are no outdated/conflicting dependencies. Building the SW without containerization would have introduced major bottlenecks because of the strict dependencies between tools, adding delays to the release lifecycle.

‍

OpenTitan comes with an experimental Dockerfile, which had mostly all of the tools that were needed to run the OpenTitan ephemeral test environment with simulated HW. We based the Dockerfile used by Uffizzi (Dockerfile.uffizzi) on OpenTitan’s experimental Dockerfile and added the ttyd dependency to it - ttyd allows terminal sharing over the web. With this dependency added, developers will have a terminal interface to the OpenTitan environment, where they can build the SW and run on-host and on-device tests. 

‍

Step 3: Orchestrating the container

‍

Created a docker-compose.uffizzi.yml file to orchestrate the OpenTitan container. This docker-compose file is where we add the Uffizzi integration, which will spin up the container using the defined image, and either deploy a new ephemeral environment on Uffizzi cloud or update an existing one. More on this in the next step. 

‍

This docker-compose defines an entrypoint-port into the terminal shared over the web through ttyd. We also used this compose to mount the OpenTitan project onto the container, the interface to the project is then provided by ttyd. Furthermore, this file also defines the resources (memory and CPU) needed by this container. Through trial and error, we found the optimal resource requirement for running OpenTitan ephemeral environment to be 4 vCPU and 16 GiBs. 

‍

Please note, this prototype requires more memory and CPU than the Uffizzi Platform defaults. Contact us if you’d like to request more resources for your project. 

‍

x-uffizzi:

 ingress:

   service: opentitan

   port: 7681

‍

docker-compose.uffizzi.yml with entrypoint into OpenTitan

‍

Step 4: Defining GitHub workflows to provision the ephemeral environments

‍

The power of ephemeral environments lies within the realms of CI/CD. So Uffizzi leveraged GitHub actions to orchestrate building the ephemeral environments, collecting feedback, updating the environments as the feature gets iterated on, and finally deleting the environment if it is no longer required. 

‍

We built a 2-stage GitHub workflow, to extend GHA-triggered ephemeral environments support for open-source projects. The 2 stages of this workflow are:

‍

• The uffizzi-build.yml action: build and push the OpenTitan image with the most recent changes of the PR.
• The uffizzi-preview.yml action:  create, update and delete the environment with the image built by the uffizzi-buiild workflow.

‍

Since embedded system projects heavily rely on HW simulators, especially early on in the development cycle, our focus with the OpenTitan ephemeral environment was to run the simulation effectively while optimizing resource consumption because running HW simulators can easily hike the resources consumption and cost. The uffizzi-preview.yml action helped achieve this by deleting the environment and freeing the resources as the PR is closed. 

‍

Step 5: Triggering environment creation

‍

After adding the above 2 GitHub workflows in our fork of OpenTitan (if you’re following this example for your project, make sure the actions are added to the default branch of your repo, else the 2-stage workflow will not work as desired.) we opened a PR in this fork. This triggered the uffizzi-build.yml, which then triggered the uffizzi-preview.yml workflow and created a new environment, ready to run tests against, interfaced through a terminal shared over the web. 

‍

Considering we leverage Docker to build our image and also use docker-layers caching from GitHub actions, the first run of the uffizzi-build action with the uffizzi-preview action took around 10 minutes. However, subsequent updates to this feature branch were 4X times faster, and the two actions combined took around 3-4.5. We ran the actions a couple of times to ensure this behavior was consistent—check here. This is significant because reviewing features and receiving feedback in isolation, parallelly and continuously, as they are being developed, is scalable and allows faster iteration, hence faster releases. 

‍

Step 6: Building OpenTitan SW and running on-device and on-host tests on the environment

Once the environment is created successfully, a comment posted on the PR will give us a link to the ephemeral environment. Visiting the environment, we see a terminal with /home/dev/src as the entrypoint. This is where the OpenTitan application (both SW and HW) is mounted. Here are the components in this directory:

The code related to building and testing the SW and the HW are in the sw and the hw directories, respectively. There are three major parts to the OpenTitan software stack:

• The device software, which runs on the primary core within the OpenTitan platform chip.
• The otbn software, which runs on the OTBN cryptographic co-processor within the OpenTitan platform chip.
• The host software, which is run on a host device and interacts with an OpenTitan device.

Building of the SW is handled by Bazel. The device software can be built for multiple OpenTitan devices and memories. Running tests on multiple OpenTitan devices (whether DV or Verilator simulation or an FPGA) which involves building several software images for multiple memories, building the necessary SW for the target device (FPGA/Verilator), etc was taken care of by Bazel.

Like many embedded systems, OpenTitan runs 2 categories of tests: on-host and on-device. On-host tests are compiled and run on the host machine, so these will be interacting directly with the container; these are hardware-independent tests and perform unit tests for device software. On-device tests are compiled and run on (simulated/emulated) OpenTitan hardware and so they’ll interact directly with the simulated OpenTitan hardware. The ephemeral test environment passed both these test suites successfully. 

To run both the test-suites, we used Bazel, which built the OpenTitan SW for our target HW (Verilator, in this case). The on-device tests could take longer, so we utilized screen-sessions to let the tests run even when SSH disconnects. 

‍

The on-host test suite does not require HW simulation and so these are faster. The suite passed successfully on our ephemeral environment and took 1–2 seconds. The on-device tests, however, require HW simulation and take more time. According to the official OpenTitan docs, Bazel can take an hour to build the simulation. The ephemeral environment, with optimal resource planning, took between 35–40 minutes for the first build and subsequent tests took around 1–8 minutes depending upon the kind of test that was running. This was remarkably faster than the time it’d typically take to run on-device tests, as OpenTitan suggests. The ephemeral environment reduced that time by maximizing resource efficiency. 

‍

Below are the results of running on-device, sensor_ctrl_alert_test  with optimized resources successfully on the ephemeral test environment. This test took 43 minutes in total, 35 minutes to build the HW, where the test would run and 8 minutes to build the sw this particular test was written for. This is at least 15 minutes faster than the time it’d take to run on-device tests the first time. Subsequent tests on this environment were faster. 

‍

As described by the OpenTitan official docs, the HW simulation takes an hour to build, add to that the time it’d take to build the sw. Our ephemeral environment solution significantly optimized the run time for on-device tests.

‍

As can be seen, the power of containerized ephemeral environments came in very handy—we were able to set the optimal amount of resources needed, depending upon the type of test that we’d run in the environment, without running the risk of over or under-provisioning. The environments had dedicated resources, which prevented throttling. 

‍

Step 7: Running Comparative Analysis and Summarizing Performance

After comparative analysis, here are some important points we noted while running OpenTitan SW on simulated OpenTitan HW in Uffizzi ephemeral environments and running it locally:

• Without these environments, each developer and stakeholder would have to go through several steps to build OpenTitan. The list of steps to build OpenTitan on a Linux machine can be found here. First off, each Developer will need a Linux system, with certain dedicated memory and CPU. Next, they’ll need to install the SW and HW dependencies. Embedded projects tend to have several dependencies and most of these dependencies must be of a specific version. Local installation alone took a long time. With Uffizzi ephemeral environments, we were able to reproduce instances of OpenTitan on the cloud, at least 10X times faster. 
• The availability of resources (CPU and memory, majorly), greatly affected performance. With fewer available resources, the HW simulation and the tests invariably either ran for much longer or failed due to a timeout. The inaccessibility to an efficient system can make testing and benchmarking time-consuming. We were able to spin new environments with needed resources quickly and deleting them did not mean wasting efforts, since a zero-effort environment could be spun again easily.
• As HW simulation required more resources, if these testing environments were overprovisioned and sat idle for long times, the operational cost could have spiked very quickly without yielding results. On the other hand, is the risk of underprovisioning. Also, most local systems are not equipped with enough memory and CPU to run the HW simulation. The ephemeral environments eliminated the repercussions of over and under-provisioning. 
• Running ephemeral test environments allowed parallelism at scale. Multiple features could be tested in-parallel without overburdening the host the tests are running on. 
• These environments allowed modularity, where features could be built and tested separately in isolation. Eliminating the need to test the HW and the SW changes (changes to the device SW, the OTBN SW, and the host SW) through the same criteria could make development faster and quicker. 
• Engaging technical and non-technical stakeholders for feedback on a pre-released feature early on was easy because the environments come fully configured and can be tested with almost zero extra effort.
• Security patches and dependency management is easier—when building the SW locally, we came across conflicting dependencies, which required fixing and rebuilding. The ephemeral test environments were pre-packaged with all needed dependencies, and hence building the SW and the HW was easy and regulated. 

Value Proposition — The Uffizzi Difference

We ran embedded systems without ephemeral environments to understand the impact and the value Uffizzi provisioned ephemeral test environments for embedded systems offer. As we saw, adding simulated ephemeral test environments into OpenTitan’s CI/CD practices not only reduced the pain encountered during embedded project development, right from planning to release and gathering feedback but also added capabilities to quicken turnaround times. Speed, resource and cost optimization, ease of maintenance and system updates, shorter development and release cycles, and continuous testing throughout development and release cycles are some of the most significant achieved benefits of developing embedded systems with ephemeral environments.

‍

These environments will help reach the needed state of scalable and efficient continuous integration, testing, and delivery which has traditionally been hard for embedded technologies. The ephemeral test environment approach to developing, testing, and releasing will accelerate the time to market, keep customers happy by engaging with their feedback and reduce costs. If you, too, work with embedded systems and would like to reduce your time to market, adopt agile practices, cut down your operational costs, and improve product quality with early feedback, check Uffizzi out, and connect with us here.

‍