In my last Emulation 101 post, I talked about why we use emulation instead of expensive real-world systems testing or abstracted software-only simulation. This time, we're talking about the design of our emulation testbed and the software under test.
Emulation Testbed
Our emulation testbed has three main components:
- The emulation software / framework
- The software (or system) under test, and
- The middleware abstraction boundary that makes it possible
I talked about the emulation software and framework in the last post. The software or system under test is the software we're writing that needs to be tested - and emulation is the means we've chosen to do it. The "middleware" is the important part I'd like to discuss today.
The software we write is not designed to be run in emulation, it is designed to be run on real world systems. The fact that we use emulation as an aid in testing during the development process should not be evident in the software design. That is, our software under test should not be aware of the underlying emulation. How do we do that? Middleware abstraction boundary.
I'll admit that's my made-up term, but it refers to additional software, not part of the emulation framework, and not part of our software under test. It provides an application programming interface (API) to our software under test that essentially "impersonates" the real system.
An Example
Many of our test scenarios include mobility - platforms moving to different GPS positions during the test which rearrange their topology and stretch and stress the virtual radio links we are emulating. The emulator takes care of managing the virtual GPS positions of all platforms and the corresponding signal to noise ratio (SNR) and pathloss along the virtual radio links. The emulator even provides a nice Python API to get this information real-time during the running scenario. It would be very easy to use that API in our software under test - it would also be very unproductive to do so.
Again, our software under test is not being written for emulation, it's being written for real world systems. It is only being tested and developed in emulation. So when we move our software to a real world system and it calls an emulator API to get the platform's GPS location, there's going to be problem - the emulator isn't on the real-world system and there's no fancy API to help us. Instead, using the GPS example, GPS is available via a NMEA serial feed on a real world system, so in emulation, we write a piece of "middleware abstraction boundary" software (henceforth called "middleware") that uses the fancy emulator API to get GPS and then makes it available via a NMEA serial feed. Our software under test can now import GPS via the NMEA serial feed in both the emulation testbed and when it's moved to a real world system.
This goes for other metrics as well. Our software may query a radio for its SNR, but real world radios do not have fancy Python APIs available through an emulator. But middleware can be written to use the emulator API to get SNR and make it available on the protocol a real radio would talk, so our software under test needs only to be written against a radio's interface specification.
Turing Test
This creates more work, more software development - and software that really won't be used in the real world system. But this approach is much more scalable in the long run. Our radio interface middleware, written once, can run on every emulated radio and make them all appear with the real radio's interface specification. Our GPS middleware can be reused by any system in the emulation that needs location information. We're essentially creating a Turing Test for our software under test - it should not be able to tell if it's running in a real world system or an emulation.
Software Design Philosophy
Using middleware to create a "realistic" emulation boundary to present to our software under test means we can have a more focused approach to our system software design.
- Build for one interface Do not design for emulation and then design separately for the real system. Take the time to make the model and emulation environment as "real" as possible, using middleware software where necessary.
- Focus on functionality Focus development work on functions the system software provides and needs, not how to integrate it with one-off interfaces in various testbed scenarios.
- Modularize Can be many flavors, simple subclasses or complex microservices - the goal is to create reusable parts that can abstract away any scant dependence on the underlying emulation
Summary
Emulation abstraction through middleware is the liner in a pool, it's the trash bag in a garbage can, it's the oven mitt on your hand - it prevents leakage of messy data and APIs from burning you when you move your software under test from emulation to real world systems.
Posts
