Jeff Feldstein is the Senior Director of Quality Assurance at Zuora

Testing applications on subscription-based software platforms versus traditionally purchased computing environments is a night and day comparison, like the difference between working on HPCloud or an old MS-DOS desktop. Subscriptions turn what can potentially be a painful slog into an flexible, intuitive process of discovery.

First some background for non-developers: when you test a new software application to make sure it can handle your processing needs, you don’t experiment with the live production version. You create a “miniature” duplicate version that mimics the capabilities of your production system, without carrying the carry the entire data load. This miniature version of your application is called a testbed, and is usually comprised of a collection of servers that are isolated from your live production environment.

Your testbed can undergo lots of operating system and application configurations. Lots. Many more than in your live production environment. That’s what makes it a rigorous testing environment. To track all these configuration changes and their effects, you take a series of snapshots of the entire network called “images.” That’s where the beauty of subscriptions comes in.

In a traditional, buy-as-you-need computer environment, you need to know in advance exactly how many machines and how much memory and storage the test process will require. The problem is that you may not even know what you need to test, and fixed processing assets leave you little or no room for creative experimentation. If you wanted to take a look at, say, a new version of an operating system, you would need to wipe a machine clean and spend hours rebuilding it.

Expanding your test lab can require additional floor space, racks to hold servers, air conditioning and even extra power capabilities. And what if you have an urgent new need to comply with a security issue found in your operating system or Java? You have to stop using at least one of the testbeds to test this new configuration, slowing down the other work.

When using a limited resource of owned machines, teams can wait hours or days for a testbed to become available (believe me, I’ve been there). Even if you received approval to purchase a new machine, its several days, at best, before its online.

Subscriptions change everything.

Need more disk or memory? No problem – just fire up a new machine, using one of your saved images and extra usage-based capacity. Want to experiment with the reliability of new feature? Need to compare performance of an enhancement across different configurations? Want to update your automation infrastructure without disturbing the servers already in use? Wondering what end user response times are using servers placed around the world? No problem – just start another machine.

You get the idea.

Naturally, all this flexibility comes with some caveats. Here are five rules for testing in subscription-based software environments :

Delete – Delete your assets when they’re longer needed. Since you are charged for all assets you’ve provisioned, you’ll pay for them whether they’re in use or not. Keep in mind that you are charged for everything you use including virtual machines machines, storage (whether attached to a VM or not), allocated ip addresses, stand-alone databases, domain name tables, etc. Diligent monitoring with assigned owners is required to make sure you are deleting machines no longer in use, and you are also deleting all the assets you’ve created that were attached to the testbed.

Archive – That being said, archive any data you may want to use in the future because once you delete it, its gone. There is nobody in IT you can ask to dig out a back-up for you. Most services providers give several choices of storage at widely varying performance and cost. Make sure you are archiving to the lowest cost storage you require.

Performance – Performance testing on subscription machines requires some thought and knowledge about CPU/Memory/Disk Usage. You need a thorough understanding of how your provider provisions these resources, which parts remain constant and which vary significantly between runs. As in any performance test, subscriptions don’t let you get away from multiple runs to get an accurate performance measurement.

Security – Ensure you are using the correct VPN technology, strong security groups, etc. Don’t forget to pay careful attention to secure all of your assets. Make sure you are on a private network and only authorized users can access the machines and the data. Consider security audit from the same team and tools that are monitoring your production systems.

Cost – You may not know the real cost of a testbed until you’ve used it for a month. Most providers of cloud servers will only bill you once a month, and they usually do not supply a running counter on usage. Go slowly that first month, and carefully watch the estimated costs before firing up a new asset. Add new machines only as you need them, and try to phase their use. Consider yearly leases for software and hardware assets that you are sure you need for at least 12 months. Most providers offer steep discounts for yearly usage.

For more on this topic, check out our Academy Guide: Subscription IT: Delivering High Performance, Availability and Scalability.