Automated Web Application Code Testing On A Budget
How to thoroughly test your code but keep costs under control.
- Automated Testing On A Budget
The Typical Corporate Automated Testing Environment
For a software development manager or architect that works for a large company with endless resources then it is easy to simply follow the idealistic automated code testing philosopy that goes something like this. "Yes boss, we have 80% code coverage with both our unit tests and our automated functional tests. Here is a chart. Look at all of this green. You can see what a great job we are doing!" This will of course make all of the non technical types exceedingly happy because green is a pleasant color and they like to see lots of green.
For the other employees that work in companies with limited budgets then this sort of approach is naive, and frankly quite expensive. There definitely is an important place for testing in any development lifecycle but like most things the real goal is to get the most bang for your buck (aka. the most for your money).
Test Driven Development Fallacies
Let's start by debunking some fallacies
- Code coverage equals code quality - Horrible code can be covered 100% by unit tests. Writing unit tests should improve code quality
but it in no way guarantees it.
- Test driven development is the prefered way to write all code - The truth is that 90% of developers took TDD training in
2015 when it was all the rage, played along and quickly went back to writing the code first. The reason for this is that it is a
huge waste of time to write tests for something that doesn't even exist, especially when the developer is in the creative process of
figuring out what they are trying to do. TDD can work for special cases where the inputs and the outputs of a method are known right from the start.
However, seeing as how in the real world a developer is actually figuring out how many methods to write, how many layers of methods to write,
and even what these method signatures are, then TDD is more or less impossible, until the thing that is to be tested has formed and by that
time most of the code is done. In actuality, a developer does TDD in their head while they code because they are imagining the paths through
the code while they write it. The smarter and more experienced they are, the better they are at visualizing this while they work, and the more they limit the code to predictable paths.
- Tests make the code easier to maintain - This depends of what is meant by easier. It is easier to change the code without breaking it,
because ideally the tests should tell the developer if a mistake was made. However, a code base with high coverage will force the developer to
break the tests with almost any code change, and therefore fixing all the tests every time something is refactored makes the code more difficult to maintain.
- Manual Testing - Testing done by human being. Your QA department. Humans are expensive but they will also find things
wrong that your developers never thought of. Never get rid of them completely, just reduce the amount of tedious testing they need to do with automation.
- Unit Testing - This is code that tests code at the method or class level. It isolates the object being tested by
faking everything around it and sends information into it and tests for expected output. Unit tests are usually written in the same
language as the code they are testing and are expected to run fast because they should be done in memory and not interface to any other systems.
Unit tests typically take 3 times as long to write as the code they are testing, so they are expensive. Furthermore, very high code coverage
percentages in unit tests actually ends up enforcing how the code is written rather than what the code does, and therefore pours a
layer of virtual concrete over your code base that needs to be jack hammered up and rebuilt every time you refactor.
- Integration Testing - Similar to unit tests but these tests target calls from one system to another. For example,
testing code that lives on the web server but queries against the database. Unlike unit tests the integration tests need to operate
against a real version of the other system because they are testing the real interaction between the two systems. For example, a
DAO (Data Access Object) Test must query against a real database with tables in it because it needs to verify that the fields
it is running against actually exist and are of the size and type it expects.
- Automated Functional Testing - Software that mimicks a manual tester. It's basically a robot program that acts as a
human user and manipulates an application from the front end.
- Client side script testing - This is a type of testing particular to web applications because it
is testing code that runs on the browser. There are actually two categories of client side script
Client side utility scripts - Deep libraries of script
that are used potentially in multiple spots and have
multiple levels of method calls.
Page level scripts - Small pieces of code that utilize
other larger libraries and are called in events in a web
page. Code that hides a div or changes the color of
something for example.
Now it's time to get the important part of this article. How can a company get the most bang for its buck and accumulate more gold bars
(see photo above), instead of pouring money down the drain.
- Target your unit testing - Forget about blanket percentages across your entire application. That is like trying
to keep every room in an entire hospital as clean as the surgery rooms, which is a waste of money and impossible.
Unit tests should target two kinds of code
- Important code - Code that simply can't break because it is core to your business.
For example, code that has a direct impact on financial transactions. Its probably a good idea to cover this code at a 100% level, but it should be written to
isolate it from all of the other less important code.
- Complex code - Code that is hard to understand and easily broken because of its complexity. This needs to be covered because
the testing framework will assist in its creation to begin with and because it needs to be protected against accidental breakage later.
- Important code - Code that simply can't break because it is core to your business. For example, code that has a direct impact on financial transactions. Its probably a good idea to cover this code at a 100% level, but it should be written to isolate it from all of the other less important code.
- Still pursue high test code coverage on integration objects - Integration testing is of high importance and should be thoroughly covered.
Also, if the code is written correctly, for example DAO classes, should have very little branching and should be easy to cover at near 100%.
Keep business logic out of the integration code so that it is as linear as possible and the writing of unit tests should be straight forward.
Integration tests are good insurance against surprises from your own company's database or from company or third party services.
- Limit Automated Functional Testing To Mostly Smoke Tests - The great thing about AFT is that it covers the entire stack.
Just bringing up that web page probably exercises a couple 1000 lines of code, all the way from the UI to the database.
Smoke tests are relatively cheap to write because they only include enough details to spider around the website the way a Googlebot would.
Detailed AFT tests on the other hand are very expensive to write and also quite brittle. Small changes to the UI can break detailed tests as can changes to the
data in your test database. Some detailed AFT tests might be in order but limit them to avoid costs incurred by having to refactoring them frequently.
unit test to test simple script like this is a very bad return on investment. A developer that edits this script but never pushes that button to see if
the code works should just be shown the door instead. On the other hand, if a script utility is written that has hundreds or thousands of lines of code
then it should be unit tested and treated like server side code. Also, large js libraries like React and Angular move a large portion of what used to be
server side to the client and therefore this script code needs to be covered with tests using the same philosophy outlined above in the unit testing section.
- Avoid testing overlap between unit tests and automated functional tests - A good portion of "less important" unit testing can be avoided if
the same code is exercised by AFT tests. For example, a lot of server side controller code and views will be fully exercised by AFT smoke tests so this
code does not need to be thoroughly covered. The same can be said, for example, for React and Angular routing code.