Moles vs. JMockit Comparison
Overview
This document compares two similar test double injection frameworks:
JMockit for Java and Moles for C#. We created a set of use cases that
capture most/many of the things programmers have to do when unit testing
code with test doubles. We then attempted to create a test case in each
framework to demonstrate each use case.
The JMockit test cases can be found in our JMockit
Evaluation. If you're unfamiliar with JMockit, you may want to
start with our JMockit: The Basics.
The Moles test cases can be found in our Moles
Evaluation. If you're unfamiliar with Moles, you may want to start
with our Pex & Moles: The Basics.
Pex is a complimentary framework for parameterized unit testing, but since
there is no Pex equivalent for JMockit, we do not cover Pex outside of
that document.
The result of this document summarizes the results of both evaluations
and draws some higher-level conclusions on their relative merits.
Evaluation Results
Our common set of use cases as well as the ability to achieve each use
case with JMockit and Moles is summarized in this table.
ID
|
Feature
|
JMockit
|
Moles
|
1
|
Inject test doubles without changing the API
|
YES |
YES |
2
|
Replace final/non-virtual functions with test doubles
|
YES |
YES |
3
|
Replace static functions with test doubles
|
YES |
YES |
4
|
Replace constructors with test
doubles
|
YES
|
YES
|
5
|
Replace anonymous inner/nested classes with test doubles
|
NO
|
NA
|
6
|
Replace inaccessible (e.g., private) inner/nested classes with
test doubles
|
NO |
NO |
7
|
Replace subclass and superclass
functions with test doubles
|
YES |
YES
|
8
|
Test doubles can delegate to real
instance
|
YES |
YES
|
9
|
Hand-coded test doubles can be
injected.
|
YES |
YES
|
10
|
Test double injection can be toggled
|
YES |
YES
|
11
|
Test double injection can be narrowly
targeted
|
YES (brittle) |
YES (brittle)
|
12
|
Replace any implementation of an
interface with a test double
|
NO
|
NO
|
13
|
Safely replace a class library class
with a test double
|
YES (brittle)
|
YES (brittle)
|
14 |
Inject async-friendly test doubles
|
YES
|
YES
|
15 |
Inject thread-friendly test doubles
|
NO
|
NO (crashes)
|
16 |
Associate failed assertions in test doubles with test cases in the
same thread
|
YES
|
YES
|
17 |
Associate failed assertions in test doubles with test cases in
different threads |
NO
|
NO (crashes)
|
Similarities
The first thing to note is how similar both frameworks' capabilities are.
They share similar strengths and weaknesses though they make use of
different mechanisms.
Both frameworks allow injection of test doubles into the code under test
without having to modify the code under test's public API. This in our
view is the chief limitation of dependency injection/inversion of control
frameworks like Guice and Spring. These frameworks add complexity to the
code by introducing additional levels of indirection and also break down the
compartmentalization provided by private data members. While they may be
appropriate for larger components, our concern is that using them to test
every class in isolation from its dependencies would create an unmanagebly
complex code base.
Furthermore both framworks allow test doubles to be introduced for
final/sealed classes, static functions, and constructors. They are not
limited to extension or realization of open classes, abstract classes, and
interfaces the way some dependency injection frameworks are.
Both frameworks are similarly limited when it comes to injecting test
doubles for symbols that are inaccessible to the test code (e.g., private
members of the code under test). In the JMockit case it is because code
that references these symbols in order to tell the JMockit runtime to
replace them with test doubles won't compile. In the Moles case it is
because Moles only generates 'Moles Types' for publicly accessible members
of each class it stubs.
Both frameworks pass the real instance to the test double so the test double
can delegate to it.
Both frameworks support the injection of arbitrary hand-coded test
doubles. For JMockit, this is the common case. For Moles, this can be
achieved by having the generated Moles Types (stubs) delegate to a
hand-coded test double. With this both frameworks can support test doubles
that call back on the code under test. This is important for testing
increasingly popular asynchronous APIs.
Both frameworks are similarly limited when it comes to scoping the injection
of test doubles. In both cases injection can be modified between calls to
the class under test, but inside the call all transitive calls to the real
dependency will be redirected to the test double. While the programmer has
some ability to restrict when the test double is invoked by counting the
number of times its called or by comparing the arguments that are passed to
it, this approach is sensitive to minor changes in the class under test
(e.g., another call to the dependency is introduced which breaks the count).
While both frameworks allow replacement of core classes / classes from the
class library, because of the scoping problem this can be dangerous. Core
classes are more likely to be used by many of a class's dependencies. If
the dependencies are not fully mocked and they share the same test double,
care has to be taken by the programmer to ensure that the test double is
compatible with all dependencies.
Neither framework can replace an unknown implementation with a test
double. That is, calls to any implementation of an interface or abstract
class cannot be redirected to the same test double. Instead test doubles
can only replace specifically designated classes.
Similarly, both frameworks require the programmer to specify the exact class
in the inheritance hierarchy that declares the symbol to be replaced with a
test double. If a test double for a derived class replaces a symbol
declared in a base class, JMockit requires the programmer to associate the
test double with the base class, not the derived class. In addition to
being inconvenient, it's somewhat brittle since it's sensitive to movement
of symbols across the inheritance hierarchy that is common in refactoring.
Moles has a similar limitation in that the Moles Types (stubs) it generates
only have symbols that match the public symbols declared in each real
class. Moles Types do not have injection points for inherited members.
Finally, both frameworks had poor support for testing multithreaded code.
Moles instrumented test cases actually crashed Visual Studio when executed
by multiple threads. While JMockit performed more reliably, it was not
able to associate test doubles with specific threads. Instead all threads
shared the same test double instance. Also, while this is a property of
the TestNG Java unit testing framework and not JMockit, failed assertions in
background threads were not associated with the test case that spawned them
nor were they considered failures by the unit test framework.
Differences
Other than Moles' poorer performance under multithreaded conditions, we can
summarize their differences as: Moles is harder than JMockit to learn but
once learnt is more natural to use.
While we got the basics of JMockit working in less than a day, thanks in
part to copious online and Open Source documentation, it took a full week to
get Moles working. The documentation for Moles is unfortunately sorely
lacking. While the Moles maintainers produced multiple guides to using
Moles, they all referenced the same examples which in turn had the same
gaps. As a result we put extra effort into our Pex
& Moles: The Basics guide because we know of no other resource
that shows you how to setup Moles without omitting important steps like
configuring Moles to stub out a referenced assembly, importing the namespace
for the generated stubs into the test case, and building the project before
trying to reference any generated stubs.
Once the basics were learnt, however, Moles pulled ahead of JMockit. Moles
did not require any strange language tricks to introduce test doubles.
Moles test cases would look like typical C# code to a C# programmer
unfamiliar with Moles. JMockit on the other hand made use of
constructions that, while they compiled, would look very strange to a Java
programmer unfamiliar with JMockit.
To illustrate our point, consider this example from the in-depth
evaluations. This is how one could replace a final/sealed dependency with
a test double using Moles:
[TestClass]
public class ClassTest02
{
[TestMethod]
[HostType("Moles")]
public void test()
{
MDependency02.AllInstances.generate = _ => 123;
Class02 clazz = new Class02();
Assert.AreEqual(2 * 123, clazz.generate());
}
}
Of course you have to be familiar with lambda expressions, but at least
these are part of the C# language. Compare the Moles test case to the
corresponding JMockit test case:
public class ClassTestAlternative {
@Test
public void testIt() {
new MockUp<Dependency>() {
@Mock
public int generate() {
return 123;
}
};
Class clazz = new Class();
assert 123 * 2 == clazz.generate();
}
}
Java programmers rarely instantiate objects yet assign them to nothing.
JMockit unfortunately makes repeated use of this idiom. Here is another
example pulled from their online
documentation:
@Test
public void doSomethingHandlesSomeCheckedException() throws Exception
{
new Expectations() {
DependencyAbc abc;
{
new DependencyAbc();
abc.intReturningMethod(); result = 3;
abc.stringReturningMethod();
returns("str1", "str2");
result = new SomeCheckedException();
}
};
new UnitUnderTest().doSomething();
}
It doesn't look like valid Java but it is. A programmer joining an existing
project that has already setup Moles should be able to get the hang of Moles
after seeing a few examples. A programmer joining an existing JMockit
project, however, would probably have to reread the online documentation
several times before figuring out how to work with it.
Moles accomplishes much with a few constructions. JMockit accomplishes
much with a large collection of ad hoc constructions. A little bit of
JMockit knowledge does not generalize as well as a little bit of Moles
knowledge.
Yes, these observations are subjective.
Conclusion
Once familiar with test double injection frameworks like Moles and JMockit,
it's hard to imaging unit testing code of any complexity without one. Yet
these frameworks such as these are relatively new, having been introduced
more a decade after the JUnit
was introduced in 1997.
Without frameworks such as these or support in the language itself,
programmers had to resort to passing in test doubles in manually or via
dependency injection frameworks like Guice and Spring. It was unfortunate
that before these frameworks were introduced core principles of simplicity
and compartmentalization/data hiding had to be violated in order to make
code unit testable.
Perhaps now that new
programming languages are incorporating unit testing into the language
itself, we can next look forward to programming language support for
injection of test doubles.