Testing HTTP Client Code with MockWebServer
Posted on January 15, 2025 by Scott Leberknight
When testing HTTP client code, it can be challenging to verify your application's behavior. For example, if you have an HTTP client that makes calls to some third-party API, or even to another service that you control, you want to make sure that you are sending the correct requests and handling the responses properly. There are various libraries available to help, and many times the library or framework you're using provides some kind of test support.
For example, I've used Dropwizard to create REST-based web services for a number of years. Dropwizard uses Jersey, which is the reference implementation of Jakarta RESTful Web Services (formerly known as JAX-RS). Dropwizard provides a way to test HTTP client implementations by creating a resource within your test that acts as a "test double" of the real server you are trying to simulate. When the test executes, a real HTTP server is started that can respond to real HTTP requests. No mocking, which is important since mocks can't easily simulate all the various things that can happen with HTTP requests.
Suppose you have an HTTP client that uses Jersey Client to call a "Math API". For now, you only care about adding two numbers, so your client looks like:
public class MathApiClient {
private final Client client;
private final URI baseUri;
public MathApiClient(Client client, URI baseUri) {
this.client = client;
this.baseUri = baseUri;
}
public int add(int a, int b) {
var response = client.target(baseUri)
.path("/math/add/{a}/{b}")
.resolveTemplate("a", a)
.resolveTemplate("b", b)
.request()
.get();
return response.readEntity(Integer.class);
}
}
You want to design the client for easy testing, so the constructor accepts a Jersey Client and a URI, which lets you easily change the target server location. That's important, since you need to be able to provide the URI of the test server.
Here's an example of a Math API test class using Dropwizard's integration testing support:
@ExtendWith(DropwizardExtensionsSupport.class)
class DropwizardMathApiClientTest {
@Path("/math")
public static class MathStubResource {
@GET
@Path("/add/{a}/{b}")
@Produces(MediaType.TEXT_PLAIN)
public Response add(@PathParam("a") int a, @PathParam("b") int b) {
var answer = a + b;
return Response.ok(answer).build();
}
}
private static final DropwizardClientExtension CLIENT_EXTENSION =
new DropwizardClientExtension(new MathStubResource());
private MathApiClient mathClient;
private Client client;
@BeforeEach
void setUp() {
client = ClientBuilder.newBuilder()
.connectTimeout(500, TimeUnit.MILLISECONDS)
.readTimeout(500, TimeUnit.MILLISECONDS)
.build();
var baseUri = CLIENT_EXTENSION.baseUri();
mathClient = new MathApiClient(client, baseUri);
}
@AfterEach
void tearDown() {
client.close();
}
@Test
void shouldAdd() {
assertThat(mathClient.add(40, 2)).isEqualTo(42);
}
}
In this code, it's the DropwizardClientExtension that provides all the real HTTP server functionality. You provide it the stub resource (a new MathStubResource instance) and it takes care of starting a real application that responds to HTTP requests and responds as you defined in the stub resource. Then you write tests that use the MathApiClient, make assertions as you normally would, and so on.
This works great, but there are some downsides. First, there is no way to (easily) verify the HTTP requests that the HTTP client made. The client makes the HTTP request and handles the response, but unless it provides some way to access the requests it has made, there's not really any way to verify this. You can add code into the stub resource to capture the requests, and provide a way for test code to access them, but that adds complexity to the stub resource.
Second, while testing the "happy path" is straightforward, things quickly become more difficult if you want to test errors, invalid input, and other "not happy path" scenarios. For example, let's say you want to test how your client responds when it receives an error response such as a 400 Bad Request or 500 Internal Server Error. How can you do this? One way is "magic input" where the server responds with a 400 when you provide one set of input (e.g., whenever a is 84) and a 500 when you provide a different input (e.g., whenever a is 142). Depending on the number of error cases you want to test, the stub resource code can quickly get complicated with conditionals. Another way is to use some kind of "flag" field inside the test stub resource class, where each test can "record" the response it wants. But this starts to become a "mini-framework" as you need more and more features.
Something else you can do is to create separate tests with different stub resources for different scenarios. But again, this can get out of control quickly if your HTTP client has a lot of methods and you want to test each one thoroughly.
Despite these shortcomings, you can still write good HTTP tests using what Dropwizard (and other similar libraries) provides. I've used the Dropwizard test support for the vast majority of HTTP client testing over the past few years. But I've recently come across the excellent MockWebServer from OkHttp. Basically, it is like a combination of a real HTTP server to test against and a mocking library such as Mockito.
To test HTTP clients using MockWebServer, you:
- Record the responses you want to receive
- Run your HTTP client code
- Make assertions about the result from the client (if any)
- Verify the client made the expected requests
This is very similar to using mocking like in Mockito, except that MockWebServer lets you test against the full HTTP/HTTPS request/response lifecycle in a realistic manner. So, rewriting the above test to use MockWebServer looks like:
class OkHttpMathApiClientTest {
private MathApiClient mathClient;
private Client client;
private MockWebServer server;
@BeforeEach
void setUp() throws URISyntaxException {
client = ClientBuilder.newBuilder()
.connectTimeout(500, TimeUnit.MILLISECONDS)
.readTimeout(500, TimeUnit.MILLISECONDS)
.build();
server = new MockWebServer();
var baseUri = server.url("/").uri();
mathClient = new MathApiClient(client, baseUri);
}
@AfterEach
void tearDown() throws IOException {
client.close();
server.close();
}
@Test
void shouldAdd() throws InterruptedException {
server.enqueue(new MockResponse()
.setResponseCode(200)
.setHeader(HttpHeaders.CONTENT_TYPE, "text/plain")
.setBody("42"));
assertThat(mathClient.add(40, 2)).isEqualTo(42);
var recordedRequest = server.takeRequest(1, TimeUnit.SECONDS);
assertThat(recordedRequest).isNotNull();
assertAll(
() -> assertThat(recordedRequest.getMethod()).isEqualTo("GET"),
() -> assertThat(recordedRequest.getPath()).isEqualTo("/math/add/40/2"),
() -> assertThat(recordedRequest.getBodySize()).isZero()
);
}
}
In this test, we first record the response (or responses) we want to receive by calling enqueue with a MockResponse. Don't let the "Mock" in the name fool you, though, since this just tells MockWebServer the response you want. It will take care of returning a real HTTP response from a real HTTP server. The next line in the test is the same as in the Dropwizard example above, where we call the HTTP client and assert the result. But after that, MockWebServer lets you get the requests that the client code made using takeRequest, so you can verify that it sent exactly what it should have, with the expected path, query parameters, headers, body, etc.
One advantage of using MockWebServer is that it is really easy to record different responses and test how your client responds. For example, suppose the Math API returns a 400 response if you provide two numbers that add up to a number higher than the maximum value of a Java int, or a 500 response if there is a server error. Here are a few tests for those situations:
@Test
void shouldThrowIllegalArgumentException_ForInvalidInput() throws InterruptedException {
server.enqueue(new MockResponse()
.setResponseCode(400)
.setHeader(HttpHeaders.CONTENT_TYPE, "text/plain")
.setBody("overflow"));
assertThatIllegalArgumentException()
.isThrownBy(() -> mathClient.add(Integer.MAX_VALUE, 1))
.withMessage("Invalid arguments: overflow");
var recordedRequest = server.takeRequest(1, TimeUnit.SECONDS);
assertThat(recordedRequest).isNotNull();
assertAll(
() -> assertThat(recordedRequest.getMethod()).isEqualTo("GET"),
() -> assertThat(recordedRequest.getPath()).isEqualTo("/math/add/%d/1", Integer.MAX_VALUE)
);
}
@Test
void shouldThrowIllegalStateException_ForServerError() throws InterruptedException {
server.enqueue(new MockResponse()
.setResponseCode(500)
.setHeader(HttpHeaders.CONTENT_TYPE, "text/plain")
.setBody("Server error: can't add right now"));
assertThatIllegalStateException()
.isThrownBy(() -> mathClient.add(2, 2))
.withMessage("Unknown error: Server error: can't add right now");
var recordedRequest = server.takeRequest(1, TimeUnit.SECONDS);
assertThat(recordedRequest).isNotNull();
assertAll(
() -> assertThat(recordedRequest.getMethod()).isEqualTo("GET"),
() -> assertThat(recordedRequest.getPath()).isEqualTo("/math/add/2/2", Integer.MAX_VALUE)
);
}
Each test defines the response(s) that the MockWebServer should sent it. This makes it possible to create clean, self-contained test code that is easy to understand and change.
To make these tests pass, we should update the original implementation with some error handling code:
public int add(int a, int b) {
var response = client.target(baseUri)
.path("/math/add/{a}/{b}")
.resolveTemplate("a", a)
.resolveTemplate("b", b)
.request()
.get();
if (successful(response)) {
return response.readEntity(Integer.class);
} else if (clientError(response)) {
throw new IllegalArgumentException("Invalid arguments: " + response.readEntity(String.class));
}
throw new IllegalStateException("Unknown error: " + response.readEntity(String.class));
}
The code examples (adding two numbers) I've used are simple. In "real life" you are probably calling more complicated and expansive APIs, and need to test various success and failure scenarios. To recap, some of the advantages of using MockWebServer in your HTTP client tests are:
- You can record different responses for each test (similar to setting up mock objects, e.g., Mockito)
- You can avoid having to implement "stub" resources that are a "shadow API" of the remote API
- Avoiding complexity in "stub" resources when adding logic to provide different responses based on inputs or other signals
- You can verify the requests that were made, like how you verify method calls with mocking (e.g., Mockito)
There are other things you can do with MockWebServer, for example you can throttle responses to simulate a slow network to test timeout and retry behavior. You can also test with and without HTTPS, requiring client certificates, and customizing the supported protocols. These are all things that can be done in custom code, but it's much nicer when it comes out of the box.
To sum up, MockWebServer makes it simple to write tests for HTTP client code, allowing you to test the "happy path" and various failure scenarios, and provides support for more advanced testing situations such as when requiring client certificate authentication or simulating network slowness.