25 October 2020 / TESTING

Unit Testing Asynchronous Code in Swift

If you are practicing unit testing, sooner or later you’ll run across testing async code. Writing such tests can be a challenge for the next reasons:

False positives and negatives. Async code ends after the unit test does, causing misleading test results.
Flakiness. Async code executes with different speed on different machines, bringing a subtle time dependency into a test suite. A common symptom of flakiness is that on your development machine all tests are passing, however, when run on a CI/CD slave, a couple of random tests are failing.
Untrustworthy. Because of the flakiness, we stop trusting our test suite. If several random tests from the test suite fail, we do not tell ourselves that these tests have detected a bug. Instead, we re-run them individually. And if they pass, we pretend that everything is all right with the test suite.
Error-prone. We all know how hard it is to get async code right. Same goes for testing it. If you run across a multithreading issue in your tests, you know that you are on for a wild ride.

The solution is to use next four patterns which allow us to test async code in a safe and reliable way:

Mocking
Testing before & after
Expectations
Busy assertion

Throughout this article, we’ll study these four patterns: what they are, when to use, and what you can get from them.

System Under Test

When you test something, you refer to it as the system under test (SUT).

MusicService will be our system under test. It is a networking service which searches music via iTunes API:

struct MusicService {
    
    func search(_ term: String, completion: @escaping (Result<[Track], Error>) -> Void) {
        URLSession.shared.dataTask(with: .search(term: term)) { data, response, error in
            DispatchQueue.main.async {
                completion(self.parse(data: data, error: error))
            }
        }.resume()
    }
    
    func parse(data: Data?, error: Error?) -> Result<[Track], Error> {
        if let data = data {
            return Result { try JSONDecoder().decode(SearchMediaResponse.self, from: data).results }
        } else {
            return .failure(error ?? URLError(.badServerResponse))
        }
    }
}

It creates a request via:

extension URLRequest {
    static func search(term: String) -> URLRequest {
        var components = URLComponents(string: "https://itunes.apple.com/search")
        components?.queryItems = [
            .init(name: "media", value: "music"),
            .init(name: "entity", value: "song"),
            .init(name: "term", value: "\(term)")
        ]
        
        return URLRequest(url: components!.url!)
    }
}

MusicService uses URLSession to fire HTTP requests, and returns an array of Tracks via the completion callback:

struct SearchMediaResponse: Codable {
    let results: [Track]
}

struct Track: Codable, Equatable {
    let trackName: String?
    let artistName: String?
}

When building iOS apps, it is typical to design networking layer using services similar to MusicService. Therefore, testing it is a common and relevant task.

After discussing the system under test, let’s move to the first pattern.

Mocking in Swift

The idea of the Mocking pattern is to use special objects called mocks.

Mock object mimics real objects for testing.

To use mock objects, we must:

Extract asynchronous work into a new type.
Delegate asynchronous work to the new type.
Replace real dependency with a mock during testing.

Passing dependencies to an object is called dependency injection. I’ve written an in-depth article about Dependency Injection in Swift, which is important to understand when practicing mocking.

Let’s highlight the parts of MusicService which perform asynchronous work:

struct MusicService {
    func search(_ term: String, completion: @escaping (Result<[Track], Error>) -> Void) {
        URLSession.shared.dataTask(with: .search(term: term)) { data, response, error in
            DispatchQueue.main.async {
                ...
            }
        }.resume()
    }
    ...
}

The responsibility of the above code is to execute HTTP requests. We extract it into a new type called HTTPClient, and use it from MusicService:

protocol HTTPClient {
    func execute(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
}

struct MusicService {
    let httpClient: HTTPClient
    
    func search(_ term: String, completion: @escaping (Result<[Track], Error>) -> Void) {
        httpClient.execute(request: .search(term: term)) { result in
            completion(self.parse(result))
        }
    }
    
    private func parse(_ result: Result<Data, Error>) -> Result<[Track], Error> { ... }
}

HTTPClient will have two implementations. The production one contains all the code from MusicService which was responsible for firing HTTP requests:

class RealHTTPClient: HTTPClient {
    func execute(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void) {
        URLSession.shared.dataTask(with: request) { data, response, error in
            DispatchQueue.main.async {
                if let data = data {
                    completion(.success(data))
                } else {
                    completion(.failure(error!))
                }
            }
        }.resume()
    }
}

The test implementation remembers the last parameter passed to execute(), and allows us to pass an arbitrary result to the completion callback, mimicking network response.

class MockHTTPClient: HTTPClient {
    var inputRequest: URLRequest?
    var executeCalled = false
    var result: Result<Data, Error>?
    
    func execute(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void) {
        executeCalled = true
        inputRequest = request
        result.map(completion)
    }
}

Now we are ready to test MusicService using a mock object. In the next test we verify that MusicService fires a correct HTTP request:

func testSearch() {
    // 1.
    let httpClient = MockHTTPClient()
    let sut = MusicService(httpClient: httpClient)
    
    // 2.
    sut.search("A") { _ in }
    
    // 3.
    XCTAssertTrue(httpClient.executeCalled)
    // 4.
    XCTAssertEqual(httpClient.inputRequest, .search(term: "A"))
}

Here is what we are doing:

Initialize the system under test with a mock implementation of HTTPClient.
Run the method search(), passing it an arbitrary query.
Verify that the method execute() has been invoked on the mock.
Verify that the correct HTTP request has been passed to the mock.

Next, we verify how MusicService handles API response. In order to do this, we mimic a successful response using MockHTTPClient:

func testSearchWithSuccessResponse() throws {
    // 1.
    let expectedTracks = [Track(trackName: "A", artistName: "B")]
    let response = try JSONEncoder().encode(SearchMediaResponse(results: expectedTracks))

    // 2.
    let httpClient = MockHTTPClient()
    httpClient.result = .success(response)

    let sut = MusicService(httpClient: httpClient)

    var result: Result<[Track], Error>?

    // 3.
    sut.search("A") { result = $0 }

    // 4.
    XCTAssertEqual(result?.value, expectedTracks)
}

Prepare test data.
Initialize a mock object, and pass it predefined response data. The mock will return that data in the execute() callback.
The method search() is synchronous since we mocked HTTPClient. Therefore, the completion callback is invoked instantly.
Verify that the correct result has been received.

We can also verify how MusicService handles failed response:

func testSearchWithFailureResponse() throws {
    // 1.
    let httpClient = MockHTTPClient()
    httpClient.result = .failure(DummyError())

    let sut = MusicService(httpClient: httpClient)

    var result: Result<[Track], Error>?

    // 2.
    sut.search("A") { result = $0 }

    // 4.
    XCTAssertTrue(result?.error is DummyError)
}

struct DummyError: Error {}

Provide an error response to the mock.
Same as before, the method search() is synchronous, and the completion callback is invoked immediately.
Verify that search() passed the correct error to the callback.

Testing Before & After

The idea of the Before & After pattern is to break verification of an asynchronous code into two tests. The first test verifies that the system under test is in correct state before async code has started. The second test validates the state of the SUT as if async code has already finished.

Let’s return to our initial MusicService implementation. That is, the one without HTTPClient:

struct MusicService {
 
    func search(_ term: String, completion: @escaping (Result<[Track], Error>) -> Void) {
        URLSession.shared.dataTask(with: .search(term: term)) { ... }.resume()
    }
 
    private func parse(data: Data?, error: Error?) -> Result<[Track], Error> { ... }
}

In the test before, validate that MusicService fires correct HTTP request:

func testSearchBefore() {
    let sut = MusicService()
    
    // 1.
    sut.search("A") { _ in }
    
    // 2.
    let lastRequest = URLSession.shared.tasks.last?.currentRequest
    XCTAssertEqual(lastRequest?.url, URLRequest.search(term: "A").url)
}

Unlike the example with Mocking, the search() method fires a real HTTP request, and invokes the completion callback as soon as it receives a response. That it, asynchronously. For this reason we are we do not handle it.
To validate that MusicService has fired a correct HTTP request, we take the last URLRequest from URLSession, and compare its url with the expected one.

An attentive reader might have noticed that URLSession does not expose a list of its tasks synchronously. Indeed, we’ve been using a helper method for this:

extension URLSession {
    var tasks: [URLSessionTask] {
        var tasks: [URLSessionTask] = []
        let group = DispatchGroup()
        group.enter()
        getAllTasks {
            tasks = $0
            group.leave()
        }
        group.wait()
        return tasks
    }
}

In the test after, pretend that MusicService has received a response, and test the method parse(). The tricky thing is that this method is private. What we are going to do is trade encapsulation for testability, and make parse() internal:

struct MusicService {
    ...
    func parse(data: Data?, error: Error?) -> Result<[Track], Error> { ... }
}

Now nothing prevents us from testing the method parse() directly:

func testSearchAfterWithSuccess() throws {
    let expectedTracks = [Track(trackName: "A", artistName: "B")]
    let response = try JSONEncoder().encode(SearchMediaResponse(results: expectedTracks))
    
    let sut = MusicService()
    
    let result = sut.parse(data: response, error: nil)
    
    XCTAssertEqual(result.value, expectedTracks)
}

We can also validate the negative case:

func testSearchAfterWithFailure() {
    let sut = MusicServiceWithoutDependency()
    
    let result = sut.parse(data: nil, error: DummyError())
    
    XCTAssertTrue(result.error is DummyError)
}

struct DummyError: Error {}

Expectations

Up to this point, we were able to verify asynchronous code with synchronous tests, avoiding the complexity of writing multithreading code in tests altogether.

What if we cannot use Mocking since it is too risky or too long to refactor an SUT? What if we also cannot use Before & After since there are no noticeable changes in the SUT that we can verify in the test before? In this case, asynchronous code in tests is unavoidable. The next two patterns lend themselves to writing multithreading tests effectively.

The Expectations pattern is based on the usage of XCTestExpectation.

Expectation is an expected outcome in an asynchronous test.

The pattern can be summarized into four steps:

Create an instance of XCTestExpectation.
Fulfill the expectation when async operation has finished.
Wait for the expectation to be fulfilled.
Assert the expected result.

We are going to test the version of MusicService with HTTPClient. Let’s recall the implementation:

protocol HTTPClient {
    func execute(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
}
 
struct MusicService {
    let httpClient: HTTPClient
    
    func search(_ term: String, completion: @escaping (Result<[Track], Error>) -> Void) {
        httpClient.execute(request: .search(term: term)) { result in
            completion(self.parse(result))
        }
    }
    
    private func parse(_ result: Result<Data, Error>) -> Result<[Track], Error> { ... }
}

The primary difference is that HTTPClient won’t be mocked. As you already might have guessed, we are going to write an integration test:

func testSearch() {
    // 1.
    let didReceiveResponse = expectation(description: #function)
    
    // 2.
    let sut = MusicService(httpClient: RealHTTPClient())
    
    // 3.
    var result: Result<[Track], Error>?
    
    // 4.
    sut.search("ANYTHING") {
        result = $0
        didReceiveResponse.fulfill()
    }
    
    // 5.
    wait(for: [didReceiveResponse], timeout: 5)
    
    // 6.
    XCTAssertNotNil(result?.value)
}

Create an instance of XCTestExpectation.
Initialize SUT with RealHTTPClient. That is, the one used in production.
Declare a variable which will hold a result of the search() method once it completes.
In the search() callback, fulfill the expectation, and store the received result into the variable.
Wait for up to 5 seconds for search() to fulfill.
Assert that a successful request has been received.

More Use Cases for Expectations

XCTestExpectation is a versatile tool and can be applied in a number of scenarios.

Inverted Expectations allows us to verify that something did not happen:

func testInvertedExpectation() {
    // 1.
    let exp = expectation(description: #function)
    exp.isInverted = true
    
    // 2.
    sut.maybeComplete {
        exp.fulfill()
    }
    
    // 3.
    wait(for: [exp], timeout: 0.1)
}

Create an inverted expectation. The test will fail if the inverted expectation is fulfilled.
Call a method that may conditionally invoke a callback.
Verify that the callback is not invoked.

Notification Expectation is fulfilled when an expected Notification is received:

func testExpectationForNotification() {
    let exp = XCTNSNotificationExpectation(name: .init("MyNotification"), object: nil)
    
    ...
    sut.postNotification()
    
    wait(for: [exp], timeout: 1)
}

Assert for over fulfillment triggers an assertion if the number of calls to fulfill() exceeds expectedFulfillmentCount:

func testExpectationFulfillmentCount() {
    let exp = expectation(description: #function)
    exp.expectedFulfillmentCount = 3
    exp.assertForOverFulfill = true
    
    ...
    sut.doSomethingThreeTimes()
    
    wait(for: [exp], timeout: 1)
}

There are even more use cases which we won’t cover in details:

Busy Assertion

The Busy Assertion pattern makes expectations on values which are updated asynchronously. The idea is to create an infinite loop which re-evaluates a condition until it holds or the timeout is reached.

The method expectToEventually() creates a busy assertion loop:

extension XCTest {
    
    func expectToEventually(
        _ isFulfilled: @autoclosure () -> Bool,
        timeout: TimeInterval,
        message: String = "",
        file: StaticString = #filePath,
        line: UInt = #line
    ) {
        func wait() { RunLoop.current.run(until: Date(timeIntervalSinceNow: 0.01)) }
        
        let timeout = Date(timeIntervalSinceNow: timeout)
        func isTimeout() -> Bool { Date() >= timeout }

        repeat {
            if isFulfilled() { return }
            wait()
        } while !isTimeout()
        
        XCTFail(message, file: file, line: line)
    }
}

The important highlight is that wait() is non-blocking. It releases RunLoop for 0.01 seconds to serve other attached input sources. After 0.01 seconds of waiting, another iteration of the busy assertion loop spins. If the isFulfilled condition does not hold for the duration of timeout, the loop ends, and XCTFail is called to signal test failure.

Here is how we can re-write the expectations-based test using busy assertion:

func testSearchBusyAssert() {
    let sut = MusicService(httpClient: RealHTTPClient())
    
    var result: Result<[Track], Error>?
    
    sut.search("ANYTHING") { result = $0 }
    
    expectToEventually(result?.value != nil, timeout: 5)
}

Notice that the number of lines reduced from 16 to 9. The reason for this is that Busy Assertion needs only one step instead of four steps required by expectations.

Summary

Let’s summarize what we just discussed, and lay out when to use each pattern.

Mocking: use whenever possible. It has several strong benefits:

Improves design of existing code since we follow good practices of object-oriented design when extracting multithreading dependencies.
Eliminates asynchronous code in tests.
Verifies edge cases. In our example, we were able to verify how MusicService handles response error.

Before & after: use with legacy code since the pattern requires minimal or no refactoring. In such situations, Mocking can be unsafe since it may lead to bugs in production due to significant refactoring.

Busy assertion: use in the most scenarios when need to test async code directly. This pattern is simpler than expectations and usually can replace them.

Expectations: use in special cases when need to test async code directly. In some non-trivial scenarios, busy assertion cannot replace expectations. Additionally, expectations come with a number of flavours.

Source Code

Source code with all examples from this article can be found here. It is published under the “Unlicense”, which allows you to do whatever you want with it.

Unit Testing Asynchronous Code in Swift

System Under Test

Mocking in Swift

Testing Before & After

Expectations

More Use Cases for Expectations

Busy Assertion

Summary

Source Code

Further reading

Thanks for reading!

SwiftUI View Lifecycle

Working with an Internet Connection on iOS with Swift: Best Practices