From bd5f6718abdb421b17b792efb9a05357aa668ce2 Mon Sep 17 00:00:00 2001 From: Bradlee Speice Date: Thu, 1 Jan 2015 14:40:31 -0500 Subject: [PATCH] Add a post on Jacoco coverage --- _posts/2015-01-01-android-and-jacoco.md | 236 ++++++++++++++++++++++++ 1 file changed, 236 insertions(+) create mode 100644 _posts/2015-01-01-android-and-jacoco.md diff --git a/_posts/2015-01-01-android-and-jacoco.md b/_posts/2015-01-01-android-and-jacoco.md new file mode 100644 index 0000000..e922b5c --- /dev/null +++ b/_posts/2015-01-01-android-and-jacoco.md @@ -0,0 +1,236 @@ +--- +layout: post +title: "Android & Jacoco" +modified: 2015-01-01 13:56:32 -0500 +tags: [jacoco,testing,code coverage] +image: + feature: + credit: + creditlink: +comments: +share: +--- + +Empirical Development +--------------------- + +For a while now, I've been wanting to get code coverage working with MinimalBible, +and it's finally at a point that I'm mostly satisfied with. I certainly wouldn't argue that it's in a "good" state, +but it's enough for now. So, I wanted to write a post outlining how this was all set up since I haven't been able to +find many people using this style. Plus, it took a very long time to set up, so I hope I can save someone else some +pain in the future! + +Before I get too much farther though, let me explain the concept of code coverage. +Code coverage is intended to answer the question of "What code have I written tests for?" +This allows you to quickly spot code that is untested by your existing suite, and let you know where is best to focus +your time. Coverage statistics are generated line-by-line and branch-by-branch - +if your test doesn't execute a specific branch of code during the test, +your coverage system will let you know that not everything is "covered" by a test. +All said, you get an easy way to see what potential problems exist and proactively solve them. + +Test Setup +---------- + +So, onward to how I set up testing in MinimalBible. +First things first, I have to say that I really changed how I did testing in Android. Instead of using the existing +Android build tools, I'm using pure-Java testing. Nothing Android. You can find the setup process +[here](http://blog.blundell-apps.com/how-to-run-robolectric-junit-tests-in-android-studio/) +(massive shoutouts to Blundell Apps, this has been incredibly useful) but I'll give a quick overview of how this is set up. + +The basic principle is that the testing support in Android is so broken as to be useless. I won't go into all my gripes, +but suffice to say I have had issues with the JUnit API (Android uses something below 3.8), code coverage, and UI tests +with [Espresso](https://code.google.com/p/android-test-kit/). So instead of trying to use the native Android tools for +testing, we create a new project based on the source code of our existing project. This allows us to run everything in a +native Java environment, without worrying about any of the platform considerations. If you're interested in what that +setup would look like, you can find my version +[here](https://github.com/MinimalBible/MinimalBible/tree/cb8ea71f620ac0a25c628920bfe5fa82b1d6cebe). + +There are two projects: `app` and `app-test`. +App is where the actual source code resides, and App-test is where the test code resides. +However, the test code includes the original code as a dependency so that we can still write tests against it. +I've included the important bits of `app-test/build.gradle` below: + +{% highlight groovy %} +apply plugin: 'java' + +def androidModule = project(':app') +dependencies { + compile androidModule + testCompile androidModule.android.applicationVariants.toList().first().javaCompile.classpath // 1 + testCompile androidModule.android.applicationVariants.toList().first().javaCompile.outputs.files // 2 + testCompile files(androidModule.plugins.findPlugin("com.android.application").getBootClasspath()) // 3 + testCompile 'junit:junit:4.+' // 4 + testCompile 'org.robolectric:robolectric:2.2' +} +{% endhighlight %} + +And a quick breakdown of what's going on: + +1. Make sure to include all dependencies of the actual project in the test project +2. Include all code for the actual project in the test project. +3. Include all dependencies of the `android` Gradle plugin in the test project. Not sure why this is here. +4. Finally, include all the other libraries and things we actually need for testing. + +So at this point we have a test-ready project setup. We're not quite ready for code coverage yet, but we're about to change that. + +Next steps: Enabling Code Coverage +---------------------------------- + +So, we have a test project set up to run the tests we put inside it. The next step is setting up Jacoco to report on +what code is being tested during the test suite. I'm going to present the solution first to make it easy on anyone +reading this - if you want a full explanation of what's going on check it out [here](#jacoco-full-explanation). + +In order to enable Jacoco testing we need to change `app-test/build.gradle` to look like +[this](https://github.com/MinimalBible/MinimalBible/blob/c29b043d2313b3653a9671c36921f6ce8e4b9348/app-test/build.gradle): + +{% highlight groovy %} +apply plugin: 'java' +apply plugin: 'jacoco' + +def androidModule = project(':app') +def firstVariant = androidModule.android.applicationVariants.toList().first() + +def testIncludes = [ + '**/*Test.class' +] +def jacocoExcludes = [ + 'android/**', + 'org/bspeice/minimalbible/R*', + '**/*$$*' +] +{% endhighlight %} + +First steps first, this is the easy part. We're defining what files we want to include in testing, +alongside the files we want to exclude from the final report. For example, we exclude the "R" file, since +it's all generated code. In addition, anything containing a "$$" is generated by Dagger/Butterknife, so we ignore +those too. + +If you want to adapt the solution I'm outlining here to your own project, these should be the only sections you +need to edit. + +The next section is a whole lot more complicated: + +{% highlight groovy %} +dependencies { + compile androidModule + testCompile 'junit:junit:4.+' + testCompile 'org.robolectric:robolectric:+' + testCompile 'org.mockito:mockito-core:+' + testCompile 'com.jayway.awaitility:awaitility:+' + testCompile 'org.jetbrains.spek:spek:+' + testCompile firstVariant.javaCompile.classpath + testCompile firstVariant.javaCompile.outputs.files + testCompile files(androidModule.plugins.findPlugin("com.android.application").getBootClasspath()) +} +def buildExcludeTree(path, excludes) { + fileTree(path).exclude(excludes) +} // 1 +jacocoTestReport { + doFirst { + // First we build a list of our base directories + def fileList = new ArrayList() + def outputsList = firstVariant.javaCompile.outputs.files + outputsList.each { fileList.add(it.absolutePath.toString()) } + + // And build a fileTree from those + def outputTree = fileList.inject { tree1, tree2 -> + buildExcludeTree(tree1, jacocoExcludes) + + buildExcludeTree(tree2, jacocoExcludes) + } + + // And finally tell Jacoco to only include said files in the report + classDirectories = outputTree + } // 3 +} +tasks.withType(Test) { + scanForTestClasses = false + includes = testIncludes +} // 2 +{% endhighlight %} + +1. Define a quick function that will exclude a *list* of file paths from a given path. +2. Set up Gradle to run the tests we defined earlier +3. Set up Jacoco to exclude the paths we specified earlier from the report. This step is so complicated +because we have to get the outputs paths from the Android project, and exclude our paths from each of those. + +Wrapping Up +----------- + +So given the above build.gradle file, we now have a project capable of testing your actual application code and +producing coverage statistics on it. While I haven't outlined it above, because the testing code is separate from the +Android project, you're free to write your tests in JUnit, [Spock](https://code.google.com/p/spock/), +or [Spek](http://jetbrains.github.io/spek/). I'm going to be using Spek moving forward. + +We can include tests using the `testIncludes` list, and make sure that classes don't get reported using the +`jacocoExcludes` list. All said, that's what we were out for in the first place, so I'll call it a success. + +If you want to take this solution further, the next step would be to add Robolectric tests into the suite, +but [I've been having issues](https://github.com/robolectric/robolectric/issues/1385) with that too. + +Appendix: Jacoco Full Explanation +============================================================= + +In order to fully understand what's going on with how Jacoco excludes things from reporting, we have to step back and +take a visit to Gradle first to understand your build lifecycle. + +Gradle: Configure, Run +---------------------- + +Gradle is an incredibly powerful tool, but it is massively confusing if you don't already know what you're doing. +In my opinion, the documentation is still missing many examples that would be super-helpful, +and is generally dense to try and get through. + +That aside, to understand what's going on, you must understand that the Gradle build process happens in two phases: +**Configuration**, and then **Build**. + +For our purposes, you don't need to understand what each one does, but understanding the semantics is crucial. +Because there's a two phase build, we can't write a `build.gradle` that tries to exclude files from Jacoco like this: + +{% highlight groovy %} +jacocoTestReport { + // First we build a list of our base directories + def fileList = new ArrayList() + def outputsList = firstVariant.javaCompile.outputs.files + outputsList.each { fileList.add(it.absolutePath.toString()) } + + // And build a fileTree from those + def outputTree = fileList.inject { tree1, tree2 -> + buildExcludeTree(tree1, jacocoExcludes) + + buildExcludeTree(tree2, jacocoExcludes) + } + + // And finally tell Jacoco to only include said files in the report + classDirectories = outputTree +} +{% endhighlight %} + +Did you notice the difference? In the second example, we're missing the `doFirst` closure. +Keep this in mind during the next sections. + +Under the hood, Jacoco reports on all classes specified in the `classDirectories` variable. So, all we need to do is +make sure that we include all the classes to report on in `classDirectories`, and exclude the ones we don't want to see. + +However, if you skip the `doFirst` closure, you'll be in deep trouble. Without that closure, Groovy will run the code +in the `jacocoTestReport` closure before testing is actually run, since it will be in the **configuration** build phase. + +What the code actually does is exclude everything in `jacocoExcludes` from the global class path. This isn't a great +solution, but I'm not sure how else to do it. + +The problem comes when you exclude files like the `android` package that we don't want to report on, but are needed for +testing. When things in `android` aren't loaded during the tests, you'll get lots of nasty `NoClassDefFoundException` +exceptions, because Java can't find the code it needs for testing. + +The solution? We need to modify the class path **only right before Jacoco runs**. This way, the tests are allowed to +run successfully, and Jacoco never knows about those classes. + +To do this, we need to move the class path configuration into the **build** phase instead of **configure**. +The way to do that? You guessed it, surround the code in a `doFirst` closure. + +So the end result is that we can exclude specific classes from reporting without interfering in the test setup process. +It took me forever to figure out how exactly to implement this, but I hope this can help someone avoid the same issues +in the future. + +Side note: Much of the above solution was adapting the procedures outlined +[here](https://issues.gradle.org/browse/GRADLE-2955) to the world of Android. Thanks to everyone for putting in the +effort to make it easier for me! +