Add a post on Jacoco coverage

_posts/2015-01-01-android-and-jacoco.md

@@ -0,0 +1,236 @@
+---
+layout: post
+title: "Android & Jacoco"
+modified: 2015-01-01 13:56:32 -0500
+tags: [jacoco,testing,code coverage]
+image:
+  feature: 
+  credit: 
+  creditlink: 
+comments: 
+share: 
+---
+
+Empirical Development
+---------------------
+
+For a while now, I've been wanting to get code coverage working with MinimalBible,
+and it's finally at a point that I'm mostly satisfied with. I certainly wouldn't argue that it's in a "good" state,
+but it's enough for now. So, I wanted to write a post outlining how this was all set up since I haven't been able to
+find many people using this style. Plus, it took a very long time to set up, so I hope I can save someone else some
+pain in the future!
+
+Before I get too much farther though, let me explain the concept of code coverage.
+Code coverage is intended to answer the question of "What code have I written tests for?"
+This allows you to quickly spot code that is untested by your existing suite, and let you know where is best to focus
+your time. Coverage statistics are generated line-by-line and branch-by-branch -
+if your test doesn't execute a specific branch of code during the test,
+your coverage system will let you know that not everything is "covered" by a test.
+All said, you get an easy way to see what potential problems exist and proactively solve them.
+
+Test Setup
+----------
+
+So, onward to how I set up testing in MinimalBible.
+First things first, I have to say that I really changed how I did testing in Android. Instead of using the existing
+Android build tools, I'm using pure-Java testing. Nothing Android. You can find the setup process
+[here](http://blog.blundell-apps.com/how-to-run-robolectric-junit-tests-in-android-studio/)
+(massive shoutouts to Blundell Apps, this has been incredibly useful) but I'll give a quick overview of how this is set up.
+
+The basic principle is that the testing support in Android is so broken as to be useless. I won't go into all my gripes,
+but suffice to say I have had issues with the JUnit API (Android uses something below 3.8), code coverage, and UI tests
+with [Espresso](https://code.google.com/p/android-test-kit/). So instead of trying to use the native Android tools for
+testing, we create a new project based on the source code of our existing project. This allows us to run everything in a
+native Java environment, without worrying about any of the platform considerations. If you're interested in what that
+setup would look like, you can find my version
+[here](https://github.com/MinimalBible/MinimalBible/tree/cb8ea71f620ac0a25c628920bfe5fa82b1d6cebe).
+
+There are two projects: `app` and `app-test`.
+App is where the actual source code resides, and App-test is where the test code resides.
+However, the test code includes the original code as a dependency so that we can still write tests against it.
+I've included the important bits of `app-test/build.gradle` below:
+
+{% highlight groovy %}
+apply plugin: 'java'
+
+def androidModule = project(':app')
+dependencies {
+    compile androidModule
+    testCompile androidModule.android.applicationVariants.toList().first().javaCompile.classpath // 1
+    testCompile androidModule.android.applicationVariants.toList().first().javaCompile.outputs.files // 2
+    testCompile files(androidModule.plugins.findPlugin("com.android.application").getBootClasspath()) // 3
+    testCompile 'junit:junit:4.+' // 4
+    testCompile 'org.robolectric:robolectric:2.2'
+}
+{% endhighlight %}
+
+And a quick breakdown of what's going on:
+
+1. Make sure to include all dependencies of the actual project in the test project
+2. Include all code for the actual project in the test project.
+3. Include all dependencies of the `android` Gradle plugin in the test project. Not sure why this is here.
+4. Finally, include all the other libraries and things we actually need for testing.
+
+So at this point we have a test-ready project setup. We're not quite ready for code coverage yet, but we're about to change that.
+
+Next steps: Enabling Code Coverage
+----------------------------------
+
+So, we have a test project set up to run the tests we put inside it. The next step is setting up Jacoco to report on
+what code is being tested during the test suite. I'm going to present the solution first to make it easy on anyone
+reading this - if you want a full explanation of what's going on check it out [here](#jacoco-full-explanation).
+
+In order to enable Jacoco testing we need to change `app-test/build.gradle` to look like
+[this](https://github.com/MinimalBible/MinimalBible/blob/c29b043d2313b3653a9671c36921f6ce8e4b9348/app-test/build.gradle):
+
+{% highlight groovy %}
+apply plugin: 'java'
+apply plugin: 'jacoco'
+
+def androidModule = project(':app')
+def firstVariant = androidModule.android.applicationVariants.toList().first()
+
+def testIncludes = [
+    '**/*Test.class'
+]
+def jacocoExcludes = [
+    'android/**',
+    'org/bspeice/minimalbible/R*',
+    '**/*$$*'
+]
+{% endhighlight %}
+
+First steps first, this is the easy part. We're defining what files we want to include in testing,
+alongside the files we want to exclude from the final report. For example, we exclude the "R" file, since
+it's all generated code. In addition, anything containing a "$$" is generated by Dagger/Butterknife, so we ignore
+those too.
+
+If you want to adapt the solution I'm outlining here to your own project, these should be the only sections you
+need to edit.
+
+The next section is a whole lot more complicated:
+
+{% highlight groovy %}
+dependencies {
+    compile androidModule
+    testCompile 'junit:junit:4.+'
+    testCompile 'org.robolectric:robolectric:+'
+    testCompile 'org.mockito:mockito-core:+'
+    testCompile 'com.jayway.awaitility:awaitility:+'
+    testCompile 'org.jetbrains.spek:spek:+'
+    testCompile firstVariant.javaCompile.classpath
+    testCompile firstVariant.javaCompile.outputs.files
+    testCompile files(androidModule.plugins.findPlugin("com.android.application").getBootClasspath())
+}
+def buildExcludeTree(path, excludes) {
+    fileTree(path).exclude(excludes)
+} // 1
+jacocoTestReport {
+    doFirst {
+        // First we build a list of our base directories
+        def fileList = new ArrayList<String>()
+        def outputsList = firstVariant.javaCompile.outputs.files
+        outputsList.each { fileList.add(it.absolutePath.toString()) }
+        
+        // And build a fileTree from those
+        def outputTree = fileList.inject { tree1, tree2 ->
+            buildExcludeTree(tree1, jacocoExcludes) +
+            buildExcludeTree(tree2, jacocoExcludes)
+        }
+        
+        // And finally tell Jacoco to only include said files in the report
+        classDirectories = outputTree
+    } // 3
+}
+tasks.withType(Test) {
+    scanForTestClasses = false
+    includes = testIncludes
+} // 2
+{% endhighlight %}
+
+1. Define a quick function that will exclude a *list* of file paths from a given path.
+2. Set up Gradle to run the tests we defined earlier
+3. Set up Jacoco to exclude the paths we specified earlier from the report. This step is so complicated
+because we have to get the outputs paths from the Android project, and exclude our paths from each of those.
+
+Wrapping Up
+-----------
+
+So given the above build.gradle file, we now have a project capable of testing your actual application code and
+producing coverage statistics on it. While I haven't outlined it above, because the testing code is separate from the
+Android project, you're free to write your tests in JUnit, [Spock](https://code.google.com/p/spock/),
+or [Spek](http://jetbrains.github.io/spek/). I'm going to be using Spek moving forward.
+
+We can include tests using the `testIncludes` list, and make sure that classes don't get reported using the
+`jacocoExcludes` list. All said, that's what we were out for in the first place, so I'll call it a success.
+
+If you want to take this solution further, the next step would be to add Robolectric tests into the suite,
+but [I've been having issues](https://github.com/robolectric/robolectric/issues/1385) with that too.
+
+Appendix: <a name="jacoco-full-explanation">Jacoco Full Explanation</a>
+=============================================================
+
+In order to fully understand what's going on with how Jacoco excludes things from reporting, we have to step back and
+take a visit to Gradle first to understand your build lifecycle.
+
+Gradle: Configure, Run
+----------------------
+
+Gradle is an incredibly powerful tool, but it is massively confusing if you don't already know what you're doing.
+In my opinion, the documentation is still missing many examples that would be super-helpful,
+and is generally dense to try and get through.
+
+That aside, to understand what's going on, you must understand that the Gradle build process happens in two phases:
+**Configuration**, and then **Build**.
+
+For our purposes, you don't need to understand what each one does, but understanding the semantics is crucial.
+Because there's a two phase build, we can't write a `build.gradle` that tries to exclude files from Jacoco like this:
+
+{% highlight groovy %}
+jacocoTestReport {
+    // First we build a list of our base directories
+    def fileList = new ArrayList<String>()
+    def outputsList = firstVariant.javaCompile.outputs.files
+    outputsList.each { fileList.add(it.absolutePath.toString()) }
+    
+    // And build a fileTree from those
+    def outputTree = fileList.inject { tree1, tree2 ->
+        buildExcludeTree(tree1, jacocoExcludes) +
+        buildExcludeTree(tree2, jacocoExcludes)
+    }
+    
+    // And finally tell Jacoco to only include said files in the report
+    classDirectories = outputTree
+}
+{% endhighlight %}
+
+Did you notice the difference? In the second example, we're missing the `doFirst` closure.
+Keep this in mind during the next sections.
+
+Under the hood, Jacoco reports on all classes specified in the `classDirectories` variable. So, all we need to do is
+make sure that we include all the classes to report on in `classDirectories`, and exclude the ones we don't want to see.
+
+However, if you skip the `doFirst` closure, you'll be in deep trouble. Without that closure, Groovy will run the code
+in the `jacocoTestReport` closure before testing is actually run, since it will be in the **configuration** build phase.
+
+What the code actually does is exclude everything in `jacocoExcludes` from the global class path. This isn't a great
+solution, but I'm not sure how else to do it.
+
+The problem comes when you exclude files like the `android` package that we don't want to report on, but are needed for
+testing. When things in `android` aren't loaded during the tests, you'll get lots of nasty `NoClassDefFoundException`
+exceptions, because Java can't find the code it needs for testing.
+
+The solution? We need to modify the class path **only right before Jacoco runs**. This way, the tests are allowed to
+run successfully, and Jacoco never knows about those classes.
+
+To do this, we need to move the class path configuration into the **build** phase instead of **configure**.
+The way to do that? You guessed it, surround the code in a `doFirst` closure.
+
+So the end result is that we can exclude specific classes from reporting without interfering in the test setup process.
+It took me forever to figure out how exactly to implement this, but I hope this can help someone avoid the same issues
+in the future.
+
+Side note: Much of the above solution was adapting the procedures outlined
+[here](https://issues.gradle.org/browse/GRADLE-2955) to the world of Android. Thanks to everyone for putting in the
+effort to make it easier for me!
+