A Year of Functional Programming

It's been a year since I first came across the concept of functional programming. To say it's changed my life, is an unjust understatement. This is a reflection on that journey.

Warning: I use the term FP quite loosely throughout this article.

Where I Was

I've been coding since the age of 8, so 26 years now. I started with BASIC & different types of assembly then moved on to C, C++, PHP and Perl (those were different times, maaaan ☮), Java, JavaScript, Ruby. That's the brief gist of it. Basically: a lot of time on the clock and absolutely no exposure to FP concepts or languages. I thought functional programming just meant recursive functions (ooooo big deal right?). I really came in blind.

How It Started: Scala

Last year, I wanted the speed and static checking (note: not “types”) of Java with the conciseness and flexibility of Ruby. I came across Scala, skimmed a little and was impressed. I bought a copy of Scala For The Impatient and just ate it for breakfast. I read the entire thing in 2 or 3 days, jotted down everything useful and then just started coding. It was awesome! At first I was just coding the same way I would Java with less than half the lines of code. It is a very efficient Java.

Exposure: Haskell

Lurking around the Scala community, I came across a joke. Someone said “Scala is a gateway drug to Haskell.” I found that amusing, although not for the reasons that the author intended. Haskell? Isn't that some toy university language? An experiment or something. Is it even still alive? Scala's awesome and so powerful, why would that lead to Haskell? How.... intriguing. Inexplicably it piqued my interest and really stuck with me. Later I decided to look it up and yes, it sure was alive and very active. I was shocked to discover that it compiles to machine code (binary) and is comparable in speed to C/C++. What?! It seems idiotic now but a year ago I thought it was some interpreted equation solver. I'm not alone in that ignorance, sadly; talking about it to some mates over lunch last year and a friend incredulously burst out laughing, “Haskell?” as if I was trying to tell him my dishwasher had an impressive static typing system. It saddens me to realise how in the dark I was, and how many people still are. Haskell is pretty frikken awesome! True to the gateway drug prophecy, I do now look to Haskell as an improvement to using Scala. But let's get back on track.

Exposure: Scalaz

I also started seeing contentious discussion of some library called Scalaz. Curious, I had a look at the code to see what people were on about, and didn't understand it at all. I'd see classes like Blah[F[_], A, B], methods with confusing names that take params like G[_], A → C, F[X → G[Y]], implementations like F.boo_(f(x))(g)(x), and I'd just think “What the hell is this? How is this useful?”. I was used to methods that did something pertaining to a goal in its domain. This Scalaz code was very alien to me and yet, very intriguing. Some obviously-smart person spent time making the alphabet soup permutations, why?

I've since discovered that answer to that question and I never could've imagined the amount of benefit it would yield. Instead of methods with domain-specific meaning, I now see functions for morphisms, in accordance with relevant laws. Simply put: changing the shape or content of types. No mention of any kind of problem-domain; applicable to all problem-domains! It's been surprising over the last year to discover just how applicable this is. This kind of abstraction is prolific, it's in your code right now, disguised, and intertwined with your business logic. Without knowledge of these kind of abstractions (and the awareness that a type-component level of abstraction is indeed possible) you are doomed to reinvent the wheel for all time and not even realise it. Identifying and using these abstractions, your code becomes more reusable, simple, readable, flexible, and testable. When you learn to recognise it, it's breathtaking.

FP: The Basics

Now that I'd been exposed to FP I started actively learning about it. At first I learned about referential transparency, purity and side-effects. I nodded agreement but had major reservations about the feasibility of adhering to such principals, and so at first I wasn't really sold on it. Or rather, I was hesitant. I may have been guilty of mumbling sentences involving the term “real-world”. Next came immutability. Now I'm a guy who used to use final religiously in Java, const on everything in C++ back in the day, and here FP is advocating for data immutability. Not just religiously advocating but providing real, elegant solutions to issues that you encounter using fully immutable data structures. Wow! So with immutability, composability, lenses it had its hooks in me.

Next came advocation for more expressive typing and (G)ADTs. That appealed in theory too and again I was hesitant about its feasibility. Once I experimentally applied it to some parts of my project, I was blown away by how well it worked. That became the gateway into thinking of code/types algebraically, which lead to...

FP: Maths

I loved maths back in school and always found it easy. Reading FP literature I started coming across lots of math and at first thought “great! I'm awesome at maths!” but then, trying to make sense of some FP math stuff, I'd find myself spending hours clicking link after link, realising that I wasn't getting it and, in many cases, still couldn't even make sense of the notation. It became daunting. Even depressing. Frequently demotivating.

The good news is that everything you need is out there; you just have to be prepared to learn more than you think you need. I persisted, I stopped viewing it as an annoying bridge and starting treating it as a fun subject on its own and, before long things made sense again. It opens new doors when you learn it.

Example: I had a browser tab (about co-Yoneda lemma) open for 3 months because I couldn't make sense of it. It took months (granted not everyday) of trying then confusion then tangents to understand background and whatever it was that threw me off. Once I learned that final piece of background info, I went from understanding only the first 5% to 100%. It was a great feeling.

Feeling Intimidated

Looking back there were times when I felt learning FP quite intimidating. When I'm in/around/reading conversations between experienced FP'ers quite often I've seriously felt like a moron. I started wondering if I gave my head a good slap, would a potato fall out. It can be intimidating when you're not used to it. But really, my advice to you, Reader, is that everyone's nice and happy to help when you're confused. I have a problem asking for help but I've seen everyone else do it and be received kindly... then I swoop in an absorb all the knowledge, hehe.

It's a mindset change. I wish I'd known this earlier as it would've saved me frustration and doubt, but you kind of need to unlearn what you think you know about coding, then go back to the basics. Imagine you've driven trains for decades, and spontaneously decide you want to be a pilot. No, you can't just read a plane manual in the morning and be in Tokyo in the afternoon. No, if you grab a beer with experienced pilots you won't be able to talk about aviation at their level. It's normal, right? Be patient, learn the basics, have fun, you'll get there.

On that note, I highly recommend Fuctional Programming in Scala, it's a phenomenal book. It helped me wade my way from total confusion to comfortable comprehension on a large number of FP topics with which I was struggling trying to learn from blogs.

Realisation: Abstractions

Recently I looked at some code I wrote 8 months ago and was shocked! I looked at one file written in “good OO-style”, lots of inheritance and code reuse, and just thought “this is just a monoid and a bunch of crap because I didn't realise this is a monoid” so I rewrote the entire thing to about a third of the code size and ended up with double the flexibility. Shortly after I saw another file and this time thought “these are all just endofunctors,” and lo and behold, rewrote it to about a third of the code size and the final product being both easier to use and more powerful.

I now see more abstractions than I used to. Amazingly, I'm also starting to see similar abstractions outside of code, like in UI design, in (software) requirements. It's brilliant! If you're not on-board but aspire to write “DRY” code, you will love this.

Realisation: Confidence. Types vs Tests

I require a degree of confidence in my code/system, that varies with the project. I do whatever must be done to achieve that. In Ruby, that often meant testing it from every angle imaginable, which cost me significant effort and negated the benefit of the language itself being concise. In Java too, I felt the need to test rigorously.

At first I was the same in Scala, but since learning more FP, I test way less and have more confidence. Why? The static type system. By combining lower-level abstractions, an expressive type system, efficient and immutable data structures, and the laws of parametricity, in most cases when something compiles, it works. Simple as that. There are hard proofs available to you, I'm not talking about fuzzy feelings here. I didn't have much respect for static types coming from Java because it's hard to get much mileage out of it (even in Java 8 – switch over enum needs a default block? Argh fuck off! Gee maybe maybe all interfaces should have a catch-all method too then. That really boiled my blood the other day. Sorry-), anyway: Java as a static typing system is like an abusive alcoholic as a parent. They may put food on the table and clothes your back, but that's a far cry from a good parent. (And you'll become damaged.) Scala on the other hand teaches you to trust again. Trust. Trust in the compiler. I've come to learn that when you trust the compiler and can express yourself appropriately, entire classes of problems go away, and with it the need for testing. It's joyous.

Sadly though, eventually you get to a point where Scala starts to struggle. It gets stupid, it can't work out what you mean, what you're saying, you have to gently hold its hand and coax it with explicit type declarations or tricks with type variance or rank-n types. Once you get to that level you start to feel like you've outgrown Scala and now need a big boy's compiler which can lead to habitual grumbling and regular reading about Haskell, Idris, Agda, Coq, et al.

However when you do need tests, you can write a single test for a bunch of functions using a single expression. How? Laws. Properties. Don't know what I mean? Pushing an item on to a stack should always increase its size by 1, the popping of which should reduce its size by 1 and return the item pushed, and return a stack equivalent to what you started with. Using libraries like ScalaCheck, turning that into a single expression like pop(push(start, item)) == (start, item) which is essentially all you need to write; ScalaCheck will generates test data for you.

Where Next?

What does the future hold for me? Well, I could never go back to dynamically-typed language again.

I will stick with Scala as I've invested a lot in it and it's still the best language I know well. I'd like to get more hands-on experience with Haskell; I don't know its trade-offs that well but its type system seems angelic. Got my eye on Idris, too.

Academia!

I used to get excited discovering new libraries. I'd always think “Great! I wonder what this will allow me to do.” Well now I feel that way about research & academic papers. They are the same thing except smarter, more lasting, more dependable, and they yield more flexibility. It's awesome and I've got decades of catching up to do! Over the next year I'll definitely spend a lot of time learning more FP and comp-sci theory. I'd also like to be able to understand most Haskell blogs I come across. They promise very intelligent constructs (which aren't specific to Haskell) but the typing usually gets a bit too dense; it'd be nice to be able to read those articles with ease.

Don't fall for the industry dogma that academia isn't applicable to the real-world. What a load of horseshit that lie is. It does apply to the real-world, it's here to help you achieve your goals, it will save you significant time and effort, even with the initial learning overhead considered. Don't say you don't have time to do things in half the time. If you're always busy and your business are super fast-paced agile scrum need-it-yesterday kinda people, well I know you don't have time, but what I'm offering is this: say you just need to get something out the door and can do it quickly and messily in 1000 lines in 6 hours with 10 bugs and 10 “abilities”, well if you spend a bit of your own time learning you could perform the same task in 400 lines in 4 hours with maybe 1 bug and 20 “abilities”. You've just saved 2 hours up-front, not to mention days of savings when adding new features, fixing bugs, etc. That's applicable to you, the “real-world” and the industry. I've spent years in the industry and not just as a coder and I wish I'd known about this stuff back then because it would've saved me so much time, effort and stress. There seems to be this odd disdain for academia throughout the industry. Reject it. It's an ignorant justification of laziness and short-sightedness. It's false. I encourage you to take the leap.

Lessons Learnt on My Second Android App - Pt.1

Hi. Recently I released my second Android app. It's a collection of game timers and bells and whistles to be used whilst playing board games, card games, party games, the like. Practically, that means you can use it to play 4 player Scrabble where each player has 2 minutes-per-turn and a total of 25 min for the entire game, for example. Or, for a game like Scattegories you can have an hourglass set to run for a random amount of time between 1~2 min and then scare the crap out of everyone when it goes off. That kind of thing.

If you're interested, you can peruse some screenshots here and find the app here: “Time Us!”

Moving on, I'd like to share and record for future-me, the lessons I learned and observations I made whilst creating this app.

Scale Bitmaps Effeciently

Those tiny graphic files in your resources consume a surprising amount of memory when in use. A 100KB PNG is compressed and when loaded can comsume 10MB of memory as a Bitmap. As we know memory is quite important on mobile devices, especially on older devices. If the user is on a small LDPI device for example, their tired old phone isn't going to have much memory to spare. If you're scaling bitmaps there's a chance that you're consuming more memory that you need to, and the amount of waste gets higher with older devices where it matters more.

Say you have a large graphic that you scale to a smaller size, you're going to find that without some special options in-place, the device will load the full graphic into memory and retain it as it was loaded prior to scaling. I originally assumed that on a small LDPI device the graphic would scale down to about 20% (or whatever) of its size and accordingly only consume 20% of the full size in memory. Right? No. By default Android loads first, scales later and doesn't let go of that full-sized bitmap.

So how can you reduce memory consumption? The first recommended solution (because it is more processor-performant as well) is to prepare different copies of your graphics for each density. Ok great, but there are scenarios where that's not the best approach so what then? It turns out that there's a whole trove of info on this issue online in the official Android docs, the specifics on this scaling issue here:
http://developer.android.com/training/displaying-bitmaps/load-bitmap.html

You can mostly just copy-and-paste the utility code from the link above. By applying the methods therein, I was able to reduce memory usage by up to 80% or so (from memory), depending on the device. I didn't know this page existed until I needed it so I suggest you bookmark it and/or keep a mental note if you don't read the above page immediately.

Free Bitmap Memory Manually

Now that we know a little more about bitmap memory consumption, let's talk about another snag I encountered.

My app's home screen has two looks: dark & light. The light-mode background consists of 2 images. On a w360dp XHDPI phone, it consumes 5.8MB for bkgd img #1, and 4.0MB for #2, making a total of 9.8MB.

(FYI: You can easily see the memory profile of, and analyse your app by doing this from Eclipse: DDMS → Devices → Select process and click “Dump HPROF file” in the toolbar.)

What would you expect to happen to that 9.8MB when the user hits a button and the app moves on to the next activity (screen)? I expected that the OS would release the memory because it's not in use. Not so. As long as the home activity is in the task history the memory will stay consumed and locked which in my case means for the entirety of the app. Now maybe technology is catching up but the old phone I had for two years before my newer Galaxy Nexus, was a HTC Magic, and that thing was constantly chugging due to memory problems and as such. Consequentially, I'm not comfortable with holding onto 10MB for no reason on a mobile device.

So what can you do? Well I could draw both images onto a single canvas but that's not really solving the problem, that would just drop the 10MB down to 6MB and it would still be unavailable later. To solve the problem I did three things:

1. Remove, dereference, recycle the images in onStop().
   I wrote a utility function to do just that, and then simply called in in my activity's onStop() method. It looks like this:
2. Set the images during onStart().
   In your activity, just use View.setBackground(Drawable) or View.setBackgroundDrawable(Drawable) in your onStart() callback to restore the images that you clear in onStop().
3. Turn off hardware acceleration.
   Devices running Android 4.2 require an additional kick in the pants to get them to let go of the memory. You'll want to turn hardware acceleration off or else you'll end up with instances of GLES20DisplayList in memory that consume exactly the same amount of space. Instructions to turn off hardware acceleration can be found here: http://developer.android.com/guide/topics/graphics/hardware-accel.html

There's actually a whole bunch of doco on bitmaps and memory on the official Android site so for more info, take a look at http://developer.android.com/training/displaying-bitmaps/index.html.

PNGOUT

If you have a moderate amount of graphics and like me you optimise (ie. re-generate) them for multiple densities, then the size of your APK can balloon fast. This situation lead me to discover a utility called PNGOUT (Windows, ArchLinux, Other Linux + OSX). It optimises your PNGs and gets them down to a smaller size, from memory, using a special compression algorithm targeted at graphics. As with most types of ultra-compression it can be slow but the results are worth it.

I ran it over my PNGs. It took 30min and reduced my total size from 7.4MB to 5.9MB, a 20% reduction. Nice.

Screen Widths

I learned a few things about screen width. Firstly I had to find a database of devices and specs. One doesn't exist but I did find two admirable attempts here and here. After some analysis, from what I can tell, most older phones have a width of 320dp and I don't think there are any phones with less than that -- AdMob's smallest banner ads are 320dp. For LDPI and MDPI, 320dp is a safe bet. Once you get to HDPI, it seems that around 70% of devices are 320dp, 25% are 360dp and the rest are larger than that. Here's the most interesting piece of news: for XHDPI it seems that the 360dp is the minimum and majority. There doesn't seem to be any 320dp XHDPI devices in existence!

What does this mean? Why do I care? It means that on some devices in some scenarios, you're going to end up with more free space that you expected and a smaller perception. Instead a larger graphic is more appropriate and that's something to consider when generating graphics. You'll need to make a conscious decision about which dimension is more important to you for each graphic, and enlarge appropriately. If you want a graphic to be perceived as being the same size on a 320dp-width and a 360dp-width HDPI device, then you might want to generate another copy at 360 ÷ 320 = 1.125x the original size. If you're supporting tablets then it might pay to do something similar for the most common tablet sizes.

Here's what I ended up with:

Dir	Factor	Content
drawable-ldpi	0.75	Everything
drawable-mdpi	1	Everything
drawable-hdpi	1.5	Everything
drawable-w360dp-hdpi	1.6875	Width-sensitive GFX only
drawable-xhdpi	2 or 2.25 depending on the gfx	Everything

If you'd like to learn more have a read of these:
Supporting Different Densities
Supporting Different Screen Sizes

More Next Time

I learnt more but I'll post that next time. If you read this far I hope I've been helpful.

Android Project Templates

Today I'd like to share some templates I've made for creating Android apps. They all use Maven and have been coerced into compatibility with Eclipse. There are a few different flavours which I will describe below. If you want to skip the reading and just want the code, np, it's all here: https://github.com/japgolly/reference.

Template #1: Java

Code available here.

This is the gist of what I use when I'm using Android + Java. It comes with a few add-ons to mitigate Java's verbosity and inflexibility and the fact that they all work together in harmony, even with Eclipse, well that's what makes this template gold. It took a while to get working way back when.

Features

• Lombok: Integrated into both Maven and Eclipse.
  Modified to play nice with Android, and added the ability to add @Inject to generated constructors.
  What is Lombok?
• AndroidAnnotations: Integrated into both Maven and Eclipse.
  What is AndroidAnnotations?
• CoFoJa: Contracts For Java provides annotations that can be used to verify pre-conditions, post-conditions and invariants. The validations are inherited so they're great for interfaces. Also they can be omitted from bytecode during compilation meaning you get more checking during tests and dev with no performance penalty in prod.
  More about CoFoJa.
• Unit testing: Robolectric is being used for unit testing (along with an example of usage). I'm actually using a test library of mine (open-source) that includes Robolectric, FEST, Mockito and some Android-specific testing utilities.
  What is Robolectric?
• Integration testing: Robotium is being used for integration testing (along with an example of usage).
  What is Robotium?
• Android Lint: There's a profile that run Android Lint. Great for CI builds.
• Findbugs: Findbugs has been integrated (not that it was effort). What's nice is that I've included a rules file to exclude certain Android-specific warnings. I'm a nice guy.
• Proguard: Dev & release configs for Proguard are included.
• Release: A release profile is included that runs Proguard, signs artifacts, and runs zipAlign.
• AdMob: It's in there, ready to go but can also be deleted just as easily if you don't want it.
• Eclipse settings: Auto-format on save. Auto-organise-imports on save. A TODOC tag for doco todos. More.

Template #2: Scala

Code available here.

Here you can use Scala instead of Java to build your app. The introduction of Scala renders obsolete the need for Lombok and AndroidAnnotations which simplifies the build process. As above, everything in this build works from CLI with Maven, and within Eclipse.

Features

• Scala: Scala app code. Scala unit tests. Scala integration tests. Scala proguard. Scala Eclipse. Scala everything!
• Unit testing: Robolectric is being used for unit testing (along with an example of usage). I'm actually using a test library of mine (open-source) that includes Robolectric, FEST, Mockito and some Android-specific testing utilities.
  What is Robolectric?
• Integration testing: Robotium is being used for integration testing (along with an example of usage).
  What is Robotium?
• Android Lint: There's a profile that run Android Lint. Great for CI builds.
• Proguard: Proguard is mandatory when Scala is used. Rules included so that Scala works. Dev & release configs for Proguard are included.
• Release: A release profile is included that runs Proguard, signs artifacts, and runs zipAlign.
• AdMob: It's in there, ready to go but can also be deleted just as easily if you don't want it.

NOTE: You cannot use Scala and Java together because it wrecks Eclipse. If you're using VIM or some other editor inferior to mighty VIM, you'll be fine with both languages simultaneously, Maven can handle it, just don't expect a happy life using Eclipse.

Template #3: Scala + Free/Paid Editions of Your App

Code available here.

Same as the Scala template above except this time instead of having a single app, you have an Android library and two separate apps that extend it.

Features

• Everything in template #2. See above.
• Shared APK Library: Anything shared between your free & paid apps will live in the shared library. It's an Android library so resources can be shared too.
• Separate Apps: Your free and paid apps are separate apps that extend the shared library. Additional code and/or resources can be dropped in as necessary.

Usage

To use one of these templates just copy the entire folder (and ensure you capture files starting with .). Next use the find_template_values.sh script to find places in the project where you need to swap out example values for real values. There's also a helper script, replace.sh that you can use to perform recursive, mass regex replacements.

To load your project into Eclipse, simply hit File → Import... → Existing Projects into Workspace. All the required Eclipse settings and files are already there in the template.

Enjoy.

Releasing an App for Android - Part 2

Continuing on from part 1...

4. Free and Paid Editions.

The Google Play marketplace doesn't allow free & paid editions of apps. If you want free & paid editions you will need to manage and release two separate apps with separate package names.

To do this I decided to go with the mainstream approach, namely, creating a separate projects. If you have shared code without shared Android resources (unlikely) then you'd simply move the shared code out into its own project and include the resulting jar as a dependency in the free & paid projects. With Android it's similar, you create an APK library. Practically this means adding android.library=true to your library's project properties and changing its pom packaging from apk to apklib. The free/paid projects wont contain much but they will need their own AndroidManifests and a reference to the library in the project properties such as android.library.reference.1=../library. That's the gist of it.

One annoying thing is that, like a few things that have come out of the Android dev camp, it's a hack. The resulting library is basically just a zip of the resources and the SOURCE CODE. When you use an Android library in a top-level project it unzips, merges and recompiles the library code. How annoying. The reason is the generated R.java will produce different ids outside of the library, hence the recompilation. As far as I understand the system is therefore a quick hack. It might be fair enough - I don't know how much pressure the Android dudes are under or how well/poorly staffed they are. But technologically speaking is still a hack. Oh well. Make sure Scala incremental compilation is on as it takes a large bite of the build pain.

In retrospect it was a good thing that I saved this step until the end. The time to alter the build & structure (especially if you've got a working reference) near the end of the project will be much less than both the time spent by managing 3 separate projects instead of 1; and the time wasted by waiting on the extra compilation, dexing and proguard'ing.

Tip #1: Use a single project for as long as you can get away with but anticipate that towards the end you'll need a little time to split out library, free, and paid projects.

Tip #2: Anything edition-specific should be separated from anything shared. For example, create an ad banner layout and then <include> it rather than using an AdView directly. Later you can configure things so that the paid edition gets a NOP layout.

5. Advertising.

I started not knowing anything about ads. Basically I did this:

1. Sign up with AdMob.
2. Fill in all your bank details so they can pay you. You'll also need a merchant account with Google Play.
3. Add your app but don't give it a market URL yet. This allows you to get an ID that you can start using before you release.
4. AdMob doesn't just show you their own ads; they connect to other ad networks and can show you their ads too. That was surprising. Advantage us because it means that once you get ads working in your app and you want to use a different ad provider like InMobi or MobFox, you can just sign up with those guys, get your ID and plug it into AdMob on their website. No code changes required. You can even configure Admob to show ads from a combination of providers, or use different providers for certain countries.
5. Get the AdMob SDK and wire it into your project.
6. Follow the instructions on this page to integrate ads into your layout: https://developers.google.com/mobile-ads-sdk/docs/admob/fundamentals. Reading all those pages is a good idea.
7. Finally, if you want ads to refresh every 60 sec or so, don't do it in code; play around in the AdMob website and you'll find settings for it. You can also customise the appearance of ads on the website or in code.

Done.

Now I don't know much about how it all works yet but this is what I've gleamed:

• Every time an ad is shown to a user it's called an “impression”. You get nothing, $0.00, for impressions.
• Every time an ad is clicked you get something small like $0.03.
• RPM = “Revenue Per Mile” = the amount of money you've made per 1000 impressions.
• Fill rate = Quote: Fill rate represents the percentage of ad requests that satisfy the ad requests sent by the app. It is a measure of AdMob's ability to serve ads in your app with the existing inventory.
• You'll see eCPM everywhere, it stands for “effective cost per mile”. It's a metric indicating how effective/profitable a particular ad network has been for you so far. Don't freak out when AdMob says eCPM $0.00 when you first sign up, that just means you haven't had any clicks yet. The formula is 1000 x Cost-per-Click x Click-Through-Rate.

And we've reached the boundary of my knowledge on the topic. Hope you enjoyed the tour.

Tip #1: Filter logcat by tag “Ads” to see all your AdMob logs.

Tip #2: Search the AdMob logs for your test device ID and then plug that into the ads:testDevices attribute of your view. From what I hear the AdMob support is notoriously bad so if you get in trouble with your account, it's far from easy to get it back online.

Tip #3: eCPM is a metric of how well an ad network has been for someone. It's not a setting that you need to configure (even though it looks that way from AdMobs UI).

6. Localisation.

Not much to say here. Android warns you constantly about externalising your strings which is good. It doesn't monitor your source code though. If paranoid or you have a large team, Eclipse can warn when it finds strings without something like //$NON-NLS-1. Checkstyle et al probably have a similar feature.

Tip #1: Either put non-translatable strings into a file called donottranslate.xml (seriously – Android Lint requirement) or give them an attribute translatable="false".

Tip #2: A split to paid/free editions will introduce new strings. Consider this before sending out text for translation.

Tip #3: The marketplace description(s) will also need translation. Write and include that before sending out text for translation.

Tip #4: Your text is easier to manage when all in one place. Therefore don't include separate strings in the free/paid versions; put them all in the library project together then just reference them differently in each app. If you don't want to change the references then reference an alias and change the alias in each project. Example:

Library project:
  <string name="app_name_free">Bananas (free)</string>
  <string name="app_name_paid">Bananas (pro)</string>
  <string name="app_name">@string/app_name_paid</string>

Free project:
  <string name="app_name">@string/app_name_free</string>

Tip #5: Use the app in each language to spot-check the layout. Certain languages might need tweaking to look good. For example, a language like Japanese that doesn't use spaces might need a manual endline (\n) so that it doesn't end up with one character dangling and the word split across lines.

7. Proguard.

If you don't know what Proguard is, it's a tool that shrinks, "validates", optimises and obfuscates your binaries.

Scala on Android demands it.
Optimisation is very slow.
Obfuscation didn't seem to do very much to my app although it happily obfuscated my dependency libraries.

I managed to speed things up by having two proguard configs: dev & release. Turn off optimisation and obfuscation in the dev config and you will save a lot of time. With optimisation enabled it can take minutes at a time. I've uploaded my configs here: https://gist.github.com/japgolly/4747423

Tip #1: Reuse existing proguard configs.

Tip #2: Use separate dev & release proguard configs.

8. Signing.

Your app will need to be signed (and then zip-aligned, in that order) in order to release it. To sign it you need to have your own certificate that doesn't expire until at least 2033 (which isn't the best for security but I don't make the rules). It's a Java thing so just google keytool and you'll be done pretty quickly. You'll need to create and save 2 passwords: one for the key, one for the store. I generated mine with this command: (tip: script it or purge it from shell history when done)

keytool -genkey -v \
  -keystore <keystore filename> \
  -alias <keystore name> \
  -keyalg RSA -keysize 4096 -validity 10000 \
  -keypass 'xxxxxxxxxx' \
  -storepass 'yyyyyyyyyy'

Once you're done, make sure you backup your keys and passwords then it's time to integrate signing into your build. For a Maven project you need to weave a massive blob of shit into your poms in order to integrate signing and zip-align. They should be in a release profile and all up it comes to around 100 lines of XML (!). I don't remember how much I was able to refactor into my parent pom (as my project is multi-module) but I don't think it was all of it. I'll post all the Maven stuff shortly anyway. Most of the time spent on signing was getting Maven to work properly.

Tip #1: Either ditch Maven for something better, or make sure you copy a working Maven Android project that has signing. (I'll post my Maven stuff shortly.)

9. Build Automation.

Last but not least, we arrive at something that is actually important from day #1 of the coding period: build automation. Very important. Obviously. I used Maven and ended up creating a multi-module project which I'll post more on later. Suffice to say Maven gave me trouble, and cost me too much time. I want to switch to something else and I have my eye on SBT.

I did some tests on SBT recently and the results supported the rumours that SBT is faster than Maven. A build of a single, simple Android project took 35 sec with Maven, verses the same project at 24 sec with SBT. That's a 31% saving. A clean build of my multi-module Maven project takes 2 minutes and I wonder how fast it would be with SBT. If I assumed flat 31% saving again, that's 120 sec down to 82 sec. Nice. I didn't investigate further because it's time-consuming and because I worry about Eclipse integration. I'm sure Eclipse + SBT = happy days, but it seems that Eclipse + ADT + SBT = pain. ADT is extremely inflexible and the sbt-android plugin is way too poorly documented for an SBT noob like me. If I changed the dir & file structure to suit ADT an expert might be able to modify the sbt-android build pretty easily but I can't. Also, it practically has a seizure if you give it a full AndroidManifest as it wants to create its own. I imagine that ADT wouldn't be too happy about that. Would loooooove to be shown the error of my ignorant ways here so if anyone knows, ping me.

Another thing is that Maven is archaic and verbose. Reuse between modules is impossible in some situations; there are 6-year-old bugs and feature requests for mixins that are still open... SBT on the other hand is concise and allows as much reuse as you can shake a stick at (and I can shake a stick at A LOT of reuse).

But any further bitching about Maven is just that at this point: bitching. So I'll suck it up and declare that though sweat and toil I've come to have a great multi-module Maven Android project setup with library, free/paid projects, and free/paid (instrumented) integration test projects working together, nuances coerced, effective for dev and release builds. I'll be posting it shortly.

In Conclusion

Looking back on all these issues I can now see how it took 5 or 6 days to go from 95% dev complete to released, despite doing 0%-95% in 2 days. I'll be applying these lessons during my other Android projects and I hope it helps others (yes you!) as well.

Releasing an App for Android - Part 1

Last week I decided I would create a little micro Android app and release it so that I could get experience with the process. I recently came to the opinion that it's a good idea to hit learning curves with something small first rather than attempting both “big” and “new” at the same time. So I created an Android app called BPM Tapper. What I found surprising was that I went from vague idea to 95% code complete in 2 days, but then it took another 5 or 6 days to get it released.

Now, my goal here with this blog post is twofold. Firstly, to document what I went through releasing my first Android application so that in future I remember everything that needs to be done; great for planning. Secondly, I want to document tips and strategies to improve next time.

1. Android Completeness

By Android Completeness I'm referring to things that are either required or expected of common Android applications. The tasks I came across are as follows.

1.1. Inspecting Android Lint warnings.

This is an easy one. The Android SDK provides a lint tool that analyses your compiled code and resources, and creates a list of warnings. Not all Lint warnings are correct but they all should be vetted and resolved if necessary.

1.2. Handling interruption.

Your app can be interrupted at any time, e.g. if the phone gets a call while your app is in use, the OS will kick your app out and data will be lost if you don't handle it. Therefore if you don't want users to lose their on-screen state when using your app (or changing orientations), you generally need to implement two methods in your activities. onRestoreInstanceState and onSaveInstanceState. Scala made this really simple; case classes are serialisable so one simply calls Bundle.[get|put]Serializable. Example:

val measure = savedInstanceState .getSerializable(BUNDLE_KEY_MEASURE).asInstanceOf[Measure]

Too easy.

1.3. Integrating with the Android Backup Manager.

My app is stateless so I just flipped the allowBackup switch off in the AndroidManifest and didn't have the need to write any code here, but it is something to be aware of. It's easy to forget but very important for stateful apps.

1.4. Supporting multiple screen densities.

Different phones have different screen densities and you should at least be targeting support for MDPI through to XHDPI. Different screen densities (loosely) require different copies of your graphics at different sizes. This wasn't a problem for me because I create all my graphics in SVG and have a script to convert them to PNGs using ImageMagick. (See the Artwork section for such a script.)

1.5. Providing landscape layouts.

Not mandatory but I've seen quite a few reviews on the marketplace where people get quite passionate (read: vehement) about not being able to turn their phone on its side. Personally I don't see why it's such a big deal but I'm not everyone. Creating a new layout for landscape-mode and having it coexist is easy. However, designing the landscape UI that is pleasing and effective is the time-consuming part.

Tip #1: Always use something like Inkscape to create the screen before you hit the code. Use real sizes and colours. Don't assume and leave out anything that you plan to do in code later.

Tip #2: Commit to doing this to all screens in your app up-front, or no none at all. Or do some. But make that decision early on and ensure you can make it work. You don't want to create 8 landscape layouts then later come across a few that won't work then decide to scrap the entire landscape orientation ability.

1.6. Ensuring layouts look acceptable in various conditions.

This means compiling a testing matrix of relevant attributes such as:

• Screen size.
• Screen density.
• Screen orientation.
• Android version.
• Ads vs no-ads (they do take up valuable screen space).
• Language.

This is pain. The theory is simple right? The screen will look different depending on each factor above. You want your beloved app to look great on all users' phones.

Screen density I've addressed above. That, language and Android API version can just be eyeballed manually I think. No need to give them their own axis in a matrix. But I did want a matrix of ads-or-not, orientation, and screen size. What was annoying is that screen size is not screen size. There are resource qualifiers for screen size, namely small, normal, large, xlarge. Now they didn't work as expected for me. Due to my immense frustration and exhaustion at the time, I don't remember the details but I found them not aligning with my expectations at all. Instead I decided to use screen density to create different settings for different screen sizes, because there is a strong correlation between density and screen size anyway. I mean, no one has a 1200x800 LDPI phone. It doesn't exist. Thus I used a test matrix like this:

	ads		no ads
	port	land	port	land
ldpi	✓	✓	✓	✓
mdpi	✓	✓	✓	✓
hdpi	✓	✓	✓	✓
xdpi	✓	✓	✓	✓

Tip #1: Create a test matrix and use it to ensure things look good.

Tip #2: Eyeball the smallest screen sizes once in each language. Button widths can be twice as long as English depending on the target language which may cause side-effects with your layouts.

Tip #3: Always start with the lowest density and work your way up. A MDPI screen will choose values-ldpi/ over values/ if it exists and values-mdpi/ doesn't.

Tip #4: Don't trust the screen heights you can choose in the ADT layout editor. I have an old HTC Magic that has around 100dp less height than the closest config available from the editor.

Tip #5: Do not leave any hardcoded dimensions in your layout XML (obvious) or your styles.xml (not obvious). When adjusting for each screen it's going to be the dimensions like the text sizes, the padding & margin sizes that you'll be adjusting.

Tip #6: Do your utmost, do your friend's utmost (!) to avoid using dimensions to affect the layout itself. For example, a dimension for the margin between elements is one thing but if you want a large river of negative space, don't use dimensions, anchor both sizes of the space to stuff so that the river size is organic. This makes for more consistent layouts on different devices and means you won't have to adjust the value for different screens.

2. Scala.

Learn it! Use it! It is a brilliant language. It allows you to write better code, more concisely. There is so much boilerplate bullshit that Java requires, so many hoops to jump through, and so many obstacles that prevent you writing the kind of code that you (well, I) want to. Scala does away with nearly all of those! Its strong focus on immutability, functions, maximum code reuse (yay for multiple inheritance again!) and conciseness, not to mention all the other modern goodness such as pattern matching/partial functions; for comprehensions; case classes; implicit conversions, classes, and parameters... The list goes on. It's very productive language and you will be more productive with it.

Like anything it does have it's shortcomings though: flaky IDE support, slow compilation time, WTF-inducing stacktraces, controversial lack of binary compatibility between major versions (which doesn't bother me at all actually), massive stdlib that Android can't handle without proguard, too many custom operators at times (look at SBT with its <+= <<= ++= <++= etc, good god!).

But all in all, Scala is worth it. IDE support will continue to improve and compilation time and stdlib modularisation should be fixed later this year.

Scala can especially be a godsend when it comes to cutting out Android/Java repetition. Consider this Java:

protected TextView nameView;
protected void onCreate(Bundle savedInstanceState) {
    ...
    nameView = (TextView) findViewById(R.id.name);
}

And now consider this Scala:

lazy val nameView = find[TextView](R.id.name)

Here's another example. Java:

view.setOnClickListener(new OnClickListener() {
    @Override
    public void onClick(View view) {
        onDelete();
    }
});

Verses Scala:

view.setOnClickListener( onDelete _ )

Scala also saved me time on the testing front. Your test code becomes amazing when you can use traits for multiple inheritance, and write methods that accept lambdas or Ruby-style blocks. You'll end up with something approaching a testing DSL before you know it, and most of your test cases will be under 3 lines which certainly taking the wind of the sails of any test-case-writing-laziness.

Tip #1: Use an incremental compiler for Scala. With the scala-maven-plugin this is a single config item.

Tip #2: Copy a proguard.cfg with Scala settings and set your build to always use proguard.

Tip #3: Use traits to create collections of test helper code. Do as little directly as possible in your immediate test case code.

Tip #4: Use implicit conversions for all things Android.

Tip #5: Create an ActivityHelperTrait or similar and load it up with implicit value classes and inlined helper methods. Example:

@inline def find[T <: View](resId: Int): T = findViewById(resId).asInstanceOf[T]

3. Artwork.

Get in the habit of using vector-based graphics and using ImageMagick's convert tool (it's command-line). Vector graphics look great at any size and you can convert to generate bitmaps at various sizes (which Android needs). If you want you can use SVGs directly in Android with svg-android but there are still places where you need fixed-size images.

You will need to create or procure a visual to use as your application icon before release. Ensure you have it as an SVG (or some other vector-based format) then use ImageMagick to create various sizes. The marketplace requires 512x512, where as your app requires 36x36, 48x48, 72x72, 92x92 for ldpi, mdpi, hdpi and xhdpi respectivelly.

For all kinds of visuals you'll need to have multiple copies in separate sizes to support different screen densities. I wrote a Ruby script to convert all my visuals. Here is an except of that script:

Tip #1: Use 32-bit PNGs if you want an alpha channel.

Tip #2: ImageMagick will be your best friend for conversion. Script it.

Tip #3: Inkscape is a great tool. Very easy to use and everything is vector-based.

Tip #4: There are plenty of good-looking, vector-based visuals out there that are public-domain. There are also sites like The Noun Project that have brilliant visuals that you can usually use free as long as you attribute the author, or can buy cheaply.

To be continued...

I have 6 more points. Read about them in part 2.

Rant: Stupid Maven

I'm back in the Java world again, and already I'm annoyed. The Ruby world isn't without its problems but Ruby problems are generally much easier to solve when they do appear. You generally don't end up wanting to punch the screen and scream "give my life baaaack!" (Thor's internals get me close though.)

Today's problem: Maven.
Now, I've been a big fan and advocate of Maven for years. Therefore, upon returning to the Java world I happily sought out my good friend Maven and started using it immediately with my new project. ...Only to have problem... after problem... after problem... So now I'm going to vent.

• TO DEPLOY to a local instance of Nexus you need to plonk big blob of shit (ie. XML) in every single project POM file. Adding <distrubutionManagement> in user profile's settings.xml doesn't work. WTF! In Maven-land, any artifact can be deployed to any repository so long as the repo configuration allows it. A beauty of Maven is the interoperability of artifacts & repositories, this is, a project that can deploy to Maven Central can also deploy to an intranet Nexus or Artifactory; no code needs to change; the protocol is standard. Forcing the storage of repo settings alongside the project's build instructions doesn't seem very bright to me. It's nice... but how to build a project should depend on where a project's gonna go after its built (especially when there's no impact on build anyway). In reality they are separate concerns but the system doesn't allow that separation. If repo info could be stored in either project or user-profile settings so that people could decide based on the project circumstances then fine, but as it stands currently, it is infuriatingly not the case and not supported.

• SO, NOW to make things work, I need to have the same parent pom for all of my projects that I plan to deploy to my local Nexus. The alternative would be to copy & paste the same 20 lines or so of XML into each project: umm no. What's annoying about this is a) it's something else to maintain (as opposed to settings.xml which wouldn't have to be deployed or available as an artifact), b) it introduces a new dependency (literally, not runtime) to all my projects now, great; and c)  Maven poms use single-inheritence -- no mixins; to inherit from a different parent I have more stuffing around to do. Bad architecture. Annoying.

• Parent POMs (multi-module or standalone) allow you to configure plugins, however those plugin settings aren't automatically inherited in child projects. For settings I think you need to redeclare the plugin in the child project (not 100%, I don't even care anymore). What really bugs me is that version specifications are not inherited. WTF GUYS! So if I have a multi-module project (and I tried this) where in I declare the version of a plugin in the parent pom, the child does not use the specified version of the project. OMG. I tried with <inherited>true</inherited>, I tried in both <plugins> and <pluginManagement>, it doesn't matter. Now I guess I'm supposed to declare the versions in as properties and specify plugin versions in every single child module instead? Not only am I averse to duplication for the obvious reasons, but now my pom files are all massive! Now I googled this and apparently this is deliberate because "you should always specify specific versions, inheriting versions is evil and disallowed", yada yada. Yep, well it's one thing in the context of isolated projects but in the context of multiple modules comprising a single project, it's the opposite. The modules are components that are meant to be built together to create a larger system. Declaring certain plugin versions at project-scope is entirely reasonable and has many advantages. Secondly, when you don't specify a plugin version, where does it come from? The build still works so it's coming from somewhere, right? It comes from the Maven super POM which is like a template that all poms inherit. In it, default plugin versions are specified left-right-and-centre, and they are inherited and used without explicit specification or "evil" consequences (although there are some unless the release plugin has been updated to hardcode all plugin vers in child poms during release - haven't used it in a few  years). Double standards.

• Finally XML. Come on. It's 2012. XML has always been a tedious, ugly, unacceptably verbose format. The reasons that made it appealing in 1998 are no longer valid in my opinion. Especially not in the context of pom.xml. One the best things about Maven is that you don't have to write a bunch of crap for every project in order to get it to build in an automated fashion; Maven allows you to simply say "I'm blah v1.0 and I need v2 of X and v3.4 of Y to build. See ya!" and it takes care of the rest so long as you follow its standards. Add a few dependencies though and add a single command (argLine) to surefire (the Maven testing plugin) and you're looking at over 100 lines of XML. Writing everything manually in Ant would only come to about 80 or so and it uses XML too! There was a project called Maven Polygot a while back that was supposed to allow the pom to work in different formats but it seems dead or at least progressing slowly enough that it is moot anyway. Progress is pretty generally slow in Maven-land. Point: In this day and age I resent having to use XML almost anywhere. Have you ever filled in a form by hand, a tax return, a car rego, and seen:

  Phone Number: ________________ /Phone Number

  I haven't. I really dislike XML and I'm glad it's finally starting to trend away. Look at how concise things could be:
• pom.yml -- https://github.com/mrdon/maven-yamlpom-plugin/wiki
• Gradle -- http://www.gradle.org/docs/current/userguide/artifact_dependencies_tutorial.html
• Buildr -- http://stackoverflow.com/questions/1015525/why-use-buildr-instead-of-ant-or-maven and scroll down a little.

Rant.stop.