Tale of a Costly Offshore Mobile App Development Experience

By John Houghton on January 7, 2014

As you may have read from previous blog posts, my company did an extensive search of many countries for mobile developer talent, both for iOS and Android.  Read my article, 12 Countries for Recruiting Mobile App Developers.  We sent out hundreds of solicitations via LinkedIn, got good responses, screened them, and it seemed that we’d found the perfect candidate.  We’ll refer to him here as Lukas.  While the name of the developer has been changed, the timelines and figures presented here have not been changed.  Lukas said he had made over 50 apps, had a Master’s Degree in Computer Science, and we were able to verify one app of his on the iTunes store.  It had no reviews, but we downloaded it and it worked well enough.  The strategy for finding a good developer is to first give them a small project or piece of a project and see how they perform.  A lot of them don’t do well, so you need to run a numbers game to find the good developers by trying out a number of developers as I’m about to describe.

I got on Skype with Lukas and he was very personable and multilingual.  It turns out we both know Italian, so we spoke in Italian and had other similar interests.  We hit it off well, so I decided to try him out.  Why not put him on a simple app as a test and see if he would be a good remote team member for small projects?  So we sent him the specs and got an estimate.  He wasn’t charging much and this was puzzling, plus we didn’t get a deadline, but at that cost, what is there to lose?  The project was an iPhone app and it was an app based on a RESTful service API.  For the purposes of this discussion, let’s say it’s an app that provides a realtime readout of the current weather at your location.  Sounds pretty simple, right?

In order to do this type of app, we wanted to use the Yahoo Weather API.  The actual API we worked off of is different, but the Weather API functions very similarly to what we really built (changed for privacy reasons).  In the West and with a good team, something like this takes about a day of design (see my article on 7 Steps to Design an iPhone App User Interface), a day of server work, and 3 days of mobile development.  (I should note that, if you add multiple stakeholders and a larger team, it will take a lot longer.)  Yahoo won’t let you hook your users up directly to their “data feed” (a RESTful service that publishes the temperature every minute) because they don’t want your volume of users, so you have to have your server ping this feed and let the users then hit your server (not theirs) to get the data.  This is because Yahoo doesn’t want to scale to your user load since they would have to pay for the extra servers to handle this.

We coded the server to do its thing, and developed graphics.  Lukas coded the user interface in a few days with the graphics we supplied, and delivered a basic working app in about four days, but it was unreliable.  To make a long story short, we had some unexpected problems.  First of all, Yahoo’s API wasn’t stable and would go dark for long periods of time.  We found out later that the API is in the process of being desupported.  It took a while for this guy to troubleshoot what was going on, but with some onshore help from my San Francisco developer and a week’s time, we switched over to another API.  This API was paid (versus free – I like that better), and it was stable.  Ok, some time lost, and we used some expensive onshore hours, but it seemed like we were ready to make progress.  Besides, Lukas is a nice guy and it’s fun to communicate with him.

Lukas moved to finish the rest of the project.  He sent us a build.  It looked alright, but the performance was extremely slow.  It took up to 9 seconds for the first screen to load, so we burned some more cycles and got him on his way.  Also, it didn’t work on the iPhone 5.  It turned out that he didn’t have an iPhone 5 and therefore had no way of testing it.  More onshore help and time was required to get him on track.

It’s now three weeks later and he keeps turning in builds with bugs.  Finally, we do one big set of comprehensive testing and send him all of the bugs.  There are dozens of them and they are all triggered by different events, many of them related to not handling the app properly when a user is in varying states of wireless connectivity.  What if the user didn’t have WiFi and opened the app, then WiFi became available – we got another specific bug.  What if we had connectivity and lost it?  A different bug cropped up there and we needed to put up an error message that WiFi was lost, and so on.  Not only that, but I wanted reliability built in where data was cached on the server and on the device (otherwise, you would get a NULL message), and the app needed to be coded to get the cached data when it couldn’t get the updated weather data.  If you’re interested in putting together an effective bug testing strategy, read my article, 4 Tips for Releasing High Quality Mobile Apps.

There are a lot of test cases here.  What are you going to show on the screen when the user turns the app on and they’re out of range?  A status message, I hope!  And I hope you only show it once, not three or six times as Lukas’ build was doing.  Another case is that of an app being freshly downloaded from the iTunes store and run for the very first time by the user.  There would be no cached data in this app to fall back on, so what would you show on the interface for values if there were no connectivity?  If the developer doesn’t handle these cases (there are 34 of them for this app), the app can freeze, crash, or do something else unexpected.  But wait, wasn’t this supposed to be a simple weather app?  Yes, it is a simple app and that’s why it’s three days of work for a skilled developer.  Give it to someone who isn’t up-to-par and it will take a great deal longer.  Compounding the problem, most stakeholders that are new to mobile development underestimate the amount of work it is to build a even a simple app.  At this point, it becomes an experiment to see how long this guy takes to finish.  It’s not a lot of money, but I didn’t realize until I added it up later how much of my time he would take.

Some time along the way Lukas realizes he’s has underestimated the effort, is putting in a lot of hours, and wants to stop work until he gets paid.  The only problem is that he negotiated to get paid at the end.  Hmmm.  Nice guy, but it doesn’t seem as though he thought ahead.  Maybe in the Second World, you can go to your boss and say, “Hey, I’ve been working hard today, how about some money?”  We get that resolved after some discussion, but I mention it here because things like this don’t happen when you’re dealing with professionals in the US.  Also, with his payment plan being so flimsy, it seems like we may very well be his first customer.

So earlier in this post I mentioned that we sent him a few dozen bugs.  He’s working hard and making progress, so I cut him slack, but it takes a few weeks of back and forth to get these bugs fixed.  He was doing the best he could, it just took a while.  It took him 7 weeks in all, and the app still had a major bug that caused it to crash hard.  I finally gave the project to my regular developer and told him to redo it.   It was done in 30 hours of work.  We tested the build thoroughly and it had no bugs (that’s what you should expect).

Even though the Eastern European guy wasn’t very expensive, what was supposed to be a 4-day project (it was a simple app) turned into a 7-week project that needed to be redone.  Luckily, we weren’t under a deadline, and we wanted to see what this guy would do until the end (plus I liked talking Italian to the guy).  For a less-experienced stakeholder on a critical project, it would be hard to know when to cut the guy from the project, because he hit some real issues, and you never really know how tough things are unless you’re in there yourself, or unless you assign somebody whose skills you know to assess the situation.  It can be hard to come in on someone else’s work, so the only real test is to have a developer you know perform the project from start to finish and that will tell you how severe their roadblocks were.  That aside, because we’ve done a lot of projects, we generally know how much work a project is, and a good developer could have plowed through these issues fairly quickly.

I had a conversation with him some time later when he asked me if he should raise his rates.  I politely told him he should first learn how to code better (without bugs and of course he would have to get much quicker).  It’s not like he couldn’t catch these bugs before sending the build.  Maybe he doesn’t have time because he’s working 2 or 3 jobs?  I don’t know.  I feel bad for the guy.  He was really nice and worked hard, but I can’t tie my company’s fortunes to developers like this, so I don’t tolerate any of that on my teams.

I added up all the time spent on a spreadsheet, revealing that although his expense was a small fraction of the total cost, the time it took to manage him was so great, that it would have been cheaper to develop the whole thing here in the US.  I shared the data with him, so that he could see my point.  As we add up the expenses on this project by way of fully burdened headcount cost, here are the two scenarios:

Cost to Develop Offshore with Lukas: $16,950.00 USD / 7-week project
Cost to Develop in US: $13,750.00 / 1-week project

Notice that not only did it cost more to go offshore, but it also took 7 times longer.  However, it doesn’t have to be developed in the US.  A good offshore team can also do this well and inexpensively, but the point here is that you can’t just walk onto the playing field and expect to make all of the right moves.  It takes a while to get your offshoring capabilities together and find the right people; it’s a lot of tedious up-front work, and if you don’t want to invest in this, you should stick to conventional methods.  One of the hidden expenses is that while the underperforming developers are burning time, the onshore stakeholder is burning salary either waiting or having to play firefighter and rescue the project.  Time-to-market is critical with apps, so you also risk missing the market.  I hear that story most of all – it took so long to develop that someone else came up with the idea and got out there first.

The biggest cost in all of this was to my time.  Normally a micro project like this requires me to write and respond to 10 emails at about 1/2 hour each.  By the end of this project I had written or responded to 120 emails and I was limited in what I could get done on other projects in this 7 week period because I needed to keep Lukas on track.

In the end, Lukas was a nice guy, worked hard and didn’t intend to cheat us, but I don’t see how he could have made 50 apps.  At the skill level he was working, his skills were similar to a web developer with no education, working on his second app, after his first app had likely taken a very long time.  Web developers don’t do well with mobile apps as HTML/CSS/PHP are pretty simple languages, while Objective C is a heavy duty object oriented language and it takes a while to learn it.  On the web, you can “hack it until it works” and this is frowned upon in programming an can lead to a very poor quality app.

This experience is about average for venturing offshore.  This guy was one of hundreds that made it through screening and he was screened as well as we could.  I find from this point in the screening process, the odds of finding a good developer are one in four, so you have to give the project to three more developers and look for the one who comes through.  We usually get one good one at the end of all this.  I suggest getting to know a project and reusing it for this purpose, that way you know where the gotchas are.  In the second and third world, the norm is people get paid for time spent, not end results, so you should pay everybody whether you think they did a good job or not.

Unfortunately, unless they’re seasoned software professionals with offshore experience, this story is the typical experience of what I see when US leaders who go offshore, except instead running of a test project like ours was, the project is real.  Apps are not simple to build, but many stakeholders think they are.  I see projects that take 7 times longer than they should.  I see unqualified developers that are learning on their customer’s “dime” and needlessly burning through the budget.  I see a lot of good apps that never make it.  I see investors, CEOs, and stakeholders idly sitting by because they aren’t technical enough to know what they’ve gotten themselves into.  I don’t have an immediate solution, other than to raise awareness about these issues. Take your time, start small, have offshore work checked by people you trust, and don’t dive fully into the pool until you know how deep the water really is.  I hope my experiences can help any of you who are considering offshoring.

Posted in Android Apps, iOS Apps, iPad Apps, iPhone Apps

Comments

Comments