AdDuplex Post-Mortem. Part 5: Growing Pains — Hiring, Technology, Business
This is part 5 of the AdDuplex (2011–2023) story. AdDuplex was the widest ad network for Windows apps. You can read this part in isolation or start at the beginning.
It is common for everyone to congratulate founders on raising capital. While it’s an important milestone it is not a cause for celebration in and of itself. It’s like congratulating a handyman on getting a new power drill. Sure, it’s an important upgrade in the toolset but it’s just that — a tool. Once you get the investment money you need to put it to work, which means more work for you, not less.
In the startup world that work usually begins with expanding your team.
Hiring
The first order of business was expanding the team up to my initial plan — getting a dedicated person for each core function I imagined. In a month or so we expanded the team to 6 people. In 2013 it wasn’t as difficult as it became in the next couple of years. Some came through our own (well, Paulius’) network and it was still possible to get an inbound funnel of candidates from the job ads.
We moved from a smaller room to a slightly bigger room in the same office building, but it was clear that it wouldn’t be enough for long and before we could expand further, we would need to move. When the team grows beyond five or so people, as a small startup CEO you quickly realize that you basically have two options: spend a significant portion of your time on handling mundane housekeeping issues or hire someone to take them off your shoulders. And if you are a funded startup the choice is obvious.
Lesson learned: Your employee number 6 should be an office manager.
The specific title or even a job function could differ in each case, but you definitely need to delegate day-to-day operations that are not directly related to growing your business. There’s no excuse to try and save on that.
We moved to a new office and continued hiring on an as-needed basis for the next two years. Overall, 20 people worked at AdDuplex directly and the team was 15 people at the peak. The numbers may seem low in abstract, but one needs to remember that different types of businesses have different people needs. There wasn’t a reasonable way to 2x AdDuplex’ core business by hiring 2x more people. Probably not even grow the business by 10%. The only way to achieve that would be expanding to different areas. But let’s leave talking about it for the next part.
Hiring is a very time-consuming process. At the same time, I feel like many huge corporations overdo the “process” part for no good reason and many startups try to emulate them blindly. I won’t talk about the hiring process for “business” roles as I clearly lack the firsthand knowledge in that, but hiring developers seemed quite straightforward to me.
We’d collect candidate resumes, filter out obvious non-matches, and then I’d set up a phone/skype interview. In the phone interview I’d go through the candidate’s experience and ask them to talk about the details and get a general feel of the person. Then I’d ask some set of standard questions that always seemed ridiculous to me (like “what are you good at” and “what are you not good at”) but surprisingly more often than not they quickly helped me understand whether the person is a match. For example, answering one of those questions one guy told me that he loses his temper when dealing with someone incompetent (from his perspective) and would start screaming and cursing them. Well, I thanked him for his candor, but passed on hiring him, as you can imagine :) After the remote interview, we would invite the remaining candidates for an in-person interview where we would ask them some technical questions to poke around their skills in the specific area, we were hiring them for. And that’s it. We never asked anyone to do some programming on a whiteboard or work on an unpaid “test project” or anything like that. And I don’t think we ever missed badly.
Lesson learned: Multi-level multi-day multi-hour interviews are some sort of HR circle jerk
Obviously, I don’t have enough experience to make general claims but through the whole AdDuplex experience we never made a terrible hiring mistake (like what those long hiring processes are supposed to avoid). Yes, some people left because they didn’t like the startup way (aka slightly chaotic) of doing things and yes, those personality traits could’ve been identified in the interview process. But I’d say that was a function of not asking some of the right questions or not having the skills to see some flags, and not because the interviews were too short or there were too few of them.
As the first time CEO I’ve read some books on hiring before embarking on the journey. I can’t remember all of them but the one that influenced me the most (if I remember correctly) was Who by Geoff Smart and Randy Street (referral link). It felt a little “too corporate” at times but had a lot of good practical advice and solid explanations of why it mattered.
A reasonable question would be why for such a global business with pretty much zero footprint in Lithuania we hired locally. Actually, in the early stages I tried to hire from my Microsoft-tech-affiliated global network but after some unsuccessful attempts I gave up. The weird thing with living in a what I jokingly call a “first and half world country” is that local salaries are notably but not order of magnitude below those in the US or Western Europe (and were more so in 2013) and, at the same time, are not so dramatically higher than in most of developing world (I mean for the skilled programmers). Therefore, unless you get to some serious scale the “savings” of going cheaper are not that substantial and going “west” would create some weird internal dynamic for questionable gains. It is a common refrain that you need Silicon Valley developers because only they have the skills to build something massive. Well, I guess this is partially true when Apple wants to hire someone to compete with Facebook. But somehow this myth is propagated down to 10-person startups and that’s where it’s a big pile of bs. Even taking our own specific situation, 200 meters from where I live stands the main development office of Adform — a company that built an ad serving infrastructure orders of magnitude larger than what AdDuplex ever did. So, there is no specific life-or-death need to go abroad for the top skill tech talent (I’m just talking about quality, not quantity here).
I’m not opposed to hiring developers remotely at all. But since I already had Paulius locally it made sense to continue hiring locally. I think either fully in-person or fully remote can work just fine. I don’t think hybrid (as in part of the team is local and part remote) is a viable option for a startup, though. It’s fine in a larger organization but when you are small it just adds the overhead you don’t need.
Lesson learned: Publicly raised VC money helps with hiring.
Your business may be doing great, but as a small team you will always face skepticism from prospective employees. Will this company go out of business tomorrow? Will I get paid on time? What if I want to get a mortgage soon, will the bank frown looking at my employer? Once you raise some institutional money and declare it publicly, it becomes much easier to overcome the skepticism. I was honestly able to tell all prospective employees that we have enough money to last 1.5 years even if we didn’t make any money, and we did. This was enough to go over that hurdle in most cases.
Business roles are another story, though. It would make total sense to hire a salesperson from the region you work in. Except AdDuplex was totally global with no massive outliers. Additionally, even if we prioritized some markets (the US would be a logical choice) we didn’t have enough scale (inventory) to start selling to coca-colas of the world. Also, Windows Phone was too “weird” of a niche to participate in any programmatic exchanges at the time (there were other reasons not to do it too). At the end of the day, I made a decision that we will only try to sell to other (bigger) Windows Phone app and game developers at first. So, I hired Dominykas — a young just-out-of-school guy with a great command of English and eager to break into the industry. Now that I think about it, he is probably the only person from AdDuplex who is continuing on the same path in the in-app advertising industry.
On the marketing side, at first, I kept the developer relations role for myself. It was clear that for this role we’d need someone from the community and going beyond me would require expanding the hiring geography and budget dramatically. And I actually wanted to do it myself in the early days. Later on, I decided that I’m overstretched, and we finally needed a native English speaker on the team. I tried to hire a few people directly but none of them were either interested in doing this full-time or were way outside of our reasonable budget. And, let’s be honest, “shilling” for an ad company, no matter how “developer friendly” it was, is a hard thing to sell to a developer. While in the process of looking for our DevRel person (well, at the time it was called that — the industry just transitioned from calling it “evangelism” to “advocacy”), I was contacted by a company that tried to provide this role as a service. They proposed that rather than hiring one person full-time we could collaborate and get three great people (whom I knew from before) on a part-time basis. Plus, a manager to run the whole thing. This sounded interesting and, to be honest, I didn’t have any other plausible prospects, so I said yes.
The thing with all such roles is that you desperately want to measure their efficacy and in reality, you just can’t. I’ve seen all kinds of organizations trying to impose various KPIs on their DevRel people and universally it either messes up their behavior due to misaligned incentives or doesn’t even measure anything meaningful. Usually both. I’ll talk about it looking from the other (customer) side below. I think in a larger organization you just decide that you want to do this, allocate a budget and hope that you’ve hired the right people who will indirectly help you grow. In a smaller one, the expense is so notable that it’s just psychologically difficult to justify it every month without being able to attribute some results to the activities directly.
Technology
As I mentioned in Part 1, I’ve started AdDuplex on a $8/month shared hosting account. Before we even raised any money we moved to Microsoft Azure (Windows Azure at the time). We were the first (or at the very least the first startup) Azure customers in Lithuania. We started using Azure even before it became officially available here. I still don’t get why big corporations geo-restrict some services that have nothing to do with geography. There’s still no Azure data center in Lithuania and I doubt there ever will be, but that’s how they roll. Azure was accessible in some preview form or something in 2012 and we decided to join.
The funny thing was that while Azure was in preview there was no Lithuania in the country dropdown on the signup page. Someone told me to pick some random EU country for now, and I chose Ireland. I figured the UI and invoices would be in English and the currency was Euro. Little did I know that changing a country field in a database is a gargantuan task (shrug) for Microsoft and we would struggle with it for several years. If I remember correctly, we ended up paying thousands in Irish VAT because of that. You are welcome, Ireland.
We happened to live in the golden age for cloud-based startups as both Microsoft and Amazon (and later Google) came up with all kinds of incentives for startups. We started with Microsoft BizSpark which gave us some minimal amount of Azure credits but then progressed to BizSpark Pro (or was it Plus?) that came with a very substantial allotment of free cloud services. In a way the “grant” in Azure credits was comparable to having an extra seed investor (with no equity). It came without any direct strings attached, on the one hand, but on the other we got quite locked in with Azure and ended up covering the initial “investment” Microsoft made over the later years.
There were a lot of organizational/structural changes at Microsoft at the time and Lithuania was moving from one cohort to the other in some regional rejuggles. With each such change we got to meet some new evangelists/technical salespeople. Naturally, they had their own goals and KPIs. As I mentioned above, it’s not that easy to define some reasonable KPIs for people who don’t sell anything directly. The people talking to startups, most of whom used Azure absolutely free, had “usage” as their KPI. In a way this was counterintuitive — why would you encourage startups to use the free products to the max? On second thought, though, it made some sense. In a completely rational world, the more you use now the more you are likely to pay once the free honeymoon is over. In practice though, we had a few weird conversations along the lines of “Why wouldn’t you keep your multi-gigabyte backup in the cloud storage? It’s free!” (with the underlying message being “so I can report more usage from you on my report card”). This was another data point in my conviction that setting KPIs for DevRel people was a utopia.
Generally speaking, though, that free ride enabled us to implement some advanced features and a more scalable architecture that worked well for us while things were going well businesswise. Having said that, it was also one of the reasons we had to eventually shutdown. I could’ve run AdDuplex as a “community service” indefinitely if it was on the same initial $8/month hosting. Or even an order of magnitude more expensive. Alas, the thing we developed over the years required infrastructure that cost orders (plural) of magnitude more to run.
Lesson learned: It’s easy to scale your infrastructure spending up, not so easy to scale it down.
In heavy load industries (such as ad-tech) adding some feature could mean expanding your infrastructure costs by orders of magnitude. It’s inevitable if you want to expand and grow your business. But since every move can potentially increase your costs dramatically, it shouldn’t be taken lightly. Some things are expensive to develop but are more or less one-time costs and you can sigh and write them off if they don’t work out. Others may not be that expensive to start with but will hound you for years to come. I’ve seen several startups basically destroyed by unsustainable cost structures. This happens on the business side more often, but it can happen on the technology side just as well.
Not exactly technical, but an adjacent story is with availability of all kinds of services. For us, primarily, it was payment processing. It is likely much better now, but back then Stripe, for example, wasn’t available here. Luckily, Braintree was, and they accepted us. If they didn’t it could’ve been way more complicated as apparently processing credit cards from worldwide customers for a non-tangible service was not something banks and payment processors were very keen on. It turned out well and Braintree served us well until the end. Having said that, even though we had a nearly flawless cooperation history, they rejected our next project. But that’s a story for another part.
Business
My initial hiring round was targeted at increasing the velocity and quality of feature development and incremental growth. Basically, doing the same thing we were doing before just faster. And this part worked out really well. We were growing steadily both in terms of apps on the network and advertisers. The challenge, however, was that our business was similar to marketplaces. We couldn’t sell what we didn’t have. You can sell ten copies of your app or a million copies of your app — the only bottleneck could be some backend you control. With a closed ad network your sales are limited to your inventory and the only way to increase your sales beyond some limit is to get more inventory. This was our main challenge throughout the most active phase of AdDuplex’ life. We had multiple potential customers ready to pay (at least proclaiming they are ready) tens of thousands of dollars a day but we couldn’t take that money as we didn’t have how to fulfill it.
Windows Phone was growing steadily. It wasn’t growing as fast as Microsoft and enthusiasts had hoped but it was growing, nevertheless. The low(ish) market share numbers meant only that Android and iOS were growing as well. So, until about 2015 this wasn’t a big concern. The problem was primarily in AdDuplex’ genesis story. It was created by an indie developer for indie developers. And that was the market where it resonated. While there were a few success stories in that world, most of apps on AdDuplex were in the long tail. That’s why I could easily claim that AdDuplex was the “widest” ad network on Windows, but not that it was the largest. One massive ad-supported game probably served more ads than thousands of indie apps on AdDuplex. And we didn’t have those. That’s why the next logical step on our path was attracting bigger apps and games.
It was clear that those won’t come on their own and can’t be incentivized by phone giveaways (something that worked quite nicely with the enthusiast crowd). We needed to do some old-school face-to-face “sales” (bizdev?) to attract the bigger fish. Being an introvert and dreading this kind of activity the most I was sure that it wasn’t me who would fit that role. And again, the challenge was that there was no specific region to focus on and to hire from. So once again I decided that since there was no predetermined geography we were going after, we should hire locally. At the same time, my primary contact point and our champion at local Microsoft wasn’t too happy with where his career at the mothership was going. And his job was literally going through technology businesses and convincing them to give Microsoft technologies a try (and/or help them succeed). It seemed like a perfect match. Matching Microsoft’s compensation package was a challenge though. But after some back-and-forth Tautvydas (aka Tony for international audiences) joined AdDuplex with a mandate to bring bigger apps and games on board.
Tony talked to potential customers and very soon it became apparent that our main value proposition didn’t connect with larger companies. For those reading this who are not aware what AdDuplex was exactly, our main product was a cross-promotion network where apps were showing ads for other apps and getting free advertising on the network in return. This proved to be attractive to solo entrepreneurs and tiny teams as the same person juggled growing the userbase and making money. The equation of exchanging some revenue today for a larger userbase and more revenue tomorrow made sense. But once we tried to move up the chain, the formula broke down.
Once you get to a level where a company has separate user acquisition and revenue generation roles it becomes a much harder sell. In a simplified form, you must convince the user acquisition person to go convince the revenue generation person to give up on the revenue so that they can acquire more users. But this is not how the companies are set up. The UA person has a budget and goes around looking for ways to deploy that budget in the most cost-efficient way. These were the people ready to pay us tens of thousands a day. The revenue person only cares about making as much money as possible from those users. These two roles seem like parts of the same unit to an outside observer. But in reality, more often than not, they don’t have much in common. The only connection is that the user acquisition person has to acquire users cheaper than what the revenue person is able to generate from them (lifetime value). And this is in a way a more fertile ground for finger-pointing rather than cooperation.
After half a year of fruitless attempts the message from the larger publishers was clear. They would be happy to pay us money to send potential customers their way. And they would consider running our ads for money. But they won’t connect the dots internally — this was our job.
Lesson learned: What works well for one customer segment may not work for the other.
Even when your potential customers produce roughly the same end-product their corporate structure (or lack of one) could mean very different demands on their suppliers.
Partnerships
Besides direct customers on both sides of the advertising equation, partners were an important part of our business. Primarily these were Microsoft and Nokia, but occasionally some other third parties interested in buying advertising not directly for themselves but as benefits for some of their projects. Microsoft would sometimes buy ads as perks for their developer marketing programs. Those deals were lucrative on one hand, as they would often come as relatively large chunks of cash, but on the other we had to fulfill those obligations and that at times clogged our inventory at unpredictable times, resulting in having nothing to sell to direct customers.
Overall, dealing with Microsoft (and some others) was the source of most business takeaways from my experience as a CEO. You quite quickly learn that there’s no such thing as “Microsoft”. The corporation could be a multi-trillion-dollar company but the team you are dealing with is often smaller than your startup and sits in a room with no windows (sic!) (true story). And it doesn’t have a budget you imagine when you hear the word Microsoft.
You can be BFF with some Windows Phone-related team but that doesn’t give you any points with, say, Azure team. Additionally, a very common career move in a corporation is to switch jobs internally. So, you build a relationship with some person and then July 1st comes (Microsoft’s fiscal year starts in July), and they suddenly move to a different team at Microsoft, and in an instant couldn’t care less about your Windows Phone thingy. It’s quite an introvert nightmare 😅 And I’m not even talking about people leaving Microsoft altogether.
Another frustrating aspect was a very normal but hard to accept fact that people working on “Windows Phone” (in a wider sense) were just going to their boring job every day, performing tasks they were given and going back home. When dealing with DevRel people and generally “faces” of some division or technology it’s easy to forget that most people working on these products are Dark Matter Developers. So, once you start working with some teams beyond the customer-facing conference circuit, you realize that you deal with people who have no idea about any events, trends, “micro celebrities”, or news in your common industry. They are just performing their tasks from the backlog to the best of their ability. And that’s normal. Just unexpected when you are in your own bubble. At the AdDuplex’ peak I’d guess 90%+ of indie Windows Phone developers and enthusiasts at the very least knew the name. But when talking with someone from Microsoft working on a Windows Phone adjacent team it was quite common to need to explain who you were and repeat the company name and what it does multiple times.
Overall, though, being an active participant in a small niche owned by a mega-corporation had its obvious benefits. We were featured in multiple global keynotes at the most important Microsoft conferences, invited to private meetings with VPs, etc. All of this would clearly count as credibility points if we were raising further rounds of financing. Except we weren’t.
Lesson learned: Dealing with corporations is a special art form.
Someone told me that when dealing with corporations you need to send a lot of written information, powerpoints, pdfs, etc. Because the person you’ve spent months communicating with will inevitably just switch jobs without notice. And at least there will be some “paper trail” for the next person to discover. I’m not sure it worked like that in my dealings with people at Microsoft, but it definitely wasn’t trivial and frustrating at times.
Conferences
One of our marketing strategies was going to conferences. We’ve tried it all. From just me speaking at conferences and user groups, to having a booth at Build (Microsoft’s premier dev conference), to going on generic startup conference tours with fellow Lithuanian entrepreneurs.
The startup conferences were great for building personal connections but unless you are raising funding or making a product for startups, not particularly useful for business.
Having a booth at a fairly well focused event was useful in isolation but when you consider the price of admission it was definitely a negative ROI in the short run. The long run is not easy to measure, and I think for most startups conference/expo booth is a hardly justifiable expense any way you look at it.
Speaking at conferences is the most effective (and cost effective) way. You need to have someone on your team willing and able to speak though. And you need to find subjects that are not directly related to your business but also interesting to the same audience as your potential customers. If you manage to have all that, it could be one of the important tools in your toolset. Besides talking to potential customers, being a conference speaker also puts you in a peer group with other speakers and expands your network. I’ve written a whole post about the benefits of speaking at conferences, so there is no point in repeating it here.
Ad Tech is Sleazy
I couldn’t figure out where to put this, but it’s something that needs to be said and something that influenced AdDuplex and me personally quite a bit. So, I’ll just cover it as a separate “essay.”
As anyone who has been on the internet professionally since the 90s, I witnessed the evolution of internet advertising firsthand. I never worked in the industry before AdDuplex though. And even AdDuplex wasn’t born from the ad-tech perspective. It was just a response to developer struggles with advertising among other things. But at the end of the day, we were an ad-tech company, whether we wanted to admit it or not. And I didn’t want to for a long time.
I have nothing against advertising per se. But in the hunt for efficiency the industry keeps crossing a moral red line after red line. The most intrusive ad-tech companies keep convincing advertisers that they can deliver the best bang for the advertising buck and consequently those advertisers demand the same or higher level of targeting and tracking from other ad-tech companies perpetuating the “abuse” cycle.
We had to tiptoe around that line throughout our existence. App/game advertisers demanded that we let them pay for installs (not impressions or clicks) or even further actions. Being able to do that on mobile requires some not-so-kosher tricks known as “fingerprinting”. And that is basically tracking users across apps using “unofficial” means and for no good practical reason. At some point we even gave in and developed our own fingerprinting system. But after some experiments I realized that I just don’t want to cross this line. So, we ended up telling advertisers to use third party services if they absolutely must attribute ad clicks to further actions, but we won’t participate directly.
Being a big fish in a small Windows pond shielded us from another disgusting trope — paying for access. On Android and iOS there were so many ad networks that savvy publishers pitted them against each other by requiring some guaranteed payouts or blatantly asking for prepayment for just adding the ad network’s SDK into the app. This was one of the reasons we never went to iOS/Android (more on that in the next part). Basically, a business model of a random ad network on iOS/Android was raising tens of millions of VC money, paying a bunch of publishers for “access”, and then hoping to deliver results so good that they turn down the next ad network with a “bribe”. We were lucky that the competition on Windows was really tiny and unfocused, and we never had to do or even talk about that.
A common refrain from companies sitting on a lot of data is that this data is one of their most valuable assets. And as ad-tech companies naturally accumulate a lot of data from all over the place it felt like this should be one of our valuable assets as well. I made some aggregate reports about the state of the Windows Phone and Windows ecosystems and those reports were covered everywhere across the world from specialized tech blogs to LA Times. After that we got some interest in our data as well. But after I told the enquirers that we don’t collect any meaningful personally identifiable data and agreed to only give access to totally anonymized datasets, the interest vanished.
When it comes to advertisers you naturally attract all kinds of scammers, malware peddlers, and other questionable characters. We had a few slips with total scams (employing tricks to pass our checks) but those were reported quickly, and we blocked them. A much harder challenge was dealing with all kinds of borderline ads — not exactly scams or malware but with some fishy smell. Luckily, as we were Windows Phone/Windows network we had Microsoft Store’s Terms of Service and Privacy Policy as our guide and also an excuse to boot a lot of questionable advertisers. We had a policy that you can only advertise Windows apps automatically through the self-service system and the rest required prior manual approval.
And if you think that bad actors are only on the big bad ad-tech or advertiser sides, I have news for you. The publishers (app developers in our case) are sometimes nasty as well. We operated in a tight and relatively small community and our main “currency” was ad impressions, not money, so that made us a less attractive target. But even we had some run ins with dishonest publishers who tried to artificially pump their numbers. Luckily, with some technical solutions and manual policing we managed to keep it under control. But from a wider industry I’ve heard multiple stories of some ad networks realizing that a substantial chunk of their inventory was fraudulent. And then you either shut down or continue to live in a lie.
It takes a particularly thick skin or a total lack of morals to exist in this space. As I’m neither particularly thick-skinned nor totally amoral, I chose a path of denial. I always [slightly delusionally] positioned AdDuplex as a developer tool and not an ad network. Was this to the detriment of the business? Maybe. I don’t know. It could’ve been what made AdDuplex last for 12 years instead of burning in flames in just a year or two. Was that a good thing? I don’t know either.
We played in a niche that failed to grow beyond being a niche, and our mandate was to grow faster. We had to do something new, and we will cover those attempts in the next part.
—
P.S.: with the announcement of AdDuplex shutting down and this blog post series I keep getting asked about what I’m up to these days. Check out marker.js if you need image annotation in your web app, and the upcoming JavaScript diagramming library.
