IT Meritocracy. Part 7: Why Don’t Our Bosses Just Let Us Do Our Job?

Someone asked me an interesting question on Quora.

As a software developer, is it normal to always feel that you are unable to do your best work, due to budgetary concerns upstream?

Not sure which IT atrocity frustrated him. Was he rushed by his non-technical boss, clueless about programming intricacies behind a seemingly simple UI screen? Only programmers are aware of tricky API calls, clever database query optimization, relentless hacker proofing, and other security measures. Was he treated like a mindless assembly line worker amidst attempts to solve complex problems via brute force? Our bosses hire cheap code monkeys instead of giving one enough time to eliminate the issue for good with the right technology.

Why don’t they leave us alone and let us do our job?

There is no black and white answer. I am my own (worst) boss now. The question simply transformed into:

Am I balancing my perfectionist urges and the business needs?

No clueless non-technical bosses to blame anymore. It’s just me.

Top (Generalist) Developers are Already Managers

The solution appears to be simple on the surface. “Management” is mostly planning, including constant adjustments depending on the progress, changing situation/goals, and unforeseen circumstances. Not only programmers with their superior analytical skills are well suited to plan the project, anything technical can only be planned and governed by a (technical) expert in that field.

The same applies to the so called “enterprise architecture”. Without being in the trenches and actually using the chosen tool: framework or API no one can make a 100% educated decision how to do it and whether to use something else instead. The lowest code monkey is de facto THE architect making all technical decisions. Otherwise it’d take forever to go back and forth, communicating any issue to the high-level “architect” and waiting for his/her answer. A typical non-programmer architect cannot know all technical intricacies discovered only after the coding starts.

The same is true for business analysis (“product management” or whatever your “IT organization” calls it). The lowest code monkey has no choice, but to learn the domain better than any user, becoming THE subject matter expert and making all decisions about the feasibility of product features and acceptable workarounds. Otherwise… yes, it is the same wasteful back and forth commotion with the “upstream”.

I’ve described the most efficient way: every generalist programmer being the project owner. Meaning an architect, SME, and of course, manager. The idea is hardly new. Every developer hates Scrum today, but how many know, that it was initially invented by developers (not brain-dead PMI-certified project managers) as a process for self-managed software engineer teams. Management is not rocket science. Can a developer draw Gantt diagrams? Can a developer call emergency meetings to brainstorm some problem and “delegate”? Being an active (technical) participant instead of a formal facilitator and mediator.

Corporate politics aside, there are two reasons developers don’t manage projects — outside Google. First, all smart ones, capable of being generalists, went to Googles: the handful of top consumer tech employers, that respects programmers and pays them well: 2–3x the IT average. Second, the man-hour centric Great IT Consulting Food Chain that’s dominated the industry (outside Googles) for the last two decades, is not keen on cutting headcount by programmers taking on more responsibilities and being more efficient.

I don’t want to keep beating the dead horse. I’ve talked about it on Quora and in this (Meritocracy) series.

Are We Hopeless Perfectionists?

A more interesting topic is what you do after you gain the authority. We, engineers, are very logical people. But we are also borderline autistic perfectionists unable to stop “in the middle of something”. Do our bosses have any ground to complain about it in the context of priorities and deadlines? Everyone knows famous examples of this personality. Steve Jobs wasn’t always right. So, how do we, prudent and logical professionals, keep it in check?

Here’s an example:

I’ve just finished coding a massive billing sub-framework, which is going to back all of our future SaaS products. Wish me luck with debugging. It’s 100% business logic (explained here). It is so massive, I could have made it a standalone service with an API, etc. hosted it on GitHub, and hyped for investors to notice me — aiming to slot between Stripe/PayPal and the app layer, since both only offer primitive subscription logic.

Real SaaS billing is impossible on its own (or tucked onto credit card gateways like Stripe) — w/o connections to the internal catalog of the offered products and services (in-app purchases) , usage limits e.g. max number of texts and emails plus abuse (DDoS if you will) protection against incoming messages, and many other things one can charge money for. There are different recurring subscription scenarios, different billing rates i.e. flat vs. up to N resources e.g. users, and true per-resource (per-user) models, plus different payment options: credit card v.s invoicing and then collecting/applying the payment. Then come exceptions like payment failures, manual refunds, and credits/adjustments.

I really, really wanted to avoid reinventing the wheel. I researched it. Nothing like that exists e.g. under QuickBooks or Stripe umbrella. No independent open-source offerings. We, programmers, tend to develop technical stuff like database admin tools, message queues, and logging frameworks — the majority of stuff developers apply to accelerators like 500 Startups with. A techie would wait for some “business person” (his/her boss) to work on some billing idea.

Here’s the kicker. Did anyone ask for better billing? No. It’s not an essential feature to sell the product. But it needs to work. It needs to be part of the foundation. It’s not easy watching our cash reserves getting dangerously low, while we are racing against the clock to release the new product, 100% our own and free of the prior mistakes (doing one-off projects for “people with ideas”, but that’s a different conversation). Still, I decided to take two weeks out of my schedule to implement the damn billing once and for all.

Was it the right call? We outgrew our initially simplistic monthly and annual subscription options (just two credit card rates as you can imagine). The biggest two requirements were per-user charges, and resource (text and email) usage, since we cannot offer mass-texting and mass-mailing for free, being charged per text and email, including received ones.

BTW, that: abuse-protected two-way text and email marketing could be a SaaS or API of its own, like MailChimp. Instead we have our eyes on the bigger prize: complete automation of several niches: auto sales, medical offices, retail insurance agencies, recruiting, and several more. We see billing, marketing, reports, and everything else as internal modules instead of standalone products — so our customers don’t need to copy-paste between some form-filling Zoho app and MailChimp.

Yes, I know, “integration”. That’s more tedious custom work over time vs. coding something from scratch once in the fire and forget manner. Don;t forget the awkwardness of asking the user to click on various checkboxes and enter parameters to “integrate“ with another system. We eliminate tens if not hundreds of checkbox-filled configuration screens in our systems. Why? Because I can. Sorry for offending anyone, I am not a Photoshop specialist just taken a JavaScript 101 class. With MS of CS and 25 years of programming, I feel I can take on something bigger than an assorted collection of widgets and plugins. 99% of world’s websites may be implemented in WordPress with various PHP plugins, but it doesn’t mean someone who leverages 100% of Java, Groovy, and JavaScript, has to follow that path.

I do respect playing by the rules: either offering WordPress, Drupal, Sharepoint and other freelancers “plugins” and APIs to do something, or being those WordPress developers to assemble “websites” out of little third-party pieces of reporting, marketing, analytics, and eCommerce nature. I use plenty of third-party APIs e.g. I could embed an open-source mail server like Apache James, but decided to use Amazon SES instead. It’s not about coding everything from scratch. It’s about having a choice between APIs and your own precise solutions. I do. Others don’t. I doubt any of my former MBA bosses fixated on “buy” over “build” would have understood that 100% educated choice.

Isn’t it ironic, how the people formally closer to the “business” cannot comprehend complex business logic at all, unlike us, engineers? Billing looks simple to them.

“Just charge a monthly fee.”
“No problem.”
“And charge it per user.”
“Wait a second. The client’s admin can add a user in the middle of the billing term (month). Do we charge prorated?”
“Uhm… Well… Just figure it out… Oh, almost forgot: I want to offer an annual discount.”
“But how do we know how many users they are going to add and delete during that year?”
“Figure it out, damn it! You are our second most expensive developer, you are certainly capable of that.”

Sorry, couldn’t resist recalling all of “we are paying you so much” ($5K over $10K lower than the average salary) I heard over the years.

Of course there are deadlines. Of course no one anticipated a seemingly simple requirement like per-user charges to throw a monkey wrench into the already designed/estimated or even built billing subsystem, which now needs to be completely redone.

Is it the “expensive” developer’s concern? Or one should throw the sh_t over the fence to the business analyst and wait for him/her to figure it out? That’s how it happens in outsourced IT. A BA would do his/her best, figuring out 90% of the stuff. The programmer will start coding, hits some obstacle (the unclear 10%) and throw it back over the fence again.

For anyone still wondering where my claims of 100x efficiency come from, one can save a lot by not doing that, since after the conversation above the generalist developer is the ultimate expert on billing.

How quickly one can implement the billing requirements I described? I can tell how long it took me: a week just to think it through. Maybe I am slow and not scientifically inclined, thus justifiably un-hirable at Google and Amazon with their real time algorithmic exercises. I don’t think anyone can figure out such flexible billing solution on the fly or over some hour-long brainstorming meeting. It wasn’t possible for me after a single shower epiphany.

I thought I figured it out, started coding, hit an obstacle e.g. discovered that breaking a month into days (daily charges) to account for users (and other resources) added/deleted daily required switching to the 100% account replenishment model vs. the traditional periodic e.g. monthly charges understood by users. Sure, a user would understand a detailed invoice explaining why his/her charges are higher or lower that month, but adding another variable: having those replenishment charges happen when the balance comes to zero — independently of the billing date? Maybe you can figure out, how to charge monthly e.g. on the first. I couldn’t — in a configurable (through FP expressions calculating everything based on the time elapsed, resource usage, and many other variables) universal manner, since I intend to use that billing module for all our SaaS projects.

No big deal? The user wondering why his/her credit card is charged in an unpredictable manner? Not your problem? It will be after that user calls the support and you are assigned a bug ticket. If I had 500 code monkeys to assign those, sure… Though they’d never comprehend the complex billing code.

I don’t want to keep boring you with details. The point I am trying to make, is half of real-world automation problems require several nights of “sleeping on it” and several shower breakthroughs to come up with an acceptable solution. Is it perfectionism? Or necessity? What’s the alternative? Drawing a line and going forward with the questionable solutions to later mitigate it by ugly workarounds, until everything falls apart at the worst possible moment?

My Favorite Fowler’s Anti-Pattern: Speculative Generality

My second favorite anti-pattern in the classic Martin Fowler’s “Refactoring” book is Shotgun Surgery if you are wondering. I’ve learned both lessons the hard way early in my career. But with all due respect, if after several projects under you belt (15 just $20M+ ones in my case) you just know something would come back to bite you in the ass, are you supposed to overlook it because the client didn’t explicitly ask for it and your boss hasn’t allocated a ticket for that feature? Write a little TODO just in case? Or solve it in the fire and forget manner?

Clients rarely worry about billing. Non-technical bosses don’t have a clue either. Everything: from security breeches and DDoS attacks to billing errors comes unexpectedly for them. No one knows internal logic like that exists. Customers buy the system for something else. Things like billing are not immediately visible to the user. When they work. Like your kidneys or liver. Billing just “happens” somewhere in the back end. Now, start having problems with your kidneys, and you’ll instantly find out their exact location in your body.

Am I too defensive thinking of the worst possible scenarios e.g. some hackers flooding my apps with emails I have to pay for whether I process them or not? Am I too defensive assuming, that every line of client-side JavaScript code will be tampered with, so I have to check the permissions (e.g. to ensure some disabled button’s action still cannot be invoked) again, after the request comes to the back end? I don’t want to custom-code it for every button/action. Am I too much of a perfectionist for taking a bit of extra time and generalizing it into a framework? Spring Security and other generic frameworks are useless against it.

The level of abstraction depends, like everything else. I don’t write one-implementation interfaces and factories. All of the frameworks inside Px100 platform are reused several times. E.g. our document-generating module is used for reports, HTTP communication with external services (mini-ESB if you will), and email templates, which in turn are used not only to send every email in the system, but also for DIY design of text and email campaigns and even consumer-facing HTML pages. It took time to do it right — once, to easily extend the templating functionality w/o worrying of access/security, HTML/PDF conversion, and other intricacies.

I don’t have a boss now, but none I used to have knew about my efforts to make something orthogonal and reusable. So what is the right way of developing a fairly complex system? Is it thinking ahead, no one gave you enough time for, or banging on the keyboard to say “oops” later? Scratching your head and thinking how to fix the issue without rewriting half of the system, which wasn’t designed for a particular situation.

Ever Been Asked About Your Weaknesses?

Everyone’s been asked classic “strengths” and “weaknesses” interview questions at least once. Most mumble something moronic like being a workaholic, answering the second one. Let me give you mine. It’s a real weakness. Whenever it didn’t get me fired, it moved me to the top of the recession layoff list.

I cannot “take it easy” when I know about a future problem. I cannot shrug my shoulders and say “dunno”, “weird”, or “how did it happen?” when I f_cking know what the problem is.

That means I automatically challenge others who make those mistakes, particularly insecure bosses. I lost count of interviews I failed because of that. I did eventually learn to keep the low laid-back profile and say those “dunnos”.

If you think about the hiring context, especially for a senior developer, brought in as a $5K-“overpriced” messiah, the situation is always bad: hourly server crashes and other fires to put out. Someone, most likely the second in command interviewing you is responsible for that mess, so when you answer his/her questions how to solve one of the critical issues, you automatically insult him/her by describing his/her mistake and offering a smart solution your future boss couldn’t figure out.

Not everyone is insecure like that. It’s just, that people who aren’t, also tend to be smarter in general and avoid severe mistakes. They don’t need messiahs. A more interesting question is who hired that bad second in command. That’s the ultimate “upstream”: the root of the problem.

It’s an interesting chicken and egg problem. The majority of first in command: “people with ideas” (non-technical founders) and corporate IT decision makers have no clue about the programming complexity. Many times a programmer doesn’t at first either, but at least we anticipate complications and are prepared for them.

For the non-technical “leadership” though… any in-house system is a “website”. Or mysterious $100M intergalactic starship bought from IBM or Oracle. Either way, being non-technical themselves, they hire equally technically incompetent second in commands: current or ex-freelancers in the former case, and dignified, but still non-programmer “enterprise architects” in the latter. Neither would let you take his/her seat.

I used to naively expect, that a year or two later, when the problems start, and I am sought as a messiah to raise their sinking ship out of the water, the founder would realize his/her initial mistake: of not hiring someone like me two years ago. I hoped he/she would at least ask me to rewrite the system right. In the second in command (CTO, etc.) capacity or with a “dev” title, doesn’t matter.

No such luck. First, I thought it was the second in command defensiveness, that blocked me. Then I got angry about the first in command disrespecting engineers. It’s simpler than that. We don’t exist for them — to respect or abuse us. They truly have no idea about the technical part. They truly think they can develop an ERP in WordPress (Access, Sharepoint, whatever) or buy an “almost-turnkey” one from Oracle. And when you come to one of them describing some complex problem, that doesn’t have an existing solution among WordPress plugins or can be ought from Oracle, they get angry, that something seemingly simple cannot be done in with a few clicks.

More Examples

Some of those problems are only known to us, engineers. A couple of examples, as always.

I’ve spent some time in the auto industry, as well, as auto insurance. What can be simpler than the three classic drop-downs to select the year, make, and model?

“Doesn’t like everyone know that Tesla wasn’t available in 2005, and Model 3 wasn’t available in 2015?”

Yes, there are APIs for that. I used Edmunds at the time. Our bosses know nothing about the source of data. How much time you think one would allocate to develop a page with those three drop-downs?

Let’s make it more interesting. How about displaying car pictures? Everything related to cars (and Internet in general) should be a visual experience. Edmunds API only gives you the make, model, and trim information. It has some collection of car pictures, but good ones come from a different, unrelated source. I am talking about all trims and colors. Since there are tons of freelancer-built car sites with generic pictures, one needs to build some poor man’s car configurator similar to (expensive) manufacturer websites to stand out.

Not only we needed to show the right trim instead of a generic picture, but also the right color. When I am shopping for a Porsche, click on GT2, and then select bright yellow, I wants to see it in all its club racing glory — instead of a generic picture of a silver base Carrera. That’s 40K+ images per model year for everything sold in the US if you are wondering. Luckily there was a company: Evox, that did just that: photographed most car trims from different angles and then recolor them in Photoshop. We could either buy that monstrous database from Evox every few months, or use an API developed on top of that by FUEL. Obviously I chose the latter.

Predictably FUEL didn’t do a good job matching pictures to “standard” (from the Edmund’s point of view) makes, models, and trims, let alone manufacturer weird color names e.g. “Polished Pewter” and alphanumeric codes.

First, models and trims are not standard throughout the industry . I learned it, when I worked at American Toyota HQ. Toyota considered different-looking Camry and Solara one model, since they were identical mechanically. Though it differentiated between same-body Camry Hybrid and regular one. Car buyers’ brains are wired differently. They look at the car shape, not what’s under the hood.

Second, the photographers didn’t do a good job to describe the car. The API had entries like “MINI Convertible” (Convertible being the “model”) and just “MINI”.

Matching between Edmunds and FUEL as is worked less than 50%. Trying to avoid unnecessary work, I first asked FUEL developers to implement a more robust logic. Predictably I got “what you want from us” attitude. They knew about the problem. It wasn’t fixable with SQL. Guess, Machine Learning wasn’t popular (VC-funded) at the time. I also suspect a lot of claimed AI “algorithms” todays are normal unscientific ones I solved that problem with. No ML required.

I spent a week and created a “human learning” UI to enter and tweak Groovy rules to recursively match two datasets: first by year and make, then by model, then by trim. Going back and retrying another branch if it reached the dead end. Taking into the account the body type, etc. Color-matching was last.

All three: colors, trims, and even models had gaps on both sides. My goal was to find the closest match e.g. to the “Sapphire Blue (color code AYR2) 2015 Audi S4 Prestige with Black Optics package and 19” Star rims”. Don’t be a smart-ass and google it. I made it up to give you an idea, what defines a car visually. And no, you cannot partially match VINs. If by any chance Evox photographers shot the car exactly configured like that, I had to find it. If I couldn’t, in the worst case I had to display a generic white or silver A4 from 2013–2015, since it had the same body. Unable to find that, I’d look among 2009–2012 A4 and S4 models with a marginally different (pre 2013 facelift) look. How did I know about facelifts and bodystyles: B8 and B8.5 in A4’s case. Every manufacturer codes them differently. I had to enter a Groovy expression to do extract that info from the Edmunds data set.

Maybe I am being naive and it is an ML rather than regular FP problem. Maybe it wasn’t a programmer’s problem at all, and I could simply send the massive spreadsheet along with the zip file of pics to India or Philippines for a 100 people to dig in that mess for a month. A non-programmer would have done it.

I am a programmer, so I solved it, like any programmer would. Alone. In a week. The semi-manual UI I created allowed me to run the matching, see what didn’t match or matched incorrectly i.e. Porsche 911 against Panamera, correct it by tweaking the Groovy expressions, and run again. The match ratio came to 96% at the end. I tracked it.

Now, was it related to the task at hand — leasing cars online via a virtual showroom. No. So how long is acceptable to spend on a doing somebody else’s poorly done job? I can find some relief in the fact, that most of my “sleep on it” multi-day design of billing, car picture matching, and other invisible to the user pieces wasn’t deadline-blowing additional programming. It was business analysis, performed by someone else in a traditional “IT organization”, so overall I did save (my own company) money.

The Higher Gear Again

Sorry for overusing the “higher gear” metaphor. It’s basically the question of one programmer making an effort to solve the 100% of the problem through technology — for good. Vs. hiring a 100 code monkeys to solve 70% or less with brute force — again and again.

Programmers are perfectionists. We want the 100% solution, because we know the user will ask for those 30% sooner or later. Our bosses see it as a problem. They are not OK with one “genius” becoming the bottleneck and breaking the schedule when a complex issue appears.

It all depends on the budget anyway. I didn’t have a choice due to no funding. At the time I was doing that car leasing system, I discovered a local competitor: a funded startup with five Harvard-educated execs and 15 or so typical narrowly specialized developers. They couldn’t figure out FUEL to Edmunds matching and used the limited collection of Edmunds images. But they were funded, and I am sure the CTO received a nice salary.

You are probably wondering who succeeded. Neither of us. My cofounder’s idea didn’t work and he disappeared. I also cannot find that startup anymore. I met a different cofounder and pivoted. This time my partners knows whom he’d sell our product and how vs. the previous guy’s slide deck. It’s not cars. But if I am to do anything in that niche again, I know how to stand up alone to my funded competition plus at least another company — the one that offered that picture API.

The question is, from the meritocracy point of view, how much that ability is worth. I am three months from finding it out.