On the Occasional Misdiagnosis of "Not Invented Here Syndrome"

Not invented here syndrome (NIHS) is a slightly tongue-in-cheek name for the tendency of both individual developers and entire organizations to reject suitable external solutions to software development problems in favor of internally developed solutions.

This phrase came up in a discussion I was a part of recently, and I wanted to speak to it. Whether to build something yourself or choose an already-built solution is one of the most vital discussions we can have, so more perspective is always useful. This phrase is a great shorthand for the kind of bias we might have in software development, where we arbitrarily trust our own solution without taking into account the cost and pain of development, and more importantly, maintenance.

Even thought this is a valid concept, it's also important to not give this idea too much extra weight. A similar concept is "don't reinvent the wheel". These might be somewhat interchangeable. I want to examine where these ideas should and should not be applied. Your mileage may vary on all of these ideas.

Where this phrase applies best

Fundamental low-level software, like databases and frameworks should generally not be re-invented without a painful need and a lot of thoughtful consideration. At DEV some examples are PostgreSQL, Ruby on Rails, Preact, Redis, etc. Other utility libraries probably are safe to not be re-invented, like HTTParty which provides a clean interface over the core HTTP functionality provided by Ruby. Sometimes new things need to be invented along these lines, but it's not always a good idea.

Many of our core external services, such as logging, payments, monitoring, should probably also not be reinvented. There is so much complexity, infrastructure, and perhaps regulation, that reinvention seems incredibly unwise.

Where this phrase doesn't apply as well

"Duplication is far cheaper than the wrong abstraction" - Sandi Metz

This is a fabulous quote and blog post which applies really well here. This debate is always about whether the topic at hand is the right abstraction.

The aforementioned PostgreSQL is simultaneously robust and incredible software, while also being the wrong abstraction for a lot of types of data. That is why other databases get invented. They, too, are leaky abstractions, as is all software. We accept this abstraction because the pain of building a concrete solution is obviously a bad idea. Natural selection among those who do and do not attempt this is probably enough to weed out this notion from our debates.

When the topic at hand is "business logic", e.g. the type of code we write to solve user problems, and deliver on the ultimate purpose of the project, "inventing here" is a really good idea a lot of the time. And when going for an external solution, having the idea in mind that you may need to migrate off of it is always worth keeping in mind. The right abstractions are incredibly hard to come by. At DEV we use plenty of libraries "not invented here", and there are always limitations that beg the question of whether we should have written the logic ourselves.

A couple of libraries, acts-as-taggable-on and acts_as_follower handle a lot of the logic of tags and followers. It wouldn't be practical to migrate away from these any time soon, but they provide us with opinions which are sometimes just different enough that we may have been better off writing these concepts into the app ourselves. Tagging and following are higher abstractions which do not rise to the complexity of databases and frameworks. Since these are areas we care about in terms of user experience, I've since adopted more of a "Yes, invent here!" approach even when the current functionality we are looking for is available in a library.

Anything related to our own team's effective workflow, especially in terms of moderation and administration, typically deserves to be written by our team and community. The way we interact with the software, especially anything with possibly destructive consequences, is begging to be written in-house in our world. It helps that we are a technology-driven organization with solid funding and an open source community to boot.

In a few cases I've copied and pasted library code into a project instead of including a package in order to adopt some of the functionality now, but also allow for clean internal hackability when it inevitably becomes needed. This is not "clean" if done for the wrong reasons, but it's not inherently dirty either.

NPM & left-pad: Have We Forgotten How To Program?

This is another article that has stuck with me. Perhaps because I personally suffered the pain of "Left Pad" A day of reconning for the JavaScript community.

The article takes on a lot of the same ideas I've outlined here, more specifically in reference to "npm culture" which is known for taking this idea to extremes.

Take on a dependency for any complex functionality that would take a lot of time, money, and/or debugging to write yourself. Things like a database access layer (ORM) or caching client should be dependencies because theyre complicated and the risk of the dependency is well worth the savings and efficiency.

But, for the love of all that is programming, write your own bloody basic programming functions.

In conclusion

"Not invented here syndrome" is real, but it's easy to swing the pendulum too far. Technology-driven companies should do a lot of invention, even when it feels like reinvention. It's what we do. These named patterns we identify with should always be the start of a conversation or debate, not the thing that ends the debate.

A lot of the specifics of my argument relate to our own organizational circumstance, but just like the phrase itself, there are broad lessons that can be applied. Individual decisions are always about resource constraints and culture, and your organization may have different ones. Google, famously, invents a lot of stuff in house, and always has. Their decision-making process is unfathomably different than ours, but the in-house invention is as much a cultural thing as it is a resource issue for them as far as I can tell.

Please let me know what you think in the comments!