How overengineering wastes time and money

Suppose a startup with huge ambitions and lots of venture capital, or an established firm that fires up a new project that will solve everything. When project stars from ground zero, business and product people see themselves as the driver of a great product. They want to have every feature, and envision a great success the new software will bring when finished. Engineers think same. They see an opportunity to create the perfect system, an engineering marvel they will be proud of, that can be a highlight in their carreer.

The bigger the scale and budget, the greater the ambitions are. But with big scale come more difficulties to oversee the whole project, to keep processes well defined, engineering and costs under control.

Everything starts with a need

Product managers, designers, internal or external customers are the drivers of business decisions, let’s refer to them as the business. They bring the feature requests and requirements. These requirements are often ill-specified or unrealistic given the time, budget and engineering constraints. These overrequirements most of the time are taken granted by programmers, i.e. no room for negotiation.

The business usually have better skills to persuade engineers: we definitely need it, we are sure you can solve it, you are smart engineers, this is a key feature. To the contrary, engineers who are by average more introverted, are less likely to convincingly parry unrealistic requests.

Problem is, the business usually don’t know how difficult it is to implement something, programmers are complaining all the times anyways, how to take them seriously? The result is a roadmap with too many features. Features that are overly complex to design and implement, and turn out to be redundant because they fell victim of the perfectionism and idealistic view of business people.

These features will affect the engineering efforts greatly resulting in unnecessarily complex and overly engineered solutions.

In these situations it’s the engineer’s responsibility to make the business understand the costs: time, money, uncertainity, complexity. They must make suggestions how the technical solution could be simplified with certain changes and compromises.

Engineers are no better

The reason of an overengineered solutions is of course engineers themselves.

Overengineering not only happens when you give in to unrealistic requirements. But also when you — as an engineer — create solutions to problems that don’t yet exist. When you base decisions on assumptions and not measurements. When you spend 2–3 times more effort on a work for perceived benefits such as performance, quality, flexibility, reusability. When you create an overcomplicated system architecture. When you choose a complex technology that require greater operational maintenance effort than anticipated, while there is no in-depth know-how about that technology at your company. The increased implementation effort directly affects cost.

Decisions stay with us long, and an overengineered solution has higher maintenance cost.

Overengineering is when you choose higher complexity that is not justified by functional or non-functional requirements

How to spot overengineered work?

Look out for:

• Early and overly generalizations
• Premature optimizations
• Designing for too much flexibility and configuration
• Bleeding edge technologies

Let’s see some concrete technical examples:

You can feel that you are overdoing when you start to generalize before collecting experience with your system. Too much generalization can happen on code level, software and system architecture level.

When you create abstractions to your modules (class, package, python module etc.) without an existing use-case. When you implement YAML configuration to your software when a Java Properties file would be enough for your 2 configuration parameters.

When you decide all communication channels should use event sourcing and CQRS pattern because that’s the most state of the art solution. Plot twist: neither of them should be seen simply as “the best” architectural pattern that solves most of the problems. Applying proper event sourcing or CQRS requires significant planning, design and implementation efforts.

When you build cache layers without performance measurements and DB query optimization first.

When you decide to use a highly distributed NoSQL database before doing any capacity planning and performance comparison with replicated RDBMS setup. And you overlook the drawbacks of loosing referencial integrity, transactional integrity, and the implications of eventual consistency on your application in case of network partition etc.

When you decide to go reactive all the way on your backend without understanding its benefits and drawbacks.

The cost of complexity

The effects of an overengineering are manifold. It can be measured in longer time to market, greater headcount requirement, lack of know-how, need for more specialized knowledge and senior staff, reduced agility and flexibility. Besides the immediate costs, a suboptimal decision will stay long and will require more effort to maintain for years. Choosing more advanced technologies, techniques, concepts requires more in-depth and specialized knowledge.

Complex solutions are more prone to errors, stochastic behavior, loss of information (message driven systems), data consistency problems, reduced performance. I had a friend telling me that the project he worked on not long ago utilized Kafka, reactive patterns, event sourcing and what not. The architecture was overly complex. The volume of data or number of users did not justify these techniques however. It turned out to be good news, since the whole system had very bad performance, just enough to server their current user base.

I’m not saying don’t use leading-edge technology, I am saying is:

Always know what you are doing and why you are doing it!

Why software engineers overengineer?

The roots of overengineering is in formal education, vainness, perfectionism, lack of capacity planning and engineering disciplines.

Engineers are taught formal methods, prooving theorems in university. Surely, they are also taught practical knowledge in programming, but a Computer Science degree gives academic knowledge, with less emphasis on pragmatism.

CS undergrads learn about abstractions and reusability; to create flexible, extensible solutions because late changes in software will be more costly. Formal education, academic approach and software engineering principles dictate perfect and future proof solutions.

Perfect abstraction, reusablity, flexibility, SOLID principles, Don’t Repeat Yourself etc. These are some of the most important principles, but should not be the only drivers, because applying them mindlessly can result in a perfectionist approach.

People are selfish, engineers look for being involved in a technologically advanced project is one of the most rewarding thing for any engineer. The best system is cutting edge, is scalable, is generic enough to be forward compatible with new requirements. It’s fault tolerant. It can handle trillions of transactions per second. It’s losely coupled and event driven because that’s the shit nowadays. The best system is written in Golang, Elixir or Node.js because nothing scales only these.

Engineers want to keep their knowledge up to date. Acquiring new knowledge gives developers better market value. What’s better than being paid to learn new technologies on the job? There is a term Résumé Driven Development, but honestly, I think most of the time RDD is rather subconscious than a conscious decision. Engineers just want to feel good about their professional knowledge and get better.

For an engineer it’s vital to always push his/her abilities and constantly learn new things.

But personal development should not jeopardize a successful project!

What you need to do as an engineer

Let’s quote Wikipedia on Engineering:

If multiple solutions exist, engineers weigh each design choice based on their merit and choose the solution that best matches the requirements. The task of the engineer is to identify, understand, and interpret the constraints on a design in order to yield a successful result. It is generally insufficient to build a technically successful product, rather, it must also meet further requirements

First and foremost be realistic, be reasonable!

Don’t design for the far future! For sure, you should have an educated guess about what requirements will look like in the future and don’t make decisions against those odds. But don’t design for the unknown.

Be pragmatic! Don’t choose a more complex solution for the sake of trends.

You need to keep your eye on the goal and don’t get dragged away by perfectionism. It doesn’t make sense to spend days on coming up with the best abstraction for the in the beginning of a project. Don’t create a generic solution for a problem that will occur at 3 places in your code. There is little value in these.

Don’t be seduced by success stories! The problem with success stories is they are prone to cherry picking bias i.e. every positive trait is placed in the window but the shortcomings and winding path of achievement are unspoken.

Rome wasn’t built in day. Chances are you are not Facebook, Amazon, Netflix or Google. Don’t try to mimic them. Likely you or your company do not have that kind of resources. But even for them it took years and billions of dollars to get where they are.

Consider return over investment as an engineer. Your investment is engineering time, the return is a completed product.

Do capacity planning before you go for the “most scalable” solution. How many business transactions per second will you have? How many DB reads and writes do they require? Is your application read- or write-heavy? Which technologies are capable to fulfill these requirements? Do your business requirements allow eventual consistency? You will be surprised what a replicated MySQL cluster and a vertically scaled Java backend can handle.

You don’t need to scale to 10 Billion users before your product is on the market. Once you have that problem, you will have the resources to solve that performance issue just as the FANG companies did.

Do incremental changes and stay agile! Despite what the books say about cost of change in software is getting higher over the course of development, software is still the easiest to adapt compared to other industries where the product is physical. This concept was tied with the waterfall model.

As an engineer, say NO to unrealistic business requirements but always offer a technically more viable solution.

Do measurements, proof of concepts and performance tests before committing to a technology.

Use simpler architecture and battle-tested tools wherever you can.

Conclusion

I don’t want to imply that everyone should get back to a LAMP ecosystem, and burn technology books. Learning and continuously adopting new technologies in software industry is a key trait as an engineer.

However as a professional engineer, it’s your responsibility to not only know the benefits but also the costs of a choice. Make a decision in the full understanding of those costs. If you use a technology that has a 500 page book about, chances are, you will not embrace it fully within a week reading 3 tutorials on the internet.

Always know what you are doing and why you are doing it! Measure, evaluate and assess. Always stack the cons against the pros and treat them with equal weight before committing to a decision!