Note: This post was not reviewed/edited/written by any LLM/“AI”.

Prelude

Prior to joining Widgetic I was a Java/Android developer turned Node.js / backend developer, had worked at a Swiss cryptocurrency startup as a backend/desktop developer, and had built more than a few MVPs in the years prior(mobile apps, electron apps etc but not a full stack application). At this point I was relatively inexperienced with web technologies and wanted to work at a product company to understand how these skills/eager learning attitude translated into the real world, especially in the challenging world of business/startups. I coveted more learning and responsibility, so when an opportunity came along to work at Widgetic, I was more than happy jump on it!

I joined Widgetic in 2019 as a backend/devops developer in a team of 3 - the other two team members - an iOS turned frontend/javascript developer and a product manager. The team was fully remote with members in India(myself), Romania, and Spain. To protect their privacy, I have referred to my colleagues with their initials - AP(product manager) and AN.

Introduction

Widgetic emerged at at a web development agency in Iasi, Romania after the they received multiple requests by different clients for similar kinds of websites sections like image carousels, image hotspots, lightweight audio players, video players etc. They decided to call them widgets. Right from the get go, they understood that website builder platforms like Wix, Weebly, Shopify etc were open to third party “plugins” where the widgets could be sold for monthly/yearly subscriptions. With the semblance of a product emerging, the web agency was wound down and was left with 3 people - a product lead, a frontend developer and myself, a backend+devops engineer (at this point I only knew node.js / hapi.js / REST/ postgresql /linux and that was enough as far as requirements went for the role). Widgets were served as iframes to client websites and integrations with different website builders provided a way to customize the widgets and embed them directly onto the client platform generated websites via native/“raw” web builder interfaces.

Tech Debt: Origins and Lifecycle

image

Nothing can be said about Widgetic without talking about the technical debt. As alluded to earlier, the antecedent to Widgetic was this web agency that decided to abstract out the frequent requests for same sort of functional websites into pluggable “widgets”. Since this was a relatively successful agency, they had a lot of interns coming in; who were eventually assigned the work of building these widgets, as a way of contributing and learning new frameworks. The interns were apparently given developer freedom to choose a framework, and interestingly, eventually all the codebases started diverging on framework/ library choice with the prevailing technologies of the time(circa 2014 iirc), including-

  • coffeescript
  • backbone.js
  • spine.js
  • svelte 1,2
  • angular 1, 2
  • react

As the widgets started materializing the agency owners were quick to start integrating with website builder platforms. However, these integrations were developed haphazardly, to reduce the time to GTM(as I reflected on this later, this was great choice business wise because they reached ramen profitability quickly). There was no architectural planning - interns/employees were building integrations and a dedicated veteran of the agency was assigned to build the “GCD” of the widget server, a massive monolith. Now all of this is far from ideal world of careful planning, architecting, securing and provisioning but this wasn’t a complete disaster too - things worked gracefully most of the time(we didn’t really measure the uptime but there was no significant downtitme either). Also, a dedicated app was created with an editor to help non integration users to create and configure widget specifications directly and then these be embedded on any html website from the embed code generated there. So, the backend integrations were created to ease the distribution on the web builder / ecommerce platforms, but with different node.js based micro applications that would basically implement oauth and other necessary functionality to bring these widgets to sale. Here’s what the comprehensive technical stack comprised of when I joined-

  • node.js + express.js/ hapi.js/ koa.js
  • php+symfony
  • mongodb 2.4.x/sqlite
  • Vagrant based dev env
  • Varnish cache
  • digitalocean

Unfortunately, the lead developer who had created the monolith backend left the agency after creating the meat of the system with proper tests and some decoupled documentation. This is the exact point in time that I joined. I had no idea what I was signing up for, never even had heard the term techincal debt at that point because of relative inexperience(only ~2 years until now, very little exposure to production load bearing systems).

With the stage set for an epic showdown between developers and technical debt by the way of missing documentation, unpredictable implementations, stack fragmentation, we were ready to bring the annual revenue (already in high 5 digits) to the next meaningful milestone and not perish under the weight of instability and frequent crashes, which were a big big problem at the time and heavily detrimental to the our clients’ experience(mostly small business owners on the web). However, our clients loved us because the widgets were designed beautifully(by AP) and were highly customizable. At this point I think I should point out that the number of end users was in the mid 6 figures on account of compounding - our clients who installed widgets did so on their own portfolio / shop websites that were their storefronts, where thousands of their own customers would visit and make purchases.

Tech debt is a financial problem

The system was spread across multiple services across even older machines that hadn’t been updated in years. As mentioned previously, there were different frameworks, different versions, different languages using just a handful of common web technologies working in tandem to produce a working but unreliable system. An unexpected benefit of this mess was that the problem of triaging just wasn’t there- once you’re done with your assigned tickets, you can find something to work on easily. Although I was working in a hierarchy, and some issues were more pressing than the others, the number of issues(not catastrophic, most of them anti patterns, fragile code, incoherent architecture, the absence of an architecture) were too many. These, on top of the feature requests we would keep getting from our customers, needed to be triaged, and were slowly passed down to us, but eventually we did keep up.

This was also the time I had the “breakthrough idea” of “just rewrite everything” several times, but the finances just didn’t permit it - or maybe because I was a developer with limited production experience, I couldn’t rewrite the integrations quickly enough and still achieve our day to day targets. Just after a couple of months, I learned to tolerate the chaos and just make do with what I had. This was fun in its own way(just ignore the code that doesn’t work and test manually) for a while but becomes dull once you just have to keep doing this without a discernible change in system quality (of course you’re slowly imparting more stability to the system but the effects won’t be readily visible). For me this period lasted about 6 months, after which we achieved a baseline stability as opposed to the constant crashes that were normal before/ immediately after I joined. But that’s the thing about technical debt– it keeps piling up and you’re not gonna completely get rid of it all, ever, just that it’ll become manageable at some point.

Even if you don’t have technical debt yet, you still have code which will go through semantic drift and thus become ready to refactor. Managing expectations is prime, but it really doesn’t hurt when you’re making bank and the system isn’t breaking down frequently to the point that it’s unusable. At one point, you can start making enough revenue to hire further talent to take care of the technical debt for you and not go under. Until that time, one needs to learn to live with it.

There’s no such thing as too much documentation

Widgetic had a love/hate relationship with documentation. Although my colleagues relied heavily on documentation that was created by themselves to effectively discharge their own responsibilities, the codebases were glaringly missing any and all documentation, especially tests.

This is one of the first things I learned - documentation, even if little, is really useful in any form/shape. When I joined, we had (non code)documentation in several places - READMEs, dedicated doc repositories like notion/coda, API documentation etc.; all scattered without a structure, some of it served as a database of quick recipes for common problems around the architecture and how to avert them for the foreseeable future. But a lack of documentation also poses risks- you might not even know what’s missing from the system, especially one that’s been developed chaotically and without a process to reach the market quickly.

As a naive programmer, I had the tendency to avoid documenting. But as anyone with a semblance of a skill in programming will tell you, this is the exact thought process you need to avoid. So I eventually learned to like documenting after a few hard lessons, and the team was really helpful in this aspect - they documented everything vigorously despite the glaring lack in the codebases as mentioned previously. In the end, it left such a deep impact on me that I started thinking in documentation terms, and this would go on to help me enormously in my future projects(think of what you can do by over documenting if you plan to use high context length llms).

A lack of documentation makes a system unpredictable, and although all systems fail - good ones fail predictably.

I was amazed to experience that even a system as unstable as widgetic was when I joined can be handled pretty well, with all its bugs / quirks, by just writing about them to have a quick reference if things break / are close to breaking. Apparently, I switched polarities vis-a-vis documentation and started every mortal effort to document as much as possible, which I believe was one of the biggest improvements to my development workflow. Writing clarifies your mind, and eventually you start writing better code with a clearer head. Slowly I also realized the truth of another adage in the programming world-

Documentation is a love letter you write to your future self.

Small teams make big dents

After the success of Midjourney ($200 million revenue, 40 people team in 2023), Cursor ($100 million revenue, < 20 people team), it might seem obvious that hyper-productive small teams are the future of startups with their default lean mindset and lean execution. But back at Widgetic, we took this for granted. Small teams are efficient by definition - there is no communication overhead, no complex management required and they can run extremely lean financially(as we were doing at the time). When Covid hit in 2020- like everything else on the internet, the traffic to our clients’ websites increased organically, so much so that our revenue rose ~40% and we were overjoyed. While this was a remarkable milestone in itself, all of the work we had put in the previous months hardening and cleaning the architecture / infrastructure paid off as our system were able to handle the gradual spike in traffic smoothly and had baseline security measures at least(no one could hack us just for the heck of it).

At this point I find myself obliged to mention some of the “issues” we worked on that led to measurable results later, much to our delight-

  • Upgrade old ubuntu 14.04 running instances to the latest LTS versions
  • Tighten the firewall rules
  • Create VPCs for tighter security within attached resources
  • Pump in more logging so as to make the system more predictable
  • Improve and augment documentation all around
  • Move related services to a single VPS
  • Create and implement a firewall policy
  • Write tests wherever possible

Another unforeseen benefit of a small team - AP would set us developers to resolve customer support tickets which would put us in touch directly with our customers(not sure if this was deliberate or because he was shorthanded trying to actively raise funds / other things that PMs do) - and it was a uniquely beautiful experience. If nothing else, it reinforces your mission and provides meaning to the work you do on a daily basis since you can see it directly affecting someone’s livelihood.

Don’t fix something that’s not broken

So far I’ve discussed in length how fixing bugs and patching etc was such a fun experience for me, yet not everything was flowery with our development process. There were some lessons we learned the hard way, trying to replace software / versions which merited different approaches-

mongodb

When I inherited the system, it comprised of a mongodb database at version 2.4.x. This was way past its official support timeline but didn’t really matter to us because all the content we store on our clients’ behalf was public anyway- except there was no UI client that worked with this version that let us monitor the data. The REST API backend that was doing the heavy lifting was built with php / symfony which was documentation less and test less (there were tests sprinkled here and there but nothing that could act as a cohesive documentation with flow that could be useful to maintainers).

Long story short, we wasted a lot of time trying to migrate to mongo 4.x which was the supported version at the time, but we never could muster the confidence because of the lack of tests to prevent regression. Eventually we resorted to hacking together a really old version of robomongo to monitor our database. And it was the right decision all along, in retrospect.

vagrant

Docker had made footfall and we tried to replace vagrant with the same but with little to no success since nothing really worked and we did not have much time to spend on it anyway. In retrospect, we did waste precious time trying to stay up to date with the latest and greatest technologies - docker was noticeably faster than vagrant once setup (by definition) and that speed attracted us but this is the wrong kind of optimization, as we realized later.

Remote work works

One unexpected side effect of this strife with technical debt, and providing great customer experience and implementing customer requests was that the team was really efficient. Now it could be because the original founders were highly motivated and I have a passion for programming, with an obsession with learning by doing, we never felt like remote work could be a detriment to progress. And it wasn’t. We the developers, leveraged Zoom meetings to jump on calls and debug issues together and connect with each other when stuck. Pair programming on these calls, we would end up solving multiple issues in a few hours, although the number of issues seemed to be countless at the time. Of course doubts are thrown at the efficacy of remote work, but for us, having a distributed team working in 3 timezones, it was never a factor. There was usual friction in the team about outcomes, the direction etc which are the hallmarks of every team, but in hindsight, remote work never limited us.

Choose boring technology

At the time I had the opportunity to read about choosing boring technology. The article was a quick hit for me (it basically argues that you have but a finite number of innovation tokens and if you spend them all on the shiny new way of doing things, you’re left with very few to spend on the actual product that you’re building which might need those innovation tokens). This lesson hit home with me and taught me further invaluable lessons. In the same vein, the most stark and valuable lesson came from our push to expand Widgetic to WordPress. I had never had experience with complex software development lifecycles, and a decision was made to target WordPress to deliver our now (relatively stable) widgets, as opposed to other dedicated website builders like Squarespace etc which were more attuned to our product with their interactive editors (Wordpress with its far reaching audience seemed like the ideal choice for expansion) but there was an issue- they were slowly transitioning into js plugin based gutenberg editor which was far from stable at the time. Despite my (late) misgivings (which ultimately led to my resignation) it was decided to target the gutenberg editor at Wordpress to supply the widgets. This is a crucial business decision because of our scarce resources and need for growth for survival and possible fundraising. In the broader market, we had competitors who were doing much better revenue wise without a better product. A team of two engineers with a product that was a relatively complex potpourri of frontend and backend features, neither of the developers were ready/confident for a claim on taking full ownership of the same- I lacked experience on the frontend part of the stack(I had gravitated towards react by now as opposed to AN who was mostly a svelte developer), and AN didn’t have much experience with complex backend/full stack architectures.

These two seem to be complementary skills, and they were yet after some serious effort put into developing the wordpress integration, while maintaining bandwidth for obscure bugfixes / feature requests on old widgets with legacy codebases / almost eol integrations- we weren’t able to implement a reasonably good wordpress integration, even after a considerable span of time. Trying to get to the bottom of it - I think gutenberg kept breaking/APIs not stable, and sometimes our system would break, because under no circumstance would I assert that Widgetic was v1 by that time. This led to friction in the team- I was really excited to be working with react on this but was violating my own principle of not trusting pre v1.0 software (gutenberg at the time). The continued stress from this eventually broke the team after no meaningful result came to be after a long time. Only after a few months of hindsight did I realize that the problem was this obvious - the established players in the website builder markets with their stable and feature rich editors were more suited to our product than the relatively up and coming gutenberg, despite WordPress’s formidable reach.

Conclusion

It’s always sad to leave a team that you’ve worked closely with, solved hard problems with, and learned from, for years. But in my experience, I’ve found failure to be far more truthful than success, so I consider it a great privilege to have been able to collaborate so effectively on a shared vision with my equally if not more passionate counterparts.

My 2 years and 4 months at Widgetic were a rollercoaster ride with innumerable challenges, overcomes, and lessons that would have taken me way more time learn otherwise at a traditional corporation by all indications. I joined as a 2yoe developer and was eventually promoted to cofounder. I was constantly pushing multiple commits a day of quality code, bug fixes, patches, new features etc. In retrospect, it was a stepping stone for me to level up my skills by shipping consistently and solving a myriad of problems at each end of the stack. Since I learn by doing, I came into contact with some germinal web development concepts viz. oauth, iframes, postmessage api, document databases, managing and securing servers in production via practice and a feel for what a real world product looks like. Although I don’t place considerable weights on titles, some devs argue that you cannot be considered a senior developer without having worked on a legacy system, I definitely would recommend that every developer work on at least one legacy project, although not as extreme as Widgetic.

Postlogue

I wrote this in 2025 and it stayed in my drafts for a year before being published this year. A lot has changed in this time, but hopefully someone gleans something valuable out of my experience. Happy coding!