Refactoring is a necessary thing. The reasons are numerous, but most of them have one thing in common: they are underestimated. As the old adage says: "the devil lies in the details". And typically these "details" surface during refactoring itself, leading to even more stuff to refactor, sometimes creeping up to dangerous levels.
This is a tale of such refactoring for Passwordless.ID, or rather how a large refactoring was aborded in the middle in favour of an intermediate solution.
Well, it turns out the refactoring was closer to a whole rewrite. But before delving into the details, let's check the "why" it was done, since the goals are what drive such refactoring.
Why?
The big refactoring started to take place, mainly driven by a follow-up of UI, API, DB: pushing "three-tier architecture" too far?.
Before, front-end and back-end were in two separate repositories, with their own lifecycle. However, from a workflow perspective, having both the API and UI in the same repository is more convenient. Changes always go hand-in-hand, you have a single URL for previews, no CORS necessary, a single pipeline and deployment, etc. The code is almost the same, it's just more convenient to work with a single repository.
The second goal was improving user experience, and specifically reducing "time-to-rendering" as much as possible. Currently, this was slowed down by:
130kb assets loading due to frontend framework (not that much, but way larger than it could be)
"Waterfall loading" (load initial assets, run script, fetch dynamic content, load other stuff, render it all ...duh, feels sluggish)
Cross-origin requests (adds one more "preflight" HTTP round-trip)
The combination of these things resulted in rendering times above a second when not cached, sometimes even two or three if the CDN caches also missed. In other words, it was slow despite being a super simple page.
Lastly, some structural changes to improve security and privacy would require changes on the data level. This includes changing the primary ID from the username to a hash and another round of encryption for the stored data. This would break the compatibility with existing data. Please note though that the privacy and security of the existing version is by no means compromised. This is about being paranoid and security/privacy in-depth by adding one more protection layer.
How?
There were three main areas related to the refactoring.
Combining both codebases with same domain previews
Making the UI more lightweight
Adapting the data layer
These are just a few sentences but represent lots of work. In particular, the UI refactoring. Making this more lightweight is easier said than done. Investigating alternatives and shifting away from the larger Vue framework turned out to be quite the burden.
The mistake made
We proceeded as follows
Merge both codebases
Refactor the backend code while we are at it 🙄
Investigate frontend framework alternatives 😅
Try out various frontend techs😬
Start porting UI🥵
What started as a refactoring was turning more and more in the direction of a full rewrite.
The refactoring of the backend code was a bit time-consuming but worth it IMHO. It's slightly more lightweight and forcing a proper file structure is a good thing. It makes the code more organized and neater. You can directly pinpoint API endpoints to source code files and the related middleware without even looking at the code.
On the other hand, exploring the vast space of technologies available to replace the front-end was a bottomless pit. Even the first steps of refactoring the UI became a huge time sink. It's not only comparing the few major frameworks, it's even broader, related to the methodology: ranging from SPA (single-page-applications), to SSR (server-side rendering), to SSG (server-side generation), to bundling tools for plain HTML/TS/JS/CSS. Each of these methodologies has its own ecosystem and tools, with many frameworks to compare. Last but not least, many frameworks are often hybrids and can be "configured" differently to span a range of SPA/SRR/SSG.
In retrospect, it was a big mistake. It slowly but surely evolved into a complete rewrite. The UI refactoring is a huge endeavour that should have been left untouched and made separately. Not because it should not be done, but because it should not have been done now.
How it should have proceeded
The whole order in which we started the "big refactoring" was wrong. Of course, it wasn't originally expected to be that time-consuming either, it was just "let's refactor that". In hindsight, we should have paid much more attention to the priorities and plan the refactoring accordingly. It should have been done in smaller steps, one after another, and in an order that makes sense according to the priorities.
In particular, the UI-related one should have been done last. Of course, from a marketing and business point of view, one might disagree. The holy UI/UX would take priority, with the slogan "make it nice first, with a slicker UI which loads super fast". Something that "looks awesome" ...but that would hide and delay upcoming breaking changes, which would just frustrate early adopters later on. I'm against these kinds of short-term wins at the price of long-term liabilities. The sooner the breaking change is done, the better. That is also why any kind of promotion is on hold until this time-consuming refactoring is completed.
What should have been prioritized is merging both codebases into one for single domain deployment, data schema refactoring to use hashes as IDs and one more encryption layer. The reason to handle them first is because it's a breaking change. These two things make it incompatible with the original first version.
Merge both codebases
Ensure single domain dev previews work fine
Apply schema changes for hashed IDs
Add extra encryption round
Once this is done, the new version can be published. And afterwards, the UI and back-end related "improvements" can take place. Those which take more time but keep compatibility.
To abord or to persevere?
We're now stuck "in the middle" of the rewrite including the new UI, so it's kind of annoying to drop the work in progress halfway. On the other hand, it looks like the road ahead is still quite long. So, despite it's annoying to let the work in progress halfway done, it's better for Passwordless.ID as a whole. The sooner the "v2" is released, the better.
Also, what we did not investigate was how to make the existing UI lighter. We aren't talking about excessive amounts here, it's ~130kb gzipped JS/CSS over the wire. But for a simple sign in/up page, it's still unnecessarily large. After some experimenting, it turns out that ~100kb can be spared, just by dropping the bootstrap-vue-next lib in favour of using plain bootstrap classes directly and a few workarounds for special components. That the lib came in with that much "baggage" and could not be properly "tree-shaken" came unsuspected. Dropping it, the page goes from ~130kb => ~30kb, already much nicer.
The full UI rewrite will still take place later on, to be even more efficient and internationalized. But for now, we will make the bare minimum UI changes and focus on the breaking change first, which will hopefully be completed sooner.
Stay tuned!