Introduction¶
This was my first job where I worked 9-9-6 for a company. Aside from other small or half-finished tasks, I made the following projects, in chronological order:
"Shazam" for Fashion¶
What it is¶
This was a feature where users would send a DM to our Instagram account containing any kind of visual media (photo or video attachments, posts, reels, whatever), and we would reply with a melian.com/outfit/:uuid link, displaying purchase links for each of the fashion items shown in the images. The goal of this feature was just to get users, as it kind of already existed with Google Lens, but not on Instagram as we made it.
It was split into two parts: first, the Instagram chat handler, in charge of detecting, parsing, and sending messages; and second, the processing pipeline, in charge of receiving media bytes and returning a URL to a nice page that showed that media along with purchase links. Luciano (CTO) made the first one, and I made the second one.
Challenges¶
The main hassle was figuring out how to find the same, or at least an actually similar, product from our database based on the image.
Before I joined Melian, Mateo (co-founder and CPO) had quickly made a rough version of the "Shazam," but it was quite inaccurate. At first, I tried patching it, which meant trying an image, seeing where it failed, adding some code to handle that kind of scenario, and repeating; however, a patch for one case ended up breaking other cases. After that, given how slow it was to do the testing manually, I made a back-office where we could enter images and the products we expected to see, with a button to run them all and get an accuracy score. In a word, evals. Still, I wasn't able to pass 60%.
That was becoming an unintelligible mess, so one day I re-thought, now from scratch, how it should work; then, I created a new branch, deleted everything, rolled up my sleeves, and re-made it. That day, evals passed over 95%.
How it works¶
The algorithm was, more or less, as follows:
- Receive the media.
- Is it a video?
- Get 10 frames per second.
- Reduce each frame's size to 64 bytes.
- Deduplicate frames based on Hamming similarity.
- Continue using the original version of the remaining frames.
- Get crop coordinates for the fashion items using the Google Vision API.
- Make a collage of all crops and photos, and ask an LLM which crops show the same product, and which crop is the most representative of it.
- Take each representative crop, and make two extra variations of it: one with padding; and one with a white background, generated by Nano Banana, as if it were a stock image.
- Find products by embeddings search, using the original crop and its two variations.
- Make a collage of the matched products per photo, and ask an LLM which products should be shown for that particular photo.
- Save that product as associated with that photo, related to a new outfit.
- Return the ID of the new outfit.
- Render the page based on the outfit ID.
Later, I realized it wasn't so important to show products from our database. Instead, what really mattered for the "Shazam" was just the purchase URL. Then I found out about SerpApi (a wrapper that lets you use Google Lens programmatically), so we decided to simply show Google Lens results before ours, as they were usually better, since we didn't have every product on the internet in our database yet.
Onboarding and home remake¶
Why¶
Having finished the "Shazam," we had to decide what my next endeavor would be. Taking advantage of the fact that it was bringing users, we listened to them, and most were complaining about the onboarding (or not even answering, as they had already churned during it).
Thus, we hired a designer to redo it properly. Meanwhile, I wouldn't stay idle. By that time, the "home screen" was really the search screen, where we showed an infinite, Pinterest-like recommendation feed; but, being the power user he is, Santiago (co-founder and CEO) felt that we were missing a way to explore products the way we do with songs on Spotify: through collections. So, while we waited for the designs, I started remaking the home screen, mostly just adapting Spotify's design to our app.
"Best back-office ever"¶
Ideally, the collections would be created for each user specifically, based on their taste. However, there was nothing made for that, and we wanted it shipped as soon as possible; so, besides creating the new home frontend, I instead created an easy-to-use back-office to create collections manually. Santiago can attest that it was the best back-office he had used thus far, and later showed it as an example to other engineers.
How I implemented it¶
After that, the designs for the onboarding were ready. At that time, even with frontier LLMs, it still (unfortunately) wasn't enough to just ask them to "remake the onboarding based on this Figma, make no mistakes." After trial and error with a few approaches, I came up with one that got around 80-90% of the work done overnight. In short, it was something like this:
0. Using the Figma MCP, check [this Figma].
1. For each screen,
1.1. Spin a subagent to code it
1.2. Spin a subagent to review it
1.3. Repeat until it has no issues
2. Repeat for all screens.
3. While there are issues,
3.1. Spin a subagent to review the entire onboarding, backend interactions included.
3.2. Spin one subagent per issue to fix it.
4. Make a PR.
Still, it wasn't that straightforward, because the Figma, even though it looked perfect at first glance, clearly shouldn't be coded literally as it was, so I had to put my criteria into words beforehand. (And no, asking the agent to "use your own criteria" was not enough.) And very importantly, I had to delete the existing code before that could work, otherwise the agents got confused and messed up.
The above took around 16 hours non-stop, and there was still around 10-20% of the work left for me to fix iteratively with the agent.
Revamped analytics¶
What was wrong¶
Last but not least, now that we had a supposedly better onboarding, we wanted to check whether it was actually converting more users. However, both our analytics tracking and our dashboard were a complete mess. We couldn't tell! So I got to work on that.
Easy win after learning¶
Having learned from the "Shazam," this time I decided to just start by deleting all existing analytics-related code, then critically think about what should actually be tracked and shown in the dashboard, and finally add only that.
That was the meaty part of my internship at Melian. Being a startup, it was normal to have to spend some time addressing fires, or trying things and then dropping them.