Table of Contents

1. Introduction
2. Diving Into New Technologies
3. Overcoming Pose Detection and 3D Transformations
4. Pushing Through the Struggles
5. The Power of Rapid Prototyping and Collaboration
6. Takeaways

Introduction

Hack&Roll 2024 was an intense, fast-paced challenge that offered an incredible learning experience. In just 24 hours, our team developed Chioset, a virtual wardrobe app allowing users to upload photos of their clothes and try them on using Augmented Reality (AR). The project was filled with technical hurdles but also presented invaluable opportunities for growth. Inspired by one of my modules on AR, I decided to dive into the challenge. Below, I’ll share the challenges we faced and the lessons learned while building the AR try-on feature.

In case you're wondering, "Chioset" is a blend of the Singlish word "chio" (meaning pretty) and "closet".

Diving Into New Technologies

When I first took on the AR try-on feature, I had no experience with Three.js or TensorFlow.js, which were critical to the AR functionality. Three.js handled the 3D rendering, and TensorFlow.js was used for pose detection, helping us track the user's body movements and overlay clothing accordingly.

This was my first real deep dive into computer vision and 3D rendering, and I had to quickly get up to speed on concepts like pose estimation and real-time video processing. Initially, things felt overwhelming, but the excitement of seeing real-time AR in action soon outweighed the technical challenges.

Overcoming Pose Detection and 3D Transformations

Pose detection was one of the toughest hurdles I faced. The goal was to identify the user's body in real-time through the camera, then position clothing on them in a way that made sense as they moved. TensorFlow.js provided a pre-trained pose detection model, but integrating it with Three.js was trickier than anticipated. The keypoints (such as shoulder and elbow positions) provided by the model were helpful, but turning those points into accurate clothing placement was a complex task.

On top of that was the scale discrepancy between the webcam feed and the video feed. This caused the clothing either to misalign with the user's body or appear off-screen entirely. The math behind the transformations became incredibly challenging. It wasn’t just about aligning the clothing; it was about properly scaling and positioning it, while factoring in the different dimensions of the camera and video feed. Getting it right required a lot of fine-tuning and trial and error.

Lastly, the clothing was only 2D, which made it nearly impossible to make it wrap naturally around the user's body. Without a 3D model, the clothes couldn’t conform to the user's shape, making it harder to simulate realistic clothing fit and positioning.

Despite these challenges, after countless iterations, the moment I saw the clothing follow the user's movements in real-time was incredibly rewarding.

Watch the AR Try-on Demo Watch the AR Try-on Demo

Pushing Through the Struggles

As with any intense project, there were moments when my team and I felt like quitting. The 24-hour timeline was overwhelming, and we faced roadblocks that made us question whether we’d ever finish. But instead of giving up, we kept pushing forward—trying, failing, and trying again. Every time we hit a roadblock, we collaborated, learned, and persevered.

I’m incredibly grateful for my teammates and friends, whose support was invaluable. Their encouragement and assistance made all the difference. Without their help, especially when I felt like giving up, I may not have made it through.

The Power of Rapid Prototyping and Collaboration

Building Chioset in just 24 hours required the entire team to collaborate effectively and pivot quickly when we hit roadblocks. Rapid prototyping enabled us to iterate on features quickly, test different approaches, and find out what worked best.

A crucial takeaway from this experience was the importance of failing fast. If something didn’t work, we didn’t dwell on it. Instead, we moved on to try a different approach until we found the right solution.

Takeaways

While the 24-hour timeline was incredibly challenging, building Chioset was an unforgettable experience that taught me:

• The importance of learning new technologies under time constraints: Despite the challenges, I now feel more confident working with Three.js and TensorFlow.js.
• The power of collaboration and rapid iteration: Working as a team to solve problems quickly taught me the importance of being flexible and adaptable when building something new.

Check out the project on Devpost to see how we brought it all together!