This project started with my colleague Anin, who noticed someone using hand gestures in church. She realized it was sign language for the hearing impaired, and we discovered a significant lack of awareness about hand language. To explore this, we did a short interviews with some people at the Apple Developer Academy @ UC. We found that many wanted to understand hand language but were hesitant to commit to learning it.
From this challenge, our Team decided to create a game that would subtly teach players about hand language while they played. For our Mini Challenge 2 , themed around exploring Apple technologies, we wanted to try something new: utilizing SpriteKit and machine learning, areas none of us were familiar with. I took on the responsibility of developing the machine learning aspect to recognize basic hand gestures.
In this challenge We faced a lot of problems, From my background, I had never utilized machine learning before, so this was my first project in that domain. Despite my inexperience, I would like to try taking that assignment. I quickly learned that creating an ML model is not straightforward. I used Create ML, which requires an understanding of various templates, such as Hand Pose Classification, Action Classification, and Hand Action Classification. Each template relates to my problem but serves different functions.
I also had to ensure I had appropriate training data, considering factors such as: • Quality: Using high-quality and diverse images or videos. • Duration: If the gesture lasts one second, the training data must be consistent in duration. • Quantity: More data generally leads to more accurate results. • Data Balance: If I create four classes, ideally each class should have an equal number of samples.
Implementing this in the app was also quite challenging, as I had to align the video data from the camera with the model's expected input. At the end of the challenge, we presented our game to UC students, showcasing a unique experience that combines gameplay with hand language learning. We shared our learning journey and provided tips and tricks for using Apple technology.
Shout out to my teammate Benedikta Anindya Bernadi who made how the game works using Sprite Kit and Citta Catherine Gozali our spectacular designer all assets made by her and our mentors who always support us, Pak Allan and Ko Jaya. !! 🔥
and on October 14th yesterday, our PTS team was given the opportunity to open a booth at the AI Super User event at Ciputra University. There, we hope to inspire our high school friends to be more innovative in using ML.