top of page

Project Overview

Personal Project 

Recently, I’ve developed a strong interest in robotics as a way to expand beyond purely mechanical design. While most of my experience had been focused on CAD and structural design, I wanted to gain hands-on exposure to electronics, embedded systems, and programming. This led me to develop HandSync, a vision-controlled robotic hand capable of replicating real-time human finger motion using only a webcam. Rather than relying on sensors or wearable devices, the system uses computer vision to detect finger positions and translate them into physical actuation. The goal of this project was to build a fully integrated system from the ground up, combining mechanical design, electrical systems, and software into a single working prototype.

  • Replicate real-time human finger motion using computer vision only (no sensors or gloves)

  • Enable independent and simultaneous control of all five fingers

  • Develop a system integrating Python-based vision processing with Arduino control

  • Implement a robust finger detection algorithm using joint position data

  • Ensure low-latency communication between vision system and actuation (real-time response)

  • Achieve smooth, controllable motion with interruptible and reversible servo control

  • Design a fully 3D-printable, lightweight, and low-cost mechanical system

image.png

About HandSync

Final Product!

1

2

3

The primary goal of this project was to develop and demonstrate hardware and software integration skills. As a result, greater emphasis was placed on electronics, control systems, and coding, with the mechanical design kept intentionally simple and functional.

This was my first project involving electronics and embedded systems, requiring me to independently learn wiring, power distribution, and Python-based control. All software development and hardware integration were completed from scratch, strengthening my ability to quickly learn and apply new technical concepts.

Future Improvements:

  1. Increase spring stiffness to improve tendon tension and overall finger responsiveness

  2. Introduce multi-joint fingers to achieve more realistic and adaptable motion

Step 1:

What even is Python?

The first step in this project was figuring out how to get a webcam feed into code and track my hand in real time. Since I had limited experience with Python, I started by setting up a basic programming environment in VS Code and confirming that Python could run correctly on my computer.

From there, I installed and tested OpenCV, which I used to access my webcam. OpenCV reads video as a continuous stream of images, so the first working test was a simple loop that repeatedly captured a frame from the camera and displayed it in a window. This helped me understand that video processing is really just image processing repeated many times per second.

Once the webcam feed was working, I researched hand-tracking libraries and found MediaPipe, which could detect a hand and overlay landmark points onto it. One important setup detail was converting each webcam frame from BGR to RGB before sending it into MediaPipe. OpenCV stores images in BGR format by default, while MediaPipe expects RGB images, so the conversion was needed to make sure the model interpreted the image colors correctly.

MediaPipe outputs landmark locations as normalized coordinates between 0 and 1, rather than direct pixel values. This means a point at x = 0.5 is halfway across the image width, and y = 0.5 is halfway down the image height. To use those points visually or compare them more intuitively, I converted them into pixel coordinates by multiplying the normalized values by the webcam frame width and height.

At this stage, the goal was not yet to control the robotic hand, but to create a reliable foundation: open the webcam, process each frame, detect one hand, and display the tracked landmark points in real time. This became the base layer for the rest of the project.

image.png

Coding Environment 

First Ever Hand Detection!

Turning 1 LED on

Turning 5 LED's on

Step 2:

Let there be light! 

Once I was able to consistently detect my hand, the next step was to send a signal to a microcontroller and trigger a physical response. I started by turning on a single LED when a finger was detected as “down,” with the goal of scaling this to all five fingers.

I chose LEDs as a starting point because they are low-risk compared to motors and provide a clear visual output for debugging both the circuit and control logic.

To interpret the hand-tracking data, I implemented a simple comparison-based algorithm. As discussed in Step 1, MediaPipe provides joint positions for each finger. For each finger, I compared the vertical (Y) positions of the fingertip and the joint below it. Since values increase downward in the image, if the fingertip appeared lower, the finger was classified as “down”; otherwise, it was “up.” This check ran continuously in a loop, updating finger states in real time.

The next challenge was connecting Python and Arduino. I implemented Serial USB Communication, allowing Python to send simple command strings (e.g., finger up/down states) to the Arduino over a USB connection. The Arduino continuously read incoming data and mapped each command to a digital output, turning an LED on or off.

I first wired a single LED to understand how to properly close a circuit, including current limiting and grounding. Once this worked reliably, I expanded the system to five LEDs, each mapped to a different finger.

At this stage, I had a complete pipeline: real-time hand detection, finger state classification, communication over USB, and independent control of multiple outputs—forming the foundation for servo-based actuation.

Step 3:

Servos And Mech Design

Once I was consistently sending signals and independently controlling five outputs, the next step was to replace the LEDs with servo motors and design a system that could physically actuate the fingers.

Wiring the servos introduced a new level of complexity. Each motor required three connections: power, ground, and a signal line. While the signal wires were connected to the Arduino, I quickly realized the board could not supply enough current to drive multiple servos at once. To solve this, I introduced an external power source to handle the load, while maintaining a shared ground between the power supply and Arduino to ensure stable communication. This was my first experience managing power distribution across multiple actuators.

On the mechanical side, the design was intentionally kept simple and functional, as the primary focus of the project was electronics and control. To transfer motion from the servos to the fingers, I implemented a tendon-driven system using fishing line, similar to a marionette. Each finger was connected to its own servo, allowing for independent motion. Initially, I attempted to tie the line directly to the servo horns, but testing showed that the rotation did not produce enough linear displacement to meaningfully move the fingers.

To address this, I designed custom servo-mounted pulleys that press-fit onto the servo horns. Increasing the pulley diameter allowed more string to be wound per rotation, significantly improving finger travel and responsiveness. I then designed a compact forearm housing to mount all five servos, ensuring proper alignment with each tendon path while keeping the assembly easy to access and assemble using standard fasteners.

A key consideration throughout was 3D printability, cost, and weight. All components were designed to fit within standard printer constraints, minimize support material, and use only the necessary amount of material for strength. This allowed for rapid iteration while keeping the overall system lightweight and low-cost.

There were also several failed iterations that informed the final design. An early forearm model was not properly toleranced, preventing the servos from fitting into their mounts, which required redesigning the geometry with appropriate clearances. Similarly, an initial pulley design was too small to properly interface with the servo horns, resulting in poor force transfer and slippage. These issues were resolved through iterative testing and refinement.

By the end of this stage, I had a fully integrated system capable of converting control signals into physical motion.

image.png
image.png

Forearm Skeleton

Forearm Full Assembly

All 5 Servo Motors Wired Up!

Palm + Finger Assembly 

image.png

Final Wiring Layout

image.png

Hand Assembly +  Webcam

Step 4:

Final Assembly 

With the design finalized, the last step was to manufacture and assemble the full system. All components were 3D printed using lightweight infill to reduce print time and material usage while maintaining enough strength for operation. After printing, parts were lightly sanded and cleaned to remove imperfections and ensure smooth assembly.

The hand and forearm were assembled using standard fasteners, with each servo mounted into the forearm housing and aligned with its corresponding tendon path. Fishing line was then routed from each fingertip down to its respective servo pulley and tied off with consistent tension. This step required careful adjustment—too much slack would reduce responsiveness, while too much tension would restrict motion or overload the servos.

Once assembled, the system was fully wired and powered, and initial testing began. While the hardware functioned correctly, a major issue emerged: only one finger would move at a time, even though multiple signals were being sent simultaneously.

After investigating, I found that the issue was not with the wiring, but with how the Arduino was executing commands. The control logic was written in a way that processed each instruction sequentially. When a command was received (e.g., move a finger to a certain position), the program would fully execute that motion before checking for the next command. Because servo movement was implemented using step-by-step position updates with delays, the microcontroller was effectively “blocked” while moving a single finger, preventing it from responding to other inputs.

To resolve this, I redesigned the control approach to be non-blocking. Instead of commanding a servo to complete its entire motion immediately, each servo was assigned a target position, and the Arduino loop was structured to incrementally move all servos toward their targets over time. By using time-based updates instead of delays, the system could continuously check for new inputs while updating all servos in parallel. This allowed simultaneous and independent movement of all five fingers, as well as the ability to interrupt and reverse motion mid-actuation.

Another issue encountered during testing was that the servos were moving too quickly, causing the fishing line to slip or fail to properly engage with the pulley walls. To improve accuracy and reliability, I introduced controlled speed reduction by adjusting the rate at which servo positions were updated. Slowing the motion ensured the tendons remained properly tensioned and allowed for more consistent and repeatable finger movement.

With these refinements, the system was able to reliably replicate real-time finger motion. Each finger could be actuated independently or simultaneously, controlled entirely through webcam-based hand tracking. This final stage brought together all aspects of the project—mechanical design, electrical integration, and software control—into a fully functional robotic hand.

HandSync Finalized!

Results

This project was a valuable experience in expanding my skillset beyond mechanical design into electronics, embedded systems, and software integration. Building the system entirely from scratch required me to learn how to connect multiple disciplines into a single working product, reinforcing my ability to quickly pick up new technologies and solve problems through iteration and testing. Some results were:

  • Achieved real-time hand tracking and control using a standard webcam (no sensors or wearables)

  • Enabled independent and simultaneous actuation of all five fingers

  • Implemented interruptible and reversible motion control, improving responsiveness

  • Successfully integrated Python (computer vision) with Arduino (embedded control) via serial communication

  • Designed and assembled a fully 3D-printed, modular robotic hand and forearm

  • Maintained total project cost under ~$200 including all electronic components and hardware

  • Achieved a lightweight system (< 2 kg) suitable for rapid prototyping and iteration

image.png
bottom of page