Introduction:

This project utilizes a local network setup where an Arduino Uno Q is connected to a USB camera and a robotic arm, with programming performed via the Arduino AppLab on our computer/laptop. The ease of using the I/O pins on the Arduino Uno Q, which are identical to those of a standard Arduino Uno, makes this project accessible and easy for anyone to carry out.

In this relatively simple project, the system will translate the position of hand detected into movements of two servos that control the pan-tilt mechanism on robotic arm, while gesture types (for example, open and close) will control the servo on the claw/gripper. The system can use an existing model (Hand Gesture) available in Arduino AppLab, or a custom model created in Edge Impulse Studio integrated within AppLab. The model will be deployed in Python code for the program logic and then passed to the Arduino Sketch to drive the servos accordingly.

This project is intended as a basic implementation of object detection in robotics, featuring real-time object tracking and intuitive robotic arm movement. It is a foundation for applications in more complex robotic systems.

Steps:

In practice, Hand Gestures Control for Robotic Arm works best when you follow a step-by-step arduino validation workflow and keep a simple checklist for wiring, power stability, and expected output behavior. This makes debugging faster and creates a practical click troubleshooting path for repeatable results.

1. Uno Q Setup and Arduino AppLab

For the initial setup, install the Arduino App Lab on your computer and connect the Arduino Uno Q via a USB-C connection (PC-hosted mode). Once AppLab is opened, the Uno Q device will appear; select it, create an access password, and configure the Wi-Fi network so that we can use it in local network mode for convenience. After completing this process, disconnect the Uno Q and connect it to a power adapter. Open the Arduino AppLab on your computer, and the Uno Q will appear for you to access. To familiarize yourself with this ecosystem, you can select 1 project from Examples and press Run, which will upload and execute the chosen program on the Uno Q.

For the next step, we will create our project by duplicating "Detect Objects from Camera" from the Examples, because it’s using similar bricks. Open it and click Copy and edit (in the upper right corner), then rename it as desired, for example: "Custom Hand Gestures Detection." Next, select the AI models tab and click Train new AI model, which will allow you to access Edge Impulse Studio using your Arduino account (sign in or create one if you don't have one yet). Then, you will be redirected to the Edge Impulse Studio by clicking Start to Train your AI model. We will discuss the stages within Edge Impulse Studio in the next step below.

2. Collecting Data ( Images)

Before we start uploading the photos, we will create three labels/classes: a closed hand (neutral), and a scissor-like hand gesture (open and close). Capture these photos using your camera or your laptop's built-in webcam. I recommend taking a variety of 30–40 photos for each type, which should be sufficient for this project. Save them in folders, and we will continue in the Edge Impulse Studio.

For those who are not familiar with Edge Impulse Studio, choose Images project option, then Object detection. In Dashboard > Project Info, choose Bounding Boxes for labeling method and Uno Q for target device. Then in Data acquisition, click on Upload Data tab. Choose you saved folder then upload.

3. Labeling

A reliable implementation also benefits from modular structure: separate input handling, processing logic, and output control so each part can be tested independently. That pattern supports low-noise project tuning, clearer model calibration decisions, and safer iteration when features evolve.

The next step is labeling, now click on Data Acquisition, click on Labeling queue tab, then start drag a box around an object and label it and Save. Repeat.. until all images labelled.

After labeling, it’s recommended to split the data into training and testing sets, around an 80/20 ratio. If you haven't done this yet, click on Train / Test Split to automate this process.

4. Train, Build, Deploy

Once your labelled dataset is ready, go to Impulse Design > Create Impulse, and set the image width and height (eg. 320x320). Choose Fit shortest axis, then select Image and Object Detection as the learning blocks, and click Save Impulse. Next, navigate to the Image Parameters section, select RGB as the color depth, and press Save parameters. After that, click on Generate, where you’ll be able to see a graphical distribution.

Now, move to the Object Detection section and configure the training settings. Select GPU and set the training cycles to around 100, learning rate to 0.001 and medium for model size. Choose YOLO-Pro or MobileNetV2 SSD FPN-Lite as the NN architecture. Once done, start training by pressing Start, and monitor the progress.

If everything goes well; eg MAP@50 = 1.0 or the precision result is more than 90%, proceed to the next step. Go to the Model Testing section, click Classify all, and if the MAP@50 = 1.0 or the accuracy is around 90%, you can move on to Deployment. Select deployment target Arduino Uno Q, select quantized (int 98) then click Build. Next step will be on Arduino AppLab.

For long-term maintainability, document baseline measurements such as response time, stability under transitions, and recovery after temporary faults. Using this measurement-driven impulse optimization style gives you a scalable applab upgrade path without turning the project into a fragile one-off demo.

5. Test on Uno Q

After the model has been built, return to the Arduino AppLab; you will find the custom model we built in Edge Impulse Studio earlier—click the Download icon and select it. Now, our custom model is ready to use. To test it, don't forget to connect the USB-C to USB-A hub and the USB camera to yout Uno Q. Then, click Run in AppLab; the upload and execution process on the Uno Q will begin. If everything goes smoothly, the WebUI-HTML program will display live object detection inference in your computer's browser. Try the 3 types of gestures from our model to ensure it is working properly.

6. Build Your System: Uno Q + usb camera + robotic arm + program

To control the robotic arm using hand gestures, you must ensure that everything is properly connected as shown in the diagram above. Additionally, before creating your Python or Sketch programs, you need to download several Sketch Libraries. Click Add Sketch Library, then search for and download them one by one: Servo 1.3.0, MsgPack 0.4.2, DebugLog 0.8.4, ArxContainer 0.7.0, and ArxTypeTraits 0.3.1 (or their latest updated versions).

Below is a screenshot of the Python code. It is important to adjust/calibrate the servo angles to match the physical configuration of your specific robotic arm. In this Python code, the logic is as follows: if the "neutral" gesture is detected, the horizontal position is mapped to the pan servo angle, while the size of the bounding box is interpreted as a change in the tilt servo angle. The "opening" and "closing" gestures translate to the movement/angle change of the claw servo. Meanwhile, the Sketch file determines which pins are used; in this case, pins 9, 10, 11. Servos angle standard setup and handles the output display on the LED matrix (a smiley face) whenever a hand gesture is detected.

All Python, Sketch and YAML files can be downloaded from Code Attachment section, then you can click Add File icon on AppLab then mod/adjust for your needs.

This project successfully demonstrates a basic Edge AI vision system that connects real-time object detection to physical actuation by deploying an Edge Impulse custom gestures model with reliable accuracy and low latency. The project validated the end-to-end pipeline from camera input, spatial mapping, and actuator control in a single embedded system. This project highlights the practical integration of Edge AI with robotics and serves as a foundation for more advanced applications.