# Metafinger

This page introduces the Metafinger, a soft robotic finger capable of multimodal proprioception. The Metafinger is a specific design of Soft Polyherial Networks[<sup>\[1, 2, 3, 4, 5, 6, 7\]</sup>](/magiclaw-docs/documentation/references/bibliography.md). The core components are the soft finger and the camera under it. We also installed a dev board with the camera, which can capture the images and send out via WiFi or USB. The metafinger is deployed at the end of the beam on MagiClaw and estimates the force and shape during interaction.

## Preview

{% embed url="<https://my1261625.autodesk360.com/g/shares/SH286ddQT78850c0d8a44e94a41d9e1e7b17?mode=embed>" %}
Metafinger
{% endembed %}

## Specifications

<table><thead><tr><th width="200">Parameters</th><th>Description</th></tr></thead><tbody><tr><td>Module</td><td>LicheeRV Nano W</td></tr><tr><td>CPU</td><td>SOPHGO SG2002;
<br>Major core: 1GHz RISC-V C906 / ARM A53 (selectable);
<br>Minor core: 700MHz RISC-V C906;
<br>Low-power core: 25～300M 8051</td></tr><tr><td>NPU</td><td>1TOPS INT8, supports BF16</td></tr><tr><td>Memory</td><td>Integrated 2Gbit (256MByte) DDR3</td></tr><tr><td>Wi-Fi</td><td>Dual-band 802.11ax</td></tr><tr><td>Bluetooth</td><td>Bluetooth 5.4/ BLE</td></tr><tr><td>Storage</td><td>1 x microSD card slot</td></tr><tr><td>USB</td><td>1 x USB 2.0 OTG Type-C</td></tr><tr><td>Camera</td><td>GC4653</td></tr><tr><td>Lens</td><td>1.8mm 1/4" 5MP</td></tr><tr><td>Frame rate</td><td>60fps@720P (max)</td></tr><tr><td>Dimensions (W x D x H)</td><td>40mm * 40mm * 115mm</td></tr><tr><td>Weight</td><td>56g</td></tr><tr><td>Operating Temperature</td><td>0 ~ 45℃</td></tr></tbody></table>

## Flash Image

The finger dev board, LicheeRV Nano W, needs a TF card with system image. Please follow the following steps to flash the image.

**Step 1.** Find the **latest** system image file on [Google Drive](https://drive.google.com/drive/folders/17qpqQkLpI-1oUO4QjZ3U-uHipoeFbyrk), for example, `metafinger-latest.img`.

**Step 2.** Download [balenaEtcher](https://www.balena.io/etcher/) (highly recommended), install and launch it.&#x20;

**Step 3.** Insert a TF card into the card reader and connect it to your PC. In balenaEtcher, select the downloaded image file, choose the TF card, and click `Flash`.

{% hint style="warning" %}

### WARNING

During or after flashing, if the PC prompts with `You need to format the disk in drive X: before you can use it`, do not format the disk. This would erase the newly flashed system. Just close the window, and eject the TF card safely.
{% endhint %}

**Step 4.** Insert the TF card into the dev board, power it on, and wait for the system to boot. The first boot may take a little longer, just be patient.

## Connect to Dev Board via SSH

Since the WiFi settings are not configured, you need to connect the dev board to the PC via a USB cable, and the board will emulate as a USB network adapter. Then the board and the PC will be on the same local network through the USB connection. By default, there are two types of USB virtual network adapter drivers (NCM and RNDIS drivers) to meet the needs of different systems.

{% tabs %}
{% tab title="Windows" %}
All Windows systems will automatically install the RNDIS driver, while only Windows 11 will automatically install the NCM driver. As long as one of the drivers works, it is sufficient.

Open Task Manager -> Performance, and you should see a virtual Ethernet with an IP address such as `10.131.167.100`, which is the computer's IP address. The device's IP address is the same but with the last digit changed to `1`, i.e., `10.131.167.1`.&#x20;

If you are using Windows 11, you will see two virtual network adapters; you can use either IP address.
{% endtab %}

{% tab title="Linux" %}
Use `ifconfig` or `ip addr` to see the `usb0` and `usb1` network interfaces, and either IP address can be used. The IP address you see, such as `10.131.167.100`, is the computer's IP address, and the device's IP address is the same but with the last digit changed to `1`, i.e., `10.131.167.1`.
{% endtab %}

{% tab title="MacOS" %}
Check for the `usb` network adapter under `System Settings` -> `Network`. The IP address you see, such as `10.131.167.100`, is the computer's IP address, and the device's IP address is the same but with the last digit changed to `1`, i.e., `10.131.167.1`.
{% endtab %}
{% endtabs %}

To configure the connection settings on the PC, open Terminal and run

```bash
# SSH by IP
ssh root@<ip address>
# SSH by hostname
ssh root@<hostname>.local
```

where `root` is the default user name,  `<ip address>` is the IP address of the dev board which can be found on the connected PC, and `<hostname>` is the hostname can be found in the dev board. Both of them are capable, just select your preferred way. Type `yes` for the following message

```bash
ECDSA key fingerprint is SHA256:XXXXXXX.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
```

Then enter the default password `root`. If you have successfully connected to the dev board, you will see the following output

<figure><img src="/files/Z9eVdikiReZOE8gKKgmq" alt=""><figcaption></figcaption></figure>

## Configure WiFi Settings

After connecting to the dev board via SSH, run the following command to set the WiFi:

```bash
python /root/wifi_connect.py --ssid <ssid> --pwd <pwd>
```

where `<ssid>` is the name of the WiFi, and `<pwd>` is the password. After connecting to the WiFi successfully, the output will show the IP address of the dev board obtained from the router.

## Assemble Camera and Lens

GC4653 is a MIPI camera with good image quality. It can achieve a high frame rate of 60fps at 720p resolution. But FOV of the original lens is not enough for the Metafinger, so another wide-angle lens is selected to replace it. Please follow the steps to assemble the camera and lens.

**Step 1.** Remove the original lens from the camera and replace it with the provided wide-angle lens.

**Step 2.** Install the camera onto the MIPI camera mount, paying attention to the wire sequence. Then power it on, and wait for the system to boot.

**Step 3.** Clone the latest `magiclaw` repository and install all dependencies on either the [control box](/magiclaw-docs/documentation/hardware/controller.md) or your PC:

```bash
git clone https://github.com/asMagiClaw/magiclaw.git
cd magiclaw
pip install -e .
```

Test whether the images captured by the camera can be streamed successfully:

```bash
python scripts/read_camera.py --addr <addr>
```

where `<addr>` is the IP address of the dev board. A window will show streaming images from the dev board.

<figure><img src="/files/amLnxsFQSCgbbZWz5AP4" alt="" width="362"><figcaption></figcaption></figure>

**Step 4.** Rotate the lens to adjust the focal length so that the camera can capture the marker clearly.

## Calculate Camera Parameters

To obtain the marker's pose, we need to calibrate the camera's intrinsic matrix and distortion coefficients. We recommend using a black-white chessboard (12 x 9, square size of 1.5mm).

**Step 1.** Power the dev board on, and prepare the chessboard ready.

**Step 2.** Collect chessboard images streamed from the dev board:

```bash
python scripts/calibrate_camera.py --name <name> --addr <addr>
```

where `<name>` is the camera name defined for saving directory, and `<addr>` is the IP address of the dev board. A window will show streaming images from the dev board.&#x20;

**Step 3.** Press `c` to capture the image, and `ESC` to quit capturing. All images will be saved under `./data/camera_calibration/<name>_<width>x<height>/`  directory, where `<name>` refers to the name defined above, and `<width>` and `<height>` refers to the size of the captured images.

**Step 4.** After pressing `ESC`, you will be asked which software is to used for calculation, either OpenCV or MATLAB. To use Open CV, enter `y`, and the script will start calibration automatically. To use MATLAB, enter `n`, and the script will exit.

If using MATLAB, first launch the MATLAB and open the Camera Calibrator app. Press `Add Images` and select the saved images. Enter `1.5` for `Size of checkerboard square` in a new window `Image and Pattern Properties`. Open `Options -> Radial Distortion`, and select `3 Coefficients`. Press `Calibrate` to start calibration.

<figure><img src="/files/Xvbzy7ka0nseI8lAm91m" alt=""><figcaption></figcaption></figure>

Then export camera parameters and a variable will be created containing calculated results. A MATLAB function is provided in `scripts/calib2yaml.m` to export the parameters to a `.yaml` file. Running following command in the MATLAB command line:

```bash
cd scripts
calib2yaml(<variable_name>, <file_name>)
```

where `<variable_name>` refers to the name of the variable containing results, and `<file_name>` refers to the name of the `.yaml` file for saving parameters. The file will be saved under the same `scripts/` directory.

**Step 5.** Modify the `.yaml` file for camera parameters. A file example is provided as `configs_example/camera/maxicam-xxxx.yaml`, containing `mode`, `host`, `width`, `height` and so on. You need to replace the values of `dist` and `mtx` with the calculated results. This file should be saved under `configs/camera` directory, named as `hostname.yaml` consistent with the hostname of the dev board.

## Marker Tracking

Metafinger uses [ArUco](https://www.uco.es/investiga/grupos/ava/portfolio/aruco/) markers for pose tracking. The camera captures the images of the marker fixed on the soft finger and streams to the control box or the PC. Then the marker is detected and the pose is estimated using [OpenCV](https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_detection.html). The value change of the pose represents the deformation of the soft finger and will be as the input of the model to infer the force and shape.

<figure><img src="/files/yR4BaZihM7cUeHQIm4Et" alt=""><figcaption></figcaption></figure>

## Model Training

The soft finger structure can "feel" the force and own shape, called as "proprioception", during physical interaction with environment. To estimate the force and shape, we provided a proprioceptive learning framework for the soft finger. Please follow the guide in [metafinger](https://github.com/asMagiClaw/meta-finger) repository to train and test the model.

## Bill of Materials

<table><thead><tr><th>Part</th><th>Specifications</th><th data-type="number">Quantity</th><th>Price (per unit)</th><th data-type="content-ref">Link</th></tr></thead><tbody><tr><td>Soft finger</td><td>Hei-Cast 8400</td><td>1</td><td>$ 25.00</td><td><a href="https://drive.google.com/file/d/1SlXp93rtElRAw1PS2mrloDCAdfeSR9Z3/view">https://drive.google.com/file/d/1SlXp93rtElRAw1PS2mrloDCAdfeSR9Z3/view</a></td></tr><tr><td>Finger interface</td><td>4 cm wide, 4 mm thick with felt</td><td>1</td><td>$ 2.00</td><td><a href="https://detail.tmall.com/item.htm?id=748792781150&#x26;skuId=5208513812728">https://detail.tmall.com/item.htm?id=748792781150&#x26;skuId=5208513812728</a></td></tr><tr><td>Finger dev board</td><td>LicheeRV Nano W</td><td>1</td><td>$ 15.29</td><td><a href="https://www.aliexpress.com/item/1005006519668532.html">https://www.aliexpress.com/item/1005006519668532.html</a></td></tr><tr><td>Finger camera</td><td>GC4653</td><td>1</td><td>$ 12.26</td><td><a href="https://www.aliexpress.com/item/1005006519668532.html">https://www.aliexpress.com/item/1005006519668532.html</a></td></tr><tr><td>Camera lens</td><td>1.8mm 1/4＂5MP</td><td>1</td><td>$ 5.00</td><td><a href="https://item.taobao.com/item.htm?id=671331312652&#x26;skuId=5891956706853">https://item.taobao.com/item.htm?id=671331312652&#x26;skuId=5891956706853</a></td></tr><tr><td>Finger base cover</td><td>PLA</td><td>1</td><td>$ 0.08</td><td><a href="https://drive.google.com/file/d/18Wws-Ww0zBYUiCdIryUHiyX5tSIjqI0E">https://drive.google.com/file/d/18Wws-Ww0zBYUiCdIryUHiyX5tSIjqI0E</a></td></tr><tr><td>Finger base</td><td>PLA</td><td>1</td><td>$ 0.46</td><td><a href="https://drive.google.com/file/d/169iu9c9Co-tBOwpOXDF6m4Wau4_oBG6O">https://drive.google.com/file/d/169iu9c9Co-tBOwpOXDF6m4Wau4_oBG6O</a></td></tr><tr><td>Finger marker</td><td>PLA</td><td>1</td><td>$ 0.21</td><td><a href="https://drive.google.com/file/d/1hJ1iUjy75cXlg5wXO78ic4KK_Y1Lnz3c">https://drive.google.com/file/d/1hJ1iUjy75cXlg5wXO78ic4KK_Y1Lnz3c</a></td></tr></tbody></table>

## CAD Models

STEP files of all 3D-printed parts are provided on [Google Drive](https://drive.google.com/drive/folders/1eggZ7Y-1Vx3dt_DY61oJZzGpe7rARkbe).

## Dev Board Resources

Datasheets, schematic diagrams, drawings, and more about the dev board can be found [here](https://wiki.sipeed.com/hardware/en/lichee/RV_Nano/1_intro.html#Hardware-Resources).


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://magiclaw.gitbook.io/magiclaw-docs/documentation/hardware/metafinger.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.