ERBIE: a robot that serves Arabic coffee by the rules

If you're finished, you wiggle the cup#

If someone is serving you Arabic coffee and you've had enough, you tell them by wiggling your cup. It's called Kafa.

We built a robot that had to understand that. The robot is ERBIE.

ERBIE is tailored for serving Arabic coffee to a group of guests “within the expected customs and culture.” That last part is why it was interesting to build. A robot that drives around and a mechanism that handles cups are both things you can picture. A robot that has to act like the person serving coffee in a room full of guests is a stranger assignment.

This was a team build. My part was all of the software on the Jetson, including the face and hand-gesture recognition. The rest of the robot came from the rest of the team, and I'll get to that at the end.

The etiquette is the spec#

A top-down view of a room with seats around its edges and a patterned carpet in the middle. — **The average Majlis,** seen from above.

I'll start with the etiquette. The person serving the coffee is called the Kahwaji. The Kahwaji goes around the room and serves the host's guests.

Two rules are universal:

Kafa. Shaking or wiggling the cup tells the server that you're finished.
The right hand. The coffee is always served with the right hand, and guests accept it and drink it with the right hand.

Our setting for all of this was a Majlis, serving a group of guests. If you're building a machine to take the Kahwaji's job, those few sentences read like requirements. It has to get to each guest in the room. It has to notice there's a person there before it does anything. And it has to pay attention to the guest's hands, because a cup shake is how a guest says “that's enough.”

So the hand gestures became the controls for the robot.

What the robot does#

ERBIE standing in a lab: a frame of white plastic pipe with plywood shelves, wiring and orange parts in the middle, and a black wheel near the floor. — **ERBIE** in the lab.

ERBIE follows a line until it detects a human, then it begins service. It does this for every person it finds. That last sentence is the Kahwaji's job turned into a loop: go around the room, stop at a person, serve them, then go find the next one.

The robot itself is a tall frame of white pipe with plywood shelves, with the wheels down at the bottom.

Service is where the hand signals come in. We implemented four:

Four icons: an open palm, one raised finger, two raised fingers, and a hand holding a cup. — **The four gestures.**

Gesture	What the guest is saying
Open palm	No service please
One finger	I would like a cup
Two fingers	Redo my selection
Kafa cup shake	I am done

Kafa is the one that comes straight from the etiquette. The other three give a guest a way to decline, to ask for a cup, or to change their mind. Kafa is also the odd one out in another way. The first three are hand shapes you can hold still. A cup shake is a movement, with a cup in your hand.

Put together, one guest's turn goes something like this. The robot reaches them and starts service. One finger asks for a cup, an open palm says no service, and two fingers takes back a choice. When they've had enough, they shake the cup.

The pictures below come from separate clips rather than one continuous run: the base on a strip of tape, the serving section on its own, and the gesture software on a monitor.

The robot's wooden base with two black wheels, circuit boards and wiring, sitting on a strip of bright green tape laid over flattened cardboard boxes. — **The base on the line.** A wooden platform with two black wheels, sitting on green tape laid over flattened cardboard boxes.

Three layers, and a Jetson in the middle#

Our system diagram splits the robot into three layers.

System diagram with three columns. Service Layer: a Tiva C Board connected to Stack Manager, Cup Grabber and Pump. Command and Control Layer: Camera into Jetson Nano, connected to Wifi and then Front-end. Mobile Layer: a Tiva C Board connected to Line Detection, Motor Control and Battery Management. The Jetson Nano connects to both Tiva C boards. — **The system diagram.**

Command & Control. A camera feeds the Jetson Nano, and the Jetson connects over WiFi to a front-end.
Service. A Tiva C board connected to the Stack Manager, the Cup Grabber and the Pump.
Mobile. A second Tiva C board connected to Line Detection, Motor Control and Battery Management.

The Jetson Nano talks to both Tiva C boards. So there's one computer in the middle with the camera, and a board on each side of it looking after one half of the robot: the half that handles the coffee and the half that moves. The Jetson is also the only part of the diagram connected to both sides.

Here's the serving side, in two frames from the same clip.

The upper section of the robot. Two patterned cups sit in an orange holder above a shelf. An orange unit is at the right end of a rail. — **The serving section, from the same clip.** In the first frame, two patterned cups sit in an orange holder above the shelf, with the orange unit at the right end of its rail. In the second, one cup sits partway down the left column and the orange unit has moved left along the rail.

The same section later. One patterned cup sits partway down the left column, and the orange unit has moved left along the rail. — **The serving section, from the same clip.** In the first frame, two patterned cups sit in an orange holder above the shelf, with the orange unit at the right end of its rail. In the second, one cup sits partway down the left column and the orange unit has moved left along the rail.

The part I worked on#

I wrote all of the software on the Jetson, including the face and hand-gesture recognition, and all of it in Python.

The libraries were OpenCV, MediaPipe and Flask.

The easiest way to show that part is the monitor.

A monitor showing a desktop with a terminal and a window titled Hand Gesture Recognition. The window has a false-colored panel on the left and a person on a gray background on the right. — **The Jetson desktop.** A terminal in the lower left and a window titled “Hand Gesture Recognition.” The face is blurred.

The window has two panels. On the left is a false-colored view of the room, where people show up as flat blue shapes against a warmer background. On the right is the person in front of the camera, with the background grayed out.

The terminal underneath shows the order of events. There's a MOBILE: GO, a long column of lines starting with Confidence:, and then this:

The last lines in the terminal, copied as they appear on screen.

expecting a face being detected moving forward now
MOBILE: STOP
Detecting gesture
selected gesture: 160
Giving person chance to redo gesture recognition

That matches the behavior I described earlier: the robot goes, it finds a person, it stops, and it looks at their hand. The last line gives that person a chance to redo the gesture.

Then the hand goes up to the camera, and green dots appear on the fingers and palm.

The Hand Gesture Recognition window: a person holds up one finger, with green dots marked along the finger and across the hand. — **Two of the four gestures on screen.** Left: one finger, “I would like a cup.” Right: open palm, “No service please.” Faces are blurred.

The Hand Gesture Recognition window: a person holds up an open palm, with green dots marked along each finger and across the palm. — **Two of the four gestures on screen.** Left: one finger, “I would like a cup.” Right: open palm, “No service please.” Faces are blurred.

The rest of the robot#

ERBIE was a team build. I wrote the Jetson software. The mechanical work, the Tiva C firmware, the line following and the motors were the rest of the team's, and without those there's no robot for the software to run on.