Introducing Pixy 2

19 Mar, 2019
Posted by: shanto shaji

Robot vision made easier

Pixy2 is smaller, faster and more capable than the original Pixy. Like its predecessor, Pixy2 can learn to detect objects that you teach it, just by pressing a button. Additionally, Pixy2 has new algorithms that detect and track lines for use with line-following robots. The new algorithms can detect intersections and “road signs” as well. The road signs can tell your robot what to do, such as turn left, turn right, slow down, etc. And Pixy2 does all of this at 60 frames-per-second, so your robot can be fast, too

Vision as a Sensor

If you want your robot to perform a task such as picking up an object, chasing a ball, locating a charging station, etc., and you want a single sensor to help accomplish all of these tasks, then vision is your sensor. Vision (image) sensors are useful because they are so flexible. With the right algorithm, an image sensor can sense or detect practically anything. But there are two drawbacks with image sensors: 1) they output lots of data, dozens of megabytes per second, and 2) processing this amount of data can overwhelm many processors. And if the processor can keep up with the data, much of its processing power won’t be available for other tasks.

Pixy2 addresses these problems by pairing a powerful dedicated processor with the image sensor. Pixy2 processes images from the image sensor and only sends the useful information (e.g. purple dinosaur detected at x=54, y=103) to your microcontroller. And it does this at frame rate (60 Hz). The information is available through one of several interfaces: UART serial, SPI, I2C, USB, or digital/analog output. So your Arduino or other microcontroller can talk easily with Pixy2 and still have plenty of CPU available for other tasks.

It’s possible to hook up multiple Pixys to your microcontroller — for example, a robot with 4 Pixy2's and omnidirectional sensing. Or use Pixy2 without a microcontroller and use the digital or analog outputs to trigger events, switches, servos, etc.

Controller support

Pixy2 can easily connect to lots of different controllers because it supports several interface options (UART serial, SPI, I2C, USB, or digital/analog output), but Pixy began its life talking to Arduinos. We’ve added support for Arduino Due, Raspberry Pi and BeagleBone Black, as well as LEGO Mindstorms EV3. Software libraries are provided for all of these platforms so you can get up and running quickly. Additionally, we’ve added a Python API if you’re using a Linux-based controller (e.g. Raspberry Pi, BeagleBone).

60 frames per second

What does “60 frames per second” mean? In short, it means Pixy2 is fast. Pixy2 processes an entire image frame every 1/60th of a second (16.7 milliseconds). This means that you get a complete update of all detected objects’ positions every 16.7 ms. At this rate, tracking the path of falling/bouncing ball is possible. (A ball traveling at 40 mph moves less than a foot in 16.7 ms.) If your robot is performing line following, your robot will typically move a small fraction of an inch between frames.

Color Connected Components

Purple dinosaurs (and other things)

Pixy2 uses a color-based filtering algorithm to detect objects called the Color Connected Components (CCC) algorithm. Color-based filtering methods are popular because they are fast, efficient, and relatively robust. Most of us are familiar with RGB (red, green, and blue) to represent colors. Pixy2 calculates the color (hue) and saturation of each RGB pixel from the image sensor and uses these as the primary filtering parameters. The hue of an object remains largely unchanged with changes in lighting and exposure. Changes in lighting and exposure can have a frustrating effect on color filtering algorithms, causing them to break. Pixy2’s filtering algorithm is robust when it comes to lighting and exposure changes.

Seven color signatures

Pixy2's CCC algorithm remembers up to 7 different color signatures, which means that if you have 7 different objects with unique colors, Pixy2’s color filtering algorithm will have no problem identifying them. If you need more than seven, you can use color codes

Hundreds of objects

Pixy2 can find literally hundreds of objects at a time. It uses a connected components algorithm to determine where one object begins and another ends. Pixy2 then compiles the sizes and locations of each object and reports them through one of its interfaces (e.g. SPI).

Line tracking for line-following

Pixy2 has added the ability to detect and track lines. Line-following is a popular robotics demo/application because it is relatively simple to implement and gives a robot simple navigation abilities. Most line-following robots use discrete photosensors to distinguish between the line and the background. This method can be effective, but it tends to work best with only thick lines, and the sensing is localized making it difficult for the robot to predict the direction of the line or deal with intersections.

Pixy2 attempts to solve the more general problem of line-following by using its image (array) sensor. When driving a car, your eyes take in lots of information about the road, the direction of the road (is there a sharp curve coming up?) and if there is an intersection ahead. This information is important! Similarly, each of Pixy2's camera frames takes in information about the line being followed, its direction, other lines, and any intersections that these lines may form. Pixy2's algorithms take care of the rest. Pixy2 can also read simple barcodes, which can inform your robot what it should do – turn left, turn right, slow down, etc. Pixy2 does all of this at 60 frames-per-second.

Detecting and tracking lines

Pixy2's algorithms can detect dark lines on a light background or vice-versa. You can also tell Pixy2 that you are only interested in lines within a certain range of widths. Pixy2 goes through each frame finding lines that meet your criteria and determining where each line begins and ends within the frame.

Normally, a robot is only interested in the line that it is currently following. If new lines appear in the frame, you typically don't want your robot to switch and start following those lines. Pixy2 tracks each line, determining where each line moves in subsequent frames, so the line your robot is following remains so unless you tell it otherwise. The line your robot is currently following is called the Vector. Pixy tells you where the Vector starts and ends in each frame, so you can use that information to determine your robot's motion. So, for example, if the Vector heads toward the right in the frame, your robot should start turning right. If the Vector heads toward the left, your robot should start turning left, etc. We have example programs that run on an Arduino that show how to do this.

Detecting Intersections and "branching"

Intersections can be challenging for a line-following robot because they can take so many different shapes and forms. Will your robot deal well with T-shaped intersections, but fail when it sees Y-shaped intersections? What about 4 or even 5-way intersections? What if the intersections and lines are hand-drawn or “curvy”? Pixy2 can handle these cases. For example, Pixy2 tells your program that it has detected an intersection ahead, how many branches are in the intersection and the angles of each branch. Your program can then decide which branch it wants to take and then communicate it to Pixy2. Pixy2 will then make the branch its new Vector line, and your robot will in essence “take the branch” by following the new Vector's direction.

Technical specs

Processor: NXP LPC4330, 204 MHz, dual core
Image sensor: Aptina MT9M114, 1296×976 resolution with integrated image flow processor
Lens field-of-view: 60 degrees horizontal, 40 degrees vertical
Power consumption: 140 mA typical
Power input: USB input (5V) or unregulated input (6V to 10V)
RAM: 264K bytes
Flash: 2M bytes
Available data outputs: UART serial, SPI, I2C, USB, digital, analog
Dimensions:1.5” x 1.65” x 0.6”
Weight: 10 grams
Integrated light source, approximately 20 lumens