On Sunday, 82,529 fans descended on MetLife Stadium in Rutherford, NJ, for what would be the most watched television event in history, to the tune of 111.5 million viewers. As the fans took their seats before the Super Bowl began, they were given a bag of goodies to help combat the expected cold temperatures, despite the balmy 49 degree kick-off temperature. In the bag of goodies, under the gloves, hand warmers, tissues, lip balm, and scarf, was a small black beanie in a bag with a warning label on the package telling fans not to open the bag until instructed to do so by the stadium announcer. Most fans followed instructions, but I had an idea what was in the bag and couldn’t resist opening it early. Most fans didn’t notice, at first, the three small LEDs and the infrared receiver mounted to the front of that hat, cleverly hidden within the logo of the Pepsi Halftime Show. Most fans didn’t realize that they would soon be a part of the largest human video screen ever created.

Back in the middle of summer of 2013, a Montreal-based audience/technology interaction company made a pitch to the producers of the Super Bowl Halftime Show. Soon after, they were invited to MetLife stadium to do a demo with 2,000 custom-produced pixels. Ten minutes into the demo, the producers were sold. A few weeks later, contracts were signed, and PixMob was off to the races, errr, Super Bowl.

I recently spoke with Vincent LeClerc, PixMob’s creative technology officer, to gain some insight into what it took to bring this show and this technology together in front of 111.5 million people. PixMob specializes in engaging the audience through video technology with which the audience interacts. They try to make the audience a part of the show, giving them a deeper experience. The genesis of the company was inspired by research Leclerc was involved with at MIT. Since then, the company has grown, and it now has a portfolio featuring work with Arcade Fire, Black Keys, Lollapalooza, Microsoft Kinect, and Celine Dion, to name a few.

The company is constantly developing various pieces of technology to do what they do. The recipe is always improving. When LeClerc and his team set their sights on the Super Bowl Halftime Show, they took what had already been prototyped, tested, and added some modifications and improvements, before ordering close to 100,000 of the devices. The scale of the show was larger than any they had done before. They faced challenges of getting batteries to perform well in cold weather and how to mass produce hats. The small chips embedded in the hats included three RGB LEDs wired together to create one pixel, not three pixels. The information for each hat was sent via infared signal. Picked up by the onboard IR receiver, the signal went through a tiny 8-bit microprocessor and then the LEDs rendered the resulting pixel.

Getting the signal to the hats was a whole new challenge. Since this technology was developed in house over the last three or four years, they knew what did and didn’t work. When PixMob first started experimenting with these systems, they would use fixtures similar to what those of us in the live entertainment industry are used to seeing: wash fixtures, lekos, etc. These fixtures came with the features we are familiar with, such as shutters, gobos, visual effects, etc. These fixtures would pan and tilt, washing the audience with invisible IR waves so the result was only seen on the pixels embedded in the audience.

The newer technology  used in the Halftime Show is more like a video projector projecting content over the audience. The projectors are fabricated entirely in house. It’s not as simple as replacing a traditional projector lamp with an IR lamp. Using these custom projectors, it now becomes possible to wash the audience with real-time content. If you take a look at PixMob’s demo video, you can see live camera content playing over the audience in a test. The PixMob VT, the IR projector, receives a video signal over SDI.

According to PixMob, it then sends out pixel information to over 100,000 locations, which sounds like a lot. But when you consider that a VGA resolution of 640x480 has three times as many pixels, we aren’t dealing with a giant raster size here. This doesn’t really matter, because even if your raster size is small, your canvas is enormous and has no trouble being breathtaking at a small resolution. While this new technology affords more creative opportunities, the fixture-controlled version is still used on smaller shows. The older technology can be more effective on a smaller audience than a projector would be.

Leclerc says that most of their work is working with people. It takes many talented minds behind the scenes to develop the technology, but if the audience doesn’t engage with the medium, or in this case, put on the hats, the show fails. The Halftime Show was a much more passive version of what PixMob is used to working with. For this show, the audience merely had to open the hat's packaging, pull a tab, and wear the hat. PixMob has developed other devices, ranging from beach balls to devices with specific audience interaction like hand clapping or head shaking. It was hard enough to get the audience to donn the hats correctly. When they performed a test at the two-minute warning, only a small percentage of the audience put on the hats.

Working in several sections of the entertainment industry, PixMob is used to having different degrees of control of a production. On some shows, they provide a full turn-key solution from technology to content to programming. This is effective because they know how the product works and what looks good on it. On the other end of the spectrum, they provide the technology, leaving content and programming to someone else. Leclerc says they are open to whatever set-up a show is looking for and happy to work with anyone.

For the Super Bowl, the content was handled by a local New York video production company. The producers decided to play it conservatively due to the scale of the show, the time crunch, and wanting to make sure it was an amazing show. If you look at the demo video of the product, you’ll see it’s capable of much more than was seen in the Halftime Show. From being at the stadium, it’s hard to compare the live experience to the TV version. TV producers are still learning how to shoot it. It’s hard to capture a circular wave running around the stadium, but it’s really impressive to see it live in person.

PixMob had about two weeks in the space when they usually ask for three, but it’s hard to get an NFL stadium for that long. They could really only do their programming at night and had to beg the NFL to shut off the lights to aid in their programming. They finally got the NFL to agree to have the cleaning and snow removal personnel do their work under dim light and not the stadium’s lights. During the tests, they had several production assistants running around the stadium placing dummy hats in different locations and constantly replacing batteries so they could test and program the system.

Leclerc believes this is only the beginning for this technology and for PixMob. The company used to do a wider array of work, but they have become so successful in this area (he says requests are pouring in “hourly”) that they are focusing on this aspect. He predicts this technology will become more mainstream in just the next year alone. There are many new projects he can’t disclose, but he promises they’ll be exciting.