The best server for the project is the one for which the user understands the capabilities and limitations.
All servers are not created equal. Each media server embodies technical contributions that have made all media servers better, and the best server for the project is the one for which the user understands the capabilities and limitations. Fortunately, the world of media servers is very open-source. For example, Nigel Sadler from Green Hippo has welcomed every insane idea with, “Why yes, I’d love to help you win first place at the mad science fair!” The same can be said for Kevin P. Morris of coolux (Pandoras Box), Matt Corke from PRG (Mbox EXtreme), Ash Nehru of United Visual Artists (d3), Curtis Cox of Martin Professional (Maxedia), and Jorge Moritz from e:cue.
There is also a growing network of independent users, self-supporting other media servers including Watchout, ArKaos, Jitter, and Isadora, just to name a few. There are almost too many servers to mention, and that’s a good thing.
As a side note, the benchmark resolution for discussion in this article is 1920x1080p 30fps at 24kbps constant bit rate, 16x9 aspect ratio, sometimes called 2K, and not the 4x3 aspect ratio 1024x768 version of HD. Some of the media servers mentioned here can output up to 4K, and at least one has no resolution restrictions at all. However, at that resolution, you will get only one layer of playback utilizing a modified clip of some sort. What’s great is that multiple servers with custom media can be ganged together to get up in the millions of pixels range, even in 3D. A good example of this is the Comcast Center in Philadelphia, designed by David Niles of Niles Creative.
As server utilization has become common practice, we have become pixel-hungry. SD has begun a death rattle, and HD is becoming the standard. HD has been a part of my daily routine since Disney’s The Little Mermaid on Broadway in 2007, where we utilized Green Hippo’s Hippotizer HD media servers. Since then, our use of servers outputting 1920x1080 HD has grown to include coolux’s Pandoras Box and PRG’s Mbox EXtreme, and I hope to include more in the near future. As for what is “El Server Supreme,” it’s virtually impossible for one to claim that status, as they are all different, and all have different applied sciences, which make them right or wrong for any given project. As always, there will be preferred servers. My current server utilization includes two servers of different pedigree to get the desired effect.
Which Server?
In formulating your own decision on what server to use, bear in mind the limitations of each. For instance, what’s the max number of layers or stills a server can playback before system performance noticeably diminishes? The common misconception is that an HD still doesn’t take any processing power because it’s a still. The truth is that most servers are only resolution-aware. HD stills or HD clips are the same to the server, because they are the same resolution. The server treats them both as HD and, therefore, consumes more system resources.
Another common misconception about HD is that it is unlimited. HD playback can be measured in the strength of the graphics card or processor. Windows-based servers, such as the Hippotizer or Pandoras Box, have the option of writing custom decoders for their preferred video cards, meaning most PC servers rely more heavily on the graphics card. Mac-based servers, like the Mbox EXtreme, rely on the processor.
How each server deals with clip-loading and decoding of clips will govern your programming. Plan on making several different test clips at different bit rates until you land on one that gives you acceptable output quality and playback capabilities. You shouldn’t blame the server if you overload it. For example, two to three 30-second HD loops can play, while other times only one giganto 30-minute clip will play. Some servers have error logs that can tell us at what point we started to overload a server. Pandoras Box has an excellent error log. While reading it, you can find exactly the moment you’ve begun stressing or crashed the server and what clip took you over the edge. If you receive what we like to call “jackpot,” the log basically reads, “Congratulations, you have crashed the server,” with “jackpot” defined as: “A system resource strain that accumulates until a release of all system resources happens, resulting in a jackpot error message.”Used in a sentence: “Holy f@#k, jackpot!”
Green Hippo has a live status tab where you can actually monitor your graphics card’s health by frame rate and available RAM, including a log that keeps track of everything affecting the server’s health, and you can monitor while programming and running back cues. HD programming requires responsibility, and you have to be prepared for a shell game of utilizing system resources to avoid the dreaded jackpot error message.
So, are you feeling lucky? Or are you ready to do some homework? On the east coast in New York City’s SoHo, you have the very helpful XL Video LED Lab, kind of the neutral ground for current, emerging, and future technologies like the UVA d3 server platform. On Long Island, you have AG Light and Sound, and in New Jersey, there’s Scharff Weisberg. If you don’t live in the tri-state area, you can always rely on TMB, as those guys have ongoing training for Green Hippo all over the country. The same can be said for coolux’s Pandoras Box and PRG’s Mbox EXtreme. In Chicago, Dunaway Designs and Upstaging have a great selection of servers in their studios. All you have to do is look up your local video or lighting vendor.
Before beginning a project in HD, you need to ask yourself the following questions:
• Why do you want to use HD?
• Will your project actually look good in HD?
• What do you expect to do with the server?
• What server, or combination of servers, can complete the whole project?
• If you’re pixel-mapping an 800-pixel wide by 100-pixel tall LED surface, do you really need HD?
• Are you pixel-pushing just to be cool?
Servers have become much more than just DMX-controlled clip or DVD players. Media servers are living pieces of the entire design. Currently, there seem to be two worlds of server programming. The first is prebaking media in the bat cave and then utilizing the server as playback only, with no effects. The second is the building-block method, using the server as a live compositing, layer-based system. The programmer edits these elements the same way a non-linear editor/effects/colorist would do in motion picture production. The major difference with live compositing is that we can edit and playback in realtime, with multiple clips as building blocks. Color changes, speed changes, even live effects or edge blend changes over time, just to name a few, can be employed. Designers, directors, and artists can ask for a change, and they can receive it almost before they are finished asking for it. In the prebake-in-the-bat-cave method, if a change is requested, you have to reedit the entire piece and then render a new complete clip. These are choices that depend on each designer’s style, and either will get the job done.
Some systems employ a single server, single output. Others employ multiple-server platforms that interact with each other via live controls for VJ takeover. We did this for WES 2009 in Orlando, with Hartmann Studios’ production designer Greg Sullivan. Sullivan controlled four layers of a Hippo (in dual-output mode) and used an ArKaos setup, VJ-style, and I employed an MA Lighting grandMA to run the other half of the Hippo, while screen-thieving the ArKaos output, featuring a live performance by Will.I.Am. Thanks to Sullivan’s incorporation of VJ-style and his production design savvy, the attendees were completely blown away. Further complexity can rear its head in the form of MIDI sync, DMX control, full-interaction of multiple users in different locations, or some variety of motion-tracking. The list goes on and on. We call it “mad science.”
Next Page: Navigating The Process
Navigating The Process
The practice of using multiple server platforms simultaneously in a group environment can be called open programming. This is similar to a DJ, a VJ, and a camera director working together to make a better audio and visual experience for the audience. Each programmer/operator has access to his portion of the system, while the system also has access to the system. This can be accomplished with a lighting console that is cued or run manually, a MIDI controller, SMPTE or MIDI timecode, server timelines, beat bridges, screen-thieving, drawing right on the media as its playing, or doing all of the above simultaneously. But to be successful, everyone has to respect the system’s resources and be able to monitor the use of those resources.
First, the designer, programmer, and compositor must work together. The programmer has to build the playback surface in a multi-touch framework. This is accomplished in a few different ways. The first is remote desktop to the servers, or, in the case of Green Hippo, you have Zookeeper dongle-protected software that can run on any computer running Windows XP Pro or even on a Mac running Windows XP Pro. For Pandoras Box, you have a laptop from coolux running the dongle-protected Media Manager application. Both of these applications allow you to look at all the servers on the system and encode/upload/distribute resources, as well as view playback in realtime, just by selecting what you want to monitor. What is also fantastic is that Pandoras Media Manager and Green Hippo Zookeeper have the ability for a networked workgroup to access the system, so the programmer can look at programmer stuff, and the designer can view designer stuff. The operator can manipulate operator stuff, and the compositor can do compositor stuff, all without interrupting each other.
Second—and some operators may cringe—is running the offline software available for most consoles on a remote computer, so the designer and stage management can see what’s going on. Usually, the cue list is sufficient for stage management. For the truly brave, you route the command line down to the designer and even provide a fixture sheet, so you can program instead of being asked, “Are the dousers closed?” I have found a digital cue list is far better than a paper one, as everyone can see when things are working and, more importantly, when to wait for the system to complete a task.
Third, audio, lighting, stage management, and orchestration all must be incorporated as a part of the workgroup. Server revolution has introduced the collaboration of all departments. A great example of this was on Robert Lepage’s La Damnation de Faust, now part of the New York City Metropolitan Opera repertoire. This production spanned spring and fall of 2008. Designer Holger Förterer and I combined the Ex Machina HD interactive servers with Hippotizer HD media servers to create a completely interactive playback system so revolutionary that we built a custom DMX512-channel profile for the grandMA, so we could add interactive elements on the fly. Each day during rehearsals, the creative team would come up with a new way to control the randomness of the interactivity.
Every evening in the edit suite of the Met, Förterer would generate the code for the interactivity, and I would build that new interactivity into the programming for the next day’s rehearsal. This would have been impossible without the creative team—Lepage, Förterer, production manager Neilson Vignola, video artist Boris Firquet, technical director John Sellars, board-op/projections department head Ted Sydor, and yours truly—working collectively with automation, audio, orchestration, lighting, and stage management. This was definitely the largest integration of workgroups I have ever been fortunate enough to experience.
The system had the Green Hippo HD media servers and Ex Machina HD Interactive servers outputting 1920x1080 interactive projections to front- and rear-projection surfaces that made up the five-story set. The Hippos provided foreground and background to the Ex Machina servers, which outputted the final composite of interactivity and final images in the projections. The inputs to the Ex Machina servers included media with live-motion tracking; live-image capture; audio feeds from the orchestra, chorus, or principles; feeds switching infrared cameras and HD cameras; feeds from the motion control system that opened and closed the front screens on the set; and projector douser controls, just to name a few. Ex Machina’s custom HD servers, running Förterer’s custom programs, generated the image, the animation, and rules that governed the interactivity, completely in realtime from code.
Therefore, you experience the image, along with its interactive elements, literally as it’s being created. We then used custom interactive controls, run from the grandMA, to govern any elements we could insert from the real world, some of which included wind, water, leaf blowing, leaf wilting, birds flying, birds’ reaction to audio, candle flicker and blow out, and everyone’s favorite, fire. This system afforded the capability of complete media immersion, which we called “controlled random.”
A brilliantly integral part of the whole system was Sydor of the Met. In order for the system to operate, custom controls had to be made, so the grandMA platform could be used down to the granular level. Every scene was its own fixture, and each fixture was capable of up to 25 custom controls. Each control governed how the physics of interactivity reacted to live input, media input, or code-generated input. The grandMA would preset the interactivity, load it for Sydor, and then he would have the option to increase or decrease the interactivity as needed. Then, on the next go, the system was programmed to turn off and preset the next interactive controls.
Because of this custom control, we now know the limit of custom attributes a single fixture can contain in the grandMA console, and the answer is 128. Hopefully that will change with the grandMA 2. Fortunately, Sydor welcomed the challenge of operating and installing the extremely complex system, which would have to be uninstalled after every show and reinstalled in the four-hour changeover before each performance. He was in complete sync with the stage manager, the performers, and the orchestra. There were moments in the show where Sydor had to track multiple performers, listen to stage management, monitor the projection system, and be ready to manually override any potential system glitch simultaneously. Bravo!
None of this would have been possible without the full support of Met general manager Peter Gelb and John Sellars. These guys understand what it takes. Met management afforded us a different approach to systems integration with the creative team and house staff, instead of the usual uncomfortable hands-off attitude, where the operator sweats out each press of the go button on opening night. Sydor and his staff were in show positions during rehearsals. This freed me up to make programming changes on the fly during rehearsals, or, as Firquet would say, VJ-style. We were even able to run a remote cue list from the grandMA in the booth, all the way to the stage management position, keeping Sydor in constant communication with stage management.
Another bravo to Scharff Weisberg’s Randy Briggs and Barry Grossman for routing KVM, projector output, and VGA signal to multiple locations all over the Met, so everyone could stay in sync. As Faust is intricately staged, this was key for the illusion. If the stage manager saw that a projections cue hadn’t completed, he could react and wait or ask Sydor to override, as needed. And when stuff broke, there was a systems specialist from Scharff or Ex Machina to walk them through the solution. Come show time, there were no butterflies in the control booth.
The cross-pollination of system design practices that occurred on this project will be felt in the media systems design world for some time to come. The all-star production team from the Metropolitan Opera in New York, Ex Machina, and Scharff Weisberg combined the DNA of multiple HD systems into a fully open interactive space station of which NASA itself would be envious.
Since Faust, numerous jumps in HD and SD manipulation have taken place. Last year, we utilized open programming for Disney’s Step Up - 3D. I used Adobe After Effects to create custom media that reacted to audio playback. Then these clips were loaded into Green Hippo HD servers.The grandMA for video and lighting was triggered by a SMPTE feed from the audio playback, which made the entire system react to the dance battles, instead of the system trying to catch up to them. Director John Chu and director of photography Ken Seng could jump to anywhere in any battle scene, and we were in perfect sync and could achieve repeatability with every take.
Mad science—are you ready to go to the next level?
Peter Vincent Acken has been pioneering media design and media server programming since 2005. He resides in Battery Park, Manhattan, and when he’s not working as a mad scientist/video programmer/live compositor/ anti-sneakernet vigilante, he’s riding his motorcycle with his wife.