First VEoE Installation – Wink Inc

I spent the last week installing a Video Editing over Ethernet (VEoE) system at Wink Inc Productions in Jackson Hole. I’ve installed 2 Xsan systems here in Jackson, but they are expensive and have a fairly limited target market. Xsan’s greatest asset is its ability to deliver super high speed data to every connected system over fibre connections. You can get enough speed to do uncompressed HD editing on every system. But my experience over the last few years has been that you don’t need that much speed except on the online editing system, if you need it at all.

History:

Every room is wired, but the top switch had no power and the bottom switch had no internet

At Wink Inc, they have a separate online system with its own Xserve RAID connected via fibre. So for them, speed was not an issue because they had speed where they needed it. Their main goal was to connect all the systems together in a Storage Area Network (SAN) that could serve DVCPRO HD to multiple systems. The first time I walked in there, I was greeted by a bookshelf full of external drives. To the tune of 30TB+. They had 4 G5 edit systems, along with various iMacs and laptops. They would offline edit their projects on these systems with the external drives at either DV or DVCPRO HD, never really collaborating between the systems or editors. Their entire facility was wired with anywhere from 2 to 6 ethernet lines per room, both Cat 5e and Cat 6, but most of the computers weren’t connected. They had 2 ethernet switches, one of which wasn’t even plugged in.

Their main use of the network was wireless access to the internet for the office computers, but not the editing systems. I had been using shared storage for about 6 years and Xsan for 3, so I was blown away by how much potential was being wasted. Now, I’m not mocking their setup, but I’m starting to realize that their network was more the rule than the exception in most small production facilities. Many office networks are installed by local ISPs that are most concerned with getting everyone on the internet, not for file sharing and certainly not for video editing. There’s also a shared notion among video editors that you never ever connect an editing system to the internet. With increasing need for editors to share their cuts with producers and clients around the world on an almost daily basis (among many other reasons), this is an old adage that needs to be put to bed.

First of all, I would like to thank Bob Zelin from CreativeCow.net for his insightful article: Build your own affordable SAN — that works! I read it back in September and it really got me thinking about how to deliver Xsan-type sharing to production houses that don’t need the speed or expense of Xsan. I would also like to thank the guys at Small-Tree Communications where I bought the ethernet cards and switch. They have been extremely helpful from the planning stages to helping me tweak settings on the smart switch. They even let me borrow a 4-port ethernet card to test in a G5, but more on that later.

Mac Pro with 10GB of RAM and 9.55TB of RAID-5 storage

The Server:

Speed test on server

This was the easiest part. Wink Inc wanted a lot of storage right away, but also room to grow. I decided to go with a Mac Pro with a RAID card from CalDigit. The card can support the 4 internal drives in the MacPro (the boot drive gets moved to the second optical bay) and up to 3 external HDElements with 4 drives each. For this setup, we went with the 4 internal drives and one HDElement. At RAID-5, this gave us 9.55TB of space. The speed tests were impressive. We also installed a 6 port Gigabit Ethernet card from Small Tree. This is the key to getting the 400MB/s of read speed to the clients. Using link aggregation, the 6 ethernet connections are condensed into one 6Gb/s connection. Well, thats not really true. What you get are six separate 1Gb/s connections to the server. This allows 6 different edit systems 1Gb/s connections, whereas without link aggregation, 6 systems would be sharing one 1Gb/s connection, or 166Mb/s per system, which is not nearly enough bandwidth for video editing.

After the installation was complete, I ran some speed tests on the clients. Knowing that 70MB/s is the working maximum for Gigabit Ethernet, I was a bit surprised by the test results:

G5 with 6GB of RAM and AJA Kona 3 capture card:

Write: 95.6 MB/s

Read: 102.8 MB/s    (7.44 Streams of DVCPRO HD 1080i60)

MacBook Pro with 3GB of RAM:

Write: 75.2 MB/s

Read: 97.1 MB/s   (7.03 Streams of DVCPRO HD 1080i60)

The Mac Pro has so much available processor headroom – serving 6 edit systems at once never takes more than 10% of the processor speed – that this computer is also used as an edit system.

48-port EdgeCore L2 Smart Switch - far left blue connections are for link aggregated server

The Mobile Server:

Earlier I mentioned borrowing a 4 port ethernet card from Small Tree. In the planning stages of this project, Wink Inc asked about creating a mobile server similar to this Mac Pro house server to use at various near-live action sports events around the country, such as the Quiksilver Natural Selection Snowboard comp held here in Jackson last year. They didn’t want to spend more money on another Mac Pro, so they asked about using a G5. I told them that a G5 could be used to serve a few video editing clients, directly attached to the 4 port card. But then they asked if the server could be a capture system with an AJA Kona 2 card, while a few laptops were connected to it for editing. It sounded like it probably wouldn’t work as it would be pushing the limits of the G5. Well, it worked! We took a Dual 2.7GHz G5 with a 2Gb/s fibre channel RAID, connected 3 edit systems to it, and set it to capture DVCPRO HD from an HDCAM deck. I opened Activity Monitor to keep an eye on everything. With 2 laptops and 1 G5 connected and editing DVCPRO HD files, the main G5 kept capturing away without dropping a frame. Using this info, Wink Inc decided to use a Quad 2.5GHz G5 (instead of the tested Dual 2.7 G5) for their mobile server. Instead of fibre, it will have a CalDigit RAID card similar to the MacPro, but also sport an AJA Kona 3 capture card. Of this entire project, the mobile server experiment turned out to be the most exciting part. In this case, the mobile server is being built with components that the company already owns. The cost of adding a multi-port ethernet card to make it a server is minimal when compared to the benefits. This fact is probably the most compelling thing about VEoE to the average production house.