In the summer of 2007, freetards (me too), the OSC and others were calling for the BBC to make iPlayer cross-platform, and the Trust had committed the BBC to doing this.

Slated for a Christmas launch, the system only supported Windows Media with DRM and P2P delivery. There were some concerns from ISPs about how this might affect them.

Cross-platform support has always been a source of grief ^[1] leading us to duplicate systems and to reduce functionality in order to support multiple formats equally.

The problem wasn't going to go away and with the expansion of video and audio into devices such as games consoles, it was getting worse.

We needed a different approach, to divorce our content production from delivery format and method - something quickly adaptable to new devices and the unmentionable "C word": convergence. ^[2]

I'd had an idea for a system to do this festering for a few years - the mother of all VoD systems. I'd proposed this at the start of the iPlayer project and now it seemed time to just get on with it.

Armed with a couple of trusty developers, Tom and Dickon, a demonstrator was quickly built. We called this project "BBC Redux".

It's a video-on-demand test bed where we can try out the systems that acquire, store, search and deliver content. It's like your PVR but much larger and can convert and stream content to suit playback on more than just TVs.

So how does it work?

To start with we ingest DTT, though sourced from multicast, as direct programme feeds take time to organise. There's a server that records each stream and segments it into programmes just as your PVR does.

Each content recorder (there may be more than one of each subsystem for redundancy) offers its files to the content manager(s) that look after storage distribution and management. The content manager(s) determine which content store to save them on. Then there are front-end servers to test applications for delivering to users and a stack of transcoders that are the glue between the neutral high quality storage format and what the users want.

In the summer of 2007, we showed our prototype to Ashley and the FM&T board.

The point was made (streaming video could work) and iPlayer was set to have Flash streaming for the Christmas launch (after some quick work by the iPlayer team). A bonus for the ISPs is that Flash streams are approximately a third the size of the downloads and people only transfer them once. Currently, around 90% of BBC iPlayer use is Flash streaming.

We also demonstrated to the board a set-top box that plays our multicast live streams and iPlayer VoD - part of the longer term aim to make internet TV more usable in the home.

We continued to experiment with BBC Redux, making it work for various games consoles - PS3; PSP; Wii - and when the iPhone came out in November 2007, we added iTouch/iPhone and later 3G phones.

Some of this work was later used for iPlayer beta sites (iPhone is 3% of use).

At Mashed08, we let the public see and use one of our development front ends for the first time with the freshly-hacked that week subtitle support - so new it was still being worked on through the Saturday.

Redux was built to support the rapid development of new services and was designed to scale in many directions.

The transparent on-demand transcoding is how we can quickly adapt to new devices. When iPlayer chose to go Wii, they wanted a slightly different encoder profile than the one we'd used for demos. No problem: we defined the coding profile, allocated a subset of the transcoder pool to them and let them trawl through the store collecting all the programmes they needed for the seven day window. A few iterations were needed as the launch deadline approached, so more transcoders were added to their pool as time ran out. The risk was too high to try all this in the iPlayer live production systems; they could be adapted later.

Redux has been quite versatile - over the past year it has grown in use as a demonstrator of a tapeless world - a way of trying out ideas like:

• instant access programme compliance store
• immediately accessible archive - no delays waiting for a DVD/VHS in the post, so programme teams are using it for research.
• a usable infinite archive - why would we ever throw programmes away again?
• adaptable to produce new content formats on demand - when www.bbc.co.uk added WM alongside Real, nobody had the content readily available to make WM versions of all the previous content; now it's not a problem
• IPTV
• closed user group access to content, either in production or, as is now being developed, for TV listing magazine reviewers who used to watch on postal DVD

Redux consists of three racks of equipment - two are storage nodes - 342TB according to the disk manufacturers, 297TB of usable space with 152K programme files so far.

Full size images in our Flickr account [1 | 2]

There is also a baby one feeding iPlayer as, being a development system, we like to break the main one regularly.

Redux had its first birthday in July. We hope it will have many more before the experiment ends.

§

^[1] Used since 1995 and selected as the most cross-platform system we could find, Real still met resistance from some Windows users: often corporate IT wouldn't let them install it or they and Linux users didn't like the free player trying to entice them to upgrade to a paid-for version or use advertising.

BBC TV and Radio services work on any manufacturers device, they compete on quality and features, not on exclusive access to content. It seemed silly to have to make special content for any particular internet device manufacturer but that's how the market, governed by commercial rather than technical interests, has developed. [Return to post]

For years, we've encouraged the adoption of standards-based systems, developing DIRAC, a royalty-free video codec to reduce commercial barriers to adoption of a common standard. It's starting to make ground in high end broadcasting, but is slow progressing to the internet. In the meantime, we've been supporting H264 and AAC, which are taking off. The aim remains - people can choose their player; we don't need to care.

^[2] Internet media are still much like that start of TV broadcasting where multiple systems were run in parallel before choosing one. [Return to post]

Brandon Butterworth is Principal Technologist, Kingswood Warren, BBC FM&T. Image of Brandon by Chris Capstick. Image of protest by Matt Cashmore. Images of Redux kit by Brandon. UPDATE 24/9/09: Brandon Butterworth is now Chief Scientist, BBC.

Imagine there's no interweb...

...that's unpossible. We have a lot to thank the internet for - besides a new language and LOLcats.

The first 10 years were the best

When I put the BBC on the net, it wasn't a 2001 moment; we had to keep banging the rocks together for a while. We had email, file transfer, BBS, Usenet and piracy since the late 1980s, ours via dial-up UUCP through Brunel University to UKnet.

The USA had proper internet.

I wanted it.

Then the web arrived. We set up www.bbc.co.uk and started playing with mostly laughable content until we found a few like-minded people around the BBC who had more time and material for producing content. Little of the content is still online - except one of the first sites which is available at t-web.co.uk.

From then on (summer 1994), programme-makers joined in or started their own projects, such as the BBC Networking Club. We grew on donated time and hardware, experimenting with technology and content.

Having drawn radio and TV content into the website, we also put internet into programmes. Email feedback seems trivial now, but being able to respond to a programme and have the presenter respond to you on air was far simpler than a phone-in. IRC questions into live political chat shows hooked News and Radio 3's Facing the Radio programme produced live from user-generated content and streamed the programme.

We had our first foreign (Hungarian) language site too, though we later found that the producer had jumped the gun on a co-ordinated effort by World Service to do many languages, such was the enthusiasm.

Some projects, including the BBC Networking Club, found the expense of running their own independent services too much to continue and returned to the main site. This aggregation was to prove essential in growing the site and making it one of the most popular and cost-effective on the net. As BBC budgets were tight, we reduced the cost of adding a new site by having it share the same resources and technology as the rest: add a new feature for one, and they all could add it. This meant less waste, too: no idle resources another needs.

Keeping it together let us negotiate better prices and had a huge impact on use - there was one brand and everyone knew where to get BBC content. There had been lots of discussion over one domain or many. This was partly decided by squatters' use of some obvious addresses: if we used random domains it was likely that we'd eventually find that an obvious one was already a porn site. Guessing a site name within the site meant the site navigation and search could help them find the right one, guessing the domain either got you an unhelpful message, someone else's site - or just an error message.

That's not to say there weren't disasters.

Radio 1 inhaled deeply for its June 1996 site relaunch. The redesign was leading-edge, dynamically generated from Dynamo - a Java server engine for publishing dynamic content. Everything came from the database, which also stored user data for the really dynamic elements such as message boards. It also crashed when more than a few people used it at once, leading to a quick hack to make a regular site from the content. Sadly, lots of interesting features lost their edge but it now worked and the Shockwave elements were fine.

This shifted the risk tolerance for many years. It was realised that a working site was more valuable than the shiniest possibility, and that what is impressive in development may not scale when faced with a BBC size audience.

It wasn't all bad, though: the site also led in other areas that did work. It featured an early version of the Radio Player. We set up streaming of 30+ live programmes per week (around 80 hours) that played on demand until next live transmission. There were also lots of live webcasts from events such as Glastonbury made up of audio with a updating webcam, as video streaming hadn't been invented yet.

The UK servers over time: one rack, to three, then two rows of nine

As the site grew, we hit the limit of our Pipex line and set up some servers in Telehouse Docklands where higher-speed connections were cheaper. This was important with all the streaming that became popular as the webcasts had content not entirely covered by Radio or TV.

"Nothing special," we thought in 1996, "who wants to listen to a boring Budget speech?" That November, though, lots of people did; the 10-times peak for events became the norm, and the traffic started to stick. It helped News win the argument for the Election 97 site, though there were only six weeks left when we got approval.

The budget was too small and bandwidth was cheaper in New York; I designed an architecture to exploit that with two sets of servers - one in Telehouse Docklands and one in Telehouse New York, with their own special DNS server than directed people to appropriate servers farm for their continent.

The growth of the New York servers

We also made a system for distributing content to the servers, called ftpborg. The servers were scavenged from other projects. Most of the site - around 8,000 pages - was made automatically from the election computer via CPS built in a few days.

The split site met a criterion that was to figure heavily later - keep the UK users' traffic independent of the rest of the world's. The UK license-fee payers got preferential access; the rest of the world got whatever was left that we could afford to give them.

Chuffed with their success, News followed on with the Politics 97 site, which had our first public video streaming of the Hong Kong handover.

So far, we all had other jobs to do as the site was neither official nor funded. Options were considered including not having one or giving others the content. Pressure mounted, meetings went on and on... and then something happened. There had been a car crash and reports claimed that Diana was in it. There was nobody around to update the homepage, so I worked with a friend at World Service, who was in on the Sunday updating their site, until News arrived. We added a video stream while they built a tribute site. On Monday, we started planning the funeral webcast and by the Saturday, we'd organised a syndicated stream fed from our site to EU ISPs and AOL.

By a week later - 10 September - the response to the Diana coverage had convinced everyone that the internet would be big and that the BBC would be there - properly. With an October deadline, there was no point continuing with meetings. A committee wasn't going to make it. A ninja squad was needed.

I got a small bucket of cash and got told to do whatever was needed.

We got a bunch of shiny new Sun servers - 64 bits, too - installed them in Docklands and New York along with large internet links and updated lots of the software ready for the launch at the end of October.

The next 10 years were the best, too

News launched first in mid-November; not expecting the main infrastructure to be ready, they'd rented servers from an ISP. They had our first video-on-demand service: the One, Six and Nine O'Clock News in streamed video - as the iPlayer does today.

The main site launched on 15 December.

1998 was year of events that were fun to webcast - Glastonbury, Cricket, Golf - though less interesting on infrastructure development, as some applications were requested that failed to deliver: frustration all round.

In April we set up a 24/7 Arabic stream for World Service in case war broke out again. It didn't, but the stream was left running as it was too popular.

News was having performance problems with its outsourced servers, including ISPs having trouble reaching them. News's own ISP didn't have an open peering policy, leading to questions from others as to why they should pay to get to the News site when they peered with us on the main site for free. This came to a head in November; our offer was to add News hosting to the New York server farm, relieving their servers of non-UK traffic. This helped, and as their ISP was pulling out of the hosting market, News set up a set on the BBC network in the UK too. This worked well with the huge growth.

The technology stayed the same for some time; we spent the next year tweaking it as applications and sites were added. I built up an operations team - one that had started in early 1998 with a couple of contractors - to handle the day-to-day support needed by the content producers and developers. An important part of their work became security checking developers' code. The BBC was promoting internet use in the UK, bringing in new, naive and vulnerable users. The last thing we wanted was tabloids running "BBC hacked" headlines and users being put off.

I was aiming for one team looking after operations on a technical and editorial (webmaster) level, but after hiring them, politics pulled them apart into two divisions - though they worked closely together.

Kingswood Warren, August 1999

The site was getting quite busy; armed with the traffic we'd generated, it was time to play the next level: our own ISP.

So far we'd gone through ISPs for internet connectivity. Rather than use one to forward all of our traffic to all of the others, we had the option of delivering it to each of them directly - but that was only sensible in certain circumstances. For the UK, it meant that we had to be an ISP and join LINX where most exchanged traffic.

See the network develop.

Above is May 2000: see also Oct 2000; Jan 2001; May 2001; Dec 2001; Nov 2003; Jul 2004

I'd been looking at this for some time, but finally in late 1999 found a way through the LINX membership rules that had prevented us so far. LINX worked well, and similar arrangements which we made in New York later expanded to EU and USA west coast. We still took transit from a couple of ISPs to cover failures and to reach ISPs not present at locations we'd built our network into - around 15% of the total traffic - as it was cheaper.

As other high traffic sites grew, many followed the same path to self-hosting.

The Domecam

Having installed new infrastructure, we spent 1998-99 using it, taking on more content and applications with entertaining live webcasts. At Glastonbury, we fitted a remote control camera to a pole by the cow shed with visibility of the whole site (nobody wanted to clean that kit when it came back). We set up a Domecam on a lighthouse - Trinity Buoy Wharf - to watch the Millennium Dome being built; it lasted longer than the Dome. We did webcasts from all over - bird sanctuaries where BT delivered a phone line to a box in the middle of a field, and the Zambia desert for the 2001 eclipse, streaming live video via a satphone link. We set up Outside Broadcasts with a IP-over-satellite link so more webcasts could be done from around the UK.

Plymouth webcam, May 2001; One Big Sunday, July 2001

Our Kingswood site did need some attention. The computer suite was completely rewired; the now inadequate wall-mounted aircon replaced by large floor standing units. This was done with everything live: replacing the power wiring and main feeder meant running for a day on a mobile generator with lots of hired-in theatre lighting wiring around the floors.

An attempt to have a unified CPS didn't work out, though quite a lot of work went into testing packages. IBM reviewed the infrastructure delivery and operation reporting we were considerably underspending compared to market norms, which quelled thoughts of outsourcing it for a while.

Y2K reared up and distracted us for a while: we were certain that the infrastructure was fine, but to comply with BBC policy we had to upgrade systems. As a test, we left one unpatched. It was fine. The only problem was content: scripts displaying 19100 instead of 2000 on web pages. It was quickly fixed as we had most of the ops team on site for our own New Year's Eve party and to be around just in case the world failed as some predicted. We didn't use the special phone BT installed.

Kingswood Warren, July 2001

In September 2001, I was sat in an operations meeting when the pager went off and didn't stop: something big was happening. There was a massive influx of traffic to the site - a DoS attack, it seemed. Damion called us back: "there was this plane...". We turned on a TV and saw a burning World Trade Center tower. Then another plane. Ops worked on keeping the servers happy, raising the webmaster and News to agree sheddable load. This was the first time, so it took a while to get a new light home page in place. Our New York server farm was two blocks from the WTC site; it survived but suffered as power failed. The dust eventually clogged the generators and there were problems getting in fuel. The only outage was in the days after; we covered that by moving all traffic to London. The sites were designed to operate as hot spares for each other. We had planned around London suffering at some point, but it was the opposite.

BBCT's first birthday party, April 2002

We resurrected podcasts in 2002. Until then, we'd been providing audio in Real, originally downloadable until we had to disable that due to rights problems. All along, we'd been trying to promote open standards for access to our content, but sadly support was low. I'd wanted to use AAC for audio streaming in 1998. Some of our audio engineers were working on the standard, but the market was dominated by proprietary systems promoting their own codecs instead.

When Ogg Vorbis started, it seemed that finally an open standard would emerge and be widely supported. We ran a trial of streamed and downloadable programmes which some used on portable audio players - podcasts in all but name. Really, we needed to do MP3 to make it easier for general use, but due its associations with piracy, it wasn't acceptable to the rights holders.

I had similar ambitions for H.264/MP4 video [pdf] as a common format which is now starting to take off many years later. To control costs and make content universally available we've always looked for common standards so that - as with TV and radio - the audience can choose which manufacturer or player to use and so that the content only needs to be produced once, rather than in many formats. There is a significant cost for additional formats that could go towards new services instead.

Packing to leave Kingswood, September 2002

So far, I'd been running the department in Kingswood which housed the operations and development teams, master content servers, streaming audio and video coding and content ingest. Then, in 2002, BBC Technology formed, bought the remains of a dead dot.com and decided to move both to a site in Maidenhead. We lost half the staff in the move and new management was brought in to look after the combined operation, later to be sold to Siemens.

The master content servers and stream encoding were at Kingswood until 2003

The infrastructure remained much the same after that, with obvious increases in capacity needed to keep up with demand. In 2007, to reduce costs, the USA network was closed down and all hosting consolidated over here...

...taking us back to where we started: commercial hosting in just the UK.

Brandon Butterworth is Principal Technologist, Kingswood Warren.