BBC Internet Blog - John O' Donovan

BBC News website's content management and publishing systems

John O' Donovan John O' Donovan — Tue, 27 Jul 2010 22:00:00 +0000

As part of the BBC News site refresh we have been making substantial changes to the underlying systems that manage and publish the content.

The BBC has one of the oldest and largest websites on the internet and one of the goals of the update to the News site was to also update some of the core systems that manage content for all our interactive services.

In this post I'll highlight a few areas where we have made some important changes.

CPS

The CPS is the system that manages content production for BBC News, BBC Sport and over 100 other websites across the BBC. It also produces the content for multi-platform journalism, such as the BBC Mobile services and Interactive TV/Red Button services and even content for the mighty Old Skool Ceefax is born in the CPS.

If the BBC website is one of the largest and oldest on the internet, then the CPS has been around nearly as long. As a rule of thumb, if you remember Bagpuss you are older than the CPS. If you grew up watching Teletubbies, you probably are not.

Let's not confuse old with legacy though. The CPS has been constantly evolving and we should say, that when looking at the requirements for the new News site and other services, we did consider whether we should take a trip to the Content Management System (CMS) Showroom and see what shiny new wheels we could get.

However there is an interesting thing about the CPS - most of our users (of which there are over 1,200) think it does a pretty good job [checks inbox for complaints]. Now I'm not saying they have a picture of it next to their kids on the mantelpiece at home, but compared to my experience with many organisations and their CMS, that is something to value highly.

The latest version of the CPS - version 6 - underpins the new News site and has made substantial changes to systems and workflow, but it is still focused on the task of managing content which fits into a general journalistic pattern. It does not try to be all things to all people, and this in no doubt plays some part in its success.

There have been a number of requests from people asking to see more of the CPS but as there is a lot of detail to go into, I'll just focus on a few headline points for now. We will be doing a more in depth blog post on it soon.

Moving to a more structured approach

Some of the major changes in approach are in the Client which is a .NET 3.5 client, taking full advantage of WPF. The screenshot below shows an example image from the CPS which illustrates some new features.

This shows a snapshot of a story editing window. Around this are site navigation and other tools (like Search).

As you can see there is a component based structure to the story content with a Video, Introduction and Quote shown. These components are predefined and can be dragged in and added to the story showing that the CPS is not primarily a WYSIWYG editor. The CPS focuses on content structure because in a world where you are publishing to many platforms that have hugely different rendering possibilities WYSIWYG becomes a pointless feature but there are previews showing the output.

Previously, users could add HTML and Custom CPS tags directly into the story body to control the content presentation and the components, similar to the way you would insert code into your content on Wikis and Blogs. This causes a lot of problems for quality and content structure though, so now these things are managed as components where the user can change the content and behaviour of the component in a controlled manner. We will come on to the importance of that next.

HTML Standards

Another part of the CPS that changed considerably was the way content is published. Requirements over the years have caused features to be added organically to the way content is published, leaving it a bit messy with a lot of layout based on HTML tables. A key goal here was to improve the technical quality of content produced and support standards as we move from to

For example, we are aiming for fully valid pages to be published based on the W3C Validation Checker.

If you look at some of the older pages published you will see they don't pass this test, and some pages, such as http://news.bbc.co.uk/sport2/hi/olympic_games/default.stm, produce a lot of errors:

Ouch!

This is especially tricky to fix where the CPS is pulling in content from other systems or services which don't comply with these standards, but though there is still some work to do here, generally we should be down to 0 or very few errors now.

You will also see in the image above, that older stories are based on the iso-8859-1 character set, whereas new stories will all be UTF-8 encoded for better international support.

Semantic Structure

We will also no longer be using tables to layout the content, instead we will be rendering the pages using CSS layout and only using tables for data.

There are lots of reasons to do this, but some include making the content more efficient, more standards compliant and faster to render. It also allows us to publish semantic XHTML, which means that content blocks are better marked up to describe what they are and has benefits like creating a better header structure to help screen readers.

Better structure also means you will see a more consistent presentation of stories in Google and search engines with, for example, story dates and author information showing more clearly.

This reflects a new content model which is now largely based around a simple and generic data model of assets and groups of assets which are typed (meaning we don't just manage blocks of content, we use metadata to describe what is in the blocks of content) and publishing through templates and services based around Velocity.

Again this is about supporting content standards better as described here for making better use of headings and lists.

Take this example showing how a component is put together.

Previously the HTML would have looked something like this:

But now it is much more structured and would look something like this with headers clearly marking out the sections of content:

Using CSS for layout also makes a big difference to our HTML and makes for a better separation of layout and content. This rather messy layout...

becomes

The table elements used in the first example are gone and the layout relies on CSS to manage the positioning of content.

URL Structure

Finally a quick note on the change in our URL structure where you may have noticed a couple of significant changes. These are the tip of an iceberg of substantial changes we have made to our networks and infrastructure also part of this relaunch.

The first is that our News URLs have moved from the http://news.bbc.co.uk to https://bbcbreakingnews.pages.dev/news/ in order to consolidate our domains. As part of the News site changes this involved us making significant updates to our networking infrastructure to allow better sharing of content across our domains. Moving all our URLs onto the https://bbcbreakingnews.pages.dev/ domain also consolidates some differences which are there largely for reasons no longer necessary.

All URLs should redirect to the appropriate place, but if you do find any broken URLs please let us know.

We also wanted to simplify our URL structure removing much of the baggage in the previous structure for managing different types of content and editions of the website.

Now the structure is basically:

https://bbcbreakingnews.pages.dev/ [SITE] / [SECTION] - [SUBSECTION] - [STORY-ID]

For example:

https://bbcbreakingnews.pages.dev/news/uk-politics-10721364

This has made URLS shorter and simpler.

We considered making even shorter URLs - you will have seen some stories were published this way while we transitioned the site to the new design, such as:

https://bbcbreakingnews.pages.dev/news/10250603

The changes we have made will allow us to make URLs more flexible, and there is more work to do yet on how we might use even shorter URLs (such as https://bbcbreakingnews.pages.dev/10250603) and longer more descriptive ones https://bbcbreakingnews.pages.dev/story-about-something-interesting.

If you would like to know more about any of this, then let me know by leaving a comment.

Thanks for reading.

John O'Donovan is Chief Technical Architect, Journalism and Knowledge, BBC Future Media & Technology.

The World Cup and a call to action around Linked Data

John O' Donovan John O' Donovan — Fri, 09 Jul 2010 13:37:59 +0000

Underneath the surface of the BBC World Cup web site there is a revolution going on in the technology and workflow being used to manage and publish our content. To some extent we have been doing this in stealth mode as we figure out a lot of challenges, but as we approach the World Cup Final we'd like to explain some more about what we have changed and why this is an important engagement for us in the development of the Semantic Web and support for the use of Linked Data .

For some time, we have been working on utilising Metadata and Linked Data to organise and manage the site dynamically, culminating in the World Cup 2010 site which uses Linked Data to manage how content is published. We have also had some News Linked Data discussions with other news organisations thinking about how to bring a critical mass to the development of the Semantic Web and what benefits it can bring.

The World Cup site is our first major statement on how we think this can work for mass market media and a showcase for the benefits it brings.

First some background on the World Cup site.

The World Cup site is a large site with over 700 aggregation pages (called index pages) designed to lead you on to the thousands of story pages and content which make up the whole site. Examples of index pages range from the Groups and Fixtures page through to detailed pages for each team or player.

Normally, managing all these index pages for the World Cup would not be possible as each of these needs to be curated by an editor, setting up automation rules or keeping it up to date with latest stories and information. To put the scale of this task in perspective, the World Cup site has more index pages than the rest of the Sport site!

So how is this possible? Clearly some form of automation is required, but search technologies and previous methods for doing this have proven to be inaccurate and there is no point in having all these pages if the quality of them is perceived to be low. You don't want to get content mixed up between different players with the same surname, for example.

The key change is we are using some advanced methods for analysing content and deciding how to tag this content with precise metadata linked to uniquely identified concepts (a concept usually being a person, place or thing). In the case of the world cup we are interested in players, teams, matches, etc... but the principle can be easily applied to anything. To do this we are using some technology from IBM (Languageware) and Ontotext (BigOWLIM) and a high level view of the process is shown in Fig 1, but we will be following up this post with more details about how this all works.

Pushing the Boundaries

Though there are lots of dynamically published sites on the internet, the difference here is in the use of RDF and Linked Data to build and manage the site. This is incredibly flexible and we are only just starting to explore the possibilities of how this allows us to present and share content. Though we have been using RDF and linked data on some other sites (such as BBC Programmes, BBC Wildlife finder, Winter Olympics) we believe this is the first large scale, mass media site to be using concept extraction, RDF and a Triple store to deliver content.

Another way to think about all this, is that we are not publishing pages, but publishing content as assets which are then organised by the metadata dynamically into pages, but could be re-organised into any format we want much more easily than we could before.

So why is this important?

The principles behind this are the ones at the foundation of the next phase of the internet, sometimes called the Semantic Web, sometimes called Web 3.0. The goal is to be able to more easily and accurately aggregate content, find it and share it across many sources. From these simple relationships and building blocks you can dynamically build up incredibly rich sites and navigation on any platform.

There is also a change in editorial workflow for creating content and managing the site. This changes from publishing stories and index pages, to one where you publish content and check the suggested tags are correct. The index pages are published automatically. This process is what assures us of the highest quality output, but still saves large amounts of time in managing the site and makes it possible for us to efficiently run so many pages for the World Cup.

To make all this possible there has been fantastic support from the Sport team, engaging with new tools and workflows. We are all looking forward to the London Olympics, where there will be over 12,000 athletes and index pages to manage and so without this type of technology, we will not be able to showcase and maximise all the content we have.

A call to action

We'd like to engage further in the development of Linked Data and feel we have a role to play in supporting this important new view of how content is published and shared. The methods talked about here will become the basis for more and more of our content publishing and we fully appreciate the work many people are doing in this area to make this possible.

There is a vision for the future here with more time spent on creating and sharing content and less on managing it. However we have had to overcome many problems in getting this far and many of these issues are related to organising and cleaning up data. Due to all the technical and data challenges we have not yet been able to expose all our data as RDF, for example, though we will start doing this soon.

As more content has Linked Data principles applied to it (as outlined here , then these problems will become less significant and the vision of a Semantic Web moves closer. Importantly, what we have been able to show with the World Cup, is that the technology behind this is ready to deliver large scale products.

This is more than just a technical exercise - we have delivered real benefits back to the business as well as establishing a future model for more dynamic publishing which we think will allow us to make best use of our content and also use Linked Data to more accurately share this content and link out to other sites and content, a key goal for the BBC.

We look forward to seeing the use of Linked Data grow as we move towards a more Semantic Web.

John O'Donovan is Chief Technical Architect, Journalism and Knowledge, BBC Future Media & Technology. Read the follow up post, BBC World Cup 2010 dynamic semantic publishing on the Internet blog.

Issues on News and Sport Video & Audio Playback

John O' Donovan John O' Donovan — Fri, 12 Jun 2009 17:14:01 +0000

We have had a few issues across the News and Sports websites (and other areas of the BBC website) which has affected video / audio playback, the ticker used at the top of some pages and picture galleries.

It was caused by an update to some shared libraries used across the BBC. The problem with these libraries was quickly rectified, but unfortunately some of the corruption caused by these changes have been cached across multiple networks and in browser caches, which has caused intermittent and continuing issues from yesterday evening to this morning.

Support teams have been working on this and we believe this has now been fixed, but apologies for any inconvenience caused. If you are still experiencing problems then let us know, but before you do, please try clearing your browser cache to see if this rectifies the issue (some details here if you are not sure how to do this - How to clear your browser Cache).

John O'Donovan is Chief Technical Architect, FM&T Journalism

(Em)Bedding In

John O' Donovan John O' Donovan — Wed, 11 Mar 2009 13:29:13 +0000

It seems like a long time since we launched embedded video across the BBC News site.

Seems strange to view it any other way now. An outstanding question has been how and when will we enable the ability to have video embedded in other sites.

Well, finally you can do it.

For some videos (starting with those in the News Technology section), we have started to roll out the Embed function.

It's taken a while because there have been a huge number of tricky little issues to sort out and most of these have been complex business issues around rights, terms and conditions, etc... But at last through the fog, a simple and subtle change finally emerges.

When you roll over the Share button at the bottom of the embedded Flash player, you will now get an Embed option which allows you to take the embed code and embed the video on your site.

Some example video links to play with are:

Chinook Helecopter Story

Internet football fans

Bafta Video Game Awards

As you can see, as well as the video there is a description panel under the video. This will roll out to more areas of the site after this initial phase and there are different flavours of embedded widget which will be made available.

There will remain some content which we won't be able to allow you to be embedded and viewed off site or outside the UK, mainly for rights reasons.

This year we are focussing on a number of projects which will make our content more open including some major changes to the News and Sport website content management and publishing systems.

I'd be Interested to hear how you would like to see our syndication and content sharing develop.

John O'Donovan is Chief Technical Architect, FM&T Journalism

BBC iPlayer Local Radio Integration

John O' Donovan John O' Donovan — Thu, 04 Dec 2008 14:20:42 +0000

Local Radio: You are waiting for one, then along come 40 all at once...

In the depths of Fortress BBC, BBC Nations and Regions (part of BBC Journalism) is proud to announce that we have finally unleashed the hounds of Local Radio into BBC iPlayer and beyond. (Also see this post from James Hewines).

Local Radio integration has been a long time coming and here is some background describing why this has been complicated to deliver.

Previously the Pop-up radio player was driven by a feed derived from our EPG metadata called What's On (editor's note; see this post about the closure of What's On).

This feed was limited in the format it was produced in and the richness of the data within it so was being switched off. Priority number one therefore was to find a new and more appropriate way to generate this data which could also be integrated with BBC iPlayer, /programmes and other services.

The data required to drive services like BBC iPlayer is more detailed than that required for an EPG, and make no mistake that metadata is expensive to look after. And if you don't look after it, caress it and love it, like a badly behaved MetaPet it will just as likely bite you on the MetaBum. So the challenge was, how do you create all the metadata required to feed a rich and structured metadata standard across a huge number of local radio stations in a cost effective way?

Bear in mind that different Local Radio stations have varied levels of support, depending on the size of the operation and the scale of BBC presence in the local area where they are based. This means that schedules were provided in a variety of different formats from each station. A cost effective way of pulling all this together was required, and a gateway to publish this information onwards needed to be built.

The answer was to look at establishing a more centralised schedule collation service and use a tool from Unique Interactive to bring together a more detailed schedule. This tool was used for driving some of our EPG and DAB text data for radio, but the opportunity was there to do more.

Suddenly of course, you will see we are no longer purely in the land of software. We are now into setting up new processes, re-engineering the business chain and finding people to use these tools to put together a consolidated data view of a Local Radio schedule which could drive interactive services. This is an area in which I am aware of the challenges after my Tour of Duty on BBC iPlayer. Designing the metadata is only part of the problem - you need to push back into the content production chain the process of creating and managing the metadata to make it consistent, reliable, relevant, etc. It's not easy and it takes a lot of effort and commitment.

Ian Myatt, who has been one of the driving forces behind this development, would say this, if he were here...

The migration of all 40 local radio stations is a massive achievement that has involved numerous teams across the BBC. We've had to take a detailed look at scheduling processes across the stations and introduced a more consistent approach with minimum disruption to existing workflows. Our Birmingham based New Media Support Team now receive weekly schedules from all our stations and are responsible for entering details onto a scheduling tool that then feeds directly into the programme information system behind /programmes and BBC iPlayer

So much for technology then. Most of this is about people and process.

Pulling all this schedule data together can now be controlled and managed centrally in a single tool for all these stations. Where does it go next?

A bit of background on Schedules. Schedules can have a very long lifespan, being planned many months in advance of TX (transmission) so schedules tend to evolve over time. The level of detail increases as you get nearer to TX based on what you know about the content of programme slots in the schedule.

As TX gets closer, there are key points when data in a schedule has to be taken as a snapshot for publications such as Radio Times and further points where it has to be published to other systems. But it might change right down to the moment a programme is broadcast and an audit process (Checking the As-Run) is used to make sure that the programme that went out was the one that we said we would broadcast. Sometimes programmes can get pulled at the last minute for reasons such as sporting events over running or breaking news.

At appropriate points, the data is synchronised from the Unique EPG Client to a central server which follows a delicate path to work through BBC Information Security restrictions. This data is then processed and sent back to PIPs, our home for Programme information, in a format it can process and understand. It can then be tweaked and managed using the Programme Information Tool which allows this data to be changed directly in PIPs as necessary. This includes making sure the data is accurate and that all the programme relationships to other data are correct. There's some info about PIPs here.

So it takes quite a lot of work to get here, but once you are the rest of the publishing chain is automatic. Programmes and data appear in iPlayer based on the metadata provided to this point, and here you can see the importance of getting the data reliable and consistent. For example if you spell programme names wrong, put the wrong genre against programmes or a series is not linked together then it automatically appears wrong to the end user. The importance of metadata is very apparent to the end user, thought they may not care to know much about it.

Finally you ask, where is the programme itself?

Well compared to the changes for metadata production, the media chain looks quite simple. Currently Local Radio follows the same process as it has previously and is mostly recorded off air into Real format by a system called BOB which lives in a semi detached house in Maidenhead. We will be changing this process soon to produce more formats for Local Radio, firstly to bring it into line with other audio programming as MP3 / Flash format.

A final interesting thought is that this process allows us an easier way to push information about non-broadcast material into PIPs. This could mean that content not in a TV schedule, such as the highlights package for a Football Match can be linked to the actual match as a version of the programme, making it easier to group together News and Sport programmes more effectively. For example, linking together:

a programme (e.g. a football match)
versions of programmes (e.g. highlights)
segments of programmes (e.g. a goal)
and clips related to programmes (e.g. interviews after a match).

The power of more metadata has yet to be realised.

So here they all are, a beauty parade of 40 new stations in iPlayer, publishing /programmes data and more to follow...
Berkshire
Bristol
Cambridgeshire
Cornwall
Coventry
Cumbria
Derby
Devon
Essex
Gloucestershire
Guernsey
Hereford and Worcester
Humberside
Jersey
Kent
Lancashire
Leeds
Leicester
Lincolnshire
London
Manchester
Merseyside
Newcastle
Norfolk
Northampton
Nottingham
Oxford
Sheffield
Shropshire
Solent
Somerset
Southern Counties
Stoke
Suffolk
Swindon
Tees
Three Counties
Wiltshire
WM
York

John O'Donovan is Chief Architect, BBC FM&T Journalism

Knowing When To "Go"

John O' Donovan John O' Donovan — Wed, 05 Nov 2008 12:26:11 +0000

Recently, in response to feedback, we have been putting more external links onto the BBC site and have also started using tracking for external links on the BBC News sites.

Essentially the reason for tracking links is to allow us to report to the BBC Trust how many click-throughs we are generating to external sites so that they can accurately monitor this.

We have had some feedback that the way we are doing this has an unintended side effect ("BBC pledges to link out - but holds back the Google juice").

Strategically there is no intention to drink all the Google Juice. The reasons for this are less sinister and I thought it worth explaining.

The system the BBC uses for tracking external links has been around for years, but we only recently added this tracking to the external links on the Right Hand Side of the BBC News site. You will find the /go/ tracking system in use across the BBC website and the way it redirects links is nothing new. You can see the mechanism working if you use a /go/ URL off the BBC site (e.g. this). (Editor's note 6.44 p.m.: To see the tracking mechanism in action, you need to be on a website that isn't the BBC. We suggest you copy this entire link into a new browser window or tab: https://bbcbreakingnews.pages.dev/go/bristol/content/features/2003/01/13/ sayingaboutus.shtml/ext/_auto/-/http://www.venue.co.uk/ )

On the BBC site you don't get this delay, but you can see what it is doing - it is basically logging that you have clicked a link from he BBC to an external site by going to the intermediary page and then sends you on to this page. Many sites use similar mechanisms and have to deal with the side effects of this.

We are rolling out improvements to the way this works, as already used on some other parts of the website. Essentially we use JavaScript to retain SEO ("Search Engine Optimisation") and Google juice for external sites, while we will still be able to track external links. Search Engines, casual observers and those without JavaScript will still see the original URL.

So thanks for your comments but because of the workload around the US election, bear with us while we improve the way this works.

I can't speak for the history of /go/ tracking at the Beeb, but there is some more background in Martin Belam's post here for those interested...

John O'Donovan is Chief Architect, BBC FM&T Journalism

Google Gulp image courtesy of Google

Election Party!

John O' Donovan John O' Donovan — Tue, 04 Nov 2008 12:34:35 +0000

We are working hard to ensure the US Election will be as hard to avoid as a dull and rainy New Year's Eve, so you may as well join us and watch it. Even better, stay up all night and go crazy with cookies and hot chocolate in your own election party, but make sure you research some US Presidential Trivia to annoy your friends and family.

The web site on election night should feel more dynamic than it has ever done before, thanks to a host of Flash, AJAX and JSON features updating results, tickers and maps in real time. You can see an example of how the homepage will look on the night below. It has all been put together by an extensive team too numerous to list here, but you know who you are.

Over on the Editors Blog, Steve Herrmann has outlined some of the features we will have running for the US election. In particular you may be wondering how the new "single" results system works and how the BBC calls the results as it gets them.

Firstly, all the BBC Election results are driven off the Election Production System (EPS, right) being operated in Washington by the staff on the BBC Washington Results Desk along with the Interactive team running the EPS and other systems in Washington and the UK. This is the first time a single coordinated system has driven all outputs TV, Internet, Radio, etc... and it ensures that all outputs get the same information form the same source.

I've worked in the studio on election nights and the combination of adrenaline, coffee, fear of results or graphics not working and sheer exhaustion after 12 hours in a studio overnight mean it's one of the most satisfying and stressful experiences you can go through. In particular I remember being terrorised by a Floor Manager with an angry outlook on life as I kept trying to get in front of the cameras "by mistake". Anyway back to this century.

First I would recommend a review of how the BBC views elections US style. It's pretty straightforward, but there are a few surprises about how the US Electoral College works and especially how the BBC results are validated because some of the early results you will see are predictions rather than results.

OK - so you've read the background info and frankly now you know as much as me about how elections work in the US.

As results are gathered by the BBC from Associated Press and other sources, the Results Desk will use the EPS input client to view the data and make a call about the result for any state. This client is more complicated than ones we have used before and allows viewing, editing and declaring of results. The Results Desk can also use the client to send messages and prepare for results coming in, sharing their insider information with journalists across the BBC. This service (called Newswire) is a crucial part of the intelligence and reporting on the night. Everyone working on the output will be monitoring it.

What does the results desk look like? It's a forest of screens with some human beings scattered between them looking busy and stressed. A bit like a bunch of frenzied stockbrokers (below).

Take particular note of the screen in the middle with the horizontal stripes on. This is the Presenter and Producer Dashboard and is one of the elements at the heart of our coverage. As results are entered and information is shared by the Results Desk, the producers and presenters around the BBC will watch this screen to receive this information.

Once the results desk decide to publish results, then a number of things happen. The TV output generates a graphic with the results or predictions published. The results on the web update, including the maps and tickers across the site showing latest results and information. Data and calculations are made so that the TV high end graphics can analyse the results - when you see David Dimbleby and Jeremy Vine playing with their graphics, the data has all come through the EPS.

Gareth Owen who developed the Results system and is running it on the night:

This is the first time the BBC has come together to manage election results output centrally, with state-by-state winners declared directly into our system's desktop client by the team in Washington, live raw vote figures coming into our system from the AP in New York, and fast, bespoke output produced for every single BBC outlet via our systems in London"

And when we say every platform we really mean it, including TV, Radio, the web, Mobile services, Interactive TV and even Ceefax (right). World Service sites in different languages and Arabic TV are also all working off the same results. The typo has never had so much potential for speedy multi-platform evil.

On the night we are hoping to cater for engagement at many levels. From simple scoreboards and tickers on the Homepage and across the BBC News site, through to detailed analysis on interactive services and TV / Radio.

An interesting development is further enhancing our live multi-stream player which was used for the Olympics. Through this, you will be able to engage with a BBC "Stream Of Consciousness" as events unfold. Multiple AV streams, results data, journalistic comment and anything else of interest will be squeezed into a vibrant page updating dynamically. You can send us comments and tell us your predictions through this as well.

The live experience around AV is something we are exploring more and more, and I hope the combination of outputs will give you all the access you desire to information about this important event.

See you on the other side...

John O'Donovan is Chief Technical Architect, FM&T Journalism.

Olympics: Numbers Update

John O' Donovan John O' Donovan — Thu, 11 Sep 2008 11:57:05 +0000

I promised you an update on numbers after the Olympics and here it is. The statistics are endless and with the analytics tools at my disposal I could prove that I am the rightful King of Sheba, so I'll just give a summary of some key messages.

Overall we served approx. 50m sport video streams during the Olympics. This averaged out at about 3m per day, but peaked at 5.5m on Tuesday 19th August. There were many athletes you were interested in and Chris Hoy and the cycling team, Rachel Adlington, Christine Ohuruogu, Usain Bolt and Michael Phelps were all athletes that generated large audiences.

As an example of what people were watching, here's what people watched (ie, started a video stream) or read (ie, viewed a web page) on Wednesday 20th when Usain Bolt was winning the 200 metres:

200m text story: 914,543
200m video: 501,943
200m heats text story: 61,257
100m video: 54,357
200m heats video: 42,986
200m interview video: 33,500
100m text story: 33,500

We found that as much as 45% of the Olympics audience engaged with video from the Olympics site. The trend is more interesting in that in general you were keen to engage with video. Looking at how we promote AV, how we create clips and how we deliver them are all things to build on from the Olympics.

For example, the opening ceremony live stream was embedded on the News and Sport front pages, as well as the Olympics index. Around 80% played the stream on the Sport indexes, while 50% played the stream on the News index. In general, a quarter of the traffic which came to the Olympics site from the UK watched video (that is, those in the UK who can access the geographically rights-restricted streams).

My favourite statistic is that you watched nearly 9.7m hours of Olympic video on the website (yes, you did!) and regularly there were over 100,000 watching at the same time though the peak stayed at just under 200,000 across live and on demand videos.

How do the online and TV viewing experiences compare? Well, online you watched live video for about 15 minutes per day, and dipped in and out of on demand clips for about 3mins20sec per day. TV figures are in Roger Mosey's post here.

The Chinese can't get enough of the Olympics, like they don't want it to end and it's interesting to note that even though they could not see the video, there was a large international audience on the BBC Olympics site, some watching the live text updates, some looking for story details.

As well as the glamour of the Olympics, day-to-day news gathering carried on and also on Wednesday 20th, we served 2.7m news AV clips, with a peak in demand following the Madrid plane crash.

These are all amazing figures and if you are interested in comparing different countries in Europe and how they engaged online, you can start by looking here.

So another lesson? If this blog post had video in it, more of you would have read it... perhaps.

Thanks to you for enjoying the Olympics with us so much. Now go watch the Paralympics... you can get back to work in October.

John O'Donovan is Chief Technical Architect, BBC FM&T Journalism.

Olympics On All Platforms

John O' Donovan John O' Donovan — Fri, 22 Aug 2008 20:47:55 +0000

You may have noticed over the last few weeks that the Olympics has invaded our screens, websites, phones and life as never before.

The access to information and content has been unprecedented, and I thought you might be interested in some of the areas where we used some Olympics special sauce.

First, some (estimated) numbers to give a rough sense of scale. (NB: these figures are UK-only because the relevant media are just UK-available.)

I talked earlier in the year about the launch of embedded video and how this has provided a better platform for our media on the web. This has really come into perspective with the Beijing Olympics.

In Athens, we served about 2.5m video plays for the entire event. For the Beijing Olympics so far, we have served nearly 40m sport video streams via the web - and the games are not over yet, with a few big events still to come.

We have at times been serving nearly 200,000 concurrent streams including live video and video on-demand. Individual clips have regularly reached over 500,000. Over 6.5m hours of Olympics watched so far. Up to 5.5m Olympics videos watched each day, averaging just under 3m.

And this is just on the Sport site on the web - not to mention BBC iPlayer, interactive TV and mobile platforms. Look out for updates with consolidated details after the games have finished.

Olympics? You love it.

What factors have created such a huge step forward in demand? You may have noticed that Team GB is third in the Olympics medal table. We've double checked the calculations, and this isn't an error or a cunning conspiracy by sporting pundits. So Team GB has been magnificent and many of you have clicked to watch them run off with the GB haul of medals. (Technically speaking, they have in fact have swam, cycled, sailed or rowed away with most of them, but let's not split hairs.)

Certainly, the internet has grown up to support this level of use. Broadband is getting better. The easy accessibility of the embedded player is playing its part. And the time of day has meant that many of you may not have access to a TV and that the web is the only route - but then perhaps the on-demand nature means you prefer the web anyway?

There are no doubt other reasons as well - let me know what you think.

Our services around the Olympics have also seen some interesting developments.

First, the live video streams have been hugely popular and we have been using them extensively across the site including a multiscreen player showing all the streams available in one place. This is an extension of the Embedded Media Player that allows you to choose which live stream the player is connected to and which redirects to this stream without a page refresh.

We have up to seven of these running at a time, based on the available streams of content. Sometimes they are live streams delivered from China; sometimes they are loops of key events or summaries. The video streams are encoded in real time to Flash On 2 VP6 video format by ~~Viewcast Powerstream Pro cards~~ Viewcast's Niagara Pro Encoders at 368Kbps which gives the best compromise for audience reach, delivery reliability, bandwidth usage and quality.

Concurrently, the editorial teams have been picking about 10-20hrs of content for BBC iPlayer each day which are sent as discreet files and encoded at a slightly higher quality (about 500Kbps). These are unusual because this is the first time we have put "off schedule" content into iPlayer - meaning content which is not in the schedule of the traditional linear TV channels (BBC ONE, BBC TWO, etc). These then link up with additional metadata added to the programme and channel information used in iPlayer so that these "off schedule" programmes are linked to programme information in iPlayer. There will be more of this to come over time as we look at which content best supplements the core TV schedule content we have in iPlayer.

Following the chain further back, you can see that we have an extensive guide to what is on the various channels and services. The coverage of events using multiple streams of video is more extensive than we have ever done before and to make this more finadable we have added more information to the TV schedules and Electronic Programme Guides to provide as much information about individual events.

Previously, you would often just see large blocks in the EPG showing "Olympics". Now you can see much more about the individual events. These data are what also drives the information showing what is on in the live streams (you can see an example feed here).

In the longer term, we hope to be able to publish much more accurate running order information and to segment our programme content in News and Sport so that individual sections of programmes can be identified.

Programme pages are available for each sport from the drop down at the top of the EPG page. These show listings of events for events by Sport and allow you to click through to the video. Where available, you can follow these links through to the Olympics content in BBC iPlayer and the astute will notice the use of the same IDs for the media (the "b00cs7sv/" part at the end of both addresses).

There is a red button sports portal on four platforms (Freesat, Freeview, DSat and DCable) all showing a choice of live action, recorded highlights and integrated text updates. Again, these are benefiting from the improved programme information available, but they are delivered through separate infrastructure dedicated to these platforms. They are using the same streams as available on the web, but they don't all have all seven streams.

On the Olympics live action page, underneath the multiscreen player, you will see the Live Text update. This allows journalists to provide a running commentary on the events occurring all around the Olympics while you are watching any one particular video stream. It's hugely addictive, largely down to the entertaining commentary and comments from the audience.

The key change on this feature is that the text updates without needing a page refresh. We have had a hugely positive response on this. It works by polling updates which are added to a JSON file and then placed in an animation queue which adds them to the page. It avoids the issue of how to deal with page refresh when the video is playing on the same page as well.

The map [click here] has been an interesting experiment using Microsoft Virtual Earth. Interesting, not only because we have been putting updates and blog posts on to it, but also because we have overlayed relevant venue information and enhanced rollovers.

Customising the maps has helped identify how we can manipulate maps more effectively and present a less generic face. There is some more detail on how this was done here and a blog post on how it was built here.

Monkey has been out and about in the form of our Adobe AIR-based desktop application and, to be honest, he's been a bit naughty.

We washed his mouth out with soap and water and he has behaved since and been downloaded by over 50,000 of you. Final numbers to be confirmed. If you are interested in why we are using AIR, you can see the start of our strategy here.

Mobile has been a challenge to get working but on those networks where we are able to stream, there has been mobile video available in both live and on-demand forms. Yep: you can watch video on your mobile showing a live stream over 3g and it works, as long as you stand in the right place. It looks pretty good too.

There is more, such as blogs, podcasting, the Flickr pool, Twitter updates, the Denise Lewis Heptathlon, HD TV, Big Screens coverage around the UK, a China 08 topic page...

I'll stop there, but if you want more detail on the specifics of some of these products, then let me know and enjoy the last few days of coverage.

John O'Donovan is Chief Technical Architect, BBC FM&T Journalism

BBC LiveUpdate

John O' Donovan John O' Donovan — Wed, 02 Jul 2008 08:57:29 +0000

It seems you can't move for widgets down on the BBC farm these days. You certainly can't have a product without one.

But it's not right, the way they are treated. Packed into websites and often given no sustenance other than poor quality, recycled feeds, they suffer a cramped existence and don't have the freedom to develop.

So we decided to start setting some of them free to roam the desktop.

A free range widget

As with all good ones, this idea is not new. We have had a desktop ticker and similar products like Mini Motty for some time; they're quite successful with hundreds of thousands of users. But there are some issues with the way they work.

Firstly, they only work on Windows and are built out of a variety of proprietary tools. We'd like these to work cross-platform but we need to build them differently to do so. They are also difficult to manage and expensive to maintain.

We considered a few approaches, but decided to grow our new widget out of Adobe Flex and Adobe AIR. This is firstly because these tools met our requirements to work cross-platform and deliver the desktop experience we wanted, and also because they linked up with in-house skills in the team which manages them, making them simpler to manage.

So here's the first of a number of desktop widgets the BBC will be releasing over the coming months. It's an example of how we could develop our current ticker. It's a beta product, so it will have some issues and it is also not fully accessible, but we'd like your thoughts and comments on how it works, what you like about it compared to our current ticker and what doesn't work so well.

You will need Adobe AIR to install and run the application. If this is not installed automatically, you can download Adobe AIR from here.

And what does it do?

Well, it runs as a desktop application in a similar way to the ticker showing the latest stories and also updating you if there is breaking news. It updates stories about once a minute and should be fairly low bandwidth, as it caches data downloaded. You can take it offline and - unlike the current ticker - it also contains images and summaries of stories.

You can have it running as a small application showing stories listed from News and Sport, or you can minimise so it just shows as a ticker, scrolling current stories.

You can open up the preferences to select which feeds you get and you can click the full story links to go to the website.

You will see a BBC icon in your toolbar when the application is running and you can turn the application off from here. The first time it runs you will see the preferences window which allows you to configure the way the application works.

And that's about it.

Installation and feedback

You can download the application here [NB Editor's Note 3.03pm: If downloading via Internet Explorer, you'll need to change the file extension from .zip to .air]• (Editor's note 12.52 Thursday - you may also find this download useful if you're a Linux user)

N.B. Editors' note 19th May 2009 - this application is no longer a live prototype, so I have removed the link to download it.

Send us your comments below or through the Backstage list.

Before you do, though, here are some of the known issues, caveats, disclaimers and apologies:

Scroll arrows are missing on scrollbar
Can be slow to startup the first time: try clicking the News or Sport Tabs
Accessibility support is not fully developed.

In terms of the last, we do take accessibility very seriously and would like to give some more detail on how this applies to this beta:

Accessibility issues

BBC Future Media & Technology's pilot widget application BBC LiveUpdate uses the Adobe AIR runtime, which is dependent on users downloading and installing a plugin to their desktop, but which unfortunately does not currently support screenreaders (or other software which relies on the Microsoft Active Accessibility layer for assistive technologies).

We're working with Adobe to make tools built with AIR more accessible than current products wherever possible and are committed to delivering accessible services.

As this is a beta product, there are also other limitations in how much we have been able to establish accessibility support in the following areas:

Colour contrast cannot be altered
Text size cannot be altered
Lacks consistent alt text
Lacks Title attributes
Is not entirely tabbable.

We are sorry if you are unable to use this BBC LiveUpdate widget fully - but the content within the BBC LiveUpdate desktop application is available through existing BBC services on bbc.co.uk.

If you would like to share your views with the BBC about accessibility and the BBC LiveUpdate widget, please do not hesitate to email the BBC Usability and Accessibility Team.

John O'Donovan is Chief Architect, BBC FM&T Journalism.

Editors' note: A tad more on accessibility in this statement from Adobe: Adobe has a long history of its commitment to accessibility and Adobe AIR 1.0 currently provides support for developers to create applications that are accessible to users with special needs. AIR applications presently support many users with disabilities, such as users who are unable to use a mouse or who rely on textual equivalents for audio. AIR applications incorporate Adobe Flash, PDF, and HTML content, and Adobe support for assistive technologies for both Adobe Flash Player and Adobe Reader ensures that developers are able to deliver accessible experiences in the individual players. Adobe is continually reviewing with customers how to improve access to applications deployed on Adobe AIR for users with disabilities, in order to ensure that future versions of Adobe AIR support accessible experiences and meet emerging global accessibility standards.

News & Sport Embedded Media

John O' Donovan John O' Donovan — Fri, 14 Mar 2008 11:30:16 +0000

Yesterday we launched embedded media (see an example here) across the BBC News and Sport sites.

We are starting with just a few stories, but over the next few weeks, we will be converting most of our video and audio to be embedded.

We're using an embedded media player which, within the BBC, is known by the secret code name of the Embedded Media Player - or EMP. This is a new site-wide media player built by the Journalism and iPlayer teams. More about this further down.

"So what does this change on the site?", you may ask.

Currently, our audio and video appears in popup windows using Real or Windows Media formats, which you get to by clicking on the Watch / Video or Listen / Audio links that you see in stories. You can see some examples in this story underneath the main picture.

This approach has served us well, but it was apparent from trials we ran in the middle of last year [for example, here], that in general, people find embedded media easier to engage with. No surprise there, as the unstoppable march of sites such as YouTube has proven. However, Pete Clifton also talked about how this may change the way we make video and provide added value to the stories it is associated with.

It has taken some time to put all the pieces in place to make this change, but there are a lot of moving parts to adapt without breaking anything that is being used on the live site.

]]>For those who are interested, these media are created as clips (30sec - 3min long) and are submitted from all around the UK (and beyond) to a system called the MPS (Media Publishing System), which routes this media to the correct destination, transcoding it to the correct output format on the way.

Previously, this would be Windows and Real in both narrowband and nroadband versions for both formats. We also keep a higher quality copy so we can make further versions if required. Now we will only be making a Flash version for most video and audio, along with the archive copy. Life has just got a little simpler for us making the video and for the audience in engaging with it.

Once media are made available by the MPS, the CPS (Content Production System) is made aware of this and allows this video to be linked to stories. A big change here is that previously, the media were "hidden" in a popup console and not integrated with the site. With embedded media, each piece of media will have its own page (called a Media Asset Page or MAP - here's an example).

Apart from our obvious Talent In Creating Acronyms (TICA), you will notice that the MAP will then act as a permalink to the video or audio and is the destination to use when linking back to media items in general, especially to display video at its full size of 512 pixels wide. This media can then be linked to one or more stories, where it is placed in context within the story and will be displayed at either 400 or 256 pixels wide sizes, depending on where it is placed.

Bearing in mind that there are hundreds of users of the MPS and CPS systems generating hundreds of media clips each week, it has taken time to adapt these substantial systems and start to train users in the changes that embedded media will bring.

Putting that little embedded player onto the site has sent ripples of change through all of BBC Journalism.

On top of this, we have built the EMP in collaboration with the iPlayer team to be a unified player for all video and audio. This has meant decommissioning some other players, but over time there is little reason to have multiple players with different technical approaches. This is all about creating a consistent and familiar user experience for as much of our audience as possible. The EMP is not the same as the one currently used in iPlayer though it may look similar. It is a new application and will support a variety of new features.

A significant change for our international users will be that they will now also have higher quality Flash-based media akin to that served in the UK, rather than lower quality streams of Real or Windows Media. If you are interested in how this all works, then let us know and we will explain in a future blog post.

So the EMP is a small but feature rich and complex little application. Once it is rolled out across the Journalism sites, it will then be rolled out across the iPlayer and other BBC sites, and so needs to support lots of features and scenarios. There will be more information on this soon. Note, you will eventually be able to syndicate and embed BBC video in your own sites.

There will be some exceptions to using the EMP for now, generally where the media experience is heavily stylised or embedded within interactive games. We are also not moving live streams over to Flash yet, but aim to do so by the summer.

So, embedded media at last: it may look like just a small lonely video on this page, but it is the start of something big. The Technology and Sport sections is where we are rolling out the first changes to include embedded video over the next few weeks, so have a look and do let us know if you have any ideas about how you would like this to develop.

Cheers...

John O'Donovan is Chief Architect, BBC FM&T Journalism