Just for any new readers, I am head of technology for BBC HD & 3D and this is an update on the next phase of the BBC 3D trial. We have now moved from sport, music and animation to drama!

Mr Stink is a TV adaptation of the book by David Walliams. I won't give anything away in the post for those who love the book and for those who have never heard of it - just watch!

In this post I want to give you an overview of some the technology and processes used during the making of Mr Stink with comments from a few of the people involved.

Mr Stink transmits in 2D on BBC One & BBC One HD on Sunday 23 December at 6.30 and is repeated on Boxing Day at 12.40. The 3D transmission is on BBC HD on Sunday 23 December at 6.30 and shortly after on BBC iPlayer for seven days

Mr Stink (Hugh Bonneville) and Chloe (Nell Toger-Free) in Mr Stink

Mr Stink is the first programme we've done during the trial where the 2D and 3D versions were shot at the same time with the 2D being taken from one of the 3D cameras. The cameras were mounted on mirror rigs and for those interested in the details, these were a 3ality Technica Atom and a P+S TECHNIK Freestyle rig, each fitted with two RED Epic cameras and Angenieux Optimo matched lenses.

This is what Philipp Blaubach, the director of photography (DoP) working on Mr Stink, thought of idea to shoot in 3D:

One of the camera rigs with DoP Philipp Blaubach!

Chris Parks from Vision3 was the stereo supervisor on the programme and also commented;

The choice of rigs and cameras was determined by a number of factors. We wanted to be able to change setups quickly so we chose to use zooms as variable primes. We also wanted a reasonably small and light rig so we could be more mobile and wasn't too off-putting to actors. We also wanted it to be cost effective so that on certain days we could afford to bring in a second unit. The Atom allowed us to put on a larger mirror box so the DoP could use a 16mm lens with the Epic at five thousand. For monitoring we used a 46in JVC as we felt it was important for the director and DoP to be able to see the playback in as near to a real world situation as possible.

Mr Stink was framed for a wider aspect ratio than the normal 16:9. Philipp shot in the cinema aspect ratio that's sometimes referred to as 21:9 but more accurately it was actually 2.35:1. This gives black bars at the top and bottom that effectively become a new screen boundary. However as it's only an electronic boundary that allowed us to try some interesting effects.

Philipp said;

We composed the film for a 2.35 aspect ratio and during the depth grade there were a couple of instances where we could move the action into negative space and virtually make it overlap the letterbox. The 3D is especially effective in those moments as you feel the objects literally coming into your living room.

Framing for 2.35:1 (3D is displayed using anaglyph if anyone wants a quick preview but, the programme will not be transmitted in anaglyph).

As I said earlier the programme was shot using a single camera rig with an occasional second rig. This is different to the multi-camera set-ups we've used for the studio and live programmes that have been in the trail so far. This meant that instead of the stereographer adjusting each camera to match the 3D depth during the shoot, a 'depth grade' process was added to the final post-production stage.

Just like a colour matching or grading session that adjusts the colour and exposure of each shot to make them match, the depth grade matches the amount of 3D in each shot so there are no sudden jumps on shot changes.

Again for interest, the programme used the company ONSIGHT's 'Mistika' equipment to carry out the colour and depth grade with colourist David Gonzalez Lozano and Philipp Blaubach (the DoP) along with Matthew Smith who was the stereographer on the project working under Chris.

Matthew was responsible for the on-set Stereo 3D decisions working closely with the director and DoP and followed this through to the depth grading session during the final post-production.

The two TX lines

Mistika screen shot (3D is displayed as anaglyph if anyone wants a quick preview but remember the programme will not be transmitted in anaglyph).

The last time I saw the programme was just before the final colour and depth grading sessions, so I won't see the final results until it's transmitted.

As usual I am really keen to get your reactions to the programme and what you thought of the 3D but I will be away over Christmas and for a week in the New Year so replies to comments may not be prompt until I'm back around 8 or 9 January. In the mean time I wish you all a very, very merry Christmas and a happy new year - in advance!

Andy Quested is chief technologist HD & 3D, BBC Technology.

Mr Stink 23 December 6.30pm
Killer Dinosaurs 25 December 2.05pm
The Queen 25th December 3.00pm

Read more about the making of Mr Stink in 3D on the About the BBC blog.

Side by Side (SbyS) image

This is a very brief post as it's more a request for comment than an explanation of a process.

On the 19th August at 17:35 the BBC HD Channel will be transmitting "Ulimate Killers".

This single programme has been made from the very successful Planet Dinosaur series.

The programme is only available in 3D so we will be testing the "Watch in 2D" application on Freeview and Freesat.

I'd like you to comment on this blog post to tell me what you think of the "Watch in 2D" application. Unfortunately the application is not available on Cable or the Sky set top boxes yet.

The programme will also be available on BBC iPlayer. There will be several different encoded versions but as we are letting the automation take care of encoding I won't know what will happen until you do on some platforms!

All versions will be side by side so we also know not all devices will be able to convert them to a viewable 3D image.

What I do know is the image via the Freesat, Freeview and Virgin TiVo iPlayer apps will be 1920 x 1080 side by side. This is a new test encoding based on the work the iPlayer team carried out before the Olympics.

If you do want to comment, can you also let me know the:

1. platform you used and the device e.g. Freeview set top box, PC or Mac via DVB card, PC or Mac via bbc.co.uk/iPlayer...
2. rough measured speed of your internet connection
3. make and model of TV and or set top box

I can't guarantee individual replies but we do want to collate as much information as possible to assess our next steps

Many thanks.

Andy Quested is Head of Technology, BBC HD & 3D

How to find it

Here are the details of how to find the Strictly Come Dancing Final in 3D on iPlayer:

By Search:

Freesat Higher Quality Option

1. Select "Search"
2. Type in "Strictly..." (the options should start to appear immediately)
3. Scroll down to "Strictly Come Dancing 3D..."
4. Select the "higher quality" option

On Freesat you must select the "Higher Quality" option. The standard quality will produce an error message.

Freeview (TVs and Set top Boxes), Virgin TiVo and other devices

By Search:

1. Select "Search"
2. Type in "Strictly..." (the options should start to appear immediately)
3. Scroll down to "Strictly Come Dancing 3D..."

Virgin V+

Navigation is through the iPlayer menu as usual.

Please let us know...

Finally this is the information we are trying to get:

• Did the programme play or did it buffer?
• If it buffered, was it intermittent (random) or every few seconds?
• Could you put your TV into top/bottom mode?
• Did the 3D menus work or where they "greyed" out?
• Was the sound in sync?
• If you watched the live programme, how did the iPlayer 3D compare to the live 3D?

With all feedback, can you let me have the make and model number of the TV or set top box you have and if it's WiFi or cable connected to the router. Also if you can, what the average speed of your connection was as you watched.

Many thanks and Merry Christmas

Andy

Andy Quested is Head of Technology, BBC HD & 3D, BBC Technology

PS - Don't forget StreetDance 3D is on the BBC HD channel at 18:25 on January 1st 2012.

Last year we did a studio based 3D test. It was a special Strictly Come Dancing sequence for Children in Need - actually it was the second time Children in Need tried 3D as I mentioned in my "What is 3D blog".

The 2010 trial was such a success we are continuing our 3D trials with the Strictly Come Dancing final from Blackpool!

As we did for the Wimbledon finals, we will be transmitting on the main broadcast platforms, Satellite (Freesat and Sky), Terrestrial (Freeview) and Cable (Virgin Media). The programmes will also be shown live in selected cinemas.

At around eleven this morning we changed the Satellite and Terrestrial encoders to 3D mode and at the same time, the BBC HD Channel Promo will have a 24' 3D section added - including an updated version of the HD testcard.

But as you know Terrestrial, Satellite and Cable are only three of the four platforms we have. This time we want to add the fourth platform - BBC iPlayer. We hope to make the 3D version of Strictly available to the version of iPlayer that appears on some of the internet connected 3D TVs and on some connected set top boxes.

It will be a technical trial and we cannot yet guarantee the results across all the different makes, models and types of receiver so we will need your help to identify some of the issues and give us feedback for any future iPlayer trials.

Here are some of the proposed technical details:

• The left and right pictures will be converted from 1080 line interlace 25 frames a second (1080i/25) to 720 line progressive at 50 frames a second (720p/50).
• Both images will be sub sampled into a single top/bottom image.
• Then finally the data rate will be encoded for iPlayer distribution - the data rate is currently a bit on the high side at around 5Mbs

What the 3d iplayer image looks like on a 2d tv

If all goes well I will post details of how to find the 3D version as soon as we make it available with details of the feedback we need.

Andy Quested is Head of Technology, BBC HD & 3D, BBC Technology

This morning after a very successful 3D trial the BBC HD Channel encoders reverted back to their “2D” mode.  As a last word on the Wimbledon 3D trials, I am trying to get a series of blogs together from everyone involved, so hopefully more over the coming weeks.

From your comments I know the topic of 1920 transmission is something some of you feel very strongly about. So I thought this would be a good opportunity to do a post about it.  The whole topic is not straightforward or we would have been 1920 from the start of the HD Channel.  Although technology has matured the debate still moves between the science, the technology to produce it, the ability to deliver it and the impact when you see it at home.

As trevorjharris pointed out in the Gearing up to Deliver Wimbledon 3D blog post, the BBC’s HD strategy states we fully support the 1920 x 1080 standard for programme making.  This position has not changed but even though all programmes are delivered this way, not that many are yet made 1920.

In the BBC HD Channel promo for example there are only two programmes acquired and post produced in 1920 and during the time the encoders were set to 1920, the majority of programmes shown were not made in 1920.
One thing to remember is, two of the current primary production formats are not 1920.  Sony’s HDCam tape format is a 1440 3:1:1 format at around 140Mbs and DVCPro100 (DVCPro HD) well – that’s slightly more difficult to define.  It depends on the camera and whether it’s being used as a tape or a file codec.   As a tape codec it is often used in the Panasonic Varicam where it’s used to record 1280 x 720.

The Varicam is one of the very few 720 cameras we allow – its “CV” goes back as far as Planet Earth and Galapagos and it’s still being used for stunning Natural History programmes.  This was one the points I made during the picture quality debates last year.

During the trial we did take the opportunity to look closely at some of the differences on each platform and how some devices handled the change.  I will report back on the findings as we get more information. 
We’re also looking at the impact of the different scaling technologies used.   Good scaling is vitally important, especially when it’s used during programme making. 

If we don’t do it well it can cause aliasing – this was something we had to guard against when we made the Side-by-Side image for 3D.  Similarly up-scaling from 1440 to 1920 in domestic equipment, can also soften the images or a add edge ringing (or both!).  The degree of softening or ringing depends upon the interpolating filters which perform the up-scaling.

We are working on other improvements to the programme and transmission chain including more use of 1920 and hope to have news later in the year.

I will keep you up to date on the work as often as possible.

Andy Quested is the Head of Technology, BBC HD & 3D. He previously wrote to explain 3D TV and the BBC's Wimbledon 3D TV trial

Stereoscopy experiments began in the 19th century starting with still images but it rapidly followed the movies into the early cinemas. Stereo camera rigs were patented around 1900 and the earliest confirmed 3-D film (The Power of Love) was shown in the Ambassador Hotel Theatre Los Angeles in September 1922!

Over the past 90 years 3D has come and gone. After each decline there have been various attempts to revive the technology: The 1950's for example, were described as the golden age of 3D with now infamous "House of Wax" released in April 1953 with stereo sound!

The 1980s were responsible not just for big hair but for a run of "part III" films ripe for the addition of a "D" at the end of the title. Jaws 3-D, Amityville 3-D and Friday 13th Part III-D, all vying to throw various dismembered body parts over an eager audience!

Each revival was usually the result of a technical advance or technique that seemed to make 3D better or a more compelling but never enough to catch a mass sustainable market.

The BBC has made 3D content in the past. In 1993 Children in Need made a Doctor Who short using the Pulfrich effect . BBC Films made the very successful Streetdance 3D and BBC Worldwide has begun exploring 3D Natural History programmes. More recently we have tried 3D 6 Nations into cinema and a special Comic Relief Strictly Come Dancing promo.

And! I'm sure you all know Sky has its own 3D channel showing a mix of sport, movies, and concerts along with a collection of other genres.

Is 3D bad for you?

There has been an incredible amount of hype and misunderstanding about alleged risks from watching 3D. Stereo 3D gives an illusion of depth but your eyes are still focused on the screen plane no matter how far in or out an object actually is. This is where some of the confusion and possible "health scares" come from!

The RNIB has done some work evaluating the experience partially sighted people have when viewing 3D and an organisation called the International Telecommunications Union (ITU) - part of the UN, has asked the World Health Organisation to report on any effects associated with watching 3DTV.

After watching a lot of 3D television myself what I can say is, well-made 3D is an enjoyable experience, poor 3D can make you feel a bit odd and can give some people headaches but quite frankly you turn poor 3D off long before that!

This site has a good explanation of parallax - I will let you guess which of the four diagrams could cause discomfort!

Some other good links for further reading are;

https://bbcbreakingnews.pages.dev/rd/publications/whitepaper173.shtml

http://en.wikipedia.org/wiki/Depth_perception

http://en.wikipedia.org/wiki/3-D_film

How do 3D TVs work?

 

Anaglyph, Color Code (and other colour variants) work by showing both left and right images at the same time, the left eye image one colour and the right eye image in another.

Coloured glasses make sure the images go to the appropriate eye.

 

Colour performance is very variable as is the 3D effect and it is difficult to focus properly on the image. After the glasses come off the world looks very strange (coloured glasses affect normal colour vision for a while) - Impossible to watch the without the correct version of the coloured glasses

Polarised Screen

 

Requires a 3D TV with a polarised screen. The left and right eye images are lined up behind the polarising screen, each image is given a different polarisation. To see the 3D image polarised glasses are needed. These are similar to polarised sun glasses but the lens of each eye has a different polarisation to make sure the correct image goes to each eye.

 

Shuttered (above)

Shuttered technology requires two active components - the screen and the glasses. Shuttered LCD glasses are controlled by an infra-red signal sent from the TV. The Left and Right eye images are flashed alternately on the screen at a high frame rate (100 times a second or more). When the Left eye image is on screen the Right eye lens of the glasses is opaque and visi-versa.

At the moment glasses from different manufactures are incompatible so you need to have the right glasses for the TV you have.

3D TV with no glasses

The technology behind 3D TVs that do not need glasses is very similar to the old 3D plastic postcards. It is based on the work done around 1840 by Sir Charles Wheatstone.

The screen is covered with tiny lenses, arranged to send zones of left/right images to the viewer.

 

The lenses direct the left/right images out of the screen in zones. If you sit in a zone at the correct distance you see 3D - if you move out of a zone you lose the 3D image.

Lenticular screen technology is still very new but developing rapidly.

Philips (left) was an early pioneer of this technology for domestic displays.

I will try and keep you updated between now and the finals weekend if there is news or anything changes but I will give a full explanation of the production as soon as possible after the matches.

Andy Quested is Head of Technology, BBC HD & 3D, BBC Future Media & Technology

It's part of the remit we have to try out and test new technologies where we can and as the main tennis coverage has moved to BBC One HD this year, we have the chance to try something new. For more details of the coverage have a look at Danielle's blog.

The Wimbledon 3D event is a collaboration between the All England Lawn Tennis Club, Sony Professional and the BBC along with many of our partners. After the finals weekend I will do an update with more details of the Outside Broadcast, camera technology and all the bits between SW19, W12 and your screen! I hope to persuade some of my R&D colleagues to write about the other technology we have used and the future of 3D TV.

What are we up to before the big weekend?

The production company working with Sony Professional and BBC Sport are trying to get the best camera positions fitted into Centre Court.

This is a schematic of the 2D and 3D camera positions on Centre Court:

More cameras are used on Centre Court during finals weekend so by the time 5 6 extra 3D camera positions have been added it looks a bit busy! Obviously we try to keep as many cameras out of shot as possible so there may be a few changes to make.

I hope to have some stills so you can see the cameras in situ as well some of the other 3D facilities on site, in time for the next post.

Back in W12, my colleagues in BBC R&D and Distribution who have been working on the satellite encoder's move to S2 and are now working on a further temporary up-date to the BBC HD Channel encoders to get them ready for 3D transmission. When they're happy, all the BBC HD Channel encoders will be updated. As we are transmitting on all platforms (Freeview, Freesat, Sky and Virgin) there's a lot of testing to do!

When the 3D update is complete a modified BBC HD Channel Promo will replace the current version. This has a short 3D clip and a change to the test card to help us check registration.

As we are trying to fit two images into the current BBC HD Channel's signal, one of the temporary changes is change the horizontal resolution from 1440 pixels to 1920 for the trial period.

If all goes to plan, the 3D changes will be made during the morning of 16th 13th June and they will change back to the current settings on 6th July.

What will you see on the BBC HD Channel?

Just like the Sky 3D channel we will transmit a Side by Side (SbyS) image. If you are watching the channel on a 2D TV when a 3D image comes on you will see two squeezed images next to each other. This is the reason we need the1920 horizontal pixels - each image (eye) will be squeezed to 960 pixels wide but will still be 1080 high.

A 3DTV takes the two half-width images and stretches them to fill the screen. How the TV displays them depends on the technology it uses.

The test card will look the same (SbyS) but when displayed on a 3D TV (in 3D mode) the two half width images should line up exactly - there should be no horizontal or vertical shift between left and right eye images. The frequency gratings will give us a good idea of the system filtering and if we are introducing alias components.

Above: SbyS Test Card as seen on a 2D TV

Above: SbyS Test Card as seen on a 3D TV - With or without glasses!

If you want to know more about 3D see my next post.

Andy Quested is Head of Technology, BBC HD & 3D, BBC Future Media & Technology

See also "3D for Wimbledon - the Future of TV" by Danielle Nagler on the About The BBC blog..

N.B. Correction - the date for the encoder change is the 13th June not the 16th. This is corrected in the text above. Apologies for this error. (NR)

N.B. Correction - the previous image of Centre Court cameras was incorrect. The correct one has now been inserted. Apologies. (NR).

I've also noticed there's been a lot of speculation about what we are up to for the last few days and we thought it would be good idea to let you know what's been happening and will be happening before BBC One HD launches on Wednesday.

We are, as you may have guessed, in test mode at the moment. Testing is not just about delivery of sound and pictures going through the chain from our playout provider through to the encoders. It is to check the service works on all platforms especially all the DTT transmitters and about testing Subtitles, Audio Description, Audio Channel Continuity, AV sync, Green Button services, Surround/Stereo switching, Up and Down Conversion ...

Our playout service is run by Red Bee Media, Siemens are responsible for managing the encoders and Arqiva manage the DTT transmitters. Virgin Media handle their own encoding so we send an uncompressed HD SDI signal from Red Bee to their coding site. Finally Siemens manage the satellite transmission bundle for Freesat and Sky.

The Red Bee engineers are currently rebuilding the BBC One playout suite in two stages. Stage 1 goes on air on the 3rd November and once this is up and running the tough job of rebuilding BBC One's complete playout suite in HD begins! The reason for doing a two stage rebuild is to get the channel on air in time for Christmas.

In the Stage 1 build, the HD service is a separate path that piggy-backs onto BBC One SD. The SD channel is completely unaffected by the changes in Stage 1. Because it is a piggy-back service some of the functions will be limited, for example some of the junctions between programmes will be simpler on the HD version than the SD. When the Stage 2 build is completed, BBC One will be HD; the SD channel will be derived from a down conversion of the HD version.

What are we up to then with the tests?

Subtitles - During Stage 1, BBC One HD will use the subtitling data from the SD service. To make sure this works properly we need to test the link and the timing between the subtitles and the images. This is why you have seen some random subtitles from BBC One SD over the HD promo!

Audio Description - We have been using Doctors to check how the audio description service works. The timing of AD service is critical as the AD audio MUST be in sync with both the HD and SD services irrespective of the timing difference between the two signals (some HD processing takes longer than SD especially if we have to compensate for Dolby E encoding and decoding.

Audio Channel Continuity - seems obvious but we need to make sure Left Surround comes out of the Left Surround Speaker etc.! We also check Stereo/Surround switching and the Stereo Mix-down (when you listen to a surround sound programme via a TV's stereo speakers).

AV Sync - this is a subject close to my heart. As many of you know I was an editor for many years and can see AV sync errors very quickly! Some of you may have noticed the sync signal interrupting the HD Channel's promo for long periods. I am using the opportunity to check the HD Channel to find out why we have some small differences between stereo and surround programmes. It's only a few milliseconds but we need to understand what's causing the difference. As a reminder the EBU standards are +10 to -20ms for the equivalent processing and we are well within that.

For BBC One HD we have the added complication of the delay caused by up-converting SD programmes. This adds a yet another 40ms of video delay to one of the possible paths through playout

A word of warning here; as we are adjusting the AV sync on BBC One HD, I cannot guarantee the AV sync signal is correct at the moment; we may be adjusting AV delays while it's going out and if you make any changes during the tests you could end up with up to a 40ms (1 frame) error. The signal on BBC HD is correct and measured at about -4ms in the transmitted transport stream.

Up-conversion - BBC One HD, just like ITV 1 HD, C4 HD Sky 1 HD etc, will be a simulcast channel so all SD programmes will be up-converted for the HD service. Up-converters don't just change the video from SD to HD; they also have to move timecode from the SD position usually in the vertical sync interval (VITC) to a VANC (Vertical ANCillary) data package. Up-converters may also have to change any embedded subtitle packages and move (shuffle) embedded audio tracks to cope with stereo and surround track positions.

Down-conversion - Viewers of BBC One SD will see SD programmes up-converted then down-converted. The criteria we set for the up/down converters was based on the quality of an SD source shown on the SD service - in other words, what was the most transparent process!

STB Compatibility - the BBC HD and BBC One HD channels must work on all DTT and D-Satellite so quite a lot of testing is about making sure the service is totally compatible and all the additional services work.

Over the next couple of days we will be tidying up and loose ends before the service becomes visible on the EPGs. Please remember to check and re-do any series links as we are changing the channel some programmes are transmitted on (BBC One HD from BBC HD) and changing the channel number of BBC HD.

Andy Quested is Head of Technology, BBC HD & 3D, BBC Future Media & Technology

N.B: You can also follow Danielle Nagler, Head of BBC HD on Twitter @naglerhd

Blog 3: Conclusions and Recommendations

Personally, viewing my own HD equipment at home, I have experienced, since August 2009, noticeably worse PQ from the BBC HD channel. Public support, in the form of hundreds of emails during my BBC complaint and subsequent Trust Appeal, together with the signatories of the No. 10 Petition and the continuing comments on these BBC Blogs, suggests that I am not alone. Certainly the rest of the viewers' group agree with me.

There is no doubt that the group's visit has given us a better understanding of all the issues. We can now see that there are, in fact, three important variables that can adversely affect the PQ of the channel, namely production decisions and programme quality assurance, encoder software and finally the bit-rate.

However, we still have to weigh what we were told during the day against the evidence of our own eyes at home since August 2009, and there's no escaping that this negative change coincided with the simultaneous reduction in broadcast bit-rate and change of encoder.

The conclusions that we've come to are based on what the group witnessed during the visit, informed by our subjective experiences as viewers of the broadcast HD transmissions.

Our joint conclusions are as follows:
Production decisions and QA - clearly the BBC still has a lot of work to do here. Production and "programme style" definitely can degrade the HD PQ to a very visible degree; that is obvious and apparent on certain programs, as we all know. The BBC does produce some excellent programmes, but their production quality needs to be much more consistent and the average standard must be improved. The failings in PQ that many of us complain about are often attributable directly to programme makers and so, clearly, the BBC has to redouble its efforts to bring them to account.

Encoder software and testing - having seen side-by-side comparisons we can agree with the BBC that, on simple scenes, the new and old encoders (at their respective bit-rates) looked similar. But, as explained in more detail yesterday, when challenged by complex pictures with particular characteristics the deficiencies of the new encoder became apparent. This is where we believe the BBC's emphasis has been misplaced. They seem to have concentrated on a scientific/technical analysis of the picture quality and placed less emphasis on the human, subjective aspect. The test we experienced, although generally conducted in accordance with international standards, is entirely dependent upon the criticality (encoding difficulty) of the material used and whether it "excites" specific problems. It would be all too easy erroneously to conclude from the limited range of materials tested that most material, including material that is 'critical but not unduly so', is similar with the new encoder, and that therefore everything is within the expected performance range. This would explain for instance why the mix/fade issue was identified only after the new encoder was put into service. It also completely underplays how distracting the problems are when they do appear. The artefacts, although transient in nature, are far more pervasive and frequent than the BBC would care to admit. It is on this basis that we feel that Andy's assertion that ''on the majority of material the new encoder is as good as or better than the old" does not represent the overall position adequately. We believe this is another key factor in the perceptions of reduced PQ since August. Ultimately, improvements to, or replacement of, the new encoder may well help to bring about the PQ we all want.

Bit-rate - not being able to see the same encoder run at different bit-rates was a big disappointment to us and we don't understand Andy's reluctance to do a side-by-side comparison of the new encoder with old and new bit-rates. As a consequence, we can only speculate on the effect of bit-rate on the HD channel's PQ and, also, point out the inconsistencies in the BBC's position with regard to its impact. Since we weren't convinced that it has no impact at all it still remains an issue of contention. Whilst we did conclude that the new encoder can deliver the same PQ at a lower bit-rate for undemanding material, the evidence we saw still pointed to the fact that the reduction in the bit-rate has been a major contributory factor in a reduction in the PQ of some material.

We believe that, together, these three issues have all had an adverse affect on PQ but nothing we saw during the visit swayed us from our view that it is the combination of the reduction in bit-rate and the new encoder's problems that has led to the viewers' perception of an overall reduction in PQ since Aug 09. We still believe that an increase in bit-rate would almost certainly provide a considerable degree of improvement in PQ. This belief is supported by the fact that the majority of the group identified the old and new set-ups in the comparison of clips during the R&D visit.

Resolution - Furthermore, we have also yet to be convinced that moving to the higher 1920 resolution will not be beneficial, despite Andy's confident assertions during the day that it wouldn't. If a higher resolution is not beneficial, then why do other broadcasters commit bandwidth and money by using the highest resolution? And since this is the resolution used by the majority of HD channels in the UK, then what have the BBC got against conforming with them? This is even more puzzling when you consider that BBC HD is available in other countries in 1920 resolution.

We note that Andy's assertions also contradict the information provided by the BBC to OFCOM regarding picture resolution on the Freeview HD platform, which recommends a transmission mode that is capable of supporting a 1920 resolution that "maximises the delivery of HD benefits on viewers' increasingly large and high-quality displays".

Recommendations

So, in the group's considered opinion, the BBC should:

• be even more pro-active in its in-house quality control with programme producers, rejecting programmes where they fail to meet the standards set by BBC HD and improving directives to producers, including those on appropriate cameras, to ensure that only the highest quality HD material is made available to, and broadcast by, the Corporation.
• investigate ways in which domestic HD productions can be funded so that they can be made using higher quality cameras (be they video or 35mm film) in order to achieve superior PQ, e.g. through partnerships and/or reinvestment of (BBC Worldwide) export profits.
• continue to work with its encoder manufacturer to improve its current deficiencies and to maximise the viewing pleasure afforded by the BBC HD channel's pictures (i.e. bring back the "wow" factor), whilst also exploring the options for use of other encoders to deliver a more consistent PQ at the available bit-rate.
• increase the bit-rate on satellite platforms temporarily, since cost is not a significant factor, to compensate for the disadvantage of having to broadcast at a constant bit-rate. The higher bit-rate to be maintained until encoder technology improvements solve PQ issues or a variable bit-rate is introduced on satellite.
• move from a picture resolution of 1440x1080 to 1920x1080 in order to maximise the delivery of HD benefits on large, high-quality displays.
• not reduce its purchasing of American-made superior source material programming, in particular drama series such as Mad Men, despite the BBC Director General's recent announcements.
• speed up the process of introduction of new HD channels so that the next BBC HD channel, the HD simulcast of BBC1, will be introduced before the stated latest date of 2012 with BBC2/3/4 HD following soon after. Otherwise, the Corporation is going to be left far behind in HD broadcasting by its commercial competitors.

Do all these things please, BBC, and nobody will have justification for complaint. Do none of them and, despite the fantastic hospitality you showed us during our visit, we are all still entirely convinced that there will be a case for the BBC to answer, be it to the BBC Trust following the appeal, or to any subsequent "licence fee payers' trust".

To the BBC viewers reading these Blogs, whether you are a seasoned HD PQ campaigner or just a potential HD viewer arriving here for the first time, we recommend that whenever you next view the BBC HD channel you do so with a critical eye. Bear in mind the issues we've raised here and if you think that there is good cause for complaint then make sure you lodge one here. On the other hand, if you are convinced that the PQ is entirely acceptable, or have some constructive criticism, then please do comment about it on these blogs; I'm sure that Andy and Danielle will be pleased to hear and receive both.

And on a final note, the whole group has nothing but praise for both Andy and Danielle for the efforts that they have gone to in engaging with us and also for offering us the opportunity to write this Blog. For that, I thank them once again. They both came across as very dedicated, as did the entire team, and we strongly recommend that those who comment here on their judgements, decisions and actions do so only from a constructive standpoint.

To conclude, we wish good luck to the BBC HD team for a bright, and hopefully complaint-free, future as we hand over to you blog readers and HD channel viewers for your comments about the visit, our Blog and our recommendations.

Postscript

Read part one and part two of this guest blog: Picture Quality on BBC HD: a Viewers' Group Visit.

"Present were: Andy Quested; Matthew Postgate (Controller, Research & Development); Phill Layton (leads the work for the BBC digital transmission chain) and Andrew Cotton (a lead technologist). As mentioned yesterday, a key part of our visit was the time we spent at R&D, and so we wanted to devote a whole Blog to what we did, saw and learned there.

ITU Tests

Our time in R&D began with a set of subjective tests comparing various clips of HD material.

The complete test was based on ITU-R Recommendation BT.500 (Methodology for the subjective assessment of the quality of television pictures), double-stimulus continuous quality-scale (DSCQS).

The method requires the assessment of two versions of each test clip. One of each pair of test clips is unimpaired while the other presentation may or may not contain impairment. The unimpaired picture is included to serve as a reference, but the observers are not told which the reference picture is. In the series of tests, the position of the reference picture is changed in pseudo-random fashion. The observers are simply asked to assess the overall PQ of each presentation by inserting a mark on a vertical scale. See ITU BT.500: Methodology for the subjective assessment of the quality of television pictures.

In our case, we discovered the actual content of the test after it had taken place. We were viewing three versions of each test clip. The 'unimpaired' original source, a version played through the old encoder and bit-rate set-up, and a version played through the current set-up. In each of some 29 tests we were comparing the source against one or the other encoder set-up, or in some cases against itself. This whole procedure was preceded by a rudimentary eye-test using a Snellen chart in which we were simply asked to record the smallest line of text we could read on our test sheets as an indication of our eyesight. The tests were recorded anonymously.

The clips were viewed on calibrated 42" Full HD Plasma displays. The tests were conducted in a darkened, windowless room. The actual test set-up is shown in the picture (and schematic) below:

The test clips were part of the 'SVT High Definition Multi Format Test Set' (taken from the programme Fairytale, made by Swedish Television (SVT)) that are routinely used for this testing. The clips can be downloaded from the Video Quality Experts Group.

In addition, a short clip of Lady GaGa on the Jonathan Ross show broadcast on BBC HD, which we had identified as suffering from PQ issues, was also used.

The test clips that were used are listed in the table below. The 'coding difficulty' as according to the ITU test requirements is also shown, since the tests are meant to use a range of material on this basis. The first three are EBU assessments.

The test clips were preceded by some other clips to 'acclimatise' us for the testing process and included examples of very high quality - a clip of source material from the Children in Need concert at the Royal Albert Hall - as well as of very poor quality - a clip from Bleak House demonstrating inappropriate use of progressive filming mode that literally juddered across the screen. At the time of writing, we are still awaiting the results of this testing from the BBC. [Please see the Postscript to Part 3 of this Blog.]

Comparisons Using our Selected Clips

These tests were conducted in a less formal way than the subjective test, with the complete group free to roam around in front of the three monitors. Each version was shown side by side on three displays, and the group members were able to discuss amongst themselves, and with Andy and the R&D team, what they thought of each version.

The versions were not swapped around, so all the source clips, for example, were shown on the same display. However, we did not know which display was showing which version until the end of the sequences, when we were invited to offer our opinions as to which display was showing which version.

The test clips used are detailed in the table below, together with an indication of the issue that that had been raised in the BBC HD blogs by group members and others, plus any comment or explanation made on the day by the BBC. It should be noted that some suggested clips, showing issues that have not been acknowledged by the BBC, were not made available.
Clips 1-3 were broadcast originally with the old encoder set-up and had been identified by the group to have degraded when broadcast again using the current encoder set-up.

Clips 4, 5, 6 and 8 have only been broadcast with the current encoder set-up but had been identified by the group as exhibiting certain problems.

It's worth stating here that the test conditions were appalling. The set-up of the screens and the group meant that all of us were getting in each other's way as we moved between the displays, it was virtually impossible to see all three at the same time, and the clips were only shown once.

This made it difficult to do meaningful comparisons, and we feel a more rigorous set-up along the lines of the ITU test would have been preferable. Nevertheless, the majority of the group correctly identified the old and new set-ups, and all bar one identified the source. After the source was rapidly identified we were asked as a group two questions in quick succession: which was the old encoder and which the new, and which did we prefer. That some people didn't correctly identify the old and new set-ups may have been partly due to the physical layout of the test, but also because of the ambiguity of these questions since in some circumstances the new encoder performed better.

For example, in the Survivors clip a car's motion across the screen was jerky with the old encoder and handled much better by the new. Further, it took a minute or two for people to decide. This may have been partly because there was a lot of information to process but group dynamics played a role here as well. Undoubtedly some were reluctant to be the first to say anything in case they were wrong, as is natural in group situations. However, others later reported that they were immediately certain which was which, but left it a little while from a sense of fairness, not wanting to deny others a chance to voice an opinion.

Further reflections on encoder performance

However, the new encoder does have overt specific problems. Its blocking was not purely related to the acknowledged mix/fade issue; it was observed on demanding scenes without particular variances in luminance (brightness).

Noise handling is also an interesting area. The source material is often more noisy than many would expect. However, when viewed on the high bit-rate source, it looks natural, one might even say "analogue" in nature. As such it did not seem particularly distracting. However, once it has been passed through the encoder its nature can change. It can look "digital" and unnatural. Hence people commenting on the backgrounds in some material looking "unstable" or even "crawling". This is distracting to the viewer. The old encoder does not produce these artefacts to the same extent; it is less overt and therefore less distracting.

In the simplest terms, if we used a scale of 1 to 10 to judge the respective encoder's performance, with 1 being poor/ showing annoying distracting artefacts and 10 being excellent/no annoying distracting artefacts, the performance range of the new encoder would straddle the old. This is a completely arbitrary scale and must not be taken literally, but one might place the old encoder's performance with a variety of material ranging between say 5 and 7. The new however might range 3 to 8. The old encoder handles the problems presented by the more difficult material in a far more palatable way.

Finally, while the encoder comparisons were very valuable and helped us consolidate a number of insights, the group would not want to subscribe to the implication arguably inherent in the two test set-ups that parity with the old encoder should be the benchmark against which success for BBC HD is measured. As our own experiences attest, the old encoder had its weaknesses as well, as was perhaps evidenced in the welter of complaints on the BBC HD blogs when the bit-rate was previously reduced from 19 to 16 Mbps approx.

Demonstration of the Latest Encoder Fix

We note that the fix is still in testing, so it would be inappropriate for us to attempt to make comment on its efficacy yet. However, what we can say is that we are all very disappointed that this resolution, to a problem evident in many programmes, has still not been implemented after 9 months (at the time of writing)

Failure to Demonstrate the New Encoder at the Old Bit-rate

Considering the effort that the BBC, and particularly its R&D team, put into the day in order to show us all other aspects pertinent to our PQ complaint we were surprised by the omission of what might be considered the one demonstration that could have proven to us, once and for all, that the difference made by a higher bit-rate is insignificant.

Thus ends our blog on the visit to R&D. Tomorrow will be where we draw conclusions from the whole day, and make some recommendations to the BBC."

Read part one of this post: Picture Quality on BBC HD: a Viewers' Group Visit.

So I can now pass you to Paul and the group's HD Blog...

Blog 1: The Television Centre Visit

"Since August 2009, I have experienced noticeably worse picture quality (PQ) from the BBC HD channel. Well, after months of campaigning about this issue on BBC blogs and elsewhere I find myself in the rather strange position of being given my own series of blogs on the BBC Internet blog to write about it. For those who don't know, I made an appeal to the BBC Trust last year about the HD channel's PQ. Following this Andy Quested invited me to gather a small group of other concerned viewers to visit Television Centre for discussions and demos. That visit took place on 30 April 2010 and he has now asked me to put something together for these blogs to let you all know what we found out from the visit.

The first thing I'd like to say is a big thank you to Andy and the rest of the BBC staff for giving us the opportunity to visit them, for being so open with us, and for entering into such enthusiastic debate during our day. I'd also like to thank all those I took with me, and a few others who weren't able to make it on the day, for their help and assistance in preparing for the visit and compiling our thoughts after the event. I'm also grateful to many of you who will have known about this event from these Blogs, and from my own website and correspondence, for your patience while I put these thoughts together.

One of the key tenets in my appeal was that the 40% reduction in bit-rate, made by the BBC on 5 Aug 2009 (from 16 Mbps down to 9.7 Mbps), has had an adverse effect on the channel's PQ. Consequently a large part of the visit was spent with the BBC's Research and Development (R&D) team dealing with this issue. Because it is such a major one I'd like to discuss that aspect in detail in my next blog, which will be published tomorrow. For today, I intend to cover other aspects of the visit. Finally, since I get to write three, the last blog in the series will cover the group's conclusions from the whole visit and also give the BBC some recommendations from us, as discerning viewers, which they may wish to consider when developing their strategy for HD broadcasting.

So, onto Television Centre, and on a bright spring morning I found myself heading for reception with the rest of the group to meet Danielle and Andy and other BBC staff. What follows is a summary prepared by the group of what we did, and what we were told, during the day.

[Back row, L to R: digital_elysium, tagmclaren, AYH20 Front row: paul_geaton, citizenloz, burnlea, mike, jtemplar Picture by HD_fan428]

Meeting with Danielle Nagler

[L to R: Danielle Nagler, Paul Eaton, Andy Quested]

First off, we were all welcomed to the BBC by Danielle Nagler who, as Head of BBC HD, gave us the opportunity to ask some of our more strategic questions regarding the channel. During this time she told us that:

• There was no long-term strategy for BBC HD before she was appointed in July 2008. At that time, programmes were selected for transmission in HD having already been made. Now, they proactively commission programmes in HD and provide guidance to producers.
• They are working hard with production companies to improve the PQ of their HD programmes. This includes giving advice on production techniques and recommendations regarding technology, such as camera choice. Without naming names she said they have themselves been critical of some of the programmes produced so far.
• Danielle said "the need for another channel is very obvious and I'm fairly confident that that will happen, and certainly will happen by 2012", adding, "My expectation is that that would be a simulcast channel and, you know, the research that we've done and the logic, I don't think there's any particular secret, is that the most obvious channel to do it would be BBC1".
• Transmission/playout costs are not within Danielle's BBC HD budget, so she isn't herself directly cutting bit-rates to save money, but she repeated that the BBC is required to use spectrum efficiently.

• Inappropriate use of filters or effects like smoke/fog that softened the picture.
• Incorrect gain, adding noise.
• Focusing not critical enough; this would not have been noticed in SD.
• Inappropriate use of the 25 frame progressive format.
• Inappropriate lighting, in reference to programmes recorded in theatres etc. where the lighting director is more concerned about a small live audience than the greater TV audience.

The Blue Room

Distribution, Bit-rates and Resolution

• Cost of bandwidth is not a significant factor in reducing bit-rates. This was not the reason BBC HD bit-rates were reduced.
• Platform neutrality has to be considered when reviewing transmission parameters.
• Lack of spectrum, in particular with a UK footprint necessary to deal with rights issues, is a constraint. Even with additional capacity coming on-line in future, this will simply be taken up by additional demand for further HD channels.
• Hence there is an obligation to always use the available spectrum efficiently.
• Increasing the bit-rates for BBC HD now, using the spare satellite capacity, would only lead to disappointment when they are reduced again to make way for further channels. (NB: Take note of this statement - there is an inconsistent message here - Why would reducing the bit-rate lead to disappointment if, as we were told elsewhere, bit-rate has minimal effect on PQ?)

• TV Panels can't fully resolve the 1920 signal so he said it would be pointless to increase BBC transmissions to this resolution.

Red Bee Tour (Playout Suite)

Though this was of little relevance to the real subject of the group's meeting with the BBC, as playout has no impact on BBC HD PQ, it was very interesting to see.

Comparison with Other Channels

This was the most controversial aspect of the day for most of the group and, generally, we felt it fell short of achieving the effect that Andy had desired since the problems highlighted were primarily in the source material, not in the transmission. We all knew that other HD channels can suffer from poor source quality too, before being shown it.

Amongst those in the group who have access to subscription HD channels, the general feeling was that those which offer their viewers consistently superior PQ do so principally because of their higher bit-rate and, in some cases, their better source material.

Andy also made the point that some other channels use artificial means to enhance perceived PQ. Overall though, the group again felt such measures would be less significant contributors to perceived PQ than the basics of better source material and higher bit-rate.

At this point in our day we rounded-up and all adjourned to a nearby bar for a drink. This is an appropriate point to end today's blog on our visit to Television Centre. Tomorrow's will be about the visit to R&D, the PQ tests and what we learned from the highly qualified, skilled, personable and enthusiastic team of people we met there."

Ed's note: We'll be publishing part two tomorrow and part three the day after. (PM)

Programme Styles

The last area to talk about is programme style and the techniques used by producers. The HD Channel is expanding the range of programmes made and transmitted in high definition all the time. As the number of programmes grows we are bound to show some that don't appeal to everyone in the audience, just as any multi-genre channel would. Similarly some of the techniques and styles used will not match some viewers' expectations of what is or is not HD.

Posts have suggested we are changing the bit rate depending on the programme because individuals have seen something they really like the look of while others have suggested programmes are up-converted when they don't like the look. Neither is true. The bit rate has been constant since the change in August and the amount of up-conversion (or non-HD) allowed is still 25%. There is though a correlation between comments posted and some of the production techniques used to make HD programmes.
Techniques that cause the most debate are:

• Motion portrayal (using 25p or so called "Film Motion")
• Depth of field or focus
• Noise & Grain
• Softness & Smear

Motion Portrayal - Many programmes are shot using a camera's 25p option i.e. film motion. Drama and Natural History moved to HD from film and wanted to keep the same look. Technically there is more detail in a progressive image than the equivalent interlace image. I have mentioned this before in several posts.

Temporal resolution describes the motion of a sequence i.e. the duration of each frame in a moving sequence and therefore how far an object moves between frames.

Spatial resolution describes the detail of the image and is about the content of a frame.
The BBC R&D white paper WHP169: High Frame-Rate Television gives a very interesting insight on the relationship between the two.

At the moment though we leave the decision about motion type to the Producer and Director of Photography of the programme but there are times when we do comment on the inappropriate choice of 25i or 25p.

Caption: These two images are from the white paper. They are frame grabs of a moving train shot at 300fps (bottom image) and translated to 50fps (top image).

It is worth reading the paper if you want to know how the detail you can see in an image is very much dependant on the frame rate - unless there is no movement. However we are not talking 25 to 50fps to make the difference in clarity you see here it's more like 25 to 300fps!

Depth of Field - Drama productions often use focus to move the point of interest in a frame. Programmes such as Cranford and Wallander also use feature film style cameras with a very small or shallow depth of field which allows the point of focus to play a major role in the story.

There have been many comments about HD being pin sharp and some people believe an HD image should be in focus from the nose of the person in close up to the trees on the horizon. I think many people would find that incredibly distracting and (as some of the posts lead me to believe) people would just be looking at the image quality not the programme. Focus is a very useful programme-making tool, and when used well it adds to the look and feel of a programme.

• Depth of Field entry on Wikipedia
• Alan Roberts' book Circles of Confusion is a far more in-depth look at HD cameras and covers depth of field in Chapter 11.

Noise & Grain - as the range of programmes on the channel increases, we get programmes that are deliberately made to look "dirty" either by style of shoot or during post production. This technique is often used in the cinema too. However if overdone for television it will stretch the encoder and cause unpredictable quality changes. Programmes that do this will usually fail a technical review.

Softness & Smear - some programme makers do not want extremely sharp images and choose to soften the picture either with lens filters or in post production. Both series of Criminal Justice for example used softening to create a very distinctive look. However if this is overdone it will significantly reduce the image resolution and can increase the amount of noise in the image. We do try to limit the amount of lens filtering programmes use but if extreme image softening is required, we encourage people to it in post production so that if the end result is unsatisfactory at least we can ask for it to be removed.

Motion blur (or smearing) is another matter. It usually occurs when the camera shutter is not set appropriately. A programme shot at 25p should use a shutter speed around 1/50th sec (or 180˚). If the shutter is not turned on or is set too long, the images will smear. This doesn't look very nice and we do try and stop people doing it. If the producer wants to add motion blur, again it is always better to do it in post production. Doing it in camera is usually fatal!

Last thoughts:
It has been a marathon blog and has taken me a long time to write! But there was a lot to go through and a lot to check before I published. Over the last six days I have covered most of the technical issues raised in the four key picture quality blogs. I hope it has answered the questions you've asked and laid to rest some of the rumour.

Running through the sections day by day

I decided audio is always forgotten or left 'till last so I put it right at the front. I covered the recent audio issues we've had and some of the measures put in place to prevent them happening again if at all possible. More programmes are being delivered in surround sound now and I hope some of the recommendations due from the EBU next year will give people more confidence to try it out.

There is one thing that does crop up from time to time that I need you to watch (or should that be, listen) out for. Very occasionally the 5.1/2.0 switching gets stuck. It works very well for months then for no apparent reason sticks in 5.1. This has no effect on the audio but it's annoying for anyone using the switch information go to the Pro Logic option on an AV amplifier. Please keep letting me know if you spot it though and which programme is affected.

On Monday and Tuesday I looked at the history of HD encoding on the channel and the EBU recommendations covering programme making and transmission decision we made when the channel started. We review these decisions regularly and changes are made as and when the technology allows us to so.

Wednesday's and Thursday's posts covered how the tests are set up and the much requested PSNR and expert viewing tests results. It is fairly clear now that the new encoder is a lot more efficient than the old and more importantly, capable of many further upgrades as the technology continues to develop.

There is no doubt this series will cause much comment and raise more questions about image quality and encoding generally. I will try and answer as many as I can but in the end we may have to admit we will never be able to satisfy all of you when it comes to what is or isn't high definition.

Andy Quested is Principal Technologist, HD, BBC Future Media and Technology.

• Read part 1 of Andy Quested's HD guide: The Hitchhiker's Guide to Encoding: Before we start
• Read part 2: The Hitchhiker's Guide to Encoding: Life, Encoders and Everything (Or a brief history of HD encoding)
• Part 3: The Hitchhiker's Guide to Encoding: So Many Tests, and Thanks for All the Recommendations (Or the BBC and the EBU)
• Part 4: The Hitchhiker's Guide to Encoding: Mostly Testing (Or how to set up an encoder test)
• Part 5: The Hitchhiker's Guide to Encoding: And Another Test...(Or PSNR and all that...)

PSNR Measurements

This bit is all maths and I make no apologies for it! It is one of the methods used to evaluate the effect processing has on signals and ultimately picture quality. PSNR is a derivative of the Signal to Noise Ratio comparing the maximum possible signal energy to the noise energy ^[1].

PSNR has been shown to have a high correlation to subjective picture quality (eyeballs) when a single codec is used and cross-references between sequences are not made ^[2].

Equation 1- the Mean Square Error (MSE) of the mth frame is calculated ^[3]. Y_in and Y_out represent the luminance of the input signal from the play out server and output from the encoder respectively, and Y(I,j,m) is the luminance value of the pixel in position (I,j) in the mth frame.

Equation 2 - the PSNR of the mth frame is calculated ^[3]. B is the number of bits per sample used in representing the video. The test procedure uses 8-bit linear pulse code quantisation.

In accordance with industry recommendations, only the luminance PSNR is measured ^[3]. Typical values for the luminance PSNR for emission encoding are between 30 and 40dB.

This is capped to a maximum figure because an 8 bit system cannot accurately represent the original, analogue video image. In practice, the industry recommendation uses a cap of 50 dB ^[3], almost 10dB lower than the theoretical maximum. Above 50dB the quality of the coded image is more than sufficient for all but the most critical applications.

The median PSNR is the value of the 50th percentile of the individual frame PSNRs of a sequence listed in ascending order. The accepted critical value for this type of measurement (as used by the Moving Picture Experts Group (MPEG) in deciding if a toolset should be included in an implementation) is 0.5dB.

0.5dB represents a visible difference in picture quality across the range of PSNR values. As the PSNR increases and coding errors become less visible, the visibility threshold increases above 0.5dB.

• ^[1] L. Hanzo, P. Cherriman, and J. Streit, Wireless Video Communications - Second to Third Generation Systems and Beyond, ser. Digital and Mobile Communication. 3 Park Avenue, New York, NY, USA: The Institution of Electrical and Electronic Engineering Press, 2001.
• ^[2] Q. Huynh-Thu and M. Ghanbari, Scope of validity of PSNR in image/video quality assessment, IET Electronics Letters, vol. 44, no. 13, pp. 800-801, June 2008.
• ^[3] Objective perceptual multimedia video quality measurement in the presence of a full reference, International Telecommunications Union Telecommunication Standardization Sector - Pre-published Recommendation J.247, August 2008

PSNR of the current encoder setting compared to the old encoder.

The material used to test the encoders was a selection from the EBU test sequence and clips from the BBC HD Channel promotion.

All test material is copied to the playout server (100Mbs MPEG2 I-frame coding) and then onto the transmission encoder. The Final Cut Pro computer is used as a store for the transmission decoded material.

The results are displayed as curves on a cumulative graph. The x-axis is the measured PSNR and the Y-axis indicates the percentage of frames with a PSNR value less than or equal to that PSNR:

The new encoder has a median PSNR figure 0.5dB greater then the old, a just noticeably improvement in perceived picture quality for the majority of the test sequences. However the very easiest, least critical material, where coding artefacts are usually not visible, coded with a better PSNR on the Old Encoder. We are looking into this at the moment but one explanation could be the new encoder handles image noise differently to the old encoder.

Reading PSNR curves is not straight forward. A difference of about 0.3dB is just visible to an expert viewer at normal viewing distance while a non-expert viewer will see a difference of 0.5dB or more, as mentioned in the last paragraph of the PSNR explanation.

Where differences occur in the curve is important, at the lower end (the further left you go) the more critical the measurement. At normal viewing distance a non-expert may see a difference in quality for a change of 0.5dB or slightly less. At the far right of the curve the picture quality is much higher and differences are more difficult to see so an expert may not see a difference under 0.5dB and a non-expert may not see any difference below around 0.75dB or even 1dB.

Mix/Fade problem

We were aware of a problem with mixes before the new encoder went into action. During tests it only appeared in certain modes and wasn't severe. The overall improvement in quality outweighed the degradation it caused.

Unfortunately one of the first live programmes to be transmitted was also a programme that would highlight the mix/fade problem.

The Match of the Day, West Bromwich Albion vs. Newcastle United game kicked off with a very high contrast change almost dead centre of the pitch. As the game moved from bright sun to deep shadow the cameras had to be racked over several stops (opening and closing the iris).

A mix as you know is a transition between two different images. Coding errors caused by the mix tend to be hidden by the changing images however racking a camera is actually a mix between two different brightness levels of the same image so there's no where for the errors to hide and they become very visible. I apologised and explained we were applying a temporary fix.

Although the temporary fix is still in place we have now seen an update that improves mixes, fades and lighting changes and are just waiting for it to be incorporated into a software upgrade.

While we had the location recordings of the match to analyse the mix error, we had a chance to compare the PSNR curves through the new and old encoders:

For the majority of the sequence, the new encoder has a higher PSNR than the old with a median increase of about 0.2 dB (not a noticeable difference). The old encoder is better for approximately 8% of easy to encode scenes and 1% of difficult to encode scenes, but this is most likely due to the camera racking i.e. the mix/fade issue itself!

PSNR testing shows the new encoder is doing better than the old except where the source material has a significant amount of noise. To help this we are testing the encoder's noise reduction options to see if adding a small amount improves the look of noisy images. I will update the blog as soon as we have some results.

Subjective expert viewer evaluation

The second part of the testing process is all about looking at pictures. We use 42" plasma and LCD displays to do this, comparing the quality of the new and old encoders against the original material on the play-out server.

Expert viewing is a tricky business and as one of our experts discovered a risky one too! It involves watching the same set of images again and again and again and...

To minimise the risk of complete insanity, it is usually better not to have the audio on.

However even this didn't prevent someone coming in to work one day and asking to be taken off picture evaluation for a while. He said was on the train just looking at the country side passing by when he was convinced he saw compression blocking in the leaves of trees. This is not something you want to happen - so be warned!

Evaluating picture quality this way means a long time spent in darkened rooms. We watched a lot of images from the EBU test material and the BBC HD promo tape, comparing the new encoder with the old encoder and the play-out server on each sequence. It is important to have the EBU standard sequences to judge picture quality but we also have a test sequence made up from material that has known problems and shots that are difficult to code, to test the encoders to the "limit".

The new encoder produces images that correlate quite closely to the PSNR results. Programmes with low or no noise are noticeably better than they were on the old encoder. However, where the original images have noise we can see it on the new encoder's output but not on the old, suggesting that the new encoder is attempting to pass on more of the original image and confirming that a bit of noise reduction should be tested.

Dark pictures are inherently noisy, either because there is gain in the camera or the signal has been stretched too far in colour grading. We actually have a very noisy sequence that has too much camera gain and was stretched too far in post production. We used it during the tests to push the system, and even turning the bit rate up to just over 16Mbs made no difference to the image. We are trying a few new and different parameters that seem to improve noise handling and reduce the effect on screen. Again I will keep you posted.

Tomorrow is the last part of this epic, I will look at some of the techniques programme makers use that can have an impact on perceived picture quality.

Andy Quested is Principal Technologist, HD, BBC Future Media and Technology.

• Read part 1 of Andy Quested's HD guide: The Hitchhiker's Guide to Encoding: Before we start
• Read part 2: The Hitchhiker's Guide to Encoding: Life, Encoders and Everything (Or a brief history of HD encoding)
• Part 3: The Hitchhiker's Guide to Encoding: So Many Tests, and Thanks for All the Recommendations (Or the BBC and the EBU)
• Part 4: The Hitchhiker's Guide to Encoding: Mostly Testing (Or how to set up an encoder test)

Test results can only be seen to be accurate if as many variables as possible have been removed or at least minimised, and the methodology produces repeatable results.

Our picture quality assessment is based on a combination of Peak Signal to Noise Ratio (PSNR) measurement and expert viewing. Obviously, if done correctly PSNR measurements are accurate and repeatable within a very small tolerance range but the results are in the end only numbers. Eyeballs on the other hand, even expert ones, are slightly less predictable but can still produce repeatable results within an agreed tolerance but eyeballs are essential to judging overall picture quality of any device.

Removing Variables
Ideally the only variable in a test should be the device actually being tested. In a delivery chain with domestic devices from many manufacturers this is not as easy as it sounds and testing every combination of is just not possible.

To minimise the number of variables we use:

Test material - EBU test sequences used by all members for subjective picture quality testing.

Test Path - this is a duplicate of the actual transmission chain and includes the playout server and the continuity path. The playout server can also be bypassed to simulate live studio programmes.

Encoder - the bit we are testing!

Receivers - this is slightly more difficult because processing in set top boxes varies between different manufacturers and different models from the same manufacturer. For encoder comparison testing though we can use broadcast quality receivers. We also use domestic set top boxes from several different manufacturers to make sure the domestic receiver technology does not cause unexpected results.

Displays - this is even more difficult than dealing with receivers. We should use a graded broadcast monitor (a display where all the parameters are know and are adjustable to give a stable accurate image). There is still some debate around flat screen Grade 1 displays - until recently CRT was the only option and they're not easy to get hold of now!

• http://broadcastengineering.com/test_measurement/broadcast_monitors_video_801/
• http://broadcastengineering.com/news/replacing-crt-1005/index.html

In practice we have to think about what people are actually watching at home so we use 42" Plasma displays for comparisons and Plasma and LCD for quality assessment.

A 42" display is slightly bigger than the current highest selling TV size so this gives us a bit of head room during the assessments. The results will remain valid until the TV size passes 42" or the panel display technology changes dramatically (e.g. OLED or Laser etc.) and a new one takes over as the primary panel technology.

Viewing distance - The ITU has set out criteria for viewing distance in the document ITU-R BT.500 "Methodology for the subjective assessment of the quality of television pictures"
In the section "General viewing conditions for subjective assessments in home environment" it suggests preferred viewing distance (PVD) measured in screen heights (h) for assessment of picture quality.

The PVD suggested for 16:9 screens are:

The BBC HD Channel expert viewing is done at 4h (four times the height of the display used) so slightly closer than the ITU recommendations. This is based on the premise that the average domestic viewing distance is somewhere between 4 and 6h. Some interesting work done by BBC Research (WHP090) suggests the most common viewing distance is actually 2.7m no matter what the screen size. This is not a "serious" piece of work, more an observation of current viewing habits!

How to calculate your viewing distance?

There are two ways to calculate the height of a 16:9 television:

1. Televisions size always refers to the diagonal so first calculate the angle of the diagonal (Θ°):

Θ°= 29.36˚ All 16:9 televisions should have the same angle no matter what the screen size.

Using the angle calculate the height (h) of the television knowing the screen size:

so (h) is near enough half the (Screen Size). Multiply this by 4 to get the viewing distance.

Unfortunately this assumes all televisions are actually 16:9 which they are not - so just in case method 2 might be more accurate!

2. Use a tape measure
Or just use the table below!

When the 4 - 6h criterion was originally devised, screen sizes around 42" would have been considered extremely unusual. Now 42" screens are common and could soon be the highest selling set size. Due to the distance between the television and the back wall of the room you watch it in; WHP090 may prove to be spot on!

In the quality viewing area at BBC Research there is a mix of set top boxes and display types. Viewing distance is set at 4h.

Eye sight - before the BBC HD test channel started, a few of us spent several days in a very hot room above a shop by Oxford Circus doing viewer testing of HD and SD. We had two top end (at that time) 40" flat screens and a 28" CRT, the highest selling TV size at the time. We also had a selection of BBC HD test material as this was before the EBU had agreed test sequences.

The pictures were coded in HD using the encoder profile we proposed to go on air with and also in SD using the BBC 1 encoder and bit rate. The HD signal was fed to one LCD and the SD signal to the other LCD and the CRT.

After the first day we had a number of results that suggested no difference between the SD and HD LCD - quite confusing because even viewing at a considerable distance there was a marked difference between the two LCD images.

The next day we took a standard eye test chart and asked anyone who couldn't see a difference between the two LCD screens if they would mind taking an eye test. Everyone who took the test should not have been driving without glasses and many of them didn't event realise they should see an optician!

Anyway the expert viewers do have eyesight within the normal sight range (with glasses if they usually use them for watching TV).

This sets out the criteria for visual picture quality assessment but as I said earlier, testing uses a combination of visual assessment and a Peak Signal to Noise Ratio (PSNR) measurement.

Tomorrow will be all about the picture quality tests and the PSNR test results.

Andy Quested is Principal Technologist, HD, BBC Future Media and Technology.

• Read part 1 of Andy Quested's HD guide: The Hitchhiker's Guide to Encoding: Before we start
• Read part 2: The Hitchhiker's Guide to Encoding: Life, Encoders and Everything (Or a brief history of HD encoding)
• Part 3: The Hitchhiker's Guide to Encoding: So Many Tests, and Thanks for All the Recommendations (Or the BBC and the EBU)

BBC Research and Development is one of a diminishing number of European laboratories with the expertise and facilities to carry out testing for, and make contributions to the EBU's technical committee. I am extremely proud to work with the experts in BBC R&D and for the opportunity to contribute to the work they do.

Many of you have mentioned our current frame size and said:
"1440 Horizontal pixels is not HD"

To answer this I can point to several EBU documents that cover this point:

EBU - TECH 3328 Current Status of High Definition Television Delivery Technology (May 2008)
...In addition, the Sony HD-CAM and Panasonic DVCPROHD formats record only 1440 samples per line (with camera scanning at 1920 x 1080i/25). There is arguably no point in broadcasting material derived from this format at more than 1440 samples per line (although on the HD-SDI interfaces, a 1920x1080i/25 signal is carried).
A similar situation exists with the DVCPROHD format that horizontally sub samples (down-filters) the 1280 x 720p/50 format to 960 x 720p/50 (though on the HD-SDI interface, the signal is 1280x720p/50).

This document mentions 1440 and 960 as used in production. The HDCam tape format and some cameras that record to the DVCPro100 format use pre-filtering to reduce horizontal resolution before the signal is compressed. These cameras and formats are still widely used but are gradually being replaced by cameras and recorders that can compress the full 1920 or 1280 horizontal resolution.

Another document:

EBU - TECH 3333 EBU HDTV Receiver Requirements (March 2009)
The following image sampling structures shall be supported (see TS 101 154 V1.9.1, which defines further formats beyond those listed here).
1920 x 1080, interlaced, 25 frame/s (50 fields)
1920 x 1080, progressive, 25 frame/s
1440 x 1080, interlaced, 25 frame/s (50 fields)
1440 x 1080, progressive, 25 frame/s
1280 x 1080, interlaced, 25 frame/s (50 fields)
1280 x 1080, progressive, 25 frame/s
1280 x 720, progressive, 50 frame/s
1280 x 720, progressive, 25 frame/s

Tech 3333 is about receivers and shows 1440x1080 is an acceptable standard for high definition transmission.

It is worth pointing out that HD-Ready does not actually specify the horizontal resolution for an HD display. HD displays must have a minimum vertical resolution of 720 pixels and display 16:9 images correctly. There is no mention of the required number of horizontal pixels!

Several other posts have suggested the BBC is not meeting EBU standards for transmission with the current bit rate and have said:
"9.5Mbs is not in line with the EBU recommended high definition bit rates"

Again there are several EBU documents covering this point but the key thing to take account of is the date of the references. This document for example:

EBU Tech 3334 Accommodation of HDTV in the GE06 Plan (Feb 2009)
EBU tests of stand-alone MPEG-4 encoders of different vendors have suggested ^[5] the following minimum fixed bitrates in order achieve an HDTV image quality providing a significantly better quality perception compared to good quality SDTV (e.g. 6 Mbit/s MPEG-2) for a wide range, including critical content:

• For the 1080i/25 HDTV format and horizontal sub-sampling to 1440 samples a minimum bitrate of 12 Mbit/s is recommended
• For the 1080i/25 HDTV format and no horizontal sub-sampling a minimum bitrate of 12 - 14 Mbit/s is recommended
• For the 720p/50 HDTV format and no horizontal sub-sampling a minimum bitrate of 10 Mbit/s is recommended.

The key here is the ^[5] after "suggested". Reading the bibliography, point [5] refers to:
EBU Tech 3328 (Current Status of High Definition Television Delivery Technology)

Tech 3328 is dated May 2008, and was published long before we started tests on the new encoders for the BBC HD Channel.

Bit rates are also mentioned in an EBU presentation by Adi Kouadio (Asian Broadcasting Union symposium in March 2009).

Trends & implementations of HDTV Broadcasting
...Minimum (video) bit rate to provide HD quality (from EBU tests - BPN085-087):

• 720p/50 - 10Mbps
• 1080i/25 - 12-14Mbps...

EBU test documents BPN085 to BPN087 detail tests carried out on specific encoders. These tests were undertaken over the last two years but none refer to the encoder we are currently using.

Tests have demonstrated that at transmission bit rates, H264 encoders should deliver approximately a 2:1 efficiency over MPEG2 encoders. Depending on the manufacturer and their current stage of development good HD at 8-10Mbs is achievable now.

The BBC has made contributions to most of the documents mentioned above and many others besides. We are also involved in testing and trials for high definition production, contribution and transmission compression and were one of the earliest activists in the latest round of HD activity in the EBU.

The majority of the documents produced by the EBU are Technical Recommendations and are based on work done at a specific point in time by the members. After publication many groups continue to work on revisions and updates to the recommendations as the technology behind the encoders (in this case) continues to develop and improve. EBU Tech 3334 acknowledges that:

...with the expected future developments in video coding, it is assumed that HD fixed bit rate requirements will be reduced to 8-10 Mbit/s per programme. There will also be advances in the transmission system such as DVB-T2...

For some strange reason the encoder manufacturers don't all bring their latest offering to market at the same time! Two or three times a year they do show off concepts or the next generation prototypes at the main trade shows but products to the broadcasters arrive as and when they are stable and ready. This year for example, several manufactures were showing early versions of HD coding at 4-6Mbs (average not minimum).

One final point worth noting here is the life cycle of the hardware. As I mentioned earlier, the old encoders had reached the end of their life but the new encoders are at the beginning. They exploit more of the AVC toolset than the old encoders even though we are on version 1 of the firmware. We will continue to add improvements as and when they are available.

Tomorrow I want to concentrate on picture quality analysis and how we set up the encoder tests.

Andy Quested is Principal Technologist, HD, BBC Future Media and Technology.

• Read part 1 of Andy Quested's HD guide: The Hitchhiker's Guide to Encoding: Before we start
• Read part 2: The Hitchhiker's Guide to Encoding: Life, Encoders and Everything (Or a brief history of HD encoding)