Feedback [PSA] The X9S outputs wrong color levels

Sadly the X9S isnt the "one set top box that does it all" after all. It has a serious flaw, namely color reproduction.

Long story short, I calibrated a fourth 3D LUT for my OLED today, verified the result with test material, hooked up the X9S again, an was presented with grossly different colors (if you use the youtube Addon in Kodi, the red of the icon should give you a clue).

So I started measuring its color output - and it is a horrorshow.

So lets get through the list.

Measured only rec 709 SDR (standard dynamic range) so far, no HDR

1. Both YCbCr options output a color space that doesnt even reach the rec 709 triangle - producing significant errors in red and magenta. (In addition to the other flaws mentioned below.) DONT USE THEM.
2. The RGB 444 color option outputs a signal, that reaches the rec 709 primaries, but like every other color space the X9S offers, has significant issues with uniformity.
3. the 10 bit and 12 bit signal output options introduce significant luminance errors as well (that one I'll have to double check the signal chain on, but those are my results after one test run, and I'm usually pretty thorough... ).

The overall error profile can be summed up as follows:

- 100% (saturated) colors have a luminance deficiency of -15% and more.
- the luminance deficiency is unevenly distributed across different saturation levels
- tint is tilted towards red (high errors in the red and magenta saturation sweep)
- brightness (of black) is too low

Now - the X9S comes with a "utility" in the quick settings that allows you to adjust picture parameters. Some of the issues can be mitigated that way - but there are still significant errors present after adjusting them.

First, here are some settings, that force the X9S to output a little closer to the actual standard.

Again, you want to use RGB 444 (make sure your TV adjust the full/limited color space toggle accordingly, if your near blacks get crushed ("become more black, than they should be"), you have to adjust a corresponding setting on your TV set), both 10bit and 12bit output options disabled (otherwise the errors are even higher).



Brightness is off by 1,5 ticks of the X9S parameter menu, i adjusted the slider by +2, as it will allow you to keep the TVs brightness slider a tick lower, otherwise you might sacrifice a bit of your TVs black level.

Contrast is mostly fine, I increased it by +1 to get better xy values at the 100% targets.

Tone (tint) is off by the largest margin. I had to adjust it by -4 to get the red saturation sweep line (skintones...) on target. This sacrificed color accuracy of the blue saturation sweep line, so you could try only -3 potentially.

Saturation is not uniform in general increasing it by +1 was a cointoss decision, you could also leave it at 32.

Before those adjustments I was greeted with Color errors of dE(2000) 8 and higher (mostly at the 100% color targets), after the adjustments, dEs went to mostly about dE 3 and below on all saturation sweeps.

This is a screenshot of the color performance of the X9S after said adjustments were done.



You can see, that there are still errors of above dE(2000) 3 present, and those are undefeatable by adjusting any of the X9S settings or any of your TVs settings.

The measurements were taken with the X9S going through a 3D Lut that with "non broken devices" shows an avg dE of 1.05 across 3000 measured points. So any error margin above that is introduced by the X9S.
I also double checked this by connecting the X9S directly to the TV, and the same error profile surfaced.

So...

You'd think the first item on the list of a "TV movie box manufacturer" would be to make sure, that that thing outputs colors according to spec and standards. Well - the Zidoo X9S doesnt. And that sucks.

You'd think if a manufacturer implements high bitrate signal options, they'd make sure, that they don't distort the signal even further - well, with the X9S thats exactly what happens if you use them.

Also, to top things off - I could actually fix all those issues, creating a specific 3D Lut for the X9S (using equipment, that far exceeds the pricepoint of the X9S), but the Zidoo box isn't compatible with Calmans pattern generator for android ( https://play.google.com/store/apps/details?id=com.spectracal.mobileforge ), because the app has issues receiving the X9S IP address in the local network.

To make it short - I condsider this a fail of major proportions, and something that simply doesn't go together with the X9S' image, of being on the "premium" end of Androud Set Top boxes.

The cake is a lie. When it comes to most of the picture toggles in in the X9S settings menu. You can show the user 5 different signal options, but they all are useless, when they only distort your picture even further.

n_p

 

False Alarm.

Sorry to everyone I set scrambling towards changing the settings as described above - It turned out to be a compatibility issue with my LUT Box, I'm troubleshooting right now.

I remeasured the X9S directly on my TV again and this time the measurements from YCbCr were just fine using all the X9S' default settings (32 across the board). I double checked this before and got the same strange readings - I don't know what happend there yet.

Again, I'm sincerly sorry for the false positve and for attributing a problem of my LUT box to the X9S. Everyone go back to your old settings.

 

Never mind, so can I lock this post?

 

Give me two more days to explore the issue. I'll PM you once I think you can lock the thread.

Currently it looks like a strange handshake issue, that causes the X9S to output incorrectly. Also there is the issue, that I once checked it against the TV natively and the issue croped up there as well is very strange. Current working theory is, that as soon as the X9S' handshake goes wrong, it outputs a strange color space.

The same error also is attributed to its HDMI IN port - so I will be able to create a LUT compensating for it after all (attaching my pattern generator to the X9S' HDMI IN) -

but the issue thats relevant for here is, If - and when the X9S does this.

Until then, don't change your on box settings, yet.

Thanks.

 

Forward to your design team, that the Zidoo X9S has a severe design flaw, that it is mixing RGB full (0-255) and limited (16-235), and that it is distorting all colors as a result of this.

The Zidoo aways outputs RGB full - but in both Kodi and the Zidoo Media Player - will mask it by internaly shifting the signals brightness.

To reproduce this, simply put the Zidoo in any off its YCbCr Modes (= the same modes it defaults to if you leave it in auto), then open a Black Clipping pattern in either Kodi, XDMC or the the Zidoo Player. The pattern thats showing is "about right" for a limited signal chain (X9S set to YCbCR (which only ever outputs a limited (16-235) signal), TV set to receive a limited (16-235) signal). With the emphasis on "about right" --

then go into your TV menu and switch the TV RGB limited/full setting to full. The image will brighten up (because the TV will now expect an even darker black level), but on the test pattern you will suddenly see PC black levels 0-15 blinking. They shouldnt be there. At this point the X9S is outtputing a full RGB (PC levels) signal, but shifting its brightness, so it appears as a limited RGB signal.

When you set the X9S to output "RGB 444" - it doesn't actually switch to PC Levels, it just shift down the brightness of the signal to where it should have been in the first place.

To top it off - if you plug a signal source that outputs limited into the X9S' HDMI port, it gets attributed a third brightness level.

You can literally watch a Black clipping pattern on the HMDI passthrough in the HDMI app, and it will apear washed out, allthough the entire signal chain is set to limited, then switch to a Black clipping pattern on ZDMC, which will appear "a little crushed" - then turn your TV to expect a full range (0-255) signal, and watch ZDMC show you the entire 0-15 brightness range, although it "supposedly" is outputing YCbCR - which is limted range exclusively.

I'm sorry, but this is a Cluster*wordwithF*.

In the process of just "brightness shifting" the entire signal - colors get remapped wrongly - causing those measurable and clearly visible errors.

Also - when you set the X9S to output RGB 444, it shifts brightness down even further - but the issue now is, that whoever was responsible for faking the fake YCbCR output to fake Kodi/ZDMC/Zidoo Player black level 16, dindt take into account that someone could actually use the full RGB setting, which means, that the players blacklevels are still shifted, producing an additional error margin.

So to put it simply - the X9S is a messed up piece of engineering that doesn't work as advertised. It isn't standard complient, and if you input it into any equipment which deals with a limted signal range only (= the standard for all AV equipment) - it will produce massive, measurable errors. Because of the brightness shifting it produces less obvious errors in the darker part of the image as well, and those are undefeatable.

I'll be returning my X9S asap - thank you very much.

If anyone from Zidoo wants to respond, be my guest - I'd love to hear the explanation for all of this.

 

First of all. You are very professional on the field.
Can I make it simple that you think 1295 adjust the brightness up mistakenly?

 

I made it work. Thanks to a series of lucky coincidences.

Thankfully, the Amazon App Store Version of Spectracals MobileForge (newer than the version on the Google Play Store) works on the X9S - which allowed me to finally get a pattern generator on the device that actually reads the same image as it is displayed in Kodi, ZDMC and the Zidoo player.

After remeasuring with the pattern generator (think profiling speed x20), it turns out that RGB 444 at 10 bit distorts the signal the least. (Starting from default values of 32 in the picture parameter settings of the Zidoo box) But the way its output is set up so the signals brightness is a little low (this is not what I'm lamenting about, I'll explain later later), which means I'd have to push my OLEDs brightness to a point, where I loose "perfect black" (no glow in a dark room environment). So thats not going to happen.

Thankfully the "picture parameter settings" on the Zidoo box exist, so using RGB 444, 10 bit - I just pushed brightness to 34 (+2) and lowered the TV level by one tick to get them to align. (This is ok on an LG Oled sice it oversaturates low Luma colors anyways - so "clipping" saturation near black using the X9S isnt that bad.) Then I created my 3D LUT, using Mobileforge. It measured fine, and I am ok with the image now.

That said here are the following issues with the X9S' color output as it stands right now.

1. YCbCr modes distort the image heavily. Delta Luminance on red explodes, Cyan gets shifted.
2. There is some seriously funky stuff happening with brightness levels across different scenarios. YCbCr Modes might still output YCbCr - but they arent clipping levels 0-15, which is strange (every other Android Box I've ever looked at does). The brightness Level for the HDMI IN is actually "set up for PC Levels (0-255)", so a limited (16-235) signal (the standard for all AV equipment) will look washed out. This is undefeatable. It also hints at the X9S using PC Levels as the internal Color Space it is working with.

This doesnt have to be the case, and isn't even something that should have to change, but if you do - you have to make sure, that you do the conversion to limited color space output (YCbCr Modes) correctly - and my current assesment is, that the X9S doesn't.

I was seriously angered when I found out, that the X9S as I had set it up before (limited Color Space) altered colors and didn't output a clean signal. I love (don't use this word for "Electronics" usually) the box in EVERY other way. Its a bold concept, an ingenious realization using very geeky but potent software concepts, heck it even plays 90% of the UHD Demos I threw at it without a hitch, and transmits the HDR flag to the TV correctly - just make sure you look into your color output options in a bit more detail.

I could as well - but I'm not sure I should at this point.. Most people usually don't obsess about that stuff - but they'd want the confidence, that it works as expected.

 

@n_p thanks for your professional Report. Finally someone who comes with facts, because the problems are also present with me and many other users.

Regards Lony

 

I haven't checked the latest firmware, but on the previous firmware the YCbCr code values (the actual pixel values) were perfect for 8/10/12 bit output. I haven't checked RGB output, so can't comment on that. When time permits, I will check the latest firmware.

 

Thats definitely NOT the case. I have finally taken some time to profile the X9S' color output, and here are the results.

Firmware 1.3.0

8/10/12bit should not make any difference to what is documented next (at least not if all devices in your chain support 12bit signal encoding (mine do)).

First, the output closest to the color standard is RGB444. So lets look at it to establish a base level.

X9S (set to RGB 444) going through my 3D LUT box (no LUT correction enabled):


X9S (set to RGB 444) going DIRECTLY to my TV:


X9S (set to RGB 444) - color profile (no LUT Correction, just the output as reproduced by the TV (LG 2016 OLED)):


Those error margins are OK, they are about "where expected" on that TV.

Now watch what happens, when you set the X9S to output in YCbCr 444 (TV set to signal space limited, so this was taken into account, according to the black level shift, the X9S also expects the TV to be set so limited (black clipping pattern in ZDMC)) -

X9S (set to YCbCr 444) going through my 3D LUT box (no LUT correction enabled):


X9S (set to YCbCr 444) going DIRECTLY to my TV:


X9S (set to YCbCr 444) - color profile (no LUT correction, just the output as reproduced by the TV (LG 2016 OLED)):


The color errors explode (dE bigger 20, while 3 is usually considered a visually significant deviation). rec 709 isn't reached anymore.

Same goes for YCbCr 422 (similar enough to 444, so I'm just posting the images as links)

YCbCr 422 through LUT box, no LUT enabled:
http://i.imgur.com/3bij9gz.png
YCbCr 422 directly to the TV:
http://i.imgur.com/2rNOL0O.png
YCbCr 422 color checker chart:
http://i.imgur.com/P8h8aB6.png

Let me repeat again, that this is a huge issue.

Further more, when you set the X9S to output in RGB 444, it will revert to a YCbCr color setting on the next reboot - without showing it in the Quick select menu - and you have to switch to a YCbCr colorspace, and then back to RGB 444 again, to get RGB enabled correctly. So the ONLY setting that outputs correct color values - doesnt survive a reboot.

Whoever designed this, this is NOT how you do color. Something here went seriously wrong.

I suggest that you first patch the bug, where RGB 444 doesnt survive/stick after a reboot (although the option is still selected in the Quick Settings menu -). And then learn how the YCbCr color space is supposed to be addressed.

Currently, in its default settings, the X9S is outputting a VERY, VERY wrong color space.

(The only fix currently is to manually set it to RGB 444 (if your TV supports it, since PC level color is NOT an AV standard, but most current ones do) - after every boot.)

Please get back to me with a response, as soon as one of your engineers could look at it.

Currently you are selling a "broken" product, that isn't standards complient and reproduces colors vastly incorrectly.

 

Also, the color errors are easily visible to the naked eye. Download the youtube app in ZDMC/Kodi, and just look at its Icon. If you use the X9S default settings, the red will not resemble youtubes red even remotely.

Do a self test. We are not talking about "white level a bit wrong" type problems, those are serious chroma issues, as in "all of a sudden the rec 709 color space isn't even reached anymore". There is something seriously wrong with the X9S here.

 

Oh my God. Thank you n_p
Zidoo team you should reply soonest.

 

n_p, which is your LG OLED TV?
Mine is a 55B6D.
Could you also suggest me a downloadable reference video to see the wrong color mapping?
(I don't own your sophisticated instruments and also I'm absolutely newbie in this "world" so can't replicate your tests...)

 

@n_p : do you realize that you are actually measuring the chromaticity matching function of your display with these tests. It is only an indirect measurement of the accuracy of the source output. What you need to be looking at is measure the code/digit value (YCbCr or RGB) of the output signal with an analyzer. If that matches with the pattern value, then there is nothing wrong with the decoding. You can use a DVDO AVLab TPG for 1080p and 4K Rec.709 up to 30Hz. For 4K, YCbCr 4:2:0 or 4:2:2 you will need a Quantum Data/AstroDesign HDMI analyzer.

As I said earlier, I have only checked it on one of the earlier firmware and it was correct on that one. When time permits, I will check this on the latest firmware (1.4.4).

 

I checked this on 1.4.4 firmware and there is absolutely nothing wrong with 8-bit YCbCr 4:4:4 output. RGB output has expanded and remapped (16-->0, 235-->255) levels. That is the only negative that I can say. Personally, I prefer that RGB output for video to be limited or at least provide the user with an option to change the dynamic range of the output. I couldn't find such an option on the X9S, may be I missed it.

The tables below show the actual YCbCr pixel values measured from X9S HDMI output. I have used Ted's test patterns for testing. The test pattern clips were played in ZDMC (16.4.3). All picture paramters were at default values (32). You can see that there isn't a single error!

Saturations (100% stimulus)


Color Checker SG

 

Thank you. So what would be the best setting) RGB 444/10 bit or what?

 

To spell it out clearly, we now have differing statements as to if and when the X9S outputs colors wrongly.

But we also have people in here throwing smoke to help in consealing the issue.

The methodology I used is not the issue, if the TV in question handles YCbCr input correctly (it does, of course, I've meassured other input devices, none of them show the described issue).

And if the pattern generator used (Calmans Mobile Forge app) puts the right patterns on screen). I've heard in the past that it might not be bit accurate, but it certainly would not produce color errors of that magnitude. Also I've independently verified the issue using avsforums free rec709 test patterns, being diplayed in ZDMC. The measurement devices used were both an i1d3 and a Colormunki Photo - which were tested to be accurate (within 2% of greyscale deviation, also remember that the issue doesnt crop up in RGB444 mode)), by compairing sample variations.

Also - and you can rely on this as a fact, the color error in 100% saturated red is visible to the naked eye, judging by youtubes brand red, which becomes a glowing orange in all YCbCr modes.

So if the testing methodology is not the issue - whats left that explains the error?

I don't know... Firmware versions? (I tested on 1.3.0 and not 1.4.4) one of those statements being produced by a marketing plant? Differences in the actual image controllers used in production (its not that uncommon for chinese manufacturers to source cheaper parts midway throught production).

I'm a bit miffeed, because no one in the calibration community whos worth their salt would suggest, that this "probably is the TVs fault" in a scenario like this. With the data published above. So this is an entirely out of left field suggestion, that immediately makes me go "PR is trying to mitigate the issue here".

Also please dont forget that you have suspicious "please don't touch this setting" messages in the UI and Zidoo has messed up framerate syncing as well, in some cases, apparently.

More independet testing would be great.

Or, just look at the youtube red with YCbCr enabled. If it isn't obvious to you from there, idk - we have different boxes...

edit: Also - if this is a handshake issue, f.e. and it provokes TVs to map the wrong color space, measuring the signal output might as well be considered a "lab test", with no real world attributability. Thats here to say, that one method isn't better than the other necessarily - but still, I - and no other "color tech" would expect that much of a color error in that measuring case.

To even suggest, that thats the reason for why I am measuring this color output is an outright misrepresentation. Simply considering error margins.

I also have to specify, that my results were taken with 1080p50 and 1080p60 enabled. So maybe its an error that effects 1080p specifically...

Thats why I insisted, that zidoo have their own engineers looking at this, so we could get their assesment of the problem as well. They have multiple boxes, they can test multiple firmwares, and multiple resolutions easily.

 

As far as verifying this "with your eyes" - take youtube red (for example dl the youtube app in ZDMC, then look at the icon), compare how it looks in RGB444 and YCbCr color modes. The difference should be stark, not subtle.

As far as "recommending a test video" goes - the human eye/mind is not that good in judging color differences accurately if you dont provide comparative samples (pull out your phone (most iphones are calibrated to a whitepoint of 7000 Kelvin, but the difference in "youtube brand red" is even bigger, you should see it becoming a bright red orange in YCbCr modes on the X9S, which is not youtubes brand color)).

 

I did another profiling session with the X9S, because I don't like to be baffled, or antagonized with "there is nothing wrong here" statements - and this is what came out of it.

- The error you see, when measuring the X9S, connected directly to the TV, in a YCbCr mode, through Spectracals Mobileforge Pattern generator app, entirely vanishes, when measured through video patterns in ZDMC/Kodi in HCFR. But without changing any setting - its there, when meassuring with Calman/Mobileforge.

Now - we know what Mobileforge does. It outputs 0-255 level patterns, assuming, that the Android device will convert them to limited 16-235 - thats how this should work. But it doesnt. The good news is, that the video Levels in ZDMC are fine, so wesk05 was correct in his assesment.

Now, what does this mean? My guess at this point is, that the X9S treats Android Color (App Level) differently from Video color. Android color is PC Level (0-255) range by default - but on a TV box that claims to have a YCbCr mode should be converted to limited (16-235), my best guess currently is, that this isnt the case. And thats also why Mobileforge is outputting color patterns that are displayed incorrectly, with the box in YCbCr mode. That said - Video Level seems to be fine at the same time - so whenever you open Kodi or ZDMC, the resulting image is color accurate.

Thats the first part of the issue, the second part is even more fun...
--

That said, as soon as I attach the X9S to my LUT box - the error above shows up again, and this time even can be measured in Kodi/ZDMC. Now, the LUT box is designed to look at the incoming signal - expand it to 0-255 regardless of wether it was limited or PC Level, then do its calculations, then convert the signal back to the level that was fed in (limited or PC level). It works correctly with every other AV equipment I own.

With the X9S it doesnt. My best guess at this point is, that the X9S still outputs a PC level signal, with video colors (output of Kodi/ZDMC) shifted to limited, and then probably does relabel the signal, or delete the label entirely. This probably works on most TVs, as they don't care about the actual signal, but about the label - but the 3D LUT box actually analyses the signal - and something is very unorthodox about the output/handshake at that point.

I don't know how "far spread" this issue is, but my guess at this point is - that most TVs will display the signal correctly.
--

The third part of this is, that if you are ina YCbCr mode, and are using the HDMI IN port - with a device that outputs a limited (16-235) signal - the video Levels are all wrong again, because black is shifted up 16 levels and your TV can and will not compensate for that. So whatever you are feeding into the HDMI IN port, has to be PC level to be displayed accuratly - regardless of what video level the X9S is set to.
--

Yes, this is all a mess, but the good news is, that if you only care to get accurate colors out of Kodi/ZDMC (and I tested with Kodi, so Kodi as a third party app is fine as well), you should be good, as long as your TV doesnt analyse the signal it gets fed too closely.

And my educated guess is, that most don't.

If you care about color accuracy in other apps I cant definitely say at this point, what happens. It all could be a Mobileforge issue (the not bit accurate part of the argument is much, much lower in error margin than the issue at hand here, so thats not an explanation, I double checked) only for all I know at this point - I could test it tomorrow with a different Android pattern generator to make sure.

My statements about Youtube red looking orange were made while assessing the X9S going through my LUT box, with no LUT applied - so I have to double check this.
-

In short - both Calman (with Mobileread as a pattern generator and the box directly connected to the TV), and the X9S going through a 3D LUT box showed similar errors, by amount and direction - that don't show up, if you connect the X9S directly to a TV (at least to mine), while measuring the Colors in Kodi/ZDMC - even while the box is set to a YCbCr mode.

In RGB444 mode the X9S, Calman, Mobileforge and the 3D LUT box agreed on how the signal should be interpreted, and a 3D LUT created from that setup showed little errors, when setting up an entirely different device and applying that LUT. (Kodi Video Levels not checked independently in this "configuration", with video patterns and HCFR, I'll do that eventually to complete the tests.) But that mode didn't survive a reboot, without you having to set it again (although the menu indicates that, ...)

Those Colors measured in HCFR, were accurate enough to pass the within normal measurement errors margin test.

Hard problem to analyse - but this is what can be said about the X9S at this point.

 

I rechecked this with firmware 1.3.0 and again there is nothing wrong with YCbCr output for video playback. I have underlined video because it looks like graphics seem to be treated differently. I did also look into YouTube icon color and it indeed doesn't match the specifications (https://www.youtube.com/yt/brand/color.html) in YCbCr mode. This is not a problem isolated to Zidoo X9S. Of the couple of devices that I checked only nVIDIA Shield has correct RGB & YCbCr values that match the specifications. So, it looks like there is some funky thing going on with desktop compositing.


The ones highlighted in green color match specification.

What settings do you use for "Expand Video to PC Levels" and "Luminance" levels in Calman/MobileForge for YCbCr and RGB modes on the X9S. There is only one combination that is correct for YCbCr, 3 combinations are correct for RGB.

Zidoo X9S RGB output is an expanded/remapped output. How do you calibrate your 3D LUT box?