Hi guys,

What did he do this time?
I made a pretty thorough analysis of input lag in the following sims:
- GPL
- LFS
- N2003
- Nkpro
- RBR
- Rfactor

How?
Using my Canon S3 IS digital camera to record, at the same time, the monitor image and a small joystick on a gamepad, engaged with a 'ghetto' spring loaded device. So I am recording the input (joystick) and the response (monitor) at 60 frames per second. I count the amount of frames it takes for the monitor image to respond to the joystick. I do this five times for each sim (at various framerates) and take the average.

Here is the 'plan' in a picture:


How accurate?
It is not highly academic. The monitor is a TFT running at 60Hz. So even with games running at 200fps, the monitor won't show all 200. Furthermore there will be some delay before the monitor responses. The camera at 60fps will probably not be in perfect sync with the TFT monitor. Still, roughly 1/60th of a second is a good accuracy and the results where consistent and there was enough difference between the sims and settings to give some interesting results!

What settings?
Various frame rate ranges (after tweaking graphics / underclocking PC to get the result within the range):
- Low framerates between 27 and 37
- Good framerates between 54 and 61fps
- Very high framerates between 90 and 140
- Insane framerates of about 230

System:
- AMD64 at 2.5ghz (3800+ abouts) underclocked for some of the low fps settings
- Nvidia GF6800GT 256MB Driver version 84.26, with AA / AF to reach target fps at times
- 1GB ddr at ~225mhz
- Windows XP sp2, full updates
- no 'usb overclocking' (standard 125hz speed. Can't yet be arsed to test this)
- Philips 20'' TFT rated at 30ms for what its worth
- Thrustmaster dual analog gamepad



Also:
- VSYNC on, 60 frames per second
- In LFS and NKpro, setting a max framerate in game (37Nkpro, 30 and 18 LFS)

Results
Compiled in Excel, the absolute delay (in seconds) versus the framerates, not limited by game limits or VSYNC are:


The relative lag, measured in simulation frames is:


How in game framerate limits (cap) and VSYNC affect the absolute input lag:


Conclusions
When not using VSYNC or in game frame rate limits, looking at the absolute lag:
- GPL makes a good show: The least amount of lag at the lowest framerate
- LFS is the best of the rest although it needs more than 100fps to get to GPL levels of response
- Rfactor is affected the most: at 27fps the lag is almost 0.25 seconds!
- RBR and NKpro aren't much better with terrible lag at lower framerates
- RBR and Rfactor need 200fps to be on a par with LFS at 100fps

When looking at how many of simulation frames the lag is:
- RFactor and RBR displays a linear ratio: double the framerate, halfs the input lag
- NKpro is unaffected, the lag is almost constant regardless of FPS
- LFS and N2003 do relatively better when the framerate is low
- GPL makes a great show again: it takes less frames AND it happens at a higher framerate (both have a positive influence on lag) to respond.

When using VSYNC or in game framerate caps:
- Vsync results are very close, between 0.1 and 0.115 seconds, except for GPL which is probably not working well with vsync as 60hz monitors and 36fps might not match..
- Framerate caps as selectable in LFS and NKpro work well
- NKpro 37fps, at 0.08 seconds is almost 2.5 times faster than when system performance limites the framerate at 37, with a whopping 0.19 seconds delay!
- LFS with a 30fps framerate cap comes in at close to 2.5 times less lag than running 'system limited' at 28fps.

Other things that struck me:
I have also looked at the number of video frames it took for the steering wheel to go from center to max lock. This was pretty constant at 3..4 video frames (making that 3.5 gives 3.5/60 = 0.058 seconds) Only N2003 was slightly slower at 4..5 frames.

In N2003 I seem to feel more lag than the numbers show. I do have a possible explanation. With all the other sims, as soon as the steering wheel moved, it showed a fair amount of movement. With N2003, very often, the first video frame showing wheel movement only showed very little movement. This seems to indicate that N2003 does not havea totally linear 'response speed'; as if there is some speed buildup.

What do I think?
Input lag is a bad thing. However, even you don't or barely see it, it is still there. The quicker a sim responds to your inputs, the more direct control you have over the car. In my opinion, there is NO reason that new games shouldn't be able to deliver the same quick response as GPL; a game from 1998!

I don't want to go deeply into a "when is it good enough?" debate. If technically a lag of 0.05 seconds is possible with a sim running at just 36fps, then this is the benchmark. A sim maker has no excuse when his sim has more lag at lower framerates. Anything has to be done to make simracing as 'immediate' as possible and reducing input lag should be very hign on the list.

Meanwhile, VSYNC doesn't help, which is sad as that looks so much nicer on TFT screens. It is obvious that, especially with Rfactor, RBR and NKpro, you are much better off sacrificing detail for speed.
Isn't it dubious that an 8 year old sim basically wins this? I can't call it a victory for RBR / Rfactor as they need to run at 6 times the framerate to beat GPL.. LFS is the best of the rest, especially at lower framerates.

Food for thought, and for the sim Devs out there! (hopefully)
Hope you enjoyed this

Niels

tfts have quite a lot of processing delay on the input ... unless you use a crt you wont get any good results

what you didn't have in mind is the controller support of the controller driver, which has to translate input into the gaming language i think you should at least do that test again with 3 or 4 more controller (also standard wheel)

as i know statistically results are significant when they have at least 30 measurements of the same experimen.

just some more inputs for this. that Vsync was interesting!


but overall it is an interesting experiment! keep up that work!

Quote from Shotglass :

tfts have quite a lot of processing delay on the input ... unless you use a crt you wont get any good results

But who uses a CRT anymore? I am not switching back to a CRT just to get better reactions by the game =P

I agree try with different controllers and see what happens. The delay might come from the controller drivers processing and sending the data back on.

since monitor and input device where constant, this makes the results comparable. They might not be 'absolute' but can be compared relative to eachother.

The differences where very small and the results very consistent. I am quite sure that any trends that came up reflect actual situations. Until I read something that convinces me of the opposite..

I get what your saying =P This is how long it took your wheel to react in the game, other controllers may have different exact numbers but their graphs should reflect those you made, albiet at a different time presumably.

This would also depend on video card settings, at least with nvidia drivers there is a setting to force triple buffering on/off and a 'max frames to render ahead' setting. Both of these should affect the 'lag', and games can request triple buffering too so the driver settings might affect one game but not another... Plus if a game uses OpenGL then the 'max frames to render ahead' setting for example does not do anything.

Quote :

Input lag is a bad thing. However, even you don't or barely see it, it is still there. The quicker a sim responds to your inputs, the more direct control you have over the car. In my opinion, there is NO reason that new games shouldn't be able to deliver the same quick response as GPL; a game from 1998!

Looking at the first graph, it seems that all the lines are going asymptotically to 0.05. Some data points are below 0.05, but within the margin of 1/60th of a second (0.0166 s). It would be interesting to know what the time is for the signal to be processed from the gamepad, to the usb port, through the mainboard, to the game, then to the screen and all that and then to your camera. So maybe the 0.05 s GPL is achiving is already the limit, and new sims can only be 'as good'.

Is the steering wheel drawn on screen really a good measure for lag? Maybe the simulation part has no lag, but the graphics part has some lag (I dont know about the inner workings of sims, but I can imagine there is an indirect coupling of the data for various reasons)?

Also, judging for this data, it might be wise for simulation manufacturers to force the sim to limit fps, so that users unaware of this effect can also benefit from it.

Quote from Viper93 :

But who uses a CRT anymore? I am not switching back to a CRT just to get better reactions by the game =P

Me for starters. I'm hoping to hold out on the TFT "revolution" as long as physically possible.

Quote from Bob Smith :

Me for starters. I'm hoping to hold out on the TFT "revolution" as long as physically possible.

:hippy: Bah ok if you insist =P

Was RBR "patched" with the steering lag fix?

I don't know about a fix like that? An URL is appreciated.

My feeling says that the 'frames rendered ahead' and 'tripple' buffer thing as often mentioned as a cure for lag is something that begs to be tested as well..

Again, this whole thing gave consistent results, it is not about absolute lag but lag comparing various sims on the same hardware running at various fps.

Btw, 0.05 seconds is very very fast, I can't witness it as visible lag I must say LFS with Vsync (so 0.1s) is still 'fast' and definitely not witnessed as a 'considerable lag'. It only clearly visible above that. That is mainly the reason why I did this test; it is hard to 'see' the difference as the more a wheel turns (like 720 in lfs) the faster it seems to turn etc.. This camera test is a fairly objective judgement.

It is pretty hard to do with wheels as you want to test the analog axis (not a button) and you'd need a 'step' response, i.e. considerable turning in an instant. I'd have to disconnect the potentiometer and connect another 'rig' that turns it very quickly.

It could be that some games do a filter for the graphical wheel however I used to have a very jittery gameport wheel which would make all sims except GP4 jitter. If the graphical wheel copies the jitter of the wheel, there is probably no filtering done. I don't know about Nkpro in this regard btw..

Quote from Niels Heusinkveld :

I don't know about a fix like that? An URL is appreciated.

From our beloved nk pro pro spokesman: http://torttu.net/temp/rbr/steering_lag_fix.txt

I hope to test 'render ahead limit' of 0 and 5, as well as tripple buffering on / off tonight, although it went on til 5am last night so I might choose the bed for some good 8 hours of 'lag'.

Quote from Bob Smith :

Me for starters.

+1

I use a 21" ViewSonic CRT monitor. The size and weight don't bother me at all, considering I'm not exactly moving my PC around all that much. It's as crisp, clear and responsive as any LCD/Plasma monitor I've used, and it's a fraction of the price.

Quote from Viper93 :

But who uses a CRT anymore? I am not switching back to a CRT just to get better reactions by the game =P

i do and i most certainly wont switch until sed or oled hits the consumer market ... and even then ill probably reluctantly give up my lovely 0 processing dealys analog inputs

Quote from Niels Heusinkveld :

since monitor and input device where constant, this makes the results comparable. They might not be 'absolute' but can be compared relative to eachother.

The differences where very small and the results very consistent. I am quite sure that any trends that came up reflect actual situations. Until I read something that convinces me of the opposite..

found the link ... as you can see the delay isnt constant and well within the range of the lowest values you measured
http://www.behardware.com/arti ... compared-to-crts-yes.html

Quote from Kegetys :

This would also depend on video card settings, at least with nvidia drivers there is a setting to force triple buffering on/off and a 'max frames to render ahead' setting. Both of these should affect the 'lag', and games can request triple buffering too so the driver settings might affect one game but not another... Plus if a game uses OpenGL then the 'max frames to render ahead' setting for example does not do anything.

the fact that the lag by frames seems to be constantly at 2-3 (one of which is the default setting for max frames to render ahead suggests that this might be a cause for some of that lag

Quote from Hyperactive :

From our beloved nk pro pro spokesman

I'm curious too that is the graphical wheel really the best possible way to measure lag... e.g. in RBR after tweaking and fixing the lag actual response is much better but the visual wheel is still clearly laggy.

Offtopic: Something in-depth experiments like this would be nice to read from ASS magazine instead of those hype yadda yadda hype blah blah hype articles.

Quote from Viper93 :

But who uses a CRT anymore? I am not switching back to a CRT just to get better reactions by the game =P

I have two of them, 4 and 10 years old respectively, one is looking like it's about to die atm but I think 10 years of use ain't bad So I'm looking for a cheap replacement. Sure TFTs can be good but TBH they still don't make a viable budget option new or used.

Good nose and bad nose..

Changing the pre render limit down to 0 does have a pretty positive effect on Rfactor. Tripple buffering showed a 0.01 second improvement at high framerate and a 0.01 second loss at low framerate, i.e. that is clearly no difference at all..

The bad news is I have to re-test everything now with max prerender set to 0

I already had that RBR 'tweak' done btw, except removing that constant force multiplier line but I think that is of no use when not using FF anyway.

It will be interesting to see the effect of this: Will it make all sims 'quicker' or only the slow ones? Stay tuned... :o

and dont forget to use a crt this time

there is a pretty significant issue that no one has touched on...you are doing all of these tests on a decent modern computer, decent video card, and a decent amount of ram, all by today's standards.

BUT...in 1998 the top of the line computers topped out at ~500mhz, the video cards to have were the TNT or the voodoo2 (tnt2 and voodoo3 were release in 99), and 256 was an ungodly amount of ram that only people with more money than sense had.

I would certainly hope that a computer 7 years newer can achieve the results that it does in GPL, and "struggle" through the newer games. they have more polygons, larger textures, perform more physics calculations, and do video effects and resolutions barely dreamed about in 1998.

GPL run on a 1998 equivalent of your computer wouldn't fair nearly as well as it does in your tests. hell, i'd wager (not a lot) that it would actually perform worse than the "modern" sims did on your computer.

One other thing I'd like to see is that is there difference between different wheels when you look at steering lag.Are Logitech wheels better or worse than Microsoft wheels etc.. This is probably something you are not able to test but I'd be interested to see this. Especially if there was a way to see how the hi-tech wheels (ECCI, BRD, VPP) compare

That is an interesting question. One way you could say that when GPL runs 'maxed out' (i.e. 36fps with a bit of headroom) that I can't imagine that things like USB joystick 'transfers' are slower on a p3 450mhz. But that might be a wrong assumption..

I could use the TVout Shotglass but I doubt that such a 50hz monster is a good idea? Plus its not the absolute lag that I'm after, its more the relative lag between sims, and really I am quite sure the measurments are 'acceptable'. There is a consistent difference measurable between 0 render ahead, 2, 4 and 8.. Small differences show consistently.

There is another thing that puzzles me though. As mentioned and probably possible (I don't understand the inner workings of sims, physics / graphics loops etc ) is what's been said about the lag just being a visual thing. What I would think is the whole sim is a 'loop'.. sure the physics are done at a higher rate than the graphics but surely the lag measured as I did means you can't SEE what happens and the RESULT of the inputs some 0.xx seconds later. Then illepall that is exactly the same as delayed inputs. The only condition I can think off is indeed that the game doesn't use smoothing filters on the graphical wheel which could make it laggy. If there is no smoothing there, any visible delay IS "sim lag" (if not inputlag) that simply puts some time between what you do and what you see. So imo, whatever causes this and whereever it goes 'wrong', I don't see the difference. Lag = lag surely? I mean even if the physics react to the input after 0.000001 seconds, if I don't see it after 0.15, what's the difference?

Edit: Hyperactive, I do have a high-end USB controller chip, probably much more advanced than those used in the expensive wheels. My gut feeling is that there is likely to be not much of a difference and the main lag comes after the controller so to speak. It is dangerous to make assumptions though like I do now though

Quote from Niels Heusinkveld :

I could use the TVout Shotglass but I doubt that such a 50hz monster is a good idea? Plus its not the absolute lag that I'm after, its more the relative lag between sims, and really I am quite sure the measurments are 'acceptable'. There is a consistent difference measurable between 0 render ahead, 2, 4 and 8.. Small differences show consistently.

sure i dont doubt that theyre consistent and meaningful results but id really like to see some absolute results
but my point is that your measurements pretty much stop in the 50ms range which according to the link i posted earlier ( http://www.behardware.com/arti ... compared-to-crts-yes.html ) is about how much a lcd lags behind
iirc you said you had an 30ms lcd which suggests thats its a very old monitor with some significant processing delay
surely a friend of yours must still have a crt in his basement that you could borrow for a day or something like that

Xbit labs has a good article on it now too. Jumping to page seven:

http://www.xbitlabs.com/articl ... lay/lcd-parameters_7.html

you see that a newish 8ms rated samsung has a 47ms delay. With Rfactor at 230fps and 0 frames 'ahead' rendered, I get consistent results of 40ms lag. This is of course for the input, joystick processing, USB, game, then out to monitor again.

So one thing seems for sure, TFT lag is a factor, but new TFT's aren't nescessarily better than old ones in this respect.

Edit: and of course, any TFT running at 60hz will *start* the frame change after 1/60th second, disregarding 'other delays'. The pixel transitions will just be slower. Considering that many 'fast' TFT screens only reach their rated speed in certain situations like:

http://www.xbitlabs.com/images/other/19inch-4/10gr2.png

for a samsung 940B rated at 8ms.

Now if anything this means, unless my Philips is specced honestly (it is too old to find measurments of) that my 30ms screen is probably a lot slower than 30ms, yet I do find repeatable results of between 2 and 3 camera frames with rfactor at 230fps, 0 render ahead.

Edit2: Its a good article imo on Xbit:

Quote :

I’ve also read forum posts saying that it was just impossible to play games on a new monitor due to the input lag, but this had a very simple explanation. The user had changed the resolution from 1280x1024 to 1680x1050 but hadn’t thought that his old graphics card wouldn’t be as fast in the higher resolution. So, be careful when reading Web forums! You can’t know the level of technical competence of people who are posting on them and you can’t say beforehand if things that seem obvious to you are as obvious to them.

The input lag problem is also aggravated by two things common of all people. First, many people are inclined to search for complex explanations of simple things. They prefer to think that a light dot in the sky is a “flying saucer” rather than an ordinary weather balloon or that the strange shadows in the NASA photographs of the Moon are proof that men have never landed there rather than are indicative of the unevenness of the moonscape. Any person who’s ever taken an interest in the activities of UFO researchers and other folks of that kind will tell you that most of their alleged discoveries are the result of thinking out excessively complex theories instead just looking for simple, earthly explanations of phenomena.

Thankfully the 'techy' threads on the LFS forum about physics etc are proof that there is a lot of sanity as well but sometimes when I'm looking at overclock / pc hardware forums I don't know if I should laugh or cry..