| [ Main index » Criticism about tests and reviews » Interpretation, objectivity, and comparison with other reviews | ] |
In 2007 I became annoyed with the poor reviews of equipment in Dutch and German cycling magazines, and also on the net about various cycling equipment, which was the reason I made the web pages with my tests of hand pumps and of dynamo lighting in 2008. I had already placed an early version of my saddle experiences page on another section of my website long before that. These page give an example (to e.g. magazine writers) on how matters should be dealt with, in a scientific way as far as possible. That doesn't necessarily require expensive equipment; even with limited resources one can do things a lot better than in magazines! The first versions of the pages with pump & lighting tests were placed on the forum of the dutch magzine 'Fiets' but neither the readers there nor the people from the magazine showed any interest. I had at least expected the readers to give some feedback on pump diameters, but no... This just shows what you can expect from the people on that forum but in particular from that magazine in future articles... (that's why I won't consider even buying loose issues). How it should be done can be seen for example on http://www.mtb-news.de/forum/ where the Germans are doing a good job of getting to the facts. Well, some of them are. There are a large number of non-experts and bullshit-artists there too.
German magazines are usually among the best, thorough, critical etc., in most areas and this is mostly true for cycling magazines too, but being the best is not good enough in this case (*). I wanted a good handpump for example and couldn't find proper information. For lamps, web sources are much better than magazines, but there you get incomplete information and you're missing important stuff. I will give an example of lighting: Somewhere in the late 90s (if I remember correctly), I was thinking about getting a Union roller dynamo. The German magazine Tour had an article about DIY lighting (using a water bottle as storage for a battery etc.), and roller dynamos, the Union in particular, got mentioned and was said to be good. Not just that, they said some people used it so much that the roller wheel was worn out. In the end I never bought one, which was a good thing, because as you can see from the problems I had with it, these dynamos are in fact rubbish. It's also apparent that those people at Tour never used one in any other conditions than good weather (or if they did then only very briefly), and those people who had worn them out also not, or perhaps the roller wheels were worn out because of use in winter! (but nothing of that nature was discussed in that article as far as I can remember). So, Tour certainly didn't do a proper in-depth article on bicycle lighting. This shows you it was not going downhill recently, it was like this for a long time. The internet has made various types of magazines more or less superfluous, and this goes for cycling magazines too. They will really need to change or they will become useless. And considering the poor tests they may not be missed by many people...
Addition 2008-11-18: (*) Perhaps there is hope yet, the magazine 'Aktiv Radfahren' looks good from various bits I've seen so far over the past 6 months or so, even more promising is the recent dynamo lamps test. 'Tour' keeps on disappointing me...
Addition 2010-7-15: For more about the problem of poor reviews, but caused by corruption, have a look at my analysis and comments on a message posted on bikeradar.com (that message is not the only indication that corruption occurs with magazines): Comments about reviews on bikeradar.
Addition 2010-12-31: Here's an example of how badly the tests in Tour are done: Tour 2011-1: Bicycle lighting test
Summary:
The rest of this page is a long story with examples of problems I had already encountered since ca. 1996-1997 in discussions on newsgroups, where people do not respond properly, either logically or by being crude and making totally nonsensical statements. I made a page with explanation of argumentation problems that you will encounter too if you engage in discussions, which is fairly short, but still has examples.
What I write in my tests/reviews is objective in the sense that from principles there is good reason to say something is bad or good, e.g. the user interface of especially Sigma bike computers and the beam shape of in particular the Cyo RT headlamp is bad, not because I 'feel' this, no they are bad and there are lots of objective reasons for that. That some people like these computers or headlamps that I criticize is no reason to say that my arguments nor conclusions are biased. On the contrary. The whole point of my tests is that almost no one properly tests and even less so explains.
So when someone on the forum of the Dutch magazine 'Fiets' posted a message not long after I posted about the problems I had with the vibration of the SON28, in which he said "this hub works fine without vibrations and that the person who complained about it probably just spun the wheel by hand" I got annoyed because in ALL my posting everywhere I give lots of details and arguments. This inane message was by a guy who cannot even understand that the details I gave implies that I am not someone who just "spins a wheel by hand", and in such cases I am not going to be friendly in my response. Such people are bullshit artists, which is a term describing people who barely know anything about a given subject but declare their limited experience as 'this is how it is'. They only pollute forums with their disinformation. Besides my reviews, forums are generally the best source of information (and not magazines!), but there you have the problem of deciding whose information is valid. You can only recognize bullshit artists once you know a lot about the subject matter...
Forums where bullshit artists are not so much a problem are most German language forums, candlepowerforums.com and the mtbr forums (They are kept mostly in check there by people who do know what they are talking about). Elsewhere it's really pathetic, also because of fanboys and people who feel the need to defend their purchase of an expensive product (esp. in case of the SON28).
With bicycle saddles it's more complicated in some sense than with bicycle lighting as each saddle's suitability depends on the sit bone width of the rider and the riding position. This means, as I state from the start, that this is a limitation in my tests. I don't know a bunch of guys similarly critical to myself (which I mean in the sense of observant about all details, not in the sense of being negative) but with different sit bone widths, who can try out all the saddles as I do, and report on them. So, my reviews on saddles are subjective only in the sense that the saddles are tested by someone (me) who has a specific sit bone width.
If you compare my reviews with those in magazines, then you have to keep in mind that magazines almost always do a very poor job of reviewing. This isn't my opinion, this is a fact. Further on I comment on some postings on a forum where some people say my conclusions are subjective because they differ in some cases from other reviews. That is so absurd, it's like reading a piece by Kafka.
See my review of the review of bicycle lighting in Tour as an example of how badly most reviews are done. This has always been my experience of magazine reviews in various stints that I read them in the last 15 years...
Well, except for the lighting reviews in Aktiv Radfahren (at least from 2010 and earlier, the latest test from end of 2011 is questionable), but then, Olaf Schultz takes part in those tests/reviews (see his website http://www.enhydralutris.de/Fahrrad/Beleuchtung/Beleuchtung.html, in German; if you read German you will understand why, as this means the tests have a good basis...). He restricts himself mostly to measurements, but makes some notes as do I, for example that Supernova lamps don't get anywhere near to what the manufacturer promises (I explain a bit more why this is so), and about reflectors he notes for example that some design decisions are a 'more is better' choice which is not good (see his section on the Philips SLD/Saferide 60). This is no different from me but I include more factors, such as light colour: No, bluish is not good, purplish is unacceptable. Reasons are given on these webpages (see for example Annoyances caused by lamps). Etc., etc. etc..
On a US website (bikeforums.net) there a bunch of people who don't know what they're talking about, but act as if they do, which resulted in this section. Lack of knowledge in itself is no problem, but if such people make it look like I don't know what I'm talking about or I am nitpicking, then I will harshly criticize them.
Here are some examples of silly behaviour on that website:
And yet, such people think (some say, others imply) I don't know what I'm talking about or that I'm a nitpicker... Well, those people who don't properly read and who are obviously not critical will think that. If it weren't so sad, I could be laughing about it. From the responses to my posts it's clear there's no point in wasting my time there any further.
Ah yes, and another few examples here, of SON28 fanclub members on google groups: http://groups.google.com/group/rbw-owners-bunch. Two noteworthy posts are:
Despite these 2 guys having read my criticism of invalid arguments, they do exactly the same. And in doing so they confirm exactly the problem on forums/newsgroups etc.: Bullshit artists.
Update 2013-1-3: Kelly Sleeper emailed me that I criticized him on an 'obscure website' rather than on the groups. Uhm, yes, that is this website, the website just about everyone in the bicycle lighting business reads... And I did post via google on that google group but perhaps something went wrong with the posting? It says 'All new members are moderated'. Perhaps the moderators didn't let my postings through. I didn't check as I didn't read that group any more as it seemed pointless due to various factors I explain on this page. In any event, I don't care what happened there with my postings and it doesn't matter: he criticizes me for supposedly only writing my criticism here, but he did the same as what he criticises me for. He criticised me/my website on some group (a truly obscure source that I only happened to stumble on) and he did not email me about that. So what the hell is he criticising me for? Even if I hadn't posted on google groups then it would have been no different than what he did. He says I'm a coward for evading discussion posting on my website only. LOL. If anything will tell you this guy is a BS artist, this is it. It should be completely obvious to anyone reading my site that I do not evade any discussions with anyone (quite the contrary), but I also have no intention of wasting too much of my time (see the rest of this page and the others on the problems you can have on some forums: if there is for example the problem of authoritarian BS-artists then I won't visit that forum any more as then it's pointless to discuss antyhing). I have criticized various people and companies directly, in forums where they post, or via email. I do not evade anything, and that I do so is another ludicrous interpretation he makes. He also claimed what he says was 'opinion based and accurate'. Wrong! False statements are not opinion, they are just wrong. The view he still has that I'm smug is also wrong, as should be clear when reading my website and this page. Then he says I use a lot of space to attack him personally without ever addressing the actual opinion. Yeah right. I quoted 3 of his statements which I criticised. I also mentioned that this type of posting by him is an example of how people give disinformation, and thus that he is a bullshit artist, yes, but I didn't 'use a lot of space to attack him personally'. So this guy, in his email does exactly the same thing as he did in that posting on google groups, writing nonsense. Note: I am willing to discuss anything (contrary to what mr. Sleeper claims) but am not interested in such inane responses. In his email he again displays no reading comprehension e.g by 'missing' the fact that I countered 3 of his arguments, and this made me think of a comment I heard when I saw a TV programme while visiting someone: It is the TV programme 'Storage wars' and one of the guys said about how to respond to someone else's bidding: "There is no strategy against crazy".
I've experienced the same things in Linux newsgroups in the 1990s. Such people are often, similar to many SON28 owners, like religious fanatics. There is no point arguing with them, and they in fact never give any real arguments, just say that people who disagree are wrong, don't know what they are talking about etc. The reverse is of course the case in both situations.
In all these cases, I have nothing against people disagreeing with me, if they give arguments that are at least reasonably well thought out, even if these arguments are wrong. But when the arguments are silly, or misrepresent what I write on on my site, or are denigrating, then I will not respond in a friendly manner. There is no point to being friendly then, as I've seen plenty of times in the past. My only concern then is to set the record straight, and if someone feels his feelings have been hurt, well, that's just too bad, and their own fault.
Btw., I talked with someone from a trade magazine not long ago and she said there was a discussion to make a forum open to all, and there was strong opposition to it, exactly because of such issues...
And btw2, someone wiser than most people told me about this problem: "The problem as it always is, is that you should never argue with a fool as observers may not know which is the fool". LOL. He is right, but I try it sometimes, as I dislike that some people give disinformation. Not necessarily about me, but about the subjects at hand in general.
Actually, I wrote something comparable about similar issues some time ago to someone else, though I wrote this in quite a different context, namely:
So, a developer for FreeBSD used an interesting line as a tag, which became more clear as I got older. It's relevant to this situation but others too (such as nagging), and makes fun of it:
'Never try to teach a pig to sing, it just makes you look stupid and irritates the pig.'
This tells you that you should consider whether there is a point in doing something, or of continuing to do something, such as to try to persuade people about your point of view, or try to get them to do something. At some point you should realise that if it didn't work after trying for a while, it will never work...
And btw3, just FYI: I don't write this commentary because I feel offended, nor do I comment about this on those forums for that reason (in case I think there might be some point), no, I do this to set the record straight, to give proper information first of all and not let others get mislead. I was annoyed enough by bad reviews in the past that did not give me good guidance on buying something, that as I said earlier on this page, I started these reviews. There was no commercial motive involved for making the reviews, I am simply interested in improving matters, this is also why I make suggestions to manufacturers. I try to set the record straight in a few other fields as well, to give counterbalance to actually-biased people. E.g. about cameras on the dpreview forums. In the case of cameras: I have made no reviews and the various review sites such as dpreview.com, imaging-resource.com, dcresource.com, photographyblog.com do a fairly good job. Each has some weak points but this is incomparable to the huge weaknesses in reviews of bicycle equipment in magazines and most websites. But there is a lot of mistrust on lots of forums about reviews, esp. if they do not praise a certain camera highly enough. Reading all that on forums gives a lot of insight into how people think, but also in how people distort the facts... Setting the record straight is also why further on I have a section on 'suspect material', which I should expand a bit perhaps as it's not really enough, but to be honest, it is really difficult to give simple criteria, as you almost need to be an expert to judge whether someone else knows what he's talking about...
The Cyo RT headlamp is a very poor lamp: very dim, and has a very bad beam shape (for psychological reasons, as I explain on my review page). The beam shape made me irritated and I have seen references to other people not liking it for similar reasons which are related to this bad beam shape: The very strict trapezoidal shape gives a feeling of driving in a tunnel, which is in some sense claustrophobic (hard to explain properly, but try any other lamp and you should be able to feel this too).
If anyone doesn't think it's a bad lamp, fine, but that doesn't change the fact that it is a bad lamp for the objective reasons given. Such people probably didn't use better headlamps or they are not critical. Note also the example I give on the review page of the E3 pro StVZO about the Philips SLD which has too much light close by: This is like an annnoying background noise. You may not consciously notice it but it can annoy subsconsciously. And I notice all these sorts of issues with all lamps. That others don't notice or don't care doesn't mean I am subjective, but that they are doing a poor job reviewing.
I don't like the SON28 hub dynamo because of the vibrations. Whatever mechanical superiority it has over other dynamos is just about moot. I suppose I haven't explained this consideration well enough, but I thought it was obvious. This is because all dynamo hubs are pretty good in quality, Shimano's and Sanyo's top dynamos are almost as good as Schmidt's dynamos, they last for many years in bad conditions. The slightly smaller drag compared to the Sanyo or Shimano dynamo hubs is just about irrelevant for me (see the calculations on my bicycle lighting test page to see how little difference a few Watt changes your speed) and the price difference makes it an uninteresting proposition to buy one for the longer warranty. What the SON28 has that they don't, is the pressure equalizing system, which btw. can fail in various circumstances. Do you need that pressure equalisation system? If not, then what remains to decide between dynamo hubs is how it looks and how nicely it works. And then we get to the subject of vibrations, which in the SON 28 are the worst of any dynamo hub I tested. This results in a hub that is mechanically the best, and it was electrically the best until SP introduced its series 8 dynamo hubs (I'm assuming here that the SP efficiency values are correct), but as a product it definitely isn't the best. I give some more comments about these issues and other reasons why some people recommend the SON28 (which are: outdated information and the need to defend their own purchase of an expensive product) on the page about vibrations with dynamo hubs.
If you want to counter arguments I give on my website, then please don't refer to 'reviews' with obviously bogus values. For example I've seen an article about bicycle lighting that includes a table that says you can cycle at 50 km/h with 500 W and 30 km/h with 130 W. That's ludicrous for considerations about bicycle lighting as this only works on a time trial bike with an aero suit while sitting in a very deep time trial position. These values are far too low for calculations for normal bicycles with wider tyres, fenders, sitting up more upright etc. This means any views, or data, in such articles are very suspect. You can apply this reasoning everywhere. Someone who gives several obviously bogus values doesn't know enough or doesn't do any proper research/calculations.
I have no brand loyalty, and brands I like or dislike are treated the same with respect to a review of those lamps. I do add my notes of what I think of how firms act/respond and for example I say about the E3-pro-StVZO headlamp that the conclusion for me is different from my review conclusion, because I don't buy from companies such as Supernova who behave in the manner as they do. I also have criticism about Philips in how they respond to email etc., but this is a problem you will encounter with all large companies: The larger a company, the less likely you are to get a (meaningful) response. This is true in all fields, not just bicycles... Trelock is even worse, they never replied to emails I sent with questions about their products.
I also don't care which country a firm is from. Philips may be Dutch, but the Philips bike lamps are German designed (by the German automotive section of Philips) and made in China (as almost everything in the world these days ;-)) so if anything they are mostly German, then Chinese, then Dutch. I don't favour a Dutch brand, in fact, I think most Dutch brands that make bicycle lighting are producing mostly rubbish (headlamps in particular) that I don't even consider reviewing. Well actually, I told some Dutch makers of lamps that I think their taillamps are not just bad, but antisocial because they shine their light as in a point source which annoys following traffic a lot and makes estimating distance impossible. These people should know better after decades of being in this business, therefore they deserve harsh criticism.
The criticisms about my reviews are not just from end-users for various reasons as explained on this page and on the overview page about discussions on forums, but there is also puzzlement by manufacturers as to why I dislike certain headlamps/taillamps (even though I name the reasons on my website everywhere, but I guess this is caused by selective reading...). This is not surprising in some sense, as they need to do proper testing (which most of them do NOT seem to do), otherwise they would know why I criticize a product. My review criteria take a lot of factors into account, far more than any other review I have seen, but all are necessary:
For more on the interpretation problems and factors involved in determining the good and bad points of a headlamp and taillamp, see these pages:
-> Camera: Interpretation of beamshots (This is important because beamshots do not show what human vision sees in reality).
-> Headlamps: Lots of issues (e.g. why lux ratings are important for headlamps with cutoff).
-> Taillamps: Lots of issues (what you need, want, don't want etc.).
From the above it should be clear that I am not interested in infantile comments such as what many SON28 owners like to spout on forums, who say I am biased, but don't give any proper arguments.
But if you disagree with something I wrote in a review, or have another viewpoint, and you are serious about discussing it, then I am certainly interested. Send me an email and with viewpoint/arguments and I'll think about it...
| To email me go to the email page |
Last modified: Tue Feb 19 20:33:11 CET 2013