In studying for the Quantitative Methods test offered by World at Work, a module is dedicated to spotting when data is presented in a misleading way. The five things it suggests looking for are:
- railroad tracks that distort intervals
- graphs showing nominal data, that should be listed alphabetically
- graphs that show data with no absolute zero (such as temperature scale)
- graphs where part of the scale is lopped off, giving a false impression of trends (you can’t see the y-axis zero point.)
A great example of this is the program Lumosity, which uses games to improve your brain in various skills such as remembering names to faces, calculating math in your head, observation, and the word at the tip of your tongue situation. I had seen a commercial for it and being one always looking to improve, decided to try the free version out.
The free version limits both the types of games that are available (maybe 9 out of dozens) and the number of games you perform daily (3 vs. 5.) I don’t recall there being any evaluation to provide a base point in the various skills that was not game related.
I participated on a nearly daily basis (missing 3 days sporadically) in the month of January. After a month I had three types of measurements provided to me through the (free version of the) program – a point value for playing the games (you get points regardless of how you perform), a series of horizontal bar charts that indicated a score in various skills, and a graph plotting the “brain performance index.”
The site provides some white-washed “science behind” it all, that throwing names of doctors at universities did not assuage my desire to understand – instead it made me look at it more askance for not providing the real science behind it. Building neurons is great, but there was no actual proof that the games built said neurons. Sure there are images of brain scans and a bar chart, but they aren’t labeled in a manner that inspired validation.
The three methods of measuring how one improved in using the service felt like examples of bad data to me. The first, where one got points for playing and showing up is reminiscent of an elementary school field day where everyone got a ribbon – even if they were in 15th place (out of 15 kids.)
The breakdown of cognitive behavior bar chart did not show which game had changed which score and by how much. You had to keep your own excel document to chart after each game which score changed and how – which defeats the purpose. I think instead this chart is just to make one feel as if they are making progress in the areas they specified one wanted to work upon when signing up for the program.
The BPI chart showed change in the total score over time – for the past 4 weeks only. It is possible that the paid version you could see the full history and zoom in and out to see trends. The fact that there was no real explanation as to what this score was (as the total was not the total of the bar charts) or how it was derived did not sit well with me.
Finally, there were no points to evaluate in a separate format performance improvement – it only had the games. This makes me feel as if there is a huge bias, in addition to all the hand waving and “ignore the man behind the curtain” that was going on. I was feeling more energized and productive while using the program, but in the two weeks since stopping I still feel the same so I believe it was due to other factors which I began at the same time (like studying for the W@W T3 test and refreshing my French skills.)
It’s pretty clear that I’ve stopped using Lumosity as I feel it’s just another time waster that is trying to milk one for money. I have better ways to spend my time and money, with better ROI. I do appreciate the refresher and skills the T3: Quantitative Methods module on identifying false representation of data to help me feel better about making the decision to stop using Lumosity. The first thing learned in T3 is – use data to make sound decisions. 😉
Morale of the story – use your brain and look closely. Data and representing data can be manipulated – beware of the snake oil!