Measuring your prior probability accuracy
When performing a subjective Bayesian analysis, one aspect of the analysis is to build a prior based on subject matter knowledge from an expert. This process is called prior elicitation. It is common that experts (and others) will underestimate their own uncertainty. From the Pharyngula blog, I recently found a website measuring what they call `risk intelligence' or RQ. The idea behind the website is to measuring how accurately you estimate your own uncertainty in your knowledge about facts in this world. This type of assessment might be valuable in adjusting the self-estimated prior uncertainty to a more realistic estimate of uncertainty. Since I will tell you how the test works below, I would suggest you try out the quiz for yourself by going here before I give away any secrets.
The test asks you to give your probability of 50 statements being true where you can choose from the probabilities 0%-100% in steps of 10%. It then lumps all the questions you gave X% to and calculates the percentage of statements in that group that were true. It compares X% to the percentage that are true. Do this for all percentages and then add up the absolute (?) differences and you have a measure of how accurately people estimate their own uncertainty. So I took the quiz and you can find the results by going here and using the drop down box to choose the test results from 2010-02-05. I scored a 60, which the test says is average. Some of the interesting results are that I choose 0% three times and 1 of the 3 times the statement was actually true. Of the times I chose 90% only half of the answers were true. In fact, of the times I choose 60,70,80, and 90% only half of all these answers were true. At least I got all the 100% answers correct.
Just to get an understanding of how the test works, I retook the quiz and answered 50% to all questions. The resulting RQ score was 52. I'm guessing you can get a zero only if you answer 0% and 100% for all answers and in exactly the incorrect way. I'd be curious if you can get an RQ of 100, if you answer all questions 0% or 100% correctly. I don't know what the quiz would do with the percentages 10%-90% since you never actually answered those to any questions.
blog comments powered by Disqus
blog comments powered by Disqus