Powered by glolg
Display Preferences Most Recent Entries Chatterbox Blog Links Site Statistics Category Tags About Me, Myself and Gilbert XML RSS Feed
Sunday, May 01, 2016 - 21:20 SGT
Posted By: Gilbert

Stat, Stat!


(Source: waterstones.com)


Revisiting this ancient art - it's book review time! Today's offering is Tetlock & Gardner's Superforecasting: The Art & Science of Prediction (yes, colon game again), with, as you can see, a cover blurb by Kahneman. 570 Baht (about S$22) from an airport bookstore, but eh, it was a holiday.

Before continuing, a warning of sorts: once one gets through enough (pop-sci) titles in the same category, one has to realise that much of the content is (unavoidably) recycled. Frankly, does any competent Big Tech interviewee not know of Fermi estimation and piano tuners, by now (Chapter Five)? Performance regression to the mean (Chapter Four)? The "Guess Two-Thirds The Average" contest and reasoning (Chapter Three)? The fire chief's intuition, and the limits of the same (Chapter Two; okay, that was from Kahneman, at a minimum)? And if I see the two arrows [Müller-Lyer] illusion (Chapter Eleven) one more time...

Leaving this aside, I daresay the book does still provide a novel angle on the subject of accurate prediction, and on this we might examine the first author, Philip Tetlock, and in particular his 2005 work Expert Political Judgment, which seems to share quite a bit of overlap with Superforecasting. The core of this story, however, got its start as recently as 2011, with the IARPA-sponsored Good Judgment Project.

You'd be right if you guessed that IARPA is analogous to the probably more-famous (and amply-funded) DARPA, if with purview over intelligence instead of defence in general. Now, American intelligence agencies have been involved in some seriously f**ked-up shit, but this particular experiment was probably as benign as could be expected. Basically, its objective was to investigate how best to produce accurate forecasts of future events - or, the wet dream of every insecure ruler who had patronized entrail-poking mumblers or smoke-huffing priestesses, since antiquity.

Fine, fine, so we've come some way since those times, though not a moment too soon; the book reminds us that barely two centuries ago, Washington died after being bled, standard best practice for the time. Actual randomized controlled trials only became common in the twentieth century, with the medical profession in particular curiously amenable to the celebrated second-century Greek physician Galen's attitude that "...all who drink of [my] treatment recover in a short time, except those whom it does not help, who all die. It is obvious, therefore, that it fails only in incurable cases."

[N.B. Which takes one back to the topic of "traditional medicines" - to clarify, the objection was never on the sophistication of the proferred cures, but on empirical proof of validity. Definitely, some may prefer the old ways, if only due to cost, but it would still be nice for a line to be drawn between helpful herbal concoctions, and tiger's whips...]


This doesn't make any sense at all?
Well, now *you* know how it feels!
(Source: gfycat.com)


Returning to the Good Judgment Project. The organizers were mainly interested in tackling Big Global Forecasts that were in the "Goldilocks zone" - not so obvious as to be pointless (e.g. will the United States actively seek to liberate a dictatorship with no known oil reserves, or influence on such reserves?), and not so ungraspable as to be purely be down to fortune (e.g. what will be the first number drawn in the next national Powerball lottery?); some provided examples were: "Will the president of Tunisia flee to a cushy exile in the next month? Will an outbreak of H5N1 in China kill more than ten in the next six months? Will the euro fall below US$1.20 in the next year?"

Clearly, one imagines that experts could provide significantly impressive projections on questions like this. Maybe an experienced diplomat to Africa for the first, a bird flu researcher who had done a rotation in a Beijing lab for the second, a finance professor specializing in international flows for the third? If nothing, the intelligence community had cachet and resources, and with it, connections. Renting the minds of such talents was not a problem, and they could even be provided proprietary classified data if need be. Surely, they would come up with dazzling oracular insights?

Well, this was actually covered in Tetlock's earlier work on Expert Political Judgment, which has been distilled into a single punchline: as with the community of stock-picking experts in general, the average political expert proved roughly as prescient as a dart-throwing chimpanzee (which might make this blog's reliance on hamsters more palatable)

If you're wondering how these experts kept their comfy jobs, Tetlock explains that the study was done under condition of anonymity, since the experts as a whole were at least smart enough to recognize that there was no payoff to potentially being exposed. Not only that, it turns out that not only was nobody keeping score, few even bothered to specify what "score" meant. For example, important-sounding analysts tended to deliver a long, winding dissertation, and then end with a Magic 8-ballesque pronouncement of "probably". But what was "probably"? 90%? 75%? 60%?

As such, Tetlock noted, popular experts tended to be "hedgehogs" who are guided by One Big Idea, since they naturally project confidence in explaining everything according to their unshakeable master worldview. "Foxes", then, who pragmatically considered multiple perspectives and tools, were viewed as wishy-washy and unreliable, although they in practice gave rather better predictions. However, if one wants to be a talk-show host, it remains better to be a sure-but-wrong hedgehog, since that's what the audience expects.

[N.B. John Oliver has hit his bullseyes, but he's not done himself any favours with his weaksauce snipe at OUR GOD-EMPEROR OF GREAT AMERICAN BORDER WALLS]

Simple. Direct. Win.

While these findings suggested that the CIA etc could replace entire departments with captive monkeys and suffer no loss in efficiency, they admirably went for productivity instead of cost-cutting, thus the Good Judgment Project. This would pit some three thousand civilian volunteers against the professional intelligence community's best and brightest, who moreover had access to additional relevant top secret data. As it was, the top 2% of the volunteers beat the spooks by a considerable margin - and maintained this advantage over years, suggesting that it was not luck, but skill. These people, then, were the superforecasters.

Without further ado, the typical superforecaster traits:
  • Intelligent (80th+ percentile on I.Q.)... but not especially so; volunteers as a whole were about 70th percentile by self-selection, and truly exceptional I.Q. levels had little additional effect on performance

  • Comfortable with numbers. One can't really run from this - for example, if one is satisfied with projecting two events merely as "probable", how is he to combine their likelihood by words alone? 90% and 90% is quite different from 60% and 60%, after all (81% vs. 36%, to be precise), and it only gets worse.

    This is a key divergence from "classical" foretelling, which often involved such side-effects as: one's firstborn, exchange of bodily appendages, creeping insanity, or even one's immortal soul. That said, there doubtless remain many who would be willing to undertake all of the above, rather than endure the depredations of differential calculus.

    However, it was noted that complicated mathematical modelling was generally not involved - any logic tended to be clear and Bayesian. Probability estimates tended to be very precise, though - while one could think that there's no big deal in going with 30% or 35% for a component estimate, superforecasters were the sort of people who would argue for 33% versus 34%.

  • Well-read. Or, news junkies. Follow Keynes' dictum: "When the facts change, I change my mind. What do you do, sir?" Constantly evaluate and process the latest information, and calibrate their internal models. [N.B. On the "how long is knowledge valid" issue brought up recently, the answer appears: if empirically-derived and in the "softer" sciences, often not very long at all] Beliefs tend to be fine-tuned incrementally - frequently, but in small steps... except when it doesn't apply.
Cribbing from Tetlock's own summary here:
  • Philosophy (worldview): Reality is uncertain, infinitely complex, and non-deterministic (unfated)

  • Personality: Open-minded, introspective, curious

  • Methods: Pragmatic, analytical, recognizes diverse perspectives, probabilistic, willing to incorporate new evidence, aware of various psychological biases

  • Work Ethic: Believe in continual self-improvement, sheer perseverance

Tetlock then notes that of the above, the most important predictor of becoming a superforecaster is what he terms "perpetual beta": the drive for constantly updating beliefs, and general work ethic (though this presumably involves deliberate ["perfect"] practice, and not mindless grinding). He put this as having about three times the influence of raw intelligence alone. And, as always, one can compensate for particular deficits with strengths in other areas.

One of the most straightforward and foolproof methods to improve forecasting accuracy, then, would be simple aggregation, or strength in numbers/wisdom of the crowd. From what I could make out, the benefits to teaming up were greater for regular forecasters than superforecasters (to the point that some regular teams beat some "superteams"), in large part because superforecasters tended already to be innately "diverse". Or in other words, unless you're Saitama, hammering out one's thought processes should be beneficial.

But wait, isn't this basically polling? Which brings us to...


Sunset of #NeverTrump?

Personal bias. The immutable bane of forecasters. Nate Silver duly referenced Tetlock, foxes and hedgehogs in his 2012 tome The Signal and The Noise, and devoted multiple paragraphs throughout to the proper recognition and handling of it.

It is only fitting then, upon observing that Silver had been knocking TRUMP since he announced his candidacy, for loyal NIMBLE NAVIGATORS to question if the über-pundit had abandoned objectivity in just the way he had warned against - see the now-whimsical Six Stages of Doom, for instance.

But wait - his subjective commentary aside, Silver has actually been sticking to (mostly) aggregating polls too, which has again proven pretty darn accurate, particularly for the Democratic race. And so, what do the polls say now?


CENTIPEDES, ASSEMBLE!
(Source: reddit.com)


Soon after taking New York, TRUMP continued his sweep of the Northeast, again averaging nearly sixty percent, and keeping on track to smash the record for most total votes in a GOP primary, which one suspects would hold even if normalizing for population growth. Including a likely forty-odd unbound delegates from Pennsylvania (aided by the lovely Ivanka), TRUMP has already passed the thousand-delegate mark, the suggested lower bound for a contested convention by a Republican National Committee insider.

If one plugs this into FiveThirtyEight's handy delegate projector, together with their "expert average", then even considering that their call for TRUMP to capture barely half of California's 172 delegates appears unreasonably conservative given he's way up in the polls, TRUMP still passes 1237 comfortably.

His remaining GOP primary opponents Cruz and Kasich - who, by the way, have less than 75% of TRUMP's delegates combined - responded with... open collusion. With neither able to actually win the nomination outright any longer, Cruz hatched a misbegotten plan for his and Kasich's supporters to vote strategically, offering to stay out of Kasich's way in Oregon and New Mexico, if Kasich yielded Indiana to him.

On this, Governor Kasich - recall, currently one-for-38 in wins, and fourth in a three-man race - appeared to agree... only to announce that his people should still vote for him in Indiana the very next day, and that he was "having the time of his life", before ending the discussion to... eat some eggs.

But, credit to Cruz - this is a guy who doesn't know when he's beaten, and his next scheme was... a Hail Carly throw, unveiling Fiorina as his vice-presidential pick. Most everyone saw it as the desperation maneuver it was, of course:

...and if he loses Indiana too,
get prepared for Cruz to proclaim the site of his Presidential Library!


Not only that, this is also a guy who doesn't recognize that, seriously, nobody likes him. I mean, with TRUMP, maybe he's not everyone's cup of tea, but one doesn't doubt that he has his own circle of friends. With Cruz, you've got his own endorser claiming that no-one in the Senate would convict his murderer, colleagues explaining that people take an instant dislike to him because it simply saves time, and former college roommates ripping him on his genital stimulation habits. Let's just say that if it's one source saying that Cruz snitched on his mates to avoid paying poker losses - an ultimate dick move - it could be a smear, but when most everyone's on the same page, one has to suppose that he's exactly what it says on the cover.

Echoing popular sentiment, a former Republican Speaker of the House has just spoken his mind on Cruz, describing him as "Lucifer in the flesh" [!], before asserting that he had "never worked with a more miserable son of a bitch in [his] life." Sure, politicians turn on each other for tactical advantage at the drop of a hat, but the heartfelt, visceral denunciations of Ted Cruz surely go well beyond the required professional pale. And, if that weren't enough, the real Satanists have taken umbrage.

[N.B. Defenders of the Star of the Morning often argue that he got dealt a crap hand by his Creator, who by self-admission very well knew all the dumb shit that would go down, anyway. As such, it is reasoned, the First of the Fallen embodies the human thirst for free will and free expression, which, come to think of it, sounds vaguely libertarian... and very not Cruz. Score one against Boehner here, then.

Aside-aside on free will and determinism: scientifically, the case for even physical determinism seems to be steadily eroding, which would appear to naturally set limits on how finely the brain can be dissected - and possibly relegate a true mapping to the bin of undecidability; whether this cop-out would qualify as a "soul", on the other hand...

Anyhow, the devil is nothing if not charismatic, and he's just snagged a Fox series of his own. This follows his own solo DC comics series, which examines his associated tropes of free will and predestiny in detail. Incidentally, the tale of Adam's nameless and least-famous second wife, whom YHWH reportedly constructed from scratch - as in a reverse autopsy - before Adam's eyes, seems to suggest that He has low E.Q.; small wonder Lucifer's squeeze retained a low opinion of Him.]


Fallen angels are not always as they seem...
(Source: Sandman #23)


But we're beginning to ramble, so back to THE TRUMPENING: with TRUMP all but having the nomination locked up, his opponents nevertheless understood that citizens are entitled to their own political opinions, that they were after all fellow Americans and could remain good neighbours, and that any protests had to be reasonable and not impose upon the public...

Nah, screw that. In reality, rabid anti-TRUMP thugs descended upon The Don's peaceful California rallies, assaulting TRUMP supporters and smashing police cars; in response, the media sneakily resorted to passive voice to mask the side that was actually effectuating the stone-throwing. Contrast the time when TRUMP's campaign manager brushed past a reporter - who had approached TRUMP against Secret Service orders - and got charged with battery. Hmm.

One could question waving Mexican flags while vandalising public property and pepper-spraying kids helps the #NeverTrump cause how, but then they were never exactly big on long-term critical thinking anyhow. This didn't STUMP THE TRUMP, obviously, with him continuing with his packed mega-rallies in the Golden State after getting Hoosiers legend Bob Knight's endorsement in Indiana, to which Cruz retaliated by talking of "basketball rings". Obama couldn't resist that one.

Ending on a numbers note, TRUMP's up 27 in California, which if it holds, means that he wouldn't have needed Indiana in the first place. And head-to-head with Hillary? It's a dead heat. Voters seem to be coming around that there's a "rally comedian" TRUMP and "serious Presidential" TRUMP, and checking their prejudices accordingly.

Okay, okay, going on technicals, you've still got to give the edge to Ms. Clinton. However, remember also the last rule of superforecasting: the rules apply, until they don't...


THE MADMAN! THE ABSOLUTE MADMAN!
[...and, talking about Xi Pinjing, he walks in like this...]
[N.B. More quality impersonations...]




comments (0) - email - share - print - direct link
trackbacks (0) - trackback url


Next: Men Against The Machine


Related Posts:
War Of The Worlds
Pregame Entertainment
Bow To The Master
Reading Moves
Rak Talk
Back to top

Copyright © 2006-2025 GLYS. All Rights Reserved.