Battling Confirmation Bias When Training TAR

Categories:   News, Syndicated Posts
Article By:  

In this age of division and widely divergent views of reality, I do a lot of reading trying to understand how so many of us fall down internet rabbit holes into such extreme perspectives. This short article and the prior one on cognitive bias resonated with me on an issue I see frequently when brought in to analyze/support a large TAR review or even search term development. Once upon a time I managed a review for a brilliant attorney who insisted we have a few reviewers pretending to be the DOJ. This classic game theory tactic turned up items that were missed by the primary review team and allowed defense counsel to proactively address them. We talk about initial random sample sets to determine rough relevance richness and QC reviews of the final non-relevant set. But all of this is conducted by reviewers subject to confirmation bias. They can become blind to the ‘unknown unknowns’ because they are not seeking them. Maybe we need to incorporate a parallel or sequential ‘black hat’ team charged with taking the adversarial retrieval task?

Written by Greg Buckles

Independent consultant focused on eDiscovery and IG solutions.

All author posts   |