API Days meet the Simpson Paradox

Business API

In early October I attended the API Days NZ Conference at the Viaduct Events Centre in Auckland. The conference focus was ‘APIs. Unifying Digital Business’, and invited those interested in improving the way New Zealand organisations share digital resources with customers using API.

The most interesting presentation which stuck with me was from Lillian Grace, founder and CEO of Figure.NZ . Figure.NZ pulls together data from the public, private, and academic sectors into one place, making it easy for people to access.

Lillian spoke about how the online world gives us the opportunity to do things differently, yet we are only starting to see changes in how we share information, communicate, organise and govern ourselves.

The takeaway message for us is we’re entering an age where there are huge amounts of data available – yet as a society we don’t have the same levels of literacy with data as we do with language.

There is a danger of misinterpreting the data and make bad decisions if we don’t put it in context and critically test what it is telling us with what we know of the subject.

The Simpson Paradox states that “trends which appear in groups of data may disappear or reverse when the groups are combined”. It’s like peeling back the layers of an onion – the top level analysis of a piece of data can lead you to one conclusion, but going down a level can lead you to the opposite conclusion. The only way to avoid it is to understand the context of the data you are looking at, and critically evaluate what it is telling you.