Happy Sunday and welcome to InsideAI! I'm Rob May, CTO at Dianthus, and active angel investor in AI. My most recent investment is a machine learning powered dating app that uses ML algorithms to match people on humor styles. It's called Smile, and I hope you will check it out. If you have an early stage AI company, let me know as I'd love to take a look.
This week I want to discuss a new Chinese AI tool that can press charges for crimes. Rather than evaluate whether this is a good or bad use of AI, I want to ask the question - what happens if it gets implemented? These tools don't exist in a vacuum, and when they are introduced, they have second order effects. In particular, I wonder if a tool like this leads police to "teach to the test."
If you are U.S. based, you may know that term, but if not, I'll briefly clarify. "Teaching to the test" is when teachers are rewarded for how their students do on some kind of standardized test. The teachers then, stop focusing more broadly on learning outcomes and start teaching primarily to make sure students do well on that test. This may includes skills that aren't that useful, like how to do better on standardized tests in general (when to guess and when not to, for example). Those skills shouldn't really replace basic learning skills but, in some classrooms they may.
Turning this back on the Chinese AI prosecutor tool, will police start to "teach to the test?" If they learn what kind of things the algorithm looks for to press charges, will they start to construct their summaries of a crime to favor those words and phrases that the algorithm likes? A human can sometimes wonder if another human is trying to play them, but will an algorithm?
It's a broader problem with AI that people don't think about often enough. When you train on one data set that is independent from the model, in the sense that the data set was created by people or machines that had no idea the model was going to use their data, will people (or machines) start to modify the way they present data to increase the likelihood of the desired outcome from the model?
It's a problem with system structure, when the inputs to the model are controlled by one type of user, but the output of the model is used by someone else - how do you provide oversight and insure model input providers aren't "teaching to the test"? I don't have a clear answer but if you've thought about this problem and have some solutions, I'd love to hear them.
Thanks for reading.
@robmay
Comments
Post a Comment