Happy Sunday and welcome to InsideAI! I'm Rob May, Founder at CTO at Dianthus. We've been buying e-comm companies and improving them with custom AI software we've built but, after 18 months of that, we are now at the point where we are ready to spin out some of the software tools and let other people use them. Today I want to talk about one of the lessons learned from that experience, and how it might influence AI tools in the future.
The problem with machine learning models is that they don't always work well at first. They make errors, and they sometimes require human supervision and correction - much like a newbie employee might. This isn't a workflow we are used to today. As a result, ML companies often have higher churn rates early in their lifecycle, compared to traditional SaaS companies, and ML products often have slower adoption cycles. The time to value for a customer is often longer, even though after that inflection point the value is much higher than in non-ML tools.
We got through this at Dianthus by having stages of tool deployment. Initially, we gather data and work behind the scenes on whatever model we are trying to build. Once something appears to work, we may run it silently in production. What I mean by silently is that the eventual user (maybe the marketing or ops team) doesn't have access to it yet. We are simply testing it out on real world data because model performance usually drops significantly when you first put it in the real world, and the AI team needs time to understand why and fix it.
After that, we hit stage 3 where we run models in parallel with the employee the model was designed to aid. Say a model is making a prediction about what text to use in an email to a customer. We would show that text but the employee can still write whatever they want. The idea here is to warm up an employee to the idea that the tool can make good suggestions.
In stage 4, the model runs on it's own and the employee approves all the outputs. To take our text-> email example, the model would write all the emails but the employee would approve them before they were sent. Often, you need to stay at this stage for a long time because the next stage is a big jump. Why? Because part of the reason you want to automate things like this is so you can make them more personalized. If you have 100,000 customers, you probably divide them, currently, into no more than a dozen personas. Maybe you craft emails for each persona, and now you let the machine draft those. But with AI/ML, you could divide them into 10,000 personas and maybe each email is targeted to just 10 highly similar people. The issue is, no human can audit 10,000 email variations. Stage 4 is about getting the employee comfortable that the machine is doing a good job.
In stage 5, the machine is pretty trusted, automates the whole process, and humans only casually review some subset of the work. In this phase, the machine tries to detect anomalies that the human can review, but otherwise runs on its own.
The idea we've been toying with as we think about rolling some of our AI marketing tools out to the general public is, how do you easily walk someone through these stages? One idea is to build a separate user interface for each stage, and help graduate the user between interfaces. First let them work alongside the AI and see how it's doing. Then have an interface that helps them approve everything the machine does. Then have an anomaly detection and review interface.
User interfaces that change as you use the tool are not something we have a lot of experience with today, so I have no idea how well it will work, or be accepted by our future customers. But I think to realize the value of AI, and really unleash what it can do, something like this is required.
The point of this newsletter is to share what I learn about AI as an investor and operator in the space, and this changing UX/UI issue is something that I've been thinking about a lot lately. If you've had similar ideas, or worked on something like this, I'd love to hear from you.
Thanks for reading.
@robmay
Comments
Post a Comment