The first time we let our slackbot, D.B. Fresh, join our Slack conversations without being invited, we disabled the feature within an hour.
Colin and I would be having a back-and-forth, and D.B. always had something to add to practically every turn. We’d given our internal bot a slightly neurotic, slightly irreverent personality, which is funny when he picks his moment, but when you’re trying to get serious work done, he becomes THAT co-worker who doesn’t know when to stop tagging a joke.
A couple of issues ago, I promised to share how we give our agents more freedom while making sure they do useful work. With D.B. Fresh, that meant teaching him when to join a conversation and when to stay quiet.
Recently, I decided to give D.B. another chance to earn his autonomy.
The original setup used a Haiku model to evaluate whether D.B. should interject. This time, we’re using jev from TypeSafe. There’s a lot of excitement around it, but I find it useful to think of jev as a fast and efficient general-purpose classifier. In other words, you give it context and then ask it to make a judgment.
For D.B., that judgment determines whether a conversation is an appropriate place to autonomously contribute. Given how quickly we regretted the last experiment, we’re starting with a tuning period where D.B. Fresh can react with 👀 when he thinks he could contribute, but needs an invitation to speak. The reactions let us determine two things: can he contribute something useful, and can he recognize when it’s welcome?
Yesterday, Colin asked me to explain type safety to him. He didn’t quite get why it was becoming so popular lately. Given that we both know our Bibles and Christian theology pretty well, I asked him if he was familiar with the concept of typology before trying an analogy. In biblical interpretation, typology connects an earlier person or event with something it prefigures later.
An 👀 reaction appeared on my message.
At first, I thought it was Colin. That’s exactly the kind of message he reacts with 👀 , and I usually read it as “that’s interesting, tell me more.” Then I noticed it was from D.B. Fresh, so I asked whether he wanted to add something.
“Typology is basically pre-modern duck typing with better literature”
He connected Adam as a type of Christ with the way a type system assigns meaning and properties to a category. He also said that biblical typology allows partial correspondence, whereas stricter typing demands stronger guarantees.
It surprised me that he connected the technical conversation with the theological one and added something I actually wanted to discuss. He even used the example I wanted to use!
The last time we let him loose, I’d been worried about getting him to stop talking. But this time, I just wanted him to participate MORE.
So far, I haven’t noticed any irrelevant eye reactions. If anything, he has missed some opportunities. But I’m away from Charlotte this week, and Colin and I have been less active in the channels where D.B. Fresh participates. It’s been a pretty bad week for tuning his autonomy.
We probably need more time before letting him reply on his own.
I also keep thinking about the reaction I mistook for Colin’s, and I kind of want to let him loose.
— Alan, Colin & the Torta team
P.S. If you’re building an agent for your team or customers, it helps to know whether people find its work useful enough to keep using it. Our D.B. Fresh field guide explains how we measure that. Get the guide here