Where the model stops and the person starts
Every product I've worked on has had to answer the same question: what should the software decide, and what should a person decide? At Monster Roster, we sold lineup recommendations for daily fantasy sports. When customers who followed our picks entered identical lineups, they split the winnings. We changed the product into a guided optimizer: the software proposed a core, and each customer made the final choices. The question matters more now, because software can do much more of the work.
These are the rules I use when I put AI into someone's working day. They come from building Agency Edge, an AI research and CRM product used by a photography talent agency and two schools, and from owning live pricing APIs at Swish Analytics.
Give the model the reading. Keep the rules people can inspect.
For a photography talent agency, the task was matching prospects across roughly 40,000 CRM contacts to the agency's photographers. A model could have ranked the whole CRM. Instead, models did what they were good at: reading company websites, extracting fields and assigning business categories. Matching stayed in plain application rules built on the agency's own taxonomy and thresholds, combining category overlap, role fit and campaign engagement. Every tag carried its reasons, so staff could see why a match appeared before deciding whether to follow up.
Make the threshold the customer's decision.
Every proposed field change in Agency Edge carries a confidence label. Staff set the level at which a change is accepted automatically inside Agency Edge, and everything below it goes to a review queue. Where that line sits depends on the team, the field and the cost of a wrong value. The people doing the work are best placed to make that call.
Accepting a change is not the same as writing it.
The first version wrote research back to Salesforce and Campaign Monitor automatically. In the current product, accepting a change inside Agency Edge and writing it to the customer's system of record are separate steps. Before anything reaches Salesforce, staff see the proposed values next to what is there now and confirm the sync. The system of record belongs to the customer.
Automation never changes consent.
Enrichment can fill in a contact's title, company or location. It never changes whether that person can be emailed. An existing opt-out stays in place, whatever the research finds.
Every automated rule needs named exceptions.
At Swish, two customers wanted college-football player results confirmed faster. We set a rule to confirm eligible results automatically 30 minutes after a game. Players without recorded statistics still needed participation data, and later stat corrections could still change a result, so we told customers to keep polling. Slow-confirmation complaints went from a couple per week to none reported from rollout through September 2026. The rule’s exceptions were part of the design from the start.
Watch what people change.
The number I find most useful is not how much the system produced. It is how often people changed it. In Agency Edge's current review flow, 88% of proposed changes are accepted as written and 12% are edited or rejected. In the Marin Academy pilot, staff found wrong-person matches. We audited and corrected the records, added confidence information so staff could see which matches needed a closer look, and split interests into categories they could filter. The 160-person pilot became a paid 5,078-record engagement.
Start with the people doing the work.
At Swish, my first users were our own traders, not our clients. Building for the people closest to the work, and measuring the games they could cover rather than the features we shipped, made the client-facing products better.
Most knowledge work does not come with a test suite. The review step, the threshold and the exceptions are the product. As models take on more of the work, I want to keep building software where people decide what good looks like and own the result.