Deepali Shinde (@Deepali_Shinde_) header
Deepali Shinde (@Deepali_Shinde_) profile picture

I build production AI systems and side projects. Writing about what I learn in engineering, productivity, and figuring out life.

San Francisco, CA 🌉
Joined March 2020
Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)Photo by Deepali Shinde (@Deepali_Shinde_)
a chatbot keeps answering from an old policy, so you filter out the old documents. then someone asks what the policy was last year. that's the example i used in my first-product guide. both questions are valid; the retrieval change needs to handle both.
3
4
126
in my learning loop repo, the judge reads saved conversations. change the agent's prompt and run that judge again, and it still scores the old answers. the changed agent hasn't answered anything yet.
2
4
128
to check the prompt change, i'd need fresh conversations from the changed agent, with matched cases and the same scoring rubric. that's a separate step. regrading the saved conversations doesn't tell me whether the new prompt helped.
12
a voice lookup can feel slow before the backend has even started. in call reviews, a long spoken introduction explained part of that wait. the agent was still talking about what it was going to do. that time belongs on the timeline too.
2
3
127
I used to see a wall of green squares on GitHub and think, damn, this person codes. Then I saw all that blue on my Codex heatmap and felt a little too proud of November. 😂 Anyone else starting to check their Codex or Claude heatmap the way they used to check GitHub?
1
74
in my order-cancellation example, the agent can look up an order while it's cancellable and try to cancel it after it ships. so the service checks again when applying the change. the earlier lookup can already be out of date. i trace it here: deepalis0.github.io/series/f…
4
144
when a voice issue shows up again after a fix, i check when the call happened against when the fix was deployed. a report can arrive later about a call on the old version. i need that context before treating it as a regression.
2
4
142
i also keep the issue open until the corrected behavior is verified on a later call. the merge and deployment are recorded, but i still have to inspect the call evidence and explicitly record that the issue is resolved.
23
i used one order cancellation to explain how an ai assistant actually does something. who checks permission? who changes the order? what can the assistant say if the service's reply gets lost? the walkthrough has diagrams for those paths: deepalis0.github.io/series/f…
1
3
163
Reached a milestone !! this makes me so happy , 350+ subs on my Newsletter in 7 days! If you’re looking for practical content on building AI workflows and products this one’s for you ! Next target 🎯- crossing 500+ subs 💪
2
1
7
251
AI evaluations can mark a correctly working AI agent as a failure. whether you're building a chat agent or an AI workflow, evaluations test whether the system behaves as expected. one thing they can check is the tool-call trajectory: which tools the agent called, with what inputs, and in what order. but two independent lookups could happen in either order. A test that demands one exact sequence can reject a perfectly valid run. before requiring an exact match, decide which steps actually need to happen before others.
3
5
143
For example - checking weather and local events can happen in either order. but getting permission must happen before making a change. choose a test that allows harmless reordering. Add a custom check for any dependency that must hold.
1
33
One way to compare tool-call paths is with LangChain's AgentEvals evaluation library, which works with LangSmith. You need to provide a reference trajectory of the tool calls you expect for your agent you evaluating and the evaluator compares the agent's actual run against that reference. you can choose how strict that comparison should be : - strict: match the message and tool-call sequence in order. - unordered: require the same tool calls, but allow any order. - subset: allow calls to be skipped, but no extra calls. - superset: require all expected calls, but allow extra calls. Choose the mode based on what your workflow actually requires. docs.langchain.com/langsmith…
1
23
my development workflow is about 10x faster with coding agents. i still spend time figuring out whether a proposed fix makes sense. especially in voice, where 'the agent ignored me' can turn out to be a cancelled tool call. here's what that workflow looks like:
4
1
5
199
then i decide where the fix belongs: prompt, turn-taking, state or infrastructure. after the code changes, i review the findings and test the behavior. when i changed the end-call mechanics, that included calls where goodbye was already working.
1
16
once the fix is verified, i update the issue history, review checks or skill instructions with what changed. even a broken command gets corrected. otherwise the next coding agent walks into the same problem and starts figuring it out again.
10