Mahesh Chulet

As an AI Security Engineer, you must build these projects. Systems that prove you can secure, cage & audit when autonomous agents touch your stack. 1.) Crescendo Attack Fuzzer Build: A multi-turn stateful harness that dynamically adapts payloads to groom context windows and bypass commercial system prompts. Why: Single-shot "ignore instructions" injections are dead. Modern attacks require stateful, multi-turn grooming. 2.) Sandboxed MCP Proxy Build: A JSON-RPC middleware that intercepts Model Context Protocol tool calls, validates parameters via Pydantic and blocks "confused deputy" privilege escalations. Why: Tool poisoning is the SQL Injection of the Agentic Era. Agents are only as secure as the tools they call. 3.) Inter-Agent Sanitization Bus Build: A message broker that strips hidden markdown and malicious instructions between Agent A and Agent B in a multi-agent swarm. Why: Swarms create internal trust boundaries. Agent A can be tricked into hijacking Agent B's tools via cross-agent injection. 4.) PII UUID-Swapping Gateway Build: A pre-flight proxy using NER to swap SSNs and credit cards for deterministic UUIDs before the LLM sees them, restoring them post-generation. Why: Zero-trust compliance demands that raw customer PII never touches a third-party frontier API. 5.) RAG Citation & Poison Detector Build: A pipeline that cross-references LLM claims against exact vector DB chunk hashes to detect shadow-context injection and citation spoofing. Why: If an attacker slips a poisoned document into your knowledge base, they own every answer the AI generates. 6.) CFG Output Caging Layer Build: Middleware enforcing Context-Free Grammar (CFG) decoding to physically block the generation of executable code, markdown links or unauthorized schemas. Why: System prompts are suggestions. Grammar constraints are physics. You must enforce security at the token-decoding level. 7.) Delegated Agent OAuth Broker Build: An auth service issuing time-limited, strictly scoped JWTs to agents so they can act on a user's behalf without holding raw passwords. Why: Agents need hands to do work, but they should never hold the master keys to the enterprise. 8.) PyRIT CI/CD Security Gate Build: A GitHub Action running 1,000 adversarial payloads via Microsoft PyRIT on every PR, blocking merges if the jailbreak rate exceeds 1%. Why: Manual red teaming doesn't scale. You need automated, continuous adversarial fuzzing integrated into your deployment pipeline. 9.) Semantic API Rate-Limiter Build: A gateway calculating embedding similarity of incoming requests to detect and block scraping bots trying to extract your model's weights. Why: Your fine-tuned weights are your IP. Competitors will try to distill your model through the public API. 10.) Computer-Use Pixel Auditor Build: A vision pipeline that scans screenshots for adversarial invisible UI elements and Set-of-Mark spoofing before a GUI agent clicks. Why: Screen-hijacking and pixel manipulation are the new privilege escalation vectors for Computer Use agents. 11.) Vector DB Sleeper-Agent Scrubber Build: An anomaly detector scanning episodic memory and vector stores for delayed-trigger payloads and semantic backdoors. Why: Long-term memory is a persistent attack vector. A poisoned memory bank will trigger exploits weeks after ingestion. 12.) Cryptographic Trajectory Ledger Build: An append-only, hash-chained audit log recording the exact prompt, retrieved context, tool calls and state mutations for every agent action. Why: When an agent makes a mistake that causes financial damage, the enterprise needs cryptographic proof of exactly why it happened. 13.) AI Supply-Chain SBOM Scanner Build: A tool auditing "vibe-coded" outputs and MCP server dependencies for hallucinated packages, typosquatting and backdoors. Why: AI-generated code and dynamic tool discovery create massive, unvetted supply chain risks. Bookmark & Repost.

Let’s connect! I am working a 9-5 while building some side projects. What are you working on, reply with your project

Vibe coded an app? Before you deploy to production, check these 10 things: 1.) No API keys or secrets in your code or repo 2.) Every route checks who the user is (auth) and what they can access 3.) Database rules lock down who can read and write each row 4.) All user input is validated on the server not just the frontend 5.) Rate limits on login, signup and any paid API calls 6.) Error messages don't leak stack traces or database details 7.) Dependencies are updated and scanned for known vulnerabilities 8.) Backups are on and you've actually tested restoring one 9.) Logging and alerts so you know when something breaks 10.) A spending cap on every cloud and AI API account AI writes code fast. It doesn't care if you get hacked. Bookmark this before your next deploy.

I was going through Forward Deployed Engineer roles on LinkedIn. Almost every one asks for the same thing: proof you've shipped agents. Here are 15 agentic AI projects that will make your resume stand out: • Customer support agent that reads docs, answers tickets and escalates to a human • Sales research agent that finds leads and writes personalised outreach • Meeting agent that joins calls, takes notes and updates the CRM • Invoice agent that reads PDFs and pushes data into accounting software • Codebase Q&A agent using RAG over a real GitHub repo • PR review agent that comments on bugs and security issues • Data analyst agent that turns plain-English questions into SQL and charts • Browser agent that fills forms and books appointments • Multi-agent research team: planner, researcher, writer, critic • MCP server that connects an AI to your company's internal tools • Onboarding agent that sets up accounts and walks new hires through docs • Recruiting agent that screens resumes and schedules interviews • Incident agent that reads logs and suggests fixes • Eval dashboard that scores your agent on accuracy, cost and latency • Guardrails layer that blocks prompt injection and risky actions Pick 2–3. Deploy them. Write about what broke. That's your resume. Bookmark this and send it to someone applying for AI roles.

Tech companies are quietly moving away from classic MCP tool calling. Here's what changed: MCP is the standard that lets AI agents use tools. Ask your agent: "Find this customer and show me their orders." • It calls tool 1 to find the customer • The answer goes back to the model • It calls tool 2 for the orders • Back to the model again Bigger jobs? Again. And again. And again. Every trip costs tokens. And with thousands of tools, the tool list alone floods the context. The fix: Code Mode. The model writes code. The code calls the tools. Everything runs in one go. Why it works: LLMs have read millions of lines of real code. They've seen very few tool-call examples. Cloudflare's API has 2,500+ endpoints. With Code Mode, the agent only needs 2 tools. Stop making agents call tools. Let them write code.

Got an AI engineer interview coming up? These 25 questions come up again and again. Prep them and you'll walk in ready: • What's the difference between a chatbot and an agent? • How does an agent decide which tool to use? • Explain the reasoning loop: think, act, observe, repeat • When would you use RAG instead of fine-tuning? • How do embeddings and vector databases work together? • How do you handle a context window that's too small? • Short-term vs long-term memory: how would you design both? • What is MCP and why does it matter? • Single agent or multi-agent: when do you pick each? • How does an orchestrator split work between sub-agents? • How do you stop an agent from looping forever? • What guardrails would you add before giving it real tools? • Where should a human approve actions? • How do you evaluate whether an agent actually works? • What metrics would you track in production? • How do you debug an agent that gave a wrong answer? • How do you reduce hallucinations? • How do you cut token cost without hurting quality? • How do you reduce latency in a multi-step agent? • How would you handle a tool call that fails? • How do you protect against prompt injection? • How would you design an agent that books flights? • How would you design a customer support agent? • What breaks first when you scale agents to 1M users? • Tell me about an agent you built and what failed Most candidates can explain an LLM. Very few can design a system around one. Bookmark this and send it to someone with an interview this month.

50 terms every AI builder should know: 1.) LLM: a large language model trained to predict the next word 2.) Token: a small chunk of text the model reads and writes 3.) Context window: how much text the model can see at once 4.) Prompt: the instruction you give the model 5.) System prompt: hidden instructions that shape every response 6.) Temperature: how random or creative the output is 7.) Inference: running a trained model to get an answer 8.) Latency: how long you wait for a response 9.) Hallucination: when the model confidently makes things up 10.) Fine-tuning: training a model further on your own data 11.) RAG: fetching relevant documents and feeding them to the model 12.) Embedding: turning text into numbers that capture meaning 13.) Vector database: storage built to search embeddings fast 14.) Chunking: splitting documents into pieces for RAG 15.) Reranking: sorting search results so the best ones go first 16.) Agent: a model that plans and takes actions on its own 17.) Tool calling: letting the model use APIs, code or search 18.) MCP: a standard way to connect models to tools and data 19.) Agent loop: think, act, observe, repeat 20.) Subagent: a smaller agent handed one piece of the job 21.) Orchestration: coordinating multiple agents or steps 22.) Memory: what an agent remembers between sessions 23.) Guardrails: rules that block unsafe or off-topic output 24.) Prompt injection: hidden text that tricks a model into misbehaving 25.) Jailbreak: a prompt that gets around a model's safety rules 26.) Evals: tests that measure how well your AI performs 27.) Benchmark: a standard test used to compare models 28.) LLM-as-judge: using one model to grade another's output 29.) Ground truth: the correct answer you test against 30.) Structured output: forcing the model to reply in JSON or a set format 31.) Function schema: the definition of a tool the model can call 32.) Streaming: showing the response word by word as it's generated 33.) Caching: reusing past results to save time and money 34.) Prompt caching: reusing a repeated prompt prefix to cut cost 35.) Rate limit: how many requests you're allowed per minute 36.) Open weights: a model you can download and run yourself 37.) Distillation: training a small model to copy a big one 38.) Quantization: shrinking a model so it runs faster and cheaper 39.) LoRA: a cheap way to fine-tune only a small part of a model 40.) GPU: the chip that runs AI models 41.) Multimodal: a model that handles text, images, audio or video 42.) Reasoning model: a model that thinks step by step before answering 43.) Chain of thought: the model writing out its reasoning first 44.) Few-shot: giving the model examples in the prompt 45.) Zero-shot: asking with no examples at all 46.) Vibe coding: building software by describing it to AI 47.) Harness: the setup of tools and rules around a coding agent 48.) CLAUDE.md: a file of project rules Claude Code reads every session 49.) Skills: reusable instructions an agent loads when it needs them 50.) Human in the loop: a person approving key steps before they run Bookmark this. You'll need it in your next AI meeting.

It’s almost impossible to comprehend just how massive the gap has become between people working at the AI frontier and everyone else. What’s even crazier is that this gap isn’t closing. It’s getting wider. If AI progress had started to plateau, most people and companies would have had time to catch up, integrate the technology, and standardize their workflows. Eventually, AI would have become a level playing field, with everyone using roughly the same tools and capabilities. The competitive advantage would have diminished. But we’re seeing the exact opposite. AI progress is accelerating at an almost vertical rate. That means the workflows you carefully built and standardized 3–6 months ago could already be outdated. Now think about companies that started adopting AI a year ago, assumed they had figured it out, and moved on. They’re potentially even further behind. I’ve never experienced a technology in my life or career that evolves this quickly. And that means our expectations need to evolve just as fast. Our brains aren’t wired to process this pace of change, and neither are traditional organizational change management processes. When you introduce recursive self-improvement into the equation, things get even more interesting. We’re entering a period where you constantly need to revisit problems you couldn’t solve a few months ago and ask whether AI can solve them today. Solutions that were impractical yesterday might suddenly become viable. AI-native companies clearly have a significant head start in this environment. The real question now is whether larger, more established organizations can adapt quickly enough to keep pace. Wild times.

Tired of Claude Code asking "Allow this command?" 50 times a day: ~/.claude/settings.json { "permissions": { "allow": [ "Bash(npm run test:*)", "Bash(git status)", "Bash(git diff:*)" ] } } Safe commands run. Everything else still asks.

Here’s how to truly master LLM evals, step by step, with real projects to demonstrate your skills: 1. Build something that can fail first Don’t study evals in theory alone. Build a small LLM application with real failure points: RAG over a documentation site, a text-to-SQL bot, or an agent that uses tools. It doesn’t need to look good. You need traces, not polish. 2. Manually inspect 100 traces Export inputs, outputs, retrieved chunks, and tool calls into a spreadsheet. Describe each failure in one plain-English sentence. Group similar failures, then count them. Most people skip this and end up building metrics for problems they don’t actually have. Project: Publish your failure taxonomy with counts for each category. 3. Create your first golden dataset Start with 50 cases covering your top 3 failure categories, 20 straightforward happy-path cases, and 10 adversarial examples. Include columns for input, expected output or rubric, category, difficulty, and segment. Commit the dataset to git and tag every prompt change with the dataset version used to evaluate it. 4. Generate synthetic data the right way Define your dimensions (persona × intent × complexity × edge case), generate combinations with a script, and use an LLM to turn each combination into a natural-language query. Review and discard low-quality examples. Use embeddings to identify near-duplicates. Project: Build a 300-case synthetic dataset with a coverage matrix showing which combinations you’ve tested. 5. Write deterministic checks before using LLM judges Use pytest and pydantic for schema validation, regex checks, required fields, and length constraints. Execute generated SQL against a real SQLite test database. Run generated code inside a sandbox. Aim to make at least half of your failure categories testable without an LLM. 6. Build an LLM judge and align it with human judgments Manually label 100+ examples as pass or fail. Split them into train, dev, and test sets. Use binary verdicts, a written rubric, and a critique-before-verdict approach. Iterate on the judge prompt using the dev set until both TPR and TNR are high, then evaluate on the test set once. Lock the judge model version. Project: Create a judge-alignment notebook containing a confusion matrix and every prompt iteration you tested. 7. Evaluate RAG retrieval and generation separately Label the relevant chunks for 50 queries. Calculate recall@5, MRR, and nDCG. Then measure faithfulness by breaking each answer into claims and verifying each claim against the retrieved context. Change the chunk size or embedding model and observe which metrics improve or decline. 8. Evaluate agent trajectories Log the tool name, arguments, results, and token usage at every step. Create reference paths for 30 tasks. Measure tool precision and recall, argument correctness, step count, and error recovery. Intentionally make a tool fail 20% of the time and see whether your agent recovers or gets stuck in a loop. 9. Simulate users for multi-turn conversations Give a simulator LLM a persona, hidden goal, and constraints (for example, “won’t share the order number unless asked twice”). Run 50 conversations with a 15-turn limit. Evaluate goal completion, turns to resolution, and policy violations. 10. Measure consistency, not just one lucky run Run every test case 5 times at your production temperature. Report pass@1 and pass^5 side by side. Cases that produce inconsistent results are the flaky behaviors you need to fix before production. 11. Learn enough statistics to avoid fooling yourself Bootstrap your confidence intervals. Compare models on the same examples using paired tests (McNemar’s test works for binary pass/fail outcomes). If your test cannot distinguish the two runs, you haven’t demonstrated an improvement. You may have just gotten lucky. Further reading: Hamel Husain and Shreya Shankar on evals, Eugene Yan’s posts on LLM evaluation, and the τ-bench paper (the source of pass^k).