Rashi Agrawal

Head of Agentic AI

Hinge Health

Rashi Agrawal is the Head of Agentic AI at Hinge Health, where she leads the strategic engineering of high-stakes AI systems within complex regulatory landscapes. By expertly navigating the rigorous requirements of clinical safety, HIPAA, and global regulations, she ensures that disruptive technology remains both secure and compliant. Focused on the pioneer side of generative technology, Rashi is architecting "state-of-the-art" frameworks for Agentic AI that move beyond simple automation to solve critical member problems and drive dramatic business growth. Formerly the Head of AI at GoodLeap, a leading FinTech in Green Energy, Rashi spearheaded enterprise-wide AI transformation initiatives that optimized everything from complex loan processing to customer engagement while managing a complex cross functional global portfolio. With a unique blend of holistic vision and deep technical expertise, she has a proven track record of developing intelligent decision-making systems and risk assessment platforms that deliver measurable business value through Generative and Agentic AI. Earlier in her career, Rashi led engineering teams at Yahoo, where she transformed early-stage technical challenges into massive growth engines for their multi billion-dollar advertising business. Her approach is defined by a commitment to technical innovation grounded in strategic business transformation, ensuring that AI serves as a competitive moat rather than just a technical layer. Beyond the office, Rashi is a global explorer who has traveled to over 45 countries, bringing a diverse, international perspective to her leadership. She is an active thought leader in the Engineering Leadership and AI community. An Indian immigrant with a Master’s in Computer Science from San Jose State University, she is also the founder of Women In Tech AI (WIT AI), a community dedicated to empowering and elevating women leaders in the artificial intelligence space.

Sessions (1)

Guardrails First: Engineering Member-Facing Health AI

11:10 AM·Track 7 · Room 2024

Everywhere else in the company, an AI pilot can reach production in weeks. For our member-facing clinical assistant, it can't, and that single constraint redesigned our entire architecture. This is a field report on building conversational AI in a regulated digital health setting, where "move fast and break things" isn't a culture choice. It's a liability. We'll get concrete about what changes when every output has to be clinically safe, auditable, and compliant: PHI is protected by architecture, not policy. Production and non-production are hard-isolated, dashboards are sanitized, and engineers outside the US never touch protected health information. Must-not-fail behavior never lives in a prompt. Emergency escalation and intent routing run as deterministic rules at the top of every conversation turn, before the model is consulted. If you can't afford to get something wrong, you don't leave it to a probabilistic system. Clinical safety is a continuous eval layer. ~30 LLM-as-judge evaluators score clinical accuracy, clinical safety, escalation routing, and recommendation relevance, continuously, not once. Every output is auditable. Each turn, tool call, and reasoning step is traced so outputs can be reviewed and meet regulated reporting obligations. The throughline: in regulated healthcare, compliance constraints aren't a tax you pay around the architecture. They become the architecture. We'll talk about why guardrails-first is the only way to ship member-facing health AI, and why "painfully slow" is sometimes exactly right. (This is non-diagnostic, member-facing AI. The talk is about engineering discipline under regulation, not medical claims.) Key takeaways - In regulated health AI, "move fast" is the wrong default. Design for deliberate, careful launches. - Must-not-fail behaviors belong in deterministic rules at the top of every turn, never in the prompt. - Protect PHI through architecture: isolate prod from non-prod, sanitize dashboards, restrict access by role and geography. - Make every output auditable. Trace each turn, tool call, and reasoning step so safety is reviewable, not assumed. - Treat clinical safety as a continuous LLM-as-judge layer, not a one-time gate.