In this panel of the HubSpot for Startups annual AI Summit in San Francisco, we explored the rapidly evolving world of AI agents. From the chatbots and copilots we interact with daily to emerging autonomous agents and robots that will transform how we live and work, this session provided valuable insights into where AI technology is heading.

Moderated by JC Mao, Distinguished Fellow and Venture Partner at Fellows Fund (ex-Google and ex-Microsoft), the panel featured an impressive lineup of industry experts:

• Tamar Yehoshua, President of Product and Technology at Glean
• Denis Yarats, Cofounder and CTO at Perplexity
• Laurie Voss, VP Developer Relations at LlamaIndex
• Tri Dao, Chief Scientist at Together.ai

Defining AI Agents

The panel kicked off by addressing a fundamental question: What exactly is an AI agent? As Tamar Yehoshua pointed out, the definition has become increasingly blurry as more companies label their products as "agents."

According to a definition pulled from Perplexity itself, an agent is something that:

• Takes input and gathers information through sensors
• Makes decisions based on that information
• Takes autonomous actions
• Is decisive in its execution

The panelists shared their favorite AI agents, highlighting both consumer and enterprise applications:

• Yehoshua mentioned Google Assistant as an early example of an agent that can handle multi-turn tasks
• Yarats (unsurprisingly) favored Perplexity's search agent with its multi-step reasoning capabilities
• Voss highlighted an accessibility agent that reads web pages aloud while summarizing content and offering tool-based interactions
• Dao pointed to Devin from Cognition Labs, an agent that functions like a junior software developer

Building Successful AI Agents

Both Glean and Perplexity shared their experiences building successful AI agent applications, highlighting key lessons learned:

Perplexity's approach:

• Started with the basics: providing accurate answers quickly
• Invested heavily in detailed evaluation systems
• Built tools to understand where the system has blind spots
• Maintained a comprehensive overview of system performance

Glean's enterprise challenges:

• Security and privacy are paramount when dealing with enterprise data
• Built a sophisticated retrieval engine and knowledge graph
• Focused on understanding what signals determine relevance
• Preserved permissions and security across various tools
• Had to combat the non-deterministic nature of LLMs for enterprise settings where consistency is expected

Infrastructure for AI Agent Development

LlamaIndex and Together.ai discussed how their infrastructure platforms support AI agent developers:

Together.ai's infrastructure priorities:

• Fast and robust model serving for low latency and high throughput
• Support for long context windows, especially for multimodal content
• Memory management for agents that need to access documents or maintain a scratch pad

LlamaIndex's framework approach:

• Originally focused on retrieval augmented generation (RAG)
• Discovered that complex retrieval strategies become increasingly agentic
• Absorbs best practices from industry research
• Provides developers with a menu of agent-building strategies
• Especially useful for creating research assistants in fields with large amounts of unstructured data

Future Developments and Breakthroughs

The panelists shared their expectations for future AI agent breakthroughs:

• Denis Yarats: Replacing auto-regressive generation with planning and reasoning over search
• Tamar Yehoshua: Improved reliability with self-reflection capabilities for multi-turn agents
• Laurie Voss: LLMs that can flawlessly write SQL queries to unlock structured data in databases
• Tri Dao: Enhanced long context capabilities, with models handling millions of tokens

The panel noted that Google has already announced support for a 2-million token context window, with internal testing reaching 10 million tokens. This expanded context capability will enable developers to create agents that can process entire codebases, lengthy documents, and multimedia content.

Competing in a Crowded AI Marketplace

As foundation models continue to improve, the panelists discussed strategies for startups to compete with tech giants:

• Do something against the grain - Glean focuses on horizontal integration across all platforms, something big companies are not incentivized to do
• Move with greater velocity - Startups can pivot quickly and maintain tight feedback loops with customers
• Focus on mission over competition - Build for the future while taking advantage of inefficiencies in larger companies
• Embrace open-source approaches - LlamaIndex integrates with new technologies as they emerge
• Lower barriers to entry - Together.ai emphasized how easy it is now to build AI demos without deep technical expertise

Mitigating Risks and Building Safeguards

The panel acknowledged that with great capabilities come significant risks:

• Perplexity focuses on providing accurate, trustworthy information by investing heavily in verification and evaluation
• Glean emphasizes permission-aware security to respect enterprise data hierarchies
• Together.ai recommends using separate guardrail models (like Meta's Llama Guard) to check outputs

Final Thoughts on the Future

The panel concluded with excitement about AI agents' potential:

• Automation will remove friction and drudgery from daily tasks
• The industry is just getting started - today's models will seem primitive in a few years
• Even modest improvements in foundation models will make agents truly reliable and capable
• The barrier to entry continues to lower, making AI development accessible to more creators

AI Disclaimer: The insights shared in this video or audio were initially distilled through advanced AI summarization technologies, with subsequent refinements made by the writer and our editorial team to ensure clarity and veracity.