Behind the insights

Our AI bot leverages the Progress Agentic RAG (retrieval-augmented generation) and a ‘walled garden’ (or ‘closed system’ or ‘knowledge box’) of resources to respond to queries. Our walled garden of resources currently includes:

  • RISE website text
  • all RISE briefs and tools prepared by RISE
  • additional documents (e.g., taxonomies and other tools) linked to from this RISE website.

Over time we will add resources from additional verified sources, including:

  • all evidence syntheses (and economic evaluations) included in Health Systems Evidence (HSE) and included in the ‘children and youth services’ and ‘community and social services’ parts of Social Systems Evidence (SSE)
  • Ontario and Ontario-relevant Canadian policy documents indexed in the Government of Canada’s Open Government Data Portal and in Overton (to support the contextualization of content to the Ontario policy landscape).

We also hope that we will be able to add synthesis- and AI-ready data from empirical studies addressing health and human services included in ESIC’s open-data system.

We have configured and iteratively adjusted the RAG search to optimize responses based on the type of question(s) being asked (e.g., research/evidence queries versus organization- and program-related queries versus policy context queries). We have done this through a combination of:

  • content-retrieval rules, including rules about filtering (e.g., focusing on evidence syntheses for questions about evidence and policy documents for questions about policy landscape) and weighting (e.g., focusing on more recent and higher quality evidence syntheses)
  • large language model (LLM) system and user prompts, including instructions about plain language, professional tone, and fidelity to content in the ‘walled garden.’

Over time we will continue to refine the RAG search, including:

  • adjusting the rules and prompts based on ongoing testing by our staff and on user feedback
  • exploring an ‘LLM as ‘judge’ approach to responses, which includes identifying the most important parts of responses (which for us are typically claims about what the evidence supports, such as ‘what works, for whom, and under what conditions’), assessing them, serving them up for a ‘human-in-the-loop’ review process, and making explicit comparisons between AI and humans
  • testing a range of third-party large-language models (LLMs) available as part of the Progress Agentic RAG (e.g., from Anthropic, Micosoft Azure, Google Gemini and Vertex AI, and OpenAI) while balancing factors such accuracy and cost.