Welcome back! 👋 We’re excited to be kicking off Tech Leading Ladies in 2024 with some amazing speakers and your favourite programs lined up for the year. Our first event was last week, and it was so great to have everyone able to join and share in the experience. In case you missed it, here’s a quick run down of the meetup, and we look forward to seeing you at the next one!
At Tech Leading Ladies February Meetup, we had the privilege of hearing AI visionary Kendra Vant speak with us about some foundational skills needed for building great AI products. We kicked off the event with a lightning talk from our very own community member Hima Tk about RAG and how this will be the next hot topic.
Key takeaways from Hima’s talk on Retrieval Augmented Generation (RAG):
- RAG Revolutionizes LLMs with External Knowledge: RAG represents a significant advancement in AI frameworks by integrating Large Language Models (LLMs) with external knowledge bases. This integration allows LLMs to supplement their internal information with factual data, leading to more informed and accurate responses.
- Two-Phased Approach: RAG operates through a two-phased approach: retrieval and content generation. In the retrieval phase, a Smart Retriever component searches the knowledge base for relevant content. This content is then used by an LLM generator to generate answers to queries, ensuring that responses are grounded in factual context.
- Trulens Framework for Evaluation: To assess the effectiveness of RAG models, the Trulens framework defines three key metrics: Context Relevance, Groundedness, and QA Relevance. These metrics help track the success of RAG results by evaluating the relevance of retrieved context to queries, the support of responses by context, and the overall relevance of answers to queries.
For more RAG content, see What is RAG, RAG Theory, and TruLens website.
Why AI needs to be built on robust data foundations
And what that means for tech leadership.
A visionary product & tech exec, Kendra Vant has 16 years of commercial experience (Xero, SEEK, Telstra, Deloitte, Commonwealth Bank) and eight years of R&D experience (MIT, Los Alamos National Laboratory), driving development of products that change people’s lives for the better. In this session, she shares with us her learnings and advice for building AI powered products on a robust data foundation.
In today’s tech landscape, the rush to integrate AI into various aspects of our lives has led to unforeseen challenges and failures, highlighting the critical need for robust data foundations. Kendra, with seven years of experience in AI-powered product development, emphasises the importance of balancing speed and product robustness, especially given the complexity of AI products and data.
Despite significant investments in AI and the pressure to move swiftly, there’s a growing awareness of the risks associated with hasty implementations. Enhancing understanding of data flows and promoting rigorous scrutiny of assumptions are imperative for both creators and consumers of AI technology. However, persistent challenges, such as the relentless pursuit of market speed and widespread knowledge gaps, continue to hinder progress and fuel subpar AI implementations. Bridging these gaps is essential for fostering resilience and efficiency within AI development teams, ultimately leading to more robust and effective AI applications.
Here are some key concepts to hopefully fire up your curiosity and think about how we can bridge our vocabulary and knowledge gaps.
- Data models are not AI models and are critical to bringing shared understanding and consistency to AI.
Too few words in the English language, so many opportunities for talking past each other! Clear communication is essential for shared understanding and consistency, prioritising specificity over brevity to minimise confusion and ensure accurate interpretation by listeners. - Bias is baked into the data used to train, and with no “fair” benchmark to compare it to, removing bias can be fraught.
A quote Kendra likes to use to get people thinking: “Show me a fair world and I’ll show you a fair algorithm.” This highlights that we will always reproduce whatever world view is contained within the data that we are training upon. And while some bias might be removed, it’s impossible to remove all of it. Bias is also relative to morals and world view of the observer. - Training and scoring AI models are different processes that require differently shaped data stores and streams.
Understanding and discussing as a team what data is needed for each process and being able to retrain any model as it gets stale over time are important points to think through and accurately estimate the costs of. - No AI model can give a guarantee of 100% accuracy, so your workflows can’t expect that.
Discuss the limitations of AI model accuracy at the project’s outset. If you need 100% accuracy, maybe you can’t use AI for that workflow. - History is critical to AI, but not so much for your application functioning quickly and correctly.
Retaining historical data is crucial for AI development. Keep your data about who did what and when so you can use them in your models and avoid lengthy delays when you need to collect outcomes data for 12 months before you can begin a build in earnest.
Some final suggestions from Kendra about how we can be better AI stewards:
Round out your own gaps.
- Continuously strive to identify and address personal knowledge gaps
- Foster curiosity and recognizing when to seek guidance from others
- Embrace vulnerability and acknowledge areas of uncertainty
Speak to be understood.
- Prioritise clarity and comprehension in communication
- Recognizing the importance of tailoring vocabulary to the listener’s comprehension
Borrow with pride.
- It can be hard to teach something foundational as an expert because you no longer remember not understanding it.
- Keep a library of exceptional foundational resources ready to share, to help others on the journey.
To sum it all up, by prioritising understanding, continuous learning, and collaboration, we can build a more inclusive and innovative future for AI technology. 🚀
If you want to keep hearing from Kendra we recommend following her Substack.
