#38 Back to Basics — RAG, Transformers, ML Optimization, and LLM Evaluation.

Author(s): Towards AI Editorial Team

Originally published on Towards AI.

Good morning, AI enthusiasts! This week, the community and I are answering some recurring questions about RAG, coding assistants, transformers, machine learning, and more. You will also find fun collaboration opportunities and memes.

Enjoy the read!

What’s AI Weekly

Many clients asked us (Towards AI), “But why would I use RAG if Gemini can process millions of tokens as input?” So, is RAG dead? That’s what I investigated in this week’s iteration of What’s AI. I explore the differences between RAG and sending all data in the input and explain why we believe RAG will remain relevant for the foreseeable future. This post should help you determine whether RAG is suitable for your application. Read the complete issue here!

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

This issue is brought to you thanks to GrowthSchool:

🦾 Master AI, ChatGPT and 20+ AI Tools in just 3 hours

Don’t pay for sh*tty AI courses when you can learn it for FREE!

This incredible 3-hour Workshop on AI & ChatGPT (worth $399) makes you a master of 25+ AI tools, hacks & prompting techniques to save 16 hours/week and do more with your time.

Sign up now (free for first 100 people) 🎁

This masterclass will teach you how to:

• Do AI-driven data analysis to make quick business decisions
• Make stunning PPTs & write content for emails, socials & more in minutes
• Build AI assistants & custom bots in minutes
• Solve complex problems, research 10x faster & make your simpler & easier

You’ll wish you knew about this FREE AI masterclass sooner 😉

Register & save your seat now! (valid for next 24 hours only!)

Learn AI Together Community section!

Featured Community post from the Discord

Aman_91095 has been working on the GenAI Career Assistant, built using LangChain and Streamlit, a project designed to experiment with AI-powered job search tools. It helps with the job search process, helps you find job listings that fit your profile, generates cover letters customized for specific applications, and provides useful information about potential employers. Check it out here and support a fellow community member. If you have any feedback or questions, share them in the thread!

AI poll of the week!

The results show a very clear reliance on ChatGPT. Are general-purpose models enough for most use cases? Are specialized models only required for proprietary applications? Let’s discuss this in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Ananya.exe is looking for a partner to collaborate on a finance-based project (which involves knowledge of multi-AI agents, RAG pipelines, information retrieval, NLP tasks, end-to-end development and deployment, etc.). If you know finance and can work with the above technical specifications, reach out in the thread!

2. Gere030199 is seeking a marketing co-founder for their Discord bot project. They need someone experienced in creating engaging content. If this sounds interesting, connect in the thread!

Meme of the week!

Meme shared by creitingameplays

TAI Curated section

Article of the week

Streamline Your LLM Evaluation: A Step-by-Step Guide to RAG Metrics with Streamlit by Maxime Jabarian

This piece presents a new Streamlit app intended for RAG evaluation. It offers an easy-to-use platform that shows chatbot performance using clear metrics and graphs. By integrating a comprehensive set of evaluation metrics beyond simple accuracy, the app ensures that users can easily understand and interpret the strengths and weaknesses of their LLM models in a clear and visually engaging manner.

Our must-read articles

1. How to use SVM in Power Systems Analysis? by Optimization team

Machine Learning has become a buzzword lately, with recruiters frequently advertising “data scientist” positions when they’re really seeking experts in optimization. This post emphasizes that many machine learning methods are fundamentally based on optimization. In other words, optimization laid the groundwork for the development of machine learning — much like the chicken laying the egg!

2. Attention is all you need: How Transformer Architecture in NLP started by Surya Maddula

This article discusses the evolution of transformer architecture in NLP, starting with the “Attention is all you need” paper. It also explores the problem of contextualized word embeddings and how transformer architecture addresses it by introducing the encoder-decoder model for translation. It also presents a few fine-tuning examples and transformer-based language models.

3. Querying SQL Database Using LLM Agents — Is It a Good Idea? by Sachin Khandewal

This blog explains different ways to query SQL Databases using Groq to access the LLMs. It also explains how to leverage LLM Agents to build an SQL Agent using an advanced DSPy framework and highlights its limitations.

If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI