For a long time, libraries like spaCy <\/strong>have been the go-to tools for pattern matching, dependency parsing, and named entity recognition. But conventional keyword-based solutions may find it difficult to keep up when organizational needs change and inquiries become more intricate.<\/p>\n\n\n\n Enter Large Language Models (LLMs)<\/strong>. Recent breakthroughs in LLMs, from OpenAI o1 to specialized models like Qwen and LLaMA variants, have given us a new toolkit. These models demonstrate an extraordinary capacity for understanding context, ambiguity, and nuance\u200a\u2014\u200akeys to unlocking high-quality intent detection pipelines. For an in-depth exploration of how prompt engineering enhances these capabilities, refer to the dedicated section later in the text.<\/p>\n\n\n\n In this article, I\u2019ll explain how we harnessed LLMs for an advanced AI management system\u2019s intent detection. We\u2019ll talk about the system\u2019s architecture, the shift from spaCy-based approaches to LLM-driven solutions, and the insights gained throughout the process.<\/p>\n\n\n\n At AIHello<\/a>, I work on InSight<\/strong>, an advanced AI management system. The long-term vision is ambitious:<\/p>\n\n\n\n Where does intent detection fit into all this? Early on, we realized that any query posed by a human user (like a manager or team lead) might need data from multiple sources\u200a\u2014\u200aemails, meeting transcripts, and internal chats. Initially, I tried using spaCy to map specific terms to an \u201cintent\u201d (such as \u201cemployee attendance\u201d or \u201ccustomer feedback\u201d) and then retrieve the relevant dataset. <\/p>\n\n\n\n However, spaCy\u2019s keyword-based strategy failed when the questions become more complicated (\u201cWhat do customers think about their campaign performance over the last quarter, and how did that correlate with employee interactions?\u201d).<\/p>\n\n\n\n Solution<\/strong>: LLMs gave us the freedom to more dynamically interpret intent. Rather than strictly defining keyword sets, LLMs allow us to make inferences regarding semantic meaning. Following a number of tests, we were able to match queries to their appropriate data sources with an accuracy of over 70% in our first LLM-based prototype. This was a significant improvement over the previous attempts using spaCy.<\/p>\n\n\n\n InSight<\/strong>, is served through FastAPI and relies on asynchronous processing via Celery. The LLMs are hosted on our servers through Ollama, and connected through their own API, a solution that simplifies running large models. The pipeline looks something like this:<\/p>\n\n\n\n Key Insight:<\/strong> The intent detection module is the linchpin\u200a\u2014\u200awithout accurate source selection, the rest of the pipeline might fetch irrelevant or incomplete data.<\/p>\n\n\n\n Intent detection<\/strong> is not done by a single prompt. Instead, we use an agentic workflow<\/strong>\u200a\u2014\u200abreaking down the logic into multiple steps so we can debug and improve incrementally.<\/p>\n\n\n\n One of the critical modules here is our Intent Processor. It is responsible for calling the Intent Detector and Intent Handlers to coordinate the logic.<\/p>\n\n\n\n The In our earliest tests, switching from spaCy to LLM-based intent detection yielded impressive improvements:<\/p>\n\n\n\n I evaluate InSight\u2019s intent detection capability by testing it on real-world queries. The evaluation involves testing its natural language interpretation and ability to identify relevant data sources. For this demonstration, I will use a sample dataset of 5 random queries.<\/p>\n\n\n\n Please note that I\u2019ve used queries with typos, improper structure and incorrect grammar as well, to reflect real-world scenarios.<\/p>\n\n\n\n Prompt engineering was a critical <\/strong>factor in unlocking the real potential<\/strong> of LLMs within our agentic workflow. By designing precise and modular prompts, we achieved higher accuracy and improved adaptability in our intent detection system. Here\u2019s a detailed breakdown of our approach:<\/p>\n\n\n\n This is the prompt I provided to the agent ( Keep in mind that this prompt is only <\/strong>for this component of the agentic workflow, and all agents should have comprehensive and well-detailed prompts for the best results.<\/p>\n\n\n\n While our experience with LLMs has been transformative, it is critical to recognize the inherent obstacles and limitations of this method. Understanding the potential problems allows us to create more robust and realistic AI systems.<\/p>\n\n\n\n We can anticipate even greater accuracy, stronger reasoning, and improved domain adaptation as LLMs develop further. Imagine connecting fine-tuned LLMs that deeply understand your company\u2019s internal knowledge base, drastically reducing the gap between a manager\u2019s natural language request and the final aggregated analytics they receive.<\/p>\n\n\n\n For us, this is just the beginning. The same LLM-driven approach that improved intent detection can extend to other aspects of the system: from automatic summarization of transcripts to generating advanced analytics code on-the-fly.<\/p>\n\n\n\n Shifting from spaCy to LLM-based intent detection in the InSight system was a game-changer. Instead of battling with strict keywords and predefined patterns, we now use models that have a comprehensive understanding of the query. The result: a smoother, more accurate, and more scalable approach to intent detection.<\/p>\n\n\n\n It\u2019s also important to acknowledge the existence of solutions like spaCy-LLM, which integrate large language models into structured workflows. While spaCy-LLM offers a quick and modular way to implement LLMs for various NLP tasks, our decision to build a custom solution stemmed from the need for greater flexibility, finer control, and the ability to tailor the system precisely to our requirements. <\/p>\n\n\n\n This bespoke approach allowed us to optimize the architecture, implement highly specific prompts, and achieve scalability aligned with our project\u2019s unique demands.<\/p>\n\n\n\n Whether you\u2019re developing a basic chatbot or an intricate organizational intelligence system, take into account the ability of LLMs to comprehend the needs of your users. LLMs have the potential to revolutionize human language interpretation and interaction at scale with thoughtful prompt design, modular workflows, and reliable logging.<\/p>\n\n\n\n Thanks for reading! If you enjoyed this deep dive, consider following for more insights into NLP, LLM integration, and real-world AI engineering solutions.<\/em><\/p>\n\n\n\n By Tanishk Narula, In collaboration with AIHello<\/a>. <\/p>\n","protected":false},"excerpt":{"rendered":" Master advanced intent detection with LLMs. Discover how Large Language Models overcome spaCy’s limitations for complex NLP queries in AI systems like InSight.<\/p>\n","protected":false},"author":33,"featured_media":10927,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[103,163,85],"tags":[561,1149,1216],"class_list":["post-10923","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","category-resources","category-tips","tag-ai-machine-learning","tag-llm","tag-llm-vs-nlp"],"_links":{"self":[{"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/posts\/10923","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/comments?post=10923"}],"version-history":[{"count":1,"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/posts\/10923\/revisions"}],"predecessor-version":[{"id":12608,"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/posts\/10923\/revisions\/12608"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/media\/10927"}],"wp:attachment":[{"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/media?parent=10923"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/categories?post=10923"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aihello.com\/resources\/wp-json\/wp\/v2\/tags?post=10923"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Project Context: The InSight System<\/h3>\n\n\n\n
\n
Traditional NLP vs. LLMs: Why the Shift?<\/h3>\n\n\n\n
\n
A rule-based system might recognize \u201ccomplaints\u201d and output \u201cemails\u201d, but miss the second half of the query related to internal management. Conversely, LLMs can consider the entire query and output a combined intent that will include all three information sources.<\/li>\n\n\n\nArchitecture of the InSight System<\/h3>\n\n\n\n
![\"A](\"https:\/\/www.aihello.com\/resources\/wp-content\/uploads\/2024\/12\/why-i-ditched-spacy-in-favor-of-llms-for-natural-language-processing.png\")
\n
The Intent Detection Workflow<\/h3>\n\n\n\n
![\"A](\"https:\/\/www.aihello.com\/resources\/wp-content\/uploads\/2024\/12\/1e-surc5vbokd0ydaax_gaw.png\")
\n
emails|transcripts,0.88<\/code> meaning we need both emails and transcripts, with a confidence of 0.88, meaning it is 88% certain regarding its answer.<\/li>\n\n\n\n# Example snippet from intent_processor.py\n\ndata_sources, confidence = self.llm_intent_detector.detect_intent(user_query)\n\nif 'unknown' in data_sources or confidence < 0.6:\n return \"Unable to determine required data sources. Please rephrase.\"\n\nraw_data = self.intent_handlers.fetch_data_sources(data_sources, query_context)\n\n# Parse and return a user-friendly response.<\/code><\/pre>\n\n\n\nLLMIntentDetector<\/code> uses a carefully crafted prompt, instructing the model to return data sources and a confidence score. This is where LLMs shine: The model reads a structured prompt, interprets the user query, and decides which data sources are relevant\u2014no more brittle keyword mappings. The aforementioned prompt can be found in the corresponding section below.<\/p>\n\n\n\nPerformance Metrics and Results<\/h3>\n\n\n\n
![\"Three](\"https:\/\/www.aihello.com\/resources\/wp-content\/uploads\/2024\/12\/15h8w6tpqzkp7fgxk2ohq6q.png\")
\n
System Evaluation on Real World Queries<\/h4>\n\n\n\n
Sample Dataset:
+---------------------------------------------------------+-------------------+
| query | sources |
+---------------------------------------------------------+-------------------+
| \"show me unhappy customers from last month\" | emails |
| \"wat was the last thing sarah worked on\" | transcripts|chats |
| \"Who's responsible for handling Mrs. Doe's complaints?\" | emails |
| \"how many tickets did rachel resolve this week?\" | emails |
| \"show me john's current project status\" | transcripts|chats |
+---------------------------------------------------------+-------------------+<\/code><\/pre>\n\n\n\n=== Intent Detection Evaluation Report ===
Exact Match Accuracy: 80.00%
Average Confidence: 91.40%
=== Performance by Data Source ===
Source: transcripts
Precision: 66.67%
Recall: 100.00%
F1-Score: 80.00%
Source: chats
Precision: 66.67%
Recall: 100.00%
F1-Score: 80.00%
Source: emails
Precision: 100.00%
Recall: 66.67%
F1-Score: 80.00%
=== Average Metrics ===
Average Precision: 77.78%
Average Recall: 88.89%
Average F1-Score: 80.00%
=== Incorrect Predictions ===
Number of incorrect predictions: 1
+------------------------------------------------+--------------+---------------------+------------+
| Query | True Sources | Predicted Sources | Confidence |
+------------------------------------------------+--------------+---------------------+------------+
| how many tickets did rachel resolve this week? | emails | chats | transcripts | 0.85 |
+------------------------------------------------+--------------+---------------------+------------+<\/code><\/pre>\n\n\n\nDeep-Dive into Prompt Engineering<\/h3>\n\n\n\n
Key Techniques in Prompt Engineering<\/h4>\n\n\n\n
\n
It is worth noting that I used Markdown to write my prompt, this is because many LLMs have a format that they best understand. You can refer to this research paper<\/a>, which talks about this topic in detail. In my case, Markdown yielded the best results.<\/li>\n\n\n\nMy Intent Detection Prompt<\/h4>\n\n\n\n
LLMIntentDetector<\/code>) responsible for detecting data sources from user queries. You can analyze how it details very specific instructions for the LLM, to ensure a semblance of deterministic output.<\/p>\n\n\n\nYou are an intent detection system for an employee data retrieval system. Given a user query, you must:
1. Classify which data sources should be fetched to answer the query
2. Return the data sources as a comma-separated list with a confidence score
Available data sources and their contents:
- emails: Contains all customer-related communications, including:
* Customer feedback and satisfaction levels
* Customer complaints and issues
* Customer interactions with employees
* Historical customer relationships
* Customer names and their sentiment
- transcripts: Contains records of formal meetings and calls, including:
* Employee task assignments and updates
* Project discussions
* Performance reviews
* Team meetings
* Client meetings
* Training sessions
- chats: Contains informal internal communications, including:
* Quick task updates
* Day-to-day employee activities
* Team discussions
* Work schedule discussions
* Break time communications
* Informal status updates
Guidelines for source selection:
1. For ANY customer-related information, ALWAYS include 'emails'
2. For employee task tracking, use 'transcripts' for formal tasks, 'chats' for informal updates
3. For comprehensive employee activity analysis, include all relevant sources
4. For work patterns or attendance, include both 'transcripts' and 'chats'
5. For performance analysis, include sources based on context (customer-facing vs internal)
Your response must be in the format: \"data_source1|data_source2,confidence_score\"
Examples with explanations:
Query: \"What do customers think about our service?\"
Response: emails,0.95
Explanation: Customer feedback is exclusively in email communications
Query: \"What tasks did John complete this week?\"
Response: transcripts|chats,0.88
Explanation: Tasks are recorded in both formal meetings and informal updates
Query: \"How is Sarah performing in customer support?\"
Response: emails|transcripts|chats,0.92
Explanation: Comprehensive performance analysis needs all sources
Query: \"Who was absent last Thursday?\"
Response: transcripts|chats,0.85
Explanation: Attendance info is found in formal records and team chats
Query: \"Which customer complained about delivery delays?\"
Response: emails,0.98
Explanation: Customer complaints are always in email records
Query: \"What project is Tom working on?\"
Response: transcripts|chats,0.90
Explanation: Project information spans formal and informal communications
If you're not confident about the intent, return a low confidence score.
If the query doesn't match any need for these data sources, return \"unknown,0.0\"
User Query: {query}
Required Data Sources and Confidence:<\/code><\/pre>\n\n\n\nBest Practices for Using LLMs in Intent Detection<\/h3>\n\n\n\n
\n
LLMIntentDetector<\/code> includes a structured list of data sources, their contents, and step-by-step instructions on how to decide. Being explicit helps the LLM reason more effectively.<\/li>\n\n\n\nPotential Limitations of LLM-Based Intent Detection<\/h3>\n\n\n\n
\n
The Future of LLM-Based Intent Detection<\/h3>\n\n\n\n
Conclusion<\/h3>\n\n\n\n
\n\n\n\n
You can find me on: LinkedIn<\/a>, Github<\/a>, Twitch<\/a>, Twitter<\/a><\/em><\/p>\n\n\n\n