Skip to main content

Identifying the type of sources from User Prompts using Machine Learning

Created
Active
Viewed 95 times
3 replies
0

I am currently working on a project where we need to map user entered prompts to predefined sources based on its content descriptions. The goal here is not to extract named entities but to understand which type of source a user prompt is referring to by comparing it with existing content descriptions of sources.

Source Descriptions:

  1. Returns and Refunds: Handles inquiries about returning purchased items and getting refunds.

  2. Product Information: Provides detailed information about products, such as specifications, compatibility, and usage instructions.

Example Customer Inquiries:

  1. "How can I return a damaged item I recently purchased?"

  2. "Can you provide the specifications for the latest model of the smart home speaker?"

Answers i am looking for.

  1. What category of machine learning does this problem belong to?

  2. Could you recommend any specific machine learning models suitable for this kind of NLP tasks.

  3. Are there any resources or tutorials that might help in setting up a text classification system for this scenario?

I just given a general e-commerce example for better understanding but actually im dealing with financial related prompts.

  • 168
  • 1
  • 3
  • 20

3 replies

Sorted by:
78997544
1
  • 13.2k
  • 4
  • 44
  • 64

1. What category of machine learning does this problem belong to?

You guessed correctly - it is a (multi-class) text classification.

2. Could you recommend any specific machine learning models suitable for this kind of NLP tasks.

The classic simplified approach would be to feed the input questions into some text embedding model, and add some classifier head to it, for example:

  • A multilayer perceptron as a head with a final number of output neurons that equals to the number of classes, and finally a softmax activation function. Train the network (only the MLP or end-to-end, depends on the quality of your embeddings, your quality requirements and the available training data) with categorical cross-entropy loss.
  • a random forest classifier
  • etc.

Nowadays, with the emerge of LLMs, you have the alternative to use a general-purpose model in zero-shot learning mode with a prompt as simple as:

Classify the following user query into one of these categories: 

 - returns_and_refunds
 - product_information
 - etc.

Return only the category name and nothing else!

User query: {query}

You might experiment providing some examples in the prompt, effectively implementing a few-shot learning approach.

Or, you can train a smaller LLM model with your examples.

Which one to choose? It really depends on the availability and the quantity of the training data, and the economic balance between development cost and runtime cost you want to invest. For example, if you have to classify millions of queries, it might worth to invest in a classical model or training a small LLM. If you need just some hundreds of classification, it will be much quicker to just use a general purpose LLM.

3. Are there any resources or tutorials that might help in setting up a text classification system for this scenario?

Sure, there are tons of resources on the Internet :). Depending on your approach of choice, search for keywords like "text classification with embeddings", "prompt engineering for text classification" or "fine-tuning LLMs for text classification".
78998099
0
  • 168
  • 1
  • 3
  • 20
Author

So the problem is not relevant to 'Semantic textual similarity'? I was under impression of checking the similarity between the prompt and content description and map the higher %.
Also, I was bit confused here because similarity will check both sentences are same or not but goal here is to check which source is relevant to the given prompt.
79002793
0
  • 13.2k
  • 4
  • 44
  • 64

What you are describing is called a k nearest neighbor classifier over the embeddings. It might work well if your classes are semantically well-separated.