p2.py : Outlier seeker for NLU problem

Overview

Data - a natural language type sentence - is given with a tagged intent.
Similar to us asking a "virtual assistant" on an app or your phone with questions like:

how do i say 'hotel' in finnish

This intent should be:

translate

However, some of the intents are not classified correctly. An example would be:

how do i say 'hotel' in finnish

and this is classified as:

translate

Main goal of this script is to weed out those outliers for each types of intent within the input data.

Approaches

After identifying the sentences are human-based, meaning it has the possibility of not being grammatically correct. This points to a more Natural Language Understanding/Processing direction. Following are the two methods I experimented. I ended up using Snips NLU as the outlier engine.

• Doc2Vec
• Snips NLU - See here for details of how I apply this approach.