CISC5950 Big Data Project 2

For each part, start the Spark cluster and run the corresponding test.sh

Part 1

Accuracy for each category:

• toxic: 0.96
• severe_toxic: 0.99
• obscene: 0.97
• threat: 0.99
• insult: 0.97
• identity_hate: 0.99

Part 2

Accuracy: 0.70

AUC ROC score: 0.70

Part 3

Accuracy: 0.88

Part 4

Random forest accuracy: 0.89

Decision tree accuracy: 0.68