Create dev.md

problem-of-the-day/day25/dev.md

@@ -0,0 +1,20 @@
+# Dev Problem of the Day
+### NLP Tokenizer
+
+For machine learning roles, companies often check your fundamentals, specifically for NLP (natural language purposes), and the interviewer may ask you about your understanding of the fundamentals. So today, we are going to cover one of those fundamental topics, which is tokenizerms can easily process. Token is a way to numerically represent a word.
+
+So, what is a tokenizer??
+The tokeniser is the first step after data cleaning, it is used for converting a text or sentence into smaller parts, known as tokens, which machine learning algorithms can easily process. Token is a way to numerically represent a word.
+
+### Resources⁠
+- [Tokenizers - Hugging Face](https://huggingface.co/learn/nlp-course/chapter2/4)
+- [Understanding BPE and Tokenization Algorithms - Hugging Face](https://huggingface.co/learn/nlp-course/chapter6/1)
+[Entire Chapter 6]
+
+This resource contains detailed discussions, videos, and examples related to tokenizers.
+
+### Tips⁠⁠
+- ⁠Understand what tokenizer is and then learn about the subword tokenization algorithm.
+- Understand BPE (Byte Pair Encoding Tokenization), as it is currently used by GPT-2 and many LLMs.
+- Go also through Fast Tokenizer, Word Piece Tokenizer, and Unigram Tokenizer.
+- Then finally, learn how to build your own tokenizer.