Natural language processing NLP pipelines

Build comprehensive NLP pipelines for text analysis, sentiment analysis, and language understanding tasks. Text preprocessing pipeline: 1. Data cleaning: remove HTML tags, normalize Unicode, handle encoding issues. 2. Tokenization: word-level, subword (BPE, SentencePiece), sentence segmentation. 3. Normalization: lowercase conversion, stopword removal, stemming/lemmatization. 4. Feature extraction: TF-IDF (max_features=10000), n-grams (1-3), word embeddings (Word2Vec, GloVe). Traditional NLP approaches: 1. Bag of Words: document-term matrix, sparse representation, baseline for classification. 2. Named Entity Recognition: spaCy, NLTK for entity extraction, custom entity types. 3. Part-of-speech tagging: grammatical analysis, dependency parsing, syntactic features. Modern approaches: 1. Pre-trained transformers: BERT (bidirectional), RoBERTa (optimized BERT), DistilBERT (lightweight). 2. Fine-tuning: task-specific adaptation, learning rate 5e-5, batch size 16-32. 3. Prompt engineering: few-shot learning, in-context learning, chain-of-thought prompting. Sentiment analysis: 1. Lexicon-based: VADER sentiment, TextBlob polarity scores, domain-specific dictionaries. 2. Machine learning: feature engineering, SVM/Random Forest classifiers, cross-validation. 3. Deep learning: LSTM with attention, BERT classification, multilingual models. Evaluation metrics: accuracy >80% for sentiment, F1 score >0.75, BLEU score for generation, perplexity for language models.