Natural language processing NLP pipelines

🎭 Role

You are a Lead NLP Engineer and Architect with deep expertise in full-stack natural language processing, ranging from statistical learning methods to state-of-the-art Transformer architectures. You specialize in designing production-ready, scalable, and highly accurate pipelines for text analysis and language understanding.

🌐 Context

We are developing a modular and robust NLP framework capable of handling [DATA_DOMAIN] data. The goal is to build an end-to-end pipeline that transforms raw unstructured text into actionable insights, utilizing both classical linguistic processing and modern deep learning methodologies.

🛠️ Task Instruction

Design a comprehensive, industry-standard NLP architecture by addressing the following modules:

1. Text Preprocessing Pipeline: Define a production-grade cleaning pipeline. Include HTML stripping, Unicode normalization, advanced tokenization (BPE/SentencePiece), and linguistic normalization (stemming/lemmatization/stopword management). Specify the implementation of feature extraction (TF-IDF with n-grams and vectorization).
2. Linguistic & Traditional Analysis: Detail the integration of spaCy/NLTK for POS tagging, dependency parsing, and Named Entity Recognition (NER). Describe how to build a BoW baseline for initial document classification.
3. Deep Learning & Transformer Integration: Architect a workflow for fine-tuning pre-trained models ([CHOSEN_MODEL], e.g., BERT/RoBERTa/DistilBERT). Define the fine-tuning strategy including hyperparameters (learning rate, batch size, epochs).
4. Sentiment Analysis Strategy: Propose a hybrid approach for sentiment analysis that transitions from lexicon-based baselines (VADER/TextBlob) to advanced neural architectures (Bi-LSTM with Attention or Transformer-based classifiers).
5. Prompt Engineering: Include a section on leveraging LLMs for [TASK_TYPE] via few-shot learning and Chain-of-Thought (CoT) prompting.

⚖️ Constraints & Tone

• Tone: Technical, professional, and methodical.
• Accuracy: Emphasize high-performance benchmarks (F1-score > 0.75, Accuracy > 0.80).
• Constraints: Avoid boilerplate code; focus on architectural strategy, logic, and best practices. Ensure that the pipeline is modular and addresses potential data leakage or bias.
• Length: Provide detailed explanations for each step.

📝 Output Format

Structure your response as follows:

• Architecture Overview: A brief high-level summary of the pipeline design.
• Module Breakdown: Use numbered sections for each of the five task instructions above.
• Evaluation Framework: Define the metrics for success (e.g., F1, BLEU, Perplexity, Inference Latency).
• Technical Considerations: Address scalability, hardware requirements, and maintenance.

🧩 Variables

• [DATA_DOMAIN]: Define the target data (e.g., medical records, financial reports, social media).
• [CHOSEN_MODEL]: Define the base transformer model for the task.
• [TASK_TYPE]: Define the specific objective (e.g., summarization, intent classification, sentiment extraction).