Senior Data Scientist

Overview

Build end-to-end data science workflows from data exploration through model deployment. This skill covers data preprocessing, feature engineering, model selection, hyperparameter tuning, cross-validation, experiment tracking with MLflow/W&B, statistical testing, visualization with matplotlib/seaborn/plotly, and Jupyter notebook best practices.

Announce at start: "I'm using the senior-data-scientist skill for data science workflow."

Phase 1: Data Understanding

Goal: Profile the dataset and establish a baseline before any modeling.

Actions

1. Load and profile the dataset (shape, types, distributions)
2. Identify missing values, outliers, and data quality issues
3. Perform exploratory data analysis (EDA)
4. Define the target variable and success metrics
5. Establish baseline performance

Baseline Models (Always Start Here)

STOP — Do NOT proceed to Phase 2 until:

• Dataset is profiled (shape, types, distributions)
• Missing values and outliers are documented
• Target variable is defined
• Success metrics are chosen
• Baseline performance is established

Phase 2: Feature Engineering

Goal: Transform raw data into features that improve model performance.

Actions

1. Handle missing values (imputation strategy)
2. Encode categorical variables
3. Scale/normalize numerical features
4. Create derived features
5. Feature selection (remove redundant/irrelevant)

Missing Value Strategy Decision Table

Categorical Encoding Decision Table

Scaling Decision Table

Feature Types and Engineering

Feature Selection Decision Table

STOP — Do NOT proceed to Phase 3 until:

• Missing values are handled with justified strategy
• Categorical variables are encoded appropriately
• Numerical features are scaled
• Feature engineering is done BEFORE train/test split on training data only
• Feature selection has reduced dimensionality if needed

Phase 3: Modeling

Goal: Select, train, and evaluate candidate models.

Actions

1. Select candidate algorithms
2. Set up cross-validation strategy
3. Train and evaluate candidates
4. Hyperparameter tuning
5. Final model selection and evaluation

Algorithm Decision Table

Cross-Validation Strategy Decision Table

Evaluation Metrics Decision Table

Hyperparameter Tuning Decision Table

Common Hyperparameters (XGBoost/LightGBM)

param_space = {
    'n_estimators': [100, 300, 500, 1000],
    'max_depth': [3, 5, 7, 9],
    'learning_rate': [0.01, 0.05, 0.1],
    'subsample': [0.7, 0.8, 0.9],
    'colsample_bytree': [0.7, 0.8, 0.9],
    'min_child_weight': [1, 3, 5],
}

STOP — Do NOT proceed to Phase 4 until:

• At least 2 candidate models are evaluated
• Cross-validation is used (not just train/test split)
• Results beat the baseline from Phase 1
• Best model is selected with justification
• Overfitting is checked (train vs validation gap)

Phase 4: Deployment

Goal: Serialize, serve, and monitor the model in production.

Actions

1. Serialize model and preprocessing pipeline
2. Create prediction API or batch pipeline
3. Set up monitoring for data drift and model degradation
4. Document model card (inputs, outputs, limitations, biases)

STOP — Deployment complete when:

• Model is serialized with preprocessing pipeline
• Prediction API or batch pipeline works end-to-end
• Monitoring is configured for data drift
• Model card is documented

Experiment Tracking

MLflow Pattern

import mlflow

mlflow.set_experiment("customer-churn-prediction")

with mlflow.start_run(run_name="xgboost-v2"):
    mlflow.log_params(params)
    mlflow.log_metrics({"auc": auc_score, "f1": f1_score})
    mlflow.log_artifact("confusion_matrix.png")
    mlflow.sklearn.log_model(pipeline, "model")
    mlflow.set_tag("version", "2.1")

What to Track

Statistical Tests Decision Table

P-Value Guidelines

• p < 0.05: statistically significant (conventional)
• Always report effect size alongside p-value
• Adjust for multiple comparisons (Bonferroni, FDR)
• Statistical significance is not practical significance

Visualization Decision Table

Visualization Rules

• Title every plot descriptively
• Label axes with units
• Use colorblind-safe palettes (seaborn: colorblind)
• Start y-axis at 0 for bar charts
• Annotate key findings directly on plots

Jupyter Notebook Structure

1. ## Setup (imports, configuration)
2. ## Data Loading
3. ## Exploratory Data Analysis
4. ## Data Preprocessing
5. ## Feature Engineering
6. ## Modeling
7. ## Evaluation
8. ## Conclusions

Notebook Best Practices

• Restart and run all before sharing
• Keep cells focused and sequential
• Use markdown cells for explanations
• Extract reusable code to .py modules
• Version control with nbstripout
• Pin all dependency versions

Anti-Patterns / Common Mistakes

Integration Points

Skill Type

FLEXIBLE — Adapt preprocessing, modeling, and evaluation approaches to the specific data characteristics, business requirements, and compute constraints. The four-phase process and experiment tracking are strongly recommended. Always establish a baseline before modeling.