~/Data Science Methodology

Introduction — John B. Rollins's 10-Step Methodology

The methodology presented here was put forward by John B. Rollins, a Data Scientist at IBM Analytics. It consists of 10 steps grouped into 5 modules, each containing two stages.

🔹 The Five Modules

1. From Problem to Approach
2. From Requirements to Collection
3. From Understanding to Preparation
4. From Modelling to Evaluation
5. From Deployment to Feedback

🔹 Key Concepts You'll Learn

1. Introduction to Data Science Methodology
2. Steps for Data Science Methodology
3. Model Validation Techniques
4. Model Performance — Evaluation Metrics

Prerequisite: Class XI AI concepts + familiarity with Capstone Projects and their objectives.

1.1 Module 1 — From Problem to Approach

❓ Step 1 · Business Understanding

"What is the problem that you are trying to solve?"

• Understand the customer's problem by asking questions and discussing with stakeholders.
• Define objectives that support the customer's goal — also called Problem Scoping and defining.
• Use the 5W1H Problem Canvas (Who, What, When, Where, Why, How) to deeply understand the issue.
• Apply the Design Thinking (DT) Framework.

🧭 Step 2 · Analytic Approach

"How can you use the data to answer the question?"

The team picks the most appropriate approach by asking:

1. How much / how many? → Regression.
2. Which category does the data belong to? → Classification.
3. Can the data be grouped? → Clustering.
4. Any unusual pattern? → Anomaly Detection.
5. Which option should we give the customer? → Recommendation.

🔹 Four Main Types of Data Analytics

1.2 Module 2 — From Requirements to Collection

📋 Step 3 · Data Requirements

"What are the data requirements?"

Requirements are driven by the analytic approach chosen. Using the 5W1H method, identify:

• Types of data required — numbers, words, images.
• Structure — table, text file, database.
• Sources of data.
• Any cleaning / organisation steps needed.

🔹 Three Types of Data

• Structured data — organised in tables (customer databases).
• Unstructured data — no predefined structure (social media posts, images).
• Semi-structured data — some organisation (emails, XML files).

📥 Step 4 · Data Collection

"What occurs during data collection?" — a systematic process of gathering observations or measurements. After this step, the team decides if more or less data is required.

🔹 Primary Data Source

Original, raw, unprocessed data collected firsthand: direct observation, experimentation, surveys, interviews. Examples — marketing campaigns, feedback forms, IoT sensor data.

🔹 Secondary Data Source

Already-stored data ready to reuse: books, journals, websites, internal transactional databases. Methods: social-media tracking, web scraping, satellite data tracking, smart forms.

Popular online sources: data.gov · World Bank Open Data · UNICEF · Open Data Network · Kaggle · WHO · Google.

1.3 Module 3 — From Understanding to Preparation

🔎 Step 5 · Data Understanding

"Is the data collected representative of the problem to be solved?"

• Evaluate relevance, comprehensiveness and suitability for the problem.
• Apply descriptive statistics (univariate analysis, pairwise correlation).
• Use visualisation (histograms) to assess content, quality and initial insights.

🧹 Step 6 · Data Preparation

"What additional work is required to manipulate and work with the data?" This is the most time-consuming step.

• Cleaning — handle invalid/missing values, remove duplicates, assign suitable formats.
• Combine data from multiple sources (archives, tables, platforms).
• Transform data into meaningful input variables.

🔹 Feature Engineering

1.4 Module 4 — From Modelling to Evaluation

🤖 Step 7 · AI Modelling

"In what way can the data be visualised to get to the required answer?" Modelling is iterative — may loop back to Data Preparation. Data scientists test multiple algorithms to find the best fit.

• Descriptive Modelling — summarises & understands data without predicting. Uses Summary Statistics (mean, median, mode, SD, variance, range, percentiles/quartiles) and visualisations (bar charts, histograms, pie charts, box plots, scatter plots).
• Predictive Modelling — uses historical data to predict future outcomes. Techniques: regression, classification, time-series forecasting. Needs a training set (historical data with known outcomes) that acts as a gauge to calibrate the model.

📊 Step 8 · Evaluation

"Does the model really answer the initial question or does it need adjustment?" Use test data to measure accuracy, precision, recall, F1 score. Two phases:

1. Diagnostic Measures — ensure the model works as intended. A decision tree can evaluate a predictive model; a testing set with known outcomes can refine a descriptive model.
2. Statistical Significance Test — verifies the model accurately processes and interprets data. Avoids second-guessing when the answer is revealed.

1.5 Module 5 — From Deployment to Feedback

🚀 Step 9 · Deployment

"How does the solution reach the hands of the user?" Once evaluated, the trained model is made available to users — sometimes first to a limited group or test environment to build confidence, then rolled out fully. Deployment often involves additional internal teams, skills and technology.

💬 Step 10 · Feedback

"Is the problem solved? Has the question been satisfactorily answered?" Feedback comes from:

• Results of deployment.
• User and client feedback on model performance.
• Observations from how the model works in the live environment.

The process from Modelling → Feedback is highly iterative. Each step sets the stage for the next, making the methodology cyclical and ensuring refinement at every stage.

2.1 Solution Representation — What Fits the Problem?

The analytic approach chosen in Step 2 dictates how the solution is represented:

🔹 Represent with Stakeholders in Mind

• Business users → dashboards, simple charts.
• Technical team → metrics, code and model parameters.
• End users → one-line recommendations or alerts.

3.1 What Is Model Validation?

🔹 Benefits of Model Validation

• Enhances model quality.
• Reduces risk of errors.
• Prevents overfitting and underfitting.

3.2 Common Validation Techniques

• Train-Test Split
• K-Fold Cross Validation
• Leave-One-Out Cross Validation (LOOCV)
• Time-Series Cross Validation

3.3 Train-Test Split

A technique for evaluating any supervised learning algorithm — works for both classification and regression.

• Train dataset → used to fit the machine-learning model.
• Test dataset → used to evaluate predictions against expected values.
• Estimates model performance on new data.

🔹 Configuring the Split

Main parameter: the size of train vs test, expressed as a ratio between 0 and 1.

• Train 67 % → test size 0.33 → Train 0.67 / Test 0.33.
• Consider: computational cost of training & evaluating · representation of each set.

🔹 Common Split Percentages

• 80 : 20 (most common)
• 70 : 30
• 67 : 33

There is no single optimal split — pick one that meets the project's objectives.

3.4 K-Fold Cross Validation

Splits data into k folds (subsets). The model is trained on some folds and tested on the remainder — the process is repeated so every fold is used as a holdout exactly once.

• Gives a more accurate measure of model quality — especially when many modelling decisions are made.
• Takes more time — estimates models once per fold.

3.5 Train-Test Split vs Cross-Validation

4.1 Why Evaluation Metrics?

Evaluation metrics help assess the performance of a trained model on the test set. They enable comparison between models to pick the best one.

• Classification problems have a finite set of target classes.
• Regression problems have a continuous target variable.

Hence, different metrics for each task.

4.2 Evaluation Metrics for Classification

📊 1. Confusion Matrix

• True Positives (TP) — model predicted Yes, real was Yes.
• True Negatives (TN) — model predicted No, real was No.
• False Positives (FP) — model predicted Yes, but actually No.
• False Negatives (FN) — model predicted No, but actually Yes.

🎯 2. Precision

What proportion of predicted positives is truly positive?

Precision = TP / (TP + FP). Should be as high as possible.

🔁 3. Recall

What proportion of actual positives is correctly classified?

Recall = TP / (TP + FN).

⚖️ 4. F1 Score

A good F1 score means low false positives and low false negatives — you correctly identify real threats and are not disturbed by false alarms. F1 = 1 is perfect; F1 = 0 is a total failure.

F1 = 2 × (Precision × Recall) / (Precision + Recall)

✅ 5. Accuracy

Accuracy = (TP + TN) / (TP + FP + FN + TN) = number of correct predictions / total predictions.

4.3 Evaluation Metrics for Regression

📐 1. MAE — Mean Absolute Error

Sum of absolute differences between predictions and actual values. MAE = 0 means perfect predictions.

📐 2. MSE — Mean Square Error

The average of squared distances between the target and predicted values. Most commonly used metric for regression. Penalises larger errors more.

📐 3. RMSE — Root Mean Square Error

Standard deviation of the residuals (prediction errors). Preferred over MSE because it's in the same units as the target variable — easier to interpret.

4.4 Practical 1 — MSE & RMSE in MS Excel

Predicted vs Actual values for 10 data points:

4.5 Practical 2 — Classification Metrics from a Confusion Matrix

Given: TP = 35, TN = 50, FP = 10, FN = 5.

• Precision = 35 / (35 + 10) = 35/45 = 77.8 %.
• Recall = 35 / (35 + 5) = 35/40 = 87.5 %.
• F1 Score = 2 × (0.778 × 0.875) / (0.778 + 0.875) ≈ 82.3 %.
• Accuracy = (35 + 50) / (35 + 10 + 5 + 50) = 85/100 = 85 %.

4.6 Practical 3 — Python Code to Evaluate a Model

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import mean_squared_error
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

df = pd.read_csv('Salary_Data.csv', sep=',')
print(df.head())
print(df.shape)           # (30, 2)
print(df.isnull().sum())  # no nulls

# Data Preparation
X = np.array(df['YearsExperience']).reshape(-1, 1)
Y = np.array(df['Salary']).reshape(-1, 1)
print(X.shape, Y.shape)

# Train-Test Split
X_train, x_test, Y_train, y_test = train_test_split(
    X, Y, test_size=0.2, shuffle=True, random_state=10)

# Fit the model
model = LinearRegression()
model.fit(X_train, Y_train)

# Score
print("Train R²:", model.score(X_train, Y_train))
print("Test  R²:", model.score(x_test,  y_test))

# Mean Squared Error
y_pred = model.predict(x_test)
print("Mean Squared Error:", mean_squared_error(y_test, y_pred))