~/Python Programming – II

Introduction — What Will We Learn?

• Review the basics of the NumPy and Pandas libraries — arrays, Series, DataFrames and essential functions.
• Efficiently import and export data between CSV files and Pandas DataFrames.
• Detect and handle missing values in a dataset.
• (Advanced) Implement a Linear Regression algorithm including data preparation and model training.

Prerequisite: Foundational understanding of Python from Class XI and familiarity with basic programming.

1.1 Python Libraries

In data science and analytics, two libraries stand out — NumPy and Pandas. They form the backbone of data manipulation and analysis in Python, enabling users to handle large datasets with ease and precision.

1.2 NumPy — Numerical Python

In NumPy, the number of dimensions of the array is called the rank of the array.

🔹 Quick NumPy Recap (from Class XI)

• Creation: np.array([1, 2, 3, 4]), np.zeros(5), np.ones((2,3)), np.arange(10), np.linspace(0,1,5).
• Shape & rank: arr.shape, arr.ndim, arr.size, arr.dtype.
• Arithmetic: element-wise + − * / between arrays of same shape.
• Statistics: np.mean, np.median, np.std, np.var, np.min, np.max, np.sum.
• Indexing / slicing: arr[0], arr[1:4], arr[:, 0] (column 0 of a 2-D array).
• NaN: np.NaN represents a missing numeric value.

import numpy as np
a = np.array([90, 100, 110, 120])
print(a.shape)   # (4,)
print(a.mean())  # 105.0

1.3 Pandas — Why and Where We Use It

Suppose we have a dataset of marketing campaigns — campaign type, budget, duration, reach, engagement metrics and sales performance. Pandas is used to:

• Load the dataset.
• Display summary statistics.
• Perform group-wise analysis to understand campaign performance.
• Pair with Matplotlib for visualising sales and average engagement per campaign type.

This capability is invaluable in AI and data-driven decision-making — it lets businesses gain actionable insights from their data.

1.4 Pandas Data Structures — Series & DataFrame

Pandas provides two main data structures for manipulating data:

• Series — a one-dimensional labelled array (a single column of data with an index).
• DataFrame — a two-dimensional labelled table (rows + columns, like a spreadsheet).

🧩 1. Creating a Series from Scalar Values

import pandas as pd
s = pd.Series([10, 20, 30, 40], index=['a','b','c','d'])
print(s)

📐 2. Creating a DataFrame from NumPy Arrays

import numpy as np
import pandas as pd
array1 = np.array([90, 100, 110, 120])
array2 = np.array([50, 60, 70])
array3 = np.array([10, 20, 30, 40])
marksDF = pd.DataFrame([array1, array2, array3], columns=['A','B','C','D'])
print(marksDF)

🗃️ 3. Creating a DataFrame from a Dictionary of Arrays / Lists

import pandas as pd
data = {'Name':['Varun','Ganesh','Joseph','Abdul','Reena'],
        'Age' :[37, 30, 38, 39, 40]}
df = pd.DataFrame(data)
print(df)

The dictionary keys become column labels and the lists become the column data.

📋 4. Creating a DataFrame from a List of Dictionaries

listDict = [{'a':10, 'b':20},
            {'a':5,  'b':10, 'c':20}]
a = pd.DataFrame(listDict)
print(a)

There will be as many rows as dictionaries in the list. Missing keys become NaN.

1.5 Dealing with Rows and Columns

➕ 1. Adding a New Column to a DataFrame

ResultSheet = {
  'Rajat'    : pd.Series([90,91,97], index=['Maths','Science','Hindi']),
  'Amrita'   : pd.Series([92,81,96], index=['Maths','Science','Hindi']),
  'Meenakshi': pd.Series([89,91,88], index=['Maths','Science','Hindi']),
  'Rose'     : pd.Series([81,71,67], index=['Maths','Science','Hindi']),
  'Karthika' : pd.Series([94,95,99], index=['Maths','Science','Hindi'])
}
Result = pd.DataFrame(ResultSheet)
Result['Fathima'] = [89, 78, 76]       # new column
print(Result)

➕ 2. Adding a New Row Using .loc[]

Result.loc['English'] = [90, 92, 89, 80, 90, 88]
print(Result)

✏️ 3. Updating a Row with .loc[]

Result.loc['Science'] = [92, 84, 90, 72, 96, 88]   # change Science marks
print(Result)

1.6 Deleting Rows or Columns — .drop()

• axis = 0 → delete a row.
• axis = 1 → delete a column.

Result = Result.drop('Hindi', axis=0)                       # delete row 'Hindi'
Result = Result.drop(['Rajat','Meenakshi','Karthika'], axis=1)  # delete columns
print(Result)

1.7 Attributes of DataFrames

import pandas as pd
dict = {
  "Student": pd.Series(["Arnav","Neha","Priya","Rahul"],
                       index=["Data 1","Data 2","Data 3","Data 4"]),
  "Marks"  : pd.Series([85, 92, 78, 83],
                       index=["Data 1","Data 2","Data 3","Data 4"]),
  "Sports" : pd.Series(["Cricket","Volleyball","Hockey","Badminton"],
                       index=["Data 1","Data 2","Data 3","Data 4"])
}
df = pd.DataFrame(dict)
print(df)

2.1 What Is a CSV File?

In Python, CSV files are fundamental for data analysis. Pandas provides powerful tools to read CSV files into DataFrames — the go-to format for data scientists.

2.2 Importing a CSV File to a DataFrame — read_csv()

Use pd.read_csv("filename.csv") to import tabular data from a CSV into a Pandas DataFrame.

☁️ 1. Using Google Colab

1. Open Google Colab at colab.research.google.com → File → New notebook.
2. Click the Folder icon on the left sidebar.
3. Click the ↑ Upload button → select the CSV file (e.g., studentsmarks.csv).
4. Execute the code:

   import pandas as pd
   df = pd.read_csv("studentsmarks.csv")
   print(df)

💻 2. Using a Local Python IDE

Give the complete path of the CSV file:

import pandas as pd
import io
df = pd.read_csv('C:/PANDAS/studentsmarks.csv', sep=",", header=0)
print(df)

2.3 Exporting a DataFrame to a CSV File — to_csv()

Use df.to_csv() to save a DataFrame to a text or CSV file.

💾 On Local Python IDE

df.to_csv(path_or_buf='C:/PANDAS/resultout.csv', sep=',')

This creates resultout.csv on the hard disk. Opening it in any text editor or spreadsheet shows the data with row labels and column headers separated by commas.

☁️ On Google Colab

df.to_csv("resultout.csv", index=False)

The index=False argument prevents the row index from being written into the CSV.

2.4 Handling Missing Values

The two most common strategies to handle missing values are:

1. Drop the row having missing values.
2. Estimate (fill) the missing value.

🔍 1. Checking Missing Values — isnull()

Pandas provides the isnull() function to check whether any value is missing. It returns True if the attribute has missing values, else False.

🗑️ 2. Drop Missing Values — dropna()

Dropping removes the entire row (object) that has missing value(s). This reduces the size of the dataset, so use it only when missing values appear on few rows.

drop = marks.dropna()
print(drop)

🔢 3. Estimate the Missing Value — fillna()

Missing values can be filled with estimations — the value just before/after, the average/minimum/maximum of that attribute, or simply 0 or 1.

FillZero = marks.fillna(0)   # replace missing with 0
print(FillZero)
FillOne  = marks.fillna(1)   # replace missing with 1

2.5 Case Study — Student Marks with Missing Values

Meera and Suhana couldn't attend Science and Hindi exams (fever). Joseph attended a national-level science exhibition on AI exam day. Their marks are missing.

import pandas as pd
import numpy as np

ResultSheet = {
  'Maths'  : pd.Series([90,91,97,89,65,93],
                       index=['Heena','Shefali','Meera','Joseph','Suhana','Bismeet']),
  'Science': pd.Series([92,81,np.NaN,87,50,88],
                       index=['Heena','Shefali','Meera','Joseph','Suhana','Bismeet']),
  'English': pd.Series([89,91,88,78,77,82],
                       index=['Heena','Shefali','Meera','Joseph','Suhana','Bismeet']),
  'Hindi'  : pd.Series([81,71,67,82,np.NaN,89],
                       index=['Heena','Shefali','Meera','Joseph','Suhana','Bismeet']),
  'AI'     : pd.Series([94,95,99,np.NaN,96,99],
                       index=['Heena','Shefali','Meera','Joseph','Suhana','Bismeet'])
}
marks = pd.DataFrame(ResultSheet)
print(marks)

🔹 Check for Missing Values

>>> print(marks.isnull())
# shows True/False table; three True values → three missing pieces
>>> print(marks['Science'].isnull().any())
True
>>> print(marks['Maths'].isnull().any())
False
>>> marks.isnull().sum().sum()    # total NaN in whole dataset
3

🔹 Apply dropna() and fillna()

# Option 1 — Drop rows with NaN
drop = marks.dropna()
print(drop)

# Option 2 — Fill NaN with 0
FillZero = marks.fillna(0)
print(FillZero)

3.1 Linear Regression — What & Why

🔹 Typical Linear Regression Workflow

1. Load the dataset (with Pandas).
2. Perform Exploratory Data Analysis (EDA) — head, info, describe, plots.
3. Split data into features (X) and target (y).
4. Split into training set (80%) and testing set (20%).
5. Train the model using sklearn.linear_model.LinearRegression.
6. Predict on the test set.
7. Evaluate actual vs predicted values.

3.2 Practical Activity — USA Housing Dataset

Dataset (USA Housing) available from the CBSE Handbook's Google-Drive link.

📥 1. Load & Peek

import pandas as pd
df = pd.read_csv('USA_Housing.csv')
df.head()
df.info()
df.describe()

Examining the count reveals that all columns contain 5000 values → no missing values anywhere.

🔍 2. Exploratory Data Analysis

Use df.corr() for correlations, and Matplotlib/Seaborn for scatter-plots between each feature and the target (Price). Strong linear trends indicate good predictors.

📊 3. Train–Test Split (80 : 20)

from sklearn.model_selection import train_test_split

X = df[['Avg. Area Income','Avg. Area House Age','Avg. Area Number of Rooms',
        'Avg. Area Number of Bedrooms','Area Population']]
y = df['Price']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# 4000 rows for training (80% of 5000) · 1000 rows for testing

🤖 4. Apply Linear Regression

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)

print("Intercept:", model.intercept_)
print("Coefficients:", model.coef_)

📈 5. Predict & Compare

predictions = model.predict(X_test)

import matplotlib.pyplot as plt
plt.scatter(y_test, predictions)
plt.xlabel('Actual Price')
plt.ylabel('Predicted Price')
plt.title('Actual vs Predicted')
plt.show()

We observe there is a difference between actual and predicted values. The next chapter (Data Science Methodology · Unit 2) covers how to calculate the error, evaluate the model and test its accuracy.