~/Data Literacy — Data Collection to Data Analysis

1.1 What is Data Literacy?

1.2 Data Collection

Data collection means pooling data by scraping, capturing or loading it from multiple offline + online sources. Key guidelines:

• Quantity depends on the number of features and model complexity. A license-plate detector needs less data; a medical AI needs huge volumes.
• Diversity matters more than raw quantity — diverse data helps the model cover more scenarios.
• Start with small batches, then scale up.
• Collection is iterative — as the project develops, data requirements may change.

Two Sources of Data

Primary Data Collection Methods

Secondary Data Collection Methods

1.3 Exploring Data — Levels of Measurement

1.4 Statistical Analysis — Measures of Central Tendency

Python statistics Library — Quick Reference

import statistics

statistics.mean(data)        # arithmetic mean
statistics.median(data)      # median (middle value)
statistics.mode(data)        # most frequent value
statistics.variance(data)    # variance
statistics.stdev(data)       # standard deviation

📏 Mean (Arithmetic Average)

where M = mean, Σ = sum, f = frequency, x = score, n = total cases.

📐 Median (Middle Value)

Arrange data in ascending / descending order — the middle value is the median. For 13 values (odd), the 7th is the median.

🏔️ Mode (Most Frequent Value)

When to Use Mean / Median / Mode?

Program 1 — Mean / Median / Mode of student heights

import statistics

heights = [145, 151, 152, 149, 147, 152, 151, 149,
           152, 151, 147, 148, 155, 147, 152, 151,
           149, 145, 147, 152, 146, 148, 150, 152, 151]

print("Mean   =", statistics.mean(heights))
print("Median =", statistics.median(heights))
print("Mode   =", statistics.mode(heights))

1.5 Statistical Analysis — Measures of Dispersion

Central tendency tells us the centre. Dispersion tells us how spread out the data is around that centre. The two key measures are Variance and Standard Deviation.

Interpreting Variance / Std Deviation

Program 2 — Variance & Standard Deviation

import statistics

heights = [145, 151, 152, 149, 147, 152, 151, 149,
           152, 151, 147, 148, 155, 147, 152, 151,
           149, 145, 147, 152, 146, 148, 150, 152, 151]

print("Variance  =", statistics.variance(heights))
print("Std. Dev. =", statistics.stdev(heights))

1.6 Data Representation & Visualisation

Data representation is classified into two categories:

Matplotlib — Python's Plotting Library

pip install matplotlib
# or:
python -m pip install -U matplotlib

Import and use:

import matplotlib.pyplot as plt

Common pyplot Functions

1.7 Five Chart Types

A. Line Graph — for continuous data / trends over time

Shows trends and changes. The line slopes up (increase) or down (decrease). Function: plt.plot().

import matplotlib.pyplot as plt

tests = ["T1", "T2", "T3", "T4", "T5"]
marks = [25, 34, 49, 40, 48]

plt.plot(tests, marks, marker="o", color="blue")
plt.title("Kavya's AI marks")
plt.xlabel("Test")
plt.ylabel("Marks")
plt.show()

B. Bar Graph — for comparison between categories

Rectangular bars with heights proportional to values. Function: plt.bar().

schools = ["Oxford", "Delhi", "Jyothis", "Sanskriti", "Bombay"]
students = [123, 87, 105, 146, 34]

plt.bar(schools, students, color="green", edgecolor="black")
plt.title("Deep Learning Seminar Attendance")
plt.show()

C. Histogram — for frequency distribution

Vertical rectangles showing frequencies of value ranges (bins). One distribution per axis. Function: plt.hist().

heights = [141, 145, 142, 147, 144, 148, 141,
           142, 149, 144, 143, 149, 146, 141,
           147, 142, 143]

plt.hist(heights, bins=5, color="skyblue", edgecolor="black")
plt.title("Height distribution — Class XII girls")
plt.show()

D. Scatter Graph — for relationships between two variables

Plots data points on x and y axes to reveal correlations, clusters, trends. Function: plt.scatter().

study_hours = [2, 4, 5, 7, 8, 10]
math_score = [55, 65, 70, 80, 85, 95]

plt.scatter(study_hours, math_score, color="red")
plt.title("Study Hours vs Math Score")
plt.xlabel("Hours per week")
plt.ylabel("Math %")
plt.show()

E. Pie Chart — for proportions / composition

Circular graph divided into segments representing percentages. Limit to ~7 categories for clarity. Cannot show zero values or changes over time. Function: plt.pie().

subjects = ["English", "Maths", "Science", "Social", "AI", "PE"]
periods = [6, 8, 8, 7, 3, 2]

plt.pie(periods, labels=subjects, autopct="%.1f%%")
plt.title("Weekly periods per subject")
plt.show()

🌧️ Practical — Using rainfall.csv

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("rainfall.csv")

# Line chart
plt.plot(df["Month"], df["Rainfall"], marker="o")
plt.title("Monthly Rainfall - Tamil Nadu")
plt.show()

# Bar chart
plt.bar(df["Month"], df["Rainfall"], color="teal")
plt.show()

# Pie chart
plt.pie(df["Rainfall"], labels=df["Month"], autopct="%.1f%%")
plt.show()

1.8 Introduction to Matrices

Example — Purchases Table

Order of a Matrix

A matrix with m rows and n columns has order m × n. Total elements = m × n. Each element is written as a_ij where i = row, j = column.

Three Matrix Operations

Applications of Matrices in AI

1.9 Data Pre-processing

Five Stages of Pre-processing

Data Cleaning — Four Common Issues

1.10 Data in Modelling & Evaluation

Train–Test Split

After pre-processing, data is split into:

• Training set — used to train the model (usually 80%).
• Testing set — used to evaluate the trained model (usually 20%).

Validation Techniques

Evaluation Metrics

1.11 Practical Programs (Syllabus)

Program A — Mean / Median / Mode of ages

import statistics

ages = [25, 28, 30, 35, 40, 45, 50, 55, 60, 65]

print("Mean   =", statistics.mean(ages))
print("Median =", statistics.median(ages))
# Note: mode fails if all values unique
print("Mode   =", statistics.mode(ages))

Program B — Variance / SD of temperatures

import statistics

temps = [20, 22, 25, 18, 23]
print("Variance  =", statistics.variance(temps))
print("Std. Dev. =", statistics.stdev(temps))

Program C — Line chart (weekly inquiries)

import matplotlib.pyplot as plt

weeks = ["Week 1", "Week 2", "Week 3", "Week 4"]
inquiries = [150, 170, 180, 200]

plt.plot(weeks, inquiries, marker="o", color="purple")
plt.title("Customer Inquiries Per Week")
plt.ylabel("Number of inquiries")
plt.show()

Program D — Bar chart (book sales by genre)

genres = ["Fiction", "Mystery", "Sci-Fi", "Romance", "Biography"]
sales  = [120, 90, 80, 110, 70]

plt.bar(genres, sales, color="orange", edgecolor="black")
plt.title("Books Sold by Genre")
plt.show()

Program E — Pie chart (city transportation)

modes = ["Car", "Public Transit", "Walking", "Bicycle"]
shares = [40, 30, 20, 10]

plt.pie(shares, labels=modes, autopct="%.1f%%")
plt.title("How people commute in the city")
plt.show()

1.12 Certification (Syllabus)