WK2: Artificial Neural Networks I: Fundamentals

• Artificial Intelligence Programming
• Higher Education (degree)
• Upper Secondary (Key Stage 4)

Welcome to Week 2

Artificial Neural Networks I: Fundamentals

Module Lecturer: Dr Raghav Kovvuri

Email: [e-mail]

Introduction to ANN

Definition: Computational models inspired by biological neural networks

Key characteristics:

• Parallel processing
• Adaptive learning
• Distributed representation

Historical context: From perceptron (1958) to 

deep learning (present)

Introduction : The City of Neurotopia

• Welcome to Neurotopia, a unique city that represents our Artificial Neural Network (ANN). As we explore this city, we'll uncover the fundamental concepts of ANNs.

• Analogy: Neurotopia is a living, learning city that processes information collectively to make decisions.

Historical Context

• 1943: McCulloch-Pitts neuron
• 1958: Rosenblatt's Perceptron
• 1969: Minsky and Papert's limitations of single-layer networks
• 1986: Rumelhart, Hinton, and Williams - Backpropagation
• 2012 onwards: Deep Learning revolution

• Analogy: Neurotopia wasn't built in a day. Let's explore its evolution from a simple village to a complex metropolis.

Biological Inspiration

Structure of biological neurons:

• Dendrites (receive signals)
• Cell body (process signals)
• Axon (transmit signals) 

Dendrites

Cell body

• Synaptic transmission

• Analogy: Citizens of Neurotopia (neurons) communicate through an elaborate postal system (synapses).

Axon

Synapse

Junction between two neurons that allows a signal to pass between them

Which part of a biological neuron is most similar to the output of an artificial neuron?

1. Dendrites
2. Cell body
3. Axon
4. Synapses

The Artificial Neuron

Components of a Artificial neuron: Neurotopia citizen

1. Inputs (x₁, x₂, ..., xₙ) - Information received
2. Weights (w₁, w₂, ..., wₙ) - Importance of each input
3. Bias (b) - Personal opinion
4. Summation function (Σ) - Combining all inputs
5. Activation function (f) - Decision to pass on information

Mathematical representation: 

y=f(​i​∑​​w​i​​x​i​​+b)

Artificial Neuron vs Perceptron

 Activation Functions

Types of Activation Functions:

Definition: Activation functions are mathematical operations applied to the weighted sum of inputs in a neuron, introducing non-linearity into the network's output

Purpose: Introduce non-linearity, allowing networks to learn complex patterns

1. Step Function: Binary decision (Yes/No)
2. Sigmoid Function:  Gradual transition (0 to 1)
3. Hyperbolic Tangent (tanh): Gradual transition (0 to 1)
4. ReLU (Rectified Linear Unit)

f(x)= ​1+e​−x​​​​1​​

f(x)= ​e​x​​+e​−x​​​​e​x​​−e​−x​​​​

f(x)=max(0,x)

• Analogy: Activation functions are like the citizens' "mood" influencing their decision to share information.

Extended Analogy

1. Step Function Councilor: Makes binary decisions. "If the majority favors it, I vote yes. Otherwise, it's a no."
2. Sigmoid Function Councilor: Considers all perspectives gradually. "I'll weigh all the information and give a nuanced opinion between 0 and 1."
3. Tanh Function Councilor: Similar to Sigmoid, but more decisive. "I'll consider both sides equally and can strongly agree (+1) or disagree (-1)."
4. ReLU Function Councilor: Focuses only on positive aspects. "I'll support good ideas with full enthusiasm, but I won't consider negative aspects at all 

The Neuron City Council:

Imagine a city council (neural network) making decisions. Each council member (neuron) receives various pieces of information (inputs) from citizens. The activation function represents how each council member processes this information before voting

Which activation function would be most suitable for a neural network tasked with sentiment analysis of movie reviews (classifying as positive or negative)?

1. Step Function
2. Sigmoid Function
3. ReLU (Rectified Linear Unit)
4. Hyperbolic Tangent (tanh)

Network Architecture

Perform intermediate computations by extracting patterns and features from the data.

Produces the final result or prediction based on the learned patterns.

Receives raw data to be processed.

 Artificial Neuron

• Input Layer: Information Gathering District
• Hidden Layer(s): Information Processing Neighborhoods
• Output Layer: Decision-Making Center

Types of NN

• Feedforward NN: Information flows from input to output
• Recurrent NN: Some information loops back (like city planning meetings)

Activity (1)

• Objective: Research and understand the differences between supervised, unsupervised, and reinforcement learning.
• Search for definitions, key differences, and real-world examples of each type of learning.
• Compare their learning processes, types of algorithms used, and applications.
• Post their findings in the Discussion Section of Canvas for this activity.

Research Task - Supervised Learning vs Unsupervised Learning vs Reinforcement Learning

Activity (2)

Research and Programming Task - Classification vs Clustering (20 min)

Download Classification1.py and Clustering.py from Canvas

Activity (3)

Research and Programming Task - Classification vs Regression (20 min)

Download Classification2.py and Regression.py from canvas