Activation functions play a crucial role in artificial neural networks by introducing non-linearity into the network, allowing it to learn complex patterns and relationships in data. Here’s a comprehensive overview of activation functions:
1. Purpose:
a. Non-Linearity:
- Introduce non-linearities to the model, enabling it to learn complex mappings from inputs to outputs.
b. Enable Learning:
- Facilitate the learning process by allowing the network to capture and represent intricate patterns in the data.
2. Common Activation Functions:
a. Sigmoid Function:
- Range: (0, 1)
- Use Case: Historically used in the output layer for binary classification problems.
- Issues: Vanishing gradient problem, output is not zero-centered.
b. Hyperbolic Tangent (tanh) Function:
- Range: (-1, 1)
- Use Case: Similar to the sigmoid but with a range from -1 to 1.
- Issues: Vanishing gradient problem.
