Linear Algebra - Introduction to Vectors

Introduction to Vectors

Real world uses of vectors

Vectors make up so much of our daily lives often without us even realising. One way you may have used vectors is in a physics class, where it’s referred to as velocity. And what is velocity? Well, it’s a way of describing speed and direction. And that’s the single, simple key to understanding vectors.

So what’s the difference between velocity and a vector?

• Velocity is a type of vector, specifically used to describe the speed at which something moves in a specific direction.
• A vector is a more general term for any quantity with magnitude and direction.

And what are some other real world uses?

Physics:

• Force | Vectors can represent a force, which has both magnitude (how strong the force is) and direction. For example, if you pushed an object then the force vector would indicate the strength of your push as well as the direction.
• Velocity | Simply the speed and direction of a moving object
• Displacement | A vector can indicate displacement by representing the change in position of an object from one point to another, showing the distance (magnitude) and direction.

Engineering:

• Analysis of Structures | Engineers can use vectors to calculate forces acting on structures to ensure they can withstand certain stresses in specific directions. An example of this could be a building or bridge.
• Electrical Engineering | Vectors can describe electric and magnetic fields, displaying the strength and direction of the fields.

Computer Graphics & Animation

• Movement and Positioning | Vectors are used to position objects on the screen and give them motion. For example, in 3D modelling, you will use vectors to determine the position and orientation of an object.
• Lighting and Shading | Vectors can help to calculate angles between light sources and surfaces to create realistic lighting and shading effects.

Data Science & Machine Learning

• Gradient Descent | In Data Science, Gradient Descent is an algorithm used to make predictions as accurate as possible. The Cost Function is a measure of how well a model’s predictions match the actual data. The Gradient Vector points in the direction of the steepest increase of the Cost Function, and the data models use this vector to go in the opposite direction and find the minimum Cost Function - which is the most accurate and errorless result. Vectors are also used to represent this movement towards optimisation.
• Feature Vectors | In Machine Learning, a feature vector is a different type of vector entirely. It has the same shape but it’s a different concept. So an example of a vector would be:

This example shows a vector of (5, 4). It has both a direction in this 2D space example as well as a magnitude (distance). The magnitude is the length of the vector, and is worked out using Pythagoras’ theorem. The magnitude (a) is calculated by doing a = $\sqrt{5^2 + 4^2}$. Essentially you’re getting the squared area of each plotted point and then the root of them combined to get the non-squared length from the origin to the destination.

You could have a feature vector of (5, 4) but the 5 could represent the number of rooms of a house, and the 4 could represent the age. So your feature vector for this specific example would be f = (5, 4). If your data set had many houses, you could represent each house with this feature vector and your machine learning model could use these vectors to learn patterns and make predictions.

Astronomy & Space Exploration

• Orbital Mechanics | Vectors are critical in calculating the trajectories of spacecraft as well as the forces acting on them.
• Stellar Navigation | Vectors help to determine the position and movement of stars, planets, comets, etc in relation to one another.

Recapping what is a vector?

A vector is a mathematical object that has both magnitude (size) and direction. They’re often visually represented as arrows, where the length is the magnitude and the direction of the arrow is the direction. In algebra, they’re typically represented as ordered lists of numbers (coordinates), such as in 2D space $\vec{v} = (x, y)$ and in 3D space $\vec{v} = (x, y, z)$.

A vector in the machine learning world (a feature vector) utilises the vector shape of being an ordered list of numbers but those numbers/features could represent things other than magnitude and direction.

Key Concepts in Vectors

What’s a scalar and how does it differ to a vector?

A scalar is a quantity that has only magnitude and no direction. Examples of scalars include temperature, mass, time, speed, etc.

You could use a scalar as part of your vector, such as the speed part of velocity.

What is a Unit Vector?

A unit vector is a vector that has a magnitude of exactly 1 unit and points in a specific direction. It is used to indicate direction without concern for magnitude. Unit vectors are often used as the building blocks for more complex vectors, allowing any vector to be expressed as a combination of unit vectors scaled by the vector’s components.

Properties of Unit Vectors:

• Magnitude: The magnitude (or length) of a unit vector is always 1.
• Direction: A unit vector points in a specific direction along one of the coordinate axes or in any other direction in space.
• Notation: Unit vectors are typically denoted with a “hat” symbol (^) above the letter, such as $\hat{i}$, $\hat{j}$, and $\hat{k}$, which represent unit vectors along the x-axis, y-axis, and z-axis, respectively.

Example:

In 2D space:

• $\hat{i}$ is the unit vector along the x-axis: $\hat{i} = (1, 0)$
• $\hat{j}$ is the unit vector along the y-axis: $\hat{j} = (0, 1)$

In 3D space:

• $\hat{i}$ (pronounced i-hat, or i) is the unit vector along the x-axis: $\hat{i} = (1, 0, 0)$
• $\hat{j}$ (pronounced j-hat, or j) is the unit vector along the y-axis: $\hat{j} = (0, 1, 0)$
• $\hat{k}$ (pronounced k-hat, or k) is the unit vector along the z-axis: $\hat{k} = (0, 0, 1)$

Normalizing a Vector:

Any vector can be converted into a unit vector (a process called “normalization”) by dividing the vector by its magnitude. For a vector $\vec{v} = (x, y, z)$, the unit vector $\hat{v}$ is calculated as: $\hat{v} = \frac{\vec{v}}{\|\vec{v}\|} = \left(\frac{x}{\|\vec{v}\|}, \frac{y}{\|\vec{v}\|}, \frac{z}{\|\vec{v}\|}\right)$ where $\|\vec{v}\|$ is the magnitude of the vector.

Unit vectors are essential in many areas of physics, engineering, and computer graphics, as they allow for the representation of direction independently of magnitude.

How are vectors represented?

A vector could be represented in several different ways, depending on the context:

• Arrows. The length represents the magnitude and the direction is the direction
• Coordinates. In 2D space it’s a simple pair such as (x, y). In 3D space it’s a triplet (x, y, z).
• Algebraic notation. Often expressed as unit vectors and their corresponding components. $\vec{v}$ is the symbol for the vector. $\hat{i}$, $\hat{j}$, $\hat{k}$ are unit vectors along the x, y, and z axes.
• Columns or row matrices. A vector in 2D can be written down as a column matrix, such as $\vec{v} = \begin{pmatrix} x \\ y \end{pmatrix}$

Vector Addition and Subtraction

Addition

Vector addition involves combining two or more vectors to produce a resultant vector. The process is performed component-wise, meaning you add the corresponding components (x with x, y with y, etc.) of the vectors.

You can visualise vector addition as being “tip-to-tail”. The origin (or tail) of the second vector starts on the tip of the first vector. The resultant vector is drawn from the tip of the first to the tip of the last.

For example, if $\vec{v} = \begin{pmatrix} 3 \\ 2 \end{pmatrix}$ and $\vec{w} = \begin{pmatrix} 1 \\ 4 \end{pmatrix}$, then:

$\vec{v} + \vec{w} = \begin{pmatrix} 3 + 1 \\ 2 + 4 \end{pmatrix} = \begin{pmatrix} 4 \\ 6 \end{pmatrix}$

This resulting vector $\vec{v} + \vec{w}$ has a component of 4 along the x-axis and 6 along the y-axis.

Subtraction

Vector subtraction is similar to addition but involves reversing the direction of the vector being subtracted before adding it. The result is the vector that “connects” the end of the vector being subtracted to the start of the other vector.

For example, if $\vec{v} = \begin{pmatrix} 5 \\ 7 \end{pmatrix}$ and $\vec{w} = \begin{pmatrix} 2 \\ 3 \end{pmatrix}$, then:

$\vec{v} - \vec{w} = \begin{pmatrix} 5 - 2 \\ 7 - 3 \end{pmatrix} = \begin{pmatrix} 3 \\ 4 \end{pmatrix}$

This resulting vector $\vec{v} - \vec{w}$ has a component of 3 along the x-axis and 4 along the y-axis.

Mathematical Operations with Vectors

Dot Product

The dot product (or scalar product) is a mathematical operation that takes two vectors and returns a scalar (a single number). The dot product is calculated by multiplying corresponding components of the vectors and then summing these products. It is often used to determine the angle between two vectors or to find projections of one vector onto another.

• Formula: The dot product of two vectors $\vec{a}$ and $\vec{b}$ is given by: $\vec{a} \cdot \vec{b} = a_x b_x + a_y b_y + a_z b_z$
• Geometric Interpretation: The dot product can also be expressed as $\vec{a} \cdot \vec{b} = \|\vec{a}\| \|\vec{b}\| \cos\theta$, where $\theta$ is the angle between the two vectors.
• Applications: The dot product is used to determine whether two vectors are perpendicular (dot product is zero) or to calculate the work done by a force.

Cross Product

Summary: The cross product (or vector product) is an operation on two vectors in three-dimensional space that results in a new vector perpendicular to both original vectors. The magnitude of the cross product is proportional to the area of the parallelogram formed by the two vectors, and its direction is given by the right-hand rule.

Tips:

• Formula: The cross product of two vectors $\vec{a}$ and $\vec{b}$ is given by: $\vec{a} \times \vec{b} = \begin{vmatrix} \hat{i} & \hat{j} & \hat{k} \\ a_x & a_y & a_z \\ b_x & b_y & b_z \end{vmatrix}$ This expands to: $\vec{a} \times \vec{b} = (a_yb_z - a_zb_y)\hat{i} - (a_xb_z - a_zb_x)\hat{j} + (a_xb_y - a_yb_x)\hat{k}$
• Geometric Interpretation: The cross product’s direction is perpendicular to both vectors, and its magnitude is related to the area of the parallelogram formed by the vectors.
• Applications: The cross product is used in physics to find torque or the magnetic force on a charged particle.

Vector Scaling

Summary: Vector scaling involves multiplying a vector by a scalar, which changes the magnitude of the vector but not its direction (unless the scalar is negative, which also reverses the direction). This operation is fundamental for resizing vectors, adjusting their magnitude according to a scalar quantity.

Tips:

• Formula: Scaling a vector $\vec{v}$ by a scalar $k$ results in: $k\vec{v} = \begin{pmatrix} k \cdot v_x \\ k \cdot v_y \end{pmatrix}$ In 3D, this extends to: $k\vec{v} = \begin{pmatrix} k \cdot v_x \\ k \cdot v_y \\ k \cdot v_z \end{pmatrix}$
• Geometric Interpretation: Scaling stretches or shrinks a vector’s length while maintaining its direction.
• Applications: Vector scaling is used in computer graphics to resize objects, in physics to adjust force vectors, or in economics to scale vectors representing different financial scenarios.