Chapter 1 Machine Learning Overview

STAT 432 is a course about machine learning? Let’s try to define machine learning.

1.1 Reading

1.2 What is Machine Learning?

Machine learning (ML) is about learning functions from data. That’s it. Really. Pretty boring, right?

To quickly address some buzzwords that come up when discussing machine learning:

  • Deep learning is just a subset of machine learning.
  • Artificial Intelligence (AI) overlaps machine learning but has much loftier goals. In general, if someone claims to be using AI, they are not. (They’re probably using function learning! For example, we will learn logistic regression in this course. People in marketing might call that AI! Someone who understands ML will simply call it function learning. Don’t buy the hype! We don’t need to call simple methods AI to make them effective.)
  • Machine learning is not data science. Data science sometimes uses machine learning.
  • Does big data exist? If it does, I would bet a lot of money that you haven’t seen it, and probably won’t see it that often.
  • Analytics is just a fancy word for doing data analysis. Machine learning can be used in analyses! When it is, it is often called “Predictive Analytics.”

What makes machine learning interesting are the uses of these functions. We could develop functions that have applications in a wide variety of fields.

In medicine, we could develop a function that helps detect skin cancer.

  • Input: Pixels from an image of mole
  • Output: A probability of skin cancer

In sport analytics, we could develop a function that helps determine player salary.

  • Input: Lifetime statistics of an NBA player
  • Output: An estimate of player’s salary

In meteorology, we could develop a function to predict the weather.

  • Input: Historic weather data in Champaign, IL
  • Output: A probability of rain tomorrow in Champaign, IL

In political science we could develop a function that predicts the mood of the president.

  • Input: The text of a tweet from President Donald Trump
  • Output: A prediction of Donald’s mood (happy, sad, angry, etc)

In urban planning we could develop a function that predicts the rental prices of Airbnbs.

  • Input: The attributes of the location for rent
  • Output: An estimate of the rent of the property

How do we learn these functions? By looking at many previous examples, that is, data! Again, we will learn functions from data. That’s what we’re here to do.

1.3 Machine Learning Tasks

When doing machine learning, we will classify our tasks into one of two categories, supervised or unsupervised learning. (There are technically other tasks such as reinforcement learning and semi-supervised learning, but they are outside the scope of this text. To understand these advanced tasks, you should first learn the basics!) Within these two broad categories of ML tasks, we will define some specifics.

1.3.1 Supervised Learning

In supervised learning, we want to “predict” a specific response variable. (The response variable might also be called the target or outcome variable.) In the following examples, this is the y variable. Supervised learning tasks are called regression if the response variable is numeric. If a supervised learning tasks has a categorical response, it is called classification.

1.3.1.1 Regression

In the regression task, we want to predict numeric response variables. The predictor variables, which we will call the feature variables, or simply features can be either categorical or numeric.

x1 x2 x3 y
A -0.66 0.48 14.09
A 1.55 0.97 2.92
A -1.19 -0.81 15.00
A 0.15 0.28 9.29
B -1.09 -0.16 17.57
B 1.61 1.94 2.12
B 0.04 1.72 8.92
A 1.31 0.36 4.40
C 0.98 0.30 4.40
C 0.88 -0.39 4.52

With the data above, our goal would be to learn a function that takes as input values of the three features (x1, x2, and x3) and returns a prediction (best guess) for the true (but usually unknown) value of the response y. For example, we could obtain some “new” data that does not contain the response.

x1 x2 x3
B -0.85 -2.41

We would then pass this data to our function, which would return a prediction of the value of y. Stated mathematically, our prediction will often be an estimate the conditional mean of \(Y\), given values of the \(\boldsymbol{X}\) variables.

\[ \mu(\boldsymbol{x}) = \mathbb{E}[Y \mid \boldsymbol{X} = \boldsymbol{x}] \]

In other words, we want to learn this function, \(\mu(\boldsymbol{x})\). Much more on this later. (You can safely ignore this for now.)

1.3.1.2 Classification

Classification is similar to regression, except it considers categorical response variables.

x1 x2 x3 y
Q 0.46 5.42 B
Q 0.72 0.83 C
Q 0.93 5.93 B
Q 0.26 5.68 A
P 0.46 0.49 B
P 0.94 3.09 B
P 0.98 2.34 C
P 0.12 5.43 C
Q 0.47 2.68 B
P 0.56 5.02 B

As before we want to learn a function from this data using the same inputs, except this time, we want it to output one of A, B, or C for predictions of the y variable. Again, consider some new data:

x1 x2 x3
P 0.96 5.33

While ultimately we would like our function to return one of A, B, or C, what we actually would like is an intermediate return of probabilities that y is A, B, or C. In other words, we are attempting to estimate the conditional probability that \(Y\) is each of the possible categories, given values of the \(\boldsymbol{X}\) values.

\[ p_k(\boldsymbol{x}) = P\left[ Y = k \mid \boldsymbol{X} = \boldsymbol{x} \right] \]

We want to learn this function, \(p_k(\boldsymbol{x})\). Much more on this later. (You can safely ignore this for now.)

1.3.2 Unsupervised Learning

Unsupervised learning is a very broad task that is rather difficult to define. Essentially, it is learning without a response variable. To get a better idea about what unsupervised learning is, consider some specific tasks.

x1 x2 x3 x4 x5
2.74 0.46 5.42 4.43 2.28
2.81 0.72 0.83 4.87 2.61
0.86 0.93 5.93 2.33 0.22
2.49 0.26 5.68 4.11 5.84
1.93 0.46 0.49 0.02 2.59
-8.44 -9.06 -6.91 -5.00 -4.25
-7.79 -9.02 -7.66 -9.96 -4.67
-9.60 -9.88 -4.57 -8.75 -6.16
-8.03 -9.53 -7.32 -4.56 -4.17
-7.88 -9.44 -4.98 -6.33 -6.29

1.3.2.1 Clustering

Clustering is essentially the task of grouping the observations of a dataset. In the above data, can you see an obvious grouping? (Hint: Compare the first five observations to the second five observations.) In general, we try to group observations that are similar.

1.3.2.2 Density Estimation

Density estimation tries to do exactly what the name implies, estimate the density. In this case, the joint density of \(X_1, X_2, X_3, X_4, X_5\). In other words, we would like to learn the function that generated this data. (You could take the position that this is the only machine learning tasks, and all other tasks are subset of this task. We’ll hold off on explaining this for a while.)

1.3.2.3 Outlier Detection

Consider some new data:

x1 x2 x3 x4 x5
67 66.68 66.26 69.49 70

Was this data generated by the same process as the data above? If no, we would call it an outlier.

1.4 Open Questions

The two previous sections were probably more confusing than helpful. But of course, because we haven’t started learning yet! Hopefully, you are currently pondering one very specific question:

  • How do we learn functions from data?

That’s what this text will be about! We will spend a lot of time on this question. It is what us statisticians call fitting a model. On some level the answer is: look at a bunch of old data before predicting on new data.

While we will dedicate a large amount of time to answering this question, sometimes, some of the details might go unanswered. Since this is an introductory text, we can only go so far. However, as long as we answer another question, this will be OK.

  • How do we evaluate how well learned functions work?

This text places a high priority on being able to do machine learning, specifically do machine learning in R. You can actually do a lot of machine learning without fully understanding how the learning is taking place. That makes the evaluation of ML models extremely important.

1.5 Source