# Classification

```{r setup, include = FALSE}
knitr::opts_chunk$set(echo = TRUE, fig.align = "center", cache = TRUE, autodep = TRUE)
```

This chapter continues our discussion of **supervised learning** by introducing the **classification** tasks. Like regression, we will focus on the conditional distribution of the response.

Specifically, we will discuss:

- The setup for the **classification** task.
- The **Bayes classifier** and **Bayes error**.
- Estimating **conditional probabilities**.
- Two simple **metrics** for the classification task.

This chapter is currently under construction. While it is being developed, the following links to the STAT 432 course notes.

- [**Notes:** Classification](files/classification.pdf)

## R Setup and Source

```{r packages, warning = FALSE, message = FALSE}
library(tibble)     # data frame printing
library(dplyr)      # data manipulation

library(knitr)      # creating tables
library(kableExtra) # styling tables
```

Additionally, objects from `ggplot2`, `GGally`, and `ISLR` are accessed. Recall that the [Welcome](index.html) chapter contains directions for installing all necessary packages for following along with the text. The R Markdown source is provided as some code, mostly for creating plots, has been suppressed from the rendered document that you are currently reading.

- **R Markdown Source:** [`classification.Rmd`](classification.Rmd)

## Data Setup

- TODO: Add data setup example.

## Mathematical Setup

## Example

```{r, echo = FALSE}
set.seed(1)
joint_probs = round(1:12 / sum(1:12), 2)
joint_probs = sample(joint_probs)
joint_dist = matrix(data  = joint_probs, nrow = 3, ncol = 4)
colnames(joint_dist) = c("$X = 1$", "$X = 2$", "$X = 3$", "$X = 4$")
rownames(joint_dist) = c("$Y = A$", "$Y = B$", "$Y = C$")
joint_dist %>%
  kable() %>%
  kable_styling("striped", full_width = FALSE) %>%
  column_spec(column = 1, bold = TRUE, background = "white", border_right = TRUE)
```

```{r, echo = FALSE}
# marginal distribution of Y
t(colSums(joint_dist)) %>% kable() %>% kable_styling(full_width = FALSE)
```

```{r, echo = FALSE}
# marginal distribution of X
t(rowSums(joint_dist)) %>% kable() %>% kable_styling(full_width = FALSE)
```

## Bayes Classifier

$$
p_k(x) = P\left[ Y = k \mid X = x \right]
$$

$$
C^B(x) = \underset{k \in \{1, 2, \ldots K\}}{\text{argmax}} P\left[ Y = k \mid X = x \right]
$$

**Warning:** The Bayes classifier should not be confused with a naive Bayes classifier. The Bayes classifier assumes that we know $P\left[ Y = k \mid X = x \right]$ which is almost never known in practice. A naive Bayes classifier is a method we will see later that learns a classifier from data.^[This author feels that the use of "Bayes" in "Bayes classifier" is actually confusing because you don't actually need to apply Bayes' theorem to state or understand the result. Meanwhile, Bayes' theorem is needed to understand the naive Bayes classifier. Oh well. Naming things is hard.]

***

### Bayes Error Rate

$$
1 - \mathbb{E}_X\left[ \underset{k}{\text{max}} \ P[Y = k \mid X = x] \right]
$$

## Building a Classifier

$$
\hat{p}_k(x) = \hat{P}\left[ Y = k \mid X = x \right]
$$

$$
\hat{C}(x) = \underset{k \in \{1, 2, \ldots K\}}{\text{argmax}} \hat{p}_k(x)
$$

- TODO: first estimation conditional distribution, then classify to label with highest probability
- TODO: do it in r with knn, tree, or glm / nnet
- TODO: note about estimating probabilities vs training a classifier

## Modeling

### Linear Models

- TODO: use `nnet::multinom`
    - in place of `glm()`? always?

### k-Nearest Neighbors

- TODO: use `caret::knn3()`

### Decision Trees

- TODO: use `rpart::rpart()`

## Classification Metrics

Like regression, classification metrics will depend on the learned function (in this case, the learned classifier $\hat{C}$) as well as an additional dataset that is being used to make classifications with the learned function.^[The metrics also implicitly depend on the dataset used to learn the classifier.] Like regression, this will make the notation for a "simple" metric like accuracy look significantly more complicated than it truly is, but it will be helpful to make the dependency on the datasets explicit.

### Misclassification

$$
\text{miclass}\left(\hat{C}_{\texttt{set_f}}, \mathcal{D}_{\texttt{set_D}} \right) = \frac{1}{n_{\texttt{set_D}}}\displaystyle\sum_{i \in {\texttt{set_D}}}^{} I\left(y_i \neq \hat{C}_{\texttt{set_f}}({x}_i)\right)
$$

```{r}
calc_misclass = function(actual, predicted) {
  mean(actual != predicted)
}
```

### Accuracy

$$
\text{accuracy}\left(\hat{C}_{\texttt{set_f}}, \mathcal{D}_{\texttt{set_D}} \right) = \frac{1}{n_{\texttt{set_D}}}\displaystyle\sum_{i \in {\texttt{set_D}}}^{} I\left(y_i = \hat{C}_{\texttt{set_f}}({x}_i)\right)
$$

```{r}
calc_accuracy = function(actual, predicted) {
  mean(actual == predicted)
}
```

Plugging in the appropriate dataset will allow for calculation of train, test, and validation metrics.















```{r}

```














```{r, eval = FALSE, echo = FALSE}
gen_data = function(n = 100) {
 x = sample(c(0, 1), prob = c(0.4, 0.6), size = n, replace = TRUE)
 y = ifelse(test = {x == 0},
            yes = sample(c("A", "B", "C"), size = n, prob = c(0.25, 0.50, 0.25), replace = TRUE),
            no = sample(c("A", "B", "C"), size = n, prob = c(0.1, 0.1, 0.4) / 0.6, replace = TRUE))
 tibble(x = x, y = factor(y))
}
test_cases = tibble(x = c(0, 1))
set.seed(42)
some_data = gen_data()
predict(knn3(y ~ x, data = some_data), test_cases)
predict(rpart(y ~ x, data = some_data), test_cases)
predict(multinom(y ~ x, data = some_data, trace = FALSE), test_cases, type = "prob")
```













```{r, eval = FALSE, echo = FALSE}
################################################################################

# load packages
library("tibble")
library("caret")
library("rpart")
library("nnet")
library("e1071")
library("mlbench")
library("purrr")
library("rsample")

################################################################################

# define data generating process (from previous class)
gen_data = function(n = 100) {
  # generate x
  x = sample(x = c(0, 1), 
             size = n, 
             prob = c(0.4, 0.6), 
             replace = TRUE)
  # generate y | x
  y = ifelse(test = {x == 0},
             yes = sample(x = c("A", "B", "C"), 
                          size = n, 
                          prob = c(0.25, 0.50, 0.25), # true P[Y = k | X = 0]
                          replace = TRUE),
             no = sample(x = c("A", "B", "C"), 
                         size = n, 
                         prob = c(0.1, 0.1, 0.4) / 0.6, # true P[Y = k | X = 1]
                         replace = TRUE))
  # return tibble
  tibble(x = x, y = factor(y))
}

# true P[Y = k | X = 0]
c(A = 0.25, B = 0.50, C = 0.25)

# true P[Y = k | X = 1]
c(A = 0.1, B = 0.1, C = 0.4) / 0.6

# generate "train" and "test" data
set.seed(42)
some_data = gen_data(n = 250)
test_cases = tibble(x = c(0, 1))

# estimating P[Y = k | X = x]
predict(knn3(y ~ x, data = some_data), test_cases)
predict(rpart(y ~ x, data = some_data), test_cases)
predict(nnet(y ~ x, data = some_data, size = 1, trace = FALSE), test_cases)
predict(naiveBayes(y ~ x, data = some_data), test_cases, type = "raw")

# making classifications
predict(knn3(y ~ x, data = some_data), test_cases, type = "class")
predict(rpart(y ~ x, data = some_data), test_cases, type = "class")
predict(nnet(y ~ x, data = some_data, size = 1, trace = FALSE), test_cases, type = "class")
predict(naiveBayes(y ~ x, data = some_data), test_cases, type = "class")

################################################################################

# simulated circle data
set.seed(42)
circle = mlbench.circle(n = 1000)
plot(circle, pch = 16, xlab = "x.1", ylab = "x.2", main = "Simulated Circle Data")
grid()

circle_df = as_tibble(as.data.frame(circle)) # list -> df -> tibble
mod = knn3(classes ~ ., data = circle_df)

# calculate misclassifaction rate
mean(predict(mod, circle_df, type = "class") != circle_df$classes)

# function to calculate misclassifaction rate
calc_misclass = function(actual, predicted) {
  mean(actual != predicted)
}

# calculate misclassifaction rate
calc_misclass(actual = circle_df$classes,
              predicted = predict(mod, circle_df, type = "class"))

################################################################################

# simulated mvn data
set.seed(42)
three_norms = mlbench.2dnormals(n = 1000, cl = 3)
plot(three_norms, pch = 16, xlab = "x.1", ylab = "x.2", main = "Simulated MVN Data")
grid()

three_norms_df = as_tibble(as.data.frame(three_norms)) # list -> df -> tibble
knn_norm = knn3(classes ~ ., data = three_norms_df)

# calculate misclassifaction rate
calc_misclass(actual = three_norms_df$classes,
              predicted = predict(knn_norm, three_norms_df, type = "class"))

# create confusion matrix
table(predicted = predict(knn_norm, three_norms_df, type = "class"),
      actual = three_norms_df$classes)

################################################################################

# simulated spiral data
set.seed(42)
spirals = mlbench.spirals(n = 1000, cycles = 3, sd = 0.1)
plot(spirals, pch = 16, xlab = "x.1", ylab = "x.2", main = "Simulated Spiral Data")
grid()

spirals_df = as_tibble(as.data.frame(spirals)) # list -> df -> tibble

set.seed(1)
# test-train split
spirals_tst_trn_split = initial_split(spirals_df, prop = 0.80)
spirals_trn = training(spirals_tst_trn_split)
spirals_tst = testing(spirals_tst_trn_split)
# estimation-validation split
spirals_est_val_split = initial_split(spirals_trn, prop = 0.80)
spirals_est = training(spirals_est_val_split)
spirals_val = testing(spirals_est_val_split)

# tuning knn with a for loop
k_values = c(1, 3, 5, 7, 9)
val_misclass = rep(0, length(k_values))
for (k in seq_along(k_values)) {
  fit = knn3(classes ~ ., data = spirals_est, k = k_values[k])
  pred = predict(fit, spirals_val, type = "class")
  val_misclass[k] = calc_misclass(actual = spirals_val$classes, predicted = pred)
}

c("Validation Misclassification" = min(val_misclass), 
  k = k_values[which.min(val_misclass)])

# tuning knn with a for purrr::map
knn_mods = map(k_values, ~ knn3(classes ~ ., data = spirals_est, k = .x))
knn_preds = map(knn_mods, predict, spirals_val, type = "class")
knn_val_misclass = map_dbl(knn_preds, calc_misclass, actual = spirals_val$classes)

c("Validation Misclassification" = min(knn_val_misclass), 
  k = k_values[which.min(knn_val_misclass)])

################################################################################
```







