This post explains what is a statistical learning framework, and common results under this framework.
We have a random variable X, another random variable Y. Now we want to determine the relationship between X and Y.
We define the relationship by a prediction function f(x). For each x, this function produces an “action” a in the action space.
Now how do we get the predictive function f? We use a loss function l(a, y), that for each a and y, we produce a “loss”. Note since X is a random variable, f(x) is a transformation, so a is a random variable, too.
Also, l(a, y) is a transformation of (a, y), so l(a, y) is a random variable too. It’s distribution is based on both X and Y.
We then calculate f by minimizing the expectation of the loss, which is called “risk”. Since the distribution of l(a, y) is based both on the distribution of X and Y, to get this expectation, we need to do integration both on X and on Y. In the case of discrete variables, we do summation based on the pmf of (x, y).
The above are about theoretical properties of Y, X, loss function and prediction function. But we usually do not know the distribution of (X, Y). Thus, we choose to minimize empirical risk instead. We calculate empirical risk by summing all the empirical loss together, divided by m. (q: does this resemble Monte Carlo method? is this about computational statistics? Need a review.)
In the case of square-loss, we have the result, a = E(y|x).
In the case of 0-1 loss, we have the result, a = arg max P(y|x)
We want to predict a student’s mid-term grade (Y). We want to know the relationship between predicted value, and whether she is hard-working (X).
We use square-loss for this continuous variable Y. Since we know that to minize square loss, for any random variable we should predict the mean value of the variable (c.f. regression analysis, in OLS scenerio we calculate the MSE — but need further connection to this framework).
Now we just observed unfortunately the student is not hard-working.
We know for a not-hardworking student the expectation of mid-term grade is 40.
We then predict the grade to be 40, as a way to minimize square-loss.