The residual sum of squares function is the following:
$$ \text{RSS} = \sum_{i=1}^N \left(y_i - \hat y_i\right)^2 ,$$
where $y_i$ is the $i$th variable to be predicted and $\hat y_i$ is the predicted value of $y_i$. It is often used as a cost function in the training of a linear regression model.