In this case, the activation function does not depend durante scores of other classes mediante \(C\) more than \(C_1 = C_i\). So the gradient respect to the each conteggio \(s_i\) in \(s\) will only depend on the loss given by its binary problem.
- Caffe: Sigmoid Ciclocross-Entropy Loss Layer
- Pytorch: BCEWithLogitsLoss
- TensorFlow: sigmoid_cross_entropy.
, from Facebook, in this paper. They claim to improve one-stage object detectors using Focal Loss sicuro train verso detector they name RetinaNet. Focal loss is verso Cross-Entropy Loss that weighs the contribution of each sample onesto the loss based durante the classification error. The timore is that, if a sample is already classified correctly by the CNN, its contribution onesto the loss decreases. With this strategy, they claim puro solve the problem of class imbalance by making the loss implicitly focus mediante those problematic classes. Moreover, they also weight the contribution of each class puro the lose per verso more explicit class balancing. They use Sigmoid activations, so Focal loss could also be considered verso Binary Ciclocross-Entropy Loss. We define it for each binary problem as:
Where \((1 – s_i)\gamma\), with the focusing parameter \(\genere >= 0\), is a modulating factor to scampato the influence of correctly classified samples in the loss. With \(\varieta = 0\), Focal Loss is equivalent sicuro Binary Ciclocampestre Entropy Loss.
Where we have separated formulation for when the class \(C_i = C_1\) is positive or negative (and therefore, the class \(C_2\) is positive). As before, we have \(s_2 = 1 – s_1\) and \(t2 = 1 – t_1\).
The gradient gets verso bit more complex due to the inclusion of the modulating factor \((1 – s_i)\gamma\) sopra the loss formulation, but it can be deduced using the Binary Cross-Entropy gradient expression.
Where \(f()\) is the sigmoid function. Puro get the gradient expression for a negative \(C_i (t_i = 0\)), we just need puro replace \(f(s_i)\) with \((1 – f(s_i))\) in the expression above.
Ratto that, if the modulating factor \(\varieta = 0\), the loss is equivalent preciso the CE Loss, and we end up with the same gradient expression.
Forward pass: Loss computation
Where logprobs[r] stores, a each element of the batch, the sum of the binary cross entropy verso each class. The focusing_parameter is \(\gamma\), which by default is 2 and should be defined as per layer parameter sopra the net prototxt. The class_balances can be used onesto introduce different loss contributions verso class, as they do per the Facebook paper.
Backward pass: Gradients computation
Sopra the specific (and usual) case of Multi-Class classification the labels are one-hot, so only the positive class \(C_p\) keeps its term sopra the loss. There is only one element of the Target vector \(t\) which is not niente \(t_i = t_p\). So discarding the elements of the summation which are nulla due preciso target labels, we can write:
This would be the pipeline for each one of the \(C\) clases. We set \(C\) independent binary classification problems \((C’ = 2)\). Then we sum up the loss over the different binary problems: We sum up the gradients of every binary problem to backpropagate, and the losses to schermo the global loss. \(s_1\) and \(t_1\) are the conteggio and the gorundtruth label for the class \(C_1\), which is also the class \(C_i\) sopra \(C\). \(s_2 = 1 – s_1\) and \(t_2 = 1 – t_1\) are the punteggio and the groundtruth label of the class \(C_2\), which is not per “class” mediante our original problem with \(C\) classes, but per class we create onesto batteria up the binary problem numero di telefono datemyage with \(C_1 = C_i\). We can understand it as per background class.