Multi label Data and MultiLabel Accuracy #523

sguada · 2014-06-20T01:44:39Z

This PR allows to use multi-label data {-1,0,1} with 0 means ignore, -1 negative and 1 positive.

@sergeyk review

shelhamer · 2014-06-20T04:42:49Z

Nice, thanks Sergio! Could you rebase to squash the fixes, linting, and the like to make this easier to read?

sguada · 2014-06-20T20:47:19Z

I will, do you use a nice tool to squash commits and make it nicer?

Sergio

2014-06-19 21:42 GMT-07:00 Evan Shelhamer notifications@github.com:

Nice, thanks Sergio! Could you rebase to squash the fixes, linting, and
the like to make this easier to read?

—
Reply to this email directly or view it on GitHub
#523 (comment).

shelhamer · 2014-06-21T08:04:47Z

I just do an interactive rebase. For any commits you want to squash you
change the status to squash or s for short instead of pick:

http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html

I'm not sure if that qualifies as a nice tool but I like it haha. Thanks
Sergio!

On Sat, Jun 21, 2014 at 4:47 AM, Sergio Guadarrama <notifications@github.com

wrote:

I will, do you use a nice tool to squash commits and make it nicer?

Sergio

2014-06-19 21:42 GMT-07:00 Evan Shelhamer notifications@github.com:

Nice, thanks Sergio! Could you rebase to squash the fixes, linting, and
the like to make this easier to read?

—
Reply to this email directly or view it on GitHub
#523 (comment).

—
Reply to this email directly or view it on GitHub
#523 (comment).

sguada · 2014-06-23T13:22:04Z

@shelhamer I have cleaned a bit the commits, could try that it works for you?
@sergeyk Can you test this PR with the PASCAL classification example?

sergeyk · 2014-06-24T00:02:55Z

Yes, but only after Tuesday.
Great PR --- I've already used it, by the way.

On Monday, June 23, 2014, Sergio Guadarrama notifications@github.com
wrote:

@shelhamer https://github.com/shelhamer I have cleaned a bit the
commits, could try that it works for you?
@sergeyk https://github.com/sergeyk Can you test this PR with the
PASCAL classification example?

—
Reply to this email directly or view it on GitHub
#523 (comment).

thuanvh · 2014-06-24T06:58:15Z

@sguada Could I use this feature for face points alignment? As you know, with face point alignment problem, the input is a face image and label output is face points coordinates. Does this feature support that? Or does caffe support that model type?

sguada · 2014-06-25T13:48:58Z

@thuanvh I wouldn't use multi-label for that problem, multi-label assumes a discrete set of labels, rather than a vector of coordinates. So this PR is useful for image tagging or image classification (when one image can be classified with two or more labels), but not for regression.

Look at #512 and #147 on how to do regression. You probably want to minimize the distance between the predicted positions and the ground truth. So using a euclidian loss seems a good first attempt. You can use it directly if all the images have the same number of points, but you would need to adapt the code if that is not the case.

shelhamer · 2014-06-25T23:59:16Z

Should this be split into a multilabel accurac 8000 y and a multilabel loss as in #522 ?

sguada · 2014-06-26T00:13:03Z

Yes, but we don't need to have a multilabel loss, since I was using sigmoid_cross_entropy_loss. However I removed the code that compute the loss from the multi-label accuracy, to keep it align with #522
New multi-label losses could be added like #88 in the future.

onauparc · 2014-06-26T10:12:43Z

@sguada I would like to use this PR because in my net I would like to make multi-class prediction.

In my case I have a groundtruth like this: (3 classes)
1st class true/false
2nd class [0, 1, 2, 3]
3rd class [0, 1, 2, 3]

How should I format my labels?

For example if the groundtruth is [True 23] I think it should be something like this:

[1], [-1, -1, 1, -1], [-1, -1, -1, 1]. (I added brackets to make my example clearer)

Is it correct?

yocox · 2014-06-27T08:25:41Z

I am trying this PR ^^ But not sure what layer type should I use for the last layer for training / validation?

SOFT_MAX or MULTINOMIAL_LOGISTIC_LOSS for training?
MULTI_LABEL_ACCURACY for validating?

Thanks in advance!

sguada · 2014-06-27T11:41:36Z

You should use SIGMOID_CROSS_ENTROPY_LOSS as a loss for training and MULTI_LABEL_ACCURACY to measure accuracy on test.

sguada · 2014-06-27T11:46:17Z

@onauparc I'm a bit confuse about your multi-label problem, since it seems a mix of multi-class and and regression. But if you think that the encoding your proposed makes sense then this PR should be able to handle it.
You should try and see if it works or not.
If the value for classes 2 and 3 is a ranking then you could also try encoding the rank information, but setting to positive any smaller value.
[True 23] = [1] [1 1 1 -1] [1 1 1 1]
[False 01] = [-1][1 -1 -1 -1][1 1 -1 -1]
Probably you should try both and see which one works better for your problem.

bhack · 2014-06-27T19:20:10Z

I think that @onauparc is doing something like multi target single class with one hot encoding GT. @sguada Is it possible to handle this kind of problem with forked multiple "parallel" softmax top layer (one for each target)?

bhack · 2014-06-29T11:44:38Z

@sguada I mean something like the schema at page 13 in this pubblication: http://arxiv.org/abs/1312.6082v4
and that I've already posted in a comment in #547

yocox · 2014-06-30T10:54:35Z

It's a long post. I had tried for several days, still can not get a working sample Q_Q

I have done follow works

modify `tools/convert_imageset.cpp`

I modified the convert_imageset.cpp from:

while (infile >> filename >> label) {
  lines.push_back(std::make_pair(filename, label));
}

to:

std::string line;
while (std::getline(infile, line)) {
  int label;
  std::istringstream iss(line);
  iss >> filename;
  std::vector<int> labels;
  while(iss >> label) {
    labels.push_back(label);
  }
  lines.push_back(std::make_pair(filename, labels));
}

so now it can take file in following format:

file1.jpg 0 1
file2.jpg 2

I composed the `train.prototxt` and `val.prototxt`

my network is a 1 hidden node layer(2 weight layer) net.

Input data:

channel: 3
width: 16
height: 16
label: [0, 1, 2] multiple tag is possible

the train.prototxt is

name: "CaffeNet"
layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label"
  data_param {
    source: "train.leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 128
  }
}
layers {
  name: "ip1"
  type: INNER_PRODUCT
  bottom: "data"
  top: "ip1"
  blobs_lr: 1
  blobs_lr: 2
  inner_product_param {
    num_output: 500
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "relu1"
  type: RELU
  bottom: "ip1"
  top: "ip1"
}
layers {
  name: "ip2"
  type: INNER_PRODUCT
  bottom: "ip1"
  top: "ip2"
  blobs_lr: 1
  blobs_lr: 2
  inner_product_param {
    num_output: 4
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "loss"
  type: SIGMOID_CROSS_ENTROPY_LOSS
  bottom: "ip2"
  bottom: "label"
}

and the val.prototxt is

name: "CaffeNet"
layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label"
  data_param {
    source: "train.leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 128
  }
}
layers {
  name: "ip1"
  type: INNER_PRODUCT
  bottom: "data"
  top: "ip1"
  inner_product_param {
    num_output: 500
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "relu1"
  type: RELU
  bottom: "ip1"
  top: "ip1"
}
layers {
  name: "ip2"
  type: INNER_PRODUCT
  bottom: "ip1"
  top: "ip2"
  inner_product_param {
    num_output: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "prob"
  type: SOFTMAX
  bottom: "ip2"
  top: "prob"
}
layers {
  name: "accuracy"
  type: MULTI_LABEL_ACCURACY
  bottom: "prob"
  bottom: "label"
  top: "accuracy"
}

When I run training, I got following error

I0630 18:37:25.089489 10070 net.cpp:65] Creating Layer loss
I0630 18:37:25.089495 10070 net.cpp:197] loss <- ip2
I0630 18:37:25.089509 10070 net.cpp:197] loss <- label
F0630 18:37:25.089534 10070 sigmoid_cross_entropy_loss_layer.cpp:18] Check failed: bottom[0]->count() == bottom[1]->count() (512 vs. 128) SigmoidCrossEntropyLoss Layer inputs must have same count.
*** Check failure stack trace: ***
    @     0x2b16f01d4a1d  google::LogMessage::Fail()
    @     0x2b16f01d68bd  google::LogMessage::SendToLog()
    @     0x2b16f01d460c  google::LogMessage::Flush()

I will be gratitude if sguada can point out my error, or just show me an working example :D and I will work from the sample!

Thanks in advance.

sguada · 2014-06-30T14:00:18Z

@yocox What happens is that the multi-label format should be a vector of {-1,0,1} one per label. So your train file should look like

file1.jpg 1 1 -1
file2.jpg -1 -1 1

What means that file1.jpg is have label 0 and 1 but no 2, and file2.jpg have label 2 but not 0 or 1.

I think your modification of conver_imageset looks good, just try to change your input file.

sguada · 2014-06-30T14:07:27Z

@bhack It is totally possible handle multiple softmax layers and then concat them. The only thing that need to be address is how the labels are going to be encoded.

One solution using this PR would be to encode the multiple labels as a vector of binary labels. For example, let's assume we want to code length 0, 1, 2, >2 and then two digits 0..9 that could be done by:

34 -> [-1 -1 1 -1] [-1 -1 -1 1 -1 -1 -1 -1 -1 -1][-1 -1 -1 -1 1 -1 -1 -1 -1 -1]
7  -> [-1 1 -1 -1] [-1 -1 -1 -1 -1 -1 -1 1 -1 -1][-1 -1 -1 -1 -1 -1 -1 -1 -1 -1]

sguada · 2014-06-30T14:12:15Z

@shelhamer I don't know how to clean this PR more, should I close it and start a new one with fewer and better grouped commits?

bhack · 2014-06-30T14:24:36Z

Seems that not merge anymore

bhack · 2014-06-30T14:33:35Z

@sguada We have already experimented this encoding also on other frameworks and works fine.
Another solution is to have an accurancy layer that support multiple bottom label and multiple top.

8000

sguada · 2014-06-30T15:10:23Z

@bhack

I think for that you just need multiple accuracy layers, each one with one
bottom one label and one top.

Sergio

2014-06-30 7:33 GMT-07:00 bhack notifications@github.com:

@sguada https://github.com/sguada We have already experimented this
encoding also on other frameworks and works fine.
Another solution is to have an accurancy layer that support multiple
bottom label and multiple top.

—
Reply to this email directly or view it on GitHub
#523 (comment).

bhack · 2014-06-30T15:21:40Z

@sguade Yes off course but this doesn't work if we want our accurancy incremented on the global goal of having all the multiple accurancy layer (target) correct.

sguada · 2014-06-30T15:25:12Z

@bhack yeah, that's correct if you want a global loss then you need to combine them. Are you interested in writing an example?

I will either rebase again or start a cleaner PR

bhack · 2014-06-30T15:49:17Z

@sguada Yes. One example is in Street View House Number (pag. 13 http://arxiv.org/pdf/1312.6082v4.pdf). The accuracy need to handle all softmax layers because the goal is to have the correct House Number sequence (5 digit max) with the right length (1 softmax). So we have six softmax layer with a global loss.

sguada · 2014-06-30T15:59:10Z

@bhack, I know the paper and that example, I meant if you are willing to
write the needed prototxts and the global loss.

On Monday, June 30, 2014, bhack notifications@github.com wrote:

@sguada https://github.com/sguada Yes. One example is in Street View
House Number (pag. 13 http://arxiv.org/pdf/1312.6082v4.pdf). The accuracy
need to handle all softmax layers because the goal is to have the correct
House Number sequence (5 digit max) with the right length (1 softmax). So
we have six softmax layer with a global loss.

—
Reply to this email directly or view it on GitHub
#523 (comment).

Sergio

bgreen-litl · 2015-01-07T17:20:33Z

Is #1414 really intended to replace this PR or is it IndirectionLayer just one part of it? (Trying to get my bearings in these two very heavily commented PRs.)

bhack · 2015-01-07T17:52:58Z

@bgreen-litl There is also #1380

elaksana · 2015-01-23T00:47:54Z

Has there been any progress on this PR? @sguada Have you been working lately on this?

rajeswar18 · 2015-01-27T13:08:49Z

@sguada I am trying to learn a model that outputs a word(more than 10 characters) so how to write the code length as a binary vector? should it be [0,..9,>9] or could you let me kknow the current trend for multi-label

danielhauagge · 2015-03-29T15:39:03Z

I'm really interested in this PR or any other one that implements it's functionality, so far #1380 and #1414 seem to be working on the same problem.

@sguada, if I were to try and do some multilabel learning with caffe right now, which pr would you recommend? If any.

gavinmh · 2015-04-19T19:14:35Z

I am also interested in data and accuracy layers that support multi-label classification. Can someone clarify the states of the PRs?

bhack · 2015-04-19T20:13:28Z

@gavinmh @danielhauagge See last commenta in #1380

lireagan · 2015-05-11T06:17:06Z

fatal error: cublas_v2.h: No such file or directory

Same problem happened when I complied the example released by @yocox , did you deal with it? @misaghkord @yocox

I have set CPU_ONLY:=1 but it still happened.
or anybody can help me? THX!

barbolo · 2015-06-18T19:32:22Z

Wow! Tomorrow is exactly a year since this PR was created.

If I need to solve a multi label classification problem, what is the approach I should follow with caffe?

ps: I'm still digging through all the references posted here.

bhack · 2015-06-23T09:00:55Z

Is this only a documentation problem now? See #1698

shelhamer · 2015-08-26T00:10:13Z

Closing this PR since it is against the defunct dev branch and we concluded that losses and layers are capable of handling multilabel problems so this is a documentation / example issue and not a technical problem that needs further layers.

ih4cku · 2015-09-14T01:32:10Z

The dev branch has been deprecated and deleted. Why didn't this feature merged into the master branch?

bhack · 2015-09-14T08:41:25Z

@ih4cku It is not needed. See #1698

ih4cku · 2015-09-14T14:21:41Z

@bhack
Thanks for responding. My problem is quite similar with Goodfellow's, which you also mentioned above.

In this multi-task setting, it's ok for training by using the multi-dimensional labels with SigmoidCrossEntropyLayer or slicing the labels to feed to multiple SoftmaxWithLoss layers.

But how about the accuracy? How to compute the total accuracy of a sample during test?
AFAIK, now the AccuracyLayer only support the target label blob of shape (N ×1×1×1), which is also mentioned in #2188.

Have you customized the accuracy layer?

bhack · 2015-09-14T14:26:54Z

I've done something cited in comments at #2935 but was never reviewed and now closed because against dev branch.

urakagi · 2015-10-13T06:19:22Z

Where is the document/example explaining how to handle multilabel problems with current system?

ColdZoo · 2015-12-10T00:50:35Z

Many thanks to @sguada , I am using your code now.

Coderx7 · 2017-10-07T09:54:53Z

Is this finalized?

sidmangla · 2018-05-15T12:31:28Z

Hi there, siddharth here, i am working on a project to recognize products and have used 200 pics each to train the 11 classes. but when i am testing it is showing only one class at a time. For labeling i have used LabelImg.
please help

sergeyk self-assigned this Jun 20, 2014

This was referenced Jun 26, 2014

Does anyone make a multi-label classfication test using caffe? #536

Closed

ImageDataParameter has no field specify whether input images are color or grayscale #538

Closed

shelhamer mentioned this pull request Jun 26, 2014

Feature Request: Multi-Label DataLayer and LevelDB converter #460

Closed

shelhamer removed this from the 1.0 milestone Mar 9, 2015

shelhamer added the ES label Mar 12, 2015

lukeyeager mentioned this pull request Jul 14, 2015

Add support for multiple and/or floating point labels NVIDIA/DIGITS#97

Closed

shelhamer mentioned this pull request Aug 25, 2015

Clear all content out of dev except README.md #2081

Closed

shelhamer closed this Aug 26, 2015

hemanth2090 mentioned this pull request Sep 23, 2015

Multi-label classification #3111

Closed

BlGene mentioned this pull request Nov 11, 2015

multi-label softmax support #3268

Closed

samratkokula mentioned this pull request Dec 11, 2015

Master #3443

Closed

elezar mentioned this pull request Feb 9, 2016

Add support for multiple labels to be specified in the image_data_layer #3653

Closed

varunagrawal mentioned this pull request May 26, 2016

Create Validation Net for each model varunagrawal/YelpNet#1

Closed

ih4cku mentioned this pull request Jul 27, 2016

multiple targets ih4cku/caffe-notes#14

Open

Multi label Data and MultiLabel Accuracy #523

Multi label Data and MultiLabel Accuracy #523

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

modify tools/convert_imageset.cpp

I composed the train.prototxt and val.prototxt

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

modify `tools/convert_imageset.cpp`

I composed the `train.prototxt` and `val.prototxt`