Description
I probably found an issue with the indexing of categories at coco evaluation. This might cause interchanged results for each category which would also affect AP(novel) and AP(base) since it is unknown which result belongs to which category #30 . The overall mAP is unaffected by this issue.
At inference, the test-class puts the detections inside the variable all_boxes where the classes are indexed by their position in imdb.classes (which is Background-Class + Base-Classes + Novel-Classes, as defined in the constructor of coco.py).
As coco.py does the evaluation, the following happens:
- its method
evaluate_detectionsgets the detections viaall_boxesfrom the test class evaluate_detectionscalls_write_coco_results_file_write_coco_results_fileiterates over its categories (self.classes) and uses the mappingcoco_cat_id = self._class_to_coco_cat_id[cls]to obtain the category id from category name. It passes each category ID to_coco_results_one_categorywhich will return detections in a different format:- x,y,w,h instead of xmin, ymin, xmax, ymax
- image id instead of image index
- category ID instead of category
- Now we have saved the detections in a different format to a json file which will be passed to
_do_detection_eval _do_detection_evalcreates an instance of COCOeval with- itself (the COCO object, initialized with the validation-annotation file)
- another COCO-object initialized with the previously created json-file (the rewritten detections)
_do_detection_evalruns evaluate and accumulate on the cocoeval object and passes it to_print_detection_eval_metrics- inside COCOeval this takes place:
- in its constructor, it sets
self.params.catIds = sorted(cocoGt.getCatIds()), where cocoGt is the COCO-object initialized with the validation annotation file - evaluate() uses those catIDs to identify categories
- accumulate() stores precision and recall of a category at the index of that category in catIDs (stores them in self.eval)
- in its constructor, it sets
Now we have two problematic situations inside _print_detection_eval_metrics() method of coco.py:
- printing of class-wise AP:
- directly accesses cocoeval.eval['precision'] with class incides from
cls_ind, cls in enumerate(self.classes), but as stated above, the metrics for a class are stored at the index of that class as in the validation annotation file. This causes the category names for per-category results to be interchanged
- directly accesses cocoeval.eval['precision'] with class incides from
- printing of summarized novel class mAP and base class mAP:
- passes range(0, len(base_classes)) for summary of base classes and range(len(base_classes), len(all_classes)) for summary of novel classes to
cocoeval.summarize. However, thesummarizemethod uses thecategoryIdargument to directly access the precision and recall of that class, but those indices are wrong (as described above for class-wise AP)
- passes range(0, len(base_classes)) for summary of base classes and range(len(base_classes), len(all_classes)) for summary of novel classes to
To solve the stated problems, I would suggest the following changes (for _print_detection_eval_metrics, line 245-259)
cat_ids = self._COCO.getCatIds()
cats = self._COCO.loadCats(cat_ids)
cat_name_to_ind = dict(list(zip([c['name'] for c in cats], range(len(cats)))))
for cls_ind, cls in enumerate(cat_name_to_ind.keys()):
# no check for cls == '__background__' needed due to new list we're iterating over
precision = coco_eval.eval['precision'][ind_lo:(ind_hi + 1), :, cls_ind, 0, 2] # no index shift necessary
ap = np.mean(precision[precision > -1])
print('{}: {:.1f}'.format(cls, 100 * ap))
print('~~~~ Summary Base metrics ~~~~')
categoryId = list(map(lambda cls: cat_name_to_ind[cls], self._base_classes)) # use correct indices now
coco_eval.summarize(categoryId)
print('~~~~ Summary Novel metrics ~~~~')
categoryId = list(map(lambda cls: cat_name_to_ind[cls], self._novel_classes)) # use correct indices now
coco_eval.summarize(categoryId)self._base_classes and self._novel_classes are the lists of base and novel class names which I used to create self._classes in the constructor.
Some final thoughts on the issue:
- I think using a variable name
categoryIdinsidecocoeval.summarizeis a bit confusing, since they treat them as indices - The crucial mistake presumably was the different order of the categories inside
self.classes(of coco.py) and the categories as in the COCO object- In coco.py of Meta-RCNN they leave the categories in the same order as they are read in from the COCO API and just prepend a background class. That's why they are able to directly iterate over
self.classes(instead of having to read in original coco categories for the correct order) and just have to do a simple index shift to obtain correct results for each class.
- In coco.py of Meta-RCNN they leave the categories in the same order as they are read in from the COCO API and just prepend a background class. That's why they are able to directly iterate over