A real-world object detection project that fine-tunes YOLOv5s on a maritime vessel dataset to detect 10 target classes, including Cabin Cruisers, Canoes/Kayaks, Commercial vessels, PWCs, Humans, and lifejackets (PFDs). Beyond training, the notebook builds a complete prediction evaluation pipeline: running inference on a new dataset, parsing YOLO’s text output into a structured DataFrame, deduplicating by confidence score, and computing accuracy, precision, and recall against ground truth labels extracted from filename conventions.
Can a single-stage object detector reliably identify vessel types and safety equipment from waterway surveillance images? And once the model is trained, how do you rigorously evaluate its per-class performance on a brand-new dataset without pre-existing annotation files?
Training - YOLOv5 was cloned from the Ultralytics repository and dependencies installed. yolov5s.pt (the small pre-trained backbone) was fine-tuned with:
--nosave for suppressed intermediate checkpoints, with --cache for faster data loading.Old dataset validation - val.py was run against the trained weights on the held-out validation set, producing per-class mAP@0.5 and mAP@0.5:0.95 metrics alongside precision and recall. A confusion matrix and PR curve were loaded from saved image files and displayed for visual analysis.
New dataset inference - A second dataset (New_DataSet) was processed by looping through all images, calling detect.py via os.popen() with conf=0.1, and inserting a 6-second time.sleep() between calls to allow subprocess completion. A second inference pass on a richer image set used detect_txt.py with --save-txt --save-conf to write per-image label files.
Post-processing pipeline - The saved .txt label files were parsed into a structured DataFrame:
class_dict mapping to retain only the target classes.- separator, with speed suffixes (under5kn, over5kn) stripped.groupby('image_name')['confidence_score'].idxmax().HalfCab was remapped to Open during ground truth normalization.Evaluation - accuracy_score, precision_score (macro), and recall_score (macro) were computed against the ground truth labels. A full confusion matrix was plotted.
Old dataset validation metrics (mAP@0.5):
| Class | mAP@0.5 |
|---|---|
| Cabin Cruiser | 99.5% |
| Canoe/Kayak | 97.8% |
| Commercial | 99.5% |
| Half Cab | 99.1% |
| Human | 90.8% |
| Open | 96.4% |
| PFD (lifejacket) | 82.7% |
| PWC | 94.7% |
| Registration Number | 89.5% |
| Structure | 89.6% |
| Overall | mAP@0.5: 0.861 / mAP@0.5:0.95: 0.625 |
Overall precision: 0.939, recall: 0.832.
New dataset evaluation - The confusion matrix revealed that on the new dataset, Cabin Cruisers and Human-Powered vessels were frequently misclassified as HalfCab/Open, likely a labeling consistency issue rather than a model failure. Commercial and Open class predictions were accurate. The notebook concludes with actionable diagnosis: remove underrepresented classes such as Dinghy, House Boat, and Windsurfer that had no training examples; remove the Other class because it had a 100% misclassification rate; and add more Human and PFD training examples to address the two weakest per-class scores.
The post-processing pipeline is the technically demanding part of this project: YOLOv5’s inference output is a folder of plain text files, and converting that into a structured, deduplicated, class-matched DataFrame ready for sklearn evaluation metrics requires non-trivial parsing logic.
The filename-as-ground-truth convention, extracting the real label from the image name, is a pragmatic engineering choice that avoids needing separate annotation files for the new dataset. But it is brittle; the speed suffix stripping and class remapping steps were both required to handle naming inconsistencies in the data.
The confusion matrix on the new dataset also illustrates an important model evaluation principle: poor performance on a held-out set is not always a model problem. When Cabin Cruisers are predicted as HalfCabs on entirely new data, the first question is whether the training labels were consistent, not whether the architecture is wrong.