Login Register Cart 0
Generic placeholder image

Recent Patents on Engineering

Editor-in-Chief

ISSN (Print): 1872-2121
ISSN (Online): 2212-4047

Back
Journal Home About Journal Editorial Board Journal Insight Current Issue Volumes/Issues
Subscribe
Research Article

Pavement Pothole Detection Based on Cascade and Fusion Convolutional Neural Network Using 2D Images under Complex Pavement Conditions

Author(s): Chen Guoqiang*, Bai Bingxin and Yi Huailong

Volume 16, Issue 1, 2022

Published on: 14 September, 2020

Article ID: e201021185886 Pages: 11

DOI: 10.2174/1872212114999200914113515

Price: $65

Abstract

Background: The development of deep learning technology has promoted the industrial intelligence, and automatic driving vehicles have become a hot research direction. As to the problem that pavement potholes threaten the safety of automatic driving vehicles, the pothole detection under complex environment conditions is studied.

Objective: The goal of the work is to propose a new model of pavement pothole detection based on convolutional neural network. The main contribution is that the Multi-level Feature Fusion Block and the Detector Cascading Block are designed and a series of detectors are cascaded together to improve the detection accuracy of the proposed model.

Methods: A pothole detection model is designed based on the original object detection model. In the study, the Transfer Connection Block in the Object Detection Module is removed and the Multi-level Feature Fusion Block is redesigned. At the same time, a Detector Cascading Block with multi-step detection is designed. Detectors are connected directly to the feature map and cascaded. In addition, the structure skips the transformation step.

Results: The proposed method can be used to detect potholes efficiently. The real-time and accuracy of the model are improved after adjusting the network parameters and redesigning the model structure. The maximum detection accuracy of the proposed model is 75.24%.

Conclusion: The Multi-level Feature Fusion Block designed enhances the fusion of high and low layer feature information and is conducive to extracting a large amount of target information. The Detector Cascade Block is a detector with cascade structure, which can realize more accurate prediction of the object. In a word, the model designed has greatly improved the detection accuracy and speed, which lays a solid foundation for pavement pothole detection under complex environmental conditions.

Keywords: Pavement pothole detection, automatic driving, security, convolutional neural network, cascade, detector

Graphical Abstract

[1]
M. Saad, and K.N. Tahar, "Identification of rut and pothole by using multirotor unmanned aerial vehicle (UAV)", Measurement, vol. 137, pp. 647-654, 2019.
[http://dx.doi.org/10.1016/j.measurement.2019.01.093]
[2]
Y.O. Ouma, and M. Hahn, "Pothole detection on asphalt pavements from 2D-colour pothole images using fuzzy c-means clustering and morphological reconstruction", Autom. Construct., vol. 83, pp. 196-211, 2017.
[http://dx.doi.org/10.1016/j.autcon.2017.08.017]
[3]
Hoye and S. Krotosky, Pothole detection. U.S. Patent 20170243370A1, 2017
[4]
K.D. Zoysa, and C. Keppitiyagama, "Busnet–a sensor network built over a public transport system", Adjunct Poster/Demo Proceedings of the Fourth 10 European Conference on Wireless and Sensor Networks (EWSN), 2007pp. 29-31 Delft, Netherlands
[5]
S.S. Rode, S. Vijay, and P. Goyal, "Pothole detection and warning system: infrastructure support and system design", 2009 International Conference on Electronic Computer Technology, 2009pp. 286-290 Macau China
[6]
A. Mohamed, M.M.M. Fouad, and E. Elhariri, "RoadMonitor: an intelligent road surface condition monitoring system", IEEE Trans. Consum. Electron., vol. 30, no. 3, pp. 251-258, 2015.
[http://dx.doi.org/10.1007/978-3-319-11310-4_33]
[7]
T. Ishikawa, and K. Fujinami, "Smartphone-based pedestrian’s avoidance behavior recognition towards opportunistic road anomaly detection", ISPRS Int. J. Geoinf., vol. 5, no. 10, p. 182, 2016.
[http://dx.doi.org/10.3390/ijgi5100182]
[8]
C. Kyriakou, S.E. Christodoulou, and L. Dimitriou, "Smartphone-based pothole detection utilizing artificial neural networks", J. Infrastruct. Syst., vol. 25, no. 3, 2019.
[http://dx.doi.org/10.1061/(ASCE)IS.1943-555X.0000489]
[9]
A. Tedeschi, and F. Benedetto, "A real-time automatic pavement crack and pothole recognition system for mobile Android-based devices", Adv. Eng. Inform., vol. 32, pp. 11-25, 2017.
[http://dx.doi.org/10.1016/j.aei.2016.12.004]
[10]
R. Du, G. Qiu, K. Gao, L. Hu, and L. Liu, "Abnormal road surface recognition based on smartphone acceleration sensor", Sensors (Basel), vol. 20, no. 2, p. E451, 2020.
[http://dx.doi.org/10.3390/s20020451] [PMID: 31941141]
[11]
B.X. Yu, and X. Yu, "Vibration-based system for pavement condition evaluation", in: 9th International Conference on Applications of Advanced Technology in Transportation, Chicago, UnitedStates, 2006, pp.183-189.
[http://dx.doi.org/10.1061/40799(213)31]
[12]
J. Eriksson, L. Girod, and B. Hull, "The pothole patrol: using a mobile sensor network for road surface monitoring", Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services, 2008pp. 29-39 Breckenridge, United States
[http://dx.doi.org/10.1145/1378600.1378605]
[13]
D. Liu, J. T. Ren, and J. Y. Zhang, A method for detecting road potholes based on acceleration sensor and machine learning. CN Patent 107167580A, 2011.
[14]
X. H. Zhang, Q. Li, and F. T. Deng, Pavement pothole recognition system of wheelchair and wheelchair. CN Patent 109512594A
[15]
V. Rishiwal, and H. Khan, "Automatic pothole and speed breaker detection using android system", 39th International Convention on Information and Communication Technology, Electronics and Microelectronic, 2016pp. 1270-1273 Opatija, Croatia
[http://dx.doi.org/10.1109/MIPRO.2016.7522334]
[16]
J.P.C.d. Azevedo, F.M.A. Neves, and R.M.C.e. Costa, Adaptive road management in the network of moving things. U.S. Patent 20180053405A1, 2018.
[17]
L. Huidrom, L.K. Das, and S.K. Sud, "Method for automated assessment of potholes, cracks and patches from road surface video clips", Procedia Soc. Behav. Sci., vol. 104, pp. 312-321, 2013.
[http://dx.doi.org/10.1016/j.sbspro.2013.11.124]
[18]
H. L. Yang, J. Wang, and H. T. Sun, Pavement pothole detection method based on vehicle binocular vision. CN Patent 104964708A
[19]
M.H. Yousaf, K. Azhar, F. Murtaza, and F. Hussain, "Visual analysis of asphalt pavement for detection and localization of potholes", Adv. Eng. Inform., vol. 38, pp. 527-537, 2018.
[http://dx.doi.org/10.1016/j.aei.2018.09.002]
[20]
C. L. Chang, and S. S. Cheng, Automatic detection method for pavement potholes. CN Patent 106978774B
[21]
G.M. Jog, C. Koch, M. Golparvar-Fard, and I. Brilakis, "Pothole properties measurement through visual 2D recognition and 3D reconstruction Computing", Civ. Eng., vol. •••, pp. 553-560, 2012.
[http://dx.doi.org/10.1061/9780784412343.0070]
[22]
G. Debreczeni, Autonomous vehicle with improved visual detec. U.S. Patent 20170285649A, 2017
[23]
Y. Hang, Y. Y. Qi, and S. Huang, Method and device for Pavement pothole detection. CN Patent 108149554A.
[24]
D. Zhang, Q. Zou, H. Lin, X. Xu, L. He, R. Gui, and Q. Li, "Automatic pavement defect detection using 3D laser profiling technology", Autom. Construct., vol. 96, pp. 350-365, 2018.
[http://dx.doi.org/10.1016/j.autcon.2018.09.019]
[25]
A.C.B. Chun, A.A. Mahmud, L.B. Theng, and A.C.W. Yen, "Wearable ground plane hazards detection and recognition system for the visually impaired", In: Proceedings of the 2019 3rd International Conference on E-Society, E-Education and ETechnology, Taipei, Taiwan, 2019, pp. 84-89.
[http://dx.doi.org/10.1145/3355966.3355968]
[26]
X. S. Zheng, Y. Ren, and G. Yan, Inspection device for vehicle-mounted pavement pothole detection system based on variable pothole shape. CN Patent 110629646A
[27]
V. Pereira, S. Tamura, and S. Hayamizu, "A deep learning-based approach for road pothole detection in timor leste", IEEE International Conference on Service Operations and Logistics, and Informatics, 2018pp. 279-284 Furama River Front, Singapore
[http://dx.doi.org/10.1109/SOLI.2018.8476795]
[28]
A. Krizhevsky, I. Sutskever, and G.E. Hinton, Imagenet classification with deep convolutional neural networksNIPS 2012: Advances in Neural Information Processing Systems, Lake Tahoe, United States. 2012, pp. 1097-1105.
[29]
Y Bhatia, R Rai, and V Gupta, Convolutional neural networks based potholes detection using thermal imaging. Journal of King Saud University-Computer and Information Sciences, 2019.
[30]
H. Wu, L. Yao, and Z. Xu, "Road pothole extraction and safety evaluation by integration of point cloud and images derived from mobile mapping sensors., vol. 42", Adv. Eng. Inform., 2019.
[31]
L.K. Suong, and K. Jangwoo, "Detection of potholes using a deep convolutional neural network", J. Univers. Comput. Sci., vol. 24, no. 9, pp. 1244-1257, 2018.
[32]
J. Redmon, and A. Farhadi, "YOLO9000: better, faster, stronger", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Honolulu, United States, 2017, pp. 7263-7271.
[33]
W. Ye, J. Wei, and T. Zheng, "Convolutional neural network for pothole detection in asphalt pavement", Road Mater. Pavement Des., vol. 22, pp. 1-17, 2019.
[http://dx.doi.org/10.1080/14680629.2019.1615533]
[34]
L. Song, and X. Wang, "Faster region convolutional neural network for automated pavement distress detection", Road Mater. Pavement Des., vol. 22, pp. 23-41, 2019.
[http://dx.doi.org/10.1080/14680629.2019.1614969]
[35]
C.V. Gonzalez, Object detection for an autonomous vehicle. U.S. Patent 20170323179A1, 2017.
[36]
L. Ale, N. Zhang, and L. Li, "Road damage detection using retinanet", 2018 IEEE International Conference on Big Data, 2018pp. 5197-5200 Seattle, United States
[http://dx.doi.org/10.1109/BigData.2018.8622025]
[37]
T. M., "Hoang, S. H. Nam and K. R. Park, “Enhanced detection and recognition of road markings based on adaptive region of interest and deep learning", IEEE Access, vol. 7, pp. 109817-109832, 2019.
[38]
J.J. Koh, T.T.V. Yap, and H. Ng, "Autonomous road potholes detection on video", International Conference of Computer Science and Technology, 2018pp. 137-143 Kota Kinabalu, Malaysia
[39]
S. Anand, S. Gupta, V. Darbari, and S. Kohli, Crack-pot: autonomous road crack and pothole detection In: DICTA 2018: Digital Image Computing: Techniques and Applications, Canberra, Australia, 2018 pp. 1-6.
[http://dx.doi.org/10.1109/DICTA.2018.8615819]
[40]
C. Koch, and I. Brilakis, "Improving pothole recognition through vision tracking for automated pavement assessment", Proc. of EGICE Workshop, 2011pp. 1-8
[41]
H. Maeda, Y. Sekimoto, and T. Seto, "Lightweight road manager: smartphone-based automatic determination of road damage status by deep neural network", Proceedings of the 5th ACM SIGSPATIAL International Workshop on Mobile Geographic Information Systems, 2016pp. 37-45 California, United States
[http://dx.doi.org/10.1145/3004725.3004729]
[42]
H. Maeda, Y. Sekimoto, and T. Seto, "Road damage detection and classification using deep neural networks with smartphone images", Comput. Aided Civ. Infrastruct. Eng., vol. 33, no. 12, pp. 1127-1141, 2018.
[http://dx.doi.org/10.1111/mice.12387]
[43]
D.G. Lowe, "Distinctive image features from scale-invariant keypoints", Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
[http://dx.doi.org/10.1023/B:VISI.0000029664.99615.94]
[44]
N. Dalal, and B. Triggs, "Histograms of oriented gradients for human detection", CVPR 2005: Proceedings of International Conference on Computer Vision and Pattern Recognition, 2005pp. 886-893 San Diego, United States
[http://dx.doi.org/10.1109/CVPR.2005.177]
[45]
R. Girshick, J. Donahue, and T. Darrell, "Rich feature hierarchies for accurate object detection and semantic segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014pp. 580-587 Columbus, United States
[http://dx.doi.org/10.1109/CVPR.2014.81]
[46]
S. Ren, K. He, R. Girshick, and J. Sun, Faster r-cnn: towards real-time object detection with region proposal networks., Montreal, Canada, 2015, pp. 91-99.Advances in Neural Information Processing Systems
[47]
Y. Tian, P. Luo, X. Wang, and X. Tang, "Deep learning strong parts for pedestrian detection", Proceedings of the IEEE International Conference on Computer Vision, 2015pp. 1904-1912 Santiago, Chile
[http://dx.doi.org/10.1109/ICCV.2015.221]
[48]
W. Nam, P. Dollár, and J.H. Han, “Local decorrelation for improved pedestrian detection”, In: Advances in Neural Information Processing Systems, Montreal, Canada. 2014, pp. 424-432.
[49]
Z. Cai, Q. Fan, R.S. Feris, and N. Vasconcelos, "A unified multi-scale deep convolutional neural network for fast object detection", European Conference on Computer Vision, 2016pp. 354-3700 Amsterdam, Holland
[http://dx.doi.org/10.1007/978-3-319-46493-0_22]
[50]
Z. Chen, K. Wu, Y. Li, M. Wang, and W. Li, SSD-MSN: An improved multi-Scale object detection network based on SSD., vol. Vol. 7. IEEE, 2019, pp. 80622-80632.
[http://dx.doi.org/10.1109/ACCESS.2019.2923016]
[51]
M. Everingham, L.V. Gool, and C.K. Williams, "The pascal visual object classes (voc) challenge", Int. J. Comput. Vis., vol. 88, no. 2, pp. 303-338, 2010.
[http://dx.doi.org/10.1007/s11263-009-0275-4]
[52]
T.Y. Lin, M. Maire, and S. Belongie, "Microsoft coco: common objects in context", European Conference on Computer Vision, 2014pp. 740-755 Zurich, Switzerland
[53]
B.J. Zhao, B.Y. Zhao, and L.B. Tang, "Multi-scale object detection by top-down and bottom-up feature pyramid network", J. Syst. Eng. Electron., vol. 30, no. 1, pp. 1-12, 2019.
[http://dx.doi.org/10.21629/JSEE.2017.01.01]
[54]
B. Kaur, and J. Bhattacharya, "A convolutional feature map-based deep network targeted towards traffic detection and classification", Expert Syst. Appl., vol. 124, pp. 119-129, 2019.
[http://dx.doi.org/10.1016/j.eswa.2019.01.014]
[55]
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: unified, real-time object detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016pp. 779-788 Las Vegas, United States
[http://dx.doi.org/10.1109/CVPR.2016.91]
[56]
B. Wu, F. Iandola, P.H. Jin, and K. Keutzer, "Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017pp. 129-137 Puerto Rico, United States
[http://dx.doi.org/10.1109/CVPRW.2017.60]
[57]
Z. Li, C. Peng, and G. Yu, Light-head r-cnn: in defense of two-stage object detector. arXiv preprint arXiv:1711.07264, 2017.
[58]
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016pp. 770-778 Las Vegas, United States
[59]
S. Zhang, L. Wen, and X. Bian, "Single-shot refinement neural network for object detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018pp. 4203-4212 Salt Lake City, United States
[http://dx.doi.org/10.1109/CVPR.2018.00442]
[60]
K. Simonyan, and A. Zisserman, "Very deep c-onvolutional networks for large-scale image recognit-ion", International Conference on Learning Representations, 2015 San Diego, United States
[61]
W. Liu, D. Anguelov, and D. Erhan, "Ssd: single shot multibox detector", European Conference on Computer Vision, 2016pp. 21-37 Amsterdam, Holland
[62]
A. Neubeck, and L.V. Gool, "Efficient non-maximum suppression", 18th International Conference on Pattern Recognition, 2006pp. 850-855 Hong Kong, China
[63]
T.Y. Lin, and P. Dollár, "Feature pyramid networks for object detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017pp. 2117-2125 Honolulu, United States
[64]
Y. Guo, Y. Xu, and S. Li, "Dense construction vehicle detection based on orientation-aware feature fusion convolutional neural network", Autom. Construct., vol. 112, no. 103124, 2020.
[http://dx.doi.org/10.1016/j.autcon.2020.103124]
[65]
Y. Gu, B. Wang, and B. Xu, "A FPN-Based Framework for Vehicle Detection in Aerial Images", Proceedings of the 2018 the 2nd International Conference on Video and Image Processing, 2018pp. 60-64 Chengdu, China
[http://dx.doi.org/10.1145/3301506.3301531]
[66]
M. Zhai, X. Xiang, R. Zhang, N. Lv, and A. El Saddik, "Optical flow estimation using channel attention mechanism and dilated convolutional neural networks", Neurocomputing, vol. 368, pp. 124-132, 2019.
[http://dx.doi.org/10.1016/j.neucom.2019.08.040]
[67]
Y.H. Li, Y.T. Chen, N.Y. Wang, and Z.X. Zhang, "Scale-aware trident networks for object detection", Proceedings of the IEEE International Conference on Computer Vision, 2019pp. 6054-6063 California, USA
[68]
W. Ma, Y.W. Wu, F. Cen, and G.H. Wang, "MDFN: Multi-scale deep feature learning network for object detection., Vol. 100", Pattern Recognit., 2020.
[69]
Z. Cai, and N. Vasconcelos, "Cascade r-cnn: delving into high quality object detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018pp. 6154-6162 Salt Lake City, United States
[http://dx.doi.org/10.1109/CVPR.2018.00644]
[70]
S. Zhang, X. Zhu, and Z. Lei, "S3fd: single shot scale-invariant face detector", Proceedings of the IEEE International Conference on Computer Vision, 2017pp. 192-201 Venice, Italy

Rights & Permissions Print Cite
Article Metrics
16
1
© 2024 Bentham Science Publishers | Privacy Policy