Personal Mobility Enforcement : A MAP-API and Mixed Reality Approach with Drones for Effective Traffic Monitoring

Ⅰ. Introduction

Since the launch of the first shared electric kick personal mobility service in South Korea in March 2018, the number of electric kick personal mobility users has increased dramatically.

However, along with this increase, there has been a rise in accidents due to violations of personal mobility usage rules, such as not wearing head gear and riding with multi-passenger. Electric kick personal mobility-related accidents have surged from 49 cases in 2016 to over 258 cases in 2018, an increase of more than five times. Recently, a newterm “킥라니” (a portmanteau of “킥보드,” meaning personal mobility, and “고라니,” meaning raccoon dog) appeared, referring to the sudden appearance of electric kick personal mobility on the streets^[1]. Personal mobility requires an on-the-spot crackdown due to the absence of license plates on kick personal mobility. However, The police can not be on-the-spot crackdown because of the current traffic police manpower faces lack.

A comprehensive survey of 242 police precincts within the Seoul area revealed that 105 out of 242 locations (43.4%) had insufficient personnel. Some police stations are still experiencing shortages ranging from 2 officers to 20 officers^[2]. As of December 2013, the population per the local police officer in Gyeonggi was 635.4 people, the issue which traffic police man is lack is increased. So, it make to the difficulties in enforcing traffic regulations^[3]. To solve this issue, the Seoul Metropolitan Government has been consistently augmenting its workforce. From 2012 to 2022, over a decade, the city has increased its police personnel by approximately 13.2%. Nevertheless, there still remains an insufficient workforce relative to the demand^[4].

Furthermore, the police tried to use real-time communication via social networking services (SNS), which allows communication without constraints of time, location, or distance in order to resolves the shortage of personnel and personal mobility enforcement issues through citizen engagement. However, due to the absence of a specialized department handling SNS tasks, this approach led to another manpower shortage issue by inadequate training and additional task^[5]. Therefore, to effectively resolves the police personnel shortage problem, efficient distribution of on-site enforcement tasks is necessary. Police officer tried to use for cracking down traffic in highway in order to solve the personnel shortage problem. In fact, the cracking down traffic performance has doubled with a drone in highway^[6]. A drone has advantage that the drone can crack down without traffic jams. However, personal mobility basically is on an urban road so it's hard to crack down with drones. If the drone doesn't follow the correct path, it can crash. So, it needs a correct and efficient paths and control in order to using a drone. However, even if the above problem is solved, it is very difficult to control the drone, so considerable effort and knowledge are required to control the drone.

This paper proposes a Map-Application Programming Interface (API) to addresses the aforementioned issues. The proposed system provides with a traffic monitoring system using wearable devices, virtual interfaces such as maps and buttons with Mixed Reality (MR) technology. If the proposed system is used, the drone can be operated based on the correct route to lower the drone accident rate and share the police's work. In addition, MR based on intuitive UI was used to reduce the difficulty of drone control. Mixed Reality is a technology that combines Virtual Reality (VR) and Augmented Reality (AR)^[7]. MR combines the advantages of VR, which presents users with a fully virtual environment, and AR, which overlays digital content onto the real world. In MR environment, users can interact with virtual UI on their actual surroundings. If we Utilize the characteristics of MR, users can conveniently use digital content within the system. Through the proposed traffic monitoring system in this paper, the evasion of enforcement through dynamic patrolling can be prevented. Simultaneously, roles between the police officers and surveillance cameras can be divided. It leads to savings in manpower and surveillance camera installation costs.

Ⅱ. System

The system architecture in this paper is illustrated in Fig. 1. The wearable device connects with the Map-Application Programming Interface (Map-API) via Wi-Fi. Users can interact with virtual interfaces, such as maps and buttons, as well as utilize voice commands through the wearable device. As the wearable device does not have the GPS sensor, it relies on a mobile device as an assistance component. Through voice commands, users can control the drone and update maps effectively within the interface. However, the current voice control tool with does not support the Korean language^[8]. Users can not use voice commands to set departure and arrival points. But, they can interact with the virtual interface's start and end points using tap gestures to configure the information. Once the information is set, the user requests the route from the Map-API and search it. The received routes are transmitted to unmanned aerial vehicles(UAVs), such as drones. Additionally, users can see real-time UAV camera view on the wearable device's screen. To control the drone, the widely used Robot Operating System(ROS) middleware is used.

Fig. 1.

System Structure

Ⅲ. Contents

The proposed system is divided into three main parts: the MR-based navigation system, the Deep Learning component, and the drone control section.

3.1 Navigation Contents based on MR

The proposed system uses the characteristics of MR to provide users with convenient access to the Map-API. The virtual interface on the wearable device is divided into three sections: drone control, map, and camera footage. Fig. 2 represents the virtual button interface for drone control, Fig. 3 represents the virtual map interface, and you can see the camera footage interface in Fig. 4.

Fig. 2.

Button UI

Fig. 3.

MR Map UI

The virtual interface buttons shown in Fig. 2 are composed of a total of 9 elements. The user's scenario for employing the virtual interface can be observed in Fig. 5 and the functions and corresponding voice commands for each button are outlined in table 1.

Users can receive GPS information from their mobile devices. Utilizing the received GPS data, users can update the virtual map and set details for the starting and destination points.

Button 1 is responsible for updating the virtual map centered around the user's location. As previously mentioned, because the wearable device dose not have its own GPS sensor, a mobile environment was added. A mobile app is created in order to transmit the user's phone GPS sensor information to the wearable device. Using the received GPS data, users can view the updated virtual map, which includes the user's current location, a drone position, and starting and destination points. Users can manipulate objects on the virtual map to set the starting and end points. By pressing Button 2, the user sends the information of the starting and destination points to the Map-API and receives navigation information. GPS data is extracted and transmitted. Users can view the received GPS-based route information by utilizing button 4. The map information is displayed on the virtual map in a line in Fig. 3. Button 3 allows users to transmit GPS information to the unmanned drone.

Fig. 4.

Camera Streaming UI

Fig. 5.

Scenario

Buttons 5, 6, 7, and 8 are associated with drone control. Button 5 sends takeoff message to the drone, and button 6 enables users to autonomously fly the unmanned drone based on the route information.

Button 7 stops the drone's movement. Button 8 send land message to the drone. By pressing Button 9, users can exit the wearable device's app. Furthermore, all buttons can be operated using voice commands. The voice commands related with each button's execution are listed in table 1. Fig. 4 displays a screen that streams camera images.

The wearable device receives video information from the camera which is attached to the drone. Through the Screen shown in Fig. 4, users can observe the real-time drone camera view. The proposed system in this paper detects illegal activities related to personal mobility based on camera footage from drones. Deep Learning is used in order to identify these unlawful actions. The training and inference processes are carried out on the GCS because GCS has a high performance compared to the wearable device. The performance of the GCS used in this paper are detailed in table 2.

A custom Deep Learning model was trained for the recognition of illegal personal mobility activities in this paper. The 'You Only Look Once Version 5 (YOLOv5)' model was used to identify individuals violating kick personal mobility traffic regulations. 'You Only Look Once (YOLO)' exists in several versions and is known for its real-time object detection capabilities and fast processing speed. Additionally, to expedite the training process, 'CUDA' and 'CUDA Deep Neural Network (cuDNN)' were employed in conjunction^[11][12]. CUDA is a toolkit that consist of a GPU-accelerated library, debugging and optimization tools, and runtime libraries. cuDNN is a GPU-accelerated library that extends CUDA to include deep neural networks. When CUDA and cuDNN are used together, GPU utilization is optimized, resulting in accelerated training. So, GPU for faster training is used with CUDA and cuDNN.

The classes used in this paper comprise persons engaged in multiple-person riding on a kick personal mobility, people not wearing helmets, and individuals correctly riding the personal mobility

The unmanned drone detects helmet usage and whether there is more than one person on the personal mobility (indicative of illegal kick personal mobility activity). Subsequently, the drone transmits this detected information to the MR wearable device.

When the MR device receives information about illegal user of personal mobility devices, the user can see a warning message.

3.2 Drone Control

In this paper, the drone used is the Parrot Bebop 2.

The specifications of the Bebop 2 are provided in table 3, and its appearance is illustrated in Fig. 6.

The Bebop 2 drone is controlled using the ROS Driver provided by Parrot^[13]. Since the offered ROS Driver supports only Ubuntu 16.04, Docker is used for compatibility^[14]. Communication between the Bebop 2 drone and the GCS is connected by Wi-Fi. The provided ROS Driver's nodes are utilized to control the drone. It receives target information from Deep Learning and performs tracking accordingly. The drone's camera identifies the target object, retrieves the size and position information of the Bounding Box (B-BOX), and maintains a consistent B-BOX value for object tracking during flight.

Fig. 6.

Bebop 2

Ⅳ. Test

Images containing individuals riding personal mobility were used as the dataset to evaluate the performance of the Deep Learning model. The model was trained by Yolo, and a set of 100 test images was utilized to assess the average accuracy across the three classes. The specifications of the testing PC are provided in table 4. The training and testing were conducted using YOLOv5 on the testing PC.

4.1 Test Environment

The Test Dataset consists of a total of 10,605 samples. While existing datasets capturing kick personal mobility users from a walker's viewpoint existed, they were not suitable for drone-based capture due to the differences in altitude. This altitude problems can potentially affect Deep Learning accuracy. Consequently, a new dataset was captured from the drone's perspective, combined with the existing dataset, resulting in an initial dataset of 707 samples. Labeling was carried out using labelImg^[15]. The classes consist of people without helmets, people with helmets, and people riding with passengers, totaling three classes. Data augmentation was performed in order to enhance accuracy and prevent overfitting^[16]. Data augmentation is techniques such as rotation and horizontal flipping to increase the dataset size, improving accuracy and mitigating overfitting. Through data augmentation, the initial dataset of 707 samples was expanded 15 times, resulting in a final dataset of 10,605 samples.

In this paper, the YOLOv5 Small (Yolov5s) model was used for training. Real-time recognition of personal mobility users was the goal. So, this system needs fast estimation speed. Therefore, for testing, both the nano model and small model with faster speeds were considered. However, due to the significantly lower accuracy of the nano model compared to the small model, the Yolov5s model was selected as the pre-trained model for training. The completed training PR curve is depicted in Fig. 7. The overall mAP@0.5 is approximately 88%, with mAP for Helmet at 82%, No-Helmet at 91%, and Two at 90%.

Fig. 7.

PR curve

4.2 Test Result

Test images from a dataset were used for evaluation.

Fig. 8 is one of the results obtained from testing the 100 images. The user can know the recognition result with the bounding box as shown in Fig. 8. Fig. 9 is the result graph of test and Table 5 is a details of test result. The set of images consisted of 35 Helmet, 46 No-Helmet, and 19 Two instances. From the test results, there were errors in 6 out of the 100 images. In these error cases, the model classified the images as Helmet or No-Helmet based on the lower body of individuals. It leads to incorrect predictions.

Fig. 8.

Test Result

Fig. 9.

Test Result Graph

Ⅴ. Conclusions

A system that utilizes Map-API and MR wearable devices is proposed and implemented. Users can autonomously give commands to the drone through voice commands and tap interactions for autonomous driving. Additionally, proposed system can enforce traffic regulations related to personal mobility users using Deep Learning. This system aims to reduce personal mobility accident rates and solves the absent problem of the traffic police manpower.

The proposed system relies on outdoor autonomous drones. So, this system requires the use of GPS sensors essential. However, GPS sensors have coverage limitations as Fig. 10, exhibiting an error range of 60 to 300 feet (approximately 18m to 91m). This limitations can lead to collision risks on roads with many obstacles^[17]. To overcome these limitation of GPS sensors, we will develop to enhance the system by incorporating depth sensor-based obstacle detection and computer vision-based lane recognition. This approach aims to develop a strength traffic enforcement system that can handle GPS inaccuracies and sudden situations effectively.

Fig. 10.

GPS Accuracy Error

Currently, the system is in the development and communicates over limited short-range external Wi-Fi. However, for real-time use, we will try to expand the communication range by equipping drones with Access Points (APs). And user who gets a wearable device can communicate with a drone in more wide range than before.

Moreover, this system will be extended the system's versatility by incorporating functions like detecting illegal parking and patrolling areas near school zones.

This will lead to the development of a regulation the drone patrolling system, connecting a single MR device with multiple patrolling drones. It aims for a more efficient the drone-based traffic enforcement system.