Consumer drones have become significant safety hazards for critical infrastructure, disrupting airports and airplanes, while also raising privacy concerns, such as unauthorized recording. Hence, in this paper, we explore the problem of detecting UAVs based on sparse point cloud data coming from a rosette scanning LiDAR sensor. Besides distinguishing between background and foreground elements, a major problem is filtering out false-positive detections due to birds and other flying objects. To address this issue, we integrate a classification network into our solution along with a semantic segmentation network, which is responsible for highlighting the foreground objects for the classifier. Our algorithm utilizes the attention mechanism throughout the object detection phase to efficiently capture the global features of the scene, while it benefits from the convolutional layer’s local feature maps during classification. Experimental results show that our solution is capable of real-time object detection and recognition while maintaining its robustness.