Object recognition (r)evolution
In this age of rapidly advancing and ever-accelerating technological progress, many ideas deemed impossible yesterday are transformed into reality today. In the domain of object detection, for example, Google’s best algorithms could only recognize a cat in a photo “just” 70% of the time in 2012. Now, Google’s AI is able to distinguish among 5,000 animal species, identify hundreds of dog and cat breeds and even manage a live phone conversation to book a vet appointment for you. This tremendous progress and other leaps in applied sciences builds our expectation for what we think technology should be able to deliver in the near future.
In such a technologically advanced world, it seems natural to think that a tech product will perform as expected or as described. For example, a smart parking, in-ground vehicle detection sensor should detect a car’s presence parked above it.
Ironically, back in 2012 when kittens were detected with 70% accuracy by leading AI scientists, the author of this blog first started to tackle the on-street parking problem with real-time availability data derived from sensors. The problem was daunting, so the approach was very pragmatic – solve it promptly, with the most modern and cutting-edge technology available. A few weeks later, when a pilot program with a leading smart parking sensor vendor was deployed, we noticed quite frequent false positives. With no car parking or driving activity whatsoever, the status of the sensors would change from unoccupied to occupied. The vendor explained that it was due to the electromagnetic anomaly of our pilot location. A week later, we noticed some cars were not being detected – “false negatives”. This was explained to us as vehicle type anomalies. This was quite bewildering, to say the least! The parking spot sensors did detect a significant number of vehicles, but, not every vehicle that was parked. Not all parking detection events translated to a certainty that a vehicle was parked. The combined observed parking detection accuracy was close to 70%, the same 70% that the best computer algorithm could boast when recognizing a cat in a picture.
A perfect trap for an electronics engineer
Unlike image processing and computer vision that require advanced math and statistics knowledge, the minimal viable product (MVP) in vehicle detection can be built based on one sensor’s readings and a quite basic “if-then” programming logic. Or so I thought.
One would think that with MVP showing very encouraging results – in the lab and in the field where it was tested with dozens of cars in a local parking lot – it could be a commercial product. Unfortunately, results in real-world deployments differed greatly. As it turns out, the real physical world is much more diverse and complex and holds a lot of “analog” surprises for digital electronics specialists that are hard to predict. The MVP approach of one vehicle detection sensor and relatively simple signal processing logic proves very hard to incrementally improve to meet the commercial demand for accuracy above 99%.
There is indeed a big gap between 90% and 99% detection accuracy in terms of electronics design, data science, and physics. Engineers on all continents seem to get caught in this trap – from North America to Australia.
To be fair, not every vehicle detection application needs an accuracy of 99%. With no data at all, even 90% accurate data is a huge leap. Traffic counting on highways often has a lower accuracy, but, still serves well in terms of providing valuable information about road utilization and average vehicle speeds. Interestingly enough, we recently discovered that a leading vendor of wireless sensors for highway vehicle counting made multiple attempts at developing a vehicle detection sensor for parking bays but was not able to achieve the accuracy and energy efficiency required for commercial deployment of these sensors.
Challenges of single sensor vehicle detection
There are plenty of physical principles and sensors that can be used for parking vehicle detection: magnetometers, infrared, ultrasonic, inductive, radar and others. We have seen more and more exotic modalities but they seem to have more challenges than value. In any case, any individual sensor has its own peculiarities and challenges that don’t seem to fit into a simple binary logic and a power profile of a battery powered device.
Just as an example, let’s consider a magnetometer, the most commonly used sensor for wireless vehicle detection system. As many are aware, a car containing ferromagnetic (iron alloy) parts distorts the Earth’s magnetic field (Figure 1). These distortions can be captured by a vehicle detection sensor, but as can be seen on the drawing, cars have some areas of positive and negative displacement and almost always have spots in between where magnetic field deviation is close to zero.
This picture also explains why traffic counting is a simpler task than parking bay vehicle detection. A vehicle driving over the sensor will trigger it with positive and negative displacements, while in parking availability detection, the sensor may get into the zero (blind) zone where magnetic field changes are minimal.
To highlight further the complexities encountered when it comes to using sensors for parking detection, it is important to note that a real-world city has many sources of electromagnetic interference. Additionally, new electric vehicles are so light and efficient that they often have carbon fiber chassis and no iron alloy (ferromagnetic) parts whatsoever.
With all the vehicle and environmental challenges affecting parking monitoring system, combining 2-3 different sensing technologies significantly increases the reliability of the overall vehicle detection sensor design. More sensor data entails more complex signal processing and statistical algorithms that take more time to develop and validate but, eventually, such approach greatly improves the occupancy data quality.
Smart parking confidence
Now that we have a better, multi-sensor car detection technology that is working, how many individual parking event tests do we need to pass to prove that our system has 99% accuracy?
Many think 100-1,000 individual parking events is all that’s needed to prove the efficacy. The reality is that a far greater number of tests are required.
If we want to present results in a statistically correct form, we need to use the correct and appropriate terms. Confidence interval and confidence level are very useful statistical concepts.
The confidence interval is the plus-or-minus error usually reported with the results. For example, if you have a confidence interval of 2 and 97% of your test sample give positive results you can be sure that on a much larger scale, positive results will be within the 95-99% range.
The confidence level shows how sure one can be that larger scale observations will be within confidence interval range. It is expressed as a percentage and most applied researchers use the 95% confidence level.
There is a very simple approximate formula for a 95% confidence level linking test sample size (N) and confidence interval (B):
N = 1 / B2
So if we want to state our statistical results with 95% confidence with +/-10% error (B=0.1) the number of samples required N = 100, for +/-5% accuracy (B=0.05), we will need N = 400.
99% detection accuracy can be meaningfully stated with confidence interval of 1% (B=0.01) which means N = 10,000.
Making 10,000 tests or observations with different individual parking events is a very time-consuming research effort. A bit harder than identifying 10,000 cat photos on the internet! Not nearly as much fun either. Due to this, Nwave developed a special in-house mobile app to speed up the collection and analysis of the 10,000 individual parking events minimizing the human input factor.
We have seen a number of companies claiming an impressive 99% detection accuracy. Unfortunately, none of them provided sample size, the methodology of data collection or other information supporting their claims.
We hope this post will stimulate a broader technical discussion about detection accuracy and help customers make informed choices. In the meantime, you can read more about the pros and cons of different types of sensors used in smart parking systems in our another post.
Keep tuned in to our blog to learn how accurate real-time parking space management is changing the world!