November 18, 2018

In the last post, I said I would post again once I had implemented some solution for fire detection and localisation. It may seem like it has taken forever, but there have been some problems along the way. And the project has a few more aspects that need to be completed fairly soon.

My robot is equipped with a Seek Thermal Compact thermal imaging device, which is supposed to detect thermal radiation from up to 1.000 ft with an FOV of 36° and a resolution of 206 by 156. Sounds like a horrendous resolution, however, the Compact is one of the few thermal cameras that exist in the lower price spectrum while still being useful. The only alternative I came across were the sensors from FLIR, though I believed the Compact would be easier to wire.

Though I may have been correct in that sense — merely requiring a converter from male micro USB connector to a regular USB cable to connect to my NUC — the major issue I came across were with driver compatibility. Initially, after having discovered a driver on github (libseekthermal) by a user named ethz-asl, though after actually setting up the camera, I could not get the driver to work. Instead, I tried another driver I found on github named libseek-thermal (confusing, right?), this time made by maartenvds. This driver proved more successful, finally providing me with the theral images I so desperately sought. The trouble, however, had only begone.

A test of the libseel-thermal driver and the Seek Thermal Compact

Once I had tested the camera with a few hot objects, I found that the driver was normailizing the temperature values in order to always have the same color spectrum (white/yellow for the hottest objects and blue/black for the coolest), meaning that as soon as I moved a lighter into the camera's FOV, I, who had previously been colored yellow and red due to my body temperature, would turn nearly black, along with everything else, while the lighter now received the coloring previsouly reserved for my body. While attempting to get rid of the normalization in order to have more control over the display, I discovered another major issue soon confirmed by the author: the driver could not provide absolute temperature values. That, of course, was a problem. How would I find fire if I could not get my thermal camera to provide me with actual temperatures and not just values that could somehow be converted into temperatures?

Luckily, I thought of a solution: I could loop over the picture and compare the values of neighboring pixels in order to find ones which differed greatly from its neighbors, suggesting a fire or similar cause of extrem but localized temperature differences. Before I was able to devise an efficient algorithm that would accomplish just that without slowing detection down to a snail's pace, someone in the issues section referenced a pull request for that project which seemed to implement absolute temperature readouts.

The pull request, however, was not perfect, as, after looking through the discussion in the request's comments, I discovered the reason for why reading absolute temperatures from the thermal camera was so difficult: there was a conversion algorithm, which Seek Thermal simply refused to release. People were measuring values produced when detecting objects at known temperatures, attempting to find the magical conversion between raw value and temperature, with increasing success, though the exact conversion remains a mystery, as factors such as the changing temperature of the camera during operation are extremely difficult to calculate. After some of my own modifications, I believed to have found a far from perfect but working solution. The readouts were far from exact, however, with common candles reaching temperatures in the hundreds of degress Celcius, an error of 10 degrees seems managable.

To detect fire, I was using OpenCV, a computer vision library, with provided me with two extremely powerful tools: Gaussian blur and minmaxloc. The first is used for preproccessing, where the image is blurred in order to negate the effect of damaged pixels, localized misreads or data corruption, which could skew the results of the second function. This function, minmaxloc, returns the position of the brightest pixel (which, in this case, is also the hottest). Since we applied the Gaussian bur filter before, this function should return the hottest region without interference. Once the position is known, we can simply determine if the approximate temperature is above a certain thershold, e.g. 120°, and act accordingly, sounding the metaphorical alarm or not.

Once a fire has been detected, the robot needs to know where it is. This part is currently sort-of working. First, I use the FOV, resolution of the camera and the offset of the hottest pixel from the center to calculate at what angle the fire is being detected. Basically, I determine the measure of the angles (x and y direction) formed by a line perpendicular to the camera and one connecting the camera and the fire. The angle describing the "left-right" offset of the fire can be used to turn the robot in the correct direction. The other angle has another use: using it, the offset of the camera from the ground (height) and some triganometry, it should be possible to calculate the robot's distance from the fire. Should.

This idea was suggested by a postdoctoral robotics researcher at TU-Darmstadt (the Technical University of Darmstadt) as a possible temporary solution. In theory, his idea is pretty good, and I thought it would be the solution I was looking for. The caculation was quite simple:
\(tan(angle)=\frac{distance}{height}\) \(\Leftrightarrow\) \(tan(angle)\cdot height=distance\)
However, when applied to an actual robot with a low-resolution camera and other factors, the result is less than ideal. It sometimes calculates a value close to the actual distance, however, it oftentimes seems to have a mind of its own. Also, as some of you might have figured out, since I am using the height of the camera as part of the calculation (building a right triangle), this only works for fires that are placed on the ground. It works good enough for a few tests, but not good enough for anything else, including submission to a competition. This is why I have a backup plan, also being inspired by my conversation with the postdoctoral researcher.

This backup plan is actually similar to some of my original ideas when planning this project, and it is related to the reason I had a RealSense R200 depth camera ordered. Originally, I wanted to create 3D maps of the robots surroundings for navigation purposes and as information for the fire brigade. I also belived that the 3D map would be used to localise the fire. While I will likely not construct 3D maps, although that is not off the table, thanks to the Octomap algorithm as well as Google Cartographer, I can use the camera to locate fire. If I can corrolate the information provided by the depth camera with that of the thermal imaging camera, I could find the distance and therefore the fire's location. Theoretically.

Before I can delve back into the technical aspects of the project, I also have to take care of some organizational issues. Since I am submitting the project to the German science fair (Jugend Forscht), I have to start writing a research paper. However, as the project is not yet completed, it is quite difficult to report on my accomplishments and methods: both work in progress. There is also the poster/booth that I have to start to design soon. SAT Subject tests are comming up, for which I have to study a lot since the German curriculum is incredibly different to the American one, basically requiring that I complete a two-year high school course in a few months while also having my own diploma and exams to worry about. I'm not even going to talk about college applications, which I also have to write, considering that I am applying to a few U.S. universities (a nightmare when having no college counselor and no one familiar with the American application process) as well as European ones. All in all, a busy schedule, but a rewarding one.