Autonomous NASA Mars helicopter has a snafu, confusions through, providing lessons for self-driving AI . .. cars. fbs-accordion> Images
Have you ever looked at a snapshot and asked someone when they took that photo ?
I'm sure you did.
You probably wanted to place the image in a date and time context. Maybe the photo was taken years ago and showcases the past. Or maybe the image is fairly recent and shows the way things are today. Overall, know when a photo was taken can be useful and sometimes essential.
In the IT fieldWe often speak of a timestamp.
When a computer is connected to a camera, taking a photo is usually accompanied by adding a time stamp to the collected image. The timestamp simply indicates the date and time of the image. This can be inserted into the data that contains the actual image or can be added as an additional data element that describes or otherwise indexes the photo.
If a series of photos is taken, the timestamp starts extremely important.
Let's explore why.
Imagine you own a car with a camera mounted on the dashboard and pointed towards the roadway. You opt for a road trip one and decide to have the camera periodically take pictures of the road in front of you. The image after the image is captured. No big deal, easy.
During the ride, you find yourself at one point in a residential area and a dog may be wandering across the street. Luckily, you see the dog and stop to let him move safely. Shortly after, a toddler runs across the street. Since you have already stopped, the child can run down the street without incident. There was nothing particularly bad about the event and you could put it down to a simple everyday car ride.
A few weeks later, you tell someone about your general driving experience. When you start to explain the difficulties, you suddenly cannot remember if the dog appeared first and then the child came next, or maybe the child was the first to cross the street and that the dog followed the little one. It would have been easy to confuse the question relatively withoutincident and the particular sequence in your mind.
Aha, you have these images stored in your dash camera!
You download the images to your laptop.
Going up each of the images, let's assume that there is no apparent timestamp. This means that each photo was without a definitive indication of the date and time it was taken. You can clearly see each image and you can attest that these are accurate representations of what you saw during your driving effort. Unfortunately, they lack a timestamp.
It would be almost as if you had scattered the images on a table and had to determine which came before which other. This can be a difficult puzzle to solve.
Indeed, you find the photo which shows the dog which was in the street, and you find the photo which shows the toddler which was in the street. But you don't know aBe sure which photo is first in order. Damn, that doesn't help you solve your quest to remember the sequence of these events.
As mentioned, timestamps can be crucial.
If you were really determined, you could take a close look at each of the two photos. It is possible that the child in the photo has the dog in the middle of the street. If the child is on the side and is running away from the street, you can deduce that the child probably left first and the dog followed. Likewise, if the photo of the kid in the middle of the street shows the dog walking towards or away from the street, you can try to deduce the sequence that must have happened.
It certainly would. much easier to have the timestamps.
Well you dig in and find out there is a timestamp built into the image data. You use a special program to find out the timestamp. The question ofdetermining the sequence seems to be solved.
Life is never this easy, of course, and you suddenly notice that the timestamp has a date of June 31, 1777, which doesn't. makes absolutely no sense. . You know it can't be right. The times shown on each of the images indicate an evening period, although you will know for sure that you have encountered the dog and toddler during the day.
Yes, the timestamps are skewed.
How willing are you to trust timestamps?
For example, you can ignore the date and assume that it was just wrongly preset. You could ignore that the time indicated was in the evening and assume that the clock was not correctly set at the start. At least you can see if the timestamp via the time shown on each photo will reveal which photo was taken first.
Are you willing to accept the idea that it will be a irrefutable proofthe to indicate which photo was taken first and therefore whether the dog or the child was the first to enter the street?
Looks like you'd be on slippery ground. The fact that the date is wrong would be worrying. The fact that the times are apparently wrong in terms of accurately rendering the time of day is also troubling. At this point you are going to assume that at least the footage is correct due to the timestamp showing once in the photo of the dog and again in the photo containing the toddler.
If you were only doing this quest out of innate curiosity, it doesn't matter whether you are willing to accept that the timestamp time was likely appropriate and now you know what event happened first. Would you be as optimistic if the question was of vital importance? Imagine that for some reason this is a major problem and needs serious fixing.
I guessthat we could play that ever popular kid's game of asking if you'd be willing to bet your life on it. A bit sketchy and not something over which a life and death decision would seem solidly viable. The photos clearly seem to show that you did see a dog and that you did see a child, something almost indisputable. The unmistakable sequence is unfortunately still a bit in the air.
Speaking of being in the air, let's turn our attention to the subject of autonomous helicopters. We'll come back to the whole issue of photos and timestamps after going through that extra terrain.
You may be aware that NASA has put an autonomous helicopter on Mars. The helicopter is part of the global mission of the Mars 2020 Perseverance rover. With the catchy name of Ingenuity, the autonomous helicopter has already set new records based on its flights to Mars, and the NA teamsSA should rightly be proud of the accomplishments thus undertaken.
The reason the helicopter is referred to being autonomous involves the aspect that it has to fend for itself when it Fly. If the human controllers here on earth tried to pilot the craft directly, the delay due to distances and ti mes transmission would not be adequate. By the time the human pilot here saw what was going on there and then issued a drive or pilot command, which was then to be transmitted and received, the helicopter could have encountered a problem that completely ruined the helicopter. 'distant machine.
Having developed an autonomous piloting system, Ingenuity can almost stand on its own.
That being said, it doesn 't imply that the craft is going willy-nilly where it wants to go. NASA and the Jet Propulsion Laboratory (JPL) have established predetermined missions and each of these missions has been carefully planned and prepared.
A series of missions are being performed. Each mission has specific goals of what needs to be accomplished. As the missions progress they get a little more complex each time, almost like crawling first, then walking and then running (or, in the case of a helicopter, short vertical flights of up and down followed by longer distances and horizontal multipath flights).
This brings us to Flight Six and an interesting in-flight anomaly that could have caused the autonomous helicopter to dive into the surface Of March. If this were to happen at any point during the Mars excursion, you can probably assume that the helicopter went out for the count. There would be no ready means to repair the machine. Alone, he would sit still, having done his part for science, and be a silent marker saying that humans have been here.
What happened to ingenuity?
I comI will start by pointing out that he was able to complete the assigned mission and exists, still intact, and ready for the next mission (well, with some adjustments to be made remotely).
Halfway through the sixth mission, the craft began to roll and pitch disturbingly. This rapid tilt and speed adjustments can be seen in video recorded on Mars. Anyone watching the recorded video must feel their heart tighten since the autonomous helicopter appears to have gone mad and appears to be flying erratically. You would probably assume that something horrible has happened to Ingenuity and that his demise (sort of) is imminent.
The basis of Savage Flight is somewhat complicated.
There is one word though that sums up the general problem: Timestamps.
Whoa, you have to be thinking, weren't we discussing timestamps a while ago? What a coincidence! In fact, the goal of this previous sagaon the timestamps was to prepare you to identify what happened to Ingenuity on Mars during its Flight Six.
Buckle up and let's jump into the fray.
When the autonomous helicopter flies, it tends to use a downward-facing camera which is usually pointed towards the surface of Mars. Every thirty seconds or so, a photo is taken. Each image is analyzed by computer by an on-board computer image processing system. Over time, a series of images can help determine where the craft is located, as well as changes in speed, altitude, attitude, position, etc.
In particular, this includes comparing one photo against another. Photo.
Tip: Remember the previous story of the photos taken of the toddler on the street and the dog on the street. That will come in handy in a moment.
This particular camera on Ingenuity is generically referred to as navcam, which means that it is a camera which is mainly used to aid the navigation of the boat. The computer that autonomously drives the system uses the images and analyzes of the images to help determine where it is, where it is heading, and more. This analysis is coupled with other navigation capabilities, including the use of an inertial measurement unit (IMU).
With a little drum roll, I now introduce you to the official statement on the anomaly (according to the NASA website): "About 54 seconds after the start of the flight, a problem occurred in the p ipeline of images delivered by the navigation camera. This issue resulted in the loss of only one image, but more importantly, it resulted in the delivery of all subsequent navigation images with inaccurate timestamps. From that moment, whenever the navigation algorithm performed a correction based on a navigation image.on, it was operating based on incorrect information about when the image was taken. The resulting inconsistencies significantly degraded the information used to pilot the helicopter, leading to estimates that were constantly "corrected" for phantom errors. Great oscillations ensued. "
I hope you can see how the analogous discussion about using a car dashcam and having to figure out if you've seen a dog and then a toddler, or if it was the toddler and then the dog, applies to this circumstance.
According to the official deion of what Ingenuity did, it appears that an image was "lost " during the capture or processing stages, and this kind of prompted or fed the rest of the images to be assigned "inaccurate" timestamps. This caused confusion for the rest of the autonomous piloting on the navigation aspects.of the gear. It doesn't seem entirely clear yet what the timestamps were and how they were misallocated or misaligned, but we'll just take this as it is for now and continue with that.
You could be. I wonder why the craft didn't become so out of whack that it crashed.
Glad you asked.
One aspect Distinct but integral to Ingenuity's autonomous piloting is that it apparently tries to keep the craft within certain preferred or reasonable operating thresholds.
This is often some kind of fail-safe mechanism for autonomous vehicles (AV).
If everything else goes crazy, at least the main driving or steering component is usually meant to keep the vehicle from drastically deviating from normally expected parameters. Programmers usually include defined thresholds which, in ordering to stay within these limits, and even s 'There is another internal component trying to force the vehicle out of these limits, the main steering system refuses to do so and tries to counter actions by blocking or counteracting what is happening.
According to NASA 's explanation of how Ingenuity overcame the time stamp snafu: "One of the reasons he was able to do this is the effort considerable amount that has been deployed to ensure that the helicopter's flight control system has ample "margin of stability ": We have designed Ingenuity to tolerate large errors without becoming unstable, including errors This built-in margin was not entirely necessary in previous Ingenuity flights, as the vehicle behaved as expected, but this margin came to the rescue in Flight Six. "
There was also a real lucky rabbit's foot that helped too.
When Ingenuity reaches the final stage of a flight and begins its descent, the navcam is no longer actively used for navigation purposes. This makes sense due to the likelihood that dust will be thrown as the autonomous helicopter approaches the surface of Mars and obscure or render the images unusable or unreliable. NASA 's deion is worded as follows: "This design decision also paid off during Flight Six: ingenuity ignored the camera footage in the final moments of the flight, ceased to 'swayed, leveled his stance, and landed at the expected speed. "
We could quibble a bit about this last aspect. As suggested, this might be more a matter of luck rather than a deliberate design basis. It seems highly unlikely that a situation envisaged is that the navcam itself could cause difficulty and that on landing, it would therefore make sense to no longer use it.iser. Instead, in this case the other reason for not using the navcam when the pier started became unintentional and yet a very welcome helping hand.
Let's stop there and let's take a short breath of relief about Ingenuity.
Additionally, a notable rule of thumb regarding any engineering effort stems from the immortal words of Thomas Jefferson, whom he has stated to be. 'He was a great believer in luck and that the harder he worked the more he seemed to have.
Some succinctly claim that luck is the result of good planning.
Now that we've covered the issue of an autonomous helicopter operating on Mars, let's turn our attention down to earth. There are going to be all kinds of autonomous vehicles here on earth, including autonomous helicopters, autonomous drones, autonomous trucks, autonomous ships, autonomous cars, etc. For convenience, considerer these autonomous vehicles as autonomous.
I would like to see what lessons we can take from the Ingenuity situation and apply them to the advent of true self-driving based on AI. driving cars.
Self-driving cars are driven via an AI driving system. There is no need for a human driver behind the wheel, and there is also no provision for a human to drive the vehicle. For my full coverage of Autonomous Vehicles (AV) and in particular self-driving cars, see link here .
Here's an intriguing question worth thinking about: How to do timestamps and processing image apply to real autonomous cars based on AI and anomaliCould e similar to ingenuity happen to a self-driving car?
Before I go into details I would like to clarify what is meant by reference to real ones self-driving cars.
Understanding the levels of self-driving cars
By way of clarification, the real self-driving cars are the ones that AI Fully drive the car on his own and there is no human assistance during the driving task.
These driverless vehicles are considered level 4 and level 5 (see my explanation on this link here ), while a car that requires a human driver to share the driving effort is generally considered to be level 2 or level 3. Cars that share the driving task are described as semi-autonomous and usually contain a variety of automated add-ons which are called ADAS (Advanced Driver-Assistance Systems).
There is no real self-driving car at level 5 yet, which we do not even know yet whether this will be possible to achieve, and nor how long will it take to get there.
Meanwhile, Level 4 efforts are gradually trying to gain some traction by undergoing very narrow and selective street testing, although there is some controversy as to whether whether these tests should be allowed per se (we are all living or - guinea pigs dead in an experiment that takes place on our highways and roads, some claim, see my coverageto this link here ).
Since semi-autonomous cars require a human driver, adopting these Car types don 't be significantly different from driving conventional vehicles, so there is not much new per se to cover on this topic (although, as you will see in a moment, the following points are generally applicable).
For semi-autonomous cars, it is important that the public be alerted to a worrying aspect that has emerged recently, namely that despite these human drivers who continue to post videos of themselves falling asleep while driving a level 2 or level 3 car we all have to watchavoid being tricked into believing that the driver can distract from the driving task by driving a semi-autonomous car.
You are the party responsible for the actions of driving the vehicle, regardless of the level of automation that can be added to level 2 or level 3.
Self-driving cars And time stamp issues
For truly self-driving level 4 and level 5 vehicles, there will be no human driver involved in the driving task.
All occupants will. be passengers.
The AI does the driving.
One aspect to be discussed immediately involves the fact that the AI involved in the driving systems of the AI of today is not sensitive. In other words, AI is quite a collective of computer programming and algorithms, and most certainly not capable of reasoning in the same way as humans.
Why does this accent sumore about the fact that AI is not sensitive?
Because I want to stress that when I discuss the role of the AI piloting system, I do not attribute no human qualities to AI. Please note that there is a continuing and dangerous trend to anthropomorphize AI these days. Essentially, people attribute a human sensitivity to today's AI, despite the undeniable and indisputable fact that such AI does not yet exist.
With this clarification , you can imagine that the AI piloting system will not work. natively somehow "know" the facets of driving. Driving and all that it involves will have to be programmed as part of the self-driving car's hardware and software.
Let’s break down into the myriad of aspects that come into play on this topic.
A suitable starting point involves the occurrence of an anomaly while an autonomous vehicle is in the field and essentiallyso on the way. This is a bad time for things to go wrong.
In the case of an autonomous Mars helicopter, there had likely been a great deal of design, construction, and careful testing long before the AV was sent to Mars. Yet despite rigorous efforts to identify in advance any potential problems that might arise, a fairly serious problem nonetheless arose.
Some might get heartburn referring to the problem as a so-called 'anomaly' which perhaps uses nauseating or cowardly semantics to obscure the aspect that it seems. be an outright error or bug in the system. The least alarming wording via the harmless "anomaly " seems to tone down the skepticism that developers, management, and the development process let the loophole pass and ultimately crop up while the contraption was at least at its fingertips. 34 millions of miles away. from home and in the middle of his mission.
Questions which, on a debriefing basis, would certainly seem justified, include how did one abandoned image lead to the consequent series of dysfunctional outcomes? Would an abandoned image not be considered part of the base design and would it have been anticipated? If not, at least it apparently should have been a test case. If this was a test case, what happened during the test? Didn't it reveal later issues associated with the timestamp? Perhaps he did but was not noticed, for which this alone is of concern. What provision did the design include for checking or validating timestamps? Weren't there some tests involved that deliberately changed the timestamps to see how the rest of the system would react? And so on.
Anyway, hFortunately, the AV was able to stay aloft and land without damage or being destroyed. We can be grateful for that.
This however seems to have happened relying on global security rather than having a separate worded provision to deal with this specific error or bug. A tote saved the day. Likewise a distinct feature which had no particular relation to the insect and perhaps to the skin of the teeth provided (unintentional) protective cover.
The fact is, The depth and intensity of design and testing that takes place for a craft like Ingenuity is some of the most impressive and robust work done on any AV of any type. In contrast, for some of the self-driving car development efforts that exist, the quality and intensity of the design and testing is not as thorough and exhaustive.
In short, if this tybug pe can slip through and end up in the final system of a particularly meticulously designed AV, we must keep a close eye on the daily efforts of self-development - the cars that drive.
One can also wonder about the consequences aspects in the case of an autonomous car. An autonomous helicopter shot down on Mars as a test experiment would apparently be bad, sad and disappointing, but no one would have died. An autonomous car here on earth that encounters a bug or serious error on the public road could spell disaster in terms of the possibility of having a fatal car crash, injuring the occupants of the autonomous car, as well as the possibility of injuring or killing pedestrians and other drivers in other nearby cars.
This covers the somewhat generic issue of hidden bugs or errors and the need to show them to the public. advance and excise them orless than having targeted provisions specifically designed to deal with it (in addition to, and not in place of, overall integrated security capabilities).
That being said, is there a risk loss of footage from video cameras used for autonomous driving cars?
You may be wondering. There might be no chance that a similar error will occur for self-driving cars.
Sorry to say that there is a non-zero chance that this will happen, or more clearly there is indeed a chance that it will. One could easily argue that there is a good chance that this will happen, for
Is there a chance that the time stamps used in self-driving cars will go wrong one way or another?
But this isn This does not necessarily imply that the various self-driving car efforts are all consciously focusing on these specific potential problems and therefore devoting substantial resources to those particular types of errors or bugs. Keep in mind that many AI development teams are already out of breath and are just trying to get their self-driving car project to move successfully from point A to point B, safely and without incident. If there are qualms about dropped images or a snafu with timestamps, this is probably a considered low-chance possibility at this time and not attracting inordinate attention just yet.
Maybe the ingenuity snafu will come in handy waking up.
Some insist that self-driving cars should be fully tested on private, closed tracks or proving grounds before beinge allowed to enter public roads. Some also insist that self-driving cars should be fully tested via computer simulations, before being allowed on public roads. Presumably, a combination of simulation and testing ground would seem relatively satisfactory for these camps (I have discussed this overall topic at length, see my columns).
Regardless of the security arrangements integrated that every automaker or auto driving technology company is selectively designing a solution sufficient to overcome unforeseen errors or bugs?
I don't know. I can't say for sure.
We know a lot, that the life of this dog and toddler running on the streets might depend on it.
Go ahead and put an indisputable timestamp on this solemn thought.