Jetson Nano and AI / Vision

 

It would be nice if a solar drone could detect ships for boarder patrol, people in the water for rescue, and clouds for maximizing solar collected by the plane.  It could then plan a route to get closer pictures of ships or people and avoid clouds.

 This paper on ship detection also compares Jetson Nano to Raspberry Pi and shows that Jetson gets more processing per watt.   We have thought this too and have been playing some with Jetson Nano.   In ArduPilot a Jetson Nano can be the companion computer for both 4G cell data and for image processing/AI stuff.  It can run at 5 watts and is about 135 grams.   

The Jetson Nano is about $100. It has a big brother the Jetson Xavier NX that is about the same but with a fan and 15 watt top instead of 10 watt.  The NX costs about $400 and has about 10 times the AI performance of the Nano.  Though with the NX you probably also want a fast SSD, which could add another $100 or so.  After we get good enough that don't lose airplanes too often, might upgrade to the NX.

 


 We are collecting links to work on some of the vision problems that we are interested in below.

 

Boat or Ship Detection

      There is plenty of work on using computer vision to detect ships.

 

Cloud Mapping

       We would like to know where the clouds are and where the blue sky is so that we can navigate a path with plenty of solar power.   Searching for "cloud" gets so many references to "the cloud" and "cloud computing" type things that it is hard to find any work in this area.


Ocean Rescue 

       Would like drone to be able to recognize someone in the water or a ship that has partially sunk.   In Anguilla ships are often made of wood and even when they "go under" they are often still near the surface and so could be recognized.

       I fear that getting a sizable dataset of partially sunken ships or people who are not in the water because they want to be will be hard.   Without  a good dataset, training a neural network is not going to work well.

 

Airplanes, whales, schools of jacks

      It would be good to be able to recognize airplanes, whales, or schools of fish that local fisherman catch.   With Yolo it can recognize many different things at the same time.   So it could just be flying alone and notice a whale...


Distance to object

      It would be nice to be able to report the GPS location of a ship that our system recognizes.   To do this we need to estimate the distance and angle from our known GPS location.    If we record a direction, then move to the side a couple hundred meters and measure the direction again, then crossing these two lines we can estimate a distance.

Object Detection/Following/Tracking

    If we can click on a boat and have the airplane fly to that boat that could be very helpful.   In this case it would not need to learn in advance what a boat looks like.   There are demos for this so this could be the easiest way to go.

     If we lock onto a boat we could do a stabilized digital zoom.  So if we have a 12 Mp camera, we might make an image that was 1 Mp but just the area of interest and stabilized.   With tracking and Jetson power we could do this.  

      Would be fun to have one airplane following another with the one behind using a gimbaled camera and AI to keep it aimed at the one in front.  Or just a fixed camera but the AI keeping the plane aimed at the one in front and the right distance behind.

NVIDIA Links

Raspberry Pi Links


Raspberry Pi is lighter.  Some versions use less power than Jetson.  Most don't seem to have the power of Jetson Nano and use liter models.   It might be possible to get things to work with it.  However,  I think that it will be more fun to experiment with Jetson Nano as it clearly has enough computational power.   It may be that a future Raspberry Pi 5 or 6 is the way to go though.

Google 

The Coral USB Accelerator can work with Raspberry Pi  - also - light and low power 

Benchmarking

FC Software on Linux

  Linux now has some realtime options - so running flight controller software linux, like Jetson Nano uses, may be reasonable now or soon.  This could save weight and be simpler for both hardware and software.  There is a Raspberry Pi Hat called Navio2 and Ardupilot has been ported to this.  The Jetson nano does not have an IMU but you can buy an IMU for $15.  Oh, you also need compass, gps, and barometer.   It seems like with realtime and and a set of sensors it should be possible to use the Jetson Nano as a flight controller.  But maybe not just yet.


Cameras - Jetson has CSI and USB connectors

    The Osmo Pocket 2 has a 64 mp sensor that can digitally zoom. It is possible for a Jetson Nano to control an Osmo Pocket 2

Arducam 12MP IMX477 Pan Tilt Zoom(PTZ) Camera  A motorized optical zoom would be nice

Arducam Mini 12.3MP HQ Camera for Nvidia Jetson Nano  $65

IMX179 8mp 10x Optical Zoom Camera Module Motorized - 57 grams $200 

USB  5.1~47mm motorized zoom lens camera for day/night- 63 grams $250 Euro, seems 2 megapixels

    Using a high MP camera and having Jetson scan the whole image and digitally zoom in to something interesting seems good.   Longer term it would be nice to also have an optical zoom.   But these are more pricey and so we would want to have very reliable survival from airplane missions before adding them. 


Streaming for public

    ipcamlive.com -  with adds will stream video for free, monthly charge otherwise

Comments

Popular posts from this blog

Switching from iNav to Ardupilot for Flight Controller

Crowd Funded and Crowd Researched

Stabilized Camera / Pan-Tilt