There are many aspects of UAV (drone) technology that require further research and development, especially in the context of developing Open Source-based solutions suitable for use around the world. Our labs page covers some of the work that OpenRelief volunteers and other projects have done in key areas.
Drones provide an “eye-in-the-sky” but need to be combined with other technologies to provide complete solutions. For example, with image-processing software a drone can “recognise” roads, people and smoke. OpenRelief worked on this technology and produced test code using the open source project called OpenCV.
Here is an example of how we taught a computer to see:
Here is an example of how we taught it to recognise what it was seeing:
Here is an example of how it learned to understand what it was seeing:
Putting it all together, here is an overview of our code recognising roads, people and smoke:
Naturally all of our code is freely available. You can find it on Gitorious.
Command and Control
An important part of managing drones is the “Command and Control” technology. This is the hardware and software that you use to manage the missing planning before the drones take off and the extraction of information after they return.
The simplest version of this is the ‘Ground Control Station’ used to program an autopilot. A great example is Mission Planner for the open source ArduPilot autopilot that we use. Mission Planner is very useful to put waypoints into the drones and to see precisely where they went in post-mission summaries.
However, a disaster relief drone needs more software or features to integrate with disaster management technology. A key example would be Sahana Eden, an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors, and which is used in response to disasters around the world.
Importing and exporting information from a Ground Control Station into a disaster management platform is a problem that has yet to be addressed effectively in the open source drove eco-system. In the OpenRelief project we considered addressing this through the creation of a little bridging server that sits in the middle and talks with everything via RESTful interfaces. Below you can find our notes on how the Drone Manager could work, though please note that the test system has not been created yet.
Drone Manager is conceptualized as a little web server. It can contact Mission Planner to load/unload data from drones. It can send mission data to Sahana Eden. Ideally Drone Manager will be very light, it will be very modular, and we can start from a small base of critical functions to gradually add more ability.
The most basic function is listed below under (1) in the use-cases section, and it basically just has Drone Manager calling Mission Planner, loading and unloading mission data from autopilots, and sending mission GPS waypoint data back to Sahana Eden.
Drone Manager Use Cases
The Drone Manager will have three use-cases, and they can be understood both in order of development priority and time to deployment.
We expect the first generation of designs to be load mission, go and return without ground to air interaction. Moving ahead, the ability to stream some data will be useful, starting with the simple “where is the drone now” and moving towards “what is it seeing?” The later end of that is likely to rely on communication hubs we will prototype off Tegra2 boards, which can make use of multiple bands via software defined radio, as well as GSM and similar methods of obtaining data from the planes and deployed sensors.
Putting that in context, here are the Drone Manager (our drone control server software) use-cases, and they can be understood both in order of development priority and time to deployment.
- Direct laptop connection to drone autopilot via USB (one to one). This is the starting situation, where one laptop will load a mission to one drone, the drone will fly away into a zone with no live communication, and will return to download telemetrics back to the laptop. Vanilla Mission Planner can do this, and I would assume that Drone Manager would do the same simply by calling Mission Planner via RESTful API to load and download data.
- Direct laptop connections to multiple drone autopilots via USB (one to many), as above, where the drones fly and return to dump mission data. This is the first type of fleet management we are likely to see in practice, and appear to involve Drone Manager calling Mission Planner via RESTful, Drone Manager having the expansion to support multiple drones via public key ID, and Drone Manager helping you visualize where the drones went and what they saw using its modules/plugins.
- Drone Manager is running at HQ and is able to talk with networked drones in the air. These are identified and secured using the public key ID system. The drones can be loaded with missions over the network, they can send back data and can (regulations and hardening permitting) have their missions altered in the air. This is the advanced use case, and it will involve a balancing of technology with regulations. It’s likely to be first deployed in a sort of (2a), where the drones get missions loaded physically, but provide live data back over the air to HQ and other interested parties over the network.
Sahana Eden Integration
The drones are quite complex. They have GPS data, telemetrics and imaging, with some of the former being able to tag the latter. A substantial amount of the complex data from the drone is managed via software called Mission Planner. That basically allows you to tell the drone where to go, and it provides a blackbox of what the drone was up to. Conceptually, we then plan to build a little server called Drone Manager that acts as a bridge between the drone itself (autopilot/computer), the mission planner, the processing done at HQ (if any) and external systems that need the GIS data (i.e. Eden).
The important point is that Eden should never talk with a drone to ensure that (a) you guys don’t get a headache as we change and twist around lower level stuff and (b) we don’t get a headache trying to upstream things which don’t actually help your mission. It would be great if we could push data to Eden through a RESTful module called by the Drone Manager. The way George and I discussed this so far, the Drone Manager will just be a little server calling the Mission Planner via RESTful, so push/pull data via RESTful to Eden fits right in.
In summary, if we conceptually have the Drone Manager pushing and pulling data from Eden, that allows us to send our waypoints and data, and it allows export of “make this point a waypoint for a drone” notes in Eden.
Conceptualizing Data Transfer
Understanding what the data going back and forth is an important point of working out the practical bridge between Drone Manager and Sahana Eden.
Some things are pretty clear.
- On a basic level, there will be missions, there will be drones, there will be locations, there will be times and there will be associated images and/or flags (“I saw smoke”).
- The Drone Manager should be able to deliver all that type of data to Eden so that relief workers know what went where, when, and why. I’m not sure if we need to send any more data than this, as Eden has all the general relief mission planning, task management and so on needed for relief worker in general. We don’t want to duplicate or pull that data. We just want to make sure relief workers know a drone mission was carried out, they can access the data from the mission, and they can identify the physical units involved in the mission.
- The export from Eden side is probably a lot thinner, consisting of the “make this point a waypoint for a drone” flag that the drone managers can load into the Mission Planner, and work out new missions with optimal waypoints/range, and perhaps a “refly mission” flag to request a simple waypoint reload on Mission Planner for reviewing previously seen land.
- One area where we probably need to duplicate is the “who assigns the mission ID” feature. Eden probably needs to be able to assign mission IDs for its normal workflow, but we probably also need to be able to create/import/export mission IDs in the Drone Manager, especially in contexts where people are not using Eden. This is where we have to be aware that while Eden is a primary target for interoperability, the OpenRelief solution needs to also function without it.
- It therefore sounds like we need to have common table modules for data import/export on both sides to allow us to push/pull data from each other seamlessly. An OpenRelief API? 😉
Drone Manager should be responsible for Mission IDs, talking with the drones and all interaction with the Mission Planner software.
Sahana Eden should be able to request “Add this point to a future mission” and “please (re) survey this area.”
Overview of the Proposed Architecture
The major components are:
- The Mission planner is existing software as outlined above.
- The Drone Manager is new software that acts as a bridge between the Mission Planner and Sahana Eden.
- Sahana Eden is the Disaster Management solution we are initially targeting. It has its own database of assets, people, etc.
- The OpenRelief module in Sahana Eden consists of the plugin to Eden that is required to link to the Drone Manager to Eden.
The Open Relief module has a number of responsibilities to perform:
- It should provide the data models used to store the GIS and mission data supplied by the Drone Manager. These models define the tables within the Eden Database.
- It should provide the controllers to allow access to the models to both humans – via an HTML interface, and to machines via a RESTful interface.
- Any necessary views to display the information.
The Eden architecture will provide a substantial amount of the controller implementation, including the RESTful interface. This leaves the bulk of the work in defining the data models and the views.
Map Data Visualisation
The OpenRelief module will also need to define the layers e.g. smoke, so that data points can be displayed on a map. Additionally, the we need to link “pins” on the map back to the original data, and possibly as “refly this location” option.
Actual implementation of data transfer between Drone Manager and Sahana Eden
It is proposed that the implementation takes place in two stages:
- We export a CSV file containing the mission data from the Drone Manager into Eden. Eden in turn will export a CVS file containing a list of way points that need to be re-flown. The Drone Manager should import this file.
- Once this has been proved to work and the model structures on both sides have settled, the full RESTful interface can be implemented on both the Eden side and the Drone Manager side.
On the Eden side a RESTFul interface should be presented that supports CRUD operations on the mission’s GIS data. Eden should return an Event ID for each mission event to the Drone Manager. On the Drone Manager side a RESTful interface should be presented that allows Eden to create a list of way points that should be used in future missions, and request that a specific mission is re-flown.
If it is done this way then on the Eden side the table structures can be defined and redefined as needed. These tables will be reused via the RESTful interface that Eden presents to the Drone Manager. The Drone manager uses this interface to push mission data into Eden via CRUD style interface. Using CSV files, initially, will also decouple the Eden OpenRelief module from the Drone Manager. Likewise on the Drone Manger side, if it initially imports way-point data via a CSV file, until a RESTFul interface is fully defined.
In both cases while the RESTFul interfaces are in a state of flux, dummy data can be served by serving static response files from the web server.
Defining a data transfer API
We need to clarify what data we want to pass to Eden from the Drone Manager. One suggestion below:
- a) For a given sensor reading we need:
- i) time (in UTC – you can take this from the GPS)
- ii) position of “event” as lat, long (+ altitude?)
- iii) mission ID
- iv) event id (note from Fran at Sahana: “UUID?”
- v) Type of event (smoke/fire/person/radiation/weather etc)
- vi) reference to further event data, if any (note from Fran at Sahana: “We’d do that the other way around normally – the further data would refer to the event by event_id (the further data is a ‘component’ of the main resource)”
- vii) drone ID (does Eden need this?)
- b) For each event the data I think is different.
- i) In the case of smoke/fire/people we need the image data
- ii) Radiation needs the sensor value
- iii) Weather needs??? (as a suggestion: current Temp, max temp, min temp, wind direction, and speed?)
User Interface Mockups
A series of prototypes have been created to serve as a basis for discussion about the way users will interact with the Drone Manager.