A software framework for agricultural and forestry robots

3.1 Logical view

Defines top‐level functionality, general modules and their relationship. In our case, this is mostly described in the specification documents for the CROPS robots. According to these, the robots should be able to determine locations of ripe and healthy fruit in the vicinity of the robot, determine appropriate picking order, plan a route to individual fruits, move to a fruit, determine how to grip a fruit, grip a fruit, determine how to cut a fruit, cut a fruit, and finally bring a fruit to a basket. All this should be done in a safe manner both for humans working close to the robot, and for fruits and plants. The same software framework should work for development of different robots for harvesting of apples, sweet pepper, and grapes. Based on this specification, the following generic functional modules are identified:

• •

  Main control program. Runs the main loop that detects fruits, plans and executes motion, gripping, and harvesting.

• •

  Virtual sensors. Abstractions of sensors that do not only measure the physical world, but also process results from one or several physical, or virtual, sensors.

• •

  Planner. Generates plans for picking order, grasp patterns, and possibly also motion planning.

• •

  Arm and gripper control. Provides an interface to the robot arm (Baur et al., 2012), gripper, and cutter.

• •

  Error manager. Detects and handles situations when things go wrong.

• •

  Resource manager. Lets the user tune and configure the system for a task (choice of sensors, algorithms, parameters, etc.).

• •

  Graphical user interface (GUI). Allows a user to start, pause, stop, and inspect the robot.

• •

  Fusion/learning. Creates and adapts virtual sensors.

• •

  Performance monitor. Checks the health of all modules and communication channels (for instance physical connections and data flows).

3.2 Development and process view

The development view describes the organization of software modules and how they communicate to fulfil non‐functional requirements such as performance and availability. This description specifies programming and software development paradigms such as object orientation, component based approaches, and model driven design. The process view describes issues related to concurrency, distribution versus centralization, and communication modes (such as point‐to‐point or publish‐subscribe). In the CROPS project, most development is done using the robot operating system (ROS) environment (Quigley et al., 2009), with programming done in C++, which encourages object‐oriented systems. ROS manages parallel execution of software modules (denoted nodes) and administrates ethernet based communication between nodes. ROS also supports transparent physical relocation of nodes and contains other useful functionality. In our case, most aspects of the development and process views are jointly described by a directed graph, with vertices representing ROS nodes, and links representing information flow (ROS message passing). The example in Figure 2 shows a small part of the system configured for a fruit harvesting robot in the CROPS project.

3.3 Physical view

This view describes placement of physical components and physical connections, and takes into account non‐functional system requirements such as availability, reliability, performance, and scalability (Kruchten, 1995). In our case, this view is illustrated as a block diagram showing how laptops, sensors and actuators communicate through a standard ethernet bus with possibility to connect also CAN based equipment (Figure 3). The hardware is used to implement the functionality described by the Logical view, and also the application specific functionality according to the specification documents. Thanks to the ROS environment, the physical location of software modules within the ROS domain (red box) is very flexible. This means that processor‐intensive computations, if necessary, can be moved to separate laptops without any changes in the programs. All laptops, sensors, and actuators communicate through a standard ethernet bus. A gateway to CAN equipment is, if necessary, available through a CAN/ethernet interface.

3.4 Robot architecture

We further develop the architecture by specifying and designing how the different modules function and interact. This development is affected by design choices described in the logical, development, process and physical views. The result is a robot architecture that follows the hybrid paradigm. The logical view of this architecture is shown in Figure 4.

The major refinement concerns the design of the Main Control Program (Figure 1). A state machine replaces the planner component commonly found in hybrid architectures. This is viewed as a good solution for the CROPS robots, and for other robots where there is no need for planning in the sense of problem solving or finding novel solutions to new tasks. This is the case if the behaviour of the robot is simple and well defined, and can be described in for instance a flow diagram. A part of a simplified flow diagram for a sweet‐pepper harvesting robot is shown in Figure 5. This flow diagram is transformed into the state diagram in Figure 6. The complete state diagram is implemented as a state machine in the robot. At each time step, the current state returns an “outcome”. Depending on the “outcome”, the state machine moves to another state or remains in the same state at next time step. In Figure 6, the label of each arrow represents a specific “outcome” of the state at the root of the arrow. The tip of the arrow points at the state that will become active at next time step.

The proposed architecture differs from the standard hybrid model also in the way behaviours are defined and interact with the state machine/planner. Each state is typically associated with a behaviour. When the state machine moves to a new state, the corresponding behaviour is activated. Behaviours are implemented as ROS nodes and execute independently of the state machine, such that the latter at each time step can decide to change state or stop execution. This is particularly important when integrating user interfaces and error handling. Behaviours can be computational and/or acting. The computational ones are not connected to any actuators while the acting ones are.

The relation between states and behaviours is shown in Figure 7. In the figure, the “Move_home” state is connected to “Move arm behaviour”, which in turn communicates with the “Manipulator” node, which is connected to the physical manipulator. The outcome and other results computed in a behaviour are communicated to the corresponding state and can also be stored such that they are made available to other states and behaviours. One example is the computation and usage of coordinates of fruits. The “Locate_fruits” state communicates with “Fruit localization behaviour”, which computes locations of fruits by using data delivered by the “Camera” node and the “Range sensor” node. The result is stored such that the states that decide on picking order (Select fruit) and move to a fruit (Pick fruit) can access them (Figure 6).

Properties of states, and conditions for transitions between states, are defined in a “Transition file” in text format. This file is read by the state machine at run time and provides a flexible and easy to use configuration tool when developing new robots. One example is shown in Figure 8. The contents of the transition file, shown in the rectangle, specify that the state machine will stay in state “Move_home” as long as the outcome of the state is “Moving to home”. When the outcome changes to “Home”, the state will change to “Locate_fruits”.

The “GUI” contains controls to start, pause, and stop the state machine. It also displays informative messages retrieved from the behaviours and other modules. A “Virtual sensor” is an abstraction of a regular sensor, and connects to one or several physical or virtual sensors. For the fruit harvesting robots, one important virtual sensor communicates with one or several cameras and identifies fruits in the images. The output of this virtual sensor is read and further processed by the “Locate fruits” behaviour. The “Resource manager” provides customization of the virtual sensors such that the system can be adapted to varying environmental conditions (such as changing weather) or changing tasks (such as harvesting a new type of fruit). At present, this kind of customization is done off‐line and stored in configuration files that are read at system initialization.

Errors are dealt with at two levels in the system. Some errors are both detected and dealt with locally where the problem appears. One example is if a node responsible for fruit localization is not able to find any fruit in the image. The node may deal with this by acquiring a new picture and trying again, moving the camera or platform, or calling for human assistance. Another example is if the robot arm fails to reach the coordinates of a fruit selected for harvesting (for instance if an obstacle blocks the motion, or if the fruit is no longer visible for the robot). As shown in Figure 6, the state “Pick_fruit” may detect this error by returning outcome “Out of reach”, causing a state shift to “Select_fruit” where another fruit is selected for harvesting.

Other errors are independent of the current state, and are more efficiently detected and dealt with at system level. The “Performance monitor” monitors the status of communication lines and hardware such as sensors and computers. If an error is detected, messages are sent to the “Error manager” that takes appropriate action. One example is a camera that suddenly stops functioning. This may be detected by the “Performance monitor” and forwarded to the “Error manager”, which then stops the state machine (and thereby the robot) and prints an error message through the GUI. Another example is an emergency stop of the robot arm if something or someone gets in the way. The “Error manager” may also tell the state machine to change state if an error is detected. The “Error manager” may sometimes need complex decision‐making, and the correct action for error recovery may depend on the current state. Just as we use a static state machine to control the normal execution of tasks, the “Error manager” may be implemented as a state machine.