Performance evaluation of autonomous weeding robots - FIRA 2019 - Challenge Rose

performance evaluation
 of autonomous weeding
 robots

FIRA 2019 Rémi Régnier (LNE), remi.regnier@lne.fr

 Goal : encourage the development of autonomous innovative
 solutions for intra-row weed control in field crops with wide
 spacing and vegetable crops in order to reduce by 50% the use of
 phytosanitary products, and thus contribute to the achievement of
 the objectives of the Ecophyto II plan.

 Inter-row

 Intra-row

 Crops

 The ROSE challenge goal 2

Develop solutions
 4 projects funded
 BIPBIP WeedElec
 Participants
 Contribute to the definition of the
 scientific and technological PEAD
 ROSEAU
 objectives of the challenge

 Leads the definition of competition
 objectives and ensures that they are
 measurable
Operational organizer
 (trust third party) Organizes and leads the challenge

 Ensures fair treatment of participants

 Finance the challenge

 Funding body
 Statue on the objectives of the
 challenge

 Challenge participants

Launch of the first evaluation
 campaign
 06/2019

 Meeting to present the results of the dry-
 run
 Opening of the ROSE challenge 06/2019
 01/01/2018
 Launch of the second evaluation
 Challenge kick-off meeting campaign
 02/28/2018 01/2020

 Validation meeting of the evaluation plan - Launch of the third evaluation
 Launch of the dry-run campaign campaign
 06/05/2018 06/2020

 12/31/2021
 2018 Jan. May Sept. 2019 May Sept. 2020 May Sept. 2021 May Sept. 2021
 10/2018 05/2020 05/2021
 Field meeting 1 - Dry-run Field meeting Field meeting
 10/2019
 Field meeting
 05/2019
 Field meeting 2 - Dry-run

Dry-run campaign 06/2018 - 06/2019

First evaluation campaign 06/2019 - 01/2020

Second evaluation campaign 01/2020 - 06/2020

Third evaluation campaign 06/2020 - 07/2021

 The macro planning of the challenge

Four evaluation campaigns

 Operational
Six meetings in the experimental field organization

An area of four hectares dedicated to
experiments
 Organisation opérationnelle du challenge ROSE

 Operational organization of the ROSE challenge

AgroTechnoPôle site : Irstea Montoldre
 Plot challenge ROSE

 Irstea Montoldre Site

 Fields meetings

•Detect and identify plants
Detection

 •Decide on the action to be
Decision taken

 •Carry out the weeding action
 Action

 Three key steps to evaluate

Types of crops planted : Types of weeds planted:
• large crop with wide spacing: maize spread out (horizontal):
 (row spacing 75 to 80 cm, foot spacing spread out (horizontal) :
 14 cm) o Model weeds : mustard
• field vegetable crops: beans (row o Natural weeds : matricaria.
 spacing 15 to 30 cm, foot spacing 3 to 8 with upright (vertical) :
 cm) o Model weeds : ray grass
 o Natural weeds : goosefoot.

 Maize Beans

 Mustard Matricaria Goosefoot

 Crops and weeds

Prototype presented by Pead in September 2019
Prototype presented by BIPBIP in September 2019

 Détection

Prototype presented by ROSEAU in September 2019
 Prototype presented by Weedelec in September 2019

 Detection evaluation

Participant Camera Light Resolution Surface d θ α β r
 RGB Artificial 5 Megapixels 45cm*5 40cm 0°
 1
 (DEL) (5 pixels/mm) 5 cm
 Visible + Natural 50 cm 60° 0° 0°
 2 hyperspectral
 (Carbon Bee)
 RGB + Infrared Natural 1024*768 2m*1.3 1.3 m 0°
 3
 pixels m
 RGB Natural 5 Megapixels 25°
 4 (night (1.5mm/pixel)
 excluded)

 Provision of  References
 Metrics
 services Error analysis and
 Definition of the comparative
 of the data performance
 evaluation task between hypothesis estimation
 and test
 and references
 environments  Hypothesis

 Four technologies for one evaluation

Comparaison
 References : manual annotations 1. Mapping
 2. Calculation
 of the error
 rate

 Plant of
 Acquisition of images by interest
 the 4 evaluated robots Hypothesis : outputs from detection systems
Objective: determine the position of weeds and/or plants of interest on the
images

 Detection evaluation

Metric

Evaluation via the EGER metric:
 
 =1 + + 
 = 
 =1 

 : costs of confusion on the image k
 : false alarm costs on the image k
 : costs of forgetting on the image k
 : number of plants detected in the reference
(weeds and plants of interest)

 Detection evaluation

Development and use of the DIANNE software

Next steps :
• January 2020: presentation of the results of the first campaign
• Presentation of the results of the first campaign
• Availability of the four annotated databases during 2020 (250 images with
 minimum annotations per technology).
• New evaluation in June 2020

Possibility to use the parcels for image acquisition on request from IRSTEA
Montoldre

To follow the progress of the challenge : http://challenge-rose.fr/

 ROSE Challenge

Thank you for your attention

Influencing factors Controllability Robustness test Measurements
 made
 Weather (rain, wind,...) No No Daily measurements by
 weather station
 Brightness No - During the image- Measurements by
 based detection task luxmeters when
 Agro- - During the field participants pass through
pedoclimatic detection task
 conditions Soil moisture content, No No Daily measurements by
 temperature, useful water ground probes
 reserve
 Clay rate measurement Yes (constant) No Measurement before the
 first meeting
 Described before the
 Technical itinerary Yes (constant) No
 start of the campaigns
 - During the field
 Crop density and detection task Taking pictures before
 Test mode distribution
 Yes
 - During weeding each meeting
 tasks
 - When detecting on
 Stage of plant development No Daily image capture
 the image database

 Global influencing factors

Title Bloc-outil et Imagerie de Perception Et binage RObotics SEnsorimotor loops Robot de désherbage localisé par
 Précision pour le Binage autonome des cultures en to weed AUtonomously procédé électrique haute tension
 Intra-rang Précoce Agriculture Durable combiné avec une gestion prédictive
 par vision hyper-spectrale et post-
 évaluation par drone
 Project acronym BIPBIP PEAD ROSEAU WeedElec

Coordinating body Laboratoire de l’Intégration Research institut Xlim SITIA (Engineering company) UMR Itap Information, Technologies,
 du Matériau au Système (UMR CNRS 7252, multi- Analyse environnementale, Procédés
 sites Limoges, Poitiers, agricoles
 (IMS, UMR5218 CNRS,
 Brive, Angoulême)
 university of Bordeaux, (Irstea, Montpellier SupAgro)
 Bordeaux INP) Team REMIX
 Teams COMIC and PEPS
 Team MOTIVE
Academic partners  Bordeaux Sciences  CNRS  INRA (UMR  Irstea
 Agro Agroécologie)
  Université de  CIRAD (AMAP, UR AIDA )
  Bordeaux INP Limoges (Xlim)  IRSEEM
  INRIA ( ZENITH, LIRMM)
  CNRS
  INRA (UMR EMMAH/UAPV)
  Université de
 Bordeaux (IMS, Labri
 équipe Rhoban)
 Technical and  Les Fermes Larrère  CARBON BEE Les chambres régionales  AGRIAL
economic partners d’Agriculture de Pays de la
  Elatec  SABI AGRI
 Loire et de Bretagne
  CTIFL

 Participating consortia
 to the ROSE challenge