KEY TECHNOLOGIES and FEATURES

The stack we are putting together to support AP30 requirements includes:

• Apache Cassandra Database. The database will house the full record details and will store the results of duplicate & outlier detection. It will also provide the persistence for the record vettings provided by Edgar. 
• Apache SOLR search indexes. These indexes will support the searching capabilities required by Edgar.
• A processing chain implemented in Scala. This will include the algorithms for detecting duplicate records and environmental outliers. This custom code will then update search indexes to allow Edgar to filter for non-duplicates, and non-outliers hence improving the quality of the model and reducing the number of records to be vetted. The code for this component is accessible in the google code repository http://code.google.com/p/ala-portal/
• Java Spring MVC web services. These web services will provide the interface for the Edgar project to download snapshots of data for modelling and vetting purposes. They will also provide a write interface for submission of the vettings of bird records by expert users. The code for this component is accessible in the google code repository http://code.google.com/p/ala-portal/. An additional important functional requirement is for Edgar to be able to query for record deletions. Services will be developed to allow Edgar to keep track of these deletions that occur periodically when the ALA harvests from data providers.

PROJECT OUTPUTS #2 - DUPLICATE DETECTION

As a part of a range of data quality checks the Atlas has identified potential duplicate records within the Biocache.   This allows users to discard duplicates records from searches, analysis and mapping where this is appropriate.  A discussion on duplicate records is available here. 

The ALA uses the scientific name, decimal latitude, decimal longitude, collector and collection date to indicate potential duplicate records: 

• Records are considered for a species (with synonyms mapped to the accepted name).
• Collection dates are duplicates when the individual components are identical (year, month and day). Empty values are considered the same.
• Collector names are compared; if one is null it is considered a duplicate, otherwise a Levenstein distance is calculated with an acceptable threshold indicating a duplicate
• Latitudes and Longitudes are duplicates when they are identical at the same precision. Null values are excluded from consideration.

When a group of records are identified as duplicates one needs to be identified as the "representative" record.  The representative record can be used to represent the entire group of duplicates and should not be considered a duplicate. 

Here is an example "representative" record in the ALA: http://biocache.ala.org.au/occurrence/3cde1570-7a38-4a58-b121-e95c35585a29#inferredOccurrenceDetails:

There is a JSON web service that returns the complete information that was used in the duplication detection:
http://biocache.ala.org.au/ws/duplicates/3cde1570-7a38-4a58-b121-e95c35585a29

It is possible to perform a search for all the records that are duplicates of a specific representative record: http://biocache.ala.org.au/occurrence/search?q=duplicate_record:3cde1570-7a38-4a58-b121-e95c35585a29

Duplicate records can be excluded from the queries submitted to the biocache by applying the negative filter &fq=-duplicate_status:D.  Example: http://biocache.ala.org.au/occurrences/search?taxa=Mountain%20Thornbill&fq=-duplicate_status:D