Content Discovery

Searching for content is a functionality that is primarily supported by Coordinating Nodes, programmatically through the CN_read.search method, and through a web browser interface that connects through a user interface implemented by Mercury which utilizes a SOLR index to support search operations.

This document describes the properties of the SOLR index, how it is populated, and how it can be used to support programmatic search and introspection into the DataONE holdings.

The SOLR index is populated from content that is copied to Coordinating Nodes, that is System Metadata, Science Metadata and Resource Maps. Each index entry represents information about the information referenced by a single identifier (PID). For PIDs that refer to science data objects the index entry will be constructed from system metadata and zero or more resource maps that reference the object. If a PID identifies a science metadata object, the index entry will have fields populated with content extracted from the system metadata for the object, the science metadata document and zero or more resource maps that reference the science metadata. Similarly, if the PID identifies a resource map, then the index entry fields for that PID will be populated by the System Metadata and resource map values for that PID.

The general sequence for indexing content is illustrated in Figure 1 below.

The general process of indexing content involves retrieving a System Metadata document, parsing system metadata properties such as permission rules to generate a SOLR index document, then optionally adding to the document if the PID refers to a science metadata or resource map object. The general sequence of operations is outlined in Figure 1. If the PID refers to a resource map, then it is necessary to update the documents and isDocumentedBy properties of the Science Metadata and data objects that appear in the data package defined by the Resource Map.

../_images/7c80f112dbe31d6f92902ab2830a8af0a5ac427e8cbf2083676a0db6e8b1448d.svg

Figure 1.

A full list of index fields is provided in Table 4 below.

Querying the SOLR Index

The SOLR query syntax is based on the Lucene query syntax and adds a couple of extensions. The SOLR search index is exposed directly as a SOLR search service with the base url of:

http(s)://server.name/solr/

System Metadata Index Properties

The main goals for indexing system metadata include:

  1. Support access control rules for search results

  2. Support efficient discovery of objects by their properties

Table 1 provides a list of SOLR attributes that are populated by values taken from System Metadata.

Table 1. Elements derived from System Metadata and available for all objects. “Multi” fields may be populated with more than a single value per document (PID) indexed.

System

Attribute

Type

Multi?

Content Origin

'SOLR'

‘FieldType’

‘Multi’

‘Reference’

'blockedReplicationMN'

‘string’

‘Yes’

Types.ReplicationPolicy.blockedMemberNode

'checksum'

‘string’

‘No’

Types.SystemMetadata.checksum

'checksumAlgorithm'

‘string’

‘No’

Types.Checksum.algorithm

'dataUrl'

‘string’

‘No’

‘Resolve REST URL created from PID’

'datasource'

‘string’

‘No’

Types.SystemMetadata.originMemberNode

'dateUploaded'

‘date’

‘No’

Types.SystemMetadata.dateUploaded

'dateModified'

‘date’

‘No’

Types.SystemMetadata.dateSysMetadataModified

'id'

‘string’

‘No’

Types.SystemMetadata.identifier

'numberReplicas'

‘string’

‘No’

Types.replicationPolicy.numberReplicas

'formatId'

‘string’

‘No’

Types.SystemMetadata.formatId

'obsoletes'

‘string’

‘No’

Types.SystemMetadata.obsoletes

'preferredReplicationMN'

‘string’

‘Yes’

Types.ReplicationPolicy.preferredMemberNode

'replicationAllowed'

‘boolean’

‘No’

Types.ReplicationPolicy.replicationAllowed

'rightsHolder'

‘string’

‘No’

SystemMetadata.rightsHolder

'size'

‘long’

‘No’

Types.SystemMetadata.size

'submitter'

‘string’

‘No’

Types.SystemMetadata.submitter

'authoritativeMN'

‘string’

‘No’

Types.SystemMetadata.authoritativeMemberNode

'replicaMN'

‘string’

‘Yes’

Types.Replica.replicationMemberNode

'readPermission'

‘string’

‘Yes’

Types.AccessRule.subject where permission = read’

'writePermission'

‘string’

‘Yes’

Types.AccessRule.subject where permission = write’

'changePermission'

‘string’

‘Yes’

Types.AccessRule.subject where permission = changePermission’

Populating Permission Fields

This process is fairly straight forward, and requires simply de-duping entries so that each of the permission fields contains a distinct list of subjects granted that permission. If the DataONE public user is present in the read permissions, then the isPublic field is set to True.

Example Queries

The following provide some example queries that may be issued directly against the SOLR index operating on the Coordinating Nodes. Note that search terms must be properly escaped for the SOLR query syntax, and additionally if submitted directly as part of a URL, must also be properly URL escaped.

  1. Index entries that match PID (should be at most one match):

    id:"PID"
    

    where PID = identifier of object being located

  2. Index entries with PID that starts with “some_prefix”:

    id:"some_prefix*"
    
  3. Index entries that match a particular format id:

    formatId:"format_a"
    

    where format_a = the formatId being located

  4. Entries with formatId matching “fmtid_1” or “fmtid_2”:

    formatId:"fmtid_1" || formatId:"fmtid_2"
    
  5. Entries with byte size less than or equal to 10000:

    size:[* TO 10000]
    
  6. Entries with byte size less than 10000:

    size:{* TO 10000}
    
  7. Objects modified before 09:56:04 on 2012-01-03 UTC:

    datemodified:{* TO 2012-01-03T09:56:04.000Z}
    
  8. Objects modified less than 10 minutes ago:

    datemodified:[NOW-10MINUTE TO *]
    
  9. Objects of a formatId equal to “format_a” and modified within one day:

    formatId:"format_a" AND datemodified:[NOW-1DAY TO *]
    
  10. Metadata associated with data objects of relevance to Photosynthesis:

    photosynthesis AND documents:[* TO *]
    

Properties of the Index Derived from Resource Maps

Table 2 provides a list of SOLR attributes that are populated by values obtained by parsing resource map objects.

Table 2. Elements derived from Resource Map documents. “Multi” fields may be populated with more than a single value per document (PID) indexed.

Resource

Attribute

Type

Multi?

Content Origin

SearchMetadata.'SOLR'

‘FieldType’

‘Multi’

‘Reference’

SearchMetadata.'isDocumentedBy'

‘string’

‘Yes’

‘None’

SearchMetadata.'documents'

‘string’

‘Yes’

‘None’

Populating Object Relation Fields

The fields resourceMap, documents, and isDocumentedBy contain identifiers of other objects in the DataONE system related to the subject of the current record. These relationships are recorded in OAI-ORE resource map documents as described in the section Data Packaging. Since the values for these fields are contained in resource maps, it is necessary to build these fields in progression as additional resource maps are synchronized with the coordinating nodes.

For each object, a query is issued against the index to retrieve the list of resource maps in which the object identifier is present, enabling population of the resourceMaps field for the object. Each of those resource maps is examined to obtain the list of identifiers for the documents field if the object is science metadata, and the isDocumentedBy field if the object is data.

Example Sequence for Index Population

The following sequence steps through the generation of index entries for three data packages, providing an example of how the index entries are updated depending on relationships between the objects being added.

Adding Package 1

digraph RM1 { graph [rankdir="LR", label="Package 1"]; A [shape=folder]; B [shape=note]; C [shape=oval]; A -> B; A -> C; }

A simple data package with a Resource Map A indicating that the metadata document B and the data object C together form a data package, and that B documents C and that C isDocumentedBy B.

Index entries after Package 1 is added.

PID

resourceMaps

documents

isDocumentedBy

A

B

A

C

C

A

B

Adding Package 2

digraph RM1 { graph [rankdir="LR", label="Package 2"]; D [shape=folder]; B [shape=note]; E [shape=oval]; D -> B; D -> E; }

A simple data package with a Resource Map D indicating that the metadata document B and the data object E together form a data package, and that B documents E and that E isDocumentedBy B. Note that the metadata document B is also used in Package 1.

Index entries after package 2 is added. New entries and changes to existing are indicated in bold.

PID

resourceMaps

documents

isDocumentedBy

A

B

A, D

C, E

C

A

B

D

E

D

B

Adding Package 3

digraph RM1 { graph [rankdir="LR", label="Package 3"]; F [shape=folder]; G [shape=note]; D [shape=folder]; F -> G; F -> D; }

A data package that references another, already indexed data package (represented by the Resource Map D) as its data object.

Index entries after package 3 is added. New entries and changes to existing are indicated in bold.

PID

resourceMaps

documents

isDocumentedBy

A

B

A, D

C, E

C

A

B

D

F

G

E

D

B

F

G

F

D

Limitations of Multi-Valued Fields in SOLR

Multi-valued fields in SOLR have an internal representation that can be likened to a delimited concatenation of all the values for the field. The fields have a configurable limit to the maximum field size, and this limit may be encountered in the various entries such as the permission fields or object relation fields where a potentially large number of subjects or identifiers may be stored. Further research has indicated this problem is unlikely since simply increasing the SOLR maxFieldLength configuration value will enable larger field content, with examples in the wild of several thousand entries equivalent to DataONE identifiers in a single field.

Values Extracted from Science Metadata

Indexing the system metadata entries provides a mechanism for selection of content using low level attributes of the objects such as the type, size and relationships, however is not useful for locating content relevant to a particular topic or purpose. The science metadata contains such information and the index records associated with these documents contain additional information described below.

The initial web user interface to DataONE utilizes the Mercury system. This in turn relies upon a SOLR index to support the search, faceting, and sub-setting operations that are exposed through Mercury. The underlying SOLR index is also exposed through as an API, so other clients (such as the file system driver) can leverage the search capabilities thus provided.

The remainder of this document describes the search properties that are extracted from the various formats and syntaxes of of science metadata that are supported by the DataONE indexing system.

Table 3. Index fields populated from science metadata documents. “Multi” fields may be populated with more than a single value per document (PID) indexed.

Date Representations in Science Metadata Documents

The following are examples of date values extracted from FGDC, ISO-19115, and EML science metadata documents currently in use. The literal value appearing in the document and the interpreted date value are shown.

Examples of date values appearing in FGDC <pubdate>:

Value

Interpreted Value

Unknown

Null

unknown

Null

Unpublished material

Null

unpublished material

Null

1993

1993-01-01 00:00:00Z

199607

1996-07-01 00:00:00Z

20000101

2000-01-01 00:00:00Z

19981231

1998-12-31 00:00:00Z

196820405

1968-01-01 00:00:00Z

1992 onwards

1992-01-01 00:00:00Z

1989 and 1990

1989-01-01 00:00:00Z

varies

Null

Present

Null

1995/1996

1995-01-01 00:00:00Z

1991-1992

1991-01-01 00:00:00Z

variouis

Null

April 1999

1999-04-01 00:00:00Z

1980 on

1980-01-01 00:00:00Z

2005-06-24

2005-06-24 00:00:00Z

NA

Null

1990- [unpublished annual reports]

1990-01-01 00:00:00Z

November, 1994

1994-11-01 00:00:00Z

Examples of date values appearing in ISO-19115 <gco:Date>:

Value

Interpreted Value

1999

1999-01-01 00:00:00Z

2010-03-03

2010-03-03 00:00:00Z

Examples of date values appearing in EML <calendarDate>:

Value

Interpreted Value

2002-06-20

2002-06-20 00:00:00Z

1998

1998-01-01 00:00:00Z

2004-02-13

2004-02-13 00:00:00Z

Attribute Descriptions and Notes

A list of all fields in search index, their origin and brief description. Source column indicates which DataONE content the values are retrieved from: Y = System Metadata, S = Science Metadata, R = Resource Map, and C = copied internally by the search index.


Creating Citations from Index Fields

Table 3. Cross reference of index fields and elements useful in citation best practices as exemplified in the interagency data stewardship wiki page of ESIP. * = Mandatory (if applicable). COinS tags provide a mechanism of embedding Z39.88-2004 elements in HTML for consumption by citation manager tools.

Citation Element

FGDC CSDGM field

DataCite Metadata Scheme ID and Property

DataONE Index Field

COinS Tag

Author or Creator*

idinfo > citation > citeinfo > “origin”

2 Creator*

author

rft.creator

Release Date*

idinfo > citation > citeinfo > “pubdate” and sometimes “othercit”

5 PublicationYear*

datePublished

rft.date

Title*

idinfo > citation > citeinfo > “title” and possibly “edition”

3 Title*

title

rft.title

Version*

15 Version

See notes below.

Archive and/or Distributor*

idinfo > citation > citeinfo > “publish”

4 Publisher*

See notes below.

rft.publisher

Locator, Identifier, or Distribution Medium*

idinfo > citation > citeinfo > “othercit” or “onlink”

1 Identifier*

pid

rft.identifier

Access Date and Time*

not applicable

8 Date

perhaps dateUploaded

Subset Used

not applicable

12 RelatedIdentifier DataCite recommends obtaining an identifier for any subset that needs to be cited as well as an identifier for the larger whole.

perhaps a reference to the resource map?

Editor or Other Important Role

idinfo > citation > citeinfo > “origin”

7 Contributor

See notes below.

rft.source

Publication Place

idinfo > citation > citeinfo > “pubplace”

17 Description

See notes below.

Distributor, Associate Archive, or other Institutional Role

idinfo > citation > citeinfo > “othercit”

7 Contributor or possibly 4 Publisher

http://dataone.org

Data Within a Larger Work

idinfo > citation > citeinfo > “othercit” or “lworkcit”

12 RelatedIdentifier

perhaps a reference to the resource map?

keyWord

rft.subject

abstract

rft.description

object format name, or “data” | “metadata” | “resource map”

rft.type

object format type

rft.format

constructed from bounding box

rft.coverage

Notes

Version

There is no clear recommendation on how to express version. For an object with a linear revision history (A is obsoleted by B is obsoleted by C), object C might be considered to have a version of “2”. However, it is possible for an object to have a tree lineage (A1 and A2 are obsoleted by B1, and B1, A3 and Z are obsoleted by C). A suggestion for a version indicator in this case may be the total number of parent objects, 5 in this case. Another option may be to indicate the number of contributing objects at each revision, 1.1 in the first case, and 2.3 in the second.

Publisher

The publisher in DataONE may in some cases be the origin member node, however this will not always be the case. For example, many data sets in the KNB are published by LTER, though KNB would be the member node. Where possible the publisher information should be pulled from the science metadata, though this may be complicated where multiple data and metadata entries occur within a data package and there are several publishers listed. In this case, a simple concatenation of the publishers (semi-colon + space delimited) may be sufficient.

Date

In the DataCite 2.2 document, a mapping between Date and the dcterms:date is suggested. The description of dcterms:date is “A point or period of time associated with an event in the lifecycle of the resource.” In DataONE there are two main dates related to an object, when the content is added to the system, and when the properties of the object (access control, and so forth) are modified. Since the content of the object does not change once submitted to DataONE, the SystemMetadata.dateUploaded value seems appropriate here, expressed as dateUploaded in the search index.

Subset Used

DataONE does not provide a mechanism for identifying sets of information other than through resource maps which describe data packages. It is possible that a resource map may contain references to additional resource maps, conceptually equivalent to subsets of content. The simplest option here would be to use the identifier of the resource map object describing the package in question.

Contributor

Described in DataCite 2.2 as “The institution or person responsible for collecting, creating, or otherwise contributing to the development of the dataset”, this optional element can be populated by the citeinfo/origin element from FGDC.

Publication Place

In CSDGM, “the name of the city (and state or province, and country, if needed to identify the city) where the data set was published or released”.

Data Within a Larger Work

This is similar to the “Subset Used” entry except in the other direction of containment. The identifier of the containing resource map relevant to the current package should be used.