Natural History of System Metadata

Revisions

Date

Comment

20100303

(D.V.) Initial draft

20100304

(D.V.) Revised draft and added scenario where system metadata is stored only on the Coordinating Nodes

Corrected DataSysMetadataModified changes in scenario B.

This document describes the generation of system metadata associated with a data object and its associated science metadata document as they are added to a DataONE Member Node and subsequently propagated through the DataONE infrastructure.

Process Overview

There are two scenarios examined here: A) the system metadata is stored on all nodes (MN and CN) and B) system metadata is stored only on the CNs. The general sequence of operations is the same for both scenarios with the exclusion of a couple of steps from scenario B.

  1. Scientist using an ITK tool generates some science data and some science metadata describing the science data.

  2. The ITK populates the required fields of the system metadata documents.

  3. Not necessarily in this order:

    1. The ITK uploads the science data

    2. The ITK uploads the science metadata

    On receipt, the MN updates the system metadata fields for which it is responsible.

  4. The CN detects new content is available on the MN using the listObjects() method.

  5. Copies of the science metadata and the two (incomplete) system metadata documents are retrieved by the CN.

    A

    The CN updates the system metadata replication information, and indicates to the MN that it should update its copy of the system metadata objects.

    B

    The CN updates the system metadata replication information.

  6. The CN issues a replication order to an MN.

    A

    The recipient MN retrieves copies of the system metadata and associated objects (science data and metadata)

    B

    The recipient MN retrieves copies of the various objects indicated by the CN

  7. The recipient MN requests content from the origin MN, which in turn verifies the request with the CN.

    A

    The origin MN and the CN update copies of system metadata

    B

    The CN updates system metadata

  8. After verifying the successful transfer of content, the recipient MN indicates completion to the CN.

    A

    The CN updates the system metadata for the objects and ensures that the MN copies of the system metadata are accurate.

    B

    The CN updates the system metadata for the objects.

The change in system metadata state is shown in the following process detail sections. In each case, the identifiers used for the various elements are indicated in table 1.

Table 1. Identifiers used in the example sequence.

Identifier

Description

sciD.1

Science Data

sciM.1

Science metadata describing sciD.1

sysD.1

System metadata for object sciD.1

sysM.1

System metadata for object sciM.1

cn1.dataone.org

A Coordinating Node

mn1.dataone.org

Member Node that is initial recipient of data

mn2.dataone.org

Member Node that receives a copy of the objects

The following fictitious script provides an example of how uploading content to a Member Node using a DataONE client library might occur.

import sys
import logging
from pyd1.client import *
from pyd1.exceptions import *

d1client = pyd1.client(identity="/home/vieglais/.d1/credentials",
                       defaults="/home/vieglais/.d1/defaults.ini")
d1client.login()

# The science data and metadata as files on disk
data_file = "/home/vieglais/data/mydata.csv"
meta_file = "/home/vieglais/data/mymeta.xml"

# Reserve a couple of identifiers. Note that this reservation will
# be system-wide.
try:
  data_id = d1client.reserve(u"sysD.1")
  meta_id = d1client.reserve(u"sysM.1")
except IdentifierInUseException, e:
  logging.fatal(repr(e))
  sys.exit()

# Generate stub system metadata documents. Default values
# for access rules etc are loaded from defaults.ini
sysm_data = d1client.generateSysMeta(id=data_id,
                                     format="text/csv",
                                     describedBy=meta_id)
sysm_meta = d1client.generateSysMeta(id=meta_id,
                                     format="eml://ecoinformatics.org/eml-2.1.0",
                                     describedBy=data_id)
# Open file handles for data and metadata
fdata = file(data_file, "r")
fmetadata = file(meta_file, "r")

# Send the data to the Member Node
target_node = 'mn1.dataone.org'
data_sysid = d1client.create(data_id, fdata, sys_data,
                             target=target_node)
meta_sysid = d1client.create(meta_id, fmeta, sys_meta,
                             target=target_node)
fdata.close()
fmeta.close()

Process Detail - Scenario A

Step A1.

The scientist generates a data set and associated science metadata.

Step A2.

Using tools in the ITK, the scientist generates two initial system metadata documents, one describing the science metadata (sysm_M.1), one describing the science data (sysm_D.1).

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

objectFormat

text/csv

eml://ecoinformatics.org/eml-2.1.0

size

1450238

8132

submitter

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

rightsHolder

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

obsoletes

obsoletedBy

derivedFrom

describes

sciD.1

describedBy

sciM.1

checksum

2e01e17467891f7c933dbaa00e1459d23db3fe4f

9967467891f7c933dbaa00e1459d23db3f342

checksumAlgorithm

SHA-1

SHA-1

accessRule[1]

.rule

Allow

Allow

.service

Read

Read

.principal

*

*

accessRule[2]

.rule

Allow

Allow

.service

Write

Write

.principal

o=NCEAS,dc=ecoinformatics,dc=org

o=NCEAS,dc=ecoinformatics,dc=org

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

dateSysMetadataModified

originMemberNode

authoritativeMemberNode

replica[1]

.replicaMemberNode

.replicationStatus

.replicaVerified

Step A3.

The client tool transmits the data, science metadata and two stub system metadata documents to a Member Node. The Member Node updates values of the system metadata for which it is responsible.

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

objectFormat

text/csv

eml://ecoinformatics.org/eml-2.1.0

size

1450238

8132

submitter

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

rightsHolder

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

obsoletes

obsoletedBy

derivedFrom

describes

sciD.1

describedBy

sciM.1

checksum

2e01e17467891f7c933dbaa00e1459d23db3fe4f

9967467891f7c933dbaa00e1459d23db3f342

checksumAlgorithm

SHA-1

SHA-1

accessRule[1]

.rule

Allow

Allow

.service

Read

Read

.principal

*

*

accessRule[2]

.rule

Allow

Allow

.service

Write

Write

.principal

o=NCEAS,dc=ecoinformatics,dc=org

o=NCEAS,dc=ecoinformatics,dc=org

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

dateSysMetadataModified

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

originMemberNode

mn1.dataone.org

mn1.dataone.org

authoritativeMemberNode

mn1.dataone.org

mn1.dataone.org

replica[1]

.replicaMemberNode

mn1.dataone.org

mn1.dataone.org

.replicationStatus

Queued

Queued

.replicaVerified

Since there are many copies of system metadata from this point onwards, only the replication properties of the sysD.1 document are considered, summarized in a table as follows. MN1, CN, MN2 = copy of system metadata on mn1.dataone.org, cn1.dataone.org, mn2.dataone.org respectively.

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

R[1].replicationStatus

Queued

R[1].replicaVerified

Step A4.

A CN detects new content available on mn1.dataone.org through the listObjects() method.

There is no change of state in system metadata resulting from this operation.

Step A5.

Copies of sciM.1, sysM.1, and sysD.1 are retrieved to the CN, and the system metadata objects are updated. There are three steps to this process:

  1. The CN requests a copy of the objects.

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

R[1].replicationStatus

Requested

R[1].replicaVerified

  1. The CN receives a copy of the metadata.

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

mn1.dataone.org

R[1].replicationStatus

Requested

Requested

R[1].replicaVerified

  1. The CN verifies the checksums and indicates to the MN that the copy is complete, providing the time stamp that it used for the verification (ensuring identical copies of the system metadata).

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

mn1.dataone.org

R[1].replicationStatus

Completed

Completed

R[1].replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

Step A6.

The CN initiates replication of content from mn1.dataone.org to mn2.dataone.org.

  1. The CN sends a replication request to MN2, indicating that it should retrieve the object from MN1.

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

mn1.dataone.org

R[1].replicationStatus

Completed

Completed

R[1].replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

R[2].replicaMemberNode

mn2.dataone.org

R[2].replicationStatus

Queued

R[2].replicaVerified

  1. MN2 requests the content from MN1.

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

mn1.dataone.org

mn1.dataone.org

R[1].replicationStatus

Completed

Completed

Completed

R[1].replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

R[2].replicaMemberNode

mn2.dataone.org

mn2.dataone.org

mn2.dataone.org

R[2].replicationStatus

Requested

Queued

Requested

R[2].replicaVerified

  1. MN1 checks with CN that the object was requested

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

mn1.dataone.org

mn1.dataone.org

R[1].replicationStatus

Completed

Completed

Completed

R[1].replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

R[2].replicaMemberNode

mn2.dataone.org

mn2.dataone.org

mn2.dataone.org

R[2].replicationStatus

Requested

Requested

Requested

R[2].replicaVerified

  1. MN2 verifies that the checksum is correct, then indicates to MN1 that the copy was correctly transferred, forwarding the timestamp for when the object was verified to MN1.

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

mn1.dataone.org

mn1.dataone.org

R[1].replicationStatus

Completed

Completed

Completed

R[1].replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

R[2].replicaMemberNode

mn2.dataone.org

mn2.dataone.org

mn2.dataone.org

R[2].replicationStatus

Completed

Requested

Completed

R[2].replicaVerified

2010-03-04T20:00:00.0Z

2010-03-04T20:00:00.0Z

Step A8.

MN2 informs the Coordinating Node that it has retrieved a valid copy of the object from MN1, forwarding the time stamp for when the verification was made.

Property

MN1

CN

MN2

R[1].replicaMemberNode

mn1.dataone.org

mn1.dataone.org

mn1.dataone.org

R[1].replicationStatus

Completed

Completed

Completed

R[1].replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

2010-03-04T19:13:51.0Z

R[2].replicaMemberNode

mn2.dataone.org

mn2.dataone.org

mn2.dataone.org

R[2].replicationStatus

Completed

Completed

Completed

R[2].replicaVerified

2010-03-04T20:00:00.0Z

2010-03-04T20:00:00.0Z

2010-03-04T20:00:00.0Z

Process Detail - Scenario B.

This scenario uses the notion that system metadata is stored on the CNs only.

Step B1.

The scientist generates a data set and associated science metadata.

Step B2.

Using tools in the ITK, the scientist generates two initial system metadata documents, one describing the science metadata (sysm_M.1), one describing the science data (sysm_D.1).

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

objectFormat

text/csv

eml://ecoinformatics.org/eml-2.1.0

size

1450238

8132

submitter

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

rightsHolder

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

obsoletes

obsoletedBy

derivedFrom

describes

sciD.1

describedBy

sciM.1

checksum

2e01e17467891f7c933dbaa00e1459d23db3fe4f

9967467891f7c933dbaa00e1459d23db3f342

checksumAlgorithm

SHA-1

SHA-1

accessRule[1]

.rule

Allow

Allow

.service

Read

Read

.principal

*

*

accessRule[2]

.rule

Allow

Allow

.service

Write

Write

.principal

o=NCEAS,dc=ecoinformatics,dc=org

o=NCEAS,dc=ecoinformatics,dc=org

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

dateSysMetadataModified

originMemberNode

authoritativeMemberNode

replica[1]

.replicaMemberNode

.replicationStatus

.replicaVerified

Step B3.

The client tool transmits the data, science metadata and two stub system metadata documents to a Member Node (mn1.dataone.org). On receipt, the Member Node sets the values of several fields as indicted in the table below.

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

objectFormat

text/csv

eml://ecoinformatics.org/eml-2.1.0

size

1450238

8132

submitter

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

rightsHolder

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

obsoletes

obsoletedBy

derivedFrom

describes

sciD.1

describedBy

sciM.1

checksum

2e01e17467891f7c933dbaa00e1459d23db3fe4f

9967467891f7c933dbaa00e1459d23db3f342

checksumAlgorithm

SHA-1

SHA-1

accessRule[1]

.ruleType

Allow

Allow

.service

Read

Read

.principal

*

*

accessRule[2]

.ruleType

Allow

Allow

.service

Write

Write

.principal

o=NCEAS,dc=ecoinformatics,dc=org

o=NCEAS,dc=ecoinformatics,dc=org

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

dateSysMetadataModified

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

originMemberNode

mn1.dataone.org

mn1.dataone.org

authoritativeMemberNode

mn1.dataone.org

mn1.dataone.org

replica[1]

.replicaMemberNode

mn1.dataone.org

mn1.dataone.org

.replicationStatus

Queued

Queued

.replicaVerified

Step B4.

A CN detects new content available on mn1.dataone.org through MN_replication.listObjects().

There is no change of state in system metadata resulting from this operation.

Step B5.

Copies of sciM.1, sysM.1, and sysD.1 are retrieved to the Coordinating Node, and the system metadata objects are updated on the Coordinating Node. For sake of brevity, only the elements from ReplicationPolicy onwards are shown in the remaining state tables since the remainder of the content does not change.

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

dateSysMetadataModified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:52.0Z

originMemberNode

mn1.dataone.org

mn1.dataone.org

authoritativeMemberNode

mn1.dataone.org

mn1.dataone.org

replica[1]

.replicaMemberNode

mn1.dataone.org

mn1.dataone.org

.replicationStatus

Completed

Completed

.replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:52.0Z

Step B6.

The Coordinating Node initiates replication of the objects sciM.1 and sciD.1 from mn1.dataone.org to mn2.dataone.org. It does this by calling the “replicate()” method on mn2.dataone.org.

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

dateSysMetadataModified

2010-03-04T20:58:00.0Z

2010-03-04T20:58:00.0Z

originMemberNode

mn1.dataone.org

mn1.dataone.org

authoritativeMemberNode

mn1.dataone.org

mn1.dataone.org

replica[1]

.replicaMemberNode

mn1.dataone.org

mn1.dataone.org

.replicationStatus

Completed

Completed

.replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:52.0Z

replica[2]

.replicaMemberNode

mn2.dataone.org

mn2.dataone.org

.replicationStatus

Queued

Queued

.replicaVerified

Step B7.

The node mn2.dataone.org requests the objects sciM.1 and sciD.1 from mn1.dataone.org. After verifying the request with a Coordinating Node (which has a side effect of setting the system metadata ReplicationStatus to Requested), mn1.dataone.org returns the objects. (Note that in actuality, each object is requested individually.)

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

dateSysMetadataModified

2010-03-04T20:59:00.0Z

2010-03-04T20:59:00.0Z

originMemberNode

mn1.dataone.org

mn1.dataone.org

authoritativeMemberNode

mn1.dataone.org

mn1.dataone.org

replica[1]

.replicaMemberNode

mn1.dataone.org

mn1.dataone.org

.replicationStatus

Completed

Completed

.replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:52.0Z

replica[2]

.replicaMemberNode

mn2.dataone.org

mn2.dataone.org

.replicationStatus

Requested

Requested

.replicaVerified

Step B8.

The node mn2.dataone.org verifies the retrieved objects against their checksums and indicates to the Coordinating Node that transfer was successful.

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

dateSysMetadataModified

2010-03-04T21:00:00.0Z

2010-03-04T21:00:00.0Z

originMemberNode

mn1.dataone.org

mn1.dataone.org

authoritativeMemberNode

mn1.dataone.org

mn1.dataone.org

replica[1]

.replicaMemberNode

mn1.dataone.org

mn1.dataone.org

.replicationStatus

Completed

Completed

.replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:52.0Z

replica[2]

.replicaMemberNode

mn2.dataone.org

mn2.dataone.org

.replicationStatus

Completed

Completed

.replicaVerified

2010-03-04T21:00:00.0Z

2010-03-04T21:00:00.0Z

Final State of System Metadata

Assuming the replication policy was for one replica to be created for sciM.1 and sciD.1, then the eventual end state of the associated system metadata documents (sysM.1 and sysD.1 respectively) after synchronization of the Coordinating Nodes, would be something like the following.

Note that synchronization between Coordinating Nodes does not trigger an update to the DataSysMetadataModified field. This is necessary to avoid a situation where the state of the metadata is never consistent between Coordinating Nodes.

Note also that even though the replication policy of sciD.1 indicates that NumberReplicas is one, there are actually three additional replicas stored on the Coordinating Nodes.

Field

sysD.1

sysM.1

identifier

sciD.1

sciM.1

objectFormat

text/csv

eml://ecoinformatics.org/eml-2.1.0

size

1450238

8132

submitter

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

rightsHolder

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

uid=jones,o=NCEAS,dc=ecoinformatics,dc=org

obsoletes

obsoletedBy

derivedFrom

describes

sciD.1

describedBy

sciM.1

checksum

2e01e17467891f7c933dbaa00e1459d23db3fe4f

9967467891f7c933dbaa00e1459d23db3f342

checksumAlgorithm

SHA-1

SHA-1

accessRule[1]

.ruleType

Allow

Allow

.service

Read

Read

.principal

*

*

accessRule[2]

.ruleType

Allow

Allow

.service

Write

Write

.principal

o=NCEAS,dc=ecoinformatics,dc=org

o=NCEAS,dc=ecoinformatics,dc=org

replicationPolicy

.preferredMemberNode

mn2.dataone.org

mn2.dataone.org

.blockedMemberNode

.replicationAllowed

true

true

.numberReplicas

1

1

dateUploaded

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

dateSysMetadataModified

2010-03-04T18:13:51.0Z

2010-03-04T18:14:45.0Z

originMemberNode

mn1.dataone.org

mn1.dataone.org

authoritativeMemberNode

mn1.dataone.org

mn1.dataone.org

replica[1]

.replicaMemberNode

mn1.dataone.org

mn1.dataone.org

.replicationStatus

Completed

Completed

.replicaVerified

2010-03-04T19:13:51.0Z

2010-03-04T19:13:52.0Z

replica[2]

.replicaMemberNode

mn2.dataone.org

mn2.dataone.org

.replicationStatus

Completed

Completed

.replicaVerified

2010-03-04T21:00:00.0Z

2010-03-04T21:00:00.0Z

Conclusions

Maintaining complete copies of system metadata on all nodes is cumbersome and seems to be overly complicated. However, it is still necessary for Member Nodes to provide accurate information for some elements of the system metadata (to assist with transfer verification, to validate a data object is intact).

As such, it seems that scenario B above is the most tractable and provides the desired functionality for tracking content in the DataONE infrastructure. The role of system metadata on Member Nodes and Coordinating Nodes is further clarified below.

The recommendations are:

  • Complete, authoritative copies of system metadata are maintained on the Coordinating Nodes.

  • A system metadata record becomes authoritative once the replicationStatus is set to complete after initial transfer of content to the Coordinating Node (step B5 above)

  • Member Nodes still provide access to system metadata through the MN_crud.getSystemMetadata() calls, though the copy of system metadata returned is authoritative for the properties controlled by the Member Nodes only.

  • Member Nodes should not return any of the SystemMetdata.replica attributes.