Month: July 2012

  • A Manifesto for Managing Metadata in an Open World…

    Metadata is produced and stored locally, published globally, consumed and aggregated locally, and finally integrated and stored locally. This is the create/publish/consume/integrate cycle.

    Providing a framework for managing metadata throughout this cycle is the goal of the Dublin Core Abstract Model and the Dublin Core Application Profile (DCAM/DCAP).

    The basic guidelines and requirements for this cycle are:

    • Metadata MUST be syntactically VALID and semantically COHERENT when it’s CREATED and PUBLISHED.
    • Globally PUBLISHED Metadata SHOULD be TRUE, based on the domain knowledge of the publisher.
    • PUBLISHERS of metadata MUST publish the semantics of the metadata, or reference publicly available semantics.
    • CONSUMERS of published metadata SHOULD assume that the global metadata is locally INVALID, INCOHERENT, and UNTRUE.
    • CONSUMED metadata MUST be checked for syntactic validity and semantic coherence before integration.
    • AGGREGATED metadata SHOULD indicate its PROVENANCE.
    • CONSUMED metadata MAY be considered TRUE based on its PROVENANCE.
    • Locally INTEGRATED global metadata MUST be syntactically VALID, semantically COHERENT, and SHOULD be TRUE based on local standards.

    The DCAM takes as its base the rdf data model because of its simplicity and flexibility of that model. The DCAM refines the rdf data model in order to support syntactic interoperability between the rdf data model and non-rdf data models.

    A DCAP defines a distinct subset of the set of all things and defines the domain-specific knowledge of the properties of those things, and the relationships those things have to other things. It expresses that knowledge through the DCAM and a set of related documentation (the Singapore Framework). A complete DCAP should provide the necessary domain-specific infrastructure to fully support the create/publish/consume/integrate cycle for any system, any domain, and any data model. A DCAP based on the DCAM should be able to be used by a machine to generate a valid data model in any modeling syntax, and any modeling syntax should be able to be expressed as a DCAP.

    We strongly recommend that metadata be published and consumed using the RDF data model. The strength of the rdf data model for publishing and aggregating metadata lies in its extreme flexibility of expression, it’s self-describing semantics, and its assumption of validity. These strengths become weaknesses when creating and validating local metadata that must conform to a set of local restrictions. An effective DCAP will provide metadata creators with the ability to create locally valid metadata, publish it as rdf, validate consumed rdf against local standards, and ultimately integrate global metadata into the local knowledge base that contains local descriptions of the things being described.

    There are many systems, many data models, many publish and subscribe models, many storage and validation models. There are many paths to integration. There are very few that provide a generic and neutral method for modeling and documenting inter-system metadata integration. The DCAM/DCAP specification has the potential to be one of those few.

  • Mapping without taggregations

    By-passing taggregations identifies those MARC21 variable data field tags whose level 0 RDF properties, representing individual subfields, can be used generally in data triples and mapping triples without losing information and the semantic coherency of the record. These tags have subfields which are independent of one another, with no need to keep them together if the tag is repeated in a record.

    RDF graph of Music format ontology
    RDF graph of Music format ontology

    This graph is another example of adding MARC fruit to the cornucopia by using the sub-property ladder. The MARC21 property used in the graph is a level 0 element. The MARC21 property’s description is equivalent to the ISBD property’s description, as hinted at in the ISBD scope note, but an unconstrained version is used to avoid the ISBD property’s domain. We could use the Web Ontology Language (OWL) ontological property owl:equivalentProperty to represent the relationship between the properties, but we can also use the rdfs:subPropertyOf property by applying it in both directions. That is, if two properties P1 and P2 are related as P1 rdfs:subPropertyOf P2 AND P2 rdfs:subProperty P1, then P1 owl:equivalentProperty P2 and P2 owl:equivalentProperty P1.

    Unfortunately, it is unsafe to use level 0 properties for dependent subfields in repeatable tags in this way, even if a specific record contains only one instance of such a tag. Triples from that record will cluster with triples from another record about the same bibliographical resource, either by sharing the same resource URI or an equivalence link. Taggregations are required to avoid semantic confusion, otherwise we wouldn’t know which “Materials specified” goes with which “Place of publication …” or “Name of publisher …” in the publication statement example given in Taggregations.

     

  • If We Were Asked

    If we were asked (as we sometimes are) what we’d like to see develop as a result of the BibFrame effort, the emphasis in our answer would have both technical and social aspects.

    First, given the technologies developing in several different places and considering what we can do now to bring Linked Open Data into our somewhat closed world, some concrete suggestions: We have the ability to share metadata expressed not just in a single common ‘vocabulary’, but to share it using many different vocabularies, expressed and distributed using RDF, OWL/RDFS, RDFa, Microdata, and other tools; we have methods of specifying the use of these ‘semantic’ building blocks (DC Application Profiles and emerging provenance specifications from W3C and DCMI) that allow machines to use, process and distribute data in ways that do not require a central enabling node; and finally, we have technologies and strategies in place to map between existing and prospective metadata schemas that flatten the fences between communities quite thoroughly.

    This is the post-MARC world of library-related metadata, where the format native to our metadata is no longer its most important characteristic; there’s no lossy crosswalking requirement to transform data to serve different needs; and there are fewer (eventually no) barriers to sharing. The tremendous value that MARC represents–the semantics built over many decades in response to an enormous corpus of use cases (all of which are fully documented on the MARBI site)–continues to be vitally important as we move into a different arena. But the mid-20th century requirements that dictated the MARC syntax, and the constricting consensus model that has been required to maintain it, no longer apply to the current and future requirements of the global library community.

    The usual caveats apply here–some of these technologies aren’t entirely ready for prime time, but in the world we live in, ‘finished’ is more likely to be used for something defunct rather than a goal for standards and tool development. The areas we’re personally most familiar with (no surprises here) are the complementary domains of vocabulary management and mapping. The first of those is up and running (although the ‘continuous improvement’ engine is going full bore) as the Open Metadata Registry; the second, mapping, is an important interest of ours and has generated papers, articles, and presentations (see below for a selection), not to mention the usual plethora of posts to blogs (including this one) and discussion lists. We believe that the OMR and mapping capabilities under development work together to enable legacy data to shift into the open linked data world efficiently, effectively, and with a minimum of loss, in the process enriching the useful data options available for everyone.*

    Often overlooked in the glitter of technology is the possibility of pulling together the communities that were torn asunder as a result of past technical limitations (and other reasons). In our common past, the library community built their data sharing conventions on a platform based on the consensus use of AACR2 and MARC. Some library communities–law and music the most prominent–were willing to compromise in order to work within the larger library community rather than strike out on their own. Others, like the art library and museum community and the archivists, did make the break, and have developed data standards that meet their needs better than the library consensus was able to do. But those specialized standards have resulted in yet more silos, in addition to the huge MARC one.

    If the intent is to ‘replace MARC’ (as it is at some level), it’s about re-placing MARC and its built-in limitations from the center of our world back into the crowd, where it becomes one-among-many. Also a value in that shift is the ability to expand the data sharing environment MARC enabled a half century ago to a broader community of interest that includes museums, publishers, and archives. Meeting the goal of dissolving those silos and making our data easily understandable and reusable in a host of ways will help initiate that ‘Web of Data’ we’ve been anticipating for many years. As Eric Miller explained so well in his Anaheim BibFrame update: by moving towards a linked data world we actually look well beyond the Library/Archives/Museum worlds by definition–it’s a very big world out there. But by collaborating with the LAM community as a whole to get there, we reap a great many benefits, not the least of which are perspectives that are much broader in some significant ways than ours. Limiting our view merely to a ‘MARC Linked Data Model’ might be an important beginning step, but it falls short of where our vision needs to extend.

    And the fact is that MARC will not be going away for a long time, if ever. There will be a lot of variation in how the transition is done by libraries, depending on institutional support, short term and long term needs, and existing partnerships. The process of moving MARC into the linked data world has already started. RDA and its RDF vocabularies was a start, as is the development of a complete RDF version of MARC, located at http://marc21rdf.info. Several years worth of pre-conferences, presentations and discussions, at ALA and beyond, have prepared the soil for these changes. But we need a plan, and some concrete steps to take–steps that include the groups who have been working in the trenches without a great deal of support, but making progress regardless. The BibFrame effort needs to be more than a playground for the technologists, because in most instances, the technology is not what’s holding us back–it’s the institutional inertia and the difficulties of finding ways forward that don’t pit us against one another. The plan we need balances the technical and social, the quick-win with the long-term momentum, and the need for speed with the public discussion that takes time and builds buy-in.

    What we have in our sights is an opportunity to reverse the long term trend towards balkanized metadata communities and to make the future one for which there are fewer fences between, and more data exchanged among these three communities with obviously similar challenges and interests. We think the time has come to use the vastly changed technology environment to do that.

    *It would be too easy for a response to this post to be in the form of “Oh, you’re just tooting your own horn here,” and indeed we are in some measure doing that. But we do this work because we believe it’s important. We don’t believe it’s important merely because it’s what we do. We believe in the value of the work we’ve done and will do, and we see a great deal of relevance for it as part of the BibFrame discussions.

    Selection of papers, and presentations:

    Jon Phipps’ presentation on mapping at ALA Anaheim
    A Reconsideration of Mapping in a Semantic World / Gordon Dunsire, Diane Ileana Hillmann, Jon Phipps, Karen Coyle
    Gordon Dunsire’s presentation at the recent London meeting “RDA, 5 years on”

    Post written by Diane Hillmann and Jon Phipps.