Automated Metadata Extraction, MARC 21 / Dublin Core Ontologies & LLM Vector Cataloging

1. From MARC 21 & Dublin Core to Dense Vector Embeddings

In November 2006, when the KickButtCatalogers working group convened on InfoIsland, the library science community was grappling with the explosion of digital web resources. Traditional bibliographic standards—primarily MARC 21 (Machine-Readable Cataloging) and Dublin Core—relied on manual data entry by trained cataloging professionals. Each title required careful human assignment of Dewey Decimal numbers, Library of Congress Control Numbers (LCCN), and Medical Subject Headings (MeSH).

Two decades later, institutional academic repositories ingest tens of thousands of preprints, datasets, and open-access monographs every week. Human-only cataloging has encountered an insurmountable scaling bottleneck.

Modern information systems solve this through hybrid cataloging fabrics: foundation models automate structured schema extraction, while dense vector databases (Qdrant, Milvus) provide multi-modal semantic discovery across full-text corpuses.

2. Empirical Benchmark: LLM Structured Metadata Extraction

To evaluate how accurately generative models extract structured bibliographic metadata compared to professional human catalogers, we tested four open and proprietary models across 2,500 academic papers:

3. Production Python Pipeline: Automated Dublin Core Extraction

The code snippet below illustrates how academic institutions deploy structured decoding constraints using Pydantic and open-weights models:

# Automated Dublin Core Metadata Extraction with Pydantic & Instructor
from pydantic import BaseModel, Field
from typing import List, Optional
import instructor
from openai import OpenAI

class DublinCoreRecord(BaseModel):
    title: str = Field(description="Canonical title of the work")
    creator: List[str] = Field(description="Authors or institutional contributors")
    subject_headings: List[str] = Field(description="Controlled LCSH or MeSH vocabulary terms")
    description_abstract: str = Field(description="Concise 150-word synthesis of core methodology")
    publisher: str = Field(description="Issuing academic body or university press")
    date_iso: str = Field(description="ISO-8601 publication date")
    rights_license: str = Field(description="Creative Commons or open-access copyright terms")

client = instructor.from_openai(OpenAI())

def catalog_paper(full_text_chunk: str) -> DublinCoreRecord:
    return client.chat.completions.create(
        model="gpt-4o-mini",
        response_model=DublinCoreRecord,
        messages=[
            {"role": "system", "content": "You are an expert academic cataloger. Extract standard Dublin Core metadata."},
            {"role": "user", "content": full_text_chunk[:4000]}
        ]
    )

4. Hybrid Semantic OPAC Architecture

Integrating structured relational metadata with vector embeddings delivers the gold standard for library discovery:

• Deterministic Filtering: Users filter by exact publication year, licensed access, and author authority IDs using relational SQL queries.
• Semantic Exploratory Search: Dense vector embeddings project natural language queries onto the document subspace, matching conceptual synonyms even when exact terms differ.
• Knowledge Graph Linking: Connecting cataloged works via shared entity identifiers to build corpus-wide citation graphs.

For a complete technical analysis of relational entity graphs vs vector embeddings, read our study on Vector DBs vs GraphRAG & Entity Ontologies.