Open infrastructure for
A harmonized, quality-scored foundation of DFT-computed structural and thermodynamic properties across 267,230 materials from Materials Project, OQMD, and JARVIS-DFT. Designed for AI-driven materials discovery.
WHY THIS DATASET
The biggest bottleneck in AI-driven materials discovery is no longer architecture — it's data. Most materials datasets are source-specific, lack quality scoring, and don't track provenance. Scandium Dataset solves this by harmonizing three major DFT databases into a single, quality-scored, provenance-tracked resource.
Materials from MP, OQMD, and JARVIS are standardized to shared schema, units, and property definitions. No more manual mapping between database APIs.
Every entry is scored across 5 sub-scores. Calibrated against cross-source agreement. Score 80+ correlates with research-grade label quality.
Every entry tracks its source, source_id, checksum, and repair history. Know exactly where each value came from and how it was processed.
DATASET STATISTICS
A comprehensive view of the dataset composition, quality distribution, and coverage across sources.
TIER DISTRIBUTION
PROPERTY COVERAGE
| Property | Coverage | Notes |
|---|---|---|
| Formation energy (eV/atom) | 267,230 (100%) | From MP, OQMD, JARVIS-DFT |
| Energy above hull (eV/atom) | 257,799 (96.5%) | 14,669 JARVIS entries computed via internal convex hull |
| Band gap (eV) | 267,079 (99.9%) | 151 OQMD entries with non-converged band gap |
| Space group | 267,214 (100%) | spglib symmetry analysis |
| Volume (ų) | 267,230 (100%) | From structure |
| Density (g/cm³) | 267,230 (100%) | From elements + volume |
| Structure JSON | 267,230 (100%) | pymatgen Structure serialization |
| Quality score | 267,230 (100%) | Composite score (0–88) |
| Provenance | 267,230 (100%) | Source, source_id, checksum |
| SSE family | 267,230 (100%) | Composition-based heuristic |
DATA PIPELINE
Every entry in the Scandium Dataset passes through a rigorous pipeline: ingestion, repair, deduplication, quality scoring, and tier classification.
Data is pulled from Materials Project, OQMD, and JARVIS-DFT APIs with checksum verification. Raw archives are frozen and never modified.
OQMD coordinate artifacts detected and repaired. 137,405 entries with known coordinate issues undergo Type A repair with 100% success rate.
Two-pass deduplication (intra-source + cross-source). 31,997 duplicate entries removed across 21,140 duplicate groups. Cross-source pairs identified.
Hierarchical scoring with 5 sub-scores: geometry, DFT convergence, metadata completeness, novelty, and chemical plausibility. Calibrated against cross-source agreement.
11-gate Strict Gold system. Entries classified as Strict Gold, Gold, Validated, Raw, or Experimental Gold based on validation gates passed.
Typed Parquet export with indexed lookup. Partitioned by source for selective loading. SHA256-checksummed releases with full MANIFEST.
DATA SOURCES
Each source contributes unique strengths. Together they provide comprehensive coverage of the inorganic crystal landscape.
Materials Project
69,279
CC BY 4.0
Highest data quality, complete metadata, reliable provenance. Serves as the gold standard reference.
OQMD
171,780
Non-Commercial
Largest source by volume. Required coordinate repair (137k entries) and symmetry computation (171k space groups).
JARVIS-DFT
25,673
CC0
Commercially safe (CC0). Missing EaH for all entries — computed via internal convex hull in v1.0.0.
OBELiX
498
Per-Article
Experimental Li-ion conductivity data. Integrated as experimental_gold tier with measured transport properties.
AVAILABLE PROPERTIES
47 columns of structural, thermodynamic, electronic, and provenance data. Standardized units and null semantics documented in SCHEMA.md.
Thermodynamic
Structural
Metadata & Provenance
QUICKSTART
The dataset is distributed as typed Parquet files for fast, memory-efficient loading.
import pyarrow.parquet as pq
schema = pq.read_schema("dataset/entries_v4_typed_sample.parquet")
for f in schema:
print(f"{f.name}: {f.type}")
import pandas as pd
df = pd.read_parquet("dataset/entries_v4_typed.parquet")
print(df.shape, list(df.columns))
pf = pq.ParquetFile("dataset/entries_v4_typed.parquet")
for batch in pf.iter_batches(batch_size=1000, columns=["formula","band_gap"]):
df_chunk = batch.to_pandas()
print(f"Loaded {len(df_chunk)} rows")
df_mp = pd.read_parquet("dataset/entries_v4_typed.parquet",
filters=[("source","=","mp")])
DOWNLOAD
Choose the format that works best for your workflow. All files are checksum-verified.
47 columns, typed, indexed
178 MB · entries_v4_typed.parquet
Stratified sample for schema inspection
1.25 MB · entries_v4_typed_sample.parquet
82,925 battery-relevant entries
830 MB · battery_subset_v1.json
41,665 strict Gold electrolyte candidates
492 MB · solid_electrolyte_candidate_subset_v1.json
LICENSE & TERMS
Each entry carries its own license based on its source. Understand the terms before using the dataset commercially.
MP Entries
69,279
CC BY 4.0
Commercial safe. 26.1% of dataset.
JARVIS Entries
25,673
CC0 1.0
Commercial safe. 9.6% of dataset.
OQMD Entries
171,780
Non-Commercial
Non-commercial only. 64.3% of dataset.
Commercial-Safe Edition
94,952
MP + JARVIS
Extract via included script.
⚠ See LICENSE_BREAKDOWN.md for full details. A Commercial-Safe edition (MP + JARVIS, ~94,952 entries) is extractable via scripts/extract_commercial_safe_edition.py.
CITATION
If you use the Scandium Dataset in your research, please cite it.
@misc{scandium-dataset-v1,
title = {Scandium Dataset v1.0: A Harmonized Quality-Scored Foundation of DFT-Computed Properties for AI-Driven Materials Discovery},
author = {Shamique Khan and Scandium Labs},
year = {2025},
howpublished = {\url{https://huggingface.co/datasets/Scandium-Labs/Scandium-Dataset}},
note = {CC BY 4.0 (MP entries), CC0 1.0 (JARVIS entries), Non-commercial (OQMD entries)}
}
DOI pending — Zenodo archival in progress.
FAQ
pd.read_parquet("dataset/entries_v4_typed.parquet", filters=[("source","=","mp")]). The dataset is also partitioned by source in the partitioned/ directory.
VERSION HISTORY
Track the evolution of the Scandium Dataset across releases.
Harmonized MP + OQMD + JARVIS + OBELiX. 267,230 entries. BVSE migration barriers for 23% of Li/Na. CC BY 4.0 / CC0 / Non-commercial licensing. Parquet format with typed columns. Frozen benchmark splits.
OQMD fixes applied (171,764 space groups, 47,807 volume/density). Strict Gold definition finalized. Quality score calibration completed. Cross-source agreement study published.
Hierarchical quality scoring with 5 sub-scores. Tier classification (Gold/Validated/Raw). Cross-source deduplication pipeline established.
First public release. Combined MP + OQMD + JARVIS. Basic validation and deduplication. JSON format.