Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

ABSTRACT

As language models grow in capability and scale, they increasingly generate outputs that reflect the biases, toxicity, and harmful stereotypes present in their internet-scraped training data. We introduce the Process for Adapting Language Models to Society (PALMS), a methodology for aligning models with specific societal values using small, meticulously hand-crafted datasets. By fine-tuning GPT-3 on a targeted dataset of only 80 human-written, values-aligned Q&A examples, we demonstrate a significant reduction in toxicity and bias across multiple demographic categories (race, gender, religion) without degrading the model's core generative capabilities.