Definition
The process of fine-tuning AI models to prevent them from generating harmful, unethical, or inappropriate content. This typically involves training the model on examples of undesirable outputs and teaching it to refuse such requests.
Why it matters (in Poovi’s context)
Crucial for ensuring AI is used responsibly, but as demonstrated, it’s not foolproof and can be bypassed. Understanding its limitations is key to improving AI safety.
Key properties or components
- Aims to prevent harmful outputs
- Often involves reinforcement learning from human feedback
- It’s a probability adjustment, not knowledge deletion
- Can be circumvented by novel inputs
Contradictions or debates
The claim that AI safety is a ‘habit painted on top’ suggests it’s a superimposed layer that doesn’t fundamentally alter the AI’s core knowledge, making it vulnerable to ‘out of distribution’ inputs.