Topics include AI red teaming, jailbreaking, prompt injection, prompt hacking, AI/ML safety and security.
This resource is provided by Learn Prompting, your go-to resource for mastering Generative AI.
Discord • Twitter (X) • LinkedIn • Newsletter • Free Intro Prompt Hacking Course • Free Advanced Prompt Hacking Course
- Introduction
- Resource Categories
- Contributing
Prompt Hacking is an emerging field that covers the intersection between AI and Cybersecurity. It involves exploring the outer edges of LLM behavior through adversarial prompts and prompt injection techniques.
Due to its novelty, online resources are few and far between.
This repository aims to provide a good overview of materials and tutorials that help expose vulnerabilities, document offensive research, and promote a better understanding of model limitations.
Stay informed with expert analyses, tutorials, and research articles on AI security.
- InjectPrompt – A comprehensive catalogue of novel jailbreaks, prompt injections, and system prompt leaks.
- LearnPrompting Prompt Hacking – A step-by-step guide on prompt injection and model exploitation.
- AIBlade – Directory of AI red teaming tools and methodologies.
- EmbraceTheRed – Practical experiments and insights from active AI red teamers.
- Joseph Thacker – First-person narratives on red teaming and LLM vulnerability research.
- Protect AI Blog – Enterprise insights on AI security along with open-source tooling.
- AWS Generative AI Security – Guidance on secure AI architectures and compliance.
- Lakera AI Blog – Interactive campaigns and red teaming experiments.
- Securiti AI Security – Articles on governance, risk, and compliance in the AI realm.
- PurpleSec AI & ML Security – Application of broad cybersecurity principles to AI/ML threat models.
- Wiz AI Security Articles – Curated executive insights on AI risk trends.
- Lasso Security Blog – In-depth research on prompt injection and adversarial LLM behavior.
- Cisco AI Safety – Strategic perspectives on embedding safety in AI innovation.
- Microsoft Security: AI & ML – Deep dives into threat modeling and responsible AI practices.
- Vectra AI Cybersecurity Blog – How AI can be used to defend against emerging AI-driven threats.
- Reversec AI Security Articles – Reversec research articles on LLM application security and prompt injection
- LearnPrompting’s Prompt Hacking Discord – A hub for focused discussions on prompt hacking and red teaming.
- Pliny's BASI Discord – Covering behavioral AI safety and integrative research.
- AI Safety & Security Discord – A space for general discussions on AI risk, safety, and adversarial testing.
- AI Village Discord – Associated with DEFCON’s AI Village, emphasizing practical red teaming.
- InfoSec Prep – For cybersecurity certifications with crossover AI security discussions.
- Hack The Box Discord – Active community blending traditional hacking with GenAI discussions.
- Laptop Hacking Coffee – Casual technical discussions on red teaming and ethical hacking.
- WhiteHat Security – Sharing insights on hacking and AI-enabled attacks.
- ChatGPT Jailbreak Reddit – Focus on testing limits of OpenAI’s models.
- ClaudeAI Jailbreak Reddit – Specifically for Anthropic’s Claude model vulnerabilities.
- NetSec Reddit – Broader network security discussions that occasionally intersect with AI.
- Cybersecurity Reddit – General cybersecurity topics, including AI/ML threats.
- CybersecurityAI Reddit – Dedicated to AI-related cybersecurity challenges.
- Artificial Reddit – Community discussions on AI safety, policy, and alignment.
- Introduction to Prompt Hacking – Beginner-focused course covering prompt injection fundamentals.
- Advanced Prompt Hacking – Explores adversarial prompting and defense strategies in detail.
- Prompt Engineering for Beginners (DeepLearning.AI) – Effective prompt crafting using OpenAI models.
- Prompt Engineering Crash Course (DataCamp) – Hands-on training for prompt engineering with ChatGPT.
- Introduction to Prompt Engineering – Techniques for writing optimized prompts.
- Intro to LLMs and Prompting (Google Cloud) – Overview of LLM concepts and practical prompting within a cloud framework.
- Prompt Engineering on LearnAI – Community-driven prompt engineering resources.
- Generative AI Prompting Basics (Google) – Foundational course for generative AI prompting.
- Prompt Engineering on Fast.ai – Integrated with broader practical machine learning applications.
- Prompt Engineering Guide (GitHub) – An open-source resource for advanced prompt design.
- Intro to AI Safety and Prompt Testing – Materials focused on the safety aspects of prompt exploitation.
- Spikee Tutorial Series – Series of tutorials on how to use spikee, an open-source tool, to test LLMs and LLM application resiliance to different jailbreaking and prompt injection attacks
- AI Red-Teaming and Security Masterclass – Comprehensive training on threat assessment, red teaming methodologies, and effective countermeasures.
- Attacking AI – An advanced course focusing on offensive AI security and adversarial strategies.
Keep updated with competitions, workshops, and summits that drive practical learning and networking:
- HackAPrompt – Online competitions aimed at discovering innovative ways to jailbreak AI systems.
- RedTeam Arena – A gamified platform to identify and exploit LLM vulnerabilities.
- AI Security Summit 2024 – Executive-level summit addressing current AI security challenges.
- AI Red-Teaming Workshop (SEI) – Workshops exploring advanced red teaming techniques.
- A 85A2 ISec Workshop – Academic insights held alongside major ML conferences.
- AI Security Symposium 2024 – Sessions on risk mitigation and AI adoption strategies.
- Black Hat USA 2024 AI Summit – Practical sessions on LLM vulnerabilities integrated in the Black Hat conference.
- AI Cybersecurity Summit 2025 (SANS) – Technical labs and sessions bridging AI and cybersecurity.
- Generative AI Red Teaming Challenge 2024 (Clova) – Competitive event to stress-test and enhance the resilience of LLMs.
A collection of repositories, tools, and research papers that document methods of bypassing LLM safeguards:
- L1B3RT4S – Repository featuring various jailbreak prompt sets and evaluation tools.
- Jailbreak Tracker – Live dashboard for monitoring known jailbreak prompts.
- Awesome GPT Super Prompting – Curated list of red teaming and jailbreak resources for GPT-based models.
- Jailbreaking in GenAI: Techniques and Ethical Implications – Guide discussing the practical methods and ethical considerations.
- Jailbreaking LLMs: A Comprehensive Guide (With Examples) – Step-by-step guide showcasing real-world examples.
- AI Jailbreak – IBM – Overview of jailbreak risks and mitigation strategies.
- AI Jailbreaking Demo: How Prompt Engineering Bypasses LLM Security Measures – Video walkthrough highlighting bypass techniques.
- Prompt Injection vs. Jailbreaking: What’s the Difference? – Comparative discussion on prompt injection and jailbreak strategies.
- GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts – Research outlining an automated framework for generating jailbreak prompts.
- DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak – Novel approach applying diffusion models in jailbreak generation.
- SoP: Unlock the Power of Social Facilitation for Automatic Jailbreak Attack – Framework that leverages social engineering concepts for jailbreaks.
- Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation – Technique to boost jailbreak effectiveness using adversarial translation.
- AI Jailbreaks: What They Are and How They Can Be Mitigated – Additional insights into mitigating jailbreak vulnerabilities.
- How Microsoft Approaches AI Red Teaming – Insight into Microsoft's methodologies.
- AI Red Teaming in 2024 and Beyond – Discussion of emerging trends.
- Red Teaming AI: What You Need To Know – Essential introduction to red teaming practices.
- Building Trust in AI: Introduction to Red-Teaming – Fundamentals of effective AI red teaming.
- What's Next for AI Red-Teaming? – Future directions and challenges.
- LLM Security Chronicles – YouTube playlist on practical LLM application security and red teaming
- How AI Jailbreaks Work and What Stops Them? – An overview of the methods behind AI jailbreaks.
- AI Jailbreaking Demo – Demonstration of bypass techniques in practice.
- How Jailbreakers Try to “Free” AI – Examines the mindset behind prompt injection attacks.
- Defending Against AI Jailbreaks – Strategies for protecting LLMs from adversarial prompts.
- AI Jailbroken in 30 Seconds?! – A demonstration of rapid prompt injection.
- Anthropic's Stunning New Jailbreak – Review of a breakthrough jailbreak technique.
- New AI Jailbreak Method Shatters Models – Exploration of experimental bypass strategies.
- Jailbreaking AI - Deepseek & Prompt Tricks – Advanced prompt engineering in real-world scenarios.
- First to Jailbreak Claude Wins $20,000 – Competitive challenge highlighting effective prompt injection.
If you have suggestions, improvements, or additional resources to include, please review our CONTRIBUTING.md guidelines and submit a pull request.
This repository aims to deliver critical, reliable resources for advancing prompt hacking research. We encourage rigorous testing, honest discussions, and the sharing of proven methodologies to foster safe and responsible exploration in this field.