Below a so-called awesome list with background reading accompanying my talk AI in Offensive and Defensive Cyber at the 2024 One Conference in The Hague. The Hype Cycle section aggregates core themes, the other sections provide for further exploration of specific topics.
- Security of AI Embeddings
- Model-stealing Attacks
- AI Sleeper Agents
- Greedy Coordinate Gradient
- AI Bill of Materials (AIBOMs)
- LLM Jacking
- Open Source Pair Programmers
- LLM 0/1-Day Exploitation
- LLM Proxies and Load Balancers
- LLM-Based (Malware) Pseudocode Analysis
- Dark LLMs and Blackhat GPTs
- Backdoored Model Files
- Mechanistic Interpretability
- AI-Based Intrusion Detection
- LLM-Based Honeypots
- LLM-Based Social Engineering
- Machine Unlearning
- Prompt Injection
- What policy makers need to know about AI (and what goes wrong if they don’t)
- NCSC-NL: The future of cyber attacks with Large Language Models
- The EU Artificial Intelligence Act
- The EU AI Act: National Security Implications
- NCSC-UK: Cyber security risks to artificial intelligence: findings
- NCSC-UK: Guidelines for secure AI system development
- National Security Agency (NSA): Deploying AI Systems Securely: Best Practices for Deploying Secure and Resilient AI Systems
- Australian Government: Policy for the responsible use of AI in government
- AI Safety Ethics & Society: Safe Design Principles
- Verzamelbrief AI en Algoritmes
- RDI (NL): Toezicht op kunstmatige intelligentie
- Monetary Authority of Singapore: Cyber Risks Associated with Generative Artificial Intelligence
- MITRE: ATLAS, a globally accessible, living knowledge base of adversary tactics and techniques against Al-enabled systems
- CyberSecEval 2 - A Comprehensive Evaluation Framework for Cybersecurity Risks and Capabilities of Large Language Models
- AI Vulnerability Database (AVID)
- AI Risk Repository: database of over 700 AI risks categorized by their cause and risk domain
- RAID (Real World AI Definitions)
- Leveraging AI for efficient incident response
- Large Language Models and Intelligence Analysis
- Cybersecurity & ChatGPT - Part 2 - Generative AI for Blue Teams
- The Power of Artificial Intelligence - From Search to Detection Rule at Light Speed
- Analyzing Malware in Binaries and Executables with AI
- Defending AI Model Files from Unauthorized Access with Canaries
- Prompt Injection Defenses
- Prepare for AI Hackers
- Automated LLM Bugfinders
- Applying LLMs to Threat Intelligence
- STRIDE GPT: AI-powered threat modelling leveraging LLMs to generate threat models and attack trees
- Sinon: creates Windows-based deception hosts and uses GPT-4 to generate content (files, emails, etc.)
- AI-Goat: a deliberately vulnerable AI infrastructure simulating the OWASP ML Top 10
- Where the Wild Things Are: Second Order Risks of AI
- Cybersecurity & ChatGPT - Part 3 - Generative AI for Red Teams
- Generative AI in Cybersecurity: Assessing impact on current and future malicious software
- Supply chain attacks and the many different ways I've backdoored your dependencies
- Machine Learning Attack Series: Backdooring Keras Models and How to Detect It
- AI-Powered Bug Hunting - Evolution and Benchmarking
- Automatic Tool Invocation when Browsing with ChatGPT - Threats and Mitigations
- Adversary use of Artifical Intelligence and LLMs and Classification of TTPs
- Red Teaming LLM Applications
- Zhang, Andy K., et al. "Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models.", 2024
- Miles, Farmer, et al. "Reinforcement Learning for Autonomous Resilient Cyber Defence." Black Hat USA 2024, Black Hat, 2024.]
- Mirsky, Yisroel, et al. "The threat of offensive ai to organizations." Computers & Security 124 (2023): 103006.
- Park, Peter S., et al. "AI deception: A survey of examples, risks, and potential solutions." Patterns 5.5 (2024).
- Marchal, Nahema, et al. "Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data." arXiv preprint arXiv:2406.13843 (2024).
- Bezzi, Michele. "Large Language Models and Security." IEEE Security & Privacy (2024).
- Bhatt, Manish, et al. "Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models." arXiv preprint arXiv:2404.13161 (2024).
- Derczynski, Leon, et al. "garak: A Framework for Security Probing Large Language Models." arXiv preprint arXiv:2406.11036 (2024).
- Anwar, Usman, et al. "Foundational challenges in assuring alignment and safety of large language models." arXiv preprint arXiv:2404.09932 (2024).
- Gibney, Elizabeth. "Not all 'open source' AI models are actually open: here's a ranking." Nature. 2024
- Acemoglu, Daron. "The Simple Macroeconomics of AI." Massachusetts Institute of Technology, 2024.
- Goldman Sachs Research. Gen AI: too much spend, too little benefit? 2024.
- Oprea, Alina, and Apostol Vassilev. Adversarial machine learning: A taxonomy and terminology of attacks and mitigations. No. NIST Artificial Intelligence (AI) 100-2 E2023. National Institute of Standards and Technology, 2023.
- Stanford University. 2024 AI Index Report: Measuring Trends in AI
- CIA director Bill Burns and MI6 chief Richard Moore talk to FT editor Roula Khalaf
- AI Model Review: Compare AI Models with Prompts Side by Side
- Cognitive Attack Taxonomy (CAT) of over 350 cognitive vulnerabilities, exploits, and TTPs
- Blog series by Harriet Farlow on the intersection of AI and national security
- I Will Fucking Piledrive You If You Mention AI Again
- How LLMs Work, Explained Without Math
- Lessons after a half-billion GPT tokens
- The “it” in AI models is the dataset
- Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy