This repository is a pytorch implementation of this paper accepted by IEEE S&P 2023.
Overall, this code conducts a series of multimodal analyses to understand (hateful) memes and the evolution process. First, we obtain multimodal meme representations, i.e., CLIP embeddings, and perform DBSCAN clustering to form meme clusters. We then conduct automatic annotation and hate assessment based on the clusters. Most importantly, this code provides a framework for identifying hateful meme variants and potential influencers. Given given a popular hateful meme and the 4chan dataset, we automatically retrieve variants by extracting visual and visual-linguistic semantic regularities.
pip install -r requirements.txt
Download 4chan meme URLs and 4chan posts
Then download memes locally with the URLs and build image-text pairs using two datasets.
Note: The semantic regularities in CLIP emerge from large datasets. If you intend to reproduce our work, please ensure to download as many memes as possible (4.3M in total).
Now we describe the running pipeline with a set of test data.
To get all image and text embeddings:
python inference.py --data_file data/4chan.txt \
--model_file the/finetuned/model/path \
--save_dir data
To get non-duplicated image embeddings only:
python inference.py --data_file data/4chan_images_only.txt \
--model_file the/finetuned/model/path \
--save_dir data
- The input
4chan.txtis a long list of dicts with image location, comment, and the posted time as keys. CLIP will compute both text and image embeddings and save into a npz file sequentially. The finetuned CLIP model can be downloaded here. By changing4chan.txtto4chan_images_only.txt, we can compute non-duplicated image embeddings only to improve computing efficiency later.4chan_images_only.txtis also a long list of dicts with image location and its phash as keys. - Output: Image-text or image embeddings saved as
embeddings.npzorimage_embeddings.npy.
python extract_visual_regularities.py --meme HappyMerchant \
--image_dict data/4chan_images_only.txt \
--image_embeddings data/image_embeddings.npy \
--image_root data/images \
--lower 0.85 \
--higher 0.91 \
--influencer_lower 0.91 \
--final_thred 0.94 \
--save_dir result/visual_regularity
- This script first identifies the meme variants given a hateful meme. Then, for each variant, it further automatically estimates the possible influencers. We use two popular hateful memes as case studies. Type
HappyMerchantorPepeTheFrogfor the input--meme.--lower,--higher,--influencer_lower, and--final_thredare four thresholds that needs to be manually determined and evaluated. We provide the detailed explaination and the default thresholds in the script. - Output: variant-influencer pairs (.npz), a graph with memes as nodes and pairing relation as edges (.gexf), visualization of variant-influencer pairs in top-20 communities (.png)
python extract_visual_linguistic_regularities.py --meme HappyMerchant \
--model_file the/finetuned/model/path \
--entity_dir data/entities \
--data_file data/4chan.txt \
--image_dict data/4chan_images_only.txt \
--image_embeddings data/image_embeddings.npy \
--image_root data/images \
--save_dir result/visual_linguistic_regularity
- This script retrieves meme variants given a hateful meme and a list of entities and further conducts temporal analysis.
--entity_diris the directory of four types (People, GPE, NORP, ORG) of extracted entities. - Output: variant-entity pairs (.npz), variant occurrence (.csv), popular variants (in top-2 variants) (.png)
Visual Semantic Regularities: 3.3K variant-influencer pairs + Annotation file
Visual-linguistic Semantic Regularities: 120 variant-entity pairs + Annotation file
Please find the resulting datasets and the annotations at link.
If you use or find this source code or dataset useful please cite the following work:
@inproceedings{QHPBZZ23,
author = {Yiting Qu and Xinlei He and Shannon Pierson and Michael Backes and Yang Zhang and Savvas Zannettou},
title = {{On the Evolution of (Hat
534F
eful) Memes by Means of Multimodal Contrastive Learning}},
booktitle = {{IEEE Symposium on Security and Privacy (S\&P)}},
publisher = {IEEE},
year = {2023}
}
- This project is partially funded by the Helmholtz Association within the project "Trustworthy Federated Data Analytics" (TFDA (funding number ZT-IOO1 4).
- We also gratefully acknowledge the support of Google (Perspective API) and Rewire for our experiments.