Chinese Menu Reader is a mobile application that takes a picture of a dish name on a Chinese menu and gives you an English translation of the dish name, a description, reviews, photos and tags.
This repository contains the code for the Python server. For the iOS application, check out this repository.
The iOS application takes the image, crops it, and uploads it to the Python server. The Python server does some basic image processing (smoothing, converting to grayscale, binarizing/thresholding) and then uses Google Tesseract OCR to get a string of Chinese characters from the image. If Tesseract fails to recognize any characters in the image, the server thins the image (using Stentiford's preprocessing steps and Scikit-Image's skeletonize() function) and runs it through Tesseract again.
If Tesseract returns characters, the server looks them up first in a dishes table. If a dish is found, it returns a JSON object representing that dish. If not, it attempts to translate the characters relying first on a list of common food words and finally on CEDICT, an open source Chinese dictionary maintained by MDBG.
-
Clone the repository.
-
Open the directory with Xcode.
-
In
Server.h, changekBaseURLto the address of your Python server (find the server code here) -
Connect your iPhone.
-
Change the target device from the simulator to your connected device (top left corner).
-
Build and run!
For best results, take a picture with your camera about 6 inches away from the menu and then zoom in and crop around the name of the dish you want to translate. Try to hold the menu straight and avoid blur if possible.
Tesseract has trouble recognizing some Chinese characters that have vertical space between two halves or radicals. I haven't yet figured out how to address this problem, so the app will have trouble recognizing dish names with these characters, especially if the font accentuates these spaces.