ICAPeg Server.
){kind=icon}
Open Source multi-vendor ICAP server
Scan files requested via a proxy server using ICAPeg ICAP server, ICAPeg is an ICAP server connecting web proxies with API-based scanning services and more soon. ICAPeg currently supports hashlocal, clhashlookup & Clamav for scanning the files following the ICAP protocol. If you don't know about the ICAP protocol, here is a bit about it:
ICAP stands for Internet Content Adaptation Protocol. If a content (for example: file) you've requested over the internet to download or whatever, needs adaptation(some kind of modification or analysis), the proxy server sends the content to the ICAP server for adaptation and after performing the required tasks on the content, the ICAP server sends it back to the proxy server so that it may return the adapted content to the destination. This can occur both during request and response.
To know more about the ICAP protocol, check this out.
Before starting to play with ICAPeg, make sure you have the following things in your machine:
-
Golang(latest enough to be able to use go mod)You should install at least go version 1.16.
You can get how to install Golang from here.
-
A proxy server
​ You should configure a Proxy server with ICAPeg, you can get an example from here.
- Clone the ICAPeg repository
$ git clone https://github.com/egirna/icapeg.git-
Run the project with the default configuration in config.toml file
-
Run the following command in the directory where you've installed ICAPeg
$ cd ~/ICAPeg- Build ICAPeg binary
$ go build .- Finally execute the file like you would for any other executable according to your OS, for Unix-based users though
$ ./icapegYou should see something like, ICAP server is running on localhost:1344 .... This tells you the ICAP server is up and running.
- echo
simply it's like when you bing url,so what we send will be received
ClamAV is an open source (GPLv2) anti-virus toolkit, designed especially for e-mail scanning on mail gateways. It provides a number of utilities including a flexible and scalable multi-threaded daemon also helps to scan file quickly.
simply it helps to scan each file we need to check before we send to API.
we will pretend that you want to download a file. Still, you don't know if it is safe or not, so using this service helps you calculate the hash value of any file you download through the icapeg, then it checks if this hash value is available or not in hash_file if it is in the hash_file returns back an exception page, so if you tried to downloadany file of eicar test vrius it will appear an exception page,cause the hash file of thosefile is foun in our hash_file.
​ You can change the default configuration file whatever you want to customize the application.
-
This feature is supported for strings, int, bool, time.duration and string slices only (every type used in this project).
Let's have an example to explain how to map, assume that there is an environment variable in your machine called PORT and you want to assign PORT value to app.port. You should change the value of (port= 1344) to (port= "$_PORT").
Note: before you use this feature please make sure that the env variable that you want to use is globally in your machine and not just exported in a local session.
-
-
-
This section includes the configuration variables that are responsible for the application overall like the port number and current services integrated with ICAPeg.
[app] port = 1344 services= ["echo", "virustotal", "clamav", "cloudmersive"] debugging_headers=true max_file_size_on_memory = 10 # in megabytes write_path_on_disk = "./tmp"
-
port
The port number that ICAPeg runs on. The default port number for any ICAP server is 1344, possible values:
- Any port number that isn't used in your machine.
-
services
The array that contains integrated services names with ICAPeg, possible values:
- Integrated services names with ICAPeg (ex: ["echo"]).
-
debugging_headers
A boolean variable which indicates if debugging headers should be displayed with ICAP headers or not. Debugging headers tell the client shadow service is enabled for example. they start with X-ICAPeg-{{HEADER_NAME}}. possible values:
-
true: Debugging headers should be displayed with ICAP headers.
-
false: Debugging headers should not be displayed with ICAP headers.
-
Any port number that isn't used in your machine.
-
-
max_file_size_on_memory (MB)
The maximum file size in megabytes that will be handled in memory. Files exceeding this size will be stored and processed on disk. The default value is
10MB. -
write_path_on_disk
The directory path where files exceeding
max_file_size_on_memorywill be stored on disk. The default path is"./tmp".
-
-
[echo] section
Note: Variables explained in echo service are mandatory with any service integrated with ICAPeg.
[echo] vendor = "echo" service_caption= "echo service" #Service service_tag = "ECHO ICAP" #ISTAG req_mode=true resp_mode=true shadow_service=false preview_enabled = true# options send preview header or not preview_bytes = "1024" #byte process_extensions = ["pdf", "zip", "com"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file reject_extensions = ["docx"] bypass_extensions = ["*"] #max file size value from 1 to 9223372036854775807, and value of zero means unlimited max_filesize = 0 #bytes return_original_if_max_file_size_exceeded=false
-
-
vendor
The name of the vendor's service, possible values:
- The vendor of that service (ex: "echo")
-
service_caption
Service caption header value.
-
service_tag
Service caption header value.
-
req_mode
A boolean variable that indicates whether request mode is enabled or not, possible values:
- true: Request mode is enabled.
- false: Request mode is disabled.
Get more details about request mode from here.
-
resp_mode
A boolean variable that indicates whether response mode is enabled or not, possible values:
- true: Response mode is enabled.
- false: Response mode is disabled.
Get more details about response mode from here.
-
shadow_service
A boolean variable that indicates whether shadow service mode is enabled or not, possible values:
- true: Shadow service mode is enabled.
- false: Shadow service mode is disabled.
Note: Shadow service mode is used for debugging purposes. it means that when user/client sent a request to ICAPeg, ICAPeg will send an ICAP response with 204 (No modifications) ICAP status code in case ICAP request has (Allow: 204) header or with 200 (OK) ICAP status code with the original HTTP message in case ICAP request hasn't (Allow: 204) header.
-
preview_enabled
A boolean variable that indicates whether message preview is enabled or not, possible values:
- true: Message Preview is enabled.
- false: Message Preview is disabled.
Get more details about Message Preview from here.
-
preview_bytes
It indicates how many bytes of preview are needed for a particular ICAP application on a per-resource basis, possible values:
- Any string numeric value
-
process_extensions
It indicates the file types that should be processed and scanned from the service, and possible values:
- Any string file types.
-
reject_extensions
It indicates the file types that should be rejected by the service, and possible values:
- Any string file types.
-
bypass_extensions
It indicates the file types that should be bypassed by the service and that means the files with these types will not be processed and will not be rejected. They will just be returned to the client as they were sent to ICAPeg from him, and possible values:
- Any string file types.
Notes about extensions arrays:
-
Asterisk sign (*****) means every file type except the ones in other arrays. example:
process_extensions = ["pdf", "zip", "com"] reject_extensions = ["docx"] bypass_extensions = ["*"]
this example means that any file type will be bypassed except docx type which in reject_extensions, pdf, zip, com types which in process_extensions.
-
Only one array from (process_extensions, reject_extensions, bypass_extensions) arrays should has Asterisk sign (*****).
process_extensions = ["pdf", "zip", "com"] reject_extensions = ["docx", "pdf", "*"] bypass_extensions = ["*"]
this configuration is not valid and ICAPeg will not run. another example:
process_extensions = ["pdf", "zip", "com"] reject_extensions = ["docx"] bypass_extensions = ["*"]
this configuration is valid and ICAPeg will run normally.
-
Two arrays can't have the same file type. example:
process_extensions = ["pdf", "zip", "com"] reject_extensions = ["docx", "pdf"] bypass_extensions = ["*"]
this configuration is not valid and ICAPeg will not run. another example:
process_extensions = ["pdf", "zip", "com"] reject_extensions = ["docx"] bypass_extensions = ["*"]
this configuration is valid and ICAPeg will run normally.
-
-
Notes:
- You may not use these variables in your service and you may use them, It depends on your service and It's up to you.
- We will pretend that this service is for file processing and it sends that file to an external API to process it then it gets it back again, So all optional variables depend on that scenario in this service. (It's just a fake scenario service that can do anything not just for processing files).
-
max_filesize
It's the maximum HTTP message file size that the service can process, possible values:
- Any valid integer value.
-
return_original_if_max_file_size_exceeded
A boolean variable that indicates to wether service should return the original file if the file size exceeds the maximum file size or not, possible values:
- true: Returning the original file.
- false: Returning 400 Bad request.
Get more details about request mode from here.
-
-
[hashlocal] section
[hashlocal] vendor = "hashlocal" service_caption= "hashlocal" #Service service_tag = "hashlocal ICAP" #ISTAG req_mode=true resp_mode=true shadow_service=false preview_bytes = "1024" #byte preview_enabled = true# options send preview header or not bypass_extensions = ["*"] process_extensions = ["pdf","exe", "zip"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file reject_extensions = ["docx"] Hash_File = "./hash_file/hash_file_path.txt" # max_filesize = 0 #bytes return_original_if_max_file_size_exceeded=true return_400_if_file_ext_rejected=false http_exception_response_code = 403 http_exception_has_body = true exception_page = "./temp/exception-page.html" # Location of the exception page for this service
-
HashFile It is a string variable has the link of the hash_file_path.txt ,where we save the hashes value we want to be compared with any file we try to download through icapeg.
-
[clhashlookup] section
[clhashlookup] vendor = "clhashlookup" service_caption= "cl-hashlookup" #Service service_tag = "cl-hashlookup ICAP" #ISTAG req_mode=true resp_mode=true shadow_service=false preview_bytes = "1024" #byte preview_enabled = true# options send preview header or not bypass_extensions = ["*"] process_extensions = ["pdf","exe", "zip"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file reject_extensions = ["docx"] scan_url = "https://hashlookup.circl.lu/lookup/sha256/" # timeout = 300 #seconds , ICAP will return 408 - Request timeout fail_threshold = 2 max_filesize = 0 #bytes return_original_if_max_file_size_exceeded=true return_400_if_file_ext_rejected=false verify_server_cert=true bypass_on_api_error=false http_exception_response_code = 403 http_exception_has_body = true exception_page = "./temp/exception-page.html" # Location of the exception page for this service
-
Scan_url It is a string variable has the link of Api ,where we send the files we scanned to.
-
[clamav] section
[clamav] vendor = "clamav" service_caption= "clamav service" #Service service_tag = "CLAMAV ICAP" #ISTAG req_mode=true resp_mode=true shadow_service=false preview_bytes = "1024" #byte preview_enabled = true# options send preview header or not process_extensions = ["pdf", "zip", "com"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file reject_extensions = ["docx"] bypass_extensions = ["*"] socket_path = "/var/run/clamav/clamd.ctl" fail_threshold = 2 timeout = 10 #seconds, the time upto which the server will wait for clamav to scan the results #max file size value from 1 to 9223372036854775807, and value of zero means unlimited max_filesize = 0 #bytes return_original_if_max_file_size_exceeded=false return_400_if_file_ext_rejected=false verify_server_cert=true bypass_on_api_error=false http_exception_response_code = 403 http_exception_has_body = true exception_page = "./temp/exception-page.html" # Location of the exception page for this service
-
socket_path It is a string variable it helps sending the HTTP msg body to the ClamAV through antivirus socket.
-
-
-
-
How to add a new service for a new vendor.
After reading the above markdown, read the next section because it may help you while implementing your new service.
-
This is a developer guide which includes a lot of functions to help the developer while implementing his new service.
-
You will have to restart the ICAP server each time you change anything in the config file.
-
You will have to restart squid whenever you restart the ICAP.
-
You need to configure your network(or your browser)'s proxy settings to go through squid.
This project is still a WIP. So you can contribute as well. See the contributions guide here.
ICAPeg is licensed under the Apache License 2.0.