An application designed to leverage the efficiency of small language models by implementing prompt engineering, and inferencing techniques to increase their accuracy.
We need to install some dependencies so that we can build and run the project. The first thing we need to install is Podman. Follow your OS specific instructions to install podman.
Create a folder for the models that you want to use. These should probably be .gguf files.
After that, if you want to run the containers with a gpu, you'll need to install the nvidia continer toolkit along with the appropriate drivers if needed. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest
Refer to the proper documentation for rocm.
This command downloads everything and builds the app for you. A models folder is still needed in the folder that SLaPE is ran in.
go install github.com/StoneG24/slape@latestTo run the app you need to turn on the docker socket. This allows the app to talk to the socket and controll its components.
For non-rootful use cases
sudo systemctl start dockerTo close the socket on linux,
sudo systemctl stop dockerFor windows this process is managed by docker desktop.
Containers are very useful for making reproduceable builds but the can take up a lot of space over time. This coupled with the fact that we also have to save storage space for models means that we need to be more cognicent of that fact. Here are some tips to remove dead resources in this project.
This command will tell you how much of your disk is currently being used by docker
docker system dfThese commands are good for cleaning up these old docker resources. SLaPE should clean these up but, currently, if errors occur it won't.
docker container prunedocker image prunedocker builder pruneOur code uses go doc comments as a way of effectively documenting our code.
This tool is included in our tool list of the project.
To run the doc server locally run this command
godoc -index -notes="BUG|TODO|NOTE"And travel to this url in your browser http://localhost:6060/pkg/github.com/StoneG24/slape/.
Here are some of the research papers that we used to aid us in development.