The Scene Identification and Tagging (SIT) algorithm based on Fuzzy Description Logics.
The theoretical background of SIT can be found in
If you did not already, in an Ubuntu machine, install Java JDK 8 with
sudo apt update
sudo apt install openjdk-8-jdkLocate the Java virtual machine installed with
jrunscript -e 'java.lang.System.out.println(java.lang.System.getProperty("java.home"));' It might return /usr/lib/jvm/java-8-openjdk-amd64/jre, which will be used later.
FuzzySIT depends on FuzzyDL, which have been tested at the version of the 09 January 2019.
The FuzzyDL library comes as a jar that should be placed in the /libs folder.
FuzzyDL depends on Gurobi which gives a free license for Academic and non-commercial purpose.
After having a Gurobi account and the related license, download the Gurobi Optimizer tools. We used the version 8.0.1, which can be downloaded in Ubuntu with
wget https://packages.gurobi.com/8.1/gurobi8.1.0_linux64.tar.gzThen, uncompress it in the /opt folder with
sudo tar -xf gurobi8.1.0_linux64.tar.gz -C /opt/Now add the following environmental variables in the .bashrc and configure JAVA_HOME as found previously
export GUROBI_HOME="/opt/gurobi810/linux64"
export PATH="${PATH}:${GUROBI_HOME}/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${GUROBI_HOME}/lib"
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/jre"Close and reopen the terminal, then generate the license file with
grbgetkey <licence_id>where <licence_id> is given by Gurobi when you activate your license.
The latter command should generate a file gurobi.lic in your home.
Now you can test FuzzyDL reasoner by referring to the README file of the FuzzyDL package.
To test FuzzySIT, clone it and (eventually) checkout in a relevant branch
git clone https://github.com/TheEngineRoom-UniGe/fuzzy_sit.gitTo compile it go in the fuzzy_sit folder just cloned and run
gradle wrapper --gradle-version 6.8which might require to install Gradle with sudo apt install gradle.
Then run
./gradlew buildFinally, run the SceneHierarchyTest example with
./gradlew runExampleIf your machine does not have a graphical user interface, the example will fail at the end of the script
If the example gives an error like
Exception in thread "main" java.lang.NullPointerException
at it.emarolab.fuzzySIT.core.SITTBox.querySubConcept(SITTBox.java:382)
it might be the case that the guroby path is not correctly configured.
To run other examples or to generate a jar of FuzzySIT as a library for your development we suggest to open the fuzzy-sit project with an IDE (e.g., Intellij IDEA) configured based on your .bashrc (e.g., opened from terminal in Ubuntu).
SIT requires a FuzzyDL ontology (i.e., a file .fuzzydl) that is formatted in a specific manner.
It requires a header as
(define-fuzzy-logic zadeh)
(define-primitive-concept Object *top*)
(define-primitive-concept Scene *top*)
(disjoint Object Scene)
(disjoint SpatialObject Scene)
(implies Object SpatialObject)
Then you should define the type of elements in the scene, e.g., for Spere and Cone use
(implies Sphere Object)
(implies Cone Object)
Later you should define the type of relations among elements, e.g., Right and Left
(range isRightOf Object)
(domain isRightOf SpatialObject)
(range isLeftOf Object)
(domain isLeftOf SpatialObject)
Note that this implementation of fuzzyDL is case-sensitive, and it parses variable names with the CamelCase standard
(a known issue occurs when variables end with a number, e.g., Right1).
Always add is to the definition of the relations, while Of is optional.
Finally, we should define the features used by SIT to learn. Those are all the combinations of the types of elements and relations above obtained through reification, e.g.,
(define-concept SphereRight (and Sphere (some isRightOf SpatialObject)))
(define-concept SphereLeft (and Sphere (some isLeftOf SpatialObject)))
(define-concept ConeRight (and Cone (some isRightOf SpatialObject)))
(define-concept ConeLeft (and Cone (some isLeftOf SpatialObject)))
In this case, the SphereRight class represents a feature in the scene where
"a sphere as something on the right".
You can configure and add as many types of elements and relations you like but be aware that computation complexity scales exponentially.
The core of FuzzySIT is composed by three main classes
FuzzySITBasewhich is a container for common constants and configuration, e.g.,FLAG_LOG_SHOW.SITABoxwhich allows encoding and recognising a scene. A newSITABoxshould be instantiated for each observed scene.SITTBoxwhich contains the scene categories in the memory. Only an instance ofSITTBoxshould be used for each application.SITTBoxallows- learning new scenes by the means of
SITABox, - access and visualize the hierarchy of scenes structured on the basis of the ontology,
- save or lead ontologies.
- learning new scenes by the means of
The FuzzySIT package comes also with some utilities used for testing purposes.
In particular, in the runnableSITutility contains the following runnable scripts.
Both the FuzzySIT core, and the utilities read and write on auxialry file stored in the resources folder.
It is a script to test the FuzzyDL reasoner by loading an ontology and solving queries.
Is an example to show the usage of the FuzzySIT API base on a toy scenario.
Is a graphical user interface to simulate 2D arranges of objects.
It is based on the JavaFx library, and it is currently commented since deprecated.
Show a simulation of object configured as shown in the paper, and it concerns the hierarchy of spatial scenes in memory. This example is based on fuzzy degree given as input.
Show a simulation of object configured as shown in the paper, and it concerns the hierarchy of spatial scenes in memory.
This example is different from SceneHierarchyTest since it uses fuzzy kernels to compute degrees based on the 2D object positions.
Is a simulation done to publish results about a cognitive like memory, involving consolidation and forgetting of scene categories learned during time.
Is a preliminary simulation based on a Montecarlo approach to reconstruct a scene given its learned category. This example is not in a stable version.
Is a simulation to log the SIT computation complexity to be later evaluated.
You can run the computation complexity test with the command
gradle clean runComputationTest -Pconcepts='2,6' -Prelations='2,4' -Pscenes='2,2,2' -Pelements='2,3,4' -PtasksLimit='2'where
-Pconceptssets the number of possible types of object in the ontology,-Prelationssets the number of possible relations among objects in the ontology,-Pscenessets the number of learning and recognition phase to perform (i.e., number of memory items at the end of the evaluation`),-Pelementssets the number of objects in each scene to evaluate. From this the number of relations in a scene is derived having, between each object added consegutively, (i) two relations plus (2) another relation with a probability of 0.5.-PtasksLimitsets the maximum number of threads to be used concurrently for the simulation.
The test is performed for each combination of concepts, relations, scenes and elements.
The results are store as csv files in the folder src/main/java/resources/ComputationComplexityTest/log.
CSV data is all express in millisecond or integer (e.g., for indices) and arranged in columns ordered as
CSV data is all express in millisecond or integer (i.e., indices) and arranged in columns as: A,B,C,...,Q where
- A) An ordered identified based on the creation timestamp,
- B) A string identifying the ontology complexity (i.e.,
C-D), - C) The Number of concepts in the ontology,
- D) The Number of relations in the ontology,
- E) Number of elements in the scene,
- F) Number of roles in the scene,
- G) Number of items in the memory (it should be an array with all equals elements, which size identifies the number of trials, and each element the nodes in memory when to stop),
- H) The encoding time before having learned a new scene,
- I) The recognition time before having learned a new scene,
- J) The learning time,
- K) The structuring time,
- L) The encoding time after having learned a new scene,
- M) The recognition time after having learned a new scene,
- N) The total time,
- O) Thread memory usage in bytes (this is an approximation based on
ThreadMXBean.getThreadAllocatedBytes, which might be unreadable), - P) Number of categories pre-recognised,
- Q) Number of categories post-recognised.
Author: Luca Buoncompagni,
Affiliation: University of Genoa,
Date: 2021
Email: luca.buoncompagni@edu.unige.it (please, prefer GitHub issue
52FE
if appropriate)
Licence: The FuzzySIT packager are under the GNU General Public License v3.0, while the dependences (i.e., Java,
FuzzyDL, Gurobi, and jgrapht) are subjected to their relative licences.