Here is a latent structure of about 60 variables, found to exist within the human handwriting of the number 7.
This is a normally distributed feature.
See Vae4 (below)
Vae1 is a foundational model. It uses a multivariate Gaussian as the latent distribution and either a continuous-bernoulli or gaussian reconstruction density, depending on the data type.
It is designed to be a very robust and explicit VAE model. This explicit coding style allows for significant access to the inner workings of the model. Thus, Vae1 can be quickly extended into novel experimental architectures. Below are some of the design choices that make this possible.
Every component of Vae1 (encoder, sampling layer, etc) passes dictionary objects between each other. This allows extensions, wherein new objects need to be passed, to be easily appended.
The latent probability layers/likelihoods are written explicitly. This reveals the simple laws a variational auto encoder must abide by. We see that the heart of most VAE models, the family of multivariate gaussian densities, contains parameterized elements that can be added and scaled, while still being inside the span of possible multivariate-Gaussian densities. This is to say, the space of parameterized latent density functions is closed under addition and scalar multiplication. Indeed, parameterized multivariate gaussians describe a mathematical Vector space.
NOTE: Although it is interesting to explicitly include the exponential-form of the probability densities to be used for the cost function, for an applied project one should always use the log-density! To see why this is, please see my R repository concerning log-density vs. exponential density cost functions at this repo.
After training a Vae1, the model state is saved. Analysis-specific variants of Vae1 are then able to load in this saved state. The analysis-specific Vae1 variants are well equipped to traverse the latent density using many parameterized, and manual algorithms. This encapsulation allows different types of analysis-specific extensions to be written apart from the training of the model.
Data is often recognized as existing in some high dimensional manifold. This manifold represents the behavior of the underlying generator function, and thus ML architectures like invertible flow networks, can learn these associated hyper-dimensional smooth structures.
However in real life, there are often many "states" that the generator function of the observed data can be in. In the study of dynamical systems, these "states" are usually intuited as different parameterizations of the underlying generator function (usually due to an unseen bifurcation event).
But that's neither here nor there, the point is these "states" exist in data.
To a geometer, these would be described as "discrete structures" in the manifold. To a statistician, the data would be described as multi-modal, or "clustered". At anyrate, Vae4 uses a categorical, multivariate gaussian joint-density as the latent probability distribution to learn this manifold without the use of any labels.
Vae4 categorizes the value as an element in the 5-cluster. To answer the question of why the anomalous data is categorized as a 5, we may investigate how it compares to a known-5 that is already an element of the 5-cluster.
Note: Vae2,3,4 are not in this repo at this time. Some of these experimental models may be publishable.