C++ transfer learning APIs

Backpropagation

namespace coral

class SoftmaxRegressionModel¶

#include <softmax_regression_model.h>

An implementation of the softmax regression function (multinominal logistic regression) that operates as the last layer of your classification model, and allows for on-device training with backpropagation (for this layer only).

Input for this layer must be an image embedding, which should be the output of your embedding extractor (the backbone of your model). Once given here, the input is fed to a fully-connected layer where weights and bias are applied, and then passed to the softmax function to receive the final probability for each class.

Public Functions

SoftmaxRegressionModel(int feature_dim = 0, int num_classes = 0, float weight_scale = 0.01, float reg = 0.0)¶

Inputs: feature_dim (int): The dimension of the input feature (length of the feature vector).

num_classes (int): The number of output classes. weight_scale (float): A weight factor for computing new weights. The backpropagated weights are drawn from standard normal distribution, then multiplied by this number to keep the scale small. reg (float): The regularization strength.

float GetAccuracy(const Eigen::MatrixXf &mat_x, const std::vector<int> &correct_labels)¶

Computes how many of the labels are classified correctly for a given input.

Inputs: tensor mat_x of size NXD of input data, Vector labels of size N. Output: a decimal float between 0 and 1 of the accuracy. Accuracy = number classified correctly / total inputs

std::vector<int> RunInference(const Eigen::MatrixXf &mat_x)¶

Runs an inference using the current weights.

Inputs: tensor mat_x of size NxD of input data (image embeddings) Outputs: Vector of inferred label index for each input embedding

void Train(const TrainingData &data, const TrainConfig &train_config, float learning_rate)¶: Finds the optimal weights and biases for the last layer according to the specifications of weight_updater (sgd_updater for this model) and train_config.

void AppendLayersToEmbeddingExtractor(const tflite::Model &embedding_extractor_model, flatbuffers::FlatBufferBuilder *fbb) const¶: Appends the learned weight and bias values to embedding extractor tflite model and saves to file.

Private Functions

float GetLoss(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &labels)¶

Calculates the loss of the current model for the given data, using a cross-entropy loss function.

Inputs: tensor mat_x of size NxD of input data (image embeddings) to test wehere N is number of inputs and D is the dimension of each input. Tensor labels of size NxC of labels in one-hot vector format where C is the number of classes possible in the model Output: value of the averaged cross entropy loss for each input embedding.

std::vector<Eigen::MatrixXf> GetGrads(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &labels)¶

Calculates the gradients of the current weights using cross entropy loss as the loss function and backpropogation using CrossEntropyGradient, SoftmaxGradient, and FullyConnectedGradient.

Uses same inputs as get_loss(). Outputs: vector of weights of the model: mat_w_ and vec_b_

Private Members

Eigen::MatrixXf mat_w_¶: Weights and biases of the model that we want to learn.

Eigen::MatrixXf vec_b_¶

float logit_min_¶

Minimum / maximum values of the last fully connected layer’s output.

These values will be learned from datga and used to quantize the output of the learned fully-connected layer.

float logit_max_¶

int feature_dim_ = 0¶: Parameters set when initializing model.

int num_classes_ = 0¶

float weight_scale_ = 0.01¶

float reg_ = 0.0¶

std::vector<Eigen::MatrixXf> caches_¶: Caches to improve backpropogation performance by storing intermediate results of forward operators, in order of logits, probs, loss.

struct TrainConfig¶

#include <softmax_regression_model.h>

A struct to define the training configuration to be used for last-layer training.

Public Functions

TrainConfig(int num_iter, int batch_size, int print_every)¶

Public Members

int num_iter¶

int batch_size¶

int print_every¶

struct TrainingData¶

#include <softmax_regression_model.h>

A struct to define the training and validation input to a model.

Public Members

Eigen::MatrixXf training_data¶

Eigen::MatrixXf validation_data¶

std::vector<int> training_labels¶

std::vector<int> validation_labels¶

[adapter.h source]

namespace coral

class MultiVariateNormalDistribution¶

#include <multi_variate_normal_distribution.h>

Multi variate normal distribution implemented with Eigen library.

Public Functions

MultiVariateNormalDistribution(const Eigen::VectorXf &mean, const Eigen::MatrixXf &cov)¶

Eigen::MatrixXf Sample(int num)¶

Samples ‘num’ samples from distribution.

Returns a [dim, num] shape matrix.

Private Members

Eigen::VectorXf mean_¶: Mean of the distribution.

Eigen::MatrixXf cov_¶: Covariance matrix of the distribution.

Eigen::MatrixXf p_¶: Multiplies this matrix with a random variable X which is drawn from N(0, I) will produce a sample drawn from N(0, cov_).

int dim_¶: The dimension of the covariance matrix.

Eigen::EigenSolver<Eigen::MatrixXf> solver_¶: Eigen solver which is used to compute eigen value and eigen vectors.

std::normal_distribution<float> rand_gaussian_¶: Gaussian random number generator.

[adapter.h source]

namespace coral

Functions

Eigen::MatrixXf CrossEntropyLoss(const Eigen::MatrixXf &c, const Eigen::MatrixXf &p)¶

Computes the cross entropy between two probability distributions using CE(c,p) = - sum(c*log(p)) and returns the average loss of the batch.

Inputs: vector of size 2 of [c, p] Both c and p tensors are size NxC where N is the number of distributions and C is the length of each distribution. When used with softmax, p is the probability output from softmax and c is a batch of one-hot vectors for class labels. Output: vector of size 1; tensor is 1x1 containing the average loss value.

Eigen::MatrixXf CrossEntropyGradient(const Eigen::MatrixXf &c, const Eigen::MatrixXf &p)¶

This class computes the gradient of the Cross Entropy Loss with respect to each of the elements in probability distribution p A good reference for this is: https://deepnotes.io/softmax-crossentropy Inputs: vector of size 2 of [c, p] c and p described in CrossEntropyLoss class; Loss is output of the Compute method in CrossEntropyLoss class.

Output: vector of size 1; tensor is NxC gradient with respect to p

Eigen::MatrixXf FullyConnected(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &mat_w, const Eigen::MatrixXf &mat_b)¶

Forward pass operator for the fully connected layer that computes Y = X*W + b A good reference for this is: http://cs231n.github.io/linear-classify/#score Input: vector of tensors in order of data mat_x, weights mat_w, and bias vec_b.

mat_x is size NxD where N is number of inputs and D is number of dimensions. mat_w is size DxC. vec_b is size 1xC. Output: vector of size 1 that is layer output Y

std::vector<Eigen::MatrixXf> FullyConnectedGradient(const Eigen::MatrixXf &mat_x, const Eigen::MatrixXf &mat_w, const Eigen::MatrixXf &b, const Eigen::MatrixXf &dmat_y)¶

Backward pass operator that computes gradients of the inputs to the fully connected layer A good reference for this is: http://cs231n.stanford.edu/2017/handouts/linear-backprop.pdf Input: vector of tensors in order of data mat_x, weights mat_w, bias b, grad dmat_y.

The tensors mat_x, mat_w, vec_b are as described in FullyConnected class, dmat_y is size NxC. Output: vector of tensors of gradients in order of dmat_x, dmat_w, dvec_b and correspond in size to mat_x, mat_w, vec_b respectively

Eigen::MatrixXf Softmax(const Eigen::MatrixXf &logits)¶

Forward pass operator for the softmax classifier layer that computes the probibilities of each sample in the tensor being in each class.

A good reference for this is: http://cs231n.github.io/linear-classify/#softmax Input: vector of size 1 of unnormalized logits; tensor is NxC array where N is number of inputs and C is number of classes. Output: vector of size 1 of normalized probabilities; tensor is NxC array.

Eigen::MatrixXf SoftmaxGradient(const Eigen::MatrixXf &logits, const Eigen::MatrixXf &dprobs)¶

This class computes the gradient of the Softmax operator with respect to each of the elements in the vector of unnormalized logits.

A good reference for this is: https://deepnotes.io/softmax-crossentropy Input: vector of size 2 of tensors [logits, dprobs]. logits is NxC array where N is number of inputs and C is number of classes. dprobs is NXC array of gradients of Loss with respect to softmax output. Output: vector of size 1; tensor is NxC gradient of Loss with respect to logits.

Eigen::MatrixXf SoftmaxLocalGradient(Eigen::MatrixXf::RowXpr prob)¶: Helper function to compute local gradient of softmax.

void SgdUpdate(const std::vector<Eigen::MatrixXf> &grads, float learning_rate, std::vector<Eigen::MatrixXf*> weights)¶

Updates the value of weights based on grads.

Inputs: grads is a vector of tensors of gradients to be used to update the weights in a particular layer of a neural net, and weights is a vector of tensors of the weights that we want to update. Each element grads[i] is the same size as its corresponding element weights[i]. When used to update a fully connected layer, the grads are dW and db from the output of FullyConnectedGradient.Compute and the weights are W and b. The learning rate is how fast the model learns; this value determines how much the weights are changed based on their gradient.

Weight imprinting

[adapter.h source]

namespace coral

struct ImprintingClass¶

Public Members

std::vector<float> weights¶: Sum of the weights generated by imprinting extractor.

bool trainable¶: Indicates whether the weights can be updated or not.

class ImprintingEngine¶

#include <imprinting_engine.h>

Class that implements Imprinted Weights transfer learning method proposed in paper https://arxiv.org/pdf/1712.07136.pdf.

Public Functions

ImprintingEngine(const ImprintingEngine&) = delete¶

ImprintingEngine &operator=(const ImprintingEngine&) = delete¶

absl::Span<const char> ExtractorModelBuffer() const¶: Returns extractor model buffer.

absl::Status SerializeModel(flatbuffers::FlatBufferBuilder *fbb)¶

Returns serialzied bytes of re-trained model.

The model will contain: [embedding_extractors] -> L2Norm -> Conv2d -> Mul -> Reshape -> Softmax

absl::Status Train(absl::Span<const float> embedding, int class_id)¶

Trains the model with the given embedding vector of a certain category.

Inputs: embedding vector

If training a new category, the class id should be exactly of the next class.

If training existing category with more images, the imprinting engine must be under keep_classes mode.

Call this function multiple times to train multiple different categories.

const std::vector<ImprintingClass> &GetClasses() const¶: Returns currently trained classes.

int embedding_dim() const¶: Returns number of embedding dimenssions for each class.

Public Static Functions

std::unique_ptr<ImprintingEngine> Create(std::unique_ptr<ImprintingModel> model, bool keep_classes)¶

Validates input model and initializes embedding_extractor.

The input model comes with L2Norm layer. It is supposed to be a full classification model.

Users can choose whether to keep previous classes by setting keep_classes to true or false.

Returns pair of EdgeTpuApiStatus and error message. Error message is only relevant when status is not kEdgeTpuApiOk.

Private Functions

ImprintingEngine(std::unique_ptr<ImprintingModel> model)¶

Private Members

std::unique_ptr<ImprintingModel> model_¶

std::vector<ImprintingClass> classes_¶

class ImprintingModel¶

Public Functions

absl::Span<const char> ExtractorModelBuffer() const¶: Returns extractor model buffer by truncating the prototype model.

std::vector<ImprintingClass> LoadExistingClasses() const¶: Returns classes from the prototype model.

absl::Status SerializeModel(const std::vector<ImprintingClass> &classes, flatbuffers::FlatBufferBuilder *fbb)¶: Serializes prototype model with the given array of classes.

int embedding_dim() const¶: Returns embedding vector dimension.

Public Static Functions

absl::Status Create(const tflite::Model &prototype, std::unique_ptr<ImprintingModel> *out_model)¶: Creates ImprintingModel instance from tflite::Model.

std::unique_ptr<ImprintingModel> CreateOrDie(const tflite::Model &prototype)¶: Creates ImprintingModel instance from tflite::Model or crashes on error.

Private Functions

ImprintingModel() = default¶

Private Members

std::unique_ptr<tflite::ModelT> model_t_¶

flatbuffers::FlatBufferBuilder extractor_fbb_¶

float fc_quant_scale_¶

int64_t fc_quant_zero_point_¶

int embedding_dim_¶

Is this content helpful?