ensmallen documentation

ensmallen is a library for function optimization. It is devoted to efficiently solving the problem

for many different types of $f(x)$. The documentation is split into four parts:

• Function type documentation: how to implement the different types of functions $f(x)$ that ensmallen can handle

• Optimizer documentation: how to use each of ensmallen’s optimizers to find $\operatorname{argmin} f(x)$

• Callback documentation: documentation for ensmallen callbacks, which can be used to specify custom behavior during optimization

• History/changelog: changes for each version of ensmallen

• Example program: simple example program demonstrating ensmallen usage

Everything in ensmallen is in the ens namespace so it is often useful to put a using directive in your code:

using namespace ens;

Citation

Please cite the following paper if you use ensmallen in your research and/or software. Citations are useful for the continued development and maintenance of the library.

• Ryan R. Curtin, Marcus Edel, Rahul Ganesh Prabhu, Suryoday Basak, Zhihao Lou, Conrad Sanderson.
  The ensmallen library for flexible numerical optimization.
  Journal of Machine Learning Research, Vol. 22, No. 166, 2021. (bibtex)

See also the following paper detailing the internal design of ensmallen:

• R.R. Curtin, M. Edel, R. Prabhu, S. Basak, Z. Lou, C. Sanderson. Flexible numerical optimization with ensmallen. arXiv preprint 2003.01403. (bibtex)

Contents

• Function type documentation
  • Arbitrary functions
  • Differentiable functions
    • Partially differentiable functions
  • Arbitrary separable functions
  • Differentiable separable functions
    • Sparse differentiable separable functions
  • Categorical functions
  • Multi-objective functions
  • Constrained functions
    • Semidefinite programs
  • Alternate matrix types
• Optimizer documentation
  • ActiveCMAES
  • AdaBelief
  • AdaBound
  • AdaDelta
  • Adagrad
  • AdaSqrt
  • Adam
  • AdaMax
  • AMSBound
  • AMSGrad
  • Augmented Lagrangian
  • Big Batch SGD
  • Coordinate Descent (CD)
  • CMAES
  • CNE
  • DE
  • DemonAdam
  • DemonSGD
  • Eve
  • Frank-Wolfe
  • FTML (Follow the Moving Leader)
  • Gradient Descent
  • Grid Search
  • Hogwild! (Parallel SGD)
  • IQN
  • Katyusha
  • L-BFGS
  • Lookahead
  • LRSDP (low-rank SDP solver)
  • Momentum SGD
  • Nadam
  • NadaMax
  • Nesterov Momentum SGD
  • MOEA/D-DE
  • NSGA2
  • OptimisticAdam
  • Padam
  • PSO
  • Primal-dual SDP Solver
  • Quasi-Hyperbolic Momentum Update SGD (QHSGD)
  • QHAdam
  • RMSProp
  • Simulated Annealing (SA)
  • Simultaneous Perturbation Stochastic Approximation (SPSA)
  • Stochastic Recursive Gradient Algorithm (SARAH/SARAH+)
  • Standard SGD
  • Stochastic Gradient Descent with Restarts (SGDR)
  • Snapshot Stochastic Gradient Descent with Restarts (SnapshotSGDR)
  • SMORMS3
  • Standard stochastic variance reduced gradient (SVRG)
  • SPALeRA Stochastic Gradient Descent (SPALeRASGD)
  • SWATS
  • WNGrad
  • Yogi
• Callback documentation
  • Built-in Callbacks
    • EarlyStopAtMinLoss
    • GradClipByNorm
    • GradClipByValue
    • PrintLoss
    • ProgressBar
    • Report
    • StoreBestCoordinates
  • Callback States
    • BeginOptimization
    • EndOptimization
    • Evaluate
    • EvaluateConstraint
    • Gradient
    • GradientConstraint
    • BeginEpoch
    • EndEpoch
    • GenerationalStepTaken
  • Custom Callbacks
    • Learning rate scheduling
    • Early stopping at minimum loss
• History/changelog
  • ensmallen 2.21.1: “Bent Antenna”
  • ensmallen 2.21.0: “Bent Antenna”
  • ensmallen 2.20.0: “Stripped Bolt Head”
  • ensmallen 2.19.1: “Eight Ball Deluxe”
  • ensmallen 2.19.0: “Eight Ball Deluxe”
  • ensmallen 2.18.2: “Fairmount Bagel”
  • ensmallen 2.18.1: “Fairmount Bagel”
  • ensmallen 2.18.0: “Fairmount Bagel”
  • ensmallen 2.17.0: “Pachis Din Me Pesa Double”
  • ensmallen 2.16.2: “Severely Dented Can Of Polyurethane”
  • ensmallen 2.16.1: “Severely Dented Can Of Polyurethane”
  • ensmallen 2.16.0: “Severely Dented Can Of Polyurethane”
  • ensmallen 2.15.1: “Why Can’t I Manage To Grow Any Plants?”
  • ensmallen 2.15.0: “Why Can’t I Manage To Grow Any Plants?”
  • ensmallen 2.14.2: “No Direction Home”
  • ensmallen 2.14.1: “No Direction Home”
  • ensmallen 2.14.0: “No Direction Home”
  • ensmallen 2.13.0: “Automatically Automated Automation”
  • ensmallen 2.12.1: “Stir Crazy”
  • ensmallen 2.12.0: “Stir Crazy”
  • ensmallen 2.11.5: “The Poster Session Is Full”
  • ensmallen 2.11.4: “The Poster Session Is Full”
  • ensmallen 2.11.3: “The Poster Session Is Full”
  • ensmallen 2.11.2: “The Poster Session Is Full”
  • ensmallen 2.11.1: “The Poster Session Is Full”
  • ensmallen 2.11.0: “The Poster Session Is Full”
  • ensmallen 2.10.5: “Fried Chicken”
  • ensmallen 2.10.4: “Fried Chicken”
  • ensmallen 2.10.3: “Fried Chicken”
  • ensmallen 2.10.2: “Fried Chicken”
  • ensmallen 2.10.1: “Fried Chicken”
  • ensmallen 2.10.0: “Fried Chicken”
  • ensmallen 1.16.2: “Loud Alarm Clock”
  • ensmallen 1.16.1: “Loud Alarm Clock”
  • ensmallen 1.16.0: “Loud Alarm Clock”
  • ensmallen 1.15.1: “Wrong Side Of The Road”
  • ensmallen 1.15.0: “Wrong Side Of The Road”
  • ensmallen 1.14.4: “Difficult Crimp”
  • ensmallen 1.14.3: “Difficult Crimp”
  • ensmallen 1.14.2: “Difficult Crimp”
  • ensmallen 1.14.1: “Difficult Crimp”
  • ensmallen 1.14.0: “Difficult Crimp”
  • ensmallen 1.13.2: “Coronavirus Invasion”
  • ensmallen 1.13.1: “Coronavirus Invasion”
  • ensmallen 1.13.0: “Coronavirus Invasion”
  • ensmallen 1.12.2: “New Year’s Party”
  • ensmallen 1.12.1: “New Year’s Party”
  • ensmallen 1.12.0: “New Year’s Party”
  • ensmallen 1.11.1: “Jet Lag”
  • ensmallen 1.11.0: “Jet Lag”
  • ensmallen 1.10.1: “Corporate Catabolism”
  • ensmallen 1.10.0: “Corporate Catabolism”

Function type documentation

Arbitrary functions

The least restrictive type of function that can be implemented in ensmallen is a function for which only the objective can be evaluated. For this, a class with the following API must be implemented:

class ArbitraryFunctionType
{
 public:
  // This should return f(x).
  double Evaluate(const arma::mat& x);
};

For this type of function, we assume that the gradient f'(x) is not computable. If it is, see differentiable functions.

The Evaluate() method is allowed to have additional cv-modifiers (static, const, etc.).

The following optimizers can be used to optimize an arbitrary function:

• Simulated Annealing
• CNE
• DE
• PSO
• SPSA

Each of these optimizers has an Optimize() function that is called as Optimize(f, x) where f is the function to be optimized (which implements Evaluate()) and x holds the initial point of the optimization. After Optimize() is called, x will hold the final result of the optimization (that is, the best x found that minimizes f(x)).

Example: squared function optimization

An example program that implements the objective function f(x) = 2 |x|^2 is shown below, using the simulated annealing optimizer.

#include <ensmallen.hpp>

class SquaredFunction
{
 public:
  // This returns f(x) = 2 |x|^2.
  double Evaluate(const arma::mat& x)
  {
    return 2 * std::pow(arma::norm(x), 2.0);
  }
};

int main()
{
  // The minimum is at x = [0 0 0].  Our initial point is chosen to be
  // [1.0, -1.0, 1.0].
  arma::mat x("1.0 -1.0 1.0");

  // Create simulated annealing optimizer with default options.
  // The ens::SA<> type can be replaced with any suitable ensmallen optimizer
  // that is able to handle arbitrary functions.
  ens::SA<> optimizer;
  SquaredFunction f; // Create function to be optimized.
  optimizer.Optimize(f, x);

  std::cout << "Minimum of squared function found with simulated annealing is "
      << x;
}

Differentiable functions

Probably the most common type of function that can be optimized with ensmallen is a differentiable function, where both f(x) and f’(x) can be calculated. To optimize a differentiable function with ensmallen, a class must be implemented that follows the API below:

class DifferentiableFunctionType
{
 public:
  // Given parameters x, return the value of f(x).
  double Evaluate(const arma::mat& x);

  // Given parameters x and a matrix g, store f'(x) in the provided matrix g.
  // g should have the same size (rows, columns) as x.
  void Gradient(const arma::mat& x, arma::mat& gradient);

  // OPTIONAL: this may be implemented in addition to---or instead
  // of---Evaluate() and Gradient().  If this is the only function implemented,
  // implementations of Evaluate() and Gradient() will be automatically
  // generated using template metaprogramming.  Often, implementing
  // EvaluateWithGradient() can result in more efficient optimizations.
  //
  // Given parameters x and a matrix g, return the value of f(x) and store
  // f'(x) in the provided matrix g.  g should have the same size (rows,
  // columns) as x.
  double EvaluateWithGradient(const arma::mat& x, arma::mat& g);
};

Note that you may implement either Evaluate() and Gradient() or EvaluateWithGradient(), but it is not mandatory to implement both. (Of course, supplying both is okay too.) It often results in faster code when EvaluateWithGradient() is implemented, especially for functions where f(x) and f’(x) compute some of the same intermediate quantities.

Each of the implemented methods is allowed to have additional cv-modifiers (static, const, etc.).

The following optimizers can be used with differentiable functions:

• L-BFGS (ens::L_BFGS)
• FrankWolfe (ens::FrankWolfe)
• GradientDescent (ens::GradientDescent)
• Any optimizer for arbitrary functions

Each of these optimizers has an Optimize() function that is called as Optimize(f, x) where f is the function to be optimized and x holds the initial point of the optimization. After Optimize() is called, x will hold the final result of the optimization (that is, the best x found that minimizes f(x)).

An example program is shown below. In this program, we optimize a linear regression model. In this setting, we have some matrix data of data points that we’ve observed, and some vector responses of the observed responses to this data. In our model, we assume that each response is the result of a weighted linear combination of the data:

where $x$ is a vector of parameters. This gives the objective function $f(x) = (\operatorname{responses} - x’\operatorname{data})^2$.

In the example program, we optimize this objective function and compare the runtime of an implementation that uses Evaluate() and Gradient(), and the runtime of an implementation that uses EvaluateWithGradient().

#include <ensmallen.hpp>

// Define a differentiable objective function by implementing both Evaluate()
// and Gradient() separately.
class LinearRegressionFunction
{
 public:
  // Construct the object with the given data matrix and responses.
  LinearRegressionFunction(const arma::mat& dataIn,
                           const arma::rowvec& responsesIn) :
      data(dataIn), responses(responsesIn) { }

  // Return the objective function for model parameters x.
  double Evaluate(const arma::mat& x)
  {
    return std::pow(arma::norm(responses - x.t() * data), 2.0);
  }

  // Compute the gradient for model parameters x.
  void Gradient(const arma::mat& x, arma::mat& g)
  {
    g = -2 * data * (responses - x.t() * data);
  }

 private:
  // The data.
  const arma::mat& data;
  // The responses to each data point.
  const arma::rowvec& responses;
};

// Define the same function, but only implement EvaluateWithGradient().
class LinearRegressionEWGFunction
{
 public:
  // Construct the object with the given data matrix and responses.
  LinearRegressionEWGFunction(const arma::mat& dataIn,
                              const arma::rowvec& responsesIn) :
      data(dataIn), responses(responsesIn) { }

  // Simultaneously compute both the objective function and gradient for model
  // parameters x.  Note that this is faster than implementing Evaluate() and
  // Gradient() individually because it caches the computation of
  // (responses - x.t() * data)!
  double EvaluateWithGradient(const arma::mat& x, arma::mat& g)
  {
    const arma::rowvec v = (responses - x.t() * data);
    g = -2 * data * v;
    return arma::accu(v % v); // equivalent to \| v \|^2
  }
};

int main()
{
  // We'll run a simple speed comparison between both objective functions.

  // First, generate some random data, with 10000 points and 10 dimensions.
  // This data has no pattern and as such will make a model that's not very
  // useful---but the purpose here is just demonstration. :)
  //
  // For a more "real world" situation, load a dataset from file using X.load()
  // and y.load() (but make sure the matrix is column-major, so that each
  // observation/data point corresponds to a *column*, *not* a row.
  arma::mat data(10, 10000, arma::fill::randn);
  arma::rowvec responses(10000, arma::fill::randn);

  // Create a starting point for our optimization randomly.  The model has 10
  // parameters, so the shape is 10x1.
  arma::mat startingPoint(10, 1, arma::fill::randn);

  // We'll use Armadillo's wall_clock class to do a timing comparison.
  arma::wall_clock clock;

  // Construct the first objective function.
  LinearRegressionFunction lrf1(data, responses);
  arma::mat lrf1Params(startingPoint);

  // Create the L_BFGS optimizer with default parameters.
  // The ens::L_BFGS type can be replaced with any ensmallen optimizer that can
  // handle differentiable functions.
  ens::L_BFGS lbfgs;

  // Time how long L-BFGS takes for the first Evaluate() and Gradient()
  // objective function.
  clock.tic();
  lbfgs.Optimize(lrf1, lrf1Params);
  const double time1 = clock.toc();

  std::cout << "LinearRegressionFunction with Evaluate() and Gradient() took "
    << time1 << " seconds to converge to the model: " << std::endl;
  std::cout << lrf1Params.t();

  // Create the second objective function, which uses EvaluateWithGradient().
  LinearRegressionEWGFunction lrf2(data, responses);
  arma::mat lrf2Params(startingPoint);

  // Time how long L-BFGS takes for the EvaluateWithGradient() objective
  // function.
  clock.tic();
  lbfgs.Optimize(lrf2, lrf2Params);
  const double time2 = clock.toc();

  std::cout << "LinearRegressionEWGFunction with EvaluateWithGradient() took "
    << time2 << " seconds to converge to the model: " << std::endl;
  std::cout << lrf2Params.t();

  // When this runs, the output parameters will be exactly on the same, but the
  // LinearRegressionEWGFunction will run more quickly!
}

Partially differentiable functions

Some differentiable functions have the additional property that the gradient f'(x) can be decomposed along a different axis j such that the gradient is sparse. This makes the most sense in machine learning applications where the function f(x) is being optimized for some dataset data, which has d dimensions. A partially differentiable separable function is partially differentiable with respect to each dimension j of data. This property is useful for coordinate descent type algorithms.

To use ensmallen optimizers to minimize these types of functions, only two functions needs to be added to the differentiable function type:

// Compute the partial gradient f'_j(x) with respect to data coordinate j and
// store it in the sparse matrix g.
void Gradient(const arma::mat& x, const size_t j, arma::sp_mat& g);

// Get the number of features that f(x) can be partially differentiated with.
size_t NumFeatures();

Note: many partially differentiable function optimizers do not require a regular implementation of the Gradient(), so that function may be omitted.

If these functions are implemented, the following partially differentiable function optimizers can be used:

• Coordinate Descent

Arbitrary separable functions

Often, an objective function f(x) may be represented as the sum of many functions:

f(x) = f_0(x) + f_1(x) + ... + f_N(x).

In this function type, we assume the gradient f'(x) is not computable. If it is, see differentiable separable functions.

For machine learning tasks, the objective function may be, e.g., the sum of a function taken across many data points. Implementing an arbitrary separable function type in ensmallen is similar to implementing an arbitrary objective function, but with a few extra utility methods:

class ArbitrarySeparableFunctionType
{
 public:
  // Given parameters x, return the sum of the individual functions
  // f_i(x) + ... + f_{i + batchSize - 1}(x).  i will always be greater than 0,
  // and i + batchSize will be less than or equal to the value of NumFunctions().
  double Evaluate(const arma::mat& x, const size_t i, const size_t batchSize);

  // Shuffle the ordering of the functions f_i(x).
  // (For machine learning problems, this would be equivalent to shuffling the
  // data points, e.g., before an epoch of learning.)
  void Shuffle();

  // Get the number of functions f_i(x).
  // (For machine learning problems, this is often just the number of points in
  // the dataset.)
  size_t NumFunctions();
};

Each of the implemented methods is allowed to have additional cv-modifiers (static, const, etc.).

The following optimizers can be used with arbitrary separable functions:

• CMAES

Each of these optimizers has an Optimize() function that is called as Optimize(f, x) where f is the function to be optimized and x holds the initial point of the optimization. After Optimize() is called, x will hold the final result of the optimization (that is, the best x found that minimizes f(x)).

Note: using an arbitrary non-separable function optimizer will call Evaluate(x, 0, NumFunctions() - 1); if this is a very computationally intensive operation for your objective function, it may be best to avoid using a non-separable arbitrary function optimizer.

Note: if possible, it’s often better to try and use a gradient-based approach. See differentiable separable functions for separable f(x) where the gradient f’(x) can be computed.

The example program below demonstrates the implementation and use of an arbitrary separable function. The function used is the linear regression objective function, described in differentiable functions. Given some dataset data and responses responses, the linear regression objective function is separable as

where $\operatorname{data}(i)$ represents the data point indexed by $i$ and $\operatorname{responses}(i)$ represents the observed response indexed by $i$.

#include <ensmallen.hpp>

// This class implements the linear regression objective function as an
// arbitrary separable function type.
class LinearRegressionFunction
{
 public:
  // Create the linear regression function with the given data and the given
  // responses.
  LinearRegressionFunction(const arma::mat& dataIn,
                           const arma::rowvec& responsesIn) :
      data(data), responses(responses) { }

  // Given parameters x, compute the sum of the separable objective
  // functions starting with f_i(x) and ending with
  // f_{i + batchSize - 1}(x).
  double Evaluate(const arma::mat& x, const size_t i, const size_t batchSize)
  {
    // A more complex implementation could avoid the for loop and use
    // submatrices, but it is easier to understand when implemented this way.
    double objective = 0.0;
    for (size_t j = i; j < i + batchSize; ++j)
    {
      objective += std::pow(responses[j] - x.t() * data.col(j), 2.0);
    }
  }

  // Shuffle the ordering of the functions f_i(x).  We do this by simply
  // shuffling the data and responses.
  void Shuffle()
  {
    // Generate a random ordering of data points.
    arma::uvec ordering = arma::shuffle(
        arma::linspace<arma::uvec>(0, data.n_cols - 1, data.n_cols));

    // This reorders the data and responses with our randomly-generated
    // ordering above.
    data = data.cols(ordering);
    responses = responses.cols(ordering);
  }

  // Return the number of functions f_i(x).  In our case this is simply the
  // number of data points.
  size_t NumFunctions() { return data.n_cols; }
};

int main()
{
  // First, generate some random data, with 10000 points and 10 dimensions.
  // This data has no pattern and as such will make a model that's not very
  // useful---but the purpose here is just demonstration. :)
  //
  // For a more "real world" situation, load a dataset from file using X.load()
  // and y.load() (but make sure the matrix is column-major, so that each
  // observation/data point corresponds to a *column*, *not* a row.
  arma::mat data(10, 10000, arma::fill::randn);
  arma::rowvec responses(10000, arma::fill::randn);

  // Create a starting point for our optimization randomly.  The model has 10
  // parameters, so the shape is 10x1.
  arma::mat params(10, 1, arma::fill::randn);

  // Use the CMAES optimizer with default parameters to minimize the
  // LinearRegressionFunction.
  // The ens::CMAES type can be replaced with any suitable ensmallen optimizer
  // that can handle arbitrary separable functions.
  ens::CMAES cmaes;
  LinearRegressionFunction lrf(data, responses);
  cmaes.Optimize(lrf, params);

  std::cout << "The optimized linear regression model found by CMAES has the "
      << "parameters " << params.t();
}

Differentiable separable functions

Likely the most important type of function to be optimized in machine learning and some other fields is the differentiable separable function. A differentiable separable function f(x) can be represented as the sum of many functions:

f(x) = f_0(x) + f_1(x) + ... + f_N(x).

And it also has a computable separable gradient f'(x):

f'(x) = f'_0(x) + f'_1(x) + ... + f'_N(x).

For machine learning tasks, the objective function may be, e.g., the sum of a function taken across many data points. Implementing a differentiable separable function type in ensmallen is similar to implementing an ordinary differentiable function, but with a few extra utility methods:

class ArbitrarySeparableFunctionType
{
 public:
  // Given parameters x, return the sum of the individual functions
  // f_i(x) + ... + f_{i + batchSize - 1}(x).  i will always be greater than 0,
  // and i + batchSize will be less than or equal to the value of NumFunctions().
  double Evaluate(const arma::mat& x, const size_t i, const size_t batchSize);

  // Given parameters x and a matrix g, store the sum of the gradient of
  // individual functions f'_i(x) + ... + f'_{i + batchSize - 1}(x) into g.  i
  // will always be greater than 0, and i + batchSize will be less than or
  // equal to the value of NumFunctions().
  void Gradient(const arma::mat& x,
                const size_t i,
                arma::mat& g,
                const size_t batchSize);

  // Shuffle the ordering of the functions f_i(x).
  // (For machine learning problems, this would be equivalent to shuffling the
  // data points, e.g., before an epoch of learning.)
  void Shuffle();

  // Get the number of functions f_i(x).
  // (For machine learning problems, this is often just the number of points in
  // the dataset.)
  size_t NumFunctions();

  // OPTIONAL: this may be implemented in addition to---or instead
  // of---Evaluate() and Gradient().  If this is the only function implemented,
  // implementations of Evaluate() and Gradient() will be automatically
  // generated using template metaprogramming.  Often, implementing
  // EvaluateWithGradient() can result in more efficient optimizations.
  //
  // Given parameters x and a matrix g, return the sum of the individual
  // functions f_i(x) + ... + f_{i + batchSize - 1}(x), and store the sum of
  // the gradient of individual functions f'_i(x) + ... +
  // f'_{i + batchSize - 1}(x) into the provided matrix g.  g should have the
  // same size (rows, columns) as x.  i will always be greater than 0, and i +
  // batchSize will be less than or equal to the value of NumFunctions().
  double EvaluateWithGradient(const arma::mat& x,
                              const size_t i,
                              arma::mat& g,
                              const size_t batchSize);
};

Note that you may implement either Evaluate() and Gradient() or EvaluateWithGradient(), but it is not mandatory to implement both. (Of course, supplying both is okay too.) It often results in faster code when EvaluateWithGradient() is implemented, especially for functions where f(x) and f’(x) compute some of the same intermediate quantities.

Each of the implemented methods is allowed to have additional cv-modifiers (static, const, etc.).

The following optimizers can be used with differentiable separable functions:

• AdaBelief
• AdaBound
• AdaDelta
• AdaGrad
• AdaSqrt
• Adam
• AdaMax
• AMSBound
• AMSGrad
• Big Batch SGD
• Eve
• FTML
• IQN
• Katyusha
• Lookahead
• Momentum SGD
• Nadam
• NadaMax
• NesterovMomentumSGD
• OptimisticAdam
• QHAdam
• QHSGD
• RMSProp
• SARAH/SARAH+
• SGD
• Stochastic Gradient Descent with Restarts (SGDR)
• Snapshot SGDR
• SMORMS3
• SPALeRA
• SWATS
• SVRG
• WNGrad

The example program below demonstrates the implementation and use of an arbitrary separable function. The function used is the linear regression objective function, described in differentiable functions. Given some dataset data and responses responses, the linear regression objective function is separable as

f_i(x) = (responses(i) - x' * data(i))^2

where data(i) represents the data point indexed by i and responses(i) represents the observed response indexed by i. This example implementation only implements EvaluateWithGradient() in order to avoid redundant calculations.

#include <ensmallen.hpp>

// This class implements the linear regression objective function as an
// arbitrary separable function type.
class LinearRegressionFunction
{
 public:
  // Create the linear regression function with the given data and the given
  // responses.
  LinearRegressionFunction(const arma::mat& dataIn,
                           const arma::rowvec& responsesIn) :
      data(data), responses(responses) { }

  // Given parameters x, compute the sum of the separable objective
  // functions starting with f_i(x) and ending with
  // f_{i + batchSize - 1}(x), and also compute the gradient of those functions
  // and store them in g.
  double EvaluateWithGradient(const arma::mat& x,
                              const size_t i,
                              arma::mat& g,
                              const size_t batchSize)
  {
    // This slightly complex implementation uses Armadillo submatrices to
    // compute the objective functions and gradients simultaneously for
    // multiple points.
    //
    // The shared computation between the objective and gradient is the term
    // (response - x * data) so we compute that first, only for points in the
    // batch.
    const arma::rowvec v = (responses.cols(i, i + batchSize - 1) - x.t() *
        data.cols(i, i + batchSize - 1));
    g = -2 * data.cols(i, i + batchSize - 1) * v;
    return arma::accu(v % v); // equivalent to |v|^2
  }

  // Shuffle the ordering of the functions f_i(x).  We do this by simply
  // shuffling the data and responses.
  void Shuffle()
  {
    // Generate a random ordering of data points.
    arma::uvec ordering = arma::shuffle(
        arma::linspace<arma::uvec>(0, data.n_cols - 1, data.n_cols));

    // This reorders the data and responses with our randomly-generated
    // ordering above.
    data = data.cols(ordering);
    responses = responses.cols(ordering);
  }

  // Return the number of functions f_i(x).  In our case this is simply the
  // number of data points.
  size_t NumFunctions() { return data.n_cols; }
};

int main()
{
  // First, generate some random data, with 10000 points and 10 dimensions.
  // This data has no pattern and as such will make a model that's not very
  // useful---but the purpose here is just demonstration. :)
  //
  // For a more "real world" situation, load a dataset from file using X.load()
  // and y.load() (but make sure the matrix is column-major, so that each
  // observation/data point corresponds to a *column*, *not* a row.
  arma::mat data(10, 10000, arma::fill::randn);
  arma::rowvec responses(10000, arma::fill::randn);

  // Create a starting point for our optimization randomly.  The model has 10
  // parameters, so the shape is 10x1.
  arma::mat params(10, 1, arma::fill::randn);

  // Use RMSprop to find the best parameters for the linear regression model.
  // The type 'ens::RMSprop' can be changed for any ensmallen optimizer able to
  // handle differentiable separable functions.
  ens::RMSProp rmsprop;
  LinearRegressionFunction lrf(data, responses);
  rmsprop.Optimize(lrf, params);

  std::cout << "The optimized linear regression model found by RMSprop has the"
      << " parameters " << params.t();
}

Sparse differentiable separable functions

Some differentiable separable functions have the additional property that the gradient f'_i(x) is sparse. When this is true, one additional method can be implemented as part of the class to be optimized:

// Add this definition to use sparse differentiable separable function
// optimizers.  Given x, store the sum of the sparse gradient f'_i(x) + ... +
// f'_{i + batchSize - 1}(x) into the provided matrix g.
void Gradient(const arma::mat& x,
              const size_t i,
              arma::sp_mat& g,
              const size_t batchSize);

It’s also possible to instead use templates to provide only one Gradient() function for both sparse and non-sparse optimizers:

// This provides Gradient() for both sparse and non-sparse optimizers.
template<typename GradType>
void Gradient(const arma::mat& x,
              const size_t i,
              GradType& g,
              const size_t batchSize);

If either of these methods are available, then any ensmallen optimizer that optimizes sparse separable differentiable functions may be used. This includes:

• Hogwild! (Parallel SGD)

Categorical functions

A categorical function is a function f(x) where some of the values of x are categorical variables (i.e. they take integer values [0, c - 1] and each value 0, 1, …, c - 1 is unrelated). In this situation, the function is not differentiable, and so only the objective f(x) can be implemented. Therefore the class requirements for a categorical function are exactly the same as for an ArbitraryFunctionType—but for any categorical dimension x_i in x, the value will be in the range [0, c_i - 1] where c_i is the number of categories in dimension x_i.

class CategoricalFunction
{
 public:
  // Return the objective function for the given parameters x.
  double Evaluate(const arma::mat& x);
};

However, when an optimizer’s Optimize() method is called, two additional parameters must be specified, in addition to the function to optimize and the matrix holding the parameters:

• const std::vector<bool>& categoricalDimensions: a vector of length equal to the number of elements in x (the number of dimensions). If an element is true, then the dimension is categorical.

• const arma::Row<size_t>& numCategories: a vector of length equal to the number of elements in x (the number of dimensions). If a dimension is categorical, then the corresponding value in numCategories should hold the number of categories in that dimension.

The following optimizers can be used in this way to optimize a categorical function:

• Grid Search (all parameters must be categorical)

An example program showing usage of categorical optimization is shown below.

#include <ensmallen.hpp>

// An implementation of a simple categorical function.  The parameters can be
// understood as x = [c1 c2 c3].  When c1 = 0, c2 = 2, and c3 = 1, the value of
// f(x) is 0.  In any other case, the value of f(x) is 10.  Therefore, the
// optimum is found at [0, 2, 1].
class SimpleCategoricalFunction
{
 public:
  // Return the objective function f(x) as described above.
  double Evaluate(const arma::mat& x)
  {
    if (size_t(x[0]) == 0 &&
        size_t(x[1]) == 2 &&
        size_t(x[2]) == 1)
      return 0.0;
    else
      return 10.0;
  }
};

int main()
{
  // Create and optimize the categorical function with the GridSearch
  // optimizer.  We must also create a std::vector<bool> that holds the types
  // of each dimension, and an arma::Row<size_t> that holds the number of
  // categories in each dimension.
  SimpleCategoricalFunction c;

  // We have three categorical dimensions only.
  std::vector<bool> categoricalDimensions;
  categoricalDimensions.push_back(true);
  categoricalDimensions.push_back(true);
  categoricalDimensions.push_back(true);

  // The first category can take 5 values; the second can take 3; the third can
  // take 12.
  arma::Row<size_t> numCategories("5 3 12");

  // The initial point for our optimization will be to set all categories to 0.
  arma::mat params("0 0 0");

  // Now create the GridSearch optimizer with default parameters, and run the
  // optimization.
  // The ens::GridSearch type can be replaced with any ensmallen optimizer that
  // is able to handle categorical functions.
  ens::GridSearch gs;
  gs.Optimize(c, params, categoricalDimensions, numCategories);

  std::cout << "The ens::GridSearch optimizer found the optimal parameters to "
      << "be " << params;
}

Multi-objective functions

A multi-objective optimizer does not return just one set of coordinates at the minimum of all objective functions, but instead finds a front or frontier of possible coordinates that are Pareto-optimal (that is, no individual objective function’s value can be reduced without increasing at least one other objective function).

In order to optimize a multi-objective function with ensmallen, a std::tuple<> containing multiple ArbitraryFunctionTypes (see here) should be passed to a multi-objective optimizer’s Optimize() function.

An example below simultaneously optimizes the generalized Rosenbrock function in 6 dimensions and the Wood function using NSGA2.

GeneralizedRosenbrockFunction rf(6);
WoodFunction wf;
std::tuple<GeneralizedRosenbrockFunction, WoodFunction> objectives(rf, wf);

// Create an initial point (a random point in 6 dimensions).
arma::mat coordinates(6, 1, arma::fill::randu);

// `coordinates` will be set to the coordinates on the best front that minimize the
// sum of objective functions, and `bestFrontSum` will be the sum of all objectives
// at that coordinate set.
NSGA2 nsga;
double bestFrontSum = nsga.Optimize(objectives, coordinates);

// Set `bestFront` to contain all of the coordinates on the best front.
arma::cube bestFront = optimizer.ParetoFront();
}

Note: all multi-objective function optimizers have both the function Optimize() to find the best front, and also the function ParetoFront() to return all sets of solutions that are on the front.

The following optimizers can be used with multi-objective functions:

• NSGA2
• MOEA/D-DE

Constrained functions

A constrained function is an objective function f(x) that is also subject to some constraints on x. (For instance, perhaps a constraint could be that x is a positive semidefinite matrix.) ensmallen is able to handle differentiable objective functions of this type—so, f'(x) must also be computable. Given some set of constraints c_0(x), …, c_M(x), we can re-express our constrained objective function as

f_C(x) = f(x) + c_0(x) + ... + c_M(x)

where the (soft) constraint c_i(x) is a positive value if it is not satisfied, and 0 if it is satisfied. The soft constraint c_i(x) should take some value representing how far from a feasible solution x is. It should be differentiable, since ensmallen’s constrained optimizers will use the gradient of the constraint to find a feasible solution.

In order to optimize a constrained function with ensmallen, a class implementing the API below is required.

class ConstrainedFunctionType
{
 public:
  // Return the objective function f(x) for the given x.
  double Evaluate(const arma::mat& x);

  // Compute the gradient of f(x) for the given x and store the result in g.
  void Gradient(const arma::mat& x, arma::mat& g);

  // Get the number of constraints on the objective function.
  size_t NumConstraints();

  // Evaluate constraint i at the parameters x.  If the constraint is
  // unsatisfied, a value greater than 0 should be returned.  If the constraint
  // is satisfied, 0 should be returned.  The optimizer will add this value to
  // its overall objective that it is trying to minimize.
  double EvaluateConstraint(const size_t i, const arma::mat& x);

  // Evaluate the gradient of constraint i at the parameters x, storing the
  // result in the given matrix g.  If the constraint is not satisfied, the
  // gradient should be set in such a way that the gradient points in the
  // direction where the constraint would be satisfied.
  void GradientConstraint(const size_t i, const arma::mat& x, arma::mat& g);
};

A constrained function can be optimized with the following optimizers:

• Augmented Lagrangian

Semidefinite programs

A special class of constrained function is a semidefinite program. ensmallen has support for creating and optimizing semidefinite programs via the ens::SDP<> class. For this, the SDP must be expressed in the primal form:

min_x dot(C, x) such that

dot(A_i, x) = b_i, for i = 1, ..., M;
x >= 0

In this case, A_i and C are square matrices (sparse or dense), and b_i is a scalar.

Once the problem is expressed in this form, a class of type ens::SDP<> can be created. SDP<arma::mat> indicates an SDP with a dense C, and SDP<arma::sp_mat> indicates an SDP with a sparse C. The class has the following useful methods:

• SDP(cMatrixSize, numDenseConstraints, numSparseConstraints): create a new SDP
• size_t NumSparseConstraints(): get number of sparse constraint matrices A_i
• size_t NumDenseConstraints(): get number of dense constraint matrices A_i
• std::vector<arma::mat>& DenseA(): get vector of dense A_i matrices
• std::vector<arma::sp_mat>& SparseA(): get vector of sparse A_i matrices
• arma::vec& DenseB(): get vector of b_i values for dense A_i constraints
• arma::vec& SparseB(): get vector of b_i values for sparse A_i constraints

Once these methods are used to set each A_i matrix and corresponding b_i value, and C objective matrix, the SDP object can be used with any ensmallen SDP solver. The list of SDP solvers is below:

• Primal-dual SDP solver
• Low-rank accelerated SDP solver (LRSDP)

Example code showing how to solve an SDP is given below.

int main()
{
  // We will build a toy semidefinite program and then use the PrimalDualSolver to find a solution

  // The semi-definite constraint looks like:
  //
  // [ 1  x_12  x_13  0  0  0  0 ]
  // [     1    x_23  0  0  0  0 ]
  // [            1   0  0  0  0 ]
  // [               s1  0  0  0 ]  >= 0
  // [                  s2  0  0 ]
  // [                     s3  0 ]
  // [                        s4 ]

  // x_11 == 0
  arma::sp_mat A0(7, 7); A0.zeros();
  A0(0, 0) = 1.;

  // x_22 == 0
  arma::sp_mat A1(7, 7); A1.zeros();
  A1(1, 1) = 1.;

  // x_33 == 0
  arma::sp_mat A2(7, 7); A2.zeros();
  A2(2, 2) = 1.;

  // x_12 <= -0.1  <==>  x_12 + s1 == -0.1, s1 >= 0
  arma::sp_mat A3(7, 7); A3.zeros();
  A3(1, 0) = A3(0, 1) = 1.; A3(3, 3) = 2.;

  // -0.2 <= x_12  <==>  x_12 - s2 == -0.2, s2 >= 0
  arma::sp_mat A4(7, 7); A4.zeros();
  A4(1, 0) = A4(0, 1) = 1.; A4(4, 4) = -2.;

  // x_23 <= 0.5  <==>  x_23 + s3 == 0.5, s3 >= 0
  arma::sp_mat A5(7, 7); A5.zeros();
  A5(2, 1) = A5(1, 2) = 1.; A5(5, 5) = 2.;

  // 0.4 <= x_23  <==>  x_23 - s4 == 0.4, s4 >= 0
  arma::sp_mat A6(7, 7); A6.zeros();
  A6(2, 1) = A6(1, 2) = 1.; A6(6, 6) = -2.;

  std::vector<arma::sp_mat> ais({A0, A1, A2, A3, A4, A5, A6});

  SDP<arma::sp_mat> sdp(7, 7 + 4 + 4 + 4 + 3 + 2 + 1, 0);

  for (size_t j = 0; j < 3; j++)
  {
    // x_j4 == x_j5 == x_j6 == x_j7 == 0
    for (size_t i = 0; i < 4; i++)
    {
      arma::sp_mat A(7, 7); A.zeros();
      A(i + 3, j) = A(j, i + 3) = 1;
      ais.emplace_back(A);
    }
  }

  // x_45 == x_46 == x_47 == 0
  for (size_t i = 0; i < 3; i++)
  {
    arma::sp_mat A(7, 7); A.zeros();
    A(i + 4, 3) = A(3, i + 4) = 1;
    ais.emplace_back(A);
  }

  // x_56 == x_57 == 0
  for (size_t i = 0; i < 2; i++)
  {
    arma::sp_mat A(7, 7); A.zeros();
    A(i + 5, 4) = A(4, i + 5) = 1;
    ais.emplace_back(A);
  }

  // x_67 == 0
  arma::sp_mat A(7, 7); A.zeros();
  A(6, 5) = A(5, 6) = 1;
  ais.emplace_back(A);

  std::swap(sdp.SparseA(), ais);

  sdp.SparseB().zeros();
  sdp.SparseB()[0] = sdp.SparseB()[1] = sdp.SparseB()[2] = 1.;
  sdp.SparseB()[3] = -0.2; sdp.SparseB()[4] = -0.4;
  sdp.SparseB()[5] = 1.; sdp.SparseB()[6] = 0.8;

  sdp.C().zeros();
  sdp.C()(0, 2) = sdp.C()(2, 0) = 1.;

  // That took a long time but we finally set up the problem right!  Now we can
  // use the PrimalDualSolver to solve it.
  // ens::PrimalDualSolver could be replaced with ens::LRSDP or other ensmallen
  // SDP solvers.
  PrimalDualSolver solver;
  arma::mat X, Z;
  arma::vec ysparse, ydense;
  // ysparse, ydense, and Z hold the primal and dual variables found during the
  // optimization.
  const double obj = solver.Optimize(sdp, X, ysparse, ydense, Z);

  std::cout << "SDP optimized with objective " << obj << "." << std::endl;
}

Alternate matrix types

All of the examples above (and throughout the rest of the documentation) generally assume that the matrix being optimized has type arma::mat. But ensmallen’s optimizers are capable of optimizing more types than just dense Armadillo matrices. In fact, the full signature of each optimizer’s Optimize() method is this:

template<typename FunctionType, typename MatType>
typename MatType::elem_type Optimize(FunctionType& function,
                                     MatType& coordinates);

The return type, typename MatType::elem_type, is just the numeric type held by the given matrix type. So, for arma::mat, the return type is just double. In addition, optimizers for differentiable functions have a third template parameter, GradType, which specifies the type of the gradient. GradType can be manually specified in the situation where, e.g., a sparse gradient is desired.

It is easy to write a function to optimize, e.g., an arma::fmat. Here is an example, adapted from the SquaredFunction example from the arbitrary function documentation.

#include <ensmallen.hpp>

class SquaredFunction
{
 public:
  // This returns f(x) = 2 |x|^2.
  float Evaluate(const arma::fmat& x)
  {
    return 2 * std::pow(arma::norm(x), 2.0);
  }

  void Gradient(const arma::fmat& x, arma::fmat& gradient)
  {
    gradient = 4 * x;
  }
};

int main()
{
  // The minimum is at x = [0 0 0].  Our initial point is chosen to be
  // [1.0, -1.0, 1.0].
  arma::fmat x("1.0 -1.0 1.0");

  // Create simulated annealing optimizer with default options.
  // The ens::SA<> type can be replaced with any suitable ensmallen optimizer
  // that is able to handle arbitrary functions.
  ens::L_BFGS optimizer;
  SquaredFunction f; // Create function to be optimized.
  optimizer.Optimize(f, x); // The optimizer will infer arma::fmat!

  std::cout << "Minimum of squared function found with simulated annealing is "
      << x;
}

Note that we have simply changed the SquaredFunction to accept arma::fmat instead of arma::mat as parameters to Evaluate(), and the return type has accordingly been changed to float from double. It would even be possible to optimize functions with sparse coordinates by having Evaluate() take a sparse matrix (i.e. arma::sp_mat).

If it were desired to represent the gradient as a sparse type, the Gradient() function would need to be modified to take a sparse matrix (i.e. arma::sp_mat or similar), and then you could call optimizer.Optimize<SquaredFunction, arma::mat, arma::sp_mat>(f, x); to perform the optimization while using sparse matrix types to represent the gradient. Using sparse MatType or GradType should only be done when it is known that the objective matrix and/or gradients will be sparse; otherwise the code may run very slow!

ensmallen will automatically infer MatType from the call to Optimize(), and check that the given FunctionType has all of the necessary functions for the given MatType, throwing a static_assert error if not. If you would like to disable these checks, define the macro ENS_DISABLE_TYPE_CHECKS before including ensmallen:

#define ENS_DISABLE_TYPE_CHECKS
#include <ensmallen.hpp>

This can be useful for situations where you know that the checks should be ignored. However, be aware that the code may fail to compile and give more confusing and difficult error messages!

Optimizer documentation

ActiveCMAES

An optimizer for separable functions.

Active CMA-ES is a variant of the stochastic search algorithm CMA-ES - Covariance Matrix Adaptation Evolution Strategy. Active CMA-ES actively reduces the uncertainty in unfavourable directions by exploiting the information about bad mutations in the covariance matrix update step. This isn’t for the purpose of accelerating progress, but instead for speeding up the adaptation of the covariance matrix (which, in turn, will lead to faster progress).

Constructors

• ActiveCMAES<SelectionPolicyType, TransformationPolicyType>()
• ActiveCMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy)
• ActiveCMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy, batchSize)
• ActiveCMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy)
• ActiveCMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize)

The SelectionPolicyType template parameter refers to the strategy used to compute the (approximate) objective function. The FullSelection and RandomSelection classes are available for use; custom behavior can be achieved by implementing a class with the same method signatures. The TransformationPolicyType template parameter refers to transformation strategy used to map decision variables to the desired domain during fitness evaluation and optimization termination. The EmptyTransformation and BoundaryBoxConstraint classes are available for use; custom behavior can be achieved by implementing a class with the same method signatures.

For convenience the following types can be used:

• ActiveCMAES<> (equivalent to ActiveCMAES<FullSelection, EmptyTransformation<>>): uses all separable functions to compute objective
• ApproxActiveCMAES<> (equivalent to ActiveCMAES<RandomSelection, EmptyTransformation<>>): uses a small amount of separable functions to compute approximate objective

Attributes

Attributes of the optimizer may also be changed via the member methods Lambda(), TransformationPolicy(), BatchSize(), MaxIterations(), Tolerance(), and SelectionPolicy().

The selectionPolicy attribute allows an instantiated SelectionPolicyType to be given. The FullSelection policy has no need to be instantiated and thus the option is not relevant when the ActiveCMAES<> optimizer type is being used; the RandomSelection policy has the constructor RandomSelection(fraction) where fraction specifies the percentage of separable functions to use to estimate the objective function. The transformationPolicy attribute allows an instantiated TransformationPolicyType to be given. The EmptyTransformation<MatType> has no need to be instantiated. BoundaryBoxConstraint<MatType> policy has the constructor BoundaryBoxConstraint(lowerBound, upperBound) where lowerBound and lowerBound are the lower bound and upper bound of the coordinates respectively.

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

// ActiveCMAES with the FullSelection and BoundaryBoxConstraint policies.
BoundaryBoxConstraint b(-1, 1);
ActiveCMAES optimizer(0, b, 32, 200, 1e-4);
optimizer.Optimize(f, coordinates);

// ActiveCMAES with the RandomSelection and BoundaryBoxConstraint policies.
ApproxActiveCMAES<BoundaryBoxConstraint<>> cmaes(0, b, 32, 200, 1e-4);
approxOptimizer.Optimize(f, coordinates);

See also:

• CMAES
• Improving Evolution Strategies through Active Covariance Matrix Adaptation
• Evolution strategy in Wikipedia

AdaBelief

An optimizer for differentiable separable functions.

AdaBelief uses a different denominator from Adam, and is orthogonal to other techniques such as recification, decoupled weight decay. The intuition for AdaBelief is to adapt the stepsize according to the “belief” in the current gradient direction.

Constructors

• AdaBelief()
• AdaBelief(stepSize, batchSize)
• AdaBelief(stepSize, batchSize, beta1, beta2, epsilon, maxIterations, tolerance, shuffle)
• AdaBelief(stepSize, batchSize, beta1, beta2, epsilon, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Beta1(), Beta2(), Epsilon(), MaxIterations(), Tolerance(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

AdaBelief optimizer(0.001, 32, 0.9, 0.999, 1e-12, 100000, 1e-5, true);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
• Differentiable separable functions

AdaBound

An optimizer for differentiable separable functions.

AdaBound is a variant of Adam which employs dynamic bounds on learning rates.

Constructors

• AdaBound()
• AdaBound(stepSize, batchSize)
• AdaBound(stepSize, batchSize, finalLr, gamma, beta1, beta2, epsilon, maxIterations, tolerance, shuffle)
• AdaBound(stepSize, batchSize, finalLr, gamma, beta1, beta2, epsilon, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Note that the AdaBound class is based on the AdaBoundType<UpdateRule> class with UpdateRule = AdaBoundUpdate.

Attributes

The attributes of the optimizer may also be modified via the member methods FinalLr(), Gamma(), StepSize(), BatchSize(), Beta1(), Beta2(), Eps(), MaxIterations(), Tolerance(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples

SphereFunction f(2);
arma::mat coordinates = f.GetInitialPoint();

AdaBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000, 1e-3);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Adam: A Method for Stochastic Optimization
• Differentiable separable functions

AdaDelta

An optimizer for differentiable separable functions.

AdaDelta is an extension of AdaGrad that adapts learning rates based on a moving window of gradient updates, instead of accumulating all past gradients. Instead of accumulating all past squared gradients, the sum of gradients is recursively defined as a decaying average of all past squared gradients.

Constructors

• AdaDelta()
• AdaDelta(stepSize)
• AdaDelta(stepSize, batchSize)
• AdaDelta(stepSize, batchSize, rho, epsilon, maxIterations, tolerance, shuffle)
• AdaDelta(stepSize, batchSize, rho, epsilon, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Attributes

Attributes of the optimizer may also be changed via the member methods StepSize(), BatchSize(), Rho(), Epsilon(), MaxIterations(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples:

AdaDelta optimizer(1.0, 1, 0.99, 1e-8, 1000, 1e-9, true);

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();
optimizer.Optimize(f, coordinates);

See also:

• Adadelta - an adaptive learning rate method
• AdaGrad
• Differentiable separable functions

Adagrad

An optimizer for differentiable separable functions.

AdaGrad is an optimizer with parameter-specific learning rates, which are adapted relative to how frequently a parameter gets updated during training. Larger updates for more sparse parameters and smaller updates for less sparse parameters.

Constructors

• AdaGrad()
• AdaGrad(stepSize)
• AdaGrad(stepSize, batchSize)
• AdaGrad(stepSize, batchSize, epsilon, maxIterations, tolerance, shuffle)
• AdaGrad(stepSize, batchSize, epsilon, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Attributes

Attributes of the optimizer may also be changed via the member methods StepSize(), BatchSize(), Epsilon(), MaxIterations(), Tolerance(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples:

AdaGrad optimizer(1.0, 1, 1e-8, 1000, 1e-9, true);

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();
optimizer.Optimize(f, coordinates);

See also:

• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
• AdaGrad in Wikipedia
• AdaDelta
• Differentiable separable functions

AdaSqrt

An optimizer for differentiable separable functions.

AdaSqrt is an optimizer with parameter-specific learning rates, which are adapted relative to how frequently a parameter gets updated during training. Larger updates for more sparse parameters and smaller updates for less sparse parameters. AdaSqrt, removes the square root in the denominator and scales the learning rate by sqrt(T).

Constructors

• AdaSqrt()
• AdaSqrt(stepSize)
• AdaSqrt(stepSize, batchSize)
• AdaSqrt(stepSize, batchSize, epsilon, maxIterations, tolerance, shuffle)
• AdaSqrt(stepSize, batchSize, epsilon, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Attributes

Attributes of the optimizer may also be changed via the member methods StepSize(), BatchSize(), Epsilon(), MaxIterations(), Tolerance(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples:

AdaSqrt optimizer(1.0, 1, 1e-8, 1000, 1e-9, true);

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();
optimizer.Optimize(f, coordinates);

See also:

• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
• AdaGrad in Wikipedia
• AdaDelta
• Differentiable separable functions

Adam

An optimizer for differentiable separable functions.

Adam is an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.

Constructors

• Adam()
• Adam(stepSize, batchSize)
• Adam(stepSize, batchSize, beta1, beta2, eps, maxIterations, tolerance, shuffle)
• Adam(stepSize, batchSize, beta1, beta2, eps, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Note that the Adam class is based on the AdamType<UpdateRule> class with UpdateRule = AdamUpdate.

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Beta1(), Beta2(), Epsilon(), MaxIterations(), Tolerance(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

Adam optimizer(0.001, 32, 0.9, 0.999, 1e-8, 100000, 1e-5, true);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• Adam: A Method for Stochastic Optimization
• Differentiable separable functions

AdaMax

An optimizer for differentiable separable functions.

AdaMax is simply a variant of Adam based on the infinity norm.

Constructors

• AdaMax()
• AdaMax(stepSize, batchSize)
• AdaMax(stepSize, batchSize, beta1, beta2, eps, maxIterations, tolerance, shuffle)
• AdaMax(stepSize, batchSize, beta1, beta2, eps, maxIterations, tolerance, shuffle, exactObjective, resetPolicy)

Note that the AdaMax class is based on the AdamType<UpdateRule> class with UpdateRule = AdaMaxUpdate.

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Beta1(), Beta2(), Eps(), MaxIterations(), Tolerance(), Shuffle(), ExactObjective(), and ResetPolicy().

Examples

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

AdaMax optimizer(0.001, 32, 0.9, 0.999, 1e-8, 100000, 1e-5, true);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• Adam: A Method for Stochastic Optimization (see section 7)
• Differentiable separable functions

AMSBound

An optimizer for differentiable separable functions.

AMSBound is a variant of Adam which employs dynamic bounds on learning rates.

Constructors

• AMSBound()
• AMSBound(stepSize, batchSize)
• AMSBound(stepSize, batchSize, finalLr, gamma, beta1, beta2, epsilon, maxIterations, tolerance, shuffle)
• AMSBound(stepSize, batchSize, finalLr, gamma, beta1, beta2, epsilon, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Note that the AMSBound class is based on the AdaBoundType<UpdateRule> class with UpdateRule = AdaBoundUpdate.

Attributes

The attributes of the optimizer may also be modified via the member methods FinalLr(), Gamma(), StepSize(), BatchSize(), Beta1(), Beta2(), Eps(), MaxIterations(), Tolerance(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples

SphereFunction f(2);
arma::mat coordinates = f.GetInitialPoint();

AMSBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000, 1e-3);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Adam: A Method for Stochastic Optimization
• Differentiable separable functions

AMSGrad

An optimizer for differentiable separable functions.

AMSGrad is a variant of Adam with guaranteed convergence.

Constructors

• AMSGrad()
• AMSGrad(stepSize, batchSize)
• AMSGrad(stepSize, batchSize, beta1, beta2, eps, maxIterations, tolerance, shuffle)
• AMSGrad(stepSize, batchSize, beta1, beta2, eps, maxIterations, tolerance, shuffle, exactObjective, resetPolicy)

Note that the AMSGrad class is based on the AdamType<UpdateRule> class with UpdateRule = AMSGradUpdate.

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Beta1(), Beta2(), Eps(), MaxIterations(), Tolerance(), Shuffle(), ExactObjective(), and ResetPolicy().

Examples

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

AMSGrad optimizer(0.001, 32, 0.9, 0.999, 1e-8, 100000, 1e-5, true);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• On the Convergence of Adam and Beyond
• Differentiable separable functions

Augmented Lagrangian

An optimizer for differentiable constrained functions.

The AugLagrangian class implements the Augmented Lagrangian method of optimization. In this scheme, a penalty term is added to the Lagrangian. This method is also called the “method of multipliers”. Internally, the optimizer uses L-BFGS.

Constructors

• AugLagrangian(maxIterations, penaltyThresholdFactor, sigmaUpdateFactor)

Attributes

The attributes of the optimizer may also be modified via the member methods MaxIterations(), PenaltyThresholdFactor(), SigmaUpdateFactor() and LBFGS().

/**
 * Optimize the function.  The value '1' is used for the initial value of each
 * Lagrange multiplier.  To set the Lagrange multipliers yourself, use the
 * other overload of Optimize().
 *
 * @tparam LagrangianFunctionType Function which can be optimized by this
 *     class.
 * @param function The function to optimize.
 * @param coordinates Output matrix to store the optimized coordinates in.
 */
template<typename LagrangianFunctionType>
bool Optimize(LagrangianFunctionType& function,
              arma::mat& coordinates);

/**
 * Optimize the function, giving initial estimates for the Lagrange
 * multipliers.  The vector of Lagrange multipliers will be modified to
 * contain the Lagrange multipliers of the final solution (if one is found).
 *
 * @tparam LagrangianFunctionType Function which can be optimized by this
 *      class.
 * @param function The function to optimize.
 * @param coordinates Output matrix to store the optimized coordinates in.
 * @param initLambda Vector of initial Lagrange multipliers.  Should have
 *     length equal to the number of constraints.
 * @param initSigma Initial penalty parameter.
 */
template<typename LagrangianFunctionType>
bool Optimize(LagrangianFunctionType& function,
              arma::mat& coordinates,
              const arma::vec& initLambda,
              const double initSigma);

Examples

GockenbachFunction f;
arma::mat coordinates = f.GetInitialPoint();

AugLagrangian optimizer;
optimizer.Optimize(f, coords);

See also:

• Augmented Lagrangian method on Wikipedia
• L-BFGS
• Constrained functions

Big Batch SGD

An optimizer for differentiable separable functions.

Big-batch stochastic gradient descent adaptively grows the batch size over time to maintain a nearly constant signal-to-noise ratio in the gradient approximation, so the Big Batch SGD optimizer is able to adaptively adjust batch sizes without user oversight.

Constructors

• BigBatchSGD<UpdatePolicy>()
• BigBatchSGD<UpdatePolicy>(stepSize)
• BigBatchSGD<UpdatePolicy>(stepSize, batchSize)
• BigBatchSGD<UpdatePolicy>(stepSize, batchSize, epsilon, maxIterations, tolerance, shuffle, exactObjective)

The UpdatePolicy template parameter refers to the way that a new step size is computed. The AdaptiveStepsize and BacktrackingLineSearch classes are available for use; custom behavior can be achieved by implementing a class with the same method signatures.

For convenience the following typedefs have been defined:

• BBS_Armijo = BigBatchSGD<BacktrackingLineSearch>
• BBS_BB = BigBatchSGD<AdaptiveStepsize>

Attributes

Attributes of the optimizer may also be changed via the member methods BatchSize(), StepSize(), BatchDelta(), MaxIterations(), Tolerance(), Shuffle(), and ExactObjective().

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

// Big-Batch SGD with the adaptive stepsize policy.
BBS_BB optimizer(10, 0.01, 0.1, 8000, 1e-4);
optimizer.Optimize(f, coordinates);

// Big-Batch SGD with backtracking line search.
BBS_Armijo optimizer2(10, 0.01, 0.1, 8000, 1e-4);
optimizer2.Optimize(f, coordinates);

See also:

• Big Batch SGD: Automated Inference using Adaptive Batch Sizes
• SGD in Wikipedia
• SGD

Coordinate Descent (CD)

An optimizer for partially differentiable functions.

Coordinate descent is a technique for minimizing a function by doing a line search along a single direction at the current point in the iteration. The direction (or “coordinate”) can be chosen cyclically, randomly or in a greedy fashion.

Constructors

• CD<DescentPolicyType>()
• CD<DescentPolicyType>(stepSize, maxIterations)
• CD<DescentPolicyType>(stepSize, maxIterations, tolerance, updateInterval)
• CD<DescentPolicyType>(stepSize, maxIterations, tolerance, updateInterval, descentPolicy)

The DescentPolicyType template parameter specifies the behavior of CD when selecting the next coordinate to descend with. The RandomDescent, GreedyDescent, and CyclicDescent classes are available for use. Custom behavior can be achieved by implementing a class with the same method signatures.

For convenience, the following typedefs have been defined:

• RandomCD (equivalent to CD<RandomDescent>): selects coordinates randomly
• GreedyCD (equivalent to CD<GreedyDescent>): selects the coordinate with the maximum guaranteed descent according to the Gauss-Southwell rule
• CyclicCD (equivalent to CD<CyclicDescent>): selects coordinates sequentially

Note: CD used to be called SCD. Use of the name SCD is deprecated, and will be removed in ensmallen 3 and later.

Attributes

Attributes of the optimizer may also be modified via the member methods StepSize(), MaxIterations(), Tolerance(), UpdateInterval(), and DescentPolicy().

Note that the default value for descentPolicy is the default constructor for DescentPolicyType.

Examples

SparseTestFunction f;
arma::mat coordinates = f.GetInitialPoint();

RandomCD randomscd(0.01, 100000, 1e-5, 1e3);
randomscd.Optimize(f, coordinates);

GreedyCD greedyscd(0.01, 100000, 1e-5, 1e3);
greedyscd.Optimize(f, coordinates);

CyclicCD cyclicscd(0.01, 100000, 1e-5, 1e3);
cyclicscd.Optimize(f, coordinates);

See also:

• Coordinate descent on Wikipedia
• Stochastic Methods for L1-Regularized Loss Minimization
• Partially differentiable functions

CMAES

An optimizer for separable functions.

CMA-ES (Covariance Matrix Adaptation Evolution Strategy) is a stochastic search algorithm. CMA-ES is a second order approach estimating a positive definite matrix within an iterative procedure using the covariance matrix.

Constructors

• CMAES<SelectionPolicyType, TransformationPolicyType>()
• CMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy)
• CMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy, batchSize)
• CMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy)
• CMAES<SelectionPolicyType, TransformationPolicyType>(lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize)

The SelectionPolicyType template parameter refers to the strategy used to compute the (approximate) objective function. The FullSelection and RandomSelection classes are available for use; custom behavior can be achieved by implementing a class with the same method signatures. The TransformationPolicyType template parameter refers to transformation strategy used to map decision variables to the desired domain during fitness evaluation and optimization termination. The EmptyTransformation and BoundaryBoxConstraint classes are available for use; custom behavior can be achieved by implementing a class with the same method signatures.

For convenience the following types can be used:

• CMAES<> (equivalent to CMAES<FullSelection, EmptyTransformation<>>): uses all separable functions to compute objective
• ApproxCMAES<> (equivalent to CMAES<RandomSelection, EmptyTransformation<>>): uses a small amount of separable functions to compute approximate objective

Attributes

Attributes of the optimizer may also be changed via the member methods Lambda(), TransformationPolicy(), BatchSize(), MaxIterations(), Tolerance(), and SelectionPolicy().

The selectionPolicy attribute allows an instantiated SelectionPolicyType to be given. The FullSelection policy has no need to be instantiated and thus the option is not relevant when the CMAES<> optimizer type is being used; the RandomSelection policy has the constructor RandomSelection(fraction) where fraction specifies the percentage of separable functions to use to estimate the objective function. The transformationPolicy attribute allows an instantiated TransformationPolicyType to be given. The EmptyTransformation<MatType> has no need to be instantiated. BoundaryBoxConstraint<MatType> policy has the constructor BoundaryBoxConstraint(lowerBound, upperBound) where lowerBound and lowerBound are the lower bound and upper bound of the coordinates respectively.

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

// CMAES with the FullSelection and BoundaryBoxConstraint policies.
BoundaryBoxConstraint b(-1, 1);
CMAES optimizer(0, b, 32, 200, 1e-4);
optimizer.Optimize(f, coordinates);

// CMAES with the RandomSelection and BoundaryBoxConstraint policies.
ApproxCMAES<BoundaryBoxConstraint<>> cmaes(0, b, 32, 200, 1e-4);
approxOptimizer.Optimize(f, coordinates);

See also:

• Completely Derandomized Self-Adaptation in Evolution Strategies
• CMA-ES in Wikipedia
• Evolution strategy in Wikipedia

CNE

An optimizer for arbitrary functions.

Conventional Neural Evolution is an optimizer that works like biological evolution which selects best candidates based on their fitness scores and creates new generation by mutation and crossover of population. The initial population is generated based on a random normal distribution centered at the given starting point.

Constructors

• CNE()
• CNE(populationSize, maxGenerations)
• CNE(populationSize, maxGenerations, mutationProb, mutationSize)
• CNE(populationSize, maxGenerations, mutationProb, mutationSize, selectPercent, tolerance)

Attributes

Attributes of the optimizer may also be changed via the member methods PopulationSize(), MaxGenerations(), MutationProb(), SelectPercent() and Tolerance().

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

CNE optimizer(200, 10000, 0.2, 0.2, 0.3, 1e-5);
optimizer.Optimize(f, coordinates);

See also:

• Neuroevolution in Wikipedia
• Arbitrary functions

DE

An optimizer for arbitrary functions.

Differential Evolution is an evolutionary optimization algorithm which selects best candidates based on their fitness scores and creates new generation by mutation and crossover of population.

Constructors

• DE()
• DE(populationSize, maxGenerations)
• DE(populationSize, maxGenerations, crossoverRate)
• DE(populationSize, maxGenerations, crossoverRate, differentialWeight)
• DE(populationSize, maxGenerations, crossoverRate, differentialWeight, tolerance)

Attributes

Attributes of the optimizer may also be changed via the member methods PopulationSize(), MaxGenerations(), CrossoverRate(), DifferentialWeight() and Tolerance().

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

DE optimizer(200, 1000, 0.6, 0.8, 1e-5);
optimizer.Optimize(f, coordinates);

See also:

• Differential Evolution - A simple and efficient adaptive scheme for global optimization over continuous spaces
• Differential Evolution in Wikipedia
• Arbitrary functions

DemonAdam

An optimizer for differentiable separable functions.

DemonAdam is an Adam based optimizer. DemonAdam is motivated by decaying the total contribution of a gradient to all future updates.

Constructors

• DemonAdam()
• DemonAdam(stepSize, batchSize)
• DemonAdam(stepSize, batchSize, momentum, beta1, beta2, eps, maxIterations, tolerance, shuffle)
• DemonAdam(stepSize, batchSize, momentum, beta1, beta2, eps, maxIterations, tolerance, shuffle, resetPolicy)

Note that the DemonAdam class is based on the DemonAdamType<UpdateRule> class with UpdateRule = AdamUpdate.

For convenience the following typedefs have been defined:

• DemonAdaMax (equivalent to DemonAdamType<AdaMaxUpdate>): DemonAdam that uses the AdaMax update rule.
• DemonAMSGrad (equivalent to DemonAdamType<AMSGradUpdate>): DemonAdam that uses the AMSGrad update rule.
• DemonNadam (equivalent to DemonAdamType<NadamUpdate>): DemonAdam that uses the Nadam update rule.
• NadamUpdate (equivalent to DemonAdamType<NadaMaxUpdate>): DemonAdam that uses the NadaMax update rule.
• DemonOptimisticAdam (equivalent to DemonAdamType<OptimisticAdamUpdate>): DemonAdam that uses the OptimisticAdam update rule.

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Momentum(), MomentumIterations(), Beta1(), Beta2(), Eps(), MaxIterations(), Tolerance(), Shuffle(), and ResetPolicy().

Examples

MatyasFunction f;
arma::mat coordinates = f.GetInitialPoint();

DemonAdam optimizer(0.5, 1, 0.9);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• Decaying momentum helps neural network training
• Differentiable separable functions

DemonSGD

An optimizer for differentiable separable functions.

DemonSGD is an SGD based optimizer. DemonSGD is motivated by decaying the total contribution of a gradient to all future updates.

For convenience ensmallen implements various Adam based versions of the Demon optimizer:

• DemonAdam (equivalent to DemonAdamType<AdamUpdate>): DemonAdam that uses the Adam update rule.
• DemonAdaMax (equivalent to DemonAdamType<AdaMaxUpdate>): DemonAdam that uses the AdaMax update rule.
• DemonAMSGrad (equivalent to DemonAdamType<AMSGradUpdate>): DemonAdam that uses the AMSGrad update rule.
• DemonNadam (equivalent to DemonAdamType<NadamUpdate>): DemonAdam that uses the Nadam update rule.
• NadamUpdate (equivalent to DemonAdamType<NadaMaxUpdate>): DemonAdam that uses the NadaMax update rule.
• DemonOptimisticAdam (equivalent to DemonAdamType<OptimisticAdamUpdate>): DemonAdam that uses the OptimisticAdam update rule.

Constructors

• DemonSGD()
• DemonSGD(stepSize, batchSize)
• DemonSGD(stepSize, batchSize, momentum, maxIterations, tolerance, shuffle)
• DemonSGD(stepSize, batchSize, momentum, maxIterations, tolerance, shuffle, resetPolicy)

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Momentum(), MomentumIterations(), MaxIterations(), Tolerance(), Shuffle(), and ResetPolicy().

Examples

MatyasFunction f;
arma::mat coordinates = f.GetInitialPoint();

DemonSGD optimizer(0.5, 1, 0.9);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• Decaying momentum helps neural network training
• Differentiable separable functions

Eve

An optimizer for differentiable separable functions.

Eve is a stochastic gradient based optimization method with locally and globally adaptive learning rates.

Constructors

• Eve()
• Eve(stepSize, batchSize)
• Eve(stepSize, batchSize, beta1, beta2, beta3, epsilon, clip, maxIterations, tolerance, shuffle)

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Beta1(), Beta2(), Beta3(), Epsilon(), Clip(), MaxIterations(), Tolerance(), Shuffle(), and ExactObjective().

Examples

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

Eve optimizer(0.001, 32, 0.9, 0.999, 0.999, 10, 1e-8, 100000, 1e-5, true);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
• Differentiable separable functions

Frank-Wolfe

An optimizer for differentiable functions that may also be constrained.

Frank-Wolfe is a technique to minimize a continuously differentiable convex function f over a compact convex subset D of a vector space. It is also known as conditional gradient method.

Constructors

• FrankWolfe<LinearConstrSolverType, UpdateRuleType>(linearConstrSolver, updateRule)
• FrankWolfe<LinearConstrSolverType, UpdateRuleType>(linearConstrSolver, updateRule, maxIterations, tolerance)

The LinearConstrSolverType template parameter specifies the constraint domain D for the problem. The ConstrLpBallSolver and ConstrStructGroupSolver<GroupLpBall> classes are available for use; the former restricts D to the unit ball of the specified l-p norm. Other constraint types may be implemented as a class with the same method signatures as either of the existing classes.

The UpdateRuleType template parameter specifies the update rule used by the optimizer. The UpdateClassic and UpdateLineSearch classes are available for use and represent a simple update step rule and a line search based update rule, respectively. The UpdateSpan and UpdateFulLCorrection classes are also available and may be used with the FuncSq function class (which is a squared matrix loss).

For convenience the following typedefs have been defined:

• OMP (equivalent to FrankWolfe<ConstrLpBallSolver, UpdateSpan>): a solver for the orthogonal matching pursuit problem
• StandardFrankWolfe (equivalent to FrankWolfe<ConstrLpBallSolver, ClassicUpdate>): the standard Frank-Wolfe algorithm with the solution restricted to lie within the unit ball

Attributes

Attributes of the optimizer may also be changed via the member methods LinearConstrSolver(), UpdateRule(), MaxIterations(), and Tolerance().

Examples:

TODO

See also:

• An algorithm for quadratic programming
• Frank-Wolfe in Wikipedia
• Differentiable functions

FTML (Follow the Moving Leader)

An optimizer for differentiable separable functions.

Follow the Moving Leader (FTML) is an optimizer where recent samples are weighted more heavily in each iteration, so FTML can adapt more quickly to changes.

Constructors

• FTML()
• FTML(stepSize, batchSize)
• FTML(stepSize, batchSize, beta1, beta2, epsilon, maxIterations, tolerance, shuffle)
• FTML(stepSize, batchSize, beta1, beta2, epsilon, maxIterations, tolerance, shuffle, resetPolicy, exactObjective)

Attributes

The attributes of the optimizer may also be modified via the member methods StepSize(), BatchSize(), Beta1(), Beta2(), Epsilon(), MaxIterations(), Tolerance(), Shuffle(), ResetPolicy(), and ExactObjective().

Examples

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

FTML optimizer(0.001, 32, 0.9, 0.999, 1e-8, 100000, 1e-5, true);
optimizer.Optimize(f, coordinates);

See also:

• Follow the Moving Leader in Deep Learning
• SGD in Wikipedia
• SGD
• Differentiable separable functions

Gradient Descent

An optimizer for differentiable functions.

Gradient Descent is a technique to minimize a function. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient of the function at the current point.

Constructors

• GradientDescent()
• GradientDescent(stepSize)
• GradientDescent(stepSize, maxIterations, tolerance)

Attributes

Attributes of the optimizer may also be changed via the member methods StepSize(), MaxIterations(), and Tolerance().

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

GradientDescent optimizer(0.001, 0, 1e-15);
optimizer.Optimize(f, coordinates);

See also:

• Gradient descent in Wikipedia
• Differentiable functions

Grid Search

An optimizer for categorical functions.

An optimizer that finds the minimum of a given function by iterating through points on a multidimensional grid.

Constructors

• GridSearch()

Attributes

The GridSearch class has no configurable attributes.

Note: the GridSearch class can only optimize categorical functions where every parameter is categorical.

See also:

• Categorical functions (includes an example for GridSearch)
• Grid search on Wikipedia

Hogwild! (Parallel SGD)

An optimizer for sparse differentiable separable functions.

An implementation of parallel stochastic gradient descent using the lock-free HOGWILD! approach. This implementation requires OpenMP to be enabled during compilation (i.e., -fopenmp specified as a compiler flag).

Note that the requirements for Hogwild! are slightly different than for most differentiable separable functions but it is often possible to use Hogwild! by implementing Gradient() with a template parameter. See the sparse differentiable separable functions documentation for more details.

Constructors

• ParallelSGD<DecayPolicyType>(maxIterations, threadShareSize)
• ParallelSGD<DecayPolicyType>(maxIterations, threadShareSize, tolerance, shuffle, decayPolicy)

The DecayPolicyType template parameter specifies the policy used to update the step size after each iteration. The ConstantStep class is available for use. Custom behavior can be achieved by implementing a class with the same method signatures.

The default type for DecayPolicyType is ConstantStep, so the shorter type ParallelSGD<> can be used instead of the equivalent ParallelSGD<ConstantStep>.

Attributes

Attributes of the optimizer may also be modified via the member methods MaxIterations(), ThreadShareSize(), Tolerance(), Shuffle(), and DecayPolicy().

Note that the default value for decayPolicy is the default constructor for the DecayPolicyType.

Examples

GeneralizedRosenbrockFunction f(50); // 50-dimensional Rosenbrock function.
arma::mat coordinates = f.GetInitialPoint();

ParallelSGD<> optimizer(100000, f.NumFunctions(), 1e-5, true);
optimizer.Optimize(f, coordinates);

See also:

• SGD in Wikipedia
• SGD
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Sparse differentiable separable functions

IQN

An optimizer for differentiable separable functions.

The Incremental Quasi-Newton belongs to the family of stochastic and incremental methods that have a cost per iteration independent of n. IQN iterations are a stochastic version of BFGS iterations that use memory to reduce the variance of stochastic approximations.

Constructors

• IQN()
• IQN(stepSize)
• IQN(stepSize, batchSize, maxIterations, tolerance)

Attributes

Attributes of the optimizer may also be changed via the member methods StepSize(), BatchSize(), MaxIterations(), and Tolerance().

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

IQN optimizer(0.01, 1, 5000, 1e-5);
optimizer.Optimize(f, coordinates);

See also:

• IQN: An Incremental Quasi-Newton Method with Local Superlinear Convergence Rate
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• Differentiable functions

Katyusha

An optimizer for differentiable separable functions.

Katyusha is a direct, primal-only stochastic gradient method which uses a “negative momentum” on top of Nesterov’s momentum. Two types are available—one that uses a proximal update step, and one that uses the standard update step.

Constructors

• KatyushaType<proximal>()
• KatyushaType<proximal>(convexity, lipschitz)
• KatyushaType<proximal>(convexity, lipschitz, batchSize)
• KatyushaType<proximal>(convexity, lipschitz, batchSize, maxIterations, innerIterations, tolerance, shuffle, exactObjective)

The proximal template parameter is a boolean value (true or false) that specifies whether or not the proximal update should be used.

For convenience the following typedefs have been defined:

• Katyusha (equivalent to KatyushaType<false>): Katyusha with the standard update step
• KatyushaProximal (equivalent to KatyushaType<true>): Katyusha with the proximal update step

Attributes

Attributes of the optimizer may also be changed via the member methods Convexity(), Lipschitz(), BatchSize(), MaxIterations(), InnerIterations(), Tolerance(), Shuffle(), and ExactObjective().

Examples:

RosenbrockFunction f;
arma::mat coordinates = f.GetInitialPoint();

// Without proximal update.
Katyusha optimizer(1.0, 10.0, 1, 100, 0, 1e-10, true);
optimizer.Optimize(f, coordinates);

// With proximal update.
KatyushaProximal proximalOptimizer(1.0, 10.0, 1, 100, 0, 1e-10, true);
proximalOptimizer.Optimize(f, coordinates);

See also:

• Katyusha: The First Direct Acceleration of Stochastic Gradient Methods
• Stochastic gradient descent in Wikipedia
• Differentiable separable functions