MemoryCompressionOptimizer Class Reference

This class is used in the function OptimizeMemoryCompression(), once we determine that there is some potential to do memory compression for this computation. More...

Collaboration diagram for MemoryCompressionOptimizer:

Classes
struct	MatrixCompressInfo

Public Member Functions
	MemoryCompressionOptimizer (const Nnet &nnet, int32 memory_compression_level, int32 middle_command, NnetComputation *computation)
void	Optimize ()

Private Member Functions
void	ProcessMatrix (int32 m)
void	ModifyComputation ()

Private Attributes
std::vector< MatrixCompressInfo >	compress_info_
const Nnet &	nnet_
int32	memory_compression_level_
int32	middle_command_
NnetComputation *	computation_
Analyzer	analyzer_

Detailed Description

This class is used in the function OptimizeMemoryCompression(), once we determine that there is some potential to do memory compression for this computation.

Definition at line 4690 of file nnet-optimize-utils.cc.

Constructor & Destructor Documentation

MemoryCompressionOptimizer()

MemoryCompressionOptimizer ( const Nnet &  nnet, int32  memory_compression_level, int32  middle_command, NnetComputation *  computation  )

inline

Parameters

[in]	nnet	The neural net the computation is for.
[in]	memory_compression_level.	The level of compression: 0 = no compression (the constructor should not be called with this value). 1 = compression that doesn't affect the results (but still takes time). 2 = compression that affects the results only very slightly 3 = compression that affects the results a little more.
[in]	middle_command	Must be the command-index of the command of type kNoOperationMarker in 'computation'.
[in,out]	computation	The computation we're optimizing.

Definition at line 4703 of file nnet-optimize-utils.cc.

References kaldi::nnet3::Optimize().

                                                          :
      nnet_(nnet), memory_compression_level_(memory_compression_level),
      middle_command_(middle_command), computation_(computation) { }

Member Function Documentation

ModifyComputation()

void ModifyComputation ( )

private

Definition at line 4769 of file nnet-optimize-utils.cc.

References MemoryCompressionOptimizer::MatrixCompressInfo::compression_command_index, MemoryCompressionOptimizer::MatrixCompressInfo::compression_type, DerivativeTimeLimiter::computation_, NnetComputation::GetWholeSubmatrices(), rnnlm::i, kaldi::nnet3::InsertCommands(), kaldi::nnet3::kCompressMatrix, kaldi::nnet3::kDecompressMatrix, MemoryCompressionOptimizer::MatrixCompressInfo::m, MemoryCompressionOptimizer::MatrixCompressInfo::range, MemoryCompressionOptimizer::MatrixCompressInfo::truncate, and MemoryCompressionOptimizer::MatrixCompressInfo::uncompression_command_index.

                                                   {
  // whole_submatrices[m] is the submatrix-index of the submatrix that
  // represents the whole of matrix m.
  std::vector<int32> whole_submatrices;
  computation_->GetWholeSubmatrices(&whole_submatrices);

  // 'pairs_to_insert' will be a list of pairs (command-index, command),
  // meaning: (command-index just before which to insert this command; command
  // to insert).
  std::vector<std::pair<int32, NnetComputation::Command> >
      pairs_to_insert;
  pairs_to_insert.reserve(compress_info_.size() * 2);
  for (size_t i = 0; i < compress_info_.size(); i++) {
    const MatrixCompressInfo &info = compress_info_[i];
    int32 s = whole_submatrices[info.m];
    // below we use compression_command_index + 1 because we want the
    // compression to go after the command in 'info.compression_command_index'
    // (which might be, for instance, a forward propagation command).
    std::pair<int32, NnetComputation::Command> p1(
        info.compression_command_index + 1,
        NnetComputation::Command(info.range, kCompressMatrix,
                                 s, static_cast<int32>(info.compression_type),
                                 info.truncate ? 1 : 0));
    pairs_to_insert.push_back(p1);
    std::pair<int32, NnetComputation::Command> p2(
        info.uncompression_command_index,
        NnetComputation::Command(1.0, kDecompressMatrix, s));
    pairs_to_insert.push_back(p2);
  }
  InsertCommands(&pairs_to_insert,
                 computation_);
}

Optimize()

void Optimize

(

)

Definition at line 4803 of file nnet-optimize-utils.cc.

References DerivativeTimeLimiter::computation_, NnetComputation::matrices, and DerivativeTimeLimiter::nnet_.

Referenced by kaldi::nnet3::OptimizeMemoryCompression().

                                          {
  analyzer_.Init(nnet_, *computation_);
  // note: matrix zero is not really a matrix.
  int32 num_matrices = computation_->matrices.size();
  for (int32 m = 1; m < num_matrices; m++)
    ProcessMatrix(m);
  if (!compress_info_.empty())
    ModifyComputation();
}

ProcessMatrix()

void ProcessMatrix ( int32  m )

private

Definition at line 4813 of file nnet-optimize-utils.cc.

References Access::access_type, NnetComputation::Command::arg1, Access::command_index, NnetComputation::Command::command_type, NnetComputation::commands, DerivativeTimeLimiter::computation_, Nnet::GetComponent(), KALDI_ASSERT, kaldi::nnet3::kBackprop, kaldi::kCompressedMatrixInt16, kaldi::kCompressedMatrixUint8, kaldi::nnet3::kReadAccess, DerivativeTimeLimiter::nnet_, and Component::Type().

                                                      {
  if (analyzer_.matrix_accesses[m].is_output) {
    return;  // We can't do this optimization for matrices that are going to be
             // output to the user.
  }

  // 'accesses' list the commands that access this matrix.
  const std::vector<Access> &accesses = analyzer_.matrix_accesses[m].accesses;
  // the 'kReadAccess' below is actually a don't-care  This is just
  // to find the position in 'accesses' that corresponds to command-index
  // 'middle_command'.
  Access middle_access(middle_command_, kReadAccess);
  std::vector<Access>::const_iterator iter = std::lower_bound(accesses.begin(),
                                                              accesses.end(),
                                                              middle_access);
  // At this point, 'iter' points to the first access in 'accesses'
  // whose command index is >= 'middle_command_' (which separates the forward
  // and backward passes), or accesses.end() if this matrix was not
  // accessed during the backward pass.
  if (iter == accesses.end()) {
    return;  // There is nothing to do: this matrix was not accessed during the
             // backward pass.
  }
  if (iter == accesses.begin()) {
    return;  // There is nothing to do: this matrix was not accessed during the
             // forward pass.
  }
  // 'backward_access' is the first access of the matrix in the backward
  // pass of the computation, and
  // 'forward_access' is the last access of the matrix in the forward pass
  // of the computation.
  const Access &backward_access = iter[0],
      &forward_access = iter[-1];
  KALDI_ASSERT(forward_access.command_index < middle_command_ &&
               backward_access.command_index > middle_command_);

  // 'backward_access_is_last_access' is going to be set to true if
  // 'backward_access' is the last command to access the matrix (apart from
  // deallocation or matrix-swap commands, which don't show up in the list of
  // accesses).
  bool backward_access_is_last_access = (accesses.end() == iter + 1);

  int32 backward_command_index = backward_access.command_index,
      forward_command_index = forward_access.command_index;
  NnetComputation::Command
      &backward_command = computation_->commands[backward_command_index];

  if (memory_compression_level_ >= 1 &&
      backward_access_is_last_access &&
      backward_access.access_type == kReadAccess &&
      backward_command.command_type == kBackprop) {
    int32 component_index = backward_command.arg1;
    const Component *component = nnet_.GetComponent(component_index);
    // this is potentially a candidate for our optimization for ReLU units,
    // where we only need to store the sign.
    if (component->Type() == "RectifiedLinearComponent") {
      compress_info_.push_back(
          MatrixCompressInfo(m, forward_command_index,
                             backward_command_index,
                             kCompressedMatrixUint8, 0.0,
                             true));
      return;
    }
  }

  // If memory_compression_level >= 2 (an "intermediate" level of compression),
  // then we'll consider compressing quantities using 16 bits in the range
  // [-10, 10].  Because of the way this compression works, exact zero will
  // still be uncompressed as exact zero, so even if this is the output
  // of a ReLU, it's OK.  (Having a few derivatives zero for ReLU outputs
  // that were very close to zero is OK.)
  if (memory_compression_level_ >= 2) {
    compress_info_.push_back(
        MatrixCompressInfo(m, forward_command_index,
                           backward_command_index,
                           kCompressedMatrixInt16, 10.0,
                           true));
    return;
  }

  // TODO: later maybe implement something for memory compression level = 3.
}

Member Data Documentation

analyzer_

Analyzer analyzer_

private

Definition at line 4765 of file nnet-optimize-utils.cc.

compress_info_

std::vector<MatrixCompressInfo> compress_info_

private

Definition at line 4759 of file nnet-optimize-utils.cc.

computation_

NnetComputation* computation_

private

Definition at line 4764 of file nnet-optimize-utils.cc.

memory_compression_level_

int32 memory_compression_level_

private

Definition at line 4762 of file nnet-optimize-utils.cc.

middle_command_

int32 middle_command_

private

Definition at line 4763 of file nnet-optimize-utils.cc.

nnet_

const Nnet& nnet_

private

Definition at line 4761 of file nnet-optimize-utils.cc.

---

The documentation for this class was generated from the following file:

• nnet3/nnet-optimize-utils.cc