ExampleMergingConfig Class Reference

#include <nnet-example-utils.h>

Collaboration diagram for ExampleMergingConfig:

Classes
struct	IntSet

Public Member Functions
	ExampleMergingConfig (const char *default_minibatch_size="256")
void	Register (OptionsItf *po)
void	ComputeDerived ()
int32	MinibatchSize (int32 size_of_eg, int32 num_available_egs, bool input_ended) const
	This function tells you what minibatch size should be used for this eg. More...

Public Attributes
bool	compress
std::string	measure_output_frames
std::string	minibatch_size
std::string	discard_partial_minibatches

Static Private Member Functions
static bool	ParseIntSet (const std::string &str, IntSet *int_set)

Private Attributes
std::vector< std::pair< int32, IntSet > >	rules

Detailed Description

Definition at line 321 of file nnet-example-utils.h.

Constructor & Destructor Documentation

ExampleMergingConfig()

ExampleMergingConfig ( const char *  default_minibatch_size = "256" )

inline

Definition at line 329 of file nnet-example-utils.h.

                                                                  :
      compress(false),
      measure_output_frames("deprecated"),
      minibatch_size(default_minibatch_size),
      discard_partial_minibatches("deprecated") { }

Member Function Documentation

ComputeDerived()

void ComputeDerived

(

)

Definition at line 958 of file nnet-example-utils.cc.

References kaldi::ConvertStringToInteger(), rnnlm::i, kaldi::IsSortedAndUniq(), KALDI_ERR, KALDI_WARN, and kaldi::SplitStringToVector().

Referenced by main().

                                          {
  if (measure_output_frames != "deprecated") {
    KALDI_WARN << "The --measure-output-frames option is deprecated "
        "and will be ignored.";
  }
  if (discard_partial_minibatches != "deprecated") {
    KALDI_WARN << "The --discard-partial-minibatches option is deprecated "
        "and will be ignored.";
  }
  std::vector<std::string> minibatch_size_split;
  SplitStringToVector(minibatch_size, "/", false, &minibatch_size_split);
  if (minibatch_size_split.empty()) {
    KALDI_ERR << "Invalid option --minibatch-size=" << minibatch_size;
  }

  rules.resize(minibatch_size_split.size());
  for (size_t i = 0; i < minibatch_size_split.size(); i++) {
    int32 &eg_size = rules[i].first;
    IntSet &int_set = rules[i].second;
    // 'this_rule' will be either something like "256" or like "64-128,256"
    // (but these two only if  minibatch_size_split.size() == 1, or something with
    // an example-size specified, like "256=64-128,256"
    std::string &this_rule = minibatch_size_split[i];
    if (this_rule.find('=') != std::string::npos) {
      std::vector<std::string> rule_split;  // split on '='
      SplitStringToVector(this_rule, "=", false, &rule_split);
      if (rule_split.size() != 2) {
        KALDI_ERR << "Could not parse option --minibatch-size="
                  << minibatch_size;
      }
      if (!ConvertStringToInteger(rule_split[0], &eg_size) ||
          !ParseIntSet(rule_split[1], &int_set))
        KALDI_ERR << "Could not parse option --minibatch-size="
                  << minibatch_size;

    } else {
      if (minibatch_size_split.size() != 1) {
        KALDI_ERR << "Could not parse option --minibatch-size="
                  << minibatch_size << " (all rules must have "
                  << "eg-size specified if >1 rule)";
      }
      if (!ParseIntSet(this_rule, &int_set))
        KALDI_ERR << "Could not parse option --minibatch-size="
                  << minibatch_size;
    }
  }
  {
    // check that no size is repeated.
    std::vector<int32> all_sizes(minibatch_size_split.size());
    for (size_t i = 0; i < minibatch_size_split.size(); i++)
      all_sizes[i] = rules[i].first;
    std::sort(all_sizes.begin(), all_sizes.end());
    if (!IsSortedAndUniq(all_sizes)) {
      KALDI_ERR << "Invalid --minibatch-size=" << minibatch_size
                << " (repeated example-sizes)";
    }
  }
}

MinibatchSize()

int32 MinibatchSize	(	int32	size_of_eg,
		int32	num_available_egs,
		bool	input_ended
	)		const

This function tells you what minibatch size should be used for this eg.

Parameters

[in]	size_of_eg	The "size" of the eg, as obtained by GetNnetExampleSize() or a similar function (up to the caller).
[in]	num_available_egs	The number of egs of this size that are currently available; should be >0. The value returned will be <= this value, possibly zero.
[in]	input_ended	True if the input has ended, false otherwise. This is important because before the input has ended, we will only batch egs into the largest possible minibatch size among the range allowed for that size of eg.

Returns

Returns the minibatch size to use in this situation, as specified by the configuration.

Definition at line 1017 of file nnet-example-utils.cc.

References rnnlm::i, KALDI_ASSERT, and KALDI_ERR.

Referenced by DiscriminativeExampleMerger::AcceptExample(), ChainExampleMerger::AcceptExample(), ExampleMerger::AcceptExample(), DiscriminativeExampleMerger::Finish(), ChainExampleMerger::Finish(), and ExampleMerger::Finish().

                                                                  {
  KALDI_ASSERT(num_available_egs > 0 && size_of_eg > 0);
  int32 num_rules = rules.size();
  if (num_rules == 0)
    KALDI_ERR << "You need to call ComputeDerived() before calling "
        "MinibatchSize().";
  int32 min_distance = std::numeric_limits<int32>::max(),
      closest_rule_index = 0;
  for (int32 i = 0; i < num_rules; i++) {
    int32 distance = std::abs(size_of_eg - rules[i].first);
    if (distance < min_distance) {
      min_distance = distance;
      closest_rule_index = i;
    }
  }
  if (!input_ended) {
    // until the input ends, we can only use the largest available
    // minibatch-size (otherwise, we could expect more later).
    int32 largest_size = rules[closest_rule_index].second.largest_size;
    if (largest_size <= num_available_egs)
      return largest_size;
    else
      return 0;
  } else {
    int32 s = rules[closest_rule_index].second.LargestValueInRange(
        num_available_egs);
    KALDI_ASSERT(s <= num_available_egs);
    return s;
  }
}

ParseIntSet()

bool ParseIntSet ( const std::string &  str, ExampleMergingConfig::IntSet *  int_set  )

staticprivate

Definition at line 936 of file nnet-example-utils.cc.

References rnnlm::i, ExampleMergingConfig::IntSet::largest_size, ExampleMergingConfig::IntSet::ranges, kaldi::SplitStringToIntegers(), and kaldi::SplitStringToVector().

                                                                            {
  std::vector<std::string> split_str;
  SplitStringToVector(str, ",", false, &split_str);
  if (split_str.empty())
    return false;
  int_set->largest_size = 0;
  int_set->ranges.resize(split_str.size());
  for (size_t i = 0; i < split_str.size(); i++) {
    std::vector<int32> split_range;
    SplitStringToIntegers(split_str[i], ":", false, &split_range);
    if (split_range.size() < 1 || split_range.size() > 2 ||
        split_range[0] > split_range.back() || split_range[0] <= 0)
      return false;
    int_set->ranges[i].first = split_range[0];
    int_set->ranges[i].second = split_range.back();
    int_set->largest_size = std::max<int32>(int_set->largest_size,
                                            split_range.back());
  }
  return true;
}

Register()

void Register ( OptionsItf *  po )

inline

Definition at line 335 of file nnet-example-utils.h.

References ExampleGenerationConfig::ComputeDerived(), and OptionsItf::Register().

Referenced by main().

                                {
    po->Register("compress", &compress, "If true, compress the output examples "
                 "(not recommended unless you are writing to disk)");
    po->Register("measure-output-frames", &measure_output_frames, "This "
                 "value will be ignored (included for back-compatibility)");
    po->Register("discard-partial-minibatches", &discard_partial_minibatches,
                 "This value will be ignored (included for back-compatibility)");
    po->Register("minibatch-size", &minibatch_size,
                 "String controlling the minibatch size.  May be just an integer, "
                 "meaning a fixed minibatch size (e.g. --minibatch-size=128). "
                 "May be a list of ranges and values, e.g. --minibatch-size=32,64 "
                 "or --minibatch-size=16:32,64,128.  All minibatches will be of "
                 "the largest size until the end of the input is reached; "
                 "then, increasingly smaller sizes will be allowed.  Only egs "
                 "with the same structure (e.g num-frames) are merged.  You may "
                 "specify different minibatch sizes for different sizes of eg "
                 "(defined as the maximum number of Indexes on any input), in "
                 "the format "
                 "--minibatch-size='eg_size1=mb_sizes1/eg_size2=mb_sizes2', e.g. "
                 "--minibatch-size=128=64:128,256/256=32:64,128.  Egs are given "
                 "minibatch-sizes based on the specified eg-size closest to "
                 "their actual size.");
  }

Member Data Documentation

compress

bool compress

Definition at line 324 of file nnet-example-utils.h.

Referenced by DiscriminativeExampleMerger::WriteMinibatch(), ChainExampleMerger::WriteMinibatch(), and ExampleMerger::WriteMinibatch().

discard_partial_minibatches

std::string discard_partial_minibatches

Definition at line 327 of file nnet-example-utils.h.

measure_output_frames

std::string measure_output_frames

Definition at line 325 of file nnet-example-utils.h.

minibatch_size

std::string minibatch_size

Definition at line 326 of file nnet-example-utils.h.

rules

std::vector<std::pair<int32, IntSet> > rules

private

Definition at line 410 of file nnet-example-utils.h.

---

The documentation for this class was generated from the following files:

• nnet3/nnet-example-utils.h
• nnet3/nnet-example-utils.cc