This class is responsible for storing, and displaying in log messages, statistics about how examples of different sizes (c.f. More...

#include <nnet-example-utils.h>

Collaboration diagram for ExampleMergingStats:

[legend]

Classes
struct	StatsForExampleSize

Public Member Functions
void	WroteExample (int32 example_size, size_t structure_hash, int32 minibatch_size)
	Users call this function to inform this class that one minibatch has been written aggregating 'minibatch_size' separate examples of original size 'example_size' (e.g. More...

void	DiscardedExamples (int32 example_size, size_t structure_hash, int32 num_discarded)
	Users call this function to inform this class that after processing all the data, for examples of original size 'example_size', 'num_discarded' examples could not be put into a minibatch and were discarded. More...

void	PrintStats () const
	Calling this will cause a log message with information about the examples to be printed. More...

Private Types
typedef unordered_map< std::pair< int32, size_t >, StatsForExampleSize, PairHasher< int32, size_t > >	StatsType

Private Member Functions
void	PrintAggregateStats () const

void	PrintSpecificStats () const

Private Attributes
StatsType	stats_

Detailed Description

This class is responsible for storing, and displaying in log messages, statistics about how examples of different sizes (c.f.

GetNnetExampleSize()) were merged into minibatches, and how many examples were left over and discarded.

Definition at line 427 of file nnet-example-utils.h.

Member Typedef Documentation

◆ StatsType

typedef unordered_map<std::pair<int32, size_t>, StatsForExampleSize, PairHasher<int32, size_t> > StatsType

private

Definition at line 464 of file nnet-example-utils.h.

Member Function Documentation

◆ DiscardedExamples()

void DiscardedExamples	(	int32	example_size,
		size_t	structure_hash,
		int32	num_discarded
	)

Users call this function to inform this class that after processing all the data, for examples of original size 'example_size', 'num_discarded' examples could not be put into a minibatch and were discarded.

Definition at line 1065 of file nnet-example-utils.cc.

Referenced by DiscriminativeExampleMerger::Finish(), ChainExampleMerger::Finish(), and ExampleMerger::Finish().

                                                               {
   std::pair<int32, size_t> p(example_size, structure_hash);
   stats_[p].num_discarded += num_discarded;
 }

◆ PrintAggregateStats()

void PrintAggregateStats ( ) const

private

Definition at line 1078 of file nnet-example-utils.cc.

References KALDI_LOG, ExampleMergingStats::StatsForExampleSize::minibatch_to_num_written, and ExampleMergingStats::StatsForExampleSize::num_discarded.

                                                     {
   // First print some aggregate stats.
   int64 num_distinct_egs_types = 0,  // number of distinct types of input egs
                                      // (differing in size or structure).
       total_discarded_egs = 0, // total number of discarded egs.
       total_discarded_egs_size = 0, // total number of discarded egs each multiplied by size
                                     // of that eg
       total_non_discarded_egs = 0,  // total over all minibatches written, of
                                     // minibatch-size, equals number of input egs
                                     // that were not discarded.
       total_non_discarded_egs_size = 0,  // total over all minibatches of size-of-eg
                                      // * minibatch-size.
       num_minibatches = 0,  // total number of minibatches
       num_distinct_minibatch_types = 0;  // total number of combination of
                                          // (type-of-eg, number of distinct
                                          // minibatch-sizes for that eg-type)-
                                          // reflects the number of time we have
                                          // to compile.
 
   StatsType::const_iterator eg_iter = stats_.begin(), eg_end = stats_.end();
 
   for (; eg_iter != eg_end; ++eg_iter) {
     int32 eg_size = eg_iter->first.first;
     const StatsForExampleSize &stats = eg_iter->second;
     num_distinct_egs_types++;
     total_discarded_egs += stats.num_discarded;
     total_discarded_egs_size += stats.num_discarded * eg_size;
 
     unordered_map<int32, int32>::const_iterator
         mb_iter = stats.minibatch_to_num_written.begin(),
         mb_end = stats.minibatch_to_num_written.end();
     for (; mb_iter != mb_end; ++mb_iter) {
       int32 mb_size = mb_iter->first,
           num_written = mb_iter->second;
       num_distinct_minibatch_types++;
       num_minibatches += num_written;
       total_non_discarded_egs += num_written * mb_size;
       total_non_discarded_egs_size += num_written * mb_size * eg_size;
     }
   }
   // the averages are written as integers- we don't really need more precision
   // than that.
   int64 total_input_egs = total_discarded_egs + total_non_discarded_egs,
       total_input_egs_size =
       total_discarded_egs_size + total_non_discarded_egs_size;
 
   float avg_input_egs_size = total_input_egs_size * 1.0 / total_input_egs;
   float percent_discarded = total_discarded_egs * 100.0 / total_input_egs;
   // note: by minibatch size we mean the number of egs per minibatch, it
   // does not take note of the size of the input egs.
   float avg_minibatch_size = total_non_discarded_egs * 1.0 / num_minibatches;
 
   std::ostringstream os;
   os << std::setprecision(4);
   os << "Processed " << total_input_egs
      << " egs of avg. size " << avg_input_egs_size
      << " into " << num_minibatches << " minibatches, discarding "
      << percent_discarded <<  "% of egs.  Avg minibatch size was "
      << avg_minibatch_size << ", #distinct types of egs/minibatches "
      << "was " << num_distinct_egs_types << "/"
      << num_distinct_minibatch_types;
   KALDI_LOG << os.str();
 }

◆ PrintSpecificStats()

void PrintSpecificStats ( ) const

private

Definition at line 1142 of file nnet-example-utils.cc.

References KALDI_LOG, ExampleMergingStats::StatsForExampleSize::minibatch_to_num_written, and ExampleMergingStats::StatsForExampleSize::num_discarded.

                                                    {
   KALDI_LOG << "Merged specific eg types as follows [format: <eg-size1>="
       "{<mb-size1>-><num-minibatches1>,<mbsize2>-><num-minibatches2>.../d=<num-discarded>}"
       ",<egs-size2>={...},... (note,egs-size == number of input "
       "frames including context).";
   std::ostringstream os;
 
   // copy from unordered map to map to get sorting, for consistent output.
   typedef std::map<std::pair<int32, size_t>, StatsForExampleSize> SortedMapType;
 
   SortedMapType stats;
   stats.insert(stats_.begin(), stats_.end());
   SortedMapType::const_iterator eg_iter = stats.begin(), eg_end = stats.end();
   for (; eg_iter != eg_end; ++eg_iter) {
     int32 eg_size = eg_iter->first.first;
     if (eg_iter != stats.begin())
       os << ",";
     os << eg_size << "={";
     const StatsForExampleSize &stats = eg_iter->second;
     unordered_map<int32, int32>::const_iterator
         mb_iter = stats.minibatch_to_num_written.begin(),
         mb_end =  stats.minibatch_to_num_written.end();
     for (; mb_iter != mb_end; ++mb_iter) {
       int32 mb_size = mb_iter->first,
           num_written = mb_iter->second;
       if (mb_iter != stats.minibatch_to_num_written.begin())
         os << ",";
       os << mb_size << "->" << num_written;
     }
     os << ",d=" << stats.num_discarded << "}";
   }
   KALDI_LOG << os.str();
 }

◆ PrintStats()

void PrintStats ( ) const

Calling this will cause a log message with information about the examples to be printed.

Definition at line 1073 of file nnet-example-utils.cc.

Referenced by DiscriminativeExampleMerger::Finish(), ChainExampleMerger::Finish(), and ExampleMerger::Finish().

                                            {
   PrintSpecificStats();
   PrintAggregateStats();
 }

◆ WroteExample()

void WroteExample	(	int32	example_size,
		size_t	structure_hash,
		int32	minibatch_size
	)

Users call this function to inform this class that one minibatch has been written aggregating 'minibatch_size' separate examples of original size 'example_size' (e.g.

as determined by GetNnetExampleSize(), but the caller does that. The 'structure_hash' is provided so that this class can distinguish between egs that have the same size but different structure. In the extremely unlikely eventuality that there is a hash collision, it will cause misleading stats to be printed out.

Definition at line 1051 of file nnet-example-utils.cc.

Referenced by DiscriminativeExampleMerger::WriteMinibatch(), ChainExampleMerger::WriteMinibatch(), and ExampleMerger::WriteMinibatch().

                                                           {
   std::pair<int32, size_t> p(example_size, structure_hash);
 
 
   unordered_map<int32, int32> &h = stats_[p].minibatch_to_num_written;
   unordered_map<int32, int32>::iterator iter = h.find(minibatch_size);
   if (iter == h.end())
     h[minibatch_size] = 1;
   else
     iter->second += 1;
 }

Member Data Documentation

◆ stats_

StatsType stats_

private

Definition at line 468 of file nnet-example-utils.h.

The documentation for this class was generated from the following files:

nnet3/nnet-example-utils.h
nnet3/nnet-example-utils.cc

Classes

Public Member Functions

Private Types

Private Member Functions

Private Attributes

Detailed Description

Member Typedef Documentation

◆ StatsType

Member Function Documentation

◆ DiscardedExamples()

◆ PrintAggregateStats()

◆ PrintSpecificStats()

◆ PrintStats()

◆ WroteExample()

Member Data Documentation

◆ stats_