#include <arpa-lm-compiler.h>

Inheritance diagram for ArpaLmCompiler:

Collaboration diagram for ArpaLmCompiler:

Public Member Functions
	ArpaLmCompiler (const ArpaParseOptions &options, int sub_eps, fst::SymbolTable *symbols)

	~ArpaLmCompiler ()

const fst::StdVectorFst &	Fst () const

fst::StdVectorFst *	MutableFst ()

Public Member Functions inherited from ArpaFileParser
	ArpaFileParser (const ArpaParseOptions &options, fst::SymbolTable *symbols)
	Constructs the parser with the given options and optional symbol table. More...

virtual	~ArpaFileParser ()

void	Read (std::istream &is)
	Read ARPA LM file from a stream. More...

const ArpaParseOptions &	Options () const
	Parser options. More...

Protected Member Functions
virtual void	HeaderAvailable ()
	Override function called to signal that ARPA header with the expected number of n-grams has been read, and ngram_counts() is now valid. More...

virtual void	ConsumeNGram (const NGram &ngram)
	Pure override that must be implemented to process current n-gram. More...

virtual void	ReadComplete ()
	Override function called after the last n-gram has been consumed. More...

Protected Member Functions inherited from ArpaFileParser
virtual void	ReadStarted ()
	Override called before reading starts. More...

const fst::SymbolTable *	Symbols () const
	Read-only access to symbol table. Not owned, do not make public. More...

int32	LineNumber () const
	Inside ConsumeNGram(), provides the current line number. More...

std::string	LineReference () const
	Inside ConsumeNGram(), returns a formatted reference to the line being compiled, to print out as part of diagnostics. More...

bool	ShouldWarn ()
	Increments warning count, and returns true if a warning should be printed or false if the count has exceeded the set maximum. More...

const std::vector< int32 > &	NgramCounts () const
	N-gram counts. Valid from the point when HeaderAvailable() is called. More...

Private Member Functions
void	RemoveRedundantStates ()

void	Check () const

Private Attributes
int	sub_eps_

ArpaLmCompilerImplInterface *	impl_

fst::StdVectorFst	fst_

Friends
template<class HistKey >
class	ArpaLmCompilerImpl

Detailed Description

Definition at line 32 of file arpa-lm-compiler.h.

Constructor & Destructor Documentation

◆ ArpaLmCompiler()

ArpaLmCompiler	(	const ArpaParseOptions &	options,
		int	sub_eps,
		fst::SymbolTable *	symbols
	)

inline

Definition at line 34 of file arpa-lm-compiler.h.

References ArpaLmCompiler::~ArpaLmCompiler().

       : ArpaFileParser(options, symbols),
         sub_eps_(sub_eps), impl_(NULL) {
   }

◆ ~ArpaLmCompiler()

~ArpaLmCompiler ( )

Definition at line 279 of file arpa-lm-compiler.cc.

Referenced by ArpaLmCompiler::ArpaLmCompiler().

                                 {
   if (impl_ != NULL)
     delete impl_;
 }

Member Function Documentation

◆ Check()

void Check ( ) const

private

Definition at line 363 of file arpa-lm-compiler.cc.

References ArpaLmCompilerImpl< HistKey >::fst_, and KALDI_ERR.

Referenced by ArpaLmCompiler::MutableFst().

                                  {
   if (fst_.Start() == fst::kNoStateId) {
     KALDI_ERR << "Arpa file did not contain the beginning-of-sentence symbol "
               << Symbols()->Find(Options().bos_symbol) << ".";
   }
 }

◆ ConsumeNGram()

void ConsumeNGram ( const NGram & )

protectedvirtual

Pure override that must be implemented to process current n-gram.

The n-grams are sent in the file order, which guarantees that all (k-1)-grams are processed before the first k-gram is.

Implements ArpaFileParser.

Definition at line 306 of file arpa-lm-compiler.cc.

References rnnlm::i, KALDI_WARN, and NGram::words.

Referenced by ArpaLmCompiler::MutableFst().

                                                     {
   // <s> is invalid in tails, </s> in heads of an n-gram.
   for (int i = 0; i < ngram.words.size(); ++i) {
     if ((i > 0 && ngram.words[i] == Options().bos_symbol) ||
         (i + 1 < ngram.words.size()
          && ngram.words[i] == Options().eos_symbol)) {
       if (ShouldWarn())
         KALDI_WARN << LineReference()
                    << " skipped: n-gram has invalid BOS/EOS placement";
       return;
     }
   }
 
   bool is_highest = ngram.words.size() == NgramCounts().size();
   impl_->ConsumeNGram(ngram, is_highest);
 }

◆ Fst()

const fst::StdVectorFst& Fst ( ) const

inline

Definition at line 41 of file arpa-lm-compiler.h.

References ArpaLmCompiler::fst_.

Referenced by kaldi::CoverageTest(), and kaldi::ScoringTest().

41 { return fst_; }

kaldi::ArpaLmCompiler::fst_

fst::StdVectorFst fst_

Definition: arpa-lm-compiler.h:59

◆ HeaderAvailable()

void HeaderAvailable ( )

protectedvirtual

Override function called to signal that ARPA header with the expected number of n-grams has been read, and ngram_counts() is now valid.

Reimplemented from ArpaFileParser.

Definition at line 284 of file arpa-lm-compiler.cc.

References ArpaLmCompilerImpl< HistKey >::fst_, ArpaParseOptions::kAddToSymbols, KALDI_ASSERT, KALDI_LOG, and ArpaLmCompilerImpl< HistKey >::sub_eps_.

Referenced by ArpaLmCompiler::MutableFst().

                                      {
   KALDI_ASSERT(impl_ == NULL);
   // Use optimized implementation if the grammar is 4-gram or less, and the
   // maximum attained symbol id will fit into the optimized range.
   int64 max_symbol = 0;
   if (Symbols() != NULL)
     max_symbol = Symbols()->AvailableKey() - 1;
   // If augmenting the symbol table, assume the worst case when all words in
   // the model being read are novel.
   if (Options().oov_handling == ArpaParseOptions::kAddToSymbols)
     max_symbol += NgramCounts()[0];
 
   if (NgramCounts().size() <= 4 && max_symbol < OptimizedHistKey::kMaxData) {
     impl_ = new ArpaLmCompilerImpl<OptimizedHistKey>(this, &fst_, sub_eps_);
   } else {
     impl_ = new ArpaLmCompilerImpl<GeneralHistKey>(this, &fst_, sub_eps_);
     KALDI_LOG << "Reverting to slower state tracking because model is large: "
               << NgramCounts().size() << "-gram with symbols up to "
               << max_symbol;
   }
 }

◆ MutableFst()

fst::StdVectorFst* MutableFst ( )

inline

Definition at line 42 of file arpa-lm-compiler.h.

References ArpaLmCompiler::Check(), ArpaLmCompiler::ConsumeNGram(), ArpaLmCompiler::fst_, ArpaLmCompiler::HeaderAvailable(), ArpaLmCompiler::ReadComplete(), and ArpaLmCompiler::RemoveRedundantStates().

Referenced by kaldi::CoverageTest().

42 { return &fst_; }

kaldi::ArpaLmCompiler::fst_

fst::StdVectorFst fst_

Definition: arpa-lm-compiler.h:59

◆ ReadComplete()

void ReadComplete ( )

protectedvirtual

Override function called after the last n-gram has been consumed.

Reimplemented from ArpaFileParser.

Definition at line 370 of file arpa-lm-compiler.cc.

References ArpaLmCompilerImpl< HistKey >::fst_.

Referenced by ArpaLmCompiler::MutableFst().

                                   {
   fst_.SetInputSymbols(Symbols());
   fst_.SetOutputSymbols(Symbols());
   RemoveRedundantStates();
   Check();
 }

◆ RemoveRedundantStates()

void RemoveRedundantStates ( )

private

Definition at line 323 of file arpa-lm-compiler.cc.

References ArpaLmCompilerImpl< HistKey >::fst_, KALDI_LOG, fst::RemoveEpsLocal(), and ArpaLmCompilerImpl< HistKey >::sub_eps_.

Referenced by ArpaLmCompiler::MutableFst().

                                            {
   fst::StdArc::Label backoff_symbol = sub_eps_;
   if (backoff_symbol == 0) {
     // The method of removing redundant states implemented in this function
     // leads to slow determinization of L o G when people use the older style of
     // usage of arpa2fst where the --disambig-symbol option was not specified.
     // The issue seems to be that it creates a non-deterministic FST, while G is
     // supposed to be deterministic.  By 'return'ing below, we just disable this
     // method if people were using an older script.  This method isn't really
     // that consequential anyway, and people will move to the newer-style
     // scripts (see current utils/format_lm.sh), so this isn't much of a
     // problem.
     return;
   }
 
   fst::StdArc::StateId num_states = fst_.NumStates();
 
 
   // replace the #0 symbols on the input of arcs out of redundant states (states
   // that are not final and have only a backoff arc leaving them), with <eps>.
   for (fst::StdArc::StateId state = 0; state < num_states; state++) {
     if (fst_.NumArcs(state) == 1 && fst_.Final(state) == fst::TropicalWeight::Zero()) {
       fst::MutableArcIterator<fst::StdVectorFst> iter(&fst_, state);
       fst::StdArc arc = iter.Value();
       if (arc.ilabel == backoff_symbol) {
         arc.ilabel = 0;
         iter.SetValue(arc);
       }
     }
   }
 
   // we could call fst::RemoveEps, and it would have the same effect in normal
   // cases, where backoff_symbol != 0 and there are no epsilons in unexpected
   // places, but RemoveEpsLocal is a bit safer in case something weird is going
   // on; it guarantees not to blow up the FST.
   fst::RemoveEpsLocal(&fst_);
   KALDI_LOG << "Reduced num-states from " << num_states << " to "
             << fst_.NumStates();
 }

Friends And Related Function Documentation

◆ ArpaLmCompilerImpl

friend class ArpaLmCompilerImpl

friend

Definition at line 60 of file arpa-lm-compiler.h.

Member Data Documentation

◆ fst_

fst::StdVectorFst fst_

private

Definition at line 59 of file arpa-lm-compiler.h.

Referenced by ArpaLmCompiler::Fst(), and ArpaLmCompiler::MutableFst().

◆ impl_

ArpaLmCompilerImplInterface* impl_

private

Definition at line 58 of file arpa-lm-compiler.h.

◆ sub_eps_

int sub_eps_

private

Definition at line 57 of file arpa-lm-compiler.h.

The documentation for this class was generated from the following files:

lm/arpa-lm-compiler.h
lm/arpa-lm-compiler.cc

Public Member Functions

Protected Member Functions

Private Member Functions

Private Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

◆ ArpaLmCompiler()

◆ ~ArpaLmCompiler()

Member Function Documentation

◆ Check()

◆ ConsumeNGram()

◆ Fst()

◆ HeaderAvailable()

◆ MutableFst()

◆ ReadComplete()

◆ RemoveRedundantStates()

Friends And Related Function Documentation

◆ ArpaLmCompilerImpl

Member Data Documentation

◆ fst_

◆ impl_

◆ sub_eps_