GrammarFst Class Reference

GrammarFst is an FST that is 'stitched together' from multiple FSTs, that can recursively incorporate each other. More...

#include <grammar-fst.h>

Collaboration diagram for GrammarFst:

Classes
struct	ExpandedState
	Represents an expanded state in an FstInstance. More...
struct	FstInstance

Public Types
typedef GrammarFstArc	Arc
typedef TropicalWeight	Weight
typedef Arc::StateId	StateId
typedef StdArc::StateId	BaseStateId
typedef Arc::Label	Label

Public Member Functions
	GrammarFst (int32 nonterm_phones_offset, std::shared_ptr< const ConstFst< StdArc > > top_fst, const std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > &ifsts)
	Constructor. More...
	GrammarFst (const GrammarFst &other)=default
	Copy constructor. More...
	GrammarFst ()
	This constructor should only be used prior to calling Read(). More...
void	Write (std::ostream &os, bool binary) const
void	Read (std::istream &os, bool binary)
StateId	Start () const
Weight	Final (StateId s) const
size_t	NumInputEpsilons (StateId s) const
std::string	Type () const
	~GrammarFst ()

Private Member Functions
void	InitNonterminalMap ()
bool	InitEntryArcs (int32 i)
void	InitInstances ()
void	Init ()
void	Destroy ()
void	InitEntryOrReentryArcs (const ConstFst< StdArc > &fst, int32 entry_state, int32 nonterminal_symbol, std::unordered_map< int32, int32 > *phone_to_arc)
int32	GetPhoneSymbolFor (enum NonterminalValues n)
void	DecodeSymbol (Label label, int32 *nonterminal_symbol, int32 *left_context_phone)
	Decodes an ilabel into a pair (nonterminal, left_context_phone). More...
ExpandedState *	ExpandState (int32 instance_id, BaseStateId state_id)
ExpandedState *	ExpandStateEnd (int32 instance_id, BaseStateId state_id)
ExpandedState *	ExpandStateUserDefined (int32 instance_id, BaseStateId state_id)
int32	GetChildInstanceId (int32 instance_id, int32 nonterminal, int32 state)
ExpandedState *	GetExpandedState (int32 instance_id, BaseStateId state_id)
	Called from the ArcIterator constructor when we encounter an FST state with nonzero final-prob, this function first looks up this state_id in 'expanded_states' member of the corresponding FstInstance, and returns it if already present; otherwise it populates the 'expanded_states' map with something for this state_id and returns the value. More...

Static Private Member Functions
static void	CombineArcs (const StdArc &leaving_arc, const StdArc &arriving_arc, float cost_correction, StdArc *arc)
	Called while expanding states, this function combines information from two arcs: one leaving one sub-fst and one arriving in another sub-fst. More...

Private Attributes
int32	nonterm_phones_offset_
std::shared_ptr< const ConstFst< StdArc > >	top_fst_
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > >	ifsts_
std::unordered_map< int32, int32 >	nonterminal_map_
std::vector< std::unordered_map< int32, int32 > >	entry_arcs_
std::vector< FstInstance >	instances_

Friends
class	ArcIterator< GrammarFst >

Detailed Description

GrammarFst is an FST that is 'stitched together' from multiple FSTs, that can recursively incorporate each other.

(This is limited to left-biphone phonetic context). This class does not inherit from fst::Fst and does not support its full interface– only the parts that are necessary for the decoder to work when templated on it.

The basic interface is inspired by OpenFst's 'ReplaceFst' (see its replace.h), except that this handles left-biphone phonetic context, which requires, essentially, having multiple exit-points and entry-points for sub-FSTs that represent nonterminals in the grammar; and multiple return points whenever we invoke a nonterminal. For more information see Support for grammars and graphs with on-the-fly parts. (i.e. ../doc/grammar.dox).

THREAD SAFETY: you can't use this object from multiple threads; you should create lightweight copies of this object using the copy constructor, e.g. `new GrammarFst(this_grammar_fst)`, if you want to decode from multiple threads using the same GrammarFst.

Definition at line 96 of file grammar-fst.h.

Member Typedef Documentation

Arc

typedef GrammarFstArc Arc

Definition at line 98 of file grammar-fst.h.

BaseStateId

typedef StdArc::StateId BaseStateId

Definition at line 107 of file grammar-fst.h.

Label

typedef Arc::Label Label

Definition at line 109 of file grammar-fst.h.

StateId

typedef Arc::StateId StateId

Definition at line 104 of file grammar-fst.h.

Weight

typedef TropicalWeight Weight

Definition at line 99 of file grammar-fst.h.

Constructor & Destructor Documentation

GrammarFst() [1/3]

GrammarFst	(	int32	nonterm_phones_offset,
		std::shared_ptr< const ConstFst< StdArc > >	top_fst,
		const std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > &	ifsts
	)

Constructor.

This constructor is very lightweight; the only immediate work it does is to iterate over the arcs in the start states of the provided FSTs in order to set up the appropriate entry points.

For simplicity (to avoid templates), we limit the input FSTs to be of type ConstFst<StdArc>; this limitation could be removed later if needed. You can always construct a ConstFst<StdArc> if you have another StdArc-based FST type. If the FST was read from disk, it may already be of type ConstFst, and dynamic_cast might be sufficient to convert the type.

Parameters

[in]	nonterm_phones_offset	The integer id of the symbol "#nonterm_bos" in phones.txt.
[in]	top_fst	The top-level FST of the grammar, which will usually invoke the fsts in 'ifsts'. The fsts in 'ifsts' may also invoke each other recursively. Even left-recursion is allowed, although if it is with zero cost, it may blow up when you decode. When an FST invokes another, the invocation point will have sequences of two special symbols which would be decoded as: (#nonterm:foo,p1) (#nonterm_reenter,p2) where p1 and p2 (which may be real phones or #nonterm_bos) represent the phonetic left-context that we enter, and leave, the sub-graph with respectively.
[in]	ifsts	ifsts is a list of pairs (nonterminal-symbol, the HCLG.fst corresponding to that symbol). The nonterminal symbols must be among the user-specified nonterminals in phones.txt, i.e. the things with names like "#nonterm:foo" and "#nonterm:bar" in phones.txt. Also no nonterminal may appear more than once in 'fsts'. ifsts may be empty, even though that doesn't make much sense.

GrammarFst() [2/3]

GrammarFst ( const GrammarFst &  other )

default

Copy constructor.

Useful because this object is not thread safe so cannot be used by multiple parallel decoder threads, but it is lightweight and can copy it without causing the stored FSTs to be copied.

GrammarFst() [3/3]

GrammarFst ( )

inline

This constructor should only be used prior to calling Read().

Definition at line 154 of file grammar-fst.h.

{ }

~GrammarFst()

~GrammarFst

(

)

Definition at line 52 of file grammar-fst.cc.

                        {
  Destroy();
}

Member Function Documentation

CombineArcs()

void CombineArcs ( const StdArc &  leaving_arc, const StdArc &  arriving_arc, float  cost_correction, StdArc *  arc  )

inlinestaticprivate

Called while expanding states, this function combines information from two arcs: one leaving one sub-fst and one arriving in another sub-fst.

Parameters

[in]	leaving_arc	The arc leaving the first FST; must have zero olabel. The ilabel will have a nonterminal symbol like #nonterm:foo or #nonterm_end on it, encoded with a phonetic context, but we ignore the ilabel.
[in]	arriving_arc	The arc arriving in the second FST. It will have an ilabel consisted of either #nonterm_begin or #nonterm_enter combined with a left-context phone, but we ignore the ilabel.
[in]	cost_correction	A correction term that we add to the cost of the arcs. This basically cancels out the "1/num_options" part of the weight that we added in L.fst when we put in all the phonetic-context options. We did that to keep the FST stochastic, so that if we ever pushed the weights, it wouldn't lead to weird effects. This takes out that correction term... things will still sum to one in the appropriate way, because in fact when we cross these FST boundaries we only take one specific phonetic context, rather than all possibilities.
[out]	arc	The arc that we output. Will have: weight equal to the product of the input arcs' weights, times a weight constructed from 'cost_correction'. olabel equal to arriving_arc.olabel (leaving_arc's olabel will be zero). ilabel equal to zero (we discard both ilabels, they are not transition-ids but special symbols). nextstate equal to the nextstate of arriving_arc.

Definition at line 199 of file grammar-fst.cc.

References KALDI_ASSERT.

                                          {
  // The following assertion shouldn't fail; we ensured this in
  // PrepareForGrammarFst(), search for 'olabel_problem'.
  KALDI_ASSERT(leaving_arc.olabel == 0);
  // 'leaving_arc' leaves one fst, and 'arriving_arcs', conceptually arrives in
  // another.  This code merges the information of the two arcs to make a
  // cross-FST arc.  The ilabel information is discarded as it was only intended
  // for the consumption of the GrammarFST code.
  arc->ilabel = 0;
  arc->olabel = arriving_arc.olabel;
  // conceptually, arc->weight =
  //  Times(Times(leaving_arc.weight, arriving_arc.weight), Weight(cost_correction)).
  // The below might be a bit faster, I hope-- avoiding checking.
  arc->weight = Weight(cost_correction + leaving_arc.weight.Value() +
                       arriving_arc.weight.Value());
  arc->nextstate = arriving_arc.nextstate;
}

DecodeSymbol()

void DecodeSymbol ( Label  label, int32 *  nonterminal_symbol, int32 *  left_context_phone  )

private

Decodes an ilabel into a pair (nonterminal, left_context_phone).

Crashes if something went wrong or ilabel did not represent that (e.g. was less than kNontermBigNumber).

Parameters

[in]	the	ilabel to be decoded. Note: the type 'Label' will in practice be int.
[out]	The	nonterminal part of the ilabel after decoding. Will be a value greater than nonterm_phones_offset_.
[out]	The	left-context-phone part of the ilabel after decoding. Will either be a phone index, or the symbol corresponding to #nonterm_bos (meaning no left-context as we are at the beginning of the sequence).

Definition at line 75 of file grammar-fst.cc.

References fst::GetEncodingMultiple(), KALDI_ASSERT, KALDI_ERR, fst::kNontermBigNumber, fst::kNontermBos, and fst::kNontermMediumNumber.

                                                         {
  // encoding_multiple will normally equal 1000 (but may be a multiple of 1000
  // if there are a lot of phones); kNontermBigNumber is 10000000.
  int32 big_number = static_cast<int32>(kNontermBigNumber),
      nonterm_phones_offset = nonterm_phones_offset_,
      encoding_multiple = GetEncodingMultiple(nonterm_phones_offset);
  // The following assertion should be optimized out as the condition is
  // statically known.
  KALDI_ASSERT(big_number % static_cast<int32>(kNontermMediumNumber) == 0);

  *nonterminal_symbol = (label - big_number) / encoding_multiple;
  *left_context_phone = label % encoding_multiple;
  if (*nonterminal_symbol <= nonterm_phones_offset ||
      *left_context_phone == 0 || *left_context_phone >
      nonterm_phones_offset + static_cast<int32>(kNontermBos))
    KALDI_ERR << "Decoding invalid label " << label
              << ": code error or invalid --nonterm-phones-offset?";

}

Destroy()

void Destroy ( )

private

Definition at line 56 of file grammar-fst.cc.

References GrammarFst::FstInstance::expanded_states, and rnnlm::i.

                         {
  for (size_t i = 0; i < instances_.size(); i++) {
    FstInstance &instance = instances_[i];
    std::unordered_map<BaseStateId, ExpandedState*>::const_iterator
        iter = instance.expanded_states.begin(),
        end = instance.expanded_states.end();
    for (; iter != end; ++iter) {
      ExpandedState *e = iter->second;
      delete e;
    }
  }
  top_fst_ = NULL;
  ifsts_.clear();
  nonterminal_map_.clear();
  entry_arcs_.clear();
  instances_.clear();
}

ExpandState()

GrammarFst::ExpandedState * ExpandState ( int32  instance_id, BaseStateId  state_id  )

private

Definition at line 171 of file grammar-fst.cc.

References fst::GetEncodingMultiple(), KALDI_ASSERT, KALDI_ERR, fst::kNontermBegin, fst::kNontermBigNumber, fst::kNontermEnd, fst::kNontermReenter, and fst::kNontermUserDefined.

                                             {
  int32 big_number = kNontermBigNumber;
  const ConstFst<StdArc> &fst = *(instances_[instance_id].fst);
  ArcIterator<ConstFst<StdArc> > aiter(fst, state_id);
  KALDI_ASSERT(!aiter.Done() && aiter.Value().ilabel > big_number &&
               "Something is not right; did you call PrepareForGrammarFst()?");

  const StdArc &arc = aiter.Value();
  int32 encoding_multiple = GetEncodingMultiple(nonterm_phones_offset_),
      nonterminal = (arc.ilabel - big_number) / encoding_multiple;
  if (nonterminal == GetPhoneSymbolFor(kNontermBegin) ||
      nonterminal == GetPhoneSymbolFor(kNontermReenter)) {
    KALDI_ERR << "Encountered unexpected type of nonterminal while "
        "expanding state.";
  } else if (nonterminal == GetPhoneSymbolFor(kNontermEnd)) {
    return ExpandStateEnd(instance_id, state_id);
  } else if (nonterminal >= GetPhoneSymbolFor(kNontermUserDefined)) {
    return ExpandStateUserDefined(instance_id, state_id);
  } else {
    KALDI_ERR << "Encountered unexpected type of nonterminal "
              << nonterminal << " while expanding state.";
  }
  return NULL;  // Suppress compiler warning
}

ExpandStateEnd()

GrammarFst::ExpandedState * ExpandStateEnd ( int32  instance_id, BaseStateId  state_id  )

private

Definition at line 220 of file grammar-fst.cc.

References GrammarFst::ExpandedState::arcs, GrammarFst::ExpandedState::dest_fst_instance, GrammarFst::FstInstance::fst, GrammarFst::FstInstance::ifst_index, KALDI_ASSERT, KALDI_ERR, fst::kNontermEnd, GrammarFst::FstInstance::parent_instance, and GrammarFst::FstInstance::parent_state.

                                             {
  if (instance_id == 0)
    KALDI_ERR << "Did not expect #nonterm_end symbol in FST-instance 0.";
  const FstInstance &instance = instances_[instance_id];
  int32 parent_instance_id = instance.parent_instance;
  const ConstFst<StdArc> &fst = *(instance.fst);
  const FstInstance &parent_instance = instances_[parent_instance_id];
  const ConstFst<StdArc> &parent_fst = *(parent_instance.fst);

  ExpandedState *ans = new ExpandedState;
  ans->dest_fst_instance = parent_instance_id;

  // parent_aiter is the arc-iterator in the state we return to.  We'll Seek()
  // to a different position 'parent_aiter' for each arc leaving this state.
  // (actually we expect just one arc to leave this state).
  ArcIterator<ConstFst<StdArc> > parent_aiter(parent_fst,
                                              instance.parent_state);

  // for explanation of cost_correction, see documentation for CombineArcs().
  float num_reentry_arcs = instances_[instance_id].parent_reentry_arcs.size(),
      cost_correction = -log(num_reentry_arcs);

  ArcIterator<ConstFst<StdArc> > aiter(fst, state_id);

  for (; !aiter.Done(); aiter.Next()) {
    const StdArc &leaving_arc = aiter.Value();
    int32 this_nonterminal, left_context_phone;
    DecodeSymbol(leaving_arc.ilabel, &this_nonterminal,
                 &left_context_phone);
    KALDI_ASSERT(this_nonterminal == GetPhoneSymbolFor(kNontermEnd) &&
                 ">1 nonterminals from a state; did you use "
                 "PrepareForGrammarFst()?");
    std::unordered_map<int32, int32>::const_iterator reentry_iter =
        instances_[instance_id].parent_reentry_arcs.find(left_context_phone),
        reentry_end = instances_[instance_id].parent_reentry_arcs.end();
    if (reentry_iter == reentry_end) {
      KALDI_ERR << "FST with index " << instance.ifst_index
                << " ends with left-context-phone " << left_context_phone
                << " but parent FST does not support that left-context "
          "at the return point.";
    }
    size_t parent_arc_index = static_cast<size_t>(reentry_iter->second);
    parent_aiter.Seek(parent_arc_index);
    const StdArc &arriving_arc = parent_aiter.Value();
    // 'arc' will combine the information on 'leaving_arc' and 'arriving_arc',
    // except that the ilabel will be set to zero.
    if (leaving_arc.olabel != 0) {
      // If the following fails it would maybe indicate you hadn't called
      // PrepareForGrammarFst(), or there was an error in that, because
      // we made sure the leaving arc does not have an olabel.  Search
      // in that code for 'olabel_problem' for more details.
      KALDI_ERR << "Leaving arc has zero olabel.";
    }
    StdArc arc;
    CombineArcs(leaving_arc, arriving_arc, cost_correction, &arc);
    ans->arcs.push_back(arc);
  }
  return ans;
}

ExpandStateUserDefined()

GrammarFst::ExpandedState * ExpandStateUserDefined ( int32  instance_id, BaseStateId  state_id  )

private

Definition at line 324 of file grammar-fst.cc.

References GrammarFst::ExpandedState::arcs, GrammarFst::ExpandedState::dest_fst_instance, GrammarFst::FstInstance::fst, GrammarFst::FstInstance::ifst_index, and KALDI_ERR.

                                             {
  const ConstFst<StdArc> &fst = *(instances_[instance_id].fst);
  ArcIterator<ConstFst<StdArc> > aiter(fst, state_id);

  ExpandedState *ans = new ExpandedState;
  int32 dest_fst_instance = -1;  // We'll set it in the loop.
                                 // and->dest_fst_instance will be set to this.

  for (; !aiter.Done(); aiter.Next()) {
    const StdArc &leaving_arc = aiter.Value();
    int32 nonterminal, left_context_phone;
    DecodeSymbol(leaving_arc.ilabel, &nonterminal,
                 &left_context_phone);
    int32 child_instance_id = GetChildInstanceId(instance_id,
                                                 nonterminal,
                                                 leaving_arc.nextstate);
    if (dest_fst_instance < 0) {
      dest_fst_instance = child_instance_id;
    } else if (dest_fst_instance != child_instance_id) {
      KALDI_ERR << "Same state leaves to different FST instances "
          "(Did you use PrepareForGrammarFst()?)";
    }
    const FstInstance &child_instance = instances_[child_instance_id];
    const ConstFst<StdArc> &child_fst = *(child_instance.fst);
    int32 child_ifst_index = child_instance.ifst_index;
    std::unordered_map<int32, int32> &entry_arcs = entry_arcs_[child_ifst_index];
    if (entry_arcs.empty()) {
      if (!InitEntryArcs(child_ifst_index)) {
        // This child-FST was the empty FST.  There are no arcs to expand.
        continue;
      }
    }
    // for explanation of cost_correction, see documentation for CombineArcs().
    float num_entry_arcs = entry_arcs.size(),
        cost_correction = -log(num_entry_arcs);

    // Get the arc-index for the arc leaving the start-state of child FST that
    // corresponds to this phonetic context.
    std::unordered_map<int32, int32>::const_iterator entry_iter =
        entry_arcs.find(left_context_phone);
    if (entry_iter == entry_arcs.end()) {
      KALDI_ERR << "FST for nonterminal " << nonterminal
                << " does not have an entry point for left-context-phone "
                << left_context_phone;
    }
    int32 arc_index = entry_iter->second;
    ArcIterator<ConstFst<StdArc> > child_aiter(child_fst, child_fst.Start());
    child_aiter.Seek(arc_index);
    const StdArc &arriving_arc = child_aiter.Value();
    StdArc arc;
    CombineArcs(leaving_arc, arriving_arc, cost_correction, &arc);
    ans->arcs.push_back(arc);
  }
  ans->dest_fst_instance = dest_fst_instance;
  return ans;
}

Final()

Weight Final ( StateId  s ) const

inline

Definition at line 171 of file grammar-fst.h.

References KALDI_GRAMMAR_FST_SPECIAL_WEIGHT.

Referenced by fst::CopyToVectorFst().

                                {
    // If the fst-id (top 32 bits of s) is nonzero, this state is not final,
    // because we need to return to the top-level FST before we can be final.
    if (s != static_cast<StateId>(static_cast<int32>(s))) {
      return Weight::Zero();
    } else {
      BaseStateId base_state = static_cast<BaseStateId>(s);
      Weight ans = top_fst_->Final(base_state);
      if (ans.Value() == KALDI_GRAMMAR_FST_SPECIAL_WEIGHT) {
        return Weight::Zero();
      } else {
        return ans;
      }
    }
  }

GetChildInstanceId()

int32 GetChildInstanceId ( int32  instance_id, int32  nonterminal, int32  state  )

inlineprivate

Definition at line 281 of file grammar-fst.cc.

References GrammarFst::FstInstance::fst, GrammarFst::FstInstance::ifst_index, KALDI_ERR, fst::kNontermReenter, GrammarFst::FstInstance::parent_instance, GrammarFst::FstInstance::parent_reentry_arcs, and GrammarFst::FstInstance::parent_state.

                                                  {
  int64 encoded_pair = (static_cast<int64>(nonterminal) << 32) + state;
  // 'new_instance_id' is the instance-id we'd assign if we had to create a new one.
  // We try to add it at once, to avoid having to do an extra map lookup in case
  // it wasn't there and we did need to add it.
  int32 child_instance_id = instances_.size();
  {
    std::pair<int64, int32> p(encoded_pair, child_instance_id);
    std::pair<std::unordered_map<int64, int32>::const_iterator, bool> ans =
        instances_[instance_id].child_instances.insert(p);
    if (!ans.second) {
      // The pair was not inserted, which means the key 'encoded_pair' did exist in the
      // map.  Return the value in the map.
      child_instance_id = ans.first->second;
      return child_instance_id;
    }
  }
  // If we reached this point, we did successfully insert 'child_instance_id' into
  // the map, because the key didn't exist.  That means we have to actually create
  // the instance.
  instances_.resize(child_instance_id + 1);
  const FstInstance &parent_instance = instances_[instance_id];
  FstInstance &child_instance = instances_[child_instance_id];

  // Work out the ifst_index for this nonterminal.
  std::unordered_map<int32, int32>::const_iterator iter =
      nonterminal_map_.find(nonterminal);
  if (iter == nonterminal_map_.end()) {
    KALDI_ERR << "Nonterminal " << nonterminal << " was requested, but "
        "there is no FST for it.";
  }
  int32 ifst_index = iter->second;
  child_instance.ifst_index = ifst_index;
  child_instance.fst = ifsts_[ifst_index].second.get();
  child_instance.parent_instance = instance_id;
  child_instance.parent_state = state;
  InitEntryOrReentryArcs(*(parent_instance.fst), state,
                         GetPhoneSymbolFor(kNontermReenter),
                         &(child_instance.parent_reentry_arcs));
  return child_instance_id;
}

GetExpandedState()

ExpandedState* GetExpandedState ( int32  instance_id, BaseStateId  state_id  )

inlineprivate

Called from the ArcIterator constructor when we encounter an FST state with nonzero final-prob, this function first looks up this state_id in 'expanded_states' member of the corresponding FstInstance, and returns it if already present; otherwise it populates the 'expanded_states' map with something for this state_id and returns the value.

Definition at line 352 of file grammar-fst.h.

Referenced by ArcIterator< GrammarFst >::ArcIterator().

                                                               {
    std::unordered_map<BaseStateId, ExpandedState*> &expanded_states =
        instances_[instance_id].expanded_states;

    std::unordered_map<BaseStateId, ExpandedState*>::iterator iter =
        expanded_states.find(state_id);
    if (iter != expanded_states.end()) {
      return iter->second;
    } else {
      ExpandedState *ans = ExpandState(instance_id, state_id);
      // Don't use the reference 'expanded_states'; it could have been
      // invalidated.
      instances_[instance_id].expanded_states[state_id] = ans;
      return ans;
    }
  }

GetPhoneSymbolFor()

int32 GetPhoneSymbolFor ( enum NonterminalValues  n )

inlineprivate

Definition at line 265 of file grammar-fst.h.

References rnnlm::n.

                                                           {
    return nonterm_phones_offset_ + static_cast<int32>(n);
  }

Init()

void Init ( )

private

Definition at line 36 of file grammar-fst.cc.

References KALDI_ASSERT.

                      {
  KALDI_ASSERT(nonterm_phones_offset_ > 1);
  InitNonterminalMap();
  entry_arcs_.resize(ifsts_.size());
  if (!ifsts_.empty()) {
    // We call this mostly so that if something is wrong with the input FSTs, the
    // problem will be detected sooner rather than later.
    // There would be no problem if we were to call InitEntryArcs(i)
    // for all 0 <= i < ifsts_size(), but we choose to call it
    // lazily on demand, to save startup time if the number of nonterminals
    // is large.
    InitEntryArcs(0);
  }
  InitInstances();
}

InitEntryArcs()

bool InitEntryArcs ( int32  i )

private

Definition at line 113 of file grammar-fst.cc.

References rnnlm::i, KALDI_ASSERT, and fst::kNontermBegin.

                                      {
  KALDI_ASSERT(static_cast<size_t>(i) < ifsts_.size());
  const ConstFst<StdArc> &fst = *(ifsts_[i].second);
  if (fst.NumStates() == 0)
    return false;  /* this was the empty FST. */
  InitEntryOrReentryArcs(fst, fst.Start(),
                         GetPhoneSymbolFor(kNontermBegin),
                         &(entry_arcs_[i]));
  return true;
}

InitEntryOrReentryArcs()

void InitEntryOrReentryArcs ( const ConstFst< StdArc > &  fst, int32  entry_state, int32  nonterminal_symbol, std::unordered_map< int32, int32 > *  phone_to_arc  )

private

Definition at line 133 of file grammar-fst.cc.

References KALDI_ERR, and fst::kNontermBigNumber.

                                                  {
  phone_to_arc->clear();
  ArcIterator<ConstFst<StdArc> > aiter(fst, entry_state);
  int32 arc_index = 0;
  for (; !aiter.Done(); aiter.Next(), ++arc_index) {
    const StdArc &arc = aiter.Value();
    int32 nonterminal, left_context_phone;
    if (arc.ilabel <= (int32)kNontermBigNumber) {
      if (entry_state == fst.Start()) {
        KALDI_ERR << "There is something wrong with the graph; did you forget to "
            "add #nonterm_begin and #nonterm_end to the non-top-level FSTs "
            "before compiling?";
      } else {
        KALDI_ERR << "There is something wrong with the graph; re-entry state is "
            "not as anticipated.";
      }
    }
    DecodeSymbol(arc.ilabel, &nonterminal, &left_context_phone);
    if (nonterminal != expected_nonterminal_symbol) {
      KALDI_ERR << "Expected arcs from this state to have nonterminal-symbol "
                << expected_nonterminal_symbol << ", but got "
                << nonterminal;
    }
    std::pair<int32, int32> p(left_context_phone, arc_index);
    if (!phone_to_arc->insert(p).second) {
      // If it was not successfully inserted in the phone_to_arc map, it means
      // there were two arcs with the same left-context phone, which does not
      // make sense; that's an error, likely a code error (or an error when the
      // input FSTs were generated).
      KALDI_ERR << "Two arcs had the same left-context phone.";
    }
  }
}

InitInstances()

void InitInstances ( )

private

Definition at line 124 of file grammar-fst.cc.

References KALDI_ASSERT.

                               {
  KALDI_ASSERT(instances_.empty());
  instances_.resize(1);
  instances_[0].ifst_index = -1;
  instances_[0].fst = top_fst_.get();
  instances_[0].parent_instance = -1;
  instances_[0].parent_state = -1;
}

InitNonterminalMap()

void InitNonterminalMap ( )

private

Definition at line 97 of file grammar-fst.cc.

References rnnlm::i, KALDI_ERR, and fst::kNontermUserDefined.

                                    {
  nonterminal_map_.clear();
  for (size_t i = 0; i < ifsts_.size(); i++) {
    int32 nonterminal = ifsts_[i].first;
    if (nonterminal_map_.count(nonterminal))
      KALDI_ERR << "Nonterminal symbol " << nonterminal
                << " is paired with two FSTs.";
    if (nonterminal < GetPhoneSymbolFor(kNontermUserDefined))
      KALDI_ERR << "Nonterminal symbol " << nonterminal
                << " in input pairs, was expected to be >= "
                << GetPhoneSymbolFor(kNontermUserDefined);
    nonterminal_map_[nonterminal] = static_cast<int32>(i);
  }
}

NumInputEpsilons()

size_t NumInputEpsilons ( StateId  s ) const

inline

Definition at line 190 of file grammar-fst.h.

References GrammarFst::FstInstance::fst, and KALDI_GRAMMAR_FST_SPECIAL_WEIGHT.

                                                  {
    // Compare with the constructor of ArcIterator.
    int32 instance_id = s >> 32;
    BaseStateId base_state = static_cast<int32>(s);
    const GrammarFst::FstInstance &instance = instances_[instance_id];
    const ConstFst<StdArc> *base_fst = instance.fst;
    if (base_fst->Final(base_state).Value() != KALDI_GRAMMAR_FST_SPECIAL_WEIGHT) {
      return base_fst->NumInputEpsilons(base_state);
    } else {
      return 1;
    }
  }

Read()

void Read	(	std::istream &	os,
		bool	binary
	)

Definition at line 420 of file grammar-fst.cc.

References kaldi::ExpectToken(), rnnlm::i, KALDI_ERR, kaldi::ReadBasicType(), and fst::ReadConstFstFromStream().

                                                 {
  using namespace kaldi;
  if (!binary)
    KALDI_ERR << "GrammarFst::Read only supports binary mode.";
  if (top_fst_ != NULL)
    Destroy();
  int32 format = 1, num_ifsts;
  ExpectToken(is, binary, "<GrammarFst>");
  ReadBasicType(is, binary, &format);
  if (format != 1)
    KALDI_ERR << "This version of the code cannot read this GrammarFst, "
        "update your code.";
  ReadBasicType(is, binary, &num_ifsts);
  ReadBasicType(is, binary, &nonterm_phones_offset_);
  top_fst_ = std::shared_ptr<const ConstFst<StdArc> >(ReadConstFstFromStream(is));
  for (int32 i = 0; i < num_ifsts; i++) {
    int32 nonterminal;
    ReadBasicType(is, binary, &nonterminal);
    std::shared_ptr<const ConstFst<StdArc> >
        this_fst(ReadConstFstFromStream(is));
    ifsts_.push_back(std::pair<int32, std::shared_ptr<const ConstFst<StdArc> > >(
        nonterminal, this_fst));
  }
  Init();
}

Start()

StateId Start ( ) const

inline

Definition at line 165 of file grammar-fst.h.

Referenced by fst::CopyToVectorFst().

                        {
    // the top 32 bits of the 64-bit state-id will be zero, because the
    // top FST instance has instance-id = 0.
    return static_cast<StateId>(top_fst_->Start());
  }

Type()

std::string Type ( ) const

inline

Definition at line 203 of file grammar-fst.h.

{ return "grammar"; }

Write()

void Write	(	std::ostream &	os,
		bool	binary
	)		const

Definition at line 383 of file grammar-fst.cc.

References rnnlm::i, KALDI_ERR, kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                        {
  using namespace kaldi;
  if (!binary)
    KALDI_ERR << "GrammarFst::Write only supports binary mode.";
  int32 format = 1,
      num_ifsts = ifsts_.size();
  WriteToken(os, binary, "<GrammarFst>");
  WriteBasicType(os, binary, format);
  WriteBasicType(os, binary, num_ifsts);
  WriteBasicType(os, binary, nonterm_phones_offset_);

  std::string stream_name("unknown");
  FstWriteOptions wopts(stream_name);
  top_fst_->Write(os, wopts);

  for (int32 i = 0; i < num_ifsts; i++) {
    int32 nonterminal = ifsts_[i].first;
    WriteBasicType(os, binary, nonterminal);
    ifsts_[i].second->Write(os, wopts);
  }
  WriteToken(os, binary, "</GrammarFst>");
}

Friends And Related Function Documentation

ArcIterator< GrammarFst >

friend class ArcIterator< GrammarFst >

friend

Definition at line 208 of file grammar-fst.h.

Member Data Documentation

entry_arcs_

std::vector<std::unordered_map<int32, int32> > entry_arcs_

private

Definition at line 480 of file grammar-fst.h.

ifsts_

std::vector<std::pair<int32, std::shared_ptr<const ConstFst<StdArc> > > > ifsts_

private

Definition at line 466 of file grammar-fst.h.

instances_

std::vector<FstInstance> instances_

private

Definition at line 485 of file grammar-fst.h.

Referenced by ArcIterator< GrammarFst >::ArcIterator().

nonterm_phones_offset_

int32 nonterm_phones_offset_

private

Definition at line 456 of file grammar-fst.h.

nonterminal_map_

std::unordered_map<int32, int32> nonterminal_map_

private

Definition at line 470 of file grammar-fst.h.

top_fst_

std::shared_ptr<const ConstFst<StdArc> > top_fst_

private

Definition at line 461 of file grammar-fst.h.

---

The documentation for this class was generated from the following files:

• decoder/grammar-fst.h
• decoder/grammar-fst.cc