GrammarFst Class Reference

GrammarFst is an FST that is 'stitched together' from multiple FSTs, that can recursively incorporate each other. More...

#include <grammar-fst.h>

Collaboration diagram for GrammarFst:

Classes

struct  ExpandedState
 Represents an expanded state in an FstInstance. More...
 
struct  FstInstance
 

Public Types

typedef GrammarFstArc Arc
 
typedef TropicalWeight Weight
 
typedef Arc::StateId StateId
 
typedef StdArc::StateId BaseStateId
 
typedef Arc::Label Label
 

Public Member Functions

 GrammarFst (int32 nonterm_phones_offset, std::shared_ptr< const ConstFst< StdArc > > top_fst, const std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > &ifsts)
 Constructor. More...
 
 GrammarFst (const GrammarFst &other)=default
 Copy constructor. More...
 
 GrammarFst ()
 This constructor should only be used prior to calling Read(). More...
 
void Write (std::ostream &os, bool binary) const
 
void Read (std::istream &os, bool binary)
 
StateId Start () const
 
Weight Final (StateId s) const
 
size_t NumInputEpsilons (StateId s) const
 
std::string Type () const
 
 ~GrammarFst ()
 

Private Member Functions

void InitNonterminalMap ()
 
bool InitEntryArcs (int32 i)
 
void InitInstances ()
 
void Init ()
 
void Destroy ()
 
void InitEntryOrReentryArcs (const ConstFst< StdArc > &fst, int32 entry_state, int32 nonterminal_symbol, std::unordered_map< int32, int32 > *phone_to_arc)
 
int32 GetPhoneSymbolFor (enum NonterminalValues n)
 
void DecodeSymbol (Label label, int32 *nonterminal_symbol, int32 *left_context_phone)
 Decodes an ilabel into a pair (nonterminal, left_context_phone). More...
 
ExpandedStateExpandState (int32 instance_id, BaseStateId state_id)
 
ExpandedStateExpandStateEnd (int32 instance_id, BaseStateId state_id)
 
ExpandedStateExpandStateUserDefined (int32 instance_id, BaseStateId state_id)
 
int32 GetChildInstanceId (int32 instance_id, int32 nonterminal, int32 state)
 
ExpandedStateGetExpandedState (int32 instance_id, BaseStateId state_id)
 Called from the ArcIterator constructor when we encounter an FST state with nonzero final-prob, this function first looks up this state_id in 'expanded_states' member of the corresponding FstInstance, and returns it if already present; otherwise it populates the 'expanded_states' map with something for this state_id and returns the value. More...
 

Static Private Member Functions

static void CombineArcs (const StdArc &leaving_arc, const StdArc &arriving_arc, float cost_correction, StdArc *arc)
 Called while expanding states, this function combines information from two arcs: one leaving one sub-fst and one arriving in another sub-fst. More...
 

Private Attributes

int32 nonterm_phones_offset_
 
std::shared_ptr< const ConstFst< StdArc > > top_fst_
 
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
 
std::unordered_map< int32, int32 > nonterminal_map_
 
std::vector< std::unordered_map< int32, int32 > > entry_arcs_
 
std::vector< FstInstanceinstances_
 

Friends

class ArcIterator< GrammarFst >
 

Detailed Description

GrammarFst is an FST that is 'stitched together' from multiple FSTs, that can recursively incorporate each other.

(This is limited to left-biphone phonetic context). This class does not inherit from fst::Fst and does not support its full interface– only the parts that are necessary for the decoder to work when templated on it.

The basic interface is inspired by OpenFst's 'ReplaceFst' (see its replace.h), except that this handles left-biphone phonetic context, which requires, essentially, having multiple exit-points and entry-points for sub-FSTs that represent nonterminals in the grammar; and multiple return points whenever we invoke a nonterminal. For more information see Support for grammars and graphs with on-the-fly parts. (i.e. ../doc/grammar.dox).

THREAD SAFETY: you can't use this object from multiple threads; you should create lightweight copies of this object using the copy constructor, e.g. `new GrammarFst(this_grammar_fst)`, if you want to decode from multiple threads using the same GrammarFst.

Definition at line 96 of file grammar-fst.h.

Member Typedef Documentation

◆ Arc

typedef GrammarFstArc Arc

Definition at line 98 of file grammar-fst.h.

◆ BaseStateId

typedef StdArc::StateId BaseStateId

Definition at line 107 of file grammar-fst.h.

◆ Label

typedef Arc::Label Label

Definition at line 109 of file grammar-fst.h.

◆ StateId

Definition at line 104 of file grammar-fst.h.

◆ Weight

typedef TropicalWeight Weight

Definition at line 99 of file grammar-fst.h.

Constructor & Destructor Documentation

◆ GrammarFst() [1/3]

GrammarFst ( int32  nonterm_phones_offset,
std::shared_ptr< const ConstFst< StdArc > >  top_fst,
const std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > &  ifsts 
)

Constructor.

This constructor is very lightweight; the only immediate work it does is to iterate over the arcs in the start states of the provided FSTs in order to set up the appropriate entry points.

For simplicity (to avoid templates), we limit the input FSTs to be of type ConstFst<StdArc>; this limitation could be removed later if needed. You can always construct a ConstFst<StdArc> if you have another StdArc-based FST type. If the FST was read from disk, it may already be of type ConstFst, and dynamic_cast might be sufficient to convert the type.

Parameters
[in]nonterm_phones_offsetThe integer id of the symbol "#nonterm_bos" in phones.txt.
[in]top_fstThe top-level FST of the grammar, which will usually invoke the fsts in 'ifsts'. The fsts in 'ifsts' may also invoke each other recursively. Even left-recursion is allowed, although if it is with zero cost, it may blow up when you decode. When an FST invokes another, the invocation point will have sequences of two special symbols which would be decoded as: (#nonterm:foo,p1) (#nonterm_reenter,p2) where p1 and p2 (which may be real phones or #nonterm_bos) represent the phonetic left-context that we enter, and leave, the sub-graph with respectively.
[in]ifstsifsts is a list of pairs (nonterminal-symbol, the HCLG.fst corresponding to that symbol). The nonterminal symbols must be among the user-specified nonterminals in phones.txt, i.e. the things with names like "#nonterm:foo" and "#nonterm:bar" in phones.txt. Also no nonterminal may appear more than once in 'fsts'. ifsts may be empty, even though that doesn't make much sense.

◆ GrammarFst() [2/3]

GrammarFst ( const GrammarFst other)
default

Copy constructor.

Useful because this object is not thread safe so cannot be used by multiple parallel decoder threads, but it is lightweight and can copy it without causing the stored FSTs to be copied.

◆ GrammarFst() [3/3]

GrammarFst ( )
inline

This constructor should only be used prior to calling Read().

Definition at line 154 of file grammar-fst.h.

154 { }

◆ ~GrammarFst()

~GrammarFst ( )

Definition at line 52 of file grammar-fst.cc.

52  {
53  Destroy();
54 }

Member Function Documentation

◆ CombineArcs()

void CombineArcs ( const StdArc leaving_arc,
const StdArc arriving_arc,
float  cost_correction,
StdArc arc 
)
inlinestaticprivate

Called while expanding states, this function combines information from two arcs: one leaving one sub-fst and one arriving in another sub-fst.

Parameters
[in]leaving_arcThe arc leaving the first FST; must have zero olabel. The ilabel will have a nonterminal symbol like #nonterm:foo or #nonterm_end on it, encoded with a phonetic context, but we ignore the ilabel.
[in]arriving_arcThe arc arriving in the second FST. It will have an ilabel consisted of either #nonterm_begin or #nonterm_enter combined with a left-context phone, but we ignore the ilabel.
[in]cost_correctionA correction term that we add to the cost of the arcs. This basically cancels out the "1/num_options" part of the weight that we added in L.fst when we put in all the phonetic-context options. We did that to keep the FST stochastic, so that if we ever pushed the weights, it wouldn't lead to weird effects. This takes out that correction term... things will still sum to one in the appropriate way, because in fact when we cross these FST boundaries we only take one specific phonetic context, rather than all possibilities.
[out]arcThe arc that we output. Will have:
  • weight equal to the product of the input arcs' weights, times a weight constructed from 'cost_correction'.
  • olabel equal to arriving_arc.olabel (leaving_arc's olabel will be zero).
  • ilabel equal to zero (we discard both ilabels, they are not transition-ids but special symbols).
  • nextstate equal to the nextstate of arriving_arc.

Definition at line 199 of file grammar-fst.cc.

References KALDI_ASSERT.

202  {
203  // The following assertion shouldn't fail; we ensured this in
204  // PrepareForGrammarFst(), search for 'olabel_problem'.
205  KALDI_ASSERT(leaving_arc.olabel == 0);
206  // 'leaving_arc' leaves one fst, and 'arriving_arcs', conceptually arrives in
207  // another. This code merges the information of the two arcs to make a
208  // cross-FST arc. The ilabel information is discarded as it was only intended
209  // for the consumption of the GrammarFST code.
210  arc->ilabel = 0;
211  arc->olabel = arriving_arc.olabel;
212  // conceptually, arc->weight =
213  // Times(Times(leaving_arc.weight, arriving_arc.weight), Weight(cost_correction)).
214  // The below might be a bit faster, I hope-- avoiding checking.
215  arc->weight = Weight(cost_correction + leaving_arc.weight.Value() +
216  arriving_arc.weight.Value());
217  arc->nextstate = arriving_arc.nextstate;
218 }
TropicalWeight Weight
Definition: grammar-fst.h:99
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:185

◆ DecodeSymbol()

void DecodeSymbol ( Label  label,
int32 nonterminal_symbol,
int32 left_context_phone 
)
private

Decodes an ilabel into a pair (nonterminal, left_context_phone).

Crashes if something went wrong or ilabel did not represent that (e.g. was less than kNontermBigNumber).

Parameters
[in]theilabel to be decoded. Note: the type 'Label' will in practice be int.
[out]Thenonterminal part of the ilabel after decoding. Will be a value greater than nonterm_phones_offset_.
[out]Theleft-context-phone part of the ilabel after decoding. Will either be a phone index, or the symbol corresponding to #nonterm_bos (meaning no left-context as we are at the beginning of the sequence).

Definition at line 75 of file grammar-fst.cc.

References fst::GetEncodingMultiple(), KALDI_ASSERT, KALDI_ERR, fst::kNontermBigNumber, fst::kNontermBos, and fst::kNontermMediumNumber.

77  {
78  // encoding_multiple will normally equal 1000 (but may be a multiple of 1000
79  // if there are a lot of phones); kNontermBigNumber is 10000000.
80  int32 big_number = static_cast<int32>(kNontermBigNumber),
81  nonterm_phones_offset = nonterm_phones_offset_,
82  encoding_multiple = GetEncodingMultiple(nonterm_phones_offset);
83  // The following assertion should be optimized out as the condition is
84  // statically known.
85  KALDI_ASSERT(big_number % static_cast<int32>(kNontermMediumNumber) == 0);
86 
87  *nonterminal_symbol = (label - big_number) / encoding_multiple;
88  *left_context_phone = label % encoding_multiple;
89  if (*nonterminal_symbol <= nonterm_phones_offset ||
90  *left_context_phone == 0 || *left_context_phone >
91  nonterm_phones_offset + static_cast<int32>(kNontermBos))
92  KALDI_ERR << "Decoding invalid label " << label
93  << ": code error or invalid --nonterm-phones-offset?";
94 
95 }
kaldi::int32 int32
#define KALDI_ERR
Definition: kaldi-error.h:147
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:185
int32 nonterm_phones_offset_
Definition: grammar-fst.h:456
int32 GetEncodingMultiple(int32 nonterm_phones_offset)

◆ Destroy()

void Destroy ( )
private

Definition at line 56 of file grammar-fst.cc.

References GrammarFst::FstInstance::expanded_states, and rnnlm::i.

56  {
57  for (size_t i = 0; i < instances_.size(); i++) {
58  FstInstance &instance = instances_[i];
59  std::unordered_map<BaseStateId, ExpandedState*>::const_iterator
60  iter = instance.expanded_states.begin(),
61  end = instance.expanded_states.end();
62  for (; iter != end; ++iter) {
63  ExpandedState *e = iter->second;
64  delete e;
65  }
66  }
67  top_fst_ = NULL;
68  ifsts_.clear();
69  nonterminal_map_.clear();
70  entry_arcs_.clear();
71  instances_.clear();
72 }
std::vector< std::unordered_map< int32, int32 > > entry_arcs_
Definition: grammar-fst.h:480
std::unordered_map< int32, int32 > nonterminal_map_
Definition: grammar-fst.h:470
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
Definition: grammar-fst.h:466
std::shared_ptr< const ConstFst< StdArc > > top_fst_
Definition: grammar-fst.h:461

◆ ExpandState()

GrammarFst::ExpandedState * ExpandState ( int32  instance_id,
BaseStateId  state_id 
)
private

Definition at line 171 of file grammar-fst.cc.

References fst::GetEncodingMultiple(), KALDI_ASSERT, KALDI_ERR, fst::kNontermBegin, fst::kNontermBigNumber, fst::kNontermEnd, fst::kNontermReenter, and fst::kNontermUserDefined.

172  {
173  int32 big_number = kNontermBigNumber;
174  const ConstFst<StdArc> &fst = *(instances_[instance_id].fst);
175  ArcIterator<ConstFst<StdArc> > aiter(fst, state_id);
176  KALDI_ASSERT(!aiter.Done() && aiter.Value().ilabel > big_number &&
177  "Something is not right; did you call PrepareForGrammarFst()?");
178 
179  const StdArc &arc = aiter.Value();
180  int32 encoding_multiple = GetEncodingMultiple(nonterm_phones_offset_),
181  nonterminal = (arc.ilabel - big_number) / encoding_multiple;
182  if (nonterminal == GetPhoneSymbolFor(kNontermBegin) ||
183  nonterminal == GetPhoneSymbolFor(kNontermReenter)) {
184  KALDI_ERR << "Encountered unexpected type of nonterminal while "
185  "expanding state.";
186  } else if (nonterminal == GetPhoneSymbolFor(kNontermEnd)) {
187  return ExpandStateEnd(instance_id, state_id);
188  } else if (nonterminal >= GetPhoneSymbolFor(kNontermUserDefined)) {
189  return ExpandStateUserDefined(instance_id, state_id);
190  } else {
191  KALDI_ERR << "Encountered unexpected type of nonterminal "
192  << nonterminal << " while expanding state.";
193  }
194  return NULL; // Suppress compiler warning
195 }
For an extended explanation of the framework of which grammar-fsts are a part, please see Support for...
Definition: graph.dox:21
fst::StdArc StdArc
kaldi::int32 int32
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485
int32 GetPhoneSymbolFor(enum NonterminalValues n)
Definition: grammar-fst.h:265
#define KALDI_ERR
Definition: kaldi-error.h:147
ExpandedState * ExpandStateUserDefined(int32 instance_id, BaseStateId state_id)
Definition: grammar-fst.cc:324
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:185
int32 nonterm_phones_offset_
Definition: grammar-fst.h:456
ExpandedState * ExpandStateEnd(int32 instance_id, BaseStateId state_id)
Definition: grammar-fst.cc:220
int32 GetEncodingMultiple(int32 nonterm_phones_offset)

◆ ExpandStateEnd()

GrammarFst::ExpandedState * ExpandStateEnd ( int32  instance_id,
BaseStateId  state_id 
)
private

Definition at line 220 of file grammar-fst.cc.

References GrammarFst::ExpandedState::arcs, GrammarFst::ExpandedState::dest_fst_instance, GrammarFst::FstInstance::fst, GrammarFst::FstInstance::ifst_index, KALDI_ASSERT, KALDI_ERR, fst::kNontermEnd, GrammarFst::FstInstance::parent_instance, and GrammarFst::FstInstance::parent_state.

221  {
222  if (instance_id == 0)
223  KALDI_ERR << "Did not expect #nonterm_end symbol in FST-instance 0.";
224  const FstInstance &instance = instances_[instance_id];
225  int32 parent_instance_id = instance.parent_instance;
226  const ConstFst<StdArc> &fst = *(instance.fst);
227  const FstInstance &parent_instance = instances_[parent_instance_id];
228  const ConstFst<StdArc> &parent_fst = *(parent_instance.fst);
229 
230  ExpandedState *ans = new ExpandedState;
231  ans->dest_fst_instance = parent_instance_id;
232 
233  // parent_aiter is the arc-iterator in the state we return to. We'll Seek()
234  // to a different position 'parent_aiter' for each arc leaving this state.
235  // (actually we expect just one arc to leave this state).
236  ArcIterator<ConstFst<StdArc> > parent_aiter(parent_fst,
237  instance.parent_state);
238 
239  // for explanation of cost_correction, see documentation for CombineArcs().
240  float num_reentry_arcs = instances_[instance_id].parent_reentry_arcs.size(),
241  cost_correction = -log(num_reentry_arcs);
242 
243  ArcIterator<ConstFst<StdArc> > aiter(fst, state_id);
244 
245  for (; !aiter.Done(); aiter.Next()) {
246  const StdArc &leaving_arc = aiter.Value();
247  int32 this_nonterminal, left_context_phone;
248  DecodeSymbol(leaving_arc.ilabel, &this_nonterminal,
249  &left_context_phone);
250  KALDI_ASSERT(this_nonterminal == GetPhoneSymbolFor(kNontermEnd) &&
251  ">1 nonterminals from a state; did you use "
252  "PrepareForGrammarFst()?");
253  std::unordered_map<int32, int32>::const_iterator reentry_iter =
254  instances_[instance_id].parent_reentry_arcs.find(left_context_phone),
255  reentry_end = instances_[instance_id].parent_reentry_arcs.end();
256  if (reentry_iter == reentry_end) {
257  KALDI_ERR << "FST with index " << instance.ifst_index
258  << " ends with left-context-phone " << left_context_phone
259  << " but parent FST does not support that left-context "
260  "at the return point.";
261  }
262  size_t parent_arc_index = static_cast<size_t>(reentry_iter->second);
263  parent_aiter.Seek(parent_arc_index);
264  const StdArc &arriving_arc = parent_aiter.Value();
265  // 'arc' will combine the information on 'leaving_arc' and 'arriving_arc',
266  // except that the ilabel will be set to zero.
267  if (leaving_arc.olabel != 0) {
268  // If the following fails it would maybe indicate you hadn't called
269  // PrepareForGrammarFst(), or there was an error in that, because
270  // we made sure the leaving arc does not have an olabel. Search
271  // in that code for 'olabel_problem' for more details.
272  KALDI_ERR << "Leaving arc has zero olabel.";
273  }
274  StdArc arc;
275  CombineArcs(leaving_arc, arriving_arc, cost_correction, &arc);
276  ans->arcs.push_back(arc);
277  }
278  return ans;
279 }
For an extended explanation of the framework of which grammar-fsts are a part, please see Support for...
Definition: graph.dox:21
fst::StdArc StdArc
kaldi::int32 int32
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485
static void CombineArcs(const StdArc &leaving_arc, const StdArc &arriving_arc, float cost_correction, StdArc *arc)
Called while expanding states, this function combines information from two arcs: one leaving one sub-...
Definition: grammar-fst.cc:199
int32 GetPhoneSymbolFor(enum NonterminalValues n)
Definition: grammar-fst.h:265
#define KALDI_ERR
Definition: kaldi-error.h:147
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:185
void DecodeSymbol(Label label, int32 *nonterminal_symbol, int32 *left_context_phone)
Decodes an ilabel into a pair (nonterminal, left_context_phone).
Definition: grammar-fst.cc:75

◆ ExpandStateUserDefined()

GrammarFst::ExpandedState * ExpandStateUserDefined ( int32  instance_id,
BaseStateId  state_id 
)
private

Definition at line 324 of file grammar-fst.cc.

References GrammarFst::ExpandedState::arcs, GrammarFst::ExpandedState::dest_fst_instance, GrammarFst::FstInstance::fst, GrammarFst::FstInstance::ifst_index, and KALDI_ERR.

325  {
326  const ConstFst<StdArc> &fst = *(instances_[instance_id].fst);
327  ArcIterator<ConstFst<StdArc> > aiter(fst, state_id);
328 
329  ExpandedState *ans = new ExpandedState;
330  int32 dest_fst_instance = -1; // We'll set it in the loop.
331  // and->dest_fst_instance will be set to this.
332 
333  for (; !aiter.Done(); aiter.Next()) {
334  const StdArc &leaving_arc = aiter.Value();
335  int32 nonterminal, left_context_phone;
336  DecodeSymbol(leaving_arc.ilabel, &nonterminal,
337  &left_context_phone);
338  int32 child_instance_id = GetChildInstanceId(instance_id,
339  nonterminal,
340  leaving_arc.nextstate);
341  if (dest_fst_instance < 0) {
342  dest_fst_instance = child_instance_id;
343  } else if (dest_fst_instance != child_instance_id) {
344  KALDI_ERR << "Same state leaves to different FST instances "
345  "(Did you use PrepareForGrammarFst()?)";
346  }
347  const FstInstance &child_instance = instances_[child_instance_id];
348  const ConstFst<StdArc> &child_fst = *(child_instance.fst);
349  int32 child_ifst_index = child_instance.ifst_index;
350  std::unordered_map<int32, int32> &entry_arcs = entry_arcs_[child_ifst_index];
351  if (entry_arcs.empty()) {
352  if (!InitEntryArcs(child_ifst_index)) {
353  // This child-FST was the empty FST. There are no arcs to expand.
354  continue;
355  }
356  }
357  // for explanation of cost_correction, see documentation for CombineArcs().
358  float num_entry_arcs = entry_arcs.size(),
359  cost_correction = -log(num_entry_arcs);
360 
361  // Get the arc-index for the arc leaving the start-state of child FST that
362  // corresponds to this phonetic context.
363  std::unordered_map<int32, int32>::const_iterator entry_iter =
364  entry_arcs.find(left_context_phone);
365  if (entry_iter == entry_arcs.end()) {
366  KALDI_ERR << "FST for nonterminal " << nonterminal
367  << " does not have an entry point for left-context-phone "
368  << left_context_phone;
369  }
370  int32 arc_index = entry_iter->second;
371  ArcIterator<ConstFst<StdArc> > child_aiter(child_fst, child_fst.Start());
372  child_aiter.Seek(arc_index);
373  const StdArc &arriving_arc = child_aiter.Value();
374  StdArc arc;
375  CombineArcs(leaving_arc, arriving_arc, cost_correction, &arc);
376  ans->arcs.push_back(arc);
377  }
378  ans->dest_fst_instance = dest_fst_instance;
379  return ans;
380 }
std::vector< std::unordered_map< int32, int32 > > entry_arcs_
Definition: grammar-fst.h:480
For an extended explanation of the framework of which grammar-fsts are a part, please see Support for...
Definition: graph.dox:21
fst::StdArc StdArc
bool InitEntryArcs(int32 i)
Definition: grammar-fst.cc:113
kaldi::int32 int32
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485
int32 GetChildInstanceId(int32 instance_id, int32 nonterminal, int32 state)
Definition: grammar-fst.cc:281
static void CombineArcs(const StdArc &leaving_arc, const StdArc &arriving_arc, float cost_correction, StdArc *arc)
Called while expanding states, this function combines information from two arcs: one leaving one sub-...
Definition: grammar-fst.cc:199
#define KALDI_ERR
Definition: kaldi-error.h:147
void DecodeSymbol(Label label, int32 *nonterminal_symbol, int32 *left_context_phone)
Decodes an ilabel into a pair (nonterminal, left_context_phone).
Definition: grammar-fst.cc:75

◆ Final()

Weight Final ( StateId  s) const
inline

Definition at line 171 of file grammar-fst.h.

References KALDI_GRAMMAR_FST_SPECIAL_WEIGHT.

Referenced by fst::CopyToVectorFst().

171  {
172  // If the fst-id (top 32 bits of s) is nonzero, this state is not final,
173  // because we need to return to the top-level FST before we can be final.
174  if (s != static_cast<StateId>(static_cast<int32>(s))) {
175  return Weight::Zero();
176  } else {
177  BaseStateId base_state = static_cast<BaseStateId>(s);
178  Weight ans = top_fst_->Final(base_state);
179  if (ans.Value() == KALDI_GRAMMAR_FST_SPECIAL_WEIGHT) {
180  return Weight::Zero();
181  } else {
182  return ans;
183  }
184  }
185  }
TropicalWeight Weight
Definition: grammar-fst.h:99
#define KALDI_GRAMMAR_FST_SPECIAL_WEIGHT
Definition: grammar-fst.h:67
StdArc::StateId BaseStateId
Definition: grammar-fst.h:107
std::shared_ptr< const ConstFst< StdArc > > top_fst_
Definition: grammar-fst.h:461

◆ GetChildInstanceId()

int32 GetChildInstanceId ( int32  instance_id,
int32  nonterminal,
int32  state 
)
inlineprivate

Definition at line 281 of file grammar-fst.cc.

References GrammarFst::FstInstance::fst, GrammarFst::FstInstance::ifst_index, KALDI_ERR, fst::kNontermReenter, GrammarFst::FstInstance::parent_instance, GrammarFst::FstInstance::parent_reentry_arcs, and GrammarFst::FstInstance::parent_state.

282  {
283  int64 encoded_pair = (static_cast<int64>(nonterminal) << 32) + state;
284  // 'new_instance_id' is the instance-id we'd assign if we had to create a new one.
285  // We try to add it at once, to avoid having to do an extra map lookup in case
286  // it wasn't there and we did need to add it.
287  int32 child_instance_id = instances_.size();
288  {
289  std::pair<int64, int32> p(encoded_pair, child_instance_id);
290  std::pair<std::unordered_map<int64, int32>::const_iterator, bool> ans =
291  instances_[instance_id].child_instances.insert(p);
292  if (!ans.second) {
293  // The pair was not inserted, which means the key 'encoded_pair' did exist in the
294  // map. Return the value in the map.
295  child_instance_id = ans.first->second;
296  return child_instance_id;
297  }
298  }
299  // If we reached this point, we did successfully insert 'child_instance_id' into
300  // the map, because the key didn't exist. That means we have to actually create
301  // the instance.
302  instances_.resize(child_instance_id + 1);
303  const FstInstance &parent_instance = instances_[instance_id];
304  FstInstance &child_instance = instances_[child_instance_id];
305 
306  // Work out the ifst_index for this nonterminal.
307  std::unordered_map<int32, int32>::const_iterator iter =
308  nonterminal_map_.find(nonterminal);
309  if (iter == nonterminal_map_.end()) {
310  KALDI_ERR << "Nonterminal " << nonterminal << " was requested, but "
311  "there is no FST for it.";
312  }
313  int32 ifst_index = iter->second;
314  child_instance.ifst_index = ifst_index;
315  child_instance.fst = ifsts_[ifst_index].second.get();
316  child_instance.parent_instance = instance_id;
317  child_instance.parent_state = state;
318  InitEntryOrReentryArcs(*(parent_instance.fst), state,
320  &(child_instance.parent_reentry_arcs));
321  return child_instance_id;
322 }
std::unordered_map< int32, int32 > nonterminal_map_
Definition: grammar-fst.h:470
void InitEntryOrReentryArcs(const ConstFst< StdArc > &fst, int32 entry_state, int32 nonterminal_symbol, std::unordered_map< int32, int32 > *phone_to_arc)
Definition: grammar-fst.cc:133
kaldi::int32 int32
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
Definition: grammar-fst.h:466
int32 GetPhoneSymbolFor(enum NonterminalValues n)
Definition: grammar-fst.h:265
#define KALDI_ERR
Definition: kaldi-error.h:147

◆ GetExpandedState()

ExpandedState* GetExpandedState ( int32  instance_id,
BaseStateId  state_id 
)
inlineprivate

Called from the ArcIterator constructor when we encounter an FST state with nonzero final-prob, this function first looks up this state_id in 'expanded_states' member of the corresponding FstInstance, and returns it if already present; otherwise it populates the 'expanded_states' map with something for this state_id and returns the value.

Definition at line 352 of file grammar-fst.h.

Referenced by ArcIterator< GrammarFst >::ArcIterator().

353  {
354  std::unordered_map<BaseStateId, ExpandedState*> &expanded_states =
355  instances_[instance_id].expanded_states;
356 
357  std::unordered_map<BaseStateId, ExpandedState*>::iterator iter =
358  expanded_states.find(state_id);
359  if (iter != expanded_states.end()) {
360  return iter->second;
361  } else {
362  ExpandedState *ans = ExpandState(instance_id, state_id);
363  // Don't use the reference 'expanded_states'; it could have been
364  // invalidated.
365  instances_[instance_id].expanded_states[state_id] = ans;
366  return ans;
367  }
368  }
ExpandedState * ExpandState(int32 instance_id, BaseStateId state_id)
Definition: grammar-fst.cc:171
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485

◆ GetPhoneSymbolFor()

int32 GetPhoneSymbolFor ( enum NonterminalValues  n)
inlineprivate

Definition at line 265 of file grammar-fst.h.

References rnnlm::n.

265  {
266  return nonterm_phones_offset_ + static_cast<int32>(n);
267  }
kaldi::int32 int32
struct rnnlm::@11::@12 n
int32 nonterm_phones_offset_
Definition: grammar-fst.h:456

◆ Init()

void Init ( )
private

Definition at line 36 of file grammar-fst.cc.

References KALDI_ASSERT.

36  {
39  entry_arcs_.resize(ifsts_.size());
40  if (!ifsts_.empty()) {
41  // We call this mostly so that if something is wrong with the input FSTs, the
42  // problem will be detected sooner rather than later.
43  // There would be no problem if we were to call InitEntryArcs(i)
44  // for all 0 <= i < ifsts_size(), but we choose to call it
45  // lazily on demand, to save startup time if the number of nonterminals
46  // is large.
47  InitEntryArcs(0);
48  }
49  InitInstances();
50 }
std::vector< std::unordered_map< int32, int32 > > entry_arcs_
Definition: grammar-fst.h:480
bool InitEntryArcs(int32 i)
Definition: grammar-fst.cc:113
void InitNonterminalMap()
Definition: grammar-fst.cc:97
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
Definition: grammar-fst.h:466
void InitInstances()
Definition: grammar-fst.cc:124
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:185
int32 nonterm_phones_offset_
Definition: grammar-fst.h:456

◆ InitEntryArcs()

bool InitEntryArcs ( int32  i)
private

Definition at line 113 of file grammar-fst.cc.

References rnnlm::i, KALDI_ASSERT, and fst::kNontermBegin.

113  {
114  KALDI_ASSERT(static_cast<size_t>(i) < ifsts_.size());
115  const ConstFst<StdArc> &fst = *(ifsts_[i].second);
116  if (fst.NumStates() == 0)
117  return false; /* this was the empty FST. */
118  InitEntryOrReentryArcs(fst, fst.Start(),
120  &(entry_arcs_[i]));
121  return true;
122 }
std::vector< std::unordered_map< int32, int32 > > entry_arcs_
Definition: grammar-fst.h:480
void InitEntryOrReentryArcs(const ConstFst< StdArc > &fst, int32 entry_state, int32 nonterminal_symbol, std::unordered_map< int32, int32 > *phone_to_arc)
Definition: grammar-fst.cc:133
For an extended explanation of the framework of which grammar-fsts are a part, please see Support for...
Definition: graph.dox:21
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
Definition: grammar-fst.h:466
int32 GetPhoneSymbolFor(enum NonterminalValues n)
Definition: grammar-fst.h:265
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:185

◆ InitEntryOrReentryArcs()

void InitEntryOrReentryArcs ( const ConstFst< StdArc > &  fst,
int32  entry_state,
int32  nonterminal_symbol,
std::unordered_map< int32, int32 > *  phone_to_arc 
)
private

Definition at line 133 of file grammar-fst.cc.

References KALDI_ERR, and fst::kNontermBigNumber.

137  {
138  phone_to_arc->clear();
139  ArcIterator<ConstFst<StdArc> > aiter(fst, entry_state);
140  int32 arc_index = 0;
141  for (; !aiter.Done(); aiter.Next(), ++arc_index) {
142  const StdArc &arc = aiter.Value();
143  int32 nonterminal, left_context_phone;
144  if (arc.ilabel <= (int32)kNontermBigNumber) {
145  if (entry_state == fst.Start()) {
146  KALDI_ERR << "There is something wrong with the graph; did you forget to "
147  "add #nonterm_begin and #nonterm_end to the non-top-level FSTs "
148  "before compiling?";
149  } else {
150  KALDI_ERR << "There is something wrong with the graph; re-entry state is "
151  "not as anticipated.";
152  }
153  }
154  DecodeSymbol(arc.ilabel, &nonterminal, &left_context_phone);
155  if (nonterminal != expected_nonterminal_symbol) {
156  KALDI_ERR << "Expected arcs from this state to have nonterminal-symbol "
157  << expected_nonterminal_symbol << ", but got "
158  << nonterminal;
159  }
160  std::pair<int32, int32> p(left_context_phone, arc_index);
161  if (!phone_to_arc->insert(p).second) {
162  // If it was not successfully inserted in the phone_to_arc map, it means
163  // there were two arcs with the same left-context phone, which does not
164  // make sense; that's an error, likely a code error (or an error when the
165  // input FSTs were generated).
166  KALDI_ERR << "Two arcs had the same left-context phone.";
167  }
168  }
169 }
For an extended explanation of the framework of which grammar-fsts are a part, please see Support for...
Definition: graph.dox:21
fst::StdArc StdArc
kaldi::int32 int32
#define KALDI_ERR
Definition: kaldi-error.h:147
void DecodeSymbol(Label label, int32 *nonterminal_symbol, int32 *left_context_phone)
Decodes an ilabel into a pair (nonterminal, left_context_phone).
Definition: grammar-fst.cc:75

◆ InitInstances()

void InitInstances ( )
private

Definition at line 124 of file grammar-fst.cc.

References KALDI_ASSERT.

124  {
125  KALDI_ASSERT(instances_.empty());
126  instances_.resize(1);
127  instances_[0].ifst_index = -1;
128  instances_[0].fst = top_fst_.get();
129  instances_[0].parent_instance = -1;
130  instances_[0].parent_state = -1;
131 }
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:185
std::shared_ptr< const ConstFst< StdArc > > top_fst_
Definition: grammar-fst.h:461

◆ InitNonterminalMap()

void InitNonterminalMap ( )
private

Definition at line 97 of file grammar-fst.cc.

References rnnlm::i, KALDI_ERR, and fst::kNontermUserDefined.

97  {
98  nonterminal_map_.clear();
99  for (size_t i = 0; i < ifsts_.size(); i++) {
100  int32 nonterminal = ifsts_[i].first;
101  if (nonterminal_map_.count(nonterminal))
102  KALDI_ERR << "Nonterminal symbol " << nonterminal
103  << " is paired with two FSTs.";
104  if (nonterminal < GetPhoneSymbolFor(kNontermUserDefined))
105  KALDI_ERR << "Nonterminal symbol " << nonterminal
106  << " in input pairs, was expected to be >= "
108  nonterminal_map_[nonterminal] = static_cast<int32>(i);
109  }
110 }
std::unordered_map< int32, int32 > nonterminal_map_
Definition: grammar-fst.h:470
kaldi::int32 int32
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
Definition: grammar-fst.h:466
int32 GetPhoneSymbolFor(enum NonterminalValues n)
Definition: grammar-fst.h:265
#define KALDI_ERR
Definition: kaldi-error.h:147

◆ NumInputEpsilons()

size_t NumInputEpsilons ( StateId  s) const
inline

Definition at line 190 of file grammar-fst.h.

References GrammarFst::FstInstance::fst, and KALDI_GRAMMAR_FST_SPECIAL_WEIGHT.

190  {
191  // Compare with the constructor of ArcIterator.
192  int32 instance_id = s >> 32;
193  BaseStateId base_state = static_cast<int32>(s);
194  const GrammarFst::FstInstance &instance = instances_[instance_id];
195  const ConstFst<StdArc> *base_fst = instance.fst;
196  if (base_fst->Final(base_state).Value() != KALDI_GRAMMAR_FST_SPECIAL_WEIGHT) {
197  return base_fst->NumInputEpsilons(base_state);
198  } else {
199  return 1;
200  }
201  }
kaldi::int32 int32
std::vector< FstInstance > instances_
Definition: grammar-fst.h:485
#define KALDI_GRAMMAR_FST_SPECIAL_WEIGHT
Definition: grammar-fst.h:67
StdArc::StateId BaseStateId
Definition: grammar-fst.h:107

◆ Read()

void Read ( std::istream &  os,
bool  binary 
)

Definition at line 420 of file grammar-fst.cc.

References kaldi::ExpectToken(), rnnlm::i, KALDI_ERR, kaldi::ReadBasicType(), and fst::ReadConstFstFromStream().

420  {
421  using namespace kaldi;
422  if (!binary)
423  KALDI_ERR << "GrammarFst::Read only supports binary mode.";
424  if (top_fst_ != NULL)
425  Destroy();
426  int32 format = 1, num_ifsts;
427  ExpectToken(is, binary, "<GrammarFst>");
428  ReadBasicType(is, binary, &format);
429  if (format != 1)
430  KALDI_ERR << "This version of the code cannot read this GrammarFst, "
431  "update your code.";
432  ReadBasicType(is, binary, &num_ifsts);
434  top_fst_ = std::shared_ptr<const ConstFst<StdArc> >(ReadConstFstFromStream(is));
435  for (int32 i = 0; i < num_ifsts; i++) {
436  int32 nonterminal;
437  ReadBasicType(is, binary, &nonterminal);
438  std::shared_ptr<const ConstFst<StdArc> >
439  this_fst(ReadConstFstFromStream(is));
440  ifsts_.push_back(std::pair<int32, std::shared_ptr<const ConstFst<StdArc> > >(
441  nonterminal, this_fst));
442  }
443  Init();
444 }
This code computes Goodness of Pronunciation (GOP) and extracts phone-level pronunciation feature for...
Definition: chain.dox:20
static ConstFst< StdArc > * ReadConstFstFromStream(std::istream &is)
Definition: grammar-fst.cc:406
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
kaldi::int32 int32
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
Definition: grammar-fst.h:466
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:191
#define KALDI_ERR
Definition: kaldi-error.h:147
std::shared_ptr< const ConstFst< StdArc > > top_fst_
Definition: grammar-fst.h:461
int32 nonterm_phones_offset_
Definition: grammar-fst.h:456

◆ Start()

StateId Start ( ) const
inline

Definition at line 165 of file grammar-fst.h.

Referenced by fst::CopyToVectorFst().

165  {
166  // the top 32 bits of the 64-bit state-id will be zero, because the
167  // top FST instance has instance-id = 0.
168  return static_cast<StateId>(top_fst_->Start());
169  }
std::shared_ptr< const ConstFst< StdArc > > top_fst_
Definition: grammar-fst.h:461
Arc::StateId StateId
Definition: grammar-fst.h:104

◆ Type()

std::string Type ( ) const
inline

Definition at line 203 of file grammar-fst.h.

203 { return "grammar"; }

◆ Write()

void Write ( std::ostream &  os,
bool  binary 
) const

Definition at line 383 of file grammar-fst.cc.

References rnnlm::i, KALDI_ERR, kaldi::WriteBasicType(), and kaldi::WriteToken().

383  {
384  using namespace kaldi;
385  if (!binary)
386  KALDI_ERR << "GrammarFst::Write only supports binary mode.";
387  int32 format = 1,
388  num_ifsts = ifsts_.size();
389  WriteToken(os, binary, "<GrammarFst>");
390  WriteBasicType(os, binary, format);
391  WriteBasicType(os, binary, num_ifsts);
393 
394  std::string stream_name("unknown");
395  FstWriteOptions wopts(stream_name);
396  top_fst_->Write(os, wopts);
397 
398  for (int32 i = 0; i < num_ifsts; i++) {
399  int32 nonterminal = ifsts_[i].first;
400  WriteBasicType(os, binary, nonterminal);
401  ifsts_[i].second->Write(os, wopts);
402  }
403  WriteToken(os, binary, "</GrammarFst>");
404 }
This code computes Goodness of Pronunciation (GOP) and extracts phone-level pronunciation feature for...
Definition: chain.dox:20
kaldi::int32 int32
std::vector< std::pair< int32, std::shared_ptr< const ConstFst< StdArc > > > > ifsts_
Definition: grammar-fst.h:466
#define KALDI_ERR
Definition: kaldi-error.h:147
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
std::shared_ptr< const ConstFst< StdArc > > top_fst_
Definition: grammar-fst.h:461
int32 nonterm_phones_offset_
Definition: grammar-fst.h:456
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34

Friends And Related Function Documentation

◆ ArcIterator< GrammarFst >

friend class ArcIterator< GrammarFst >
friend

Definition at line 208 of file grammar-fst.h.

Member Data Documentation

◆ entry_arcs_

std::vector<std::unordered_map<int32, int32> > entry_arcs_
private

Definition at line 480 of file grammar-fst.h.

◆ ifsts_

std::vector<std::pair<int32, std::shared_ptr<const ConstFst<StdArc> > > > ifsts_
private

Definition at line 466 of file grammar-fst.h.

◆ instances_

std::vector<FstInstance> instances_
private

Definition at line 485 of file grammar-fst.h.

Referenced by ArcIterator< GrammarFst >::ArcIterator().

◆ nonterm_phones_offset_

int32 nonterm_phones_offset_
private

Definition at line 456 of file grammar-fst.h.

◆ nonterminal_map_

std::unordered_map<int32, int32> nonterminal_map_
private

Definition at line 470 of file grammar-fst.h.

◆ top_fst_

std::shared_ptr<const ConstFst<StdArc> > top_fst_
private

Definition at line 461 of file grammar-fst.h.


The documentation for this class was generated from the following files: