All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
ConfigLine Class Reference

This class is responsible for parsing input like hi-there xx=yyy a=b c empty= f-oo=Append(bar, sss) ba_z=123 bing='a b c' baz="a b c d='a b' e" and giving you access to the fields, in this case. More...

#include <nnet-parse.h>

Collaboration diagram for ConfigLine:

Public Member Functions

bool ParseLine (const std::string &line)
 
bool GetValue (const std::string &key, std::string *value)
 
bool GetValue (const std::string &key, BaseFloat *value)
 
bool GetValue (const std::string &key, int32 *value)
 
bool GetValue (const std::string &key, std::vector< int32 > *value)
 
bool GetValue (const std::string &key, bool *value)
 
bool HasUnusedValues () const
 
std::string UnusedValues () const
 returns e.g. More...
 
const std::string & FirstToken () const
 
const std::string WholeLine ()
 

Private Attributes

std::string whole_line_
 
std::string first_token_
 
std::map< std::string,
std::pair< std::string, bool > > 
data_
 

Detailed Description

This class is responsible for parsing input like hi-there xx=yyy a=b c empty= f-oo=Append(bar, sss) ba_z=123 bing='a b c' baz="a b c d='a b' e" and giving you access to the fields, in this case.

FirstToken() == "hi-there", and key->value pairs:

xx->yyy, a->"b c", empty->"", f-oo->"Append(bar, sss)", ba_z->"123", bing->"a b c", baz->"a b c d='a b' e"

The first token is optional, if the line started with a key-value pair then FirstValue() will be empty.

Note: it can parse value fields with space inside them only if they are free of the '=' character. If values are going to contain the '=' character, you need to quote them with either single or double quotes.

Key values may contain -_a-zA-Z0-9, but must begin with a-zA-Z_.

Definition at line 48 of file nnet-parse.h.

Member Function Documentation

bool GetValue ( const std::string &  key,
std::string *  value 
)

Definition at line 115 of file nnet-parse.cc.

References ConfigLine::data_, and KALDI_ASSERT.

Referenced by PnormComponent::InitFromConfig(), DistributeComponent::InitFromConfig(), DropoutComponent::InitFromConfig(), ElementwiseProductComponent::InitFromConfig(), NormalizeComponent::InitFromConfig(), StatisticsExtractionComponent::InitFromConfig(), TimeHeightConvolutionComponent::InitFromConfig(), StatisticsPoolingComponent::InitFromConfig(), AffineComponent::InitFromConfig(), BackpropTruncationComponent::InitFromConfig(), BlockAffineComponent::InitFromConfig(), NonlinearComponent::InitFromConfig(), RepeatedAffineComponent::InitFromConfig(), ConstantComponent::InitFromConfig(), DropoutMaskComponent::InitFromConfig(), NaturalGradientAffineComponent::InitFromConfig(), FixedAffineComponent::InitFromConfig(), SumGroupComponent::InitFromConfig(), FixedScaleComponent::InitFromConfig(), FixedBiasComponent::InitFromConfig(), SumBlockComponent::InitFromConfig(), ClipGradientComponent::InitFromConfig(), PermuteComponent::InitFromConfig(), PerElementScaleComponent::InitFromConfig(), PerElementOffsetComponent::InitFromConfig(), ConstantFunctionComponent::InitFromConfig(), NaturalGradientPerElementScaleComponent::InitFromConfig(), ConvolutionComponent::InitFromConfig(), LstmNonlinearityComponent::InitFromConfig(), MaxpoolingComponent::InitFromConfig(), BatchNormComponent::InitFromConfig(), CompositeComponent::InitFromConfig(), UpdatableComponent::InitLearningRatesFromConfig(), Nnet::ProcessComponentConfigLine(), Nnet::ProcessComponentNodeConfigLine(), Nnet::ProcessDimRangeNodeConfigLine(), Nnet::ProcessInputNodeConfigLine(), Nnet::ProcessOutputNodeConfigLine(), kaldi::nnet3::ReadEditConfig(), Nnet::RemoveRedundantConfigLines(), and kaldi::nnet3::UnitTestConfigLineParse().

115  {
116  KALDI_ASSERT(value != NULL);
117  std::map<std::string, std::pair<std::string, bool> >::iterator it = data_.begin();
118  for (; it != data_.end(); ++it) {
119  if (it->first == key) {
120  *value = (it->second).first;
121  (it->second).second = true;
122  return true;
123  }
124  }
125  return false;
126 }
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
bool GetValue ( const std::string &  key,
BaseFloat value 
)

Definition at line 128 of file nnet-parse.cc.

References kaldi::ConvertStringToReal(), ConfigLine::data_, and KALDI_ASSERT.

128  {
129  KALDI_ASSERT(value != NULL);
130  std::map<std::string, std::pair<std::string, bool> >::iterator it = data_.begin();
131  for (; it != data_.end(); ++it) {
132  if (it->first == key) {
133  if (!ConvertStringToReal((it->second).first, value))
134  return false;
135  (it->second).second = true;
136  return true;
137  }
138  }
139  return false;
140 }
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
bool ConvertStringToReal(const std::string &str, T *out)
ConvertStringToReal converts a string into either float or double and returns false if there was any ...
Definition: text-utils.cc:238
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
bool GetValue ( const std::string &  key,
int32 *  value 
)

Definition at line 142 of file nnet-parse.cc.

References kaldi::ConvertStringToInteger(), ConfigLine::data_, and KALDI_ASSERT.

142  {
143  KALDI_ASSERT(value != NULL);
144  std::map<std::string, std::pair<std::string, bool> >::iterator it = data_.begin();
145  for (; it != data_.end(); ++it) {
146  if (it->first == key) {
147  if (!ConvertStringToInteger((it->second).first, value))
148  return false;
149  (it->second).second = true;
150  return true;
151  }
152  }
153  return false;
154 }
bool ConvertStringToInteger(const std::string &str, Int *out)
Converts a string into an integer via strtoll and returns false if there was any kind of problem (i...
Definition: text-utils.h:118
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
bool GetValue ( const std::string &  key,
std::vector< int32 > *  value 
)

Definition at line 156 of file nnet-parse.cc.

References ConfigLine::data_, KALDI_ASSERT, and kaldi::SplitStringToIntegers().

156  {
157  KALDI_ASSERT(value != NULL);
158  value->clear();
159  std::map<std::string, std::pair<std::string, bool> >::iterator it = data_.begin();
160  for (; it != data_.end(); ++it) {
161  if (it->first == key) {
162  if (!SplitStringToIntegers((it->second).first, ":,", true, value)) {
163  // KALDI_WARN << "Bad option " << (it->second).first;
164  return false;
165  }
166  (it->second).second = true;
167  return true;
168  }
169  }
170  return false;
171 }
bool SplitStringToIntegers(const std::string &full, const char *delim, bool omit_empty_strings, std::vector< I > *out)
Split a string (e.g.
Definition: text-utils.h:68
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
bool GetValue ( const std::string &  key,
bool *  value 
)

Definition at line 173 of file nnet-parse.cc.

References ConfigLine::data_, and KALDI_ASSERT.

173  {
174  KALDI_ASSERT(value != NULL);
175  std::map<std::string, std::pair<std::string, bool> >::iterator it = data_.begin();
176  for (; it != data_.end(); ++it) {
177  if (it->first == key) {
178  if ((it->second).first.size() == 0) return false;
179  switch (((it->second).first)[0]) {
180  case 'F':
181  case 'f':
182  *value = false;
183  break;
184  case 'T':
185  case 't':
186  *value = true;
187  break;
188  default:
189  return false;
190  }
191  (it->second).second = true;
192  return true;
193  }
194  }
195  return false;
196 }
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
bool HasUnusedValues ( ) const

Definition at line 198 of file nnet-parse.cc.

References ConfigLine::data_.

Referenced by PnormComponent::InitFromConfig(), DistributeComponent::InitFromConfig(), DropoutComponent::InitFromConfig(), ElementwiseProductComponent::InitFromConfig(), NormalizeComponent::InitFromConfig(), StatisticsExtractionComponent::InitFromConfig(), StatisticsPoolingComponent::InitFromConfig(), AffineComponent::InitFromConfig(), BackpropTruncationComponent::InitFromConfig(), BlockAffineComponent::InitFromConfig(), NonlinearComponent::InitFromConfig(), RepeatedAffineComponent::InitFromConfig(), ConstantComponent::InitFromConfig(), NaturalGradientAffineComponent::InitFromConfig(), FixedAffineComponent::InitFromConfig(), SumGroupComponent::InitFromConfig(), FixedScaleComponent::InitFromConfig(), FixedBiasComponent::InitFromConfig(), SumBlockComponent::InitFromConfig(), ClipGradientComponent::InitFromConfig(), PermuteComponent::InitFromConfig(), PerElementScaleComponent::InitFromConfig(), PerElementOffsetComponent::InitFromConfig(), ConstantFunctionComponent::InitFromConfig(), NaturalGradientPerElementScaleComponent::InitFromConfig(), ConvolutionComponent::InitFromConfig(), LstmNonlinearityComponent::InitFromConfig(), MaxpoolingComponent::InitFromConfig(), BatchNormComponent::InitFromConfig(), CompositeComponent::InitFromConfig(), Nnet::ProcessComponentConfigLine(), Nnet::ProcessComponentNodeConfigLine(), Nnet::ProcessDimRangeNodeConfigLine(), Nnet::ProcessInputNodeConfigLine(), Nnet::ProcessOutputNodeConfigLine(), kaldi::nnet3::ReadEditConfig(), and kaldi::nnet3::UnitTestConfigLineParse().

198  {
199  std::map<std::string, std::pair<std::string, bool> >::const_iterator it = data_.begin();
200  for (; it != data_.end(); ++it) {
201  if (!(it->second).second) return true;
202  }
203  return false;
204 }
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
bool ParseLine ( const std::string &  line)

Definition at line 31 of file nnet-parse.cc.

References ConfigLine::data_, ConfigLine::first_token_, kaldi::nnet3::IsValidName(), KALDI_WARN, and ConfigLine::whole_line_.

Referenced by kaldi::nnet3::GenerateRandomSimpleComponent(), CompositeComponent::InitFromConfig(), kaldi::nnet3::ModifyNnetIvectorPeriod(), and kaldi::nnet3::UnitTestConfigLineParse().

31  {
32  data_.clear();
33  whole_line_ = line;
34  if (line.size() == 0) return false; // Empty line
35  size_t pos = 0, size = line.size();
36  while (isspace(line[pos]) && pos < size) pos++;
37  if (pos == size)
38  return false; // whitespace-only line
39  size_t first_token_start_pos = pos;
40  // first get first_token_.
41  while (!isspace(line[pos]) && pos < size) {
42  if (line[pos] == '=') {
43  // If the first block of non-whitespace looks like "foo-bar=...",
44  // then we ignore it: there is no initial token, and FirstToken()
45  // is empty.
46  pos = first_token_start_pos;
47  break;
48  }
49  pos++;
50  }
51  first_token_ = std::string(line, first_token_start_pos, pos - first_token_start_pos);
52  // first_token_ is expected to be either empty or something like
53  // "component-node", which actually is a slightly more restrictive set of
54  // strings than IsValidName() checks for this is a convenient way to check it.
55  if (!first_token_.empty() && !IsValidName(first_token_))
56  return false;
57 
58  while (pos < size) {
59  if (isspace(line[pos])) {
60  pos++;
61  continue;
62  }
63 
64  // OK, at this point we know that we are pointing at nonspace.
65  size_t next_equals_sign = line.find_first_of("=", pos);
66  if (next_equals_sign == pos || next_equals_sign == std::string::npos) {
67  // we're looking for something like 'key=value'. If there is no equals sign,
68  // or it's not preceded by something, it's a parsing failure.
69  return false;
70  }
71  std::string key(line, pos, next_equals_sign - pos);
72  if (!IsValidName(key)) return false;
73 
74  // handle any quotes. we support key='blah blah' or key="foo bar".
75  // no escaping is supported.
76  if (line[next_equals_sign+1] == '\'' || line[next_equals_sign+1] == '"') {
77  char my_quote = line[next_equals_sign+1];
78  size_t next_quote = line.find_first_of(my_quote, next_equals_sign + 2);
79  if (next_quote == std::string::npos) { // no matching quote was found.
80  KALDI_WARN << "No matching quote for " << my_quote << " in config line '"
81  << line << "'";
82  return false;
83  } else {
84  std::string value(line, next_equals_sign + 2,
85  next_quote - next_equals_sign - 2);
86  data_.insert(std::make_pair(key, std::make_pair(value, false)));
87  pos = next_quote + 1;
88  continue;
89  }
90  } else {
91  // we want to be able to parse something like "... input=Offset(a, -1) foo=bar":
92  // in general, config values with spaces in them, even without quoting.
93 
94  size_t next_next_equals_sign = line.find_first_of("=", next_equals_sign + 1),
95  terminating_space = size;
96 
97  if (next_next_equals_sign != std::string::npos) { // found a later equals sign.
98  size_t preceding_space = line.find_last_of(" \t", next_next_equals_sign);
99  if (preceding_space != std::string::npos &&
100  preceding_space > next_equals_sign)
101  terminating_space = preceding_space;
102  }
103  while (isspace(line[terminating_space - 1]) && terminating_space > 0)
104  terminating_space--;
105 
106  std::string value(line, next_equals_sign + 1,
107  terminating_space - (next_equals_sign + 1));
108  data_.insert(std::make_pair(key, std::make_pair(value, false)));
109  pos = terminating_space;
110  }
111  }
112  return true;
113 }
std::string first_token_
Definition: nnet-parse.h:80
std::string whole_line_
Definition: nnet-parse.h:76
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
#define KALDI_WARN
Definition: kaldi-error.h:130
bool IsValidName(const std::string &name)
Returns true if 'name' would be a valid name for a component or node in a Nnet.
Definition: nnet-parse.cc:425
std::string UnusedValues ( ) const

returns e.g.

foo=bar xxx=yyy if foo and xxx were not consumed by one of the GetValue() functions.

Definition at line 206 of file nnet-parse.cc.

References ConfigLine::data_.

Referenced by StatisticsExtractionComponent::InitFromConfig(), StatisticsPoolingComponent::InitFromConfig(), AffineComponent::InitFromConfig(), RepeatedAffineComponent::InitFromConfig(), NaturalGradientAffineComponent::InitFromConfig(), SumBlockComponent::InitFromConfig(), PermuteComponent::InitFromConfig(), PerElementScaleComponent::InitFromConfig(), PerElementOffsetComponent::InitFromConfig(), ConvolutionComponent::InitFromConfig(), LstmNonlinearityComponent::InitFromConfig(), MaxpoolingComponent::InitFromConfig(), BatchNormComponent::InitFromConfig(), CompositeComponent::InitFromConfig(), Nnet::ProcessComponentConfigLine(), Nnet::ProcessComponentNodeConfigLine(), Nnet::ProcessDimRangeNodeConfigLine(), Nnet::ProcessInputNodeConfigLine(), Nnet::ProcessOutputNodeConfigLine(), kaldi::nnet3::ReadEditConfig(), and kaldi::nnet3::UnitTestConfigLineParse().

206  {
207  std::string unused_str;
208  std::map<std::string, std::pair<std::string, bool> >::const_iterator it = data_.begin();
209  for (; it != data_.end(); ++it) {
210  if (!(it->second).second) {
211  if (unused_str == "")
212  unused_str = it->first + "=" + (it->second).first;
213  else
214  unused_str += " " + it->first + "=" + (it->second).first;
215  }
216  }
217  return unused_str;
218 }
std::map< std::string, std::pair< std::string, bool > > data_
Definition: nnet-parse.h:83
const std::string WholeLine ( )
inline

Definition at line 73 of file nnet-parse.h.

References ConfigLine::whole_line_.

Referenced by PnormComponent::InitFromConfig(), DistributeComponent::InitFromConfig(), DropoutComponent::InitFromConfig(), ElementwiseProductComponent::InitFromConfig(), NormalizeComponent::InitFromConfig(), StatisticsExtractionComponent::InitFromConfig(), TimeHeightConvolutionComponent::InitFromConfig(), StatisticsPoolingComponent::InitFromConfig(), AffineComponent::InitFromConfig(), BackpropTruncationComponent::InitFromConfig(), BlockAffineComponent::InitFromConfig(), NonlinearComponent::InitFromConfig(), RepeatedAffineComponent::InitFromConfig(), ConstantComponent::InitFromConfig(), NaturalGradientAffineComponent::InitFromConfig(), FixedAffineComponent::InitFromConfig(), SumGroupComponent::InitFromConfig(), FixedScaleComponent::InitFromConfig(), FixedBiasComponent::InitFromConfig(), ClipGradientComponent::InitFromConfig(), PermuteComponent::InitFromConfig(), ConstantFunctionComponent::InitFromConfig(), NaturalGradientPerElementScaleComponent::InitFromConfig(), ConvolutionComponent::InitFromConfig(), LstmNonlinearityComponent::InitFromConfig(), MaxpoolingComponent::InitFromConfig(), CompositeComponent::InitFromConfig(), UpdatableComponent::InitLearningRatesFromConfig(), Nnet::ProcessComponentConfigLine(), Nnet::ProcessComponentNodeConfigLine(), Nnet::ProcessDimRangeNodeConfigLine(), Nnet::ProcessInputNodeConfigLine(), Nnet::ProcessOutputNodeConfigLine(), kaldi::nnet3::ReadEditConfig(), and Nnet::RemoveRedundantConfigLines().

73 { return whole_line_; }
std::string whole_line_
Definition: nnet-parse.h:76

Member Data Documentation

std::map<std::string, std::pair<std::string, bool> > data_
private
std::string first_token_
private

Definition at line 80 of file nnet-parse.h.

Referenced by ConfigLine::FirstToken(), and ConfigLine::ParseLine().

std::string whole_line_
private

Definition at line 76 of file nnet-parse.h.

Referenced by ConfigLine::ParseLine(), and ConfigLine::WholeLine().


The documentation for this class was generated from the following files: