All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
io-funcs.h
Go to the documentation of this file.
1 // base/io-funcs.h
2 
3 // Copyright 2009-2011 Microsoft Corporation; Saarland University;
4 // Jan Silovsky; Yanmin Qian
5 // 2016 Xiaohui Zhang
6 
7 // See ../../COPYING for clarification regarding multiple authors
8 //
9 // Licensed under the Apache License, Version 2.0 (the "License");
10 // you may not use this file except in compliance with the License.
11 // You may obtain a copy of the License at
12 
13 // http://www.apache.org/licenses/LICENSE-2.0
14 
15 // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16 // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
17 // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
18 // MERCHANTABLITY OR NON-INFRINGEMENT.
19 // See the Apache 2 License for the specific language governing permissions and
20 // limitations under the License.
21 
22 #ifndef KALDI_BASE_IO_FUNCS_H_
23 #define KALDI_BASE_IO_FUNCS_H_
24 
25 // This header only contains some relatively low-level I/O functions.
26 // The full Kaldi I/O declarations are in ../util/kaldi-io.h
27 // and ../util/kaldi-table.h
28 // They were put in util/ in order to avoid making the Matrix library
29 // dependent on them.
30 
31 #include <cctype>
32 #include <vector>
33 #include <string>
34 #include "base/kaldi-common.h"
35 
36 namespace kaldi {
37 
38 
39 
40 /*
41  This comment describes the Kaldi approach to I/O. All objects can be written
42  and read in two modes: binary and text. In addition we want to make the I/O
43  work if we redefine the typedef "BaseFloat" between floats and doubles.
44  We also want to have control over whitespace in text mode without affecting
45  the meaning of the file, for pretty-printing purposes.
46 
47  Errors are handled by throwing an exception (std::runtime_error).
48 
49  For integer and floating-point types (and boolean values):
50 
51  WriteBasicType(std::ostream &, bool binary, const T&);
52  ReadBasicType(std::istream &, bool binary, T*);
53 
54  and we expect these functions to be defined in such a way that they work when
55  the type T changes between float and double, so you can read float into double
56  and vice versa]. Note that for efficiency and space-saving reasons, the Vector
57  and Matrix classes do not use these functions [but they preserve the type
58  interchangeability in their own way]
59 
60  For a class (or struct) C:
61  class C {
62  ..
63  Write(std::ostream &, bool binary, [possibly extra optional args for specific classes]) const;
64  Read(std::istream &, bool binary, [possibly extra optional args for specific classes]);
65  ..
66  }
67  NOTE: The only actual optional args we used are the "add" arguments in
68  Vector/Matrix classes, which specify whether we should sum the data already
69  in the class with the data being read.
70 
71  For types which are typedef's involving stl classes, I/O is as follows:
72  typedef std::vector<std::pair<A, B> > MyTypedefName;
73 
74  The user should define something like:
75 
76  WriteMyTypedefName(std::ostream &, bool binary, const MyTypedefName &t);
77  ReadMyTypedefName(std::ostream &, bool binary, MyTypedefName *t);
78 
79  The user would have to write these functions.
80 
81  For a type std::vector<T>:
82 
83  void WriteIntegerVector(std::ostream &os, bool binary, const std::vector<T> &v);
84  void ReadIntegerVector(std::istream &is, bool binary, std::vector<T> *v);
85 
86  For other types, e.g. vectors of pairs, the user should create a routine of the
87  type WriteMyTypedefName. This is to avoid introducing confusing templated functions;
88  we could easily create templated functions to handle most of these cases but they
89  would have to share the same name.
90 
91  It also often happens that the user needs to write/read special tokens as part
92  of a file. These might be class headers, or separators/identifiers in the class.
93  We provide special functions for manipulating these. These special tokens must
94  be nonempty and must not contain any whitespace.
95 
96  void WriteToken(std::ostream &os, bool binary, const char*);
97  void WriteToken(std::ostream &os, bool binary, const std::string & token);
98  int Peek(std::istream &is, bool binary);
99  void ReadToken(std::istream &is, bool binary, std::string *str);
100  void PeekToken(std::istream &is, bool binary, std::string *str);
101 
102  WriteToken writes the token and one space (whether in binary or text mode).
103 
104  Peek returns the first character of the next token, by consuming whitespace
105  (in text mode) and then returning the peek() character. It returns -1 at EOF;
106  it doesn't throw. It's useful if a class can have various forms based on
107  typedefs and virtual classes, and wants to know which version to read.
108 
109  ReadToken allow the caller to obtain the next token. PeekToken works just
110  like ReadToken, but seeks back to the beginning of the token. A subsequent
111  call to ReadToken will read the same token again. This is useful when
112  different object types are written to the same file; using PeekToken one can
113  decide which of the objects to read.
114 
115  There is currently no special functionality for writing/reading strings (where the strings
116  contain data rather than "special tokens" that are whitespace-free and nonempty). This is
117  because Kaldi is structured in such a way that strings don't appear, except as OpenFst symbol
118  table entries (and these have their own format).
119 
120 
121  NOTE: you should not call ReadIntegerType and WriteIntegerType with types,
122  such as int and size_t, that are machine-independent -- at least not
123  if you want your file formats to port between machines. Use int32 and
124  int64 where necessary. There is no way to detect this using compile-time
125  assertions because C++ only keeps track of the internal representation of
126  the type.
127 */
128 
131 
132 
135 template<class T> void WriteBasicType(std::ostream &os, bool binary, T t);
136 
139 template<class T> void ReadBasicType(std::istream &is, bool binary, T *t);
140 
141 
142 // Declare specialization for bool.
143 template<>
144 void WriteBasicType<bool>(std::ostream &os, bool binary, bool b);
145 
146 template <>
147 void ReadBasicType<bool>(std::istream &is, bool binary, bool *b);
148 
149 // Declare specializations for float and double.
150 template<>
151 void WriteBasicType<float>(std::ostream &os, bool binary, float f);
152 
153 template<>
154 void WriteBasicType<double>(std::ostream &os, bool binary, double f);
155 
156 template<>
157 void ReadBasicType<float>(std::istream &is, bool binary, float *f);
158 
159 template<>
160 void ReadBasicType<double>(std::istream &is, bool binary, double *f);
161 
162 // Define ReadBasicType that accepts an "add" parameter to add to
163 // the destination. Caution: if used in Read functions, be careful
164 // to initialize the parameters concerned to zero in the default
165 // constructor.
166 template<class T>
167 inline void ReadBasicType(std::istream &is, bool binary, T *t, bool add) {
168  if (!add) {
169  ReadBasicType(is, binary, t);
170  } else {
171  T tmp = T(0);
172  ReadBasicType(is, binary, &tmp);
173  *t += tmp;
174  }
175 }
176 
178 template<class T> inline void WriteIntegerVector(std::ostream &os, bool binary,
179  const std::vector<T> &v);
180 
182 template<class T> inline void ReadIntegerVector(std::istream &is, bool binary,
183  std::vector<T> *v);
184 
186 template<class T>
187 inline void WriteIntegerPairVector(std::ostream &os, bool binary,
188  const std::vector<std::pair<T, T> > &v);
189 
191 template<class T>
192 inline void ReadIntegerPairVector(std::istream &is, bool binary,
193  std::vector<std::pair<T, T> > *v);
194 
197 void WriteToken(std::ostream &os, bool binary, const char *token);
198 void WriteToken(std::ostream &os, bool binary, const std::string & token);
199 
202 int Peek(std::istream &is, bool binary);
203 
205 void ReadToken(std::istream &is, bool binary, std::string *token);
206 
211 int PeekToken(std::istream &is, bool binary);
212 
215 void ExpectToken(std::istream &is, bool binary, const char *token);
216 void ExpectToken(std::istream &is, bool binary, const std::string & token);
217 
221 void ExpectPretty(std::istream &is, bool binary, const char *token);
222 void ExpectPretty(std::istream &is, bool binary, const std::string & token);
223 
225 
226 
230 inline void InitKaldiOutputStream(std::ostream &os, bool binary);
231 
235 inline bool InitKaldiInputStream(std::istream &is, bool *binary);
236 
237 } // end namespace kaldi.
238 
239 #include "base/io-funcs-inl.h"
240 
241 #endif // KALDI_BASE_IO_FUNCS_H_
Relabels neural network egs with the read pdf-id alignments.
Definition: chain.dox:20
void WriteBasicType< float >(std::ostream &os, bool binary, float f)
Definition: io-funcs.cc:51
bool InitKaldiInputStream(std::istream &is, bool *binary)
Initialize an opened stream for reading by detecting the binary header and.
Definition: io-funcs-inl.h:306
void WriteIntegerPairVector(std::ostream &os, bool binary, const std::vector< std::pair< T, T > > &v)
Function for writing STL vectors of pairs of integer types.
Definition: io-funcs-inl.h:93
void ExpectPretty(std::istream &is, bool binary, const char *token)
ExpectPretty attempts to read the text in "token", but only in non-binary mode.
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void WriteBasicType< bool >(std::ostream &os, bool binary, bool b)
Definition: io-funcs.cc:26
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
void ReadIntegerPairVector(std::istream &is, bool binary, std::vector< std::pair< T, T > > *v)
Function for reading STL vector of pairs of integer types.
Definition: io-funcs-inl.h:131
int Peek(std::istream &is, bool binary)
Peek consumes whitespace (if binary == false) and then returns the peek() value of the stream...
Definition: io-funcs.cc:145
void WriteBasicType< double >(std::ostream &os, bool binary, double f)
Definition: io-funcs.cc:62
void ReadIntegerVector(std::istream &is, bool binary, std::vector< T > *v)
Function for reading STL vector of integer types.
Definition: io-funcs-inl.h:232
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
int PeekToken(std::istream &is, bool binary)
PeekToken will return the first character of the next token, or -1 if end of file.
Definition: io-funcs.cc:170
void ReadBasicType< bool >(std::istream &is, bool binary, bool *b)
Definition: io-funcs.cc:34
void WriteIntegerVector(std::ostream &os, bool binary, const std::vector< T > &v)
Function for writing STL vectors of integer types.
Definition: io-funcs-inl.h:198
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
void InitKaldiOutputStream(std::ostream &os, bool binary)
InitKaldiOutputStream initializes an opened stream for writing by writing an optional binary header a...
Definition: io-funcs-inl.h:291
void ReadBasicType< double >(std::istream &is, bool binary, double *d)
Definition: io-funcs.cc:98
void ReadBasicType< float >(std::istream &is, bool binary, float *f)
Definition: io-funcs.cc:73