mirror-github-bspeice/dtparse
1
0
Fork 0
mirror of https://github.com/bspeice/dtparse synced 2026-01-08 21:20:28 -05:00
Code Issues Releases Wiki Activity

Redo the tokenization

Browse Source 
Still has issues with one test case for fuzzy
This commit is contained in:
Bradlee Speice
2018-07-07 23:37:02 -04:00
committed by Bradlee Speice
parent 9135962839
commit e049618fff
9 changed files with 1029 additions and 155 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/lib.rs
View File

@ -87,7 +87,7 @@ type ParseResult<I> = Result<I, ParseError>;
type ParseIResult<I> = Result<I, ParseInternalError>;
pub fn tokenize(parse_string: &str) -> Vec<String> {
let tokenizer = Tokenizer::new(parse_string.to_owned());
let tokenizer = Tokenizer::new(parse_string);
tokenizer.collect()
}