ParseJS update: Tokens are now sorted by length and dictionary order.

A new commit was made to the ParseJS repository, it contains an update that will prevent smaller tokens from potentially wrongfully being placed.

This is the bug in question:

parse_string(“Hello, world!”, [“Hell”, “Hello”]);
// Produces: [Symbol(He…


This content originally appeared on DEV Community and was authored by Calin Baenen

A new commit was made to the ParseJS repository, it contains an update that will prevent smaller tokens from potentially wrongfully being placed.

This is the bug in question:

parse_string("Hello, world!", ["Hell", "Hello"]);
// Produces: [Symbol(Hell), 'o', ',', ' ', 'w', ...]

// But, this--
parse_string("Hello, world!", ["Hello", "Hell"]);
// -- produces: [Symbol(Hello), ',', ' ', 'w', 'o', 'r', ...]

Now, in this update, the order won't matter, the longer token is always chosen if it can be.


This content originally appeared on DEV Community and was authored by Calin Baenen


Print Share Comment Cite Upload Translate Updates
APA

Calin Baenen | Sciencx (2021-12-11T21:58:07+00:00) ParseJS update: Tokens are now sorted by length and dictionary order.. Retrieved from https://www.scien.cx/2021/12/11/parsejs-update-tokens-are-now-sorted-by-length-and-dictionary-order/

MLA
" » ParseJS update: Tokens are now sorted by length and dictionary order.." Calin Baenen | Sciencx - Saturday December 11, 2021, https://www.scien.cx/2021/12/11/parsejs-update-tokens-are-now-sorted-by-length-and-dictionary-order/
HARVARD
Calin Baenen | Sciencx Saturday December 11, 2021 » ParseJS update: Tokens are now sorted by length and dictionary order.., viewed ,<https://www.scien.cx/2021/12/11/parsejs-update-tokens-are-now-sorted-by-length-and-dictionary-order/>
VANCOUVER
Calin Baenen | Sciencx - » ParseJS update: Tokens are now sorted by length and dictionary order.. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2021/12/11/parsejs-update-tokens-are-now-sorted-by-length-and-dictionary-order/
CHICAGO
" » ParseJS update: Tokens are now sorted by length and dictionary order.." Calin Baenen | Sciencx - Accessed . https://www.scien.cx/2021/12/11/parsejs-update-tokens-are-now-sorted-by-length-and-dictionary-order/
IEEE
" » ParseJS update: Tokens are now sorted by length and dictionary order.." Calin Baenen | Sciencx [Online]. Available: https://www.scien.cx/2021/12/11/parsejs-update-tokens-are-now-sorted-by-length-and-dictionary-order/. [Accessed: ]
rf:citation
» ParseJS update: Tokens are now sorted by length and dictionary order. | Calin Baenen | Sciencx | https://www.scien.cx/2021/12/11/parsejs-update-tokens-are-now-sorted-by-length-and-dictionary-order/ |

Please log in to upload a file.




There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.