finite state machine tokenizer

• What is this?
• When should I use this?
• Install
• Use
• API
• Types
• Contribute

This package is a tokenization utility. The underlying tokenizer is implemented as a state machine that produces tokens with positional information, as well as any other fields attached when the token was opened. Tokens are turned into events, with the tokenizer calling any defined resolvers in the process.

This package can be used to tokenize a file, Uint8Array, string, or a list containing these values, like command-line arguments. Developers familiar with micromark will find that tokenizers and constructs (objects used to define how to parse text) in this package work similarly.

This package is ESM only.

In Node.js with yarn:

yarn add @flex-development/mark-parser

See Git - Protocols | Yarn  for details regarding installing from Git.

In Deno with esm.sh:

import { tokenize } from 'https://esm.sh/@flex-development/mark-parser'

In browsers with esm.sh:

<script type="module">
  import { tokenize } from 'https://esm.sh/@flex-development/mark-parser'
</script>

TODO: use

TODO: api

Please refer to the source code for documentation.

This package is fully typed with TypeScript.

See CONTRIBUTING.md.

This project has a code of conduct. By interacting with this repository, organization, or community you agree to abide by its terms.