README.Rmd

---
output: github_document
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# piecemaker

<!-- badges: start -->
[![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental)
[![CRAN status](https://www.r-pkg.org/badges/version/piecemaker)](https://CRAN.R-project.org/package=piecemaker)
[![Codecov test coverage](https://codecov.io/gh/macmillancontentscience/piecemaker/branch/main/graph/badge.svg)](https://app.codecov.io/gh/macmillancontentscience/piecemaker?branch=main)
<!-- badges: end -->

Tokenizers break text into pieces that are more usable by machine learning models. 
While writing [wordpiece](https://github.com/macmillancontentscience/wordpiece) and [morphemepiece](https://github.com/macmillancontentscience/morphemepiece), we found that many steps were shared between those packages.
This package provides those shared steps.

## Installation

You can install the released version of piecemaker from [CRAN](https://CRAN.R-project.org) with:

``` r
install.packages("piecemaker")
```

And the development version from [GitHub](https://github.com/) with:

``` r
# install.packages("devtools")
devtools::install_github("macmillancontentscience/piecemaker")
```

## Example

{piecemaker} helps to prepare text for tokenization. 
For example, it can help you clean out strange encoding, whitespace, and special characters.

```{r example}
library(piecemaker)

piece1 <- " This is a    \n\nfa\xE7ile\n\n    example.\n"
# Specify encoding so this example behaves the same on all systems.
Encoding(piece1) <- "latin1"
example_text <- paste(
  piece1,
  "It has the bell character, \a, and the replacement character,",
  intToUtf8(65533)
)
prepare_text(example_text)
prepare_text(example_text, squish_whitespace = FALSE)
prepare_text(example_text, remove_control_characters = FALSE)
prepare_text(example_text, remove_replacement_characters = FALSE)
prepare_text(example_text, remove_diacritics = FALSE)
```

## Code of Conduct

Please note that the piecemaker project is released with a [Contributor Code of Conduct](https://contributor-covenant.org/version/2/0/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.

## Disclaimer

This is not an officially supported Macmillan Learning product.

## Contact information

Questions or comments should be directed to Jonathan Bratt (jonathan.bratt@macmillan.com) and Jon Harmon (jonthegeek@gmail.com).