EVOLUTION-MANAGER
Edit File: basic-tokenizers.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Basic tokenizers</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" type="text/css" href="R.css" /> </head><body> <table width="100%" summary="page for basic-tokenizers {tokenizers}"><tr><td>basic-tokenizers {tokenizers}</td><td style="text-align: right;">R Documentation</td></tr></table> <h2>Basic tokenizers</h2> <h3>Description</h3> <p>These functions perform basic tokenization into words, sentences, paragraphs, lines, and characters. The functions can be piped into one another to create at most two levels of tokenization. For instance, one might split a text into paragraphs and then word tokens, or into sentences and then word tokens. </p> <h3>Usage</h3> <pre> tokenize_characters( x, lowercase = TRUE, strip_non_alphanum = TRUE, simplify = FALSE ) tokenize_words( x, lowercase = TRUE, stopwords = NULL, strip_punct = TRUE, strip_numeric = FALSE, simplify = FALSE ) tokenize_sentences(x, lowercase = FALSE, strip_punct = FALSE, simplify = FALSE) tokenize_lines(x, simplify = FALSE) tokenize_paragraphs(x, paragraph_break = "\n\n", simplify = FALSE) tokenize_regex(x, pattern = "\\s+", simplify = FALSE) tokenize_tweets( x, lowercase = TRUE, stopwords = NULL, strip_punct = TRUE, strip_url = FALSE, simplify = FALSE ) </pre> <h3>Arguments</h3> <table summary="R argblock"> <tr valign="top"><td><code>x</code></td> <td> <p>A character vector or a list of character vectors to be tokenized. If <code>x</code> is a character vector, it can be of any length, and each element will be tokenized separately. If <code>x</code> is a list of character vectors, where each element of the list should have a length of 1.</p> </td></tr> <tr valign="top"><td><code>lowercase</code></td> <td> <p>Should the tokens be made lower case? The default value varies by tokenizer; it is only <code>TRUE</code> by default for the tokenizers that you are likely to use last.</p> </td></tr> <tr valign="top"><td><code>strip_non_alphanum</code></td> <td> <p>Should punctuation and white space be stripped?</p> </td></tr> <tr valign="top"><td><code>simplify</code></td> <td> <p><code>FALSE</code> by default so that a consistent value is returned regardless of length of input. If <code>TRUE</code>, then an input with a single element will return a character vector of tokens instead of a list.</p> </td></tr> <tr valign="top"><td><code>stopwords</code></td> <td> <p>A character vector of stop words to be excluded.</p> </td></tr> <tr valign="top"><td><code>strip_punct</code></td> <td> <p>Should punctuation be stripped?</p> </td></tr> <tr valign="top"><td><code>strip_numeric</code></td> <td> <p>Should numbers be stripped?</p> </td></tr> <tr valign="top"><td><code>paragraph_break</code></td> <td> <p>A string identifying the boundary between two paragraphs.</p> </td></tr> <tr valign="top"><td><code>pattern</code></td> <td> <p>A regular expression that defines the split.</p> </td></tr> <tr valign="top"><td><code>strip_url</code></td> <td> <p>Should URLs (starting with <code>http(s)</code>) be preserved intact, or removed entirely?</p> </td></tr> </table> <h3>Value</h3> <p>A list of character vectors containing the tokens, with one element in the list for each element that was passed as input. If <code>simplify = TRUE</code> and only a single element was passed as input, then the output is a character vector of tokens. </p> <h3>Examples</h3> <pre> song <- paste0("How many roads must a man walk down\n", "Before you call him a man?\n", "How many seas must a white dove sail\n", "Before she sleeps in the sand?\n", "\n", "How many times must the cannonballs fly\n", "Before they're forever banned?\n", "The answer, my friend, is blowin' in the wind.\n", "The answer is blowin' in the wind.\n") tokenize_words(song) tokenize_words(song, strip_punct = FALSE) tokenize_sentences(song) tokenize_paragraphs(song) tokenize_lines(song) tokenize_characters(song) tokenize_tweets("@rOpenSci and #rstats see: https://cran.r-project.org", strip_punct = TRUE) tokenize_tweets("@rOpenSci and #rstats see: https://cran.r-project.org", strip_punct = FALSE) </pre> <hr /><div style="text-align: center;">[Package <em>tokenizers</em> version 0.2.3 <a href="00Index.html">Index</a>]</div> </body></html>