EVOLUTION-MANAGER
Edit File: ngram-tokenizers.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: N-gram tokenizers</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" type="text/css" href="R.css" /> </head><body> <table width="100%" summary="page for ngram-tokenizers {tokenizers}"><tr><td>ngram-tokenizers {tokenizers}</td><td style="text-align: right;">R Documentation</td></tr></table> <h2>N-gram tokenizers</h2> <h3>Description</h3> <p>These functions tokenize their inputs into different kinds of n-grams. The input can be a character vector of any length, or a list of character vectors where each character vector in the list has a length of 1. See details for an explanation of what each function does. </p> <h3>Usage</h3> <pre> tokenize_ngrams( x, lowercase = TRUE, n = 3L, n_min = n, stopwords = character(), ngram_delim = " ", simplify = FALSE ) tokenize_skip_ngrams( x, lowercase = TRUE, n_min = 1, n = 3, k = 1, stopwords = character(), simplify = FALSE ) </pre> <h3>Arguments</h3> <table summary="R argblock"> <tr valign="top"><td><code>x</code></td> <td> <p>A character vector or a list of character vectors to be tokenized into n-grams. If <code>x</code> is a character vector, it can be of any length, and each element will be tokenized separately. If <code>x</code> is a list of character vectors, each element of the list should have a length of 1.</p> </td></tr> <tr valign="top"><td><code>lowercase</code></td> <td> <p>Should the tokens be made lower case?</p> </td></tr> <tr valign="top"><td><code>n</code></td> <td> <p>The number of words in the n-gram. This must be an integer greater than or equal to 1.</p> </td></tr> <tr valign="top"><td><code>n_min</code></td> <td> <p>The minimum number of words in the n-gram. This must be an integer greater than or equal to 1, and less than or equal to <code>n</code>.</p> </td></tr> <tr valign="top"><td><code>stopwords</code></td> <td> <p>A character vector of stop words to be excluded from the n-grams.</p> </td></tr> <tr valign="top"><td><code>ngram_delim</code></td> <td> <p>The separator between words in an n-gram.</p> </td></tr> <tr valign="top"><td><code>simplify</code></td> <td> <p><code>FALSE</code> by default so that a consistent value is returned regardless of length of input. If <code>TRUE</code>, then an input with a single element will return a character vector of tokens instead of a list.</p> </td></tr> <tr valign="top"><td><code>k</code></td> <td> <p>For the skip n-gram tokenizer, the maximum skip distance between words. The function will compute all skip n-grams between <code>0</code> and <code>k</code>.</p> </td></tr> </table> <h3>Details</h3> <dl> <dt><code>tokenize_ngrams</code>:</dt><dd><p> Basic shingled n-grams. A contiguous subsequence of <code>n</code> words. This will compute shingled n-grams for every value of between <code>n_min</code> (which must be at least 1) and <code>n</code>. </p> </dd> <dt><code>tokenize_skip_ngrams</code>:</dt><dd><p>Skip n-grams. A subsequence of <code>n</code> words which are at most a gap of <code>k</code> words between them. The skip n-grams will be calculated for all values from <code>0</code> to <code>k</code>. </p> </dd> </dl> <p>These functions will strip all punctuation and normalize all whitespace to a single space character. </p> <h3>Value</h3> <p>A list of character vectors containing the tokens, with one element in the list for each element that was passed as input. If <code>simplify = TRUE</code> and only a single element was passed as input, then the output is a character vector of tokens. </p> <h3>Examples</h3> <pre> song <- paste0("How many roads must a man walk down\n", "Before you call him a man?\n", "How many seas must a white dove sail\n", "Before she sleeps in the sand?\n", "\n", "How many times must the cannonballs fly\n", "Before they're forever banned?\n", "The answer, my friend, is blowin' in the wind.\n", "The answer is blowin' in the wind.\n") tokenize_ngrams(song, n = 4) tokenize_ngrams(song, n = 4, n_min = 1) tokenize_skip_ngrams(song, n = 4, k = 2) </pre> <hr /><div style="text-align: center;">[Package <em>tokenizers</em> version 0.2.3 <a href="00Index.html">Index</a>]</div> </body></html>