EVOLUTION-MANAGER

Edit File: fread.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Fast and friendly file finagler</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<link rel="stylesheet" type="text/css" href="R.css" />
</head><body>

<table width="100%" summary="page for fread {data.table}"><tr><td>fread {data.table}</td><td style="text-align: right;">R Documentation</td></tr></table>

<h2> Fast and friendly file finagler </h2>

<h3>Description</h3>

Similar to <code>read.table</code> but faster and more convenient. All controls such as <code>sep</code>, <code>colClasses</code> and <code>nrows</code> are automatically detected.

<code>bit64::integer64</code>, <code><a href="IDateTime.html">IDate</a></code>, and <code><a href="../../base/html/DateTimeClasses.html">POSIXct</a></code> types are also detected and read directly without needing to read as character before converting.

<code>fread</code> is for regular delimited files; i.e., where every row has the same number of columns. In future, secondary separator (<code>sep2</code>) may be specified within each column. Such columns will be read as type <code>list</code> where each cell is itself a vector.

<h3>Usage</h3>

<pre>
fread(input, file, text, cmd, sep="auto", sep2="auto", dec=".", quote="\"",
nrows=Inf, header="auto",
na.strings=getOption("datatable.na.strings","NA"), # due to change to ""; see NEWS
stringsAsFactors=FALSE, verbose=getOption("datatable.verbose", FALSE),
skip="__auto__", select=NULL, drop=NULL, colClasses=NULL,
integer64=getOption("datatable.integer64", "integer64"),
col.names,
check.names=FALSE, encoding="unknown",
strip.white=TRUE, fill=FALSE, blank.lines.skip=FALSE,
key=NULL, index=NULL,
showProgress=getOption("datatable.showProgress", interactive()),
data.table=getOption("datatable.fread.datatable", TRUE),
nThread=getDTthreads(verbose),
logical01=getOption("datatable.logical01", FALSE), # due to change to TRUE; see NEWS
keepLeadingZeros = getOption("datatable.keepLeadingZeros", FALSE),
yaml=FALSE, autostart=NA, tmpdir=tempdir(), tz="UTC"
)
</pre>

<h3>Arguments</h3>

<table summary="R argblock">
<tr valign="top"><td><code>input</code></td>
<td>
 A single character string. The value is inspected and deferred to either <code>file=</code> (if no \n present), <code>text=</code> (if at least one \n is present) or <code>cmd=</code> (if no \n is present, at least one space is present, and it isn't a file name). Exactly one of <code>input=</code>, <code>file=</code>, <code>text=</code>, or <code>cmd=</code> should be used in the same call. 
</td></tr>
<tr valign="top"><td><code>file</code></td>
<td>
 File name in working directory, path to file (passed through <code><a href="../../base/html/path.expand.html">path.expand</a></code> for convenience), or a URL starting http://, file://, etc. Compressed files with extension &lsquo;.gz&rsquo; and &lsquo;.bz2&rsquo; are supported if the <code>R.utils</code> package is installed. 
</td></tr>
<tr valign="top"><td><code>text</code></td>
<td>
 The input data itself as a character vector of one or more lines, for example as returned by <code>readLines()</code>. 
</td></tr>
<tr valign="top"><td><code>cmd</code></td>
<td>
 A shell command that pre-processes the file; e.g. <code>fread(cmd=paste("grep",word,"filename"))</code>. See Details. 
</td></tr>
<tr valign="top"><td><code>sep</code></td>
<td>
 The separator between columns. Defaults to the character in the set <code>[,\t |;:]</code> that separates the sample of rows into the most number of lines with the same number of fields. Use <code>NULL</code> or <code>""</code> to specify no separator; i.e. each line a single character column like <code>base::readLines</code> does.
</td></tr>
<tr valign="top"><td><code>sep2</code></td>
<td>
 The separator within columns. A <code>list</code> column will be returned where each cell is a vector of values. This is much faster using less working memory than <code>strsplit</code> afterwards or similar techniques. For each column <code>sep2</code> can be different and is the first character in the same set above [<code>,\t |;</code>], other than <code>sep</code>, that exists inside each field outside quoted regions in the sample. NB: <code>sep2</code> is not yet implemented. 
</td></tr>
<tr valign="top"><td><code>nrows</code></td>
<td>
 The maximum number of rows to read. Unlike <code>read.table</code>, you do not need to set this to an estimate of the number of rows in the file for better speed because that is already automatically determined by <code>fread</code> almost instantly using the large sample of lines. <code>nrows=0</code> returns the column names and typed empty columns determined by the large sample; useful for a dry run of a large file or to quickly check format consistency of a set of files before starting to read any of them. 
</td></tr>
<tr valign="top"><td><code>header</code></td>
<td>
 Does the first data line contain column names? Defaults according to whether every non-empty field on the first data line is type character. If so, or TRUE is supplied, any empty column names are given a default name. 
</td></tr>
<tr valign="top"><td><code>na.strings</code></td>
<td>
 A character vector of strings which are to be interpreted as <code>NA</code> values. By default, <code>",,"</code> for columns of all types, including type <code>character</code> is read as <code>NA</code> for consistency. <code>,"",</code> is unambiguous and read as an empty string. To read <code>,NA,</code> as <code>NA</code>, set <code>na.strings="NA"</code>. To read <code>,,</code> as blank string <code>""</code>, set <code>na.strings=NULL</code>. When they occur in the file, the strings in <code>na.strings</code> should not appear quoted since that is how the string literal <code>,"NA",</code> is distinguished from <code>,NA,</code>, for example, when <code>na.strings="NA"</code>. 
</td></tr>
<tr valign="top"><td><code>stringsAsFactors</code></td>
<td>
 Convert all character columns to factors? 
</td></tr>
<tr valign="top"><td><code>verbose</code></td>
<td>
 Be chatty and report timings? 
</td></tr>
<tr valign="top"><td><code>skip</code></td>
<td>
 If 0 (default) start on the first line and from there finds the first row with a consistent number of columns. This automatically avoids irregular header information before the column names row. <code>skip&gt;0</code> means ignore the first <code>skip</code> rows manually. <code>skip="string"</code> searches for <code>"string"</code> in the file (e.g. a substring of the column names row) and starts on that line (inspired by read.xls in package gdata). 
</td></tr>
<tr valign="top"><td><code>select</code></td>
<td>
 A vector of column names or numbers to keep, drop the rest. <code>select</code> may specify types too in the same way as <code>colClasses</code>; i.e., a vector of <code>colname=type</code> pairs, or a <code>list</code> of <code>type=col(s)</code> pairs. In all forms of <code>select</code>, the order that the columns are specified determines the order of the columns in the result. 
</td></tr>
<tr valign="top"><td><code>drop</code></td>
<td>
 Vector of column names or numbers to drop, keep the rest. 
</td></tr>
<tr valign="top"><td><code>colClasses</code></td>
<td>
 As in <code><a href="../../utils/html/read.table.html">utils::read.csv</a></code>; i.e., an unnamed vector of types corresponding to the columns in the file, or a named vector specifying types for a subset of the columns by name. The default, <code>NULL</code> means types are inferred from the data in the file. Further, <code>data.table</code> supports a named <code>list</code> of vectors of column names or numbers where the <code>list</code> names are the class names; see examples. The <code>list</code> form makes it easier to set a batch of columns to be a particular class. When column numbers are used in the <code>list</code> form, they refer to the column number in the file not the column number after <code>select</code> or <code>drop</code> has been applied.
If type coercion results in an error, introduces <code>NA</code>s, or would result in loss of accuracy, the coercion attempt is aborted for that column with warning and the column's type is left unchanged. If you really desire data loss (e.g. reading <code>3.14</code> as <code>integer</code>) you have to truncate such columns afterwards yourself explicitly so that this is clear to future readers of your code.

</td></tr>
<tr valign="top"><td><code>integer64</code></td>
<td>
 &quot;integer64&quot; (default) reads columns detected as containing integers larger than 2^31 as type <code>bit64::integer64</code>. Alternatively, <code>"double"|"numeric"</code> reads as <code>utils::read.csv</code> does; i.e., possibly with loss of precision and if so silently. Or, &quot;character&quot;. 
</td></tr>
<tr valign="top"><td><code>dec</code></td>
<td>
 The decimal separator as in <code>utils::read.csv</code>. If not &quot;.&quot; (default) then usually &quot;,&quot;. See details. 
</td></tr>
<tr valign="top"><td><code>col.names</code></td>
<td>
 A vector of optional names for the variables (columns). The default is to use the header column if present or detected, or if not &quot;V&quot; followed by the column number. This is applied after <code>check.names</code> and before <code>key</code> and <code>index</code>. 
</td></tr>
<tr valign="top"><td><code>check.names</code></td>
<td>
default is <code>FALSE</code>. If <code>TRUE</code> then the names of the variables in the <code>data.table</code> are checked to ensure that they are syntactically valid variable names. If necessary they are adjusted (by <code><a href="../../base/html/make.names.html">make.names</a></code>) so that they are, and also to ensure that there are no duplicates.
</td></tr>
<tr valign="top"><td><code>encoding</code></td>
<td>
 default is <code>"unknown"</code>. Other possible options are <code>"UTF-8"</code> and <code>"Latin-1"</code>. Note: it is not used to re-encode the input, rather enables handling of encoded strings in their native encoding. 
</td></tr>
<tr valign="top"><td><code>quote</code></td>
<td>
 By default (<code>"\""</code>), if a field starts with a double quote, <code>fread</code> handles embedded quotes robustly as explained under <code>Details</code>. If it fails, then another attempt is made to read the field as is, i.e., as if quotes are disabled. By setting <code>quote=""</code>, the field is always read as if quotes are disabled. It is not expected to ever need to pass anything other than \&quot;\&quot; to quote; i.e., to turn it off. 
</td></tr>
<tr valign="top"><td><code>strip.white</code></td>
<td>
 default is <code>TRUE</code>. Strips leading and trailing whitespaces of unquoted fields. If <code>FALSE</code>, only header trailing spaces are removed. 
</td></tr>
<tr valign="top"><td><code>fill</code></td>
<td>
logical (default is <code>FALSE</code>). If <code>TRUE</code> then in case the rows have unequal length, blank fields are implicitly filled.
</td></tr>
<tr valign="top"><td><code>blank.lines.skip</code></td>
<td>
<code>logical</code>, default is <code>FALSE</code>. If <code>TRUE</code> blank lines in the input are ignored.
</td></tr>
<tr valign="top"><td><code>key</code></td>
<td>
Character vector of one or more column names which is passed to <code><a href="setkey.html">setkey</a></code>. It may be a single comma separated string such as <code>key="x,y,z"</code>, or a vector of names such as <code>key=c("x","y","z")</code>. Only valid when argument <code>data.table=TRUE</code>. Where applicable, this should refer to column names given in <code>col.names</code>. 
</td></tr>
<tr valign="top"><td><code>index</code></td>
<td>
 Character vector or list of character vectors of one or more column names which is passed to <code><a href="setkey.html">setindexv</a></code>. As with <code>key</code>, comma-separated notation like <code>index="x,y,z"</code> is accepted for convenience. Only valid when argument <code>data.table=TRUE</code>. Where applicable, this should refer to column names given in <code>col.names</code>. 
</td></tr>
<tr valign="top"><td><code>showProgress</code></td>
<td>
 <code>TRUE</code> displays progress on the console if the ETA is greater than 3 seconds. It is produced in fread's C code where the very nice (but R level) txtProgressBar and tkProgressBar are not easily available. 
</td></tr>
<tr valign="top"><td><code>data.table</code></td>
<td>
 TRUE returns a <code>data.table</code>. FALSE returns a <code>data.frame</code>. The default for this argument can be changed with <code>options(datatable.fread.datatable=FALSE)</code>.
</td></tr>
<tr valign="top"><td><code>nThread</code></td>
<td>
The number of threads to use. Experiment to see what works best for your data on your hardware.
</td></tr>
<tr valign="top"><td><code>logical01</code></td>
<td>
If TRUE a column containing only 0s and 1s will be read as logical, otherwise as integer.
</td></tr>
<tr valign="top"><td><code>keepLeadingZeros</code></td>
<td>
If TRUE a column containing numeric data with leading zeros will be read as character, otherwise leading zeros will be removed and converted to numeric.
</td></tr>
<tr valign="top"><td><code>yaml</code></td>
<td>
 If <code>TRUE</code>, <code>fread</code> will attempt to parse (using <code><a href="../../yaml/html/yaml.load.html">yaml.load</a></code>) the top of the input as YAML, and further to glean parameters relevant to improving the performance of <code>fread</code> on the data itself. The entire YAML section is returned as parsed into a <code>list</code> in the <code>yaml_metadata</code> attribute. See <code>Details</code>. 
</td></tr>
<tr valign="top"><td><code>autostart</code></td>
<td>
 Deprecated and ignored with warning. Please use <code>skip</code> instead. 
</td></tr>
<tr valign="top"><td><code>tmpdir</code></td>
<td>
 Directory to use as the <code>tmpdir</code> argument for any <code>tempfile</code> calls, e.g. when the input is a URL or a shell command. The default is <code>tempdir()</code> which can be controlled by setting <code>TMPDIR</code> before starting the R session; see <code><a href="../../base/html/tempfile.html">base::tempdir</a></code>. 
</td></tr>
<tr valign="top"><td><code>tz</code></td>
<td>
 Relevant to datetime values which have no Z or UTC-offset at the end, i.e. unmarked datetime, as written by <code><a href="../../utils/html/write.table.html">utils::write.csv</a></code>. The default <code>tz="UTC"</code> reads unmarked datetime as UTC POSIXct efficiently. <code>tz=""</code> reads unmarked datetime as type character (slowly) so that <code>as.POSIXct</code> can interpret (slowly) the character datetimes in local timezone; e.g. by using <code>"POSIXct"</code> in <code>colClasses=</code>. Note that <code>fwrite()</code> by default writes datetime in UTC including the final Z and therefore <code>fwrite</code>'s output will be read by <code>fread</code> consistently and quickly without needing to use <code>tz=</code> or <code>colClasses=</code>. If the <code>TZ</code> environment variable is set to <code>"UTC"</code> (or <code>""</code> on non-Windows where unset vs &lsquo;&quot;&quot;' is significant) then the R session&rsquo;s timezone is already UTC and <code>tz=""</code> will result in unmarked datetimes being read as UTC POSIXct. For more information, please see the news items from v1.13.0 and v1.14.0. 
</td></tr>
</table>

<h3>Details</h3>

A sample of 10,000 rows is used for a very good estimate of column types. 100 contiguous rows are read from 100 equally spaced points throughout the file including the beginning, middle and the very end. This results in a better guess when a column changes type later in the file (e.g. blank at the beginning/only populated near the end, or 001 at the start but 0A0 later on). This very good type guess enables a single allocation of the correct type up front once for speed, memory efficiency and convenience of avoiding the need to set <code>colClasses</code> after an error. Even though the sample is large and jumping over the file, it is almost instant regardless of the size of the file because a lazy on-demand memory map is used. If a jump lands inside a quoted field containing newlines, each newline is tested until 5 lines are found following it with the expected number of fields. The lowest type for each column is chosen from the ordered list: <code>logical</code>, <code>integer</code>, <code>integer64</code>, <code>double</code>, <code>character</code>. Rarely, the file may contain data of a higher type in rows outside the sample (referred to as an out-of-sample type exception). In this event <code>fread</code> will automatically reread just those columns from the beginning so that you don't have the inconvenience of having to set <code>colClasses</code> yourself; particularly helpful if you have a lot of columns. Such columns must be read from the beginning to correctly distinguish &quot;00&quot; from &quot;000&quot; when those have both been interpreted as integer 0 due to the sample but 00A occurs out of sample. Set <code>verbose=TRUE</code> to see a detailed report of the logic deployed to read your file.

There is no line length limit, not even a very large one. Since we are encouraging <code>list</code> columns (i.e. <code>sep2</code>) this has the potential to encourage longer line lengths. So the approach of scanning each line into a buffer first and then rescanning that buffer is not used. There are no buffers used in <code>fread</code>'s C code at all. The field width limit is limited by R itself: the maximum width of a character string (currently 2^31-1 bytes, 2GB).

The filename extension (such as .csv) is irrelevant for &quot;auto&quot; <code>sep</code> and <code>sep2</code>. Separator detection is entirely driven by the file contents. This can be useful when loading a set of different files which may not be named consistently, or may not have the extension .csv despite being csv. Some datasets have been collected over many years, one file per day for example. Sometimes the file name format has changed at some point in the past or even the format of the file itself. So the idea is that you can loop <code>fread</code> through a set of files and as long as each file is regular and delimited, <code>fread</code> can read them all. Whether they all stack is another matter but at least each one is read quickly without you needing to vary <code>colClasses</code> in <code>read.table</code> or <code>read.csv</code>.

If an empty line is encountered then reading stops there with warning if any text exists after the empty line such as a footer. The first line of any text discarded is included in the warning message. Unless, it is single-column input. In that case blank lines are significant (even at the very end) and represent NA in the single column. So that <code>fread(fwrite(DT))==DT</code>. This default behaviour can be controlled using <code>blank.lines.skip=TRUE|FALSE</code>.

Line endings: All known line endings are detected automatically: <code>\n</code> (*NIX including Mac), <code>\r\n</code> (Windows CRLF), <code>\r</code> (old Mac) and <code>\n\r</code> (just in case). There is no need to convert input files first. <code>fread</code> running on any architecture will read a file from any architecture. Both <code>\r</code> and <code>\n</code> may be embedded in character strings (including column names) provided the field is quoted.

Decimal separator and locale: <code>fread(...,dec=",")</code> should just work. <code>fread</code> uses C function <code>strtod</code> to read numeric data; e.g., <code>1.23</code> or <code>1,23</code>. <code>strtod</code> retrieves the decimal separator (<code>.</code> or <code>,</code> usually) from the locale of the R session rather than as an argument passed to the <code>strtod</code> function. So for <code>fread(...,dec=",")</code> to work, <code>fread</code> changes this (and only this) R session's locale temporarily to a locale which provides the desired decimal separator.

On Windows, &quot;French_France.1252&quot; is tried which should be available as standard (any locale with comma decimal separator would suffice) and on unix &quot;fr_FR.utf8&quot; (you may need to install this locale on unix). <code>fread()</code> is very careful to set the locale back again afterwards, even if the function fails with an error. The choice of locale is determined by <code>options()$datatable.fread.dec.locale</code>. This may be a vector of locale names and if so they will be tried in turn until the desired <code>dec</code> is obtained; thus allowing more than two different decimal separators to be selected. This is a new feature in v1.9.6 and is experimental. In case of problems, turn it off with <code>options(datatable.fread.dec.experiment=FALSE)</code>.

Quotes:

When <code>quote</code> is a single character,

<ul>
<li>Spaces and other whitespace (other than <code>sep</code> and <code>\n</code>) may appear in unquoted character fields, e.g., <code>...,2,Joe Bloggs,3.14,...</code>.

</li>
<li>When <code>character</code> columns are quoted, they must start and end with that quoting character immediately followed by <code>sep</code> or <code>\n</code>, e.g., <code>...,2,"Joe Bloggs",3.14,...</code>.

In essence quoting character fields are required only if <code>sep</code> or <code>\n</code> appears in the string value. Quoting may be used to signify that numeric data should be read as text. Unescaped quotes may be present in a quoted field, e.g., <code>...,2,"Joe, "Bloggs"",3.14,...</code>, as well as escaped quotes, e.g., <code>...,2,"Joe \",Bloggs\"",3.14,...</code>.

If an embedded quote is followed by the separator inside a quoted field, the embedded quotes up to that point in that field must be balanced; e.g. <code>...,2,"www.blah?x="one",y="two"",3.14,...</code>.

On those fields that do not satisfy these conditions, e.g., fields with unbalanced quotes, <code>fread</code> re-attempts that field as if it isn't quoted. This is quite useful in reading files that contains fields with unbalanced quotes as well, automatically.

</li></ul>

To read fields as is instead, use <code>quote = ""</code>.

CSVY Support:

Currently, the <code>yaml</code> setting is somewhat inflexible with respect to incorporating metadata to facilitate file reading. Information on column classes should be stored at the top level under the heading <code>schema</code> and subheading <code>fields</code>; those with both a <code>type</code> and a <code>name</code> sub-heading will be merged into <code>colClasses</code>. Other supported elements are as follows:

<ul>
<li> <code>sep</code> (or alias <code>delimiter</code>) 

</li>
<li> <code>header</code> 

</li>
<li> <code>quote</code> (or aliases <code>quoteChar</code>, <code>quote_char</code>) 

</li>
<li> <code>dec</code> (or alias <code>decimal</code>) 

</li>
<li> <code>na.strings</code> 

</li></ul>

File Download:

When <code>input</code> begins with http://, https://, ftp://, ftps://, or file://, <code>fread</code> detects this and downloads the target to a temporary file (at <code>tempfile()</code>) before proceeding to read the file as usual. Secure URLS (ftps:// and https://) are downloaded with <code>curl::curl_download</code>; ftp:// and http:// paths are downloaded with <code>download.file</code> and <code>method</code> set to <code>getOption("download.file.method")</code>, defaulting to <code>"auto"</code>; and file:// is downloaded with <code>download.file</code> with <code>method="internal"</code>. NB: this implies that for file://, even files found on the current machine will be &quot;downloaded&quot; (i.e., hard-copied) to a temporary file. See <code><a href="../../utils/html/download.file.html">download.file</a></code> for more details.

Shell commands:

<code>fread</code> accepts shell commands for convenience. The input command is run and its output written to a file in <code>tmpdir</code> (<code><a href="../../base/html/tempfile.html">tempdir</a>()</code> by default) to which <code>fread</code> is applied &quot;as normal&quot;. The details are platform dependent &ndash; <code>system</code> is used on UNIX environments, <code>shell</code> otherwise; see <code><a href="../../base/html/system.html">system</a></code>.

<h3>Value</h3>

A <code>data.table</code> by default, otherwise a <code>data.frame</code> when argument <code>data.table=FALSE</code>.

<h3>References</h3>

Background : 
<a href="https://cran.r-project.org/doc/manuals/R-data.html">https://cran.r-project.org/doc/manuals/R-data.html</a> 
<a href="https://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r">https://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r</a> 
<a href="https://cerebralmastication.com/2009/11/loading-big-data-into-r/">https://cerebralmastication.com/2009/11/loading-big-data-into-r/</a> 
<a href="https://stackoverflow.com/questions/9061736/faster-than-scan-with-rcpp">https://stackoverflow.com/questions/9061736/faster-than-scan-with-rcpp</a> 
<a href="https://stackoverflow.com/questions/415515/how-can-i-read-and-manipulate-csv-file-data-in-c">https://stackoverflow.com/questions/415515/how-can-i-read-and-manipulate-csv-file-data-in-c</a> 
<a href="https://stackoverflow.com/questions/9352887/strategies-for-reading-in-csv-files-in-pieces">https://stackoverflow.com/questions/9352887/strategies-for-reading-in-csv-files-in-pieces</a> 
<a href="https://stackoverflow.com/questions/11782084/reading-in-large-text-files-in-r">https://stackoverflow.com/questions/11782084/reading-in-large-text-files-in-r</a> 
<a href="https://stackoverflow.com/questions/45972/mmap-vs-reading-blocks">https://stackoverflow.com/questions/45972/mmap-vs-reading-blocks</a> 
<a href="https://stackoverflow.com/questions/258091/when-should-i-use-mmap-for-file-access">https://stackoverflow.com/questions/258091/when-should-i-use-mmap-for-file-access</a> 
<a href="https://stackoverflow.com/a/9818473/403310">https://stackoverflow.com/a/9818473/403310</a> 
<a href="https://stackoverflow.com/questions/9608950/reading-huge-files-using-memory-mapped-files">https://stackoverflow.com/questions/9608950/reading-huge-files-using-memory-mapped-files</a>

finagler = &quot;to get or achieve by guile or manipulation&quot; <a href="https://dictionary.reference.com/browse/finagler">https://dictionary.reference.com/browse/finagler</a>

On YAML, see <a href="https://yaml.org/">https://yaml.org/</a>; on csvy, see <a href="https://csvy.org/">https://csvy.org/</a>.

<code><a href="../../utils/html/read.table.html">read.csv</a></code>, <code><a href="../../base/html/connections.html">url</a></code>, <code><a href="../../base/html/locales.html">Sys.setlocale</a></code>, <code><a href="openmp-utils.html">setDTthreads</a></code>, <code><a href="fwrite.html">fwrite</a></code>, <a href="https://CRAN.R-project.org/package=bit64"><code>bit64::integer64</code></a>

<h3>Examples</h3>

<pre>
# Reads text input directly :
fread("A,B\n1,2\n3,4")

# Reads pasted input directly :
fread("A,B
1,2
3,4
")

# Finds the first data line automatically :
fread("
This is perhaps a banner line or two or ten.
A,B
1,2
3,4
")

# Detects whether column names are present automatically :
fread("
1,2
3,4
")

# Numerical precision :

DT = fread("A\n1.010203040506070809010203040506\n")
# TODO: add numerals=c("allow.loss", "warn.loss", "no.loss") from base::read.table, +"use.Rmpfr"
typeof(DT$A)=="double"   # currently "allow.loss" with no option

DT = fread("A\n1.46761e-313\n")   # read as 'numeric'
DT[,sprintf("%.15E",A)]   # beyond what double precision can store accurately to 15 digits
# For greater accuracy use colClasses to read as character, then package Rmpfr.

# colClasses
data = "A,B,C,D\n1,3,5,7\n2,4,6,8\n"
fread(data, colClasses=c(B="character",C="character",D="character"))  # as read.csv
fread(data, colClasses=list(character=c("B","C","D")))    # saves typing
fread(data, colClasses=list(character=2:4))     # same using column numbers

# drop
fread(data, colClasses=c("B"="NULL","C"="NULL"))   # as read.csv
fread(data, colClasses=list(NULL=c("B","C")))      #
fread(data, drop=c("B","C"))      # same but less typing, easier to read
fread(data, drop=2:3)             # same using column numbers

# select
# (in read.csv you need to work out which to drop)
fread(data, select=c("A","D"))    # less typing, easier to read
fread(data, select=c(1,4))        # same using column numbers

# select and types combined
fread(data, select=c(A="numeric", D="character"))
fread(data, select=list(numeric="A", character="D"))

# skip blank lines
fread("a,b\n1,a\n2,b\n\n\n3,c\n", blank.lines.skip=TRUE)
# fill
fread("a,b\n1,a\n2\n3,c\n", fill=TRUE)
fread("a,b\n\n1,a\n2\n\n3,c\n\n", fill=TRUE)

# fill with skip blank lines
fread("a,b\n\n1,a\n2\n\n3,c\n\n", fill=TRUE, blank.lines.skip=TRUE)

# check.names usage
fread("a b,a b\n1,2\n")
fread("a b,a b\n1,2\n", check.names=TRUE) # no duplicates + syntactically valid names

## Not run: 
# Demo speed-up
n = 1e6
DT = data.table( a=sample(1:1000,n,replace=TRUE),
                 b=sample(1:1000,n,replace=TRUE),
                 c=rnorm(n),
                 d=sample(c("foo","bar","baz","qux","quux"),n,replace=TRUE),
                 e=rnorm(n),
                 f=sample(1:1000,n,replace=TRUE) )
DT[2,b:=NA_integer_]
DT[4,c:=NA_real_]
DT[3,d:=NA_character_]
DT[5,d:=""]
DT[2,e:=+Inf]
DT[3,e:=-Inf]

write.table(DT,"test.csv",sep=",",row.names=FALSE,quote=FALSE)
cat("File size (MB):", round(file.info("test.csv")$size/1024^2),"\n")
# 50 MB (1e6 rows x 6 columns)

system.time(DF1 &lt;-read.csv("test.csv",stringsAsFactors=FALSE))
# 5.4 sec (first time in fresh R session)

system.time(DF1 &lt;- read.csv("test.csv",stringsAsFactors=FALSE))
# 3.9 sec (immediate repeat is faster, varies)

system.time(DF2 &lt;- read.table("test.csv",header=TRUE,sep=",",quote="",
    stringsAsFactors=FALSE,comment.char="",nrows=n,
    colClasses=c("integer","integer","numeric",
                 "character","numeric","integer")))
# 1.2 sec (consistently). All known tricks and known nrows, see references.

system.time(DT &lt;- fread("test.csv"))
# 0.1 sec (faster and friendlier)

identical(DF1, DF2)
all.equal(as.data.table(DF1), DT)

# Scaling up ...
l = vector("list",10)
for (i in 1:10) l[[i]] = DT
DTbig = rbindlist(l)
tables()
write.table(DTbig,"testbig.csv",sep=",",row.names=FALSE,quote=FALSE)
# 500MB csv (10 million rows x 6 columns)

system.time(DF &lt;- read.table("testbig.csv",header=TRUE,sep=",",
    quote="",stringsAsFactors=FALSE,comment.char="",nrows=1e7,
    colClasses=c("integer","integer","numeric",
                 "character","numeric","integer")))
# 17.0 sec (varies)

system.time(DT &lt;- fread("testbig.csv"))
#  0.8 sec

all(mapply(all.equal, DF, DT))

# Real data example (Airline data)
# https://stat-computing.org/dataexpo/2009/the-data.html

download.file("https://stat-computing.org/dataexpo/2009/2008.csv.bz2",
              destfile="2008.csv.bz2")
# 109MB (compressed)

system("bunzip2 2008.csv.bz2")
# 658MB (7,009,728 rows x 29 columns)

colClasses = sapply(read.csv("2008.csv",nrows=100,stringsAsFactors=FALSE),class)
# 4 character, 24 integer, 1 logical. Incorrect.

colClasses = sapply(read.csv("2008.csv",nrows=200,stringsAsFactors=FALSE),class)
# 5 character, 24 integer. Correct. Might have missed data only using 100 rows
# since read.table assumes colClasses is correct.

system.time(DF &lt;- read.table("2008.csv", header=TRUE, sep=",",
    quote="",stringsAsFactors=FALSE,comment.char="",nrows=7009730,
    colClasses=colClasses))
# 24.4 secs

system.time(DT &lt;- fread("2008.csv"))
#  1.9 secs

table(sapply(DT,class))
# 5 character and 24 integer columns. Correct without needing to worry about colClasses
# issue above.

# Reads URLs directly :
fread("https://www.stats.ox.ac.uk/pub/datasets/csb/ch11b.dat")

# Decompresses .gz and .bz2 automatically :
fread("https://stat-computing.org/dataexpo/2009/1987.csv.bz2")

## End(Not run)
</pre>

<hr /><div style="text-align: center;">[Package data.table version 1.14.4 <a href="00Index.html">Index</a>]</div>
</body></html>