EVOLUTION-MANAGER
Edit File: datatable-faq.html
<!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <meta name="generator" content="pandoc" /> <meta http-equiv="X-UA-Compatible" content="IE=EDGE" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <meta name="date" content="2022-10-09" /> <title>Frequently Asked Questions about data.table</title> <script>// Pandoc 2.9 adds attributes on both header and div. We remove the former (to // be compatible with the behavior of Pandoc < 2.8). document.addEventListener('DOMContentLoaded', function(e) { var hs = document.querySelectorAll("div.section[class*='level'] > :first-child"); var i, h, a; for (i = 0; i < hs.length; i++) { h = hs[i]; if (!/^h[1-6]$/i.test(h.tagName)) continue; // it should be a header h1-h6 a = h.attributes; while (a.length > 0) h.removeAttribute(a[0].name); } }); </script> <script>// Hide empty <a> tag within highlighted CodeBlock for screen reader accessibility (see https://github.com/jgm/pandoc/issues/6352#issuecomment-626106786) --> // v0.0.1 // Written by JooYoung Seo (jooyoung@psu.edu) and Atsushi Yasumoto on June 1st, 2020. document.addEventListener('DOMContentLoaded', function() { const codeList = document.getElementsByClassName("sourceCode"); for (var i = 0; i < codeList.length; i++) { var linkList = codeList[i].getElementsByTagName('a'); for (var j = 0; j < linkList.length; j++) { if (linkList[j].innerHTML === "") { linkList[j].setAttribute('aria-hidden', 'true'); } } } }); </script> <style type="text/css"> code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;} div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} ul.task-list{list-style: none;} </style> <style type="text/css"> code { white-space: pre; } .sourceCode { overflow: visible; } </style> <style type="text/css" data-origin="pandoc"> pre > code.sourceCode { white-space: pre; position: relative; } pre > code.sourceCode > span { display: inline-block; line-height: 1.25; } pre > code.sourceCode > span:empty { height: 1.2em; } code.sourceCode > span { color: inherit; text-decoration: inherit; } div.sourceCode { margin: 1em 0; } pre.sourceCode { margin: 0; } @media screen { div.sourceCode { overflow: auto; } } @media print { pre > code.sourceCode { white-space: pre-wrap; } pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; } } pre.numberSource code { counter-reset: source-line 0; } pre.numberSource code > span { position: relative; left: -4em; counter-increment: source-line; } pre.numberSource code > span > a:first-child::before { content: counter(source-line); position: relative; left: -1em; text-align: right; vertical-align: baseline; border: none; display: inline-block; -webkit-touch-callout: none; -webkit-user-select: none; -khtml-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; padding: 0 4px; width: 4em; color: #aaaaaa; } pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } div.sourceCode { } @media screen { pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } } code span.al { color: #ff0000; font-weight: bold; } /* Alert */ code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ code span.at { color: #7d9029; } /* Attribute */ code span.bn { color: #40a070; } /* BaseN */ code span.bu { } /* BuiltIn */ code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ code span.ch { color: #4070a0; } /* Char */ code span.cn { color: #880000; } /* Constant */ code span.co { color: #60a0b0; font-style: italic; } /* Comment */ code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ code span.do { color: #ba2121; font-style: italic; } /* Documentation */ code span.dt { color: #902000; } /* DataType */ code span.dv { color: #40a070; } /* DecVal */ code span.er { color: #ff0000; font-weight: bold; } /* Error */ code span.ex { } /* Extension */ code span.fl { color: #40a070; } /* Float */ code span.fu { color: #06287e; } /* Function */ code span.im { } /* Import */ code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ code span.kw { color: #007020; font-weight: bold; } /* Keyword */ code span.op { color: #666666; } /* Operator */ code span.ot { color: #007020; } /* Other */ code span.pp { color: #bc7a00; } /* Preprocessor */ code span.sc { color: #4070a0; } /* SpecialChar */ code span.ss { color: #bb6688; } /* SpecialString */ code span.st { color: #4070a0; } /* String */ code span.va { color: #19177c; } /* Variable */ code span.vs { color: #4070a0; } /* VerbatimString */ code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ </style> <script> // apply pandoc div.sourceCode style to pre.sourceCode instead (function() { var sheets = document.styleSheets; for (var i = 0; i < sheets.length; i++) { if (sheets[i].ownerNode.dataset["origin"] !== "pandoc") continue; try { var rules = sheets[i].cssRules; } catch (e) { continue; } var j = 0; while (j < rules.length) { var rule = rules[j]; // check if there is a div.sourceCode rule if (rule.type !== rule.STYLE_RULE || rule.selectorText !== "div.sourceCode") { j++; continue; } var style = rule.style.cssText; // check if color or background-color is set if (rule.style.color === '' && rule.style.backgroundColor === '') { j++; continue; } // replace div.sourceCode by a pre.sourceCode rule sheets[i].deleteRule(j); sheets[i].insertRule('pre.sourceCode{' + style + '}', j); } } })(); </script> <style type="text/css">body { background-color: #fff; margin: 1em auto; max-width: 700px; overflow: visible; padding-left: 2em; padding-right: 2em; font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 1.35; } #TOC { clear: both; margin: 0 0 10px 10px; padding: 4px; width: 400px; border: 1px solid #CCCCCC; border-radius: 5px; background-color: #f6f6f6; font-size: 13px; line-height: 1.3; } #TOC .toctitle { font-weight: bold; font-size: 15px; margin-left: 5px; } #TOC ul { padding-left: 40px; margin-left: -1.5em; margin-top: 5px; margin-bottom: 5px; } #TOC ul ul { margin-left: -2em; } #TOC li { line-height: 16px; } table { margin: 1em auto; border-width: 1px; border-color: #DDDDDD; border-style: outset; border-collapse: collapse; } table th { border-width: 2px; padding: 5px; border-style: inset; } table td { border-width: 1px; border-style: inset; line-height: 18px; padding: 5px 5px; } table, table th, table td { border-left-style: none; border-right-style: none; } table thead, table tr.even { background-color: #f7f7f7; } p { margin: 0.5em 0; } blockquote { background-color: #f6f6f6; padding: 0.25em 0.75em; } hr { border-style: solid; border: none; border-top: 1px solid #777; margin: 28px 0; } dl { margin-left: 0; } dl dd { margin-bottom: 13px; margin-left: 13px; } dl dt { font-weight: bold; } ul { margin-top: 0; } ul li { list-style: circle outside; } ul ul { margin-bottom: 0; } pre, code { background-color: #f7f7f7; border-radius: 3px; color: #333; white-space: pre-wrap; } pre { border-radius: 3px; margin: 5px 0px 10px 0px; padding: 10px; } pre:not([class]) { background-color: #f7f7f7; } code { font-family: Consolas, Monaco, 'Courier New', monospace; font-size: 85%; } p > code, li > code { padding: 2px 0px; } div.figure { text-align: center; } img { background-color: #FFFFFF; padding: 2px; border: 1px solid #DDDDDD; border-radius: 3px; border: 1px solid #CCCCCC; margin: 0 5px; } h1 { margin-top: 0; font-size: 35px; line-height: 40px; } h2 { border-bottom: 4px solid #f7f7f7; padding-top: 10px; padding-bottom: 2px; font-size: 145%; } h3 { border-bottom: 2px solid #f7f7f7; padding-top: 10px; font-size: 120%; } h4 { border-bottom: 1px solid #f7f7f7; margin-left: 8px; font-size: 105%; } h5, h6 { border-bottom: 1px solid #ccc; font-size: 105%; } a { color: #0033dd; text-decoration: none; } a:hover { color: #6666ff; } a:visited { color: #800080; } a:visited:hover { color: #BB00BB; } a[href^="http:"] { text-decoration: underline; } a[href^="https:"] { text-decoration: underline; } code > span.kw { color: #555; font-weight: bold; } code > span.dt { color: #902000; } code > span.dv { color: #40a070; } code > span.bn { color: #d14; } code > span.fl { color: #d14; } code > span.ch { color: #d14; } code > span.st { color: #d14; } code > span.co { color: #888888; font-style: italic; } code > span.ot { color: #007020; } code > span.al { color: #ff0000; font-weight: bold; } code > span.fu { color: #900; font-weight: bold; } code > span.er { color: #a61717; background-color: #e3d2d2; } </style> </head> <body> <h1 class="title toc-ignore">Frequently Asked Questions about data.table</h1> <h4 class="date">2022-10-09</h4> <div id="TOC"> <ul> <li><a href="#beginner-faqs"><span class="toc-section-number">1</span> Beginner FAQs</a> <ul> <li><a href="#j-num"><span class="toc-section-number">1.1</span> Why do <code>DT[ , 5]</code> and <code>DT[2, 5]</code> return a 1-column data.table rather than vectors like <code>data.frame</code>?</a></li> <li><a href="#why-does-dtregion-return-a-1-column-data.table-rather-than-a-vector"><span class="toc-section-number">1.2</span> Why does <code>DT[,"region"]</code> return a 1-column data.table rather than a vector?</a></li> <li><a href="#why-does-dt-region-return-a-vector-for-the-region-column-id-like-a-1-column-data.table."><span class="toc-section-number">1.3</span> Why does <code>DT[, region]</code> return a vector for the “region” column? I’d like a 1-column data.table.</a></li> <li><a href="#why-does-dt-x-y-z-not-work-i-wanted-the-3-columns-xy-and-z."><span class="toc-section-number">1.4</span> Why does <code>DT[ , x, y, z]</code> not work? I wanted the 3 columns <code>x</code>,<code>y</code> and <code>z</code>.</a></li> <li><a href="#i-assigned-a-variable-mycol-x-but-then-dt-mycol-returns-x.-how-do-i-get-it-to-look-up-the-column-name-contained-in-the-mycol-variable"><span class="toc-section-number">1.5</span> I assigned a variable <code>mycol = "x"</code> but then <code>DT[ , mycol]</code> returns <code>"x"</code>. How do I get it to look up the column name contained in the <code>mycol</code> variable?</a></li> <li><a href="#what-are-the-benefits-of-being-able-to-use-column-names-as-if-they-are-variables-inside-dt..."><span class="toc-section-number">1.6</span> What are the benefits of being able to use column names as if they are variables inside <code>DT[...]</code>?</a></li> <li><a href="#ok-im-starting-to-see-what-data.table-is-about-but-why-didnt-you-just-enhance-data.frame-in-r-why-does-it-have-to-be-a-new-package"><span class="toc-section-number">1.7</span> OK, I’m starting to see what data.table is about, but why didn’t you just enhance <code>data.frame</code> in R? Why does it have to be a new package?</a></li> <li><a href="#why-are-the-defaults-the-way-they-are-why-does-it-work-the-way-it-does"><span class="toc-section-number">1.8</span> Why are the defaults the way they are? Why does it work the way it does?</a></li> <li><a href="#isnt-this-already-done-by-with-and-subset-in-base"><span class="toc-section-number">1.9</span> Isn’t this already done by <code>with()</code> and <code>subset()</code> in <code>base</code>?</a></li> <li><a href="#why-does-xy-return-all-the-columns-from-y-too-shouldnt-it-return-a-subset-of-x"><span class="toc-section-number">1.10</span> Why does <code>X[Y]</code> return all the columns from <code>Y</code> too? Shouldn’t it return a subset of <code>X</code>?</a></li> <li><a href="#MergeDiff"><span class="toc-section-number">1.11</span> What is the difference between <code>X[Y]</code> and <code>merge(X, Y)</code>?</a></li> <li><a href="#anything-else-about-xy-sumfoobar"><span class="toc-section-number">1.12</span> Anything else about <code>X[Y, sum(foo*bar)]</code>?</a></li> <li><a href="#thats-nice.-how-did-you-manage-to-change-it-given-that-users-depended-on-the-old-behaviour"><span class="toc-section-number">1.13</span> That’s nice. How did you manage to change it given that users depended on the old behaviour?</a></li> </ul></li> <li><a href="#general-syntax"><span class="toc-section-number">2</span> General Syntax</a> <ul> <li><a href="#how-can-i-avoid-writing-a-really-long-j-expression-youve-said-that-i-should-use-the-column-names-but-ive-got-a-lot-of-columns."><span class="toc-section-number">2.1</span> How can I avoid writing a really long <code>j</code> expression? You’ve said that I should use the column <em>names</em>, but I’ve got a lot of columns.</a></li> <li><a href="#why-is-the-default-for-mult-now-all"><span class="toc-section-number">2.2</span> Why is the default for <code>mult</code> now <code>"all"</code>?</a></li> <li><a href="#im-using-c-in-j-and-getting-strange-results."><span class="toc-section-number">2.3</span> I’m using <code>c()</code> in <code>j</code> and getting strange results.</a></li> <li><a href="#i-have-built-up-a-complex-table-with-many-columns.-i-want-to-use-it-as-a-template-for-a-new-table-i.e.-create-a-new-table-with-no-rows-but-with-the-column-names-and-types-copied-from-my-table.-can-i-do-that-easily"><span class="toc-section-number">2.4</span> I have built up a complex table with many columns. I want to use it as a template for a new table; <em>i.e.</em>, create a new table with no rows, but with the column names and types copied from my table. Can I do that easily?</a></li> <li><a href="#is-a-null-data.table-the-same-as-dt0"><span class="toc-section-number">2.5</span> Is a null data.table the same as <code>DT[0]</code>?</a></li> <li><a href="#DTremove1"><span class="toc-section-number">2.6</span> Why has the <code>DT()</code> alias been removed?</a></li> <li><a href="#DTremove2"><span class="toc-section-number">2.7</span> But my code uses <code>j = DT(...)</code> and it works. The previous FAQ says that <code>DT()</code> has been removed.</a></li> <li><a href="#what-are-the-scoping-rules-for-j-expressions"><span class="toc-section-number">2.8</span> What are the scoping rules for <code>j</code> expressions?</a></li> <li><a href="#j-trace"><span class="toc-section-number">2.9</span> Can I trace the <code>j</code> expression as it runs through the groups?</a></li> <li><a href="#inside-each-group-why-are-the-group-variables-length-1"><span class="toc-section-number">2.10</span> Inside each group, why are the group variables length-1?</a></li> <li><a href="#only-the-first-10-rows-are-printed-how-do-i-print-more"><span class="toc-section-number">2.11</span> Only the first 10 rows are printed, how do I print more?</a></li> <li><a href="#with-an-xy-join-what-if-x-contains-a-column-called-y"><span class="toc-section-number">2.12</span> With an <code>X[Y]</code> join, what if <code>X</code> contains a column called <code>"Y"</code>?</a></li> <li><a href="#xzy-is-failing-because-x-contains-a-column-y.-id-like-it-to-use-the-table-y-in-calling-scope."><span class="toc-section-number">2.13</span> <code>X[Z[Y]]</code> is failing because <code>X</code> contains a column <code>"Y"</code>. I’d like it to use the table <code>Y</code> in calling scope.</a></li> <li><a href="#can-you-explain-further-why-data.table-is-inspired-by-ab-syntax-in-base"><span class="toc-section-number">2.14</span> Can you explain further why data.table is inspired by <code>A[B]</code> syntax in <code>base</code>?</a></li> <li><a href="#can-base-be-changed-to-do-this-then-rather-than-a-new-package"><span class="toc-section-number">2.15</span> Can base be changed to do this then, rather than a new package?</a></li> <li><a href="#ive-heard-that-data.table-syntax-is-analogous-to-sql."><span class="toc-section-number">2.16</span> I’ve heard that data.table syntax is analogous to SQL.</a></li> <li><a href="#SmallerDiffs"><span class="toc-section-number">2.17</span> What are the smaller syntax differences between <code>data.frame</code> and data.table</a></li> <li><a href="#im-using-j-for-its-side-effect-only-but-im-still-getting-data-returned.-how-do-i-stop-that"><span class="toc-section-number">2.18</span> I’m using <code>j</code> for its side effect only, but I’m still getting data returned. How do I stop that?</a></li> <li><a href="#why-does-.data.table-now-have-a-drop-argument-from-v1.5"><span class="toc-section-number">2.19</span> Why does <code>[.data.table</code> now have a <code>drop</code> argument from v1.5?</a></li> <li><a href="#rolling-joins-are-cool-and-very-fast-was-that-hard-to-program"><span class="toc-section-number">2.20</span> Rolling joins are cool and very fast! Was that hard to program?</a></li> <li><a href="#why-does-dti-col-value-return-the-whole-of-dt-i-expected-either-no-visible-value-consistent-with---or-a-message-or-return-value-containing-how-many-rows-were-updated.-it-isnt-obvious-that-the-data-has-indeed-been-updated-by-reference."><span class="toc-section-number">2.21</span> Why does <code>DT[i, col := value]</code> return the whole of <code>DT</code>? I expected either no visible value (consistent with <code><-</code>), or a message or return value containing how many rows were updated. It isn’t obvious that the data has indeed been updated by reference.</a></li> <li><a href="#ok-thanks.-what-was-so-difficult-about-the-result-of-dti-col-value-being-returned-invisibly"><span class="toc-section-number">2.22</span> OK, thanks. What was so difficult about the result of <code>DT[i, col := value]</code> being returned invisibly?</a></li> <li><a href="#why-do-i-have-to-type-dt-sometimes-twice-after-using-to-print-the-result-to-console"><span class="toc-section-number">2.23</span> Why do I have to type <code>DT</code> sometimes twice after using <code>:=</code> to print the result to console?</a></li> <li><a href="#ive-noticed-that-basecbind.data.frame-and-baserbind.data.frame-appear-to-be-changed-by-data.table.-how-is-this-possible-why"><span class="toc-section-number">2.24</span> I’ve noticed that <code>base::cbind.data.frame</code> (and <code>base::rbind.data.frame</code>) appear to be changed by data.table. How is this possible? Why?</a></li> <li><a href="#r-dispatch"><span class="toc-section-number">2.25</span> I’ve read about method dispatch (<em>e.g.</em> <code>merge</code> may or may not dispatch to <code>merge.data.table</code>) but <em>how</em> does R know how to dispatch? Are dots significant or special? How on earth does R know which function to dispatch and when?</a></li> </ul></li> <li><a href="#questions-relating-to-compute-time"><span class="toc-section-number">3</span> Questions relating to compute time</a> <ul> <li><a href="#i-have-20-columns-and-a-large-number-of-rows.-why-is-an-expression-of-one-column-so-quick"><span class="toc-section-number">3.1</span> I have 20 columns and a large number of rows. Why is an expression of one column so quick?</a></li> <li><a href="#i-dont-have-a-key-on-a-large-table-but-grouping-is-still-really-quick.-why-is-that"><span class="toc-section-number">3.2</span> I don’t have a <code>key</code> on a large table, but grouping is still really quick. Why is that?</a></li> <li><a href="#why-is-grouping-by-columns-in-the-key-faster-than-an-ad-hoc-by"><span class="toc-section-number">3.3</span> Why is grouping by columns in the key faster than an <em>ad hoc</em> <code>by</code>?</a></li> <li><a href="#what-are-primary-and-secondary-indexes-in-data.table"><span class="toc-section-number">3.4</span> What are primary and secondary indexes in data.table?</a></li> </ul></li> <li><a href="#error-messages"><span class="toc-section-number">4</span> Error messages</a> <ul> <li><a href="#could-not-find-function-dt"><span class="toc-section-number">4.1</span> “Could not find function <code>DT</code>”</a></li> <li><a href="#unused-arguments-mysum-sumv"><span class="toc-section-number">4.2</span> “unused argument(s) (<code>MySum = sum(v)</code>)”</a></li> <li><a href="#translatecharutf8-must-be-called-on-a-charsxp"><span class="toc-section-number">4.3</span> “<code>translateCharUTF8</code> must be called on a <code>CHARSXP</code>”</a></li> <li><a href="#cbinderror"><span class="toc-section-number">4.4</span> <code>cbind(DT, DF)</code> returns a strange format, <em>e.g.</em> <code>Integer,5</code></a></li> <li><a href="#cannot-change-value-of-locked-binding-for-.sd"><span class="toc-section-number">4.5</span> “cannot change value of locked binding for <code>.SD</code>”</a></li> <li><a href="#cannot-change-value-of-locked-binding-for-.n"><span class="toc-section-number">4.6</span> “cannot change value of locked binding for <code>.N</code>”</a></li> </ul></li> <li><a href="#warning-messages"><span class="toc-section-number">5</span> Warning messages</a> <ul> <li><a href="#the-following-objects-are-masked-from-packagebase-cbind-rbind"><span class="toc-section-number">5.1</span> “The following object(s) are masked from <code>package:base</code>: <code>cbind</code>, <code>rbind</code>”</a></li> <li><a href="#coerced-numeric-rhs-to-integer-to-match-the-columns-type"><span class="toc-section-number">5.2</span> “Coerced numeric RHS to integer to match the column’s type”</a></li> <li><a href="#reading-data.table-from-rds-or-rdata-file"><span class="toc-section-number">5.3</span> Reading data.table from RDS or RData file</a></li> </ul></li> <li><a href="#general-questions-about-the-package"><span class="toc-section-number">6</span> General questions about the package</a> <ul> <li><a href="#v1.3-appears-to-be-missing-from-the-cran-archive"><span class="toc-section-number">6.1</span> v1.3 appears to be missing from the CRAN archive?</a></li> <li><a href="#is-data.table-compatible-with-s-plus"><span class="toc-section-number">6.2</span> Is data.table compatible with S-plus?</a></li> <li><a href="#is-it-available-for-linux-mac-and-windows"><span class="toc-section-number">6.3</span> Is it available for Linux, Mac and Windows?</a></li> <li><a href="#i-think-its-great.-what-can-i-do"><span class="toc-section-number">6.4</span> I think it’s great. What can I do?</a></li> <li><a href="#i-think-its-not-great.-how-do-i-warn-others-about-my-experience"><span class="toc-section-number">6.5</span> I think it’s not great. How do I warn others about my experience?</a></li> <li><a href="#i-have-a-question.-i-know-the-r-help-posting-guide-tells-me-to-contact-the-maintainer-not-r-help-but-is-there-a-larger-group-of-people-i-can-ask"><span class="toc-section-number">6.6</span> I have a question. I know the r-help posting guide tells me to contact the maintainer (not r-help), but is there a larger group of people I can ask?</a></li> <li><a href="#where-are-the-datatable-help-archives"><span class="toc-section-number">6.7</span> Where are the datatable-help archives?</a></li> <li><a href="#id-prefer-not-to-post-on-the-issues-page-can-i-mail-just-one-or-two-people-privately"><span class="toc-section-number">6.8</span> I’d prefer not to post on the Issues page, can I mail just one or two people privately?</a></li> <li><a href="#i-have-created-a-package-that-uses-data.table.-how-do-i-ensure-my-package-is-data.table-aware-so-that-inheritance-from-data.frame-works"><span class="toc-section-number">6.9</span> I have created a package that uses data.table. How do I ensure my package is data.table-aware so that inheritance from <code>data.frame</code> works?</a></li> </ul></li> </ul> </div> <style> h2 { font-size: 20px; } #TOC { width: 100%; } </style> <p>The first section, Beginner FAQs, is intended to be read in order, from start to finish. It’s just written in a FAQ style to be digested more easily. It isn’t really the most frequently asked questions. A better measure for that is looking on Stack Overflow.</p> <p>This FAQ is required reading and considered core documentation. Please do not ask questions on Stack Overflow or raise issues on GitHub until you have read it. We can all tell when you ask that you haven’t read it. So if you do ask and haven’t read it, don’t use your real name.</p> <p>This document has been quickly revised given the changes in v1.9.8 released Nov 2016. Please do submit pull requests to fix mistakes or improvements. If anyone knows why the table of contents comes out so narrow and squashed when displayed by CRAN, please let us know. This document used to be a PDF and we changed it recently to HTML.</p> <div id="beginner-faqs" class="section level1" number="1"> <h1 number="1"><span class="header-section-number">1</span> Beginner FAQs</h1> <div id="j-num" class="section level2" number="1.1"> <h2 number="1.1"><span class="header-section-number">1.1</span> Why do <code>DT[ , 5]</code> and <code>DT[2, 5]</code> return a 1-column data.table rather than vectors like <code>data.frame</code>?</h2> <p>For consistency so that when you use data.table in functions that accept varying inputs, you can rely on <code>DT[...]</code> returning a data.table. You don’t have to remember to include <code>drop=FALSE</code> like you do in data.frame. data.table was first released in 2006 and this difference to data.frame has been a feature since the very beginning.</p> <p>You may have heard that it is generally bad practice to refer to columns by number rather than name, though. If your colleague comes along and reads your code later they may have to hunt around to find out which column is number 5. If you or they change the column ordering higher up in your R program, you may produce wrong results with no warning or error if you forget to change all the places in your code which refer to column number 5. That is your fault not R’s or data.table’s. It’s really really bad. Please don’t do it. It’s the same mantra as professional SQL developers have: never use <code>select *</code>, always explicitly select by column name to at least try to be robust to future changes.</p> <p>Say column 5 is named <code>"region"</code> and you really must extract that column as a vector not a data.table. It is more robust to use the column name and write <code>DT$region</code> or <code>DT[["region"]]</code>; i.e., the same as base R. Using base R’s <code>$</code> and <code>[[</code> on data.table is encouraged. Not when combined with <code><-</code> to assign (use <code>:=</code> instead for that) but just to select a single column by name they are encouraged.</p> <p>There are some circumstances where referring to a column by number seems like the only way, such as a sequence of columns. In these situations just like data.frame, you can write <code>DT[, 5:10]</code> and <code>DT[,c(1,4,10)]</code>. However, again, it is more robust (to future changes in your data’s number of and ordering of columns) to use a named range such as <code>DT[,columnRed:columnViolet]</code> or name each one <code>DT[,c("columnRed","columnOrange","columnYellow")]</code>. It is harder work up front, but you will probably thank yourself and your colleagues might thank you in the future. At least you can say you tried your best to write robust code if something does go wrong.</p> <p>However, what we really want you to do is <code>DT[,.(columnRed,columnOrange,columnYellow)]</code>; i.e., use column names as if they are variables directly inside <code>DT[...]</code>. You don’t have to prefix each column with <code>DT$</code> like you do in data.frame. The <code>.()</code> part is just an alias for <code>list()</code> and you can use <code>list()</code> instead if you prefer. You can place any R expression of column names, using any R package, returning different types of different lengths, right there. We wanted to encourage you to do that so strongly in the past that we deliberately didn’t make <code>DT[,5]</code> work at all. Before v1.9.8 released Nov 2016, <code>DT[,5]</code> used to just return <code>5</code>. The thinking was that we could more simply teach one fact that the parts inside <code>DT[...]</code> get evaluated within the frame of DT always (they see column names as if they are variables). And <code>5</code> evaluates to <code>5</code> so that behaviour was consistent with the single rule. We asked you to go through an extra deliberate hurdle <code>DT[,5,with=FALSE]</code> if you really wanted to select a column by name or number. Going forward from Nov 2016, you don’t need to use <code>with=FALSE</code> and we’ll see how greater consistency with data.frame in this regard will help or hinder both new and long-time users. The new users who don’t read this FAQ, not even this very first entry, will hopefully not stumble as soon with data.table as they did before if they had expected it to work like data.frame. Hopefully they will not miss out on understanding our intent and recommendation to place expressions of columns inside <code>DT[i, j, by]</code>. If they use data.table like data.frame they won’t gain any benefits. If you know anyone like that, please give them a friendly nudge to read this document like you are.</p> <p>Reminder: you can place <em>any</em> R expression inside <code>DT[...]</code> using column names as if they are variables; e.g., try <code>DT[, colA*colB/2]</code>. That does return a vector because you used column names as if they are variables. Wrap with <code>.()</code> to return a data.table; i.e. <code>DT[,.(colA*colB/2)]</code>. Name it: <code>DT[,.(myResult = colA*colB/2)]</code>. And we’ll leave it to you to guess how to return two things from this query. It’s also quite common to do a bunch of things inside an anonymous body: <code>DT[, { x<-colA+10; x*x/2 }]</code> or call another package’s function: <code>DT[ , fitdistr(columnA, "normal")]</code>.</p> </div> <div id="why-does-dtregion-return-a-1-column-data.table-rather-than-a-vector" class="section level2" number="1.2"> <h2 number="1.2"><span class="header-section-number">1.2</span> Why does <code>DT[,"region"]</code> return a 1-column data.table rather than a vector?</h2> <p>See the <a href="#j-num">answer above</a>. Try <code>DT$region</code> instead. Or <code>DT[["region"]]</code>.</p> </div> <div id="why-does-dt-region-return-a-vector-for-the-region-column-id-like-a-1-column-data.table." class="section level2" number="1.3"> <h2 number="1.3"><span class="header-section-number">1.3</span> Why does <code>DT[, region]</code> return a vector for the “region” column? I’d like a 1-column data.table.</h2> <p>Try <code>DT[ , .(region)]</code> instead. <code>.()</code> is an alias for <code>list()</code> and ensures a data.table is returned.</p> <p>Also continue reading and see the FAQ after next. Skim whole documents before getting stuck in one part.</p> </div> <div id="why-does-dt-x-y-z-not-work-i-wanted-the-3-columns-xy-and-z." class="section level2" number="1.4"> <h2 number="1.4"><span class="header-section-number">1.4</span> Why does <code>DT[ , x, y, z]</code> not work? I wanted the 3 columns <code>x</code>,<code>y</code> and <code>z</code>.</h2> <p>The <code>j</code> expression is the 2nd argument. Try <code>DT[ , c("x","y","z")]</code> or <code>DT[ , .(x,y,z)]</code>.</p> </div> <div id="i-assigned-a-variable-mycol-x-but-then-dt-mycol-returns-x.-how-do-i-get-it-to-look-up-the-column-name-contained-in-the-mycol-variable" class="section level2" number="1.5"> <h2 number="1.5"><span class="header-section-number">1.5</span> I assigned a variable <code>mycol = "x"</code> but then <code>DT[ , mycol]</code> returns <code>"x"</code>. How do I get it to look up the column name contained in the <code>mycol</code> variable?</h2> <p>In v1.9.8 released Nov 2016 there is an ability to turn on new behaviour: <code>options(datatable.WhenJisSymbolThenCallingScope=TRUE)</code>. It will then work as you expected, just like data.frame. If you are a new user of data.table, you should probably do this. You can place this command in your .Rprofile file so you don’t have to remember again. See the long item in release notes about this. The release notes are linked at the top of the data.table homepage: <a href="https://github.com/Rdatatable/data.table/blob/master/NEWS.md">NEWS</a>.</p> <p>Without turning on that new behaviour, what’s happening is that the <code>j</code> expression sees objects in the calling scope. The variable <code>mycol</code> does not exist as a column name of <code>DT</code> so data.table then looked in the calling scope and found <code>mycol</code> there and returned its value <code>"x"</code>. This is correct behaviour currently. Had <code>mycol</code> been a column name, then that column’s data would have been returned. What has been done to date has been <code>DT[ , mycol, with = FALSE]</code> which will return the <code>x</code> column’s data as required. That will still work in the future, too. Alternatively, since a data.table <em>is</em> a <code>list</code>, too, you have been and still will be able to write and rely on <code>DT[[mycol]]</code>.</p> </div> <div id="what-are-the-benefits-of-being-able-to-use-column-names-as-if-they-are-variables-inside-dt..." class="section level2" number="1.6"> <h2 number="1.6"><span class="header-section-number">1.6</span> What are the benefits of being able to use column names as if they are variables inside <code>DT[...]</code>?</h2> <p><code>j</code> doesn’t have to be just column names. You can write any R <em>expression</em> of column names directly in <code>j</code>, <em>e.g.</em>, <code>DT[ , mean(x*y/z)]</code>. The same applies to <code>i</code>, <em>e.g.</em>, <code>DT[x>1000, sum(y*z)]</code>.</p> <p>This runs the <code>j</code> expression on the set of rows where the <code>i</code> expression is true. You don’t even need to return data, <em>e.g.</em>, <code>DT[x>1000, plot(y, z)]</code>. You can do <code>j</code> by group simply by adding <code>by =</code>; e.g., <code>DT[x>1000, sum(y*z), by = w]</code>. This runs <code>j</code> for each group in column <code>w</code> but just over the rows where <code>x>1000</code>. By placing the 3 parts of the query (i=where, j=select and by=group by) inside the square brackets, data.table sees this query as a whole before any part of it is evaluated. Thus it can optimize the combined query for performance. It can do this because the R language uniquely has lazy evaluation (Python and Julia do not). data.table sees the expressions inside <code>DT[...]</code> before they are evaluated and optimizes them before evaluation. For example, if data.table see that you’re only using 2 columns out of 100, it won’t bother to subset the 98 that aren’t needed by your j expression.</p> </div> <div id="ok-im-starting-to-see-what-data.table-is-about-but-why-didnt-you-just-enhance-data.frame-in-r-why-does-it-have-to-be-a-new-package" class="section level2" number="1.7"> <h2 number="1.7"><span class="header-section-number">1.7</span> OK, I’m starting to see what data.table is about, but why didn’t you just enhance <code>data.frame</code> in R? Why does it have to be a new package?</h2> <p>As <a href="#j-num">highlighted above</a>, <code>j</code> in <code>[.data.table</code> is fundamentally different from <code>j</code> in <code>[.data.frame</code>. Even if something as simple as <code>DF[ , 1]</code> was changed in base R to return a data.frame rather than a vector, that would break existing code in many 1000’s of CRAN packages and user code. As soon as we took the step to create a new class that inherited from data.frame, we had the opportunity to change a few things and we did. We want data.table to be slightly different and to work this way for more complicated syntax to work. There are other differences, too (see <a href="#SmallerDiffs">below</a> ).</p> <p>Furthermore, data.table <em>inherits</em> from <code>data.frame</code>. It <em>is</em> a <code>data.frame</code>, too. A data.table can be passed to any package that only accepts <code>data.frame</code> and that package can use <code>[.data.frame</code> syntax on the data.table. See <a href="https://stackoverflow.com/a/10529888/403310">this answer</a> for how that is achieved.</p> <p>We <em>have</em> proposed enhancements to R wherever possible, too. One of these was accepted as a new feature in R 2.12.0 :</p> <blockquote> <p><code>unique()</code> and <code>match()</code> are now faster on character vectors where all elements are in the global CHARSXP cache and have unmarked encoding (ASCII). Thanks to Matt Dowle for suggesting improvements to the way the hash code is generated in unique.c.</p> </blockquote> <p>A second proposal was to use <code>memcpy</code> in duplicate.c, which is much faster than a for loop in C. This would improve the <em>way</em> that R copies data internally (on some measures by 13 times). The thread on r-devel is <a href="https://stat.ethz.ch/pipermail/r-devel/2010-April/057249.html">here</a>.</p> <p>A third more significant proposal that was accepted is that R now uses data.table’s radix sort code as from R 3.3.0 :</p> <blockquote> <p>The radix sort algorithm and implementation from data.table (forder) replaces the previous radix (counting) sort and adds a new method for order(). Contributed by Matt Dowle and Arun Srinivasan, the new algorithm supports logical, integer (even with large values), real, and character vectors. It outperforms all other methods, but there are some caveats (see ?sort).</p> </blockquote> <p>This was big event for us and we celebrated until the cows came home. (Not really.)</p> </div> <div id="why-are-the-defaults-the-way-they-are-why-does-it-work-the-way-it-does" class="section level2" number="1.8"> <h2 number="1.8"><span class="header-section-number">1.8</span> Why are the defaults the way they are? Why does it work the way it does?</h2> <p>The simple answer is because the main author originally designed it for his own use. He wanted it that way. He finds it a more natural, faster way to write code, which also executes more quickly.</p> </div> <div id="isnt-this-already-done-by-with-and-subset-in-base" class="section level2" number="1.9"> <h2 number="1.9"><span class="header-section-number">1.9</span> Isn’t this already done by <code>with()</code> and <code>subset()</code> in <code>base</code>?</h2> <p>Some of the features discussed so far are, yes. The package builds upon base functionality. It does the same sorts of things but with less code required and executes many times faster if used correctly.</p> </div> <div id="why-does-xy-return-all-the-columns-from-y-too-shouldnt-it-return-a-subset-of-x" class="section level2" number="1.10"> <h2 number="1.10"><span class="header-section-number">1.10</span> Why does <code>X[Y]</code> return all the columns from <code>Y</code> too? Shouldn’t it return a subset of <code>X</code>?</h2> <p>This was changed in v1.5.3 (Feb 2011). Since then <code>X[Y]</code> includes <code>Y</code>’s non-join columns. We refer to this feature as <em>join inherited scope</em> because not only are <code>X</code> columns available to the <code>j</code> expression, so are <code>Y</code> columns. The downside is that <code>X[Y]</code> is less efficient since every item of <code>Y</code>’s non-join columns are duplicated to match the (likely large) number of rows in <code>X</code> that match. We therefore strongly encourage <code>X[Y, j]</code> instead of <code>X[Y]</code>. See <a href="#MergeDiff">next FAQ</a>.</p> </div> <div id="MergeDiff" class="section level2" number="1.11"> <h2 number="1.11"><span class="header-section-number">1.11</span> What is the difference between <code>X[Y]</code> and <code>merge(X, Y)</code>?</h2> <p><code>X[Y]</code> is a join, looking up <code>X</code>’s rows using <code>Y</code> (or <code>Y</code>’s key if it has one) as an index.</p> <p><code>Y[X]</code> is a join, looking up <code>Y</code>’s rows using <code>X</code> (or <code>X</code>’s key if it has one) as an index.</p> <p><code>merge(X,Y)</code><a href="#fn1" class="footnote-ref" id="fnref1"><sup>1</sup></a> does both ways at the same time. The number of rows of <code>X[Y]</code> and <code>Y[X]</code> usually differ, whereas the number of rows returned by <code>merge(X, Y)</code> and <code>merge(Y, X)</code> is the same.</p> <p><em>BUT</em> that misses the main point. Most tasks require something to be done on the data after a join or merge. Why merge all the columns of data, only to use a small subset of them afterwards? You may suggest <code>merge(X[ , ColsNeeded1], Y[ , ColsNeeded2])</code>, but that requires the programmer to work out which columns are needed. <code>X[Y, j]</code> in data.table does all that in one step for you. When you write <code>X[Y, sum(foo*bar)]</code>, data.table automatically inspects the <code>j</code> expression to see which columns it uses. It will subset those columns only; the others are ignored. Memory is only created for the columns <code>j</code> uses and <code>Y</code> columns enjoy standard R recycling rules within the context of each group. Let’s say <code>foo</code> is in <code>X</code> and <code>bar</code> is in <code>Y</code> (along with 20 other columns in <code>Y</code>). Isn’t <code>X[Y, sum(foo*bar)]</code> quicker to program and quicker to run than a <code>merge</code> of everything wastefully followed by a <code>subset</code>?</p> </div> <div id="anything-else-about-xy-sumfoobar" class="section level2" number="1.12"> <h2 number="1.12"><span class="header-section-number">1.12</span> Anything else about <code>X[Y, sum(foo*bar)]</code>?</h2> <p>This behaviour changed in v1.9.4 (Sep 2014). It now does the <code>X[Y]</code> join and then runs <code>sum(foo*bar)</code> over all the rows; i.e., <code>X[Y][ , sum(foo*bar)]</code>. It used to run <code>j</code> for each <em>group</em> of <code>X</code> that each row of <code>Y</code> matches to. That can still be done as it’s very useful but you now need to be explicit and specify <code>by = .EACHI</code>, <em>i.e.</em>, <code>X[Y, sum(foo*bar), by = .EACHI]</code>. We call this <em>grouping by each <code>i</code></em>.</p> <p>For example, (further complicating it by using <em>join inherited scope</em>, too):</p> <div class="sourceCode" id="cb1"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true"></a>X =<span class="st"> </span><span class="kw">data.table</span>(<span class="dt">grp =</span> <span class="kw">c</span>(<span class="st">"a"</span>, <span class="st">"a"</span>, <span class="st">"b"</span>,</span> <span id="cb1-2"><a href="#cb1-2" aria-hidden="true"></a> <span class="st">"b"</span>, <span class="st">"b"</span>, <span class="st">"c"</span>, <span class="st">"c"</span>), <span class="dt">foo =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">7</span>)</span> <span id="cb1-3"><a href="#cb1-3" aria-hidden="true"></a><span class="kw">setkey</span>(X, grp)</span> <span id="cb1-4"><a href="#cb1-4" aria-hidden="true"></a>Y =<span class="st"> </span><span class="kw">data.table</span>(<span class="kw">c</span>(<span class="st">"b"</span>, <span class="st">"c"</span>), <span class="dt">bar =</span> <span class="kw">c</span>(<span class="dv">4</span>, <span class="dv">2</span>))</span> <span id="cb1-5"><a href="#cb1-5" aria-hidden="true"></a>X</span> <span id="cb1-6"><a href="#cb1-6" aria-hidden="true"></a><span class="co"># grp foo</span></span> <span id="cb1-7"><a href="#cb1-7" aria-hidden="true"></a><span class="co"># 1: a 1</span></span> <span id="cb1-8"><a href="#cb1-8" aria-hidden="true"></a><span class="co"># 2: a 2</span></span> <span id="cb1-9"><a href="#cb1-9" aria-hidden="true"></a><span class="co"># 3: b 3</span></span> <span id="cb1-10"><a href="#cb1-10" aria-hidden="true"></a><span class="co"># 4: b 4</span></span> <span id="cb1-11"><a href="#cb1-11" aria-hidden="true"></a><span class="co"># 5: b 5</span></span> <span id="cb1-12"><a href="#cb1-12" aria-hidden="true"></a><span class="co"># 6: c 6</span></span> <span id="cb1-13"><a href="#cb1-13" aria-hidden="true"></a><span class="co"># 7: c 7</span></span> <span id="cb1-14"><a href="#cb1-14" aria-hidden="true"></a>Y</span> <span id="cb1-15"><a href="#cb1-15" aria-hidden="true"></a><span class="co"># V1 bar</span></span> <span id="cb1-16"><a href="#cb1-16" aria-hidden="true"></a><span class="co"># 1: b 4</span></span> <span id="cb1-17"><a href="#cb1-17" aria-hidden="true"></a><span class="co"># 2: c 2</span></span> <span id="cb1-18"><a href="#cb1-18" aria-hidden="true"></a>X[Y, <span class="kw">sum</span>(foo<span class="op">*</span>bar)]</span> <span id="cb1-19"><a href="#cb1-19" aria-hidden="true"></a><span class="co"># [1] 74</span></span> <span id="cb1-20"><a href="#cb1-20" aria-hidden="true"></a>X[Y, <span class="kw">sum</span>(foo<span class="op">*</span>bar), by =<span class="st"> </span>.EACHI]</span> <span id="cb1-21"><a href="#cb1-21" aria-hidden="true"></a><span class="co"># grp V1</span></span> <span id="cb1-22"><a href="#cb1-22" aria-hidden="true"></a><span class="co"># 1: b 48</span></span> <span id="cb1-23"><a href="#cb1-23" aria-hidden="true"></a><span class="co"># 2: c 26</span></span></code></pre></div> </div> <div id="thats-nice.-how-did-you-manage-to-change-it-given-that-users-depended-on-the-old-behaviour" class="section level2" number="1.13"> <h2 number="1.13"><span class="header-section-number">1.13</span> That’s nice. How did you manage to change it given that users depended on the old behaviour?</h2> <p>The request to change came from users. The feeling was that if a query is doing grouping then an explicit <code>by=</code> should be present for code readability reasons. An option was provided to return the old behaviour: <code>options(datatable.old.bywithoutby)</code>, by default <code>FALSE</code>. This enabled upgrading to test the other new features / bug fixes in v1.9.4, with later migration of any by-without-by queries when ready by adding <code>by=.EACHI</code> to them. We retained 47 pre-change tests and added them back as new tests, tested under <code>options(datatable.old.bywithoutby=TRUE)</code>. We added a startup message about the change and how to revert to the old behaviour. After 1 year the option was deprecated with warning when used. After 2 years the option to revert to old behaviour was removed.</p> <p>Of the 66 packages on CRAN or Bioconductor that depended on or import data.table at the time of releasing v1.9.4 (it is now over 300), only one was affected by the change. That could be because many packages don’t have comprehensive tests, or just that grouping by each row in <code>i</code> wasn’t being used much by downstream packages. We always test the new version with all dependent packages before release and coordinate any changes with those maintainers. So this release was quite straightforward in that regard.</p> <p>Another compelling reason to make the change was that previously, there was no efficient way to achieve what <code>X[Y, sum(foo*bar)]</code> does now. You had to write <code>X[Y][ , sum(foo*bar)]</code>. That was suboptimal because <code>X[Y]</code> joined all the columns and passed them all to the second compound query without knowing that only <code>foo</code> and <code>bar</code> are needed. To solve that efficiency problem, extra programming effort was required: <code>X[Y, list(foo, bar)][ , sum(foo*bar)]</code>. The change to <code>by = .EACHI</code> has simplified this by allowing both queries to be expressed inside a single <code>DT[...]</code> query for efficiency.</p> </div> </div> <div id="general-syntax" class="section level1" number="2"> <h1 number="2"><span class="header-section-number">2</span> General Syntax</h1> <div id="how-can-i-avoid-writing-a-really-long-j-expression-youve-said-that-i-should-use-the-column-names-but-ive-got-a-lot-of-columns." class="section level2" number="2.1"> <h2 number="2.1"><span class="header-section-number">2.1</span> How can I avoid writing a really long <code>j</code> expression? You’ve said that I should use the column <em>names</em>, but I’ve got a lot of columns.</h2> <p>When grouping, the <code>j</code> expression can use column names as variables, as you know, but it can also use a reserved symbol <code>.SD</code> which refers to the <strong>S</strong>ubset of the <strong>D</strong>ata.table for each group (excluding the grouping columns). So to sum up all your columns it’s just <code>DT[ , lapply(.SD, sum), by = grp]</code>. It might seem tricky, but it’s fast to write and fast to run. Notice you don’t have to create an anonymous function. The <code>.SD</code> object is efficiently implemented internally and more efficient than passing an argument to a function. But if the <code>.SD</code> symbol appears in <code>j</code> then data.table has to populate <code>.SD</code> fully for each group even if <code>j</code> doesn’t use all of it.</p> <p>So please don’t do, for example, <code>DT[ , sum(.SD[["sales"]]), by = grp]</code>. That works but is inefficient and inelegant. <code>DT[ , sum(sales), by = grp]</code> is what was intended, and it could be 100s of times faster. If you use <em>all</em> of the data in <code>.SD</code> for each group (such as in <code>DT[ , lapply(.SD, sum), by = grp]</code>) then that’s very good usage of <code>.SD</code>. If you’re using <em>several</em> but not <em>all</em> of the columns, you can combine <code>.SD</code> with <code>.SDcols</code>; see <code>?data.table</code>.</p> </div> <div id="why-is-the-default-for-mult-now-all" class="section level2" number="2.2"> <h2 number="2.2"><span class="header-section-number">2.2</span> Why is the default for <code>mult</code> now <code>"all"</code>?</h2> <p>In v1.5.3 the default was changed to <code>"all"</code>. When <code>i</code> (or <code>i</code>’s key if it has one) has fewer columns than <code>x</code>’s key, <code>mult</code> was already set to <code>"all"</code> automatically. Changing the default makes this clearer and easier for users as it came up quite often.</p> <p>In versions up to v1.3, <code>"all"</code> was slower. Internally, <code>"all"</code> was implemented by joining using <code>"first"</code>, then again from scratch using <code>"last"</code>, after which a diff between them was performed to work out the span of the matches in <code>x</code> for each row in <code>i</code>. Most often we join to single rows, though, where <code>"first"</code>,<code>"last"</code> and <code>"all"</code> return the same result. We preferred maximum performance for the majority of situations so the default chosen was <code>"first"</code>. When working with a non-unique key (generally a single column containing a grouping variable), <code>DT["A"]</code> returned the first row of that group so <code>DT["A", mult = "all"]</code> was needed to return all the rows in that group.</p> <p>In v1.4 the binary search in C was changed to branch at the deepest level to find first and last. That branch will likely occur within the same final pages of RAM so there should no longer be a speed disadvantage in defaulting <code>mult</code> to <code>"all"</code>. We warned that the default might change and made the change in v1.5.3.</p> <p>A future version of data.table may allow a distinction between a key and a <em>unique key</em>. Internally <code>mult = "all"</code> would perform more like <code>mult = "first"</code> when all <code>x</code>’s key columns were joined to and <code>x</code>’s key was a unique key. data.table would need checks on insert and update to make sure a unique key is maintained. An advantage of specifying a unique key would be that data.table would ensure no duplicates could be inserted, in addition to performance.</p> </div> <div id="im-using-c-in-j-and-getting-strange-results." class="section level2" number="2.3"> <h2 number="2.3"><span class="header-section-number">2.3</span> I’m using <code>c()</code> in <code>j</code> and getting strange results.</h2> <p>This is a common source of confusion. In <code>data.frame</code> you are used to, for example:</p> <div class="sourceCode" id="cb2"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true"></a>DF =<span class="st"> </span><span class="kw">data.frame</span>(<span class="dt">x =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">3</span>, <span class="dt">y =</span> <span class="dv">4</span><span class="op">:</span><span class="dv">6</span>, <span class="dt">z =</span> <span class="dv">7</span><span class="op">:</span><span class="dv">9</span>)</span> <span id="cb2-2"><a href="#cb2-2" aria-hidden="true"></a>DF</span> <span id="cb2-3"><a href="#cb2-3" aria-hidden="true"></a><span class="co"># x y z</span></span> <span id="cb2-4"><a href="#cb2-4" aria-hidden="true"></a><span class="co"># 1 1 4 7</span></span> <span id="cb2-5"><a href="#cb2-5" aria-hidden="true"></a><span class="co"># 2 2 5 8</span></span> <span id="cb2-6"><a href="#cb2-6" aria-hidden="true"></a><span class="co"># 3 3 6 9</span></span> <span id="cb2-7"><a href="#cb2-7" aria-hidden="true"></a>DF[ , <span class="kw">c</span>(<span class="st">"y"</span>, <span class="st">"z"</span>)]</span> <span id="cb2-8"><a href="#cb2-8" aria-hidden="true"></a><span class="co"># y z</span></span> <span id="cb2-9"><a href="#cb2-9" aria-hidden="true"></a><span class="co"># 1 4 7</span></span> <span id="cb2-10"><a href="#cb2-10" aria-hidden="true"></a><span class="co"># 2 5 8</span></span> <span id="cb2-11"><a href="#cb2-11" aria-hidden="true"></a><span class="co"># 3 6 9</span></span></code></pre></div> <p>which returns the two columns. In data.table you know you can use the column names directly and might try:</p> <div class="sourceCode" id="cb3"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true"></a>DT =<span class="st"> </span><span class="kw">data.table</span>(DF)</span> <span id="cb3-2"><a href="#cb3-2" aria-hidden="true"></a>DT[ , <span class="kw">c</span>(y, z)]</span> <span id="cb3-3"><a href="#cb3-3" aria-hidden="true"></a><span class="co"># [1] 4 5 6 7 8 9</span></span></code></pre></div> <p>but this returns one vector. Remember that the <code>j</code> expression is evaluated within the environment of <code>DT</code> and <code>c()</code> returns a vector. If 2 or more columns are required, use <code>list()</code> or <code>.()</code> instead:</p> <div class="sourceCode" id="cb4"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true"></a>DT[ , .(y, z)]</span> <span id="cb4-2"><a href="#cb4-2" aria-hidden="true"></a><span class="co"># y z</span></span> <span id="cb4-3"><a href="#cb4-3" aria-hidden="true"></a><span class="co"># 1: 4 7</span></span> <span id="cb4-4"><a href="#cb4-4" aria-hidden="true"></a><span class="co"># 2: 5 8</span></span> <span id="cb4-5"><a href="#cb4-5" aria-hidden="true"></a><span class="co"># 3: 6 9</span></span></code></pre></div> <p><code>c()</code> can be useful in a data.table too, but its behaviour is different from that in <code>[.data.frame</code>.</p> </div> <div id="i-have-built-up-a-complex-table-with-many-columns.-i-want-to-use-it-as-a-template-for-a-new-table-i.e.-create-a-new-table-with-no-rows-but-with-the-column-names-and-types-copied-from-my-table.-can-i-do-that-easily" class="section level2" number="2.4"> <h2 number="2.4"><span class="header-section-number">2.4</span> I have built up a complex table with many columns. I want to use it as a template for a new table; <em>i.e.</em>, create a new table with no rows, but with the column names and types copied from my table. Can I do that easily?</h2> <p>Yes. If your complex table is called <code>DT</code>, try <code>NEWDT = DT[0]</code>.</p> </div> <div id="is-a-null-data.table-the-same-as-dt0" class="section level2" number="2.5"> <h2 number="2.5"><span class="header-section-number">2.5</span> Is a null data.table the same as <code>DT[0]</code>?</h2> <p>No. By “null data.table” we mean the result of <code>data.table(NULL)</code> or <code>as.data.table(NULL)</code>; <em>i.e.</em>,</p> <div class="sourceCode" id="cb5"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true"></a><span class="kw">data.table</span>(<span class="ot">NULL</span>)</span> <span id="cb5-2"><a href="#cb5-2" aria-hidden="true"></a><span class="co"># Null data.table (0 rows and 0 cols)</span></span> <span id="cb5-3"><a href="#cb5-3" aria-hidden="true"></a><span class="kw">data.frame</span>(<span class="ot">NULL</span>)</span> <span id="cb5-4"><a href="#cb5-4" aria-hidden="true"></a><span class="co"># data frame with 0 columns and 0 rows</span></span> <span id="cb5-5"><a href="#cb5-5" aria-hidden="true"></a><span class="kw">as.data.table</span>(<span class="ot">NULL</span>)</span> <span id="cb5-6"><a href="#cb5-6" aria-hidden="true"></a><span class="co"># Null data.table (0 rows and 0 cols)</span></span> <span id="cb5-7"><a href="#cb5-7" aria-hidden="true"></a><span class="kw">as.data.frame</span>(<span class="ot">NULL</span>)</span> <span id="cb5-8"><a href="#cb5-8" aria-hidden="true"></a><span class="co"># data frame with 0 columns and 0 rows</span></span> <span id="cb5-9"><a href="#cb5-9" aria-hidden="true"></a><span class="kw">is.null</span>(<span class="kw">data.table</span>(<span class="ot">NULL</span>))</span> <span id="cb5-10"><a href="#cb5-10" aria-hidden="true"></a><span class="co"># [1] FALSE</span></span> <span id="cb5-11"><a href="#cb5-11" aria-hidden="true"></a><span class="kw">is.null</span>(<span class="kw">data.frame</span>(<span class="ot">NULL</span>))</span> <span id="cb5-12"><a href="#cb5-12" aria-hidden="true"></a><span class="co"># [1] FALSE</span></span></code></pre></div> <p>The null data.table|<code>frame</code> is <code>NULL</code> with some attributes attached, which means it’s no longer <code>NULL</code>. In R only pure <code>NULL</code> is <code>NULL</code> as tested by <code>is.null()</code>. When referring to the “null data.table” we use lower case null to help distinguish from upper case <code>NULL</code>. To test for the null data.table, use <code>length(DT) == 0</code> or <code>ncol(DT) == 0</code> (<code>length</code> is slightly faster as it’s a primitive function).</p> <p>An <em>empty</em> data.table (<code>DT[0]</code>) has one or more columns, all of which are empty. Those empty columns still have names and types.</p> <div class="sourceCode" id="cb6"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true"></a>DT =<span class="st"> </span><span class="kw">data.table</span>(<span class="dt">a =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">3</span>, <span class="dt">b =</span> <span class="kw">c</span>(<span class="dv">4</span>, <span class="dv">5</span>, <span class="dv">6</span>), <span class="dt">d =</span> <span class="kw">c</span>(7L,8L,9L))</span> <span id="cb6-2"><a href="#cb6-2" aria-hidden="true"></a>DT[<span class="dv">0</span>]</span> <span id="cb6-3"><a href="#cb6-3" aria-hidden="true"></a><span class="co"># Empty data.table (0 rows and 3 cols): a,b,d</span></span> <span id="cb6-4"><a href="#cb6-4" aria-hidden="true"></a><span class="kw">sapply</span>(DT[<span class="dv">0</span>], class)</span> <span id="cb6-5"><a href="#cb6-5" aria-hidden="true"></a><span class="co"># a b d </span></span> <span id="cb6-6"><a href="#cb6-6" aria-hidden="true"></a><span class="co"># "integer" "numeric" "integer"</span></span></code></pre></div> </div> <div id="DTremove1" class="section level2" number="2.6"> <h2 number="2.6"><span class="header-section-number">2.6</span> Why has the <code>DT()</code> alias been removed?</h2> <p><code>DT</code> was introduced originally as a wrapper for a list of <code>j</code>expressions. Since <code>DT</code> was an alias for data.table, this was a convenient way to take care of silent recycling in cases where each item of the <code>j</code> list evaluated to different lengths. The alias was one reason grouping was slow, though.</p> <p>As of v1.3, <code>list()</code> or <code>.()</code> should be passed instead to the <code>j</code> argument. These are much faster, especially when there are many groups. Internally, this was a non-trivial change. Vector recycling is now done internally, along with several other speed enhancements for grouping.</p> </div> <div id="DTremove2" class="section level2" number="2.7"> <h2 number="2.7"><span class="header-section-number">2.7</span> But my code uses <code>j = DT(...)</code> and it works. The previous FAQ says that <code>DT()</code> has been removed.</h2> <p>Then you are using a version prior to 1.5.3. Prior to 1.5.3 <code>[.data.table</code> detected use of <code>DT()</code> in the <code>j</code> and automatically replaced it with a call to <code>list()</code>. This was to help the transition for existing users.</p> </div> <div id="what-are-the-scoping-rules-for-j-expressions" class="section level2" number="2.8"> <h2 number="2.8"><span class="header-section-number">2.8</span> What are the scoping rules for <code>j</code> expressions?</h2> <p>Think of the subset as an environment where all the column names are variables. When a variable <code>foo</code> is used in the <code>j</code> of a query such as <code>X[Y, sum(foo)]</code>, <code>foo</code> is looked for in the following order :</p> <ol style="list-style-type: decimal"> <li>The scope of <code>X</code>’s subset; <em>i.e.</em>, <code>X</code>’s column names.</li> <li>The scope of each row of <code>Y</code>; <em>i.e.</em>, <code>Y</code>’s column names (<em>join inherited scope</em>)</li> <li>The scope of the calling frame; <em>e.g.</em>, the line that appears before the data.table query.</li> <li>Exercise for reader: does it then ripple up the calling frames, or go straight to <code>globalenv()</code>?</li> <li>The global environment</li> </ol> <p>This is <em>lexical scoping</em> as explained in <a href="https://cran.r-project.org/doc/FAQ/R-FAQ.html#Lexical-scoping">R FAQ 3.3.1</a>. The environment in which the function was created is not relevant, though, because there is <em>no function</em>. No anonymous <em>function</em> is passed to <code>j</code>. Instead, an anonymous <em>body</em> is passed to <code>j</code>; for example,</p> <div class="sourceCode" id="cb7"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true"></a>DT =<span class="st"> </span><span class="kw">data.table</span>(<span class="dt">x =</span> <span class="kw">rep</span>(<span class="kw">c</span>(<span class="st">"a"</span>, <span class="st">"b"</span>), <span class="kw">c</span>(<span class="dv">2</span>, <span class="dv">3</span>)), <span class="dt">y =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">5</span>)</span> <span id="cb7-2"><a href="#cb7-2" aria-hidden="true"></a>DT</span> <span id="cb7-3"><a href="#cb7-3" aria-hidden="true"></a><span class="co"># x y</span></span> <span id="cb7-4"><a href="#cb7-4" aria-hidden="true"></a><span class="co"># 1: a 1</span></span> <span id="cb7-5"><a href="#cb7-5" aria-hidden="true"></a><span class="co"># 2: a 2</span></span> <span id="cb7-6"><a href="#cb7-6" aria-hidden="true"></a><span class="co"># 3: b 3</span></span> <span id="cb7-7"><a href="#cb7-7" aria-hidden="true"></a><span class="co"># 4: b 4</span></span> <span id="cb7-8"><a href="#cb7-8" aria-hidden="true"></a><span class="co"># 5: b 5</span></span> <span id="cb7-9"><a href="#cb7-9" aria-hidden="true"></a>DT[ , {z =<span class="st"> </span><span class="kw">sum</span>(y); z <span class="op">+</span><span class="st"> </span><span class="dv">3</span>}, by =<span class="st"> </span>x]</span> <span id="cb7-10"><a href="#cb7-10" aria-hidden="true"></a><span class="co"># x V1</span></span> <span id="cb7-11"><a href="#cb7-11" aria-hidden="true"></a><span class="co"># 1: a 6</span></span> <span id="cb7-12"><a href="#cb7-12" aria-hidden="true"></a><span class="co"># 2: b 15</span></span></code></pre></div> <p>Some programming languages call this a <em>lambda</em>.</p> </div> <div id="j-trace" class="section level2" number="2.9"> <h2 number="2.9"><span class="header-section-number">2.9</span> Can I trace the <code>j</code> expression as it runs through the groups?</h2> <p>Try something like this:</p> <div class="sourceCode" id="cb8"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true"></a>DT[ , {</span> <span id="cb8-2"><a href="#cb8-2" aria-hidden="true"></a> <span class="kw">cat</span>(<span class="st">"Objects:"</span>, <span class="kw">paste</span>(<span class="kw">objects</span>(), <span class="dt">collapse =</span> <span class="st">","</span>), <span class="st">"</span><span class="ch">\n</span><span class="st">"</span>)</span> <span id="cb8-3"><a href="#cb8-3" aria-hidden="true"></a> <span class="kw">cat</span>(<span class="st">"Trace: x="</span>, <span class="kw">as.character</span>(x), <span class="st">" y="</span>, y, <span class="st">"</span><span class="ch">\n</span><span class="st">"</span>)</span> <span id="cb8-4"><a href="#cb8-4" aria-hidden="true"></a> <span class="kw">sum</span>(y)},</span> <span id="cb8-5"><a href="#cb8-5" aria-hidden="true"></a> by =<span class="st"> </span>x]</span> <span id="cb8-6"><a href="#cb8-6" aria-hidden="true"></a><span class="co"># Objects: -.POSIXt,Cfastmean,print,strptime,x,y </span></span> <span id="cb8-7"><a href="#cb8-7" aria-hidden="true"></a><span class="co"># Trace: x= a y= 1 2 </span></span> <span id="cb8-8"><a href="#cb8-8" aria-hidden="true"></a><span class="co"># Objects: -.POSIXt,Cfastmean,print,strptime,x,y </span></span> <span id="cb8-9"><a href="#cb8-9" aria-hidden="true"></a><span class="co"># Trace: x= b y= 3 4 5</span></span> <span id="cb8-10"><a href="#cb8-10" aria-hidden="true"></a><span class="co"># x V1</span></span> <span id="cb8-11"><a href="#cb8-11" aria-hidden="true"></a><span class="co"># 1: a 3</span></span> <span id="cb8-12"><a href="#cb8-12" aria-hidden="true"></a><span class="co"># 2: b 12</span></span></code></pre></div> </div> <div id="inside-each-group-why-are-the-group-variables-length-1" class="section level2" number="2.10"> <h2 number="2.10"><span class="header-section-number">2.10</span> Inside each group, why are the group variables length-1?</h2> <p><a href="#j-trace">Above</a>, <code>x</code> is a grouping variable and (as from v1.6.1) has <code>length</code> 1 (if inspected or used in <code>j</code>). It’s for efficiency and convenience. Therefore, there is no difference between the following two statements:</p> <div class="sourceCode" id="cb9"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true"></a>DT[ , .(<span class="dt">g =</span> <span class="dv">1</span>, <span class="dt">h =</span> <span class="dv">2</span>, <span class="dt">i =</span> <span class="dv">3</span>, <span class="dt">j =</span> <span class="dv">4</span>, <span class="dt">repeatgroupname =</span> x, <span class="kw">sum</span>(y)), by =<span class="st"> </span>x]</span> <span id="cb9-2"><a href="#cb9-2" aria-hidden="true"></a><span class="co"># x g h i j repeatgroupname V6</span></span> <span id="cb9-3"><a href="#cb9-3" aria-hidden="true"></a><span class="co"># 1: a 1 2 3 4 a 3</span></span> <span id="cb9-4"><a href="#cb9-4" aria-hidden="true"></a><span class="co"># 2: b 1 2 3 4 b 12</span></span> <span id="cb9-5"><a href="#cb9-5" aria-hidden="true"></a>DT[ , .(<span class="dt">g =</span> <span class="dv">1</span>, <span class="dt">h =</span> <span class="dv">2</span>, <span class="dt">i =</span> <span class="dv">3</span>, <span class="dt">j =</span> <span class="dv">4</span>, <span class="dt">repeatgroupname =</span> x[<span class="dv">1</span>], <span class="kw">sum</span>(y)), by =<span class="st"> </span>x]</span> <span id="cb9-6"><a href="#cb9-6" aria-hidden="true"></a><span class="co"># x g h i j repeatgroupname V6</span></span> <span id="cb9-7"><a href="#cb9-7" aria-hidden="true"></a><span class="co"># 1: a 1 2 3 4 a 3</span></span> <span id="cb9-8"><a href="#cb9-8" aria-hidden="true"></a><span class="co"># 2: b 1 2 3 4 b 12</span></span></code></pre></div> <p>If you need the size of the current group, use <code>.N</code> rather than calling <code>length()</code> on any column.</p> </div> <div id="only-the-first-10-rows-are-printed-how-do-i-print-more" class="section level2" number="2.11"> <h2 number="2.11"><span class="header-section-number">2.11</span> Only the first 10 rows are printed, how do I print more?</h2> <p>There are two things happening here. First, if the number of rows in a data.table are large (<code>> 100</code> by default), then a summary of the data.table is printed to the console by default. Second, the summary of a large data.table is printed by taking the top and bottom <code>n</code> (<code>= 5</code> by default) rows of the data.table and only printing those. Both of these parameters (when to trigger a summary and how much of a table to use as a summary) are configurable by R’s <code>options</code> mechanism, or by calling the <code>print</code> function directly.</p> <p>For instance, to enforce the summary of a data.table to only happen when a data.table is greater than 50 rows, you could <code>options(datatable.print.nrows = 50)</code>. To disable the summary-by-default completely, you could <code>options(datatable.print.nrows = Inf)</code>. You could also call <code>print</code> directly, as in <code>print(your.data.table, nrows = Inf)</code>.</p> <p>If you want to show more than just the top (and bottom) 10 rows of a data.table summary (say you like 20), set <code>options(datatable.print.topn = 20)</code>, for example. Again, you could also just call <code>print</code> directly, as in <code>print(your.data.table, topn = 20)</code>.</p> </div> <div id="with-an-xy-join-what-if-x-contains-a-column-called-y" class="section level2" number="2.12"> <h2 number="2.12"><span class="header-section-number">2.12</span> With an <code>X[Y]</code> join, what if <code>X</code> contains a column called <code>"Y"</code>?</h2> <p>When <code>i</code> is a single name such as <code>Y</code> it is evaluated in the calling frame. In all other cases such as calls to <code>.()</code> or other expressions, <code>i</code> is evaluated within the scope of <code>X</code>. This facilitates easy <em>self-joins</em> such as <code>X[J(unique(colA)), mult = "first"]</code>.</p> </div> <div id="xzy-is-failing-because-x-contains-a-column-y.-id-like-it-to-use-the-table-y-in-calling-scope." class="section level2" number="2.13"> <h2 number="2.13"><span class="header-section-number">2.13</span> <code>X[Z[Y]]</code> is failing because <code>X</code> contains a column <code>"Y"</code>. I’d like it to use the table <code>Y</code> in calling scope.</h2> <p>The <code>Z[Y]</code> part is not a single name so that is evaluated within the frame of <code>X</code> and the problem occurs. Try <code>tmp = Z[Y]; X[tmp]</code>. This is robust to <code>X</code> containing a column <code>"tmp"</code> because <code>tmp</code> is a single name. If you often encounter conflicts of this type, one simple solution may be to name all tables in uppercase and all column names in lowercase, or some similar scheme.</p> </div> <div id="can-you-explain-further-why-data.table-is-inspired-by-ab-syntax-in-base" class="section level2" number="2.14"> <h2 number="2.14"><span class="header-section-number">2.14</span> Can you explain further why data.table is inspired by <code>A[B]</code> syntax in <code>base</code>?</h2> <p>Consider <code>A[B]</code> syntax using an example matrix <code>A</code> :</p> <div class="sourceCode" id="cb10"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb10-1"><a href="#cb10-1" aria-hidden="true"></a>A =<span class="st"> </span><span class="kw">matrix</span>(<span class="dv">1</span><span class="op">:</span><span class="dv">12</span>, <span class="dt">nrow =</span> <span class="dv">4</span>)</span> <span id="cb10-2"><a href="#cb10-2" aria-hidden="true"></a>A</span> <span id="cb10-3"><a href="#cb10-3" aria-hidden="true"></a><span class="co"># [,1] [,2] [,3]</span></span> <span id="cb10-4"><a href="#cb10-4" aria-hidden="true"></a><span class="co"># [1,] 1 5 9</span></span> <span id="cb10-5"><a href="#cb10-5" aria-hidden="true"></a><span class="co"># [2,] 2 6 10</span></span> <span id="cb10-6"><a href="#cb10-6" aria-hidden="true"></a><span class="co"># [3,] 3 7 11</span></span> <span id="cb10-7"><a href="#cb10-7" aria-hidden="true"></a><span class="co"># [4,] 4 8 12</span></span></code></pre></div> <p>To obtain cells <code>(1, 2) = 5</code> and <code>(3, 3) = 11</code> many users (we believe) may try this first :</p> <div class="sourceCode" id="cb11"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true"></a>A[<span class="kw">c</span>(<span class="dv">1</span>, <span class="dv">3</span>), <span class="kw">c</span>(<span class="dv">2</span>, <span class="dv">3</span>)]</span> <span id="cb11-2"><a href="#cb11-2" aria-hidden="true"></a><span class="co"># [,1] [,2]</span></span> <span id="cb11-3"><a href="#cb11-3" aria-hidden="true"></a><span class="co"># [1,] 5 9</span></span> <span id="cb11-4"><a href="#cb11-4" aria-hidden="true"></a><span class="co"># [2,] 7 11</span></span></code></pre></div> <p>However, this returns the union of those rows and columns. To reference the cells, a 2-column matrix is required. <code>?Extract</code> says :</p> <blockquote> <p>When indexing arrays by <code>[</code> a single argument <code>i</code> can be a matrix with as many columns as there are dimensions of <code>x</code>; the result is then a vector with elements corresponding to the sets of indices in each row of <code>i</code>.</p> </blockquote> <p>Let’s try again.</p> <div class="sourceCode" id="cb12"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true"></a>B =<span class="st"> </span><span class="kw">cbind</span>(<span class="kw">c</span>(<span class="dv">1</span>, <span class="dv">3</span>), <span class="kw">c</span>(<span class="dv">2</span>, <span class="dv">3</span>))</span> <span id="cb12-2"><a href="#cb12-2" aria-hidden="true"></a>B</span> <span id="cb12-3"><a href="#cb12-3" aria-hidden="true"></a><span class="co"># [,1] [,2]</span></span> <span id="cb12-4"><a href="#cb12-4" aria-hidden="true"></a><span class="co"># [1,] 1 2</span></span> <span id="cb12-5"><a href="#cb12-5" aria-hidden="true"></a><span class="co"># [2,] 3 3</span></span> <span id="cb12-6"><a href="#cb12-6" aria-hidden="true"></a>A[B]</span> <span id="cb12-7"><a href="#cb12-7" aria-hidden="true"></a><span class="co"># [1] 5 11</span></span></code></pre></div> <p>A matrix is a 2-dimensional structure with row names and column names. Can we do the same with names?</p> <div class="sourceCode" id="cb13"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true"></a><span class="kw">rownames</span>(A) =<span class="st"> </span>letters[<span class="dv">1</span><span class="op">:</span><span class="dv">4</span>]</span> <span id="cb13-2"><a href="#cb13-2" aria-hidden="true"></a><span class="kw">colnames</span>(A) =<span class="st"> </span>LETTERS[<span class="dv">1</span><span class="op">:</span><span class="dv">3</span>]</span> <span id="cb13-3"><a href="#cb13-3" aria-hidden="true"></a>A</span> <span id="cb13-4"><a href="#cb13-4" aria-hidden="true"></a><span class="co"># A B C</span></span> <span id="cb13-5"><a href="#cb13-5" aria-hidden="true"></a><span class="co"># a 1 5 9</span></span> <span id="cb13-6"><a href="#cb13-6" aria-hidden="true"></a><span class="co"># b 2 6 10</span></span> <span id="cb13-7"><a href="#cb13-7" aria-hidden="true"></a><span class="co"># c 3 7 11</span></span> <span id="cb13-8"><a href="#cb13-8" aria-hidden="true"></a><span class="co"># d 4 8 12</span></span> <span id="cb13-9"><a href="#cb13-9" aria-hidden="true"></a>B =<span class="st"> </span><span class="kw">cbind</span>(<span class="kw">c</span>(<span class="st">"a"</span>, <span class="st">"c"</span>), <span class="kw">c</span>(<span class="st">"B"</span>, <span class="st">"C"</span>))</span> <span id="cb13-10"><a href="#cb13-10" aria-hidden="true"></a>A[B]</span> <span id="cb13-11"><a href="#cb13-11" aria-hidden="true"></a><span class="co"># [1] 5 11</span></span></code></pre></div> <p>So yes, we can. Can we do the same with a <code>data.frame</code>?</p> <div class="sourceCode" id="cb14"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true"></a>A =<span class="st"> </span><span class="kw">data.frame</span>(<span class="dt">A =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">4</span>, <span class="dt">B =</span> letters[<span class="dv">11</span><span class="op">:</span><span class="dv">14</span>], <span class="dt">C =</span> pi<span class="op">*</span><span class="dv">1</span><span class="op">:</span><span class="dv">4</span>)</span> <span id="cb14-2"><a href="#cb14-2" aria-hidden="true"></a><span class="kw">rownames</span>(A) =<span class="st"> </span>letters[<span class="dv">1</span><span class="op">:</span><span class="dv">4</span>]</span> <span id="cb14-3"><a href="#cb14-3" aria-hidden="true"></a>A</span> <span id="cb14-4"><a href="#cb14-4" aria-hidden="true"></a><span class="co"># A B C</span></span> <span id="cb14-5"><a href="#cb14-5" aria-hidden="true"></a><span class="co"># a 1 k 3.141593</span></span> <span id="cb14-6"><a href="#cb14-6" aria-hidden="true"></a><span class="co"># b 2 l 6.283185</span></span> <span id="cb14-7"><a href="#cb14-7" aria-hidden="true"></a><span class="co"># c 3 m 9.424778</span></span> <span id="cb14-8"><a href="#cb14-8" aria-hidden="true"></a><span class="co"># d 4 n 12.566371</span></span> <span id="cb14-9"><a href="#cb14-9" aria-hidden="true"></a>B</span> <span id="cb14-10"><a href="#cb14-10" aria-hidden="true"></a><span class="co"># [,1] [,2]</span></span> <span id="cb14-11"><a href="#cb14-11" aria-hidden="true"></a><span class="co"># [1,] "a" "B" </span></span> <span id="cb14-12"><a href="#cb14-12" aria-hidden="true"></a><span class="co"># [2,] "c" "C"</span></span> <span id="cb14-13"><a href="#cb14-13" aria-hidden="true"></a>A[B]</span> <span id="cb14-14"><a href="#cb14-14" aria-hidden="true"></a><span class="co"># [1] "k" " 9.424778"</span></span></code></pre></div> <p>But, notice that the result was coerced to <code>character.</code> R coerced <code>A</code> to <code>matrix</code> first so that the syntax could work, but the result isn’t ideal. Let’s try making <code>B</code> a <code>data.frame</code>.</p> <div class="sourceCode" id="cb15"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb15-1"><a href="#cb15-1" aria-hidden="true"></a>B =<span class="st"> </span><span class="kw">data.frame</span>(<span class="kw">c</span>(<span class="st">"a"</span>, <span class="st">"c"</span>), <span class="kw">c</span>(<span class="st">"B"</span>, <span class="st">"C"</span>))</span> <span id="cb15-2"><a href="#cb15-2" aria-hidden="true"></a><span class="kw">cat</span>(<span class="kw">try</span>(A[B], <span class="dt">silent =</span> <span class="ot">TRUE</span>))</span> <span id="cb15-3"><a href="#cb15-3" aria-hidden="true"></a><span class="co"># Error in `[.default`(A, B) : invalid subscript type 'list'</span></span></code></pre></div> <p>So we can’t subset a <code>data.frame</code> by a <code>data.frame</code> in base R. What if we want row names and column names that aren’t <code>character</code> but <code>integer</code> or <code>float</code>? What if we want more than 2 dimensions of mixed types? Enter data.table.</p> <p>Furthermore, matrices, especially sparse matrices, are often stored in a 3-column tuple: <code>(i, j, value)</code>. This can be thought of as a key-value pair where <code>i</code> and <code>j</code> form a 2-column key. If we have more than one value, perhaps of different types, it might look like <code>(i, j, val1, val2, val3, ...)</code>. This looks very much like a <code>data.frame</code>. Hence data.table extends <code>data.frame</code> so that a <code>data.frame</code> <code>X</code> can be subset by a <code>data.frame</code> <code>Y</code>, leading to the <code>X[Y]</code> syntax.</p> </div> <div id="can-base-be-changed-to-do-this-then-rather-than-a-new-package" class="section level2" number="2.15"> <h2 number="2.15"><span class="header-section-number">2.15</span> Can base be changed to do this then, rather than a new package?</h2> <p><code>data.frame</code> is used <em>everywhere</em> and so it is very difficult to make <em>any</em> changes to it. data.table <em>inherits</em> from <code>data.frame</code>. It <em>is</em> a <code>data.frame</code>, too. A data.table <em>can</em> be passed to any package that <em>only</em> accepts <code>data.frame</code>. When that package uses <code>[.data.frame</code> syntax on the data.table, it works. It works because <code>[.data.table</code> looks to see where it was called from. If it was called from such a package, <code>[.data.table</code> diverts to <code>[.data.frame</code>.</p> </div> <div id="ive-heard-that-data.table-syntax-is-analogous-to-sql." class="section level2" number="2.16"> <h2 number="2.16"><span class="header-section-number">2.16</span> I’ve heard that data.table syntax is analogous to SQL.</h2> <p>Yes :</p> <ul> <li><code>i</code> <span class="math inline">\(\Leftrightarrow\)</span> where</li> <li><code>j</code> <span class="math inline">\(\Leftrightarrow\)</span> select</li> <li><code>:=</code> <span class="math inline">\(\Leftrightarrow\)</span> update</li> <li><code>by</code> <span class="math inline">\(\Leftrightarrow\)</span> group by</li> <li><code>i</code> <span class="math inline">\(\Leftrightarrow\)</span> order by (in compound syntax)</li> <li><code>i</code> <span class="math inline">\(\Leftrightarrow\)</span> having (in compound syntax)</li> <li><code>nomatch = NA</code> <span class="math inline">\(\Leftrightarrow\)</span> outer join</li> <li><code>nomatch = NULL</code> <span class="math inline">\(\Leftrightarrow\)</span> inner join</li> <li><code>mult = "first"|"last"</code> <span class="math inline">\(\Leftrightarrow\)</span> N/A because SQL is inherently unordered</li> <li><code>roll = TRUE</code> <span class="math inline">\(\Leftrightarrow\)</span> N/A because SQL is inherently unordered</li> </ul> <p>The general form is :</p> <div class="sourceCode" id="cb16"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb16-1"><a href="#cb16-1" aria-hidden="true"></a>DT[where, select<span class="op">|</span>update, group by][order by][...] ... [...]</span></code></pre></div> <p>A key advantage of column vectors in R is that they are <em>ordered</em>, unlike SQL<a href="#fn2" class="footnote-ref" id="fnref2"><sup>2</sup></a>. We can use ordered functions in <code>data.table</code> queries such as <code>diff()</code> and we can use <em>any</em> R function from any package, not just the functions that are defined in SQL. A disadvantage is that R objects must fit in memory, but with several R packages such as <code>ff</code>, <code>bigmemory</code>, <code>mmap</code> and <code>indexing</code>, this is changing.</p> </div> <div id="SmallerDiffs" class="section level2" number="2.17"> <h2 number="2.17"><span class="header-section-number">2.17</span> What are the smaller syntax differences between <code>data.frame</code> and data.table</h2> <ul> <li><code>DT[3]</code> refers to the 3rd <em>row</em>, but <code>DF[3]</code> refers to the 3rd <em>column</em></li> <li><code>DT[3, ] == DT[3]</code>, but <code>DF[ , 3] == DF[3]</code> (somewhat confusingly in data.frame, whereas data.table is consistent)</li> <li>For this reason we say the comma is <em>optional</em> in <code>DT</code>, but not optional in <code>DF</code></li> <li><code>DT[[3]] == DF[, 3] == DF[[3]]</code></li> <li><code>DT[i, ]</code>, where <code>i</code> is a single integer, returns a single row, just like <code>DF[i, ]</code>, but unlike a matrix single-row subset which returns a vector.</li> <li><code>DT[ , j]</code> where <code>j</code> is a single integer returns a one-column data.table, unlike <code>DF[, j]</code> which returns a vector by default</li> <li><code>DT[ , "colA"][[1]] == DF[ , "colA"]</code>.</li> <li><code>DT[ , colA] == DF[ , "colA"]</code> (currently in data.table v1.9.8 but is about to change, see release notes)</li> <li><code>DT[ , list(colA)] == DF[ , "colA", drop = FALSE]</code></li> <li><code>DT[NA]</code> returns 1 row of <code>NA</code>, but <code>DF[NA]</code> returns an entire copy of <code>DF</code> containing <code>NA</code> throughout. The symbol <code>NA</code> is type <code>logical</code> in R and is therefore recycled by <code>[.data.frame</code>. The user’s intention was probably <code>DF[NA_integer_]</code>. <code>[.data.table</code> diverts to this probable intention automatically, for convenience.</li> <li><code>DT[c(TRUE, NA, FALSE)]</code> treats the <code>NA</code> as <code>FALSE</code>, but <code>DF[c(TRUE, NA, FALSE)]</code> returns <code>NA</code> rows for each <code>NA</code></li> <li><code>DT[ColA == ColB]</code> is simpler than <code>DF[!is.na(ColA) & !is.na(ColB) & ColA == ColB, ]</code></li> <li><code>data.frame(list(1:2, "k", 1:4))</code> creates 3 columns, data.table creates one <code>list</code> column.</li> <li><code>check.names</code> is by default <code>TRUE</code> in <code>data.frame</code> but <code>FALSE</code> in data.table, for convenience.</li> <li><code>stringsAsFactors</code> is by default <code>TRUE</code> in <code>data.frame</code> but <code>FALSE</code> in data.table, for efficiency. Since a global string cache was added to R, characters items are a pointer to the single cached string and there is no longer a performance benefit of converting to <code>factor</code>.</li> <li>Atomic vectors in <code>list</code> columns are collapsed when printed using <code>", "</code> in <code>data.frame</code>, but <code>","</code> in data.table with a trailing comma after the 6th item to avoid accidental printing of large embedded objects.</li> </ul> <p>In <code>[.data.frame</code> we very often set <code>drop = FALSE</code>. When we forget, bugs can arise in edge cases where single columns are selected and all of a sudden a vector is returned rather than a single column <code>data.frame</code>. In <code>[.data.table</code> we took the opportunity to make it consistent and dropped <code>drop</code>.</p> <p>When a data.table is passed to a data.table-unaware package, that package is not concerned with any of these differences; it just works.</p> </div> <div id="im-using-j-for-its-side-effect-only-but-im-still-getting-data-returned.-how-do-i-stop-that" class="section level2" number="2.18"> <h2 number="2.18"><span class="header-section-number">2.18</span> I’m using <code>j</code> for its side effect only, but I’m still getting data returned. How do I stop that?</h2> <p>In this case <code>j</code> can be wrapped with <code>invisible()</code>; e.g., <code>DT[ , invisible(hist(colB)), by = colA]</code><a href="#fn3" class="footnote-ref" id="fnref3"><sup>3</sup></a></p> </div> <div id="why-does-.data.table-now-have-a-drop-argument-from-v1.5" class="section level2" number="2.19"> <h2 number="2.19"><span class="header-section-number">2.19</span> Why does <code>[.data.table</code> now have a <code>drop</code> argument from v1.5?</h2> <p>So that data.table can inherit from <code>data.frame</code> without using <code>...</code>. If we used <code>...</code> then invalid argument names would not be caught.</p> <p>The <code>drop</code> argument is never used by <code>[.data.table</code>. It is a placeholder for non-data.table-aware packages when they use the <code>[.data.frame</code> syntax directly on a data.table.</p> </div> <div id="rolling-joins-are-cool-and-very-fast-was-that-hard-to-program" class="section level2" number="2.20"> <h2 number="2.20"><span class="header-section-number">2.20</span> Rolling joins are cool and very fast! Was that hard to program?</h2> <p>The prevailing row on or before the <code>i</code> row is the final row the binary search tests anyway. So <code>roll = TRUE</code> is essentially just a switch in the binary search C code to return that row.</p> </div> <div id="why-does-dti-col-value-return-the-whole-of-dt-i-expected-either-no-visible-value-consistent-with---or-a-message-or-return-value-containing-how-many-rows-were-updated.-it-isnt-obvious-that-the-data-has-indeed-been-updated-by-reference." class="section level2" number="2.21"> <h2 number="2.21"><span class="header-section-number">2.21</span> Why does <code>DT[i, col := value]</code> return the whole of <code>DT</code>? I expected either no visible value (consistent with <code><-</code>), or a message or return value containing how many rows were updated. It isn’t obvious that the data has indeed been updated by reference.</h2> <p>This has changed in v1.8.3 to meet your expectations. Please upgrade.</p> <p>The whole of <code>DT</code> is returned (now invisibly) so that compound syntax can work; <em>e.g.</em>, <code>DT[i, done := TRUE][ , sum(done)]</code>. The number of rows updated is returned when <code>verbose</code> is <code>TRUE</code>, either on a per-query basis or globally using <code>options(datatable.verbose = TRUE)</code>.</p> </div> <div id="ok-thanks.-what-was-so-difficult-about-the-result-of-dti-col-value-being-returned-invisibly" class="section level2" number="2.22"> <h2 number="2.22"><span class="header-section-number">2.22</span> OK, thanks. What was so difficult about the result of <code>DT[i, col := value]</code> being returned invisibly?</h2> <p>R internally forces visibility on for <code>[</code>. The value of FunTab’s eval column (see <a href="https://github.com/wch/r-source/blob/trunk/src/main/names.c">src/main/names.c</a>) for <code>[</code> is <code>0</code> meaning “force <code>R_Visible</code> on” (see <a href="https://cran.r-project.org/doc/manuals/r-release/R-ints.html#Autoprinting">R-Internals section 1.6</a> ). Therefore, when we tried <code>invisible()</code> or setting <code>R_Visible</code> to <code>0</code> directly ourselves, <code>eval</code> in <a href="https://github.com/wch/r-source/blob/trunk/src/main/eval.c">src/main/eval.c</a> would force it on again.</p> <p>To solve this problem, the key was to stop trying to stop the print method running after a <code>:=</code>. Instead, inside <code>:=</code> we now (from v1.8.3) set a global flag which the print method uses to know whether to actually print or not.</p> </div> <div id="why-do-i-have-to-type-dt-sometimes-twice-after-using-to-print-the-result-to-console" class="section level2" number="2.23"> <h2 number="2.23"><span class="header-section-number">2.23</span> Why do I have to type <code>DT</code> sometimes twice after using <code>:=</code> to print the result to console?</h2> <p>This is an unfortunate downside to get <a href="https://github.com/Rdatatable/data.table/issues/869">#869</a> to work. If a <code>:=</code> is used inside a function with no <code>DT[]</code> before the end of the function, then the next time <code>DT</code> is typed at the prompt, nothing will be printed. A repeated <code>DT</code> will print. To avoid this: include a <code>DT[]</code> after the last <code>:=</code> in your function. If that is not possible (e.g., it’s not a function you can change) then <code>print(DT)</code> and <code>DT[]</code> at the prompt are guaranteed to print. As before, adding an extra <code>[]</code> on the end of <code>:=</code> query is a recommended idiom to update and then print; e.g.> <code>DT[,foo:=3L][]</code>.</p> </div> <div id="ive-noticed-that-basecbind.data.frame-and-baserbind.data.frame-appear-to-be-changed-by-data.table.-how-is-this-possible-why" class="section level2" number="2.24"> <h2 number="2.24"><span class="header-section-number">2.24</span> I’ve noticed that <code>base::cbind.data.frame</code> (and <code>base::rbind.data.frame</code>) appear to be changed by data.table. How is this possible? Why?</h2> <p>It was a temporary, last resort solution before rbind and cbind S3 method dispatch was fixed in R >= 4.0.0. Essentially, the issue was that <code>data.table</code> inherits from <code>data.frame</code>, <em>and</em> <code>base::cbind</code> and <code>base::rbind</code> (uniquely) do their own S3 dispatch internally as documented by <code>?cbind</code>. The <code>data.table</code> workaround was adding one <code>for</code> loop to the start of each function directly in <code>base</code>. That modification was made dynamically, <em>i.e.</em>, the <code>base</code> definition of <code>cbind.data.frame</code> was fetched, the <code>for</code> loop added to the beginning, and then assigned back to <code>base</code>. This solution was designed to be robust to different definitions of <code>base::cbind.data.frame</code> in different versions of R, including unknown future changes. It worked well. The competing requirements were:</p> <ul> <li><p><code>cbind(DT, DF)</code> needs to work. Defining <code>cbind.data.table</code> didn’t work because <code>base::cbind</code> does its own S3 dispatch and required (before R 4.0.0) that the <em>first</em> <code>cbind</code> method for each object it is passed is <em>identical</em>. This is not true in <code>cbind(DT, DF)</code> because the first method for <code>DT</code> is <code>cbind.data.table</code> but the first method for <code>DF</code> is <code>cbind.data.frame</code>. <code>base::cbind</code> then fell through to its internal <code>bind</code> code which appears to treat <code>DT</code> as a regular <code>list</code> and returns very odd looking and unusable <code>matrix</code> output. See <a href="#cbinderror">below</a>. We cannot just advise users not to call <code>cbind(DT, DF)</code> because packages such as <code>ggplot2</code> make such a call (<a href="https://github.com/Rdatatable/data.table/blob/master/inst/tests/tests.Rraw#L444-L447">test 167.2</a>).</p></li> <li><p>This naturally led to trying to mask <code>cbind.data.frame</code> instead. Since a data.table is a <code>data.frame</code>, <code>cbind</code> would find the same method for both <code>DT</code> and <code>DF</code>. However, this didn’t work either because <code>base::cbind</code> appears to find methods in <code>base</code> first; <em>i.e.</em>, <code>base::cbind.data.frame</code> isn’t maskable.</p></li> <li><p>Finally, we tried masking <code>cbind</code> itself (v1.6.5 and v1.6.6). This allowed <code>cbind(DT, DF)</code> to work, but introduced compatibility issues with package <code>IRanges</code>, since <code>IRanges</code> also masks <code>cbind</code>. It worked if <code>IRanges</code> was lower on the <code>search()</code> path than data.table, but if <code>IRanges</code> was higher then data.table’s, <code>cbind</code> would never be called and the strange-looking <code>matrix</code> output occurs again (see <a href="#cbinderror">below</a>).</p></li> </ul> <p>Many thanks to the R core team for fixing the issue in Sep 2019. data.table v1.12.6+ no longer applies the workaround in R >= 4.0.0.</p> </div> <div id="r-dispatch" class="section level2" number="2.25"> <h2 number="2.25"><span class="header-section-number">2.25</span> I’ve read about method dispatch (<em>e.g.</em> <code>merge</code> may or may not dispatch to <code>merge.data.table</code>) but <em>how</em> does R know how to dispatch? Are dots significant or special? How on earth does R know which function to dispatch and when?</h2> <p>This comes up quite a lot but it’s really earth-shatteringly simple. A function such as <code>merge</code> is <em>generic</em> if it consists of a call to <code>UseMethod</code>. When you see people talking about whether or not functions are <em>generic</em> functions they are merely typing the function without <code>()</code> afterwards, looking at the program code inside it and if they see a call to <code>UseMethod</code> then it is <em>generic</em>. What does <code>UseMethod</code> do? It literally slaps the function name together with the class of the first argument, separated by period (<code>.</code>) and then calls that function, passing along the same arguments. It’s that simple. For example, <code>merge(X, Y)</code> contains a <code>UseMethod</code> call which means it then <em>dispatches</em> (i.e. calls) <code>paste("merge", class(X), sep = ".")</code>. Functions with dots in their name may or may not be methods. The dot is irrelevant really, other than dot being the separator that <code>UseMethod</code> uses. Knowing this background should now highlight why, for example, it is obvious to R folk that <code>as.data.table.data.frame</code> is the <code>data.frame</code> method for the <code>as.data.table</code> generic function. Further, it may help to elucidate that, yes, you are correct, it is not obvious from its name alone that <code>ls.fit</code> is not the fit method of the <code>ls</code> generic function. You only know that by typing <code>ls</code> (not <code>ls()</code>) and observing it isn’t a single call to <code>UseMethod</code>.</p> <p>You might now ask: where is this documented in R? Answer: it’s quite clear, but, you need to first know to look in <code>?UseMethod</code> and <em>that</em> help file contains :</p> <blockquote> <p>When a function calling <code>UseMethod('fun')</code> is applied to an object with class attribute <code>c('first', 'second')</code>, the system searches for a function called <code>fun.first</code> and, if it finds it, applies it to the object. If no such function is found a function called <code>fun.second</code> is tried. If no class name produces a suitable function, the function <code>fun.default</code> is used, if it exists, or an error results.</p> </blockquote> <p>Happily, an internet search for “How does R method dispatch work” (at the time of this writing) returns the <code>?UseMethod</code> help page in the top few links. Admittedly, other links rapidly descend into the intricacies of S3 vs S4, internal generics and so on.</p> <p>However, features like basic S3 dispatch (pasting the function name together with the class name) is why some R folk love R. It’s so simple. No complicated registration or signature is required. There isn’t much needed to learn. To create the <code>merge</code> method for data.table all that was required, literally, was to merely create a function called <code>merge.data.table</code>.</p> </div> </div> <div id="questions-relating-to-compute-time" class="section level1" number="3"> <h1 number="3"><span class="header-section-number">3</span> Questions relating to compute time</h1> <div id="i-have-20-columns-and-a-large-number-of-rows.-why-is-an-expression-of-one-column-so-quick" class="section level2" number="3.1"> <h2 number="3.1"><span class="header-section-number">3.1</span> I have 20 columns and a large number of rows. Why is an expression of one column so quick?</h2> <p>Several reasons:</p> <ul> <li>Only that column is grouped, the other 19 are ignored because data.table inspects the <code>j</code> expression and realises it doesn’t use the other columns.</li> <li>One memory allocation is made for the largest group only, then that memory is re-used for the other groups. There is very little garbage to collect.</li> <li>R is an in-memory column store; i.e., the columns are contiguous in RAM. Page fetches from RAM into L2 cache are minimised.</li> </ul> </div> <div id="i-dont-have-a-key-on-a-large-table-but-grouping-is-still-really-quick.-why-is-that" class="section level2" number="3.2"> <h2 number="3.2"><span class="header-section-number">3.2</span> I don’t have a <code>key</code> on a large table, but grouping is still really quick. Why is that?</h2> <p>data.table uses radix sorting. This is significantly faster than other sort algorithms. See <a href="https://github.com/Rdatatable/data.table/wiki/Presentations">our presentations</a> for more information, in particular from useR!2015 Denmark.</p> <p>This is also one reason why <code>setkey()</code> is quick.</p> <p>When no <code>key</code> is set, or we group in a different order from that of the key, we call it an <em>ad hoc</em> <code>by</code>.</p> </div> <div id="why-is-grouping-by-columns-in-the-key-faster-than-an-ad-hoc-by" class="section level2" number="3.3"> <h2 number="3.3"><span class="header-section-number">3.3</span> Why is grouping by columns in the key faster than an <em>ad hoc</em> <code>by</code>?</h2> <p>Because each group is contiguous in RAM, thereby minimising page fetches and memory can be copied in bulk (<code>memcpy</code> in C) rather than looping in C.</p> </div> <div id="what-are-primary-and-secondary-indexes-in-data.table" class="section level2" number="3.4"> <h2 number="3.4"><span class="header-section-number">3.4</span> What are primary and secondary indexes in data.table?</h2> <p>Manual: <a href="https://www.rdocumentation.org/packages/data.table/functions/setkey"><code>?setkey</code></a> S.O. : <a href="https://stackoverflow.com/questions/20039335/what-is-the-purpose-of-setting-a-key-in-data-table/20057411#20057411">What is the purpose of setting a key in data.table?</a></p> <p><code>setkey(DT, col1, col2)</code> orders the rows by column <code>col1</code> then within each group of <code>col1</code> it orders by <code>col2</code>. This is a <em>primary index</em>. The row order is changed <em>by reference</em> in RAM. Subsequent joins and groups on those key columns then take advantage of the sort order for efficiency. (Imagine how difficult looking for a phone number in a printed telephone directory would be if it wasn’t sorted by surname then forename. That’s literally all <code>setkey</code> does. It sorts the rows by the columns you specify.) The index doesn’t use any RAM. It simply changes the row order in RAM and marks the key columns. Analogous to a <em>clustered index</em> in SQL.</p> <p>However, you can only have one primary key because data can only be physically sorted in RAM in one way at a time. Choose the primary index to be the one you use most often (e.g. <code>[id,date]</code>). Sometimes there isn’t an obvious choice for the primary key or you need to join and group many different columns in different orders. Enter a secondary index. This does use memory (<code>4*nrow</code> bytes regardless of the number of columns in the index) to store the order of the rows by the columns you specify, but doesn’t actually reorder the rows in RAM. Subsequent joins and groups take advantage of the secondary key’s order but need to <em>hop</em> via that index so aren’t as efficient as primary indexes. But still, a lot faster than a full vector scan. There is no limit to the number of secondary indexes since each one is just a different ordering vector. Typically you don’t need to create secondary indexes. They are created automatically and used for you automatically by using data.table normally; <em>e.g.</em> <code>DT[someCol == someVal, ]</code> and <code>DT[someCol %in% someVals, ]</code> will create, attach and then use the secondary index. This is faster in data.table than a vector scan so automatic indexing is on by default since there is no up-front penalty. There is an option to turn off automatic indexing; <em>e.g.</em>, if somehow many indexes are being created and even the relatively small amount of extra memory becomes too large.</p> <p>We use the words <em>index</em> and <em>key</em> interchangeably.</p> </div> </div> <div id="error-messages" class="section level1" number="4"> <h1 number="4"><span class="header-section-number">4</span> Error messages</h1> <div id="could-not-find-function-dt" class="section level2" number="4.1"> <h2 number="4.1"><span class="header-section-number">4.1</span> “Could not find function <code>DT</code>”</h2> <p>See above <a href="#DTremove1">here</a> and <a href="#DTremove2">here</a>.</p> </div> <div id="unused-arguments-mysum-sumv" class="section level2" number="4.2"> <h2 number="4.2"><span class="header-section-number">4.2</span> “unused argument(s) (<code>MySum = sum(v)</code>)”</h2> <p>This error is generated by <code>DT[ , MySum = sum(v)]</code>. <code>DT[ , .(MySum = sum(v))]</code> was intended, or <code>DT[ , j = .(MySum = sum(v))]</code>.</p> </div> <div id="translatecharutf8-must-be-called-on-a-charsxp" class="section level2" number="4.3"> <h2 number="4.3"><span class="header-section-number">4.3</span> “<code>translateCharUTF8</code> must be called on a <code>CHARSXP</code>”</h2> <p>This error (and similar, <em>e.g.</em>, “<code>getCharCE</code> must be called on a <code>CHARSXP</code>”) may be nothing do with character data or locale. Instead, this can be a symptom of an earlier memory corruption. To date these have been reproducible and fixed (quickly). Please report it to our <a href="https://github.com/Rdatatable/data.table/issues">issues tracker</a>.</p> </div> <div id="cbinderror" class="section level2" number="4.4"> <h2 number="4.4"><span class="header-section-number">4.4</span> <code>cbind(DT, DF)</code> returns a strange format, <em>e.g.</em> <code>Integer,5</code></h2> <p>This occurs prior to v1.6.5, for <code>rbind(DT, DF)</code> too. Please upgrade to v1.6.7 or later.</p> </div> <div id="cannot-change-value-of-locked-binding-for-.sd" class="section level2" number="4.5"> <h2 number="4.5"><span class="header-section-number">4.5</span> “cannot change value of locked binding for <code>.SD</code>”</h2> <p><code>.SD</code> is locked by design. See <code>?data.table</code>. If you’d like to manipulate <code>.SD</code> before using it, or returning it, and don’t wish to modify <code>DT</code> using <code>:=</code>, then take a copy first (see <code>?copy</code>), <em>e.g.</em>,</p> <div class="sourceCode" id="cb17"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb17-1"><a href="#cb17-1" aria-hidden="true"></a>DT =<span class="st"> </span><span class="kw">data.table</span>(<span class="dt">a =</span> <span class="kw">rep</span>(<span class="dv">1</span><span class="op">:</span><span class="dv">3</span>, <span class="dv">1</span><span class="op">:</span><span class="dv">3</span>), <span class="dt">b =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">6</span>, <span class="dt">c =</span> <span class="dv">7</span><span class="op">:</span><span class="dv">12</span>)</span> <span id="cb17-2"><a href="#cb17-2" aria-hidden="true"></a>DT</span> <span id="cb17-3"><a href="#cb17-3" aria-hidden="true"></a><span class="co"># a b c</span></span> <span id="cb17-4"><a href="#cb17-4" aria-hidden="true"></a><span class="co"># 1: 1 1 7</span></span> <span id="cb17-5"><a href="#cb17-5" aria-hidden="true"></a><span class="co"># 2: 2 2 8</span></span> <span id="cb17-6"><a href="#cb17-6" aria-hidden="true"></a><span class="co"># 3: 2 3 9</span></span> <span id="cb17-7"><a href="#cb17-7" aria-hidden="true"></a><span class="co"># 4: 3 4 10</span></span> <span id="cb17-8"><a href="#cb17-8" aria-hidden="true"></a><span class="co"># 5: 3 5 11</span></span> <span id="cb17-9"><a href="#cb17-9" aria-hidden="true"></a><span class="co"># 6: 3 6 12</span></span> <span id="cb17-10"><a href="#cb17-10" aria-hidden="true"></a>DT[ , { mySD =<span class="st"> </span><span class="kw">copy</span>(.SD)</span> <span id="cb17-11"><a href="#cb17-11" aria-hidden="true"></a> mySD[<span class="dv">1</span>, b <span class="op">:</span><span class="er">=</span><span class="st"> </span>99L]</span> <span id="cb17-12"><a href="#cb17-12" aria-hidden="true"></a> mySD},</span> <span id="cb17-13"><a href="#cb17-13" aria-hidden="true"></a> by =<span class="st"> </span>a]</span> <span id="cb17-14"><a href="#cb17-14" aria-hidden="true"></a><span class="co"># a b c</span></span> <span id="cb17-15"><a href="#cb17-15" aria-hidden="true"></a><span class="co"># 1: 1 99 7</span></span> <span id="cb17-16"><a href="#cb17-16" aria-hidden="true"></a><span class="co"># 2: 2 99 8</span></span> <span id="cb17-17"><a href="#cb17-17" aria-hidden="true"></a><span class="co"># 3: 2 3 9</span></span> <span id="cb17-18"><a href="#cb17-18" aria-hidden="true"></a><span class="co"># 4: 3 99 10</span></span> <span id="cb17-19"><a href="#cb17-19" aria-hidden="true"></a><span class="co"># 5: 3 5 11</span></span> <span id="cb17-20"><a href="#cb17-20" aria-hidden="true"></a><span class="co"># 6: 3 6 12</span></span></code></pre></div> </div> <div id="cannot-change-value-of-locked-binding-for-.n" class="section level2" number="4.6"> <h2 number="4.6"><span class="header-section-number">4.6</span> “cannot change value of locked binding for <code>.N</code>”</h2> <p>Please upgrade to v1.8.1 or later. From this version, if <code>.N</code> is returned by <code>j</code> it is renamed to <code>N</code> to avoid any ambiguity in any subsequent grouping between the <code>.N</code> special variable and a column called <code>".N"</code>.</p> <p>The old behaviour can be reproduced by forcing <code>.N</code> to be called <code>.N</code>, like this :</p> <div class="sourceCode" id="cb18"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb18-1"><a href="#cb18-1" aria-hidden="true"></a>DT =<span class="st"> </span><span class="kw">data.table</span>(<span class="dt">a =</span> <span class="kw">c</span>(<span class="dv">1</span>,<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">2</span>), <span class="dt">b =</span> <span class="kw">c</span>(<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">1</span>))</span> <span id="cb18-2"><a href="#cb18-2" aria-hidden="true"></a>DT</span> <span id="cb18-3"><a href="#cb18-3" aria-hidden="true"></a><span class="co"># a b</span></span> <span id="cb18-4"><a href="#cb18-4" aria-hidden="true"></a><span class="co"># 1: 1 1</span></span> <span id="cb18-5"><a href="#cb18-5" aria-hidden="true"></a><span class="co"># 2: 1 2</span></span> <span id="cb18-6"><a href="#cb18-6" aria-hidden="true"></a><span class="co"># 3: 2 2</span></span> <span id="cb18-7"><a href="#cb18-7" aria-hidden="true"></a><span class="co"># 4: 2 2</span></span> <span id="cb18-8"><a href="#cb18-8" aria-hidden="true"></a><span class="co"># 5: 2 1</span></span> <span id="cb18-9"><a href="#cb18-9" aria-hidden="true"></a>DT[ , <span class="kw">list</span>(<span class="dt">.N =</span> .N), <span class="kw">list</span>(a, b)] <span class="co"># show intermediate result for exposition</span></span> <span id="cb18-10"><a href="#cb18-10" aria-hidden="true"></a><span class="co"># a b .N</span></span> <span id="cb18-11"><a href="#cb18-11" aria-hidden="true"></a><span class="co"># 1: 1 1 1</span></span> <span id="cb18-12"><a href="#cb18-12" aria-hidden="true"></a><span class="co"># 2: 1 2 1</span></span> <span id="cb18-13"><a href="#cb18-13" aria-hidden="true"></a><span class="co"># 3: 2 2 2</span></span> <span id="cb18-14"><a href="#cb18-14" aria-hidden="true"></a><span class="co"># 4: 2 1 1</span></span> <span id="cb18-15"><a href="#cb18-15" aria-hidden="true"></a><span class="kw">cat</span>(<span class="kw">try</span>(</span> <span id="cb18-16"><a href="#cb18-16" aria-hidden="true"></a> DT[ , <span class="kw">list</span>(<span class="dt">.N =</span> .N), <span class="dt">by =</span> <span class="kw">list</span>(a, b)][ , <span class="kw">unique</span>(.N), <span class="dt">by =</span> a] <span class="co"># compound query more typical</span></span> <span id="cb18-17"><a href="#cb18-17" aria-hidden="true"></a>, <span class="dt">silent =</span> <span class="ot">TRUE</span>))</span> <span id="cb18-18"><a href="#cb18-18" aria-hidden="true"></a><span class="co"># Error in `[.data.table`(DT[, list(.N = .N), by = list(a, b)], , unique(.N), : </span></span> <span id="cb18-19"><a href="#cb18-19" aria-hidden="true"></a><span class="co"># The column '.N' can't be grouped because it conflicts with the special .N variable. Try setnames(DT,'.N','N') first.</span></span></code></pre></div> <p>If you are already running v1.8.1 or later then the error message is now more helpful than the “cannot change value of locked binding” error, as you can see above, since this vignette was produced using v1.8.1 or later.</p> <p>The more natural syntax now works :</p> <div class="sourceCode" id="cb19"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb19-1"><a href="#cb19-1" aria-hidden="true"></a><span class="cf">if</span> (<span class="kw">packageVersion</span>(<span class="st">"data.table"</span>) <span class="op">>=</span><span class="st"> "1.8.1"</span>) {</span> <span id="cb19-2"><a href="#cb19-2" aria-hidden="true"></a> DT[ , .N, by =<span class="st"> </span><span class="kw">list</span>(a, b)][ , <span class="kw">unique</span>(N), by =<span class="st"> </span>a]</span> <span id="cb19-3"><a href="#cb19-3" aria-hidden="true"></a> }</span> <span id="cb19-4"><a href="#cb19-4" aria-hidden="true"></a><span class="co"># a V1</span></span> <span id="cb19-5"><a href="#cb19-5" aria-hidden="true"></a><span class="co"># 1: 1 1</span></span> <span id="cb19-6"><a href="#cb19-6" aria-hidden="true"></a><span class="co"># 2: 2 2</span></span> <span id="cb19-7"><a href="#cb19-7" aria-hidden="true"></a><span class="co"># 3: 2 1</span></span> <span id="cb19-8"><a href="#cb19-8" aria-hidden="true"></a><span class="cf">if</span> (<span class="kw">packageVersion</span>(<span class="st">"data.table"</span>) <span class="op">>=</span><span class="st"> "1.9.3"</span>) {</span> <span id="cb19-9"><a href="#cb19-9" aria-hidden="true"></a> DT[ , .N, by =<span class="st"> </span>.(a, b)][ , <span class="kw">unique</span>(N), by =<span class="st"> </span>a] <span class="co"># same</span></span> <span id="cb19-10"><a href="#cb19-10" aria-hidden="true"></a>}</span> <span id="cb19-11"><a href="#cb19-11" aria-hidden="true"></a><span class="co"># a V1</span></span> <span id="cb19-12"><a href="#cb19-12" aria-hidden="true"></a><span class="co"># 1: 1 1</span></span> <span id="cb19-13"><a href="#cb19-13" aria-hidden="true"></a><span class="co"># 2: 2 2</span></span> <span id="cb19-14"><a href="#cb19-14" aria-hidden="true"></a><span class="co"># 3: 2 1</span></span></code></pre></div> </div> </div> <div id="warning-messages" class="section level1" number="5"> <h1 number="5"><span class="header-section-number">5</span> Warning messages</h1> <div id="the-following-objects-are-masked-from-packagebase-cbind-rbind" class="section level2" number="5.1"> <h2 number="5.1"><span class="header-section-number">5.1</span> “The following object(s) are masked from <code>package:base</code>: <code>cbind</code>, <code>rbind</code>”</h2> <p>This warning was present in v1.6.5 and v.1.6.6 only, when loading the package. The motivation was to allow <code>cbind(DT, DF)</code> to work, but as it transpired, this broke (full) compatibility with package <code>IRanges</code>. Please upgrade to v1.6.7 or later.</p> </div> <div id="coerced-numeric-rhs-to-integer-to-match-the-columns-type" class="section level2" number="5.2"> <h2 number="5.2"><span class="header-section-number">5.2</span> “Coerced numeric RHS to integer to match the column’s type”</h2> <p>Hopefully, this is self explanatory. The full message is:</p> <p>Coerced numeric RHS to integer to match the column’s type; may have truncated precision. Either change the column to numeric first by creating a new numeric vector length 5 (nrows of entire table) yourself and assigning that (i.e. ‘replace’ column), or coerce RHS to integer yourself (e.g. 1L or as.integer) to make your intent clear (and for speed). Or, set the column type correctly up front when you create the table and stick to it, please.</p> <p>To generate it, try :</p> <div class="sourceCode" id="cb20"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb20-1"><a href="#cb20-1" aria-hidden="true"></a>DT =<span class="st"> </span><span class="kw">data.table</span>(<span class="dt">a =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">5</span>, <span class="dt">b =</span> <span class="dv">1</span><span class="op">:</span><span class="dv">5</span>)</span> <span id="cb20-2"><a href="#cb20-2" aria-hidden="true"></a><span class="kw">suppressWarnings</span>(</span> <span id="cb20-3"><a href="#cb20-3" aria-hidden="true"></a>DT[<span class="dv">2</span>, <span class="dt">b :=</span> <span class="dv">6</span>] <span class="co"># works (slower) with warning</span></span> <span id="cb20-4"><a href="#cb20-4" aria-hidden="true"></a>)</span> <span id="cb20-5"><a href="#cb20-5" aria-hidden="true"></a><span class="kw">class</span>(<span class="dv">6</span>) <span class="co"># numeric not integer</span></span> <span id="cb20-6"><a href="#cb20-6" aria-hidden="true"></a><span class="co"># [1] "numeric"</span></span> <span id="cb20-7"><a href="#cb20-7" aria-hidden="true"></a>DT[<span class="dv">2</span>, b <span class="op">:</span><span class="er">=</span><span class="st"> </span>7L] <span class="co"># works (faster) without warning</span></span> <span id="cb20-8"><a href="#cb20-8" aria-hidden="true"></a><span class="kw">class</span>(7L) <span class="co"># L makes it an integer</span></span> <span id="cb20-9"><a href="#cb20-9" aria-hidden="true"></a><span class="co"># [1] "integer"</span></span> <span id="cb20-10"><a href="#cb20-10" aria-hidden="true"></a>DT[ , b <span class="op">:</span><span class="er">=</span><span class="st"> </span><span class="kw">rnorm</span>(<span class="dv">5</span>)] <span class="co"># 'replace' integer column with a numeric column</span></span></code></pre></div> </div> <div id="reading-data.table-from-rds-or-rdata-file" class="section level2" number="5.3"> <h2 number="5.3"><span class="header-section-number">5.3</span> Reading data.table from RDS or RData file</h2> <p><code>*.RDS</code> and <code>*.RData</code> are file types which can store in-memory R objects on disk efficiently. However, storing data.table into the binary file loses its column over-allocation. This isn’t a big deal – your data.table will be copied in memory on the next <em>by reference</em> operation and throw a warning. Therefore it is recommended to call <code>setalloccol()</code> on each data.table loaded with <code>readRDS()</code> or <code>load()</code> calls.</p> </div> </div> <div id="general-questions-about-the-package" class="section level1" number="6"> <h1 number="6"><span class="header-section-number">6</span> General questions about the package</h1> <div id="v1.3-appears-to-be-missing-from-the-cran-archive" class="section level2" number="6.1"> <h2 number="6.1"><span class="header-section-number">6.1</span> v1.3 appears to be missing from the CRAN archive?</h2> <p>That is correct. v1.3 was available on R-Forge only. There were several large changes internally and these took some time to test in development.</p> </div> <div id="is-data.table-compatible-with-s-plus" class="section level2" number="6.2"> <h2 number="6.2"><span class="header-section-number">6.2</span> Is data.table compatible with S-plus?</h2> <p>Not currently.</p> <ul> <li>A few core parts of the package are written in C and use internal R functions and R structures.</li> <li>The package uses lexical scoping which is one of the differences between R and <strong>S-plus</strong> explained by <a href="https://cran.r-project.org/doc/FAQ/R-FAQ.html#Lexical-scoping">R FAQ 3.3.1</a></li> </ul> </div> <div id="is-it-available-for-linux-mac-and-windows" class="section level2" number="6.3"> <h2 number="6.3"><span class="header-section-number">6.3</span> Is it available for Linux, Mac and Windows?</h2> <p>Yes, for both 32-bit and 64-bit on all platforms. Thanks to CRAN. There are no special or OS-specific libraries used.</p> </div> <div id="i-think-its-great.-what-can-i-do" class="section level2" number="6.4"> <h2 number="6.4"><span class="header-section-number">6.4</span> I think it’s great. What can I do?</h2> <p>Please file suggestions, bug reports and enhancement requests on our <a href="https://github.com/Rdatatable/data.table/issues">issues tracker</a>. This helps make the package better.</p> <p>Please do star the package on <a href="https://github.com/Rdatatable/data.table/wiki">GitHub</a>. This helps encourage the developers and helps other R users find the package.</p> <p>You can submit pull requests to change the code and/or documentation yourself; see our <a href="https://github.com/Rdatatable/data.table/wiki/Contributing">Contribution Guidelines</a>.</p> </div> <div id="i-think-its-not-great.-how-do-i-warn-others-about-my-experience" class="section level2" number="6.5"> <h2 number="6.5"><span class="header-section-number">6.5</span> I think it’s not great. How do I warn others about my experience?</h2> <p>We add all articles we know about (whether positive or negative) to the <a href="https://github.com/Rdatatable/data.table/wiki/Articles">Articles</a> page. All pages in the project’s wiki on GitHub are open-access with no modify restrictions. Feel free to write an article, link to a negative one someone else wrote that you found, or add a new page to our wiki to collect your criticisms. Please make it constructive so we have a chance to improve.</p> </div> <div id="i-have-a-question.-i-know-the-r-help-posting-guide-tells-me-to-contact-the-maintainer-not-r-help-but-is-there-a-larger-group-of-people-i-can-ask" class="section level2" number="6.6"> <h2 number="6.6"><span class="header-section-number">6.6</span> I have a question. I know the r-help posting guide tells me to contact the maintainer (not r-help), but is there a larger group of people I can ask?</h2> <p>Please see the <a href="https://github.com/Rdatatable/data.table/wiki/Support">support guide</a> on the project’s homepage which contains up-to-date links.</p> </div> <div id="where-are-the-datatable-help-archives" class="section level2" number="6.7"> <h2 number="6.7"><span class="header-section-number">6.7</span> Where are the datatable-help archives?</h2> <p>The <a href="https://github.com/Rdatatable/data.table/wiki">homepage</a> contains links to the archives in several formats.</p> </div> <div id="id-prefer-not-to-post-on-the-issues-page-can-i-mail-just-one-or-two-people-privately" class="section level2" number="6.8"> <h2 number="6.8"><span class="header-section-number">6.8</span> I’d prefer not to post on the Issues page, can I mail just one or two people privately?</h2> <p>Sure. You’re more likely to get a faster answer from the Issues page or Stack Overflow, though. Further, asking publicly in those places helps build the general knowledge base.</p> </div> <div id="i-have-created-a-package-that-uses-data.table.-how-do-i-ensure-my-package-is-data.table-aware-so-that-inheritance-from-data.frame-works" class="section level2" number="6.9"> <h2 number="6.9"><span class="header-section-number">6.9</span> I have created a package that uses data.table. How do I ensure my package is data.table-aware so that inheritance from <code>data.frame</code> works?</h2> <p>Please see <a href="https://stackoverflow.com/a/10529888/403310">this answer</a>.</p> </div> </div> <div class="footnotes"> <hr /> <ol> <li id="fn1"><p>Here we mean either the <code>merge</code> <em>method</em> for data.table or the <code>merge</code> method for <code>data.frame</code> since both methods work in the same way in this respect. See <code>?merge.data.table</code> and <a href="#r-dispatch">below</a> for more information about method dispatch.<a href="#fnref1" class="footnote-back">↩︎</a></p></li> <li id="fn2"><p>It may be a surprise to learn that <code>select top 10 * from ...</code> does <em>not</em> reliably return the same rows over time in SQL. You do need to include an <code>order by</code> clause, or use a clustered index to guarantee row order; <em>i.e.</em>, SQL is inherently unordered.<a href="#fnref2" class="footnote-back">↩︎</a></p></li> <li id="fn3"><p><em>e.g.</em>, <code>hist()</code> returns the breakpoints in addition to plotting to the graphics device.<a href="#fnref3" class="footnote-back">↩︎</a></p></li> </ol> </div> <!-- code folding --> <!-- dynamically load mathjax for compatibility with self-contained --> <script> (function () { var script = document.createElement("script"); script.type = "text/javascript"; script.src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; document.getElementsByTagName("head")[0].appendChild(script); })(); </script> </body> </html>