From: Jim Porter Date: Sat, 19 Oct 2024 18:52:42 +0000 (-0700) Subject: Lazily convert numeric strings to Lisp numbers in Eshell X-Git-Url: http://git.eshelyaron.com/gitweb/?a=commitdiff_plain;h=b9b788d0d9d5419ad4cb4e967b6d676f0c9897ba;p=emacs.git Lazily convert numeric strings to Lisp numbers in Eshell This should reduce the number of issues with Eshell converting strings to numbers too aggressively and losing information (e.g. "001" -> 1) while still allowing almost all of the beneficial uses, like summing a list of numeric strings with '+'. * lisp/eshell/esh-util.el (eshell--do-mark-numeric-string): New function. (eshell-convert-to-number): Make obsolete in favor of... (eshell-mark-numeric-string): ... this. Update callers. * lisp/eshell/esh-arg.el (eshell--numberlike-p): New function... (eshell-concat-1): ... use it. * test/lisp/eshell/esh-util-tests.el: Reimplement type conversion tests to use 'eshell-convertible-to-number-p' instead. * test/lisp/eshell/esh-var-tests.el (esh-var-test/interp-var-splice-concat, esh-var-test/interp-concat-cmd) (esh-var-test/interp-convert-var-split-indices) (esh-var-test/interp-convert-quoted-var-split-indices) (esh-var-test/interp-convert-cmd-multiline) (esh-var-test/interp-convert-cmd-split-indices): Adjust tests to check the new behavior. * doc/misc/eshell.texi (Type Conversion): New section. (Expansion): Clarify concatenation behavior. (cherry picked from commit 43d5b7a04c4b1a8a7d57f25824df2e8720d2c567) --- diff --git a/doc/misc/eshell.texi b/doc/misc/eshell.texi index feaa7584845..5470d519687 100644 --- a/doc/misc/eshell.texi +++ b/doc/misc/eshell.texi @@ -432,16 +432,6 @@ This command writes a list of all files matching the glob pattern @node Arguments @section Arguments -Ordinarily, Eshell parses arguments in command form as either strings -or numbers, depending on what the parser thinks they look like. To -specify an argument of some other data type, you can use a Lisp form -(@pxref{Invocation}): - -@example -~ $ echo (list 1 2 3) -(1 2 3) -@end example - When calling external commands (and many built-in Eshell commands, too) Eshell will flatten the arguments the command receives, so passing a list as an argument will ``spread'' the elements into @@ -454,13 +444,15 @@ multiple arguments: 3 @end example -@subsection Quoting and escaping +@subsection Quoting and Escaping As with other shells, you can escape special characters and spaces by prefixing the character with a backslash (@samp{\}), or by surrounding the string with apostrophes (@samp{''}) or double quotes (@samp{""}). This is needed especially for file names with special characters like pipe (@samp{|}) or square brackets (@samp{[} or @samp{]}), which could -be part of remote file names. +be part of remote file names. In addition, quoting or escaping an +argument will prevent it from being converted to a number when passed to +a Lisp function. When you escape a character with @samp{\} outside of any quotes, the result is the literal character immediately following it. For @@ -495,7 +487,46 @@ When using expansions (@pxref{Expansion}) in an Eshell command, the result may potentially be of any data type. To ensure that the result is always a string, the expansion can be surrounded by double quotes. -@subsection Special argument types +@subsection Type Conversion +When invoking a Lisp function via command form, Eshell automatically +converts string arguments that look like numbers to actual Lisp +numbers in order to make it easier to work with numeric values. You can +prevent this conversion on a case-by-case basis by quoting or escaping +the argument: + +@example +~ $ type-of 1 +integer +~ $ type-of "1" +string +@end example + +When invoking a subcommand in command form, Eshell will split the output +line-by-line into a list. Additionally, if every line looks like a +number, then Eshell will mark them as numeric so that passing them to a +Lisp function will convert them to Lisp numbers: + +@example +~ $ cat numbers.txt +01 +02 +03 +~ $ + $@@@{cat numbers.txt@} +6 +@end example + +If you find this behavior inconvenient for certain functions, you can +tell Eshell not to perform this conversion for that function: + +@example +(put \\='find-file \\='eshell-no-numeric-conversions t) +@end example + +@vindex eshell-convert-numeric-arguments +You can also disable this conversion behavior entirely by setting +@code{eshell-convert-numeric-arguments} to @code{nil}. + +@subsection Special Argument Types In addition to strings and numbers, Eshell supports a number of special argument types. These let you refer to various other Emacs Lisp data types, such as lists or buffers. @@ -1764,8 +1795,8 @@ behavior depends on the types of each value being concatenated: Concatenate both values together. @item one or both numbers -Concatenate the string representation of each value, converting back to -a number if possible. +Concatenate the string representation of each value. If either value is +numeric, mark the concatenated value as numeric if possible. @item one or both (non-@code{nil}) lists Concatenate ``adjacent'' elements of each value (possibly converting diff --git a/lisp/eshell/esh-arg.el b/lisp/eshell/esh-arg.el index b441cbfc274..6e4c0df7526 100644 --- a/lisp/eshell/esh-arg.el +++ b/lisp/eshell/esh-arg.el @@ -271,15 +271,21 @@ would produce (\"abc\" \"d\")." (t (setq result (eshell-concat-1 quoted result i)))))))) +(defsubst eshell--numberlike-p (object) + (or (numberp object) + (and (stringp object) (get-text-property 0 'number object)))) + (defun eshell-concat-1 (quoted first second) "Concatenate FIRST and SECOND. -If QUOTED is nil and either FIRST or SECOND are numbers, try to -convert the result to a number as well." +If QUOTED is nil and either FIRST or SECOND are numberlike, try to mark +the result as a number as well." (let ((result (concat (eshell-stringify first) (eshell-stringify second)))) - (if (and (not quoted) - (or (numberp first) (numberp second))) - (eshell-convert-to-number result) - result))) + (remove-text-properties 0 (length result) '(number) result) + (when (and (not quoted) + (or (eshell--numberlike-p first) + (eshell--numberlike-p second))) + (eshell-mark-numeric-string result)) + result)) (defun eshell-concat-groups (quoted &rest args) "Concatenate groups of arguments in ARGS and return the result. diff --git a/lisp/eshell/esh-util.el b/lisp/eshell/esh-util.el index 46083184aaa..180f049e495 100644 --- a/lisp/eshell/esh-util.el +++ b/lisp/eshell/esh-util.el @@ -343,11 +343,22 @@ If `eshell-convert-numeric-arguments', always return nil." (concat "\\`\\s-*" eshell-number-regexp "\\s-*\\'") string))) +(defsubst eshell--do-mark-numeric-string (string) + (put-text-property 0 (length string) 'number t string)) + +(defun eshell-mark-numeric-string (string) + "If STRING is convertible to a number, add a text property indicating so. +See `eshell-convertible-to-number-p'." + (when (eshell-convertible-to-number-p string) + (eshell--do-mark-numeric-string string)) + string) + (defun eshell-convert-to-number (string) "Try to convert STRING to a number. If STRING doesn't look like a number (or `eshell-convert-numeric-arguments' is nil), just return STRING unchanged." + (declare (obsolete 'eshell-mark-numeric-string "31.1")) (if (eshell-convertible-to-number-p string) (string-to-number string) string)) @@ -376,10 +387,10 @@ trailing newlines removed. Otherwise, this behaves as follows: (setq string (substring string 0 (1- len)))) (if (string-search "\n" string) (let ((lines (split-string string "\n"))) - (if (seq-every-p #'eshell-convertible-to-number-p lines) - (mapcar #'string-to-number lines) - lines)) - (eshell-convert-to-number string))))))) + (when (seq-every-p #'eshell-convertible-to-number-p lines) + (mapc #'eshell--do-mark-numeric-string lines)) + lines) + (eshell-mark-numeric-string string))))))) (defvar-local eshell-path-env (getenv "PATH") "Content of $PATH. diff --git a/lisp/eshell/esh-var.el b/lisp/eshell/esh-var.el index 059bba03ee4..df490dd4c6c 100644 --- a/lisp/eshell/esh-var.el +++ b/lisp/eshell/esh-var.el @@ -765,11 +765,10 @@ Otherwise, each INT-OR-NAME refers to an element of the list value. Integers imply a direct index, and names, an associate lookup using `assoc'. -If QUOTED is non-nil, this was invoked inside double-quotes. -This affects the behavior of splitting strings: without quoting, -the split values are converted to numbers via -`eshell-convert-to-number' if possible; with quoting, they're -left as strings. +If QUOTED is non-nil, this was invoked inside double-quotes. This +affects the behavior of splitting strings: without quoting, the split +values are marked as numbers via `eshell-mark-numeric-string' if +possible; with quoting, they're left as plain strings. For example, to retrieve the second element of a user's record in '/etc/passwd', the variable reference would look like: @@ -785,7 +784,7 @@ For example, to retrieve the second element of a user's record in refs (cdr refs))) (setq value (split-string value separator)) (unless quoted - (setq value (mapcar #'eshell-convert-to-number value))))) + (setq value (mapcar #'eshell-mark-numeric-string value))))) (cond ((< (length refs) 0) (error "Invalid array variable index: %s" diff --git a/test/lisp/eshell/esh-util-tests.el b/test/lisp/eshell/esh-util-tests.el index 031de558d1f..4a0874bff39 100644 --- a/test/lisp/eshell/esh-util-tests.el +++ b/test/lisp/eshell/esh-util-tests.el @@ -66,70 +66,70 @@ "Test that `eshell-stringify' correctly stringifies complex objects." (should (equal (eshell-stringify (list 'quote 'hello)) "'hello"))) -(ert-deftest esh-util-test/eshell-convert-to-number/integer () - "Test that `eshell-convert-to-number' correctly converts integers." - (should (equal (eshell-convert-to-number "123") 123)) - (should (equal (eshell-convert-to-number "-123") -123)) +(ert-deftest esh-util-test/eshell-convertible-to-number-p/integer () + "Test that `eshell-convertible-to-number-p' matches integers." + (should (eshell-convertible-to-number-p "123")) + (should (eshell-convertible-to-number-p "-123")) ;; These are technially integers, since Emacs Lisp requires at least ;; one digit after the "." to be a float: - (should (equal (eshell-convert-to-number "123.") 123)) - (should (equal (eshell-convert-to-number "-123.") -123))) - -(ert-deftest esh-util-test/eshell-convert-to-number/floating-point () - "Test that `eshell-convert-to-number' correctly converts floats." - (should (equal (eshell-convert-to-number "1.23") 1.23)) - (should (equal (eshell-convert-to-number "-1.23") -1.23)) - (should (equal (eshell-convert-to-number ".1") 0.1)) - (should (equal (eshell-convert-to-number "-.1") -0.1))) - -(ert-deftest esh-util-test/eshell-convert-to-number/floating-point-exponent () - "Test that `eshell-convert-to-number' correctly converts exponent notation." + (should (eshell-convertible-to-number-p "123.")) + (should (eshell-convertible-to-number-p "-123."))) + +(ert-deftest esh-util-test/eshell-convertible-to-number-p/float () + "Test that `eshell-convertible-to-number-p' matches floats." + (should (eshell-convertible-to-number-p "1.23")) + (should (eshell-convertible-to-number-p "-1.23")) + (should (eshell-convertible-to-number-p ".1")) + (should (eshell-convertible-to-number-p "-.1"))) + +(ert-deftest esh-util-test/eshell-convertible-to-number-p/float-exponent () + "Test that `eshell-convertible-to-number-p' matches exponent notation." ;; Positive exponent: (dolist (exp '("e2" "e+2" "E2" "E+2")) - (should (equal (eshell-convert-to-number (concat "123" exp)) 12300.0)) - (should (equal (eshell-convert-to-number (concat "-123" exp)) -12300.0)) - (should (equal (eshell-convert-to-number (concat "1.23" exp)) 123.0)) - (should (equal (eshell-convert-to-number (concat "-1.23" exp)) -123.0)) - (should (equal (eshell-convert-to-number (concat "1." exp)) 100.0)) - (should (equal (eshell-convert-to-number (concat "-1." exp)) -100.0)) - (should (equal (eshell-convert-to-number (concat ".1" exp)) 10.0)) - (should (equal (eshell-convert-to-number (concat "-.1" exp)) -10.0))) + (should (eshell-convertible-to-number-p (concat "123" exp))) + (should (eshell-convertible-to-number-p (concat "-123" exp))) + (should (eshell-convertible-to-number-p (concat "1.23" exp))) + (should (eshell-convertible-to-number-p (concat "-1.23" exp))) + (should (eshell-convertible-to-number-p (concat "1." exp))) + (should (eshell-convertible-to-number-p (concat "-1." exp))) + (should (eshell-convertible-to-number-p (concat ".1" exp))) + (should (eshell-convertible-to-number-p (concat "-.1" exp)))) ;; Negative exponent: (dolist (exp '("e-2" "E-2")) - (should (equal (eshell-convert-to-number (concat "123" exp)) 1.23)) - (should (equal (eshell-convert-to-number (concat "-123" exp)) -1.23)) - (should (equal (eshell-convert-to-number (concat "1.23" exp)) 0.0123)) - (should (equal (eshell-convert-to-number (concat "-1.23" exp)) -0.0123)) - (should (equal (eshell-convert-to-number (concat "1." exp)) 0.01)) - (should (equal (eshell-convert-to-number (concat "-1." exp)) -0.01)) - (should (equal (eshell-convert-to-number (concat ".1" exp)) 0.001)) - (should (equal (eshell-convert-to-number (concat "-.1" exp)) -0.001)))) - -(ert-deftest esh-util-test/eshell-convert-to-number/floating-point/infinite () - "Test that `eshell-convert-to-number' correctly converts infinite floats." - (should (equal (eshell-convert-to-number "1.0e+INF") 1.0e+INF)) - (should (equal (eshell-convert-to-number "2.e+INF") 1.0e+INF)) - (should (equal (eshell-convert-to-number "-1.0e+INF") -1.0e+INF)) - (should (equal (eshell-convert-to-number "-2.e+INF") -1.0e+INF))) - -(ert-deftest esh-util-test/eshell-convert-to-number/floating-point/nan () - "Test that `eshell-convert-to-number' correctly converts NaNs." - (should (equal (eshell-convert-to-number "1.0e+NaN") 1.0e+NaN)) - (should (equal (eshell-convert-to-number "2.e+NaN") 2.0e+NaN)) - (should (equal (eshell-convert-to-number "-1.0e+NaN") -1.0e+NaN)) - (should (equal (eshell-convert-to-number "-2.e+NaN") -2.0e+NaN))) - -(ert-deftest esh-util-test/eshell-convert-to-number/non-numeric () - "Test that `eshell-convert-to-number' does nothing to non-numeric values." - (should (equal (eshell-convert-to-number "foo") "foo")) - (should (equal (eshell-convert-to-number "") "")) - (should (equal (eshell-convert-to-number "123foo") "123foo"))) - -(ert-deftest esh-util-test/eshell-convert-to-number/no-convert () - "Test that `eshell-convert-to-number' does nothing when disabled." + (should (eshell-convertible-to-number-p (concat "123" exp))) + (should (eshell-convertible-to-number-p (concat "-123" exp))) + (should (eshell-convertible-to-number-p (concat "1.23" exp))) + (should (eshell-convertible-to-number-p (concat "-1.23" exp))) + (should (eshell-convertible-to-number-p (concat "1." exp))) + (should (eshell-convertible-to-number-p (concat "-1." exp))) + (should (eshell-convertible-to-number-p (concat ".1" exp))) + (should (eshell-convertible-to-number-p (concat "-.1" exp))))) + +(ert-deftest esh-util-test/eshell-convertible-to-number-p/float/infinite () + "Test that `eshell-convertible-to-number-p' matches infinite floats." + (should (eshell-convertible-to-number-p "1.0e+INF")) + (should (eshell-convertible-to-number-p "2.e+INF")) + (should (eshell-convertible-to-number-p "-1.0e+INF")) + (should (eshell-convertible-to-number-p "-2.e+INF"))) + +(ert-deftest esh-util-test/eshell-convertible-to-number-p/float/nan () + "Test that `eshell-convertible-to-number-p' matches NaNs." + (should (eshell-convertible-to-number-p "1.0e+NaN")) + (should (eshell-convertible-to-number-p "2.e+NaN")) + (should (eshell-convertible-to-number-p "-1.0e+NaN")) + (should (eshell-convertible-to-number-p "-2.e+NaN"))) + +(ert-deftest esh-util-test/eshell-convertible-to-number-p/non-numeric () + "Test that `eshell-convertible-to-number-p' returns nil for non-numerics." + (should-not (eshell-convertible-to-number-p "foo")) + (should-not (eshell-convertible-to-number-p "")) + (should-not (eshell-convertible-to-number-p "123foo"))) + +(ert-deftest esh-util-test/eshell-convertible-to-number-p/no-convert () + "Test that `eshell-convertible-to-number-p' returns nil when disabled." (let ((eshell-convert-numeric-arguments nil)) - (should (equal (eshell-convert-to-number "123") "123")) - (should (equal (eshell-convert-to-number "1.23") "1.23")))) + (should-not (eshell-convertible-to-number-p "123")) + (should-not (eshell-convertible-to-number-p "1.23")))) (ert-deftest esh-util-test/eshell-printable-size () (should (equal (eshell-printable-size (expt 2 16)) "65536")) diff --git a/test/lisp/eshell/esh-var-tests.el b/test/lisp/eshell/esh-var-tests.el index 7ac9807a1a7..70f6e9c7777 100644 --- a/test/lisp/eshell/esh-var-tests.el +++ b/test/lisp/eshell/esh-var-tests.el @@ -231,7 +231,7 @@ nil, use FUNCTION instead." ;; into the first value of the non-spliced list. (eshell-command-result-equal "echo it is $@'eshell-test-value'$eshell-test-value" - '("it" "is" 1 2 (31 2 3))))) + '("it" "is" 1 2 ("31" 2 3))))) (ert-deftest esh-var-test/interp-lisp () "Interpolate Lisp form evaluation." @@ -280,12 +280,16 @@ nil, use FUNCTION instead." (eshell-command-result-equal "+ ${+ 1 2}3 3" 36) (eshell-command-result-equal "echo ${*echo \"foo\nbar\"}-baz" '("foo" "bar-baz")) - ;; Concatenating to a number in a list should produce a number... + ;; Concatenating to a number in a list should produce a numeric value... (eshell-command-result-equal "echo ${*echo \"1\n2\"}3" + '("1" "23")) + (eshell-command-result-equal "echo $@{*echo \"1\n2\"}3" '(1 23)) ;; ... but concatenating to a string that looks like a number in a list ;; should produce a string. (eshell-command-result-equal "echo ${*echo \"hi\n2\"}3" + '("hi" "23")) + (eshell-command-result-equal "echo $@{*echo \"hi\n2\"}3" '("hi" "23"))) (ert-deftest esh-var-test/interp-concat-cmd2 () @@ -491,11 +495,16 @@ nil, use FUNCTION instead." (ert-deftest esh-var-test/interp-convert-var-split-indices () "Interpolate and convert string variable with indices." - ;; Check that numeric forms are converted to numbers. + ;; Check that numeric forms are marked as numeric. (let ((eshell-test-value "000 010 020 030 040")) + ;; `eshell/echo' converts numeric strings to Lisp numbers... (eshell-command-result-equal "echo $eshell-test-value[0]" 0) + ;; ... but not lists of numeric strings... (eshell-command-result-equal "echo $eshell-test-value[0 2]" + '("000" "020")) + ;; ... unless each element is a separate argument to `eshell/echo'. + (eshell-command-result-equal "echo $@eshell-test-value[0 2]" '(0 20))) ;; Check that multiline forms are preserved as-is. (let ((eshell-test-value "foo\nbar:baz\n")) @@ -515,9 +524,14 @@ nil, use FUNCTION instead." (ert-deftest esh-var-test/interp-convert-quoted-var-split-indices () "Interpolate and convert quoted string variable with indices." (let ((eshell-test-value "000 010 020 030 040")) + ;; `eshell/echo' converts numeric strings to Lisp numbers... (eshell-command-result-equal "echo $'eshell-test-value'[0]" 0) + ;; ... but not lists of numeric strings... (eshell-command-result-equal "echo $'eshell-test-value'[0 2]" + '("000" "020")) + ;; ... unless each element is a separate argument to `eshell/echo'. + (eshell-command-result-equal "echo $@'eshell-test-value'[0 2]" '(0 20)))) (ert-deftest esh-var-test/interp-convert-cmd-string-newline () @@ -530,9 +544,13 @@ nil, use FUNCTION instead." '("foo" "bar")) ;; Numeric output should be converted to numbers... (eshell-command-result-equal "echo ${echo \"01\n02\n03\"}" + '("01" "02" "03")) + (eshell-command-result-equal "echo $@{echo \"01\n02\n03\"}" '(1 2 3)) ;; ... but only if every line is numeric. (eshell-command-result-equal "echo ${echo \"01\n02\nhi\"}" + '("01" "02" "hi")) + (eshell-command-result-equal "echo $@{echo \"01\n02\nhi\"}" '("01" "02" "hi"))) (ert-deftest esh-var-test/interp-convert-cmd-number () @@ -541,9 +559,14 @@ nil, use FUNCTION instead." (ert-deftest esh-var-test/interp-convert-cmd-split-indices () "Interpolate command result with indices." + ;; `eshell/echo' converts numeric strings to Lisp numbers... (eshell-command-result-equal "echo ${echo \"000 010 020\"}[0]" 0) + ;; ... but not lists of numeric strings... (eshell-command-result-equal "echo ${echo \"000 010 020\"}[0 2]" + '("000" "020")) + ;; ... unless each element is a separate argument to `eshell/echo'. + (eshell-command-result-equal "echo $@{echo \"000 010 020\"}[0 2]" '(0 20))) (ert-deftest esh-var-test/quoted-interp-convert-var-number ()