Why doesn't ls combined with awk show exact sizes?

I am trying to find the size of the files on my hard drive in exact bytes, but whenever the size gets too big the number turns all weird (like 1.98329e+12). Can I stop it from doing this or convert this into exact bytes?

The command is:

ls -lR | grep -v '^d' | awk '{total += $5} END {print "Total:", total}'

Picture of exact bytes:

Picture of weird number:

• The cut-off point before it stops showing exact bytes seems to be around 500gb
• The command du -sb properly shows exact bytes no matter how big the directory is.
• I have tried Ubuntu Gnome 15.10 64bit (Japanese and English) and Linux Mint 17.3 Cinnamon 64bit (Japanese)
• My drives are ntfs so I tried formatting one as ext4 and copying my files over. The results are same as ntfs.

4 Answers

The problem is that MAWK (the AWK variant installed on Ubuntu) by default prints integers bigger than 2147483647 (2³¹-1) in scientific notation:

% awk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan
compiled limits:
max NF 32767
sprintf buffer 2040
% printf '2147483647\n' | awk '{x += $1; print x}'
2147483647
% printf '2147483648\n' | awk '{x += $1; print x}'
2.14748e+09

You could use printf with a format specifer instead of print*:

printf '2147483648\n' | awk '{x += $1; printf "%.0f\n", x}'

% printf '2147483648\n' | awk '{x += $1; printf "%.0f\n", x}'
2147483648

In your case:

ls -lR | grep -v '^d' | awk '{total += $5} END {printf "Total:%.0f\n", total}'

ls -lR | grep -v '^d' | awk ' { total += $5 } END { printf "Total:%.0f\n", total } '

That will force AWK to print total in decimal notation instead of in scientific notation.

However, on another note, you should never parse ls.

A more sensitive way to do that would be using find + stat:

find . -type f -exec stat -c '%s' {} + | awk '{total += $1} END {printf "Total:%.0f\n", total}'

find . -type f -exec stat -c '%s' {} + | awk ' { total += $1 } END { printf "Total:%.0f\n", total } '

*%.0f is a trick to make printf print numbers bigger than 2147483647 (2³¹-1), which when using %d as the format specifier would always print as 2147483647. The limit of %.0f is that one will start losing precision after 9007199254740992 (2⁵³), if that's ever a concern (thanks to Rotsor for the useful information).

TL;DR: ls and awk are unnecessary for your purpose. Use du -cb or du -bs on the directory that you want to analyse.

Your purpose is to

1. Find all files
2. find their size (in bytes)
3. produce grand total for all of them

All these actions can be performed by du.

$ du -bs $HOME 2>/dev/null
76709521942 /home/xieerqi

It's worth noticing that du has two "modes" - it can either show how much file is in size OR how much actual disk space it takes up (the real , physical real-estate). Since you are interested in total size of all files, you want the apparent file size. -b flag gives exactly that ( -b is alias for --apparent-size --block-size=1 ).

Perhaps even more concise and appropriate solution would be to use du -bc directly on the directory you want. For instance, my home directory is about 76 GB in size

$ du -bc $HOME 2> /dev/null | tail -1
76694582570 total

For some reason you worry about difference in folder size and file size.You said in the comments:

I would prefer ls because directory sizes vary while file sizes are constant

du is recursive, and sums up the file sizes. Also, a directory does have a static size of 4096 bytes ( 4k ) , but with du it will be included in the result of du -bs directory_name . Consider this:

$ du -b suse/openSUSE-Leap-42.1-DVD-x86_64.iso
4648337408 suse/openSUSE-Leap-42.1-DVD-x86_64.iso
$ du -b suse/
4648341504 suse/
$ bc <<< "4648337408+4096"
4648341504
$ mkdir suse/another_dir
$ du -b suse/another_dir
4096 suse/another_dir
$ du -bs suse/
4648345600 suse/

Under the hood, awk does all calculations using double-precision floating point numbers. By default it prints them using printf(3) format specifier %.6g, which means that if the number is more than six digits wide it will switch over to E-notation, which is what you saw. You can work around this by setting the variable OFMT:

ls -lR | awk 'BEGIN { OFMT = "%d" } /^-/ { total += $5 } END { print "Total:", total }'

But there is an upper limit, beyond which it can't give you an exact number of bytes; it will start rounding off the low bits of the sum. 500 gigabytes = 500 * 1024 * 1024 * 1024 = 536870912000 ≈ 2³⁹. With the usual IEEE floating point, this is safely below that limit (which is roughly 2⁵²). However, it is large enough that I personally would feel better using a programming language that had proper "bignums" (unlimited-size integers). For instance, Python:

#! /usr/bin/python
import os
import sys
space = 0L # L means "long" - not necessary in Python 3
for subdir, dirs, files in os.walk(sys.argv[1]): for f in files: space += os.lstat(os.path.join(subdir, f)).st_size
sys.stdout.write("Total: {:d}\n".format(space))

This is also completely immune to problems with files with unusual characters in their names. And it counts space consumed by hidden files.

This computes the number of bytes visible in each file, which is the same as what ls -l prints. If you want number of bytes actually occupied on disk instead (what du prints), replace .st_size with .st_blocks * 512. (Yes, the multiplier is always 512, even if st_blksize is a different number.)

What you see here is a way to display large numbers. For example:

1.23e+3 = 1.23*10^3 = 1230

As far as I know, you cannot turn this off, but as you wrote in your question, du does behave differently, so I would recommend to use this. Otherwise, you would have to convert the numbers.