current community

chat
log out

Unix & Linux

Unix & Linux Meta

your communities

edit
cancel

Stack Overflow
1,088

Ask Ubuntu
8

Cryptography
101

Bitcoin

reset to default list

more stack exchange communities

company blog

Stack Exchange

Inbox

Reputation and Badges

+10

101

1

help

Tour

Start here for a quick overview of the site

Help Center

Detailed answers to any questions you might have

Meta

Discuss the workings and policies of this site

About Us

Learn more about Stack Overflow the company

Business

Learn more about hiring developers or posting ads with us

Unix & Linux

Questions

Tags

Users

Badges

Unanswered

Ask Question

How can I get the size of stdin?

up vote
4
down vote

favorite

I'm about to compress a large directory and I want to know how large, exactly, the resulting file will be.

I've tried using du:

$ tar -cv dir | du -h -
du: cannot access '-': No such file or directory

Then I tried using the file version of '-':

$ tar -cv dir | du -h /dev/stdin
1.0K

I'm certain this number isn't accurate. How can I get the size of stdin?

files pipe disk-usage stdin size

share|edit|flag

edited Jun 1 '15 at 21:52

Gilles

431k868041296

asked Jun 1 '15 at 1:36

strugee

7,12794487

add a comment |

start a bounty

4 Answers
4

active

oldest

votes

up vote
6
down vote

accepted

tl;dr: tar -cv dir | wc -c - | cut -d' ' -f 1 | awk '{print $1/1000"K"}'

du doesn't actually count the size of the file itself. It just asks the kernel to query the filesystem, which already keeps track of file size. This is why it's so fast. Because of that, and the fact that you're counting a stream, not a file, du doesn't work. My guess is that 1.0K is a hardcoded size for /dev/std* in the kernel.

The solution is to use wc -c, which counts bytes itself instead of querying the kernel:

$ tar -cv dir | wc -c

If you want output similar to du -h:

$ tar -cv dir | wc -c | awk '{print $1/1000"K"}'

The awk turns the number into a human-readable result.

share|edit|flag

edited Jun 1 '15 at 18:18

answered Jun 1 '15 at 1:36

strugee

7,12794487

8

upvote

flag

Note that if you just omit wc's superfluous - then you don't need the subsequent cut command either.
– Janis
Jun 1 '15 at 2:11

add a comment |

up vote
5
down vote

I'd suggest:

tar cf - dir | wc -c

A simple c (no leading - is required) is used to create a tar archive, f specifies an output file and - denotes that it be stdout. (Note that if you want just the size and there are many files beneath dir you may rather omit tar's v for performance reasons.)

share|edit|flag

answered Jun 1 '15 at 2:05

Janis

8,58111032

upvote

flag

@mikeserv; I seem to recall that I worked with tars in the past where specifying a tar-file (thus f and -) were required. - Just looked it up; without f - the tar assumed /etc/mt0 as default.
– Janis
Jun 1 '15 at 3:00

1

upvote

flag

I've looked it up in a book that I used as a manual at that time, and I think was based on SysV R4. Few folks certainly recall what /etc/mt0 actually means - "magnetic tape" ;-) I'd be interested how Solaris' tar behaves (because Solaris is one of the contemporary OSes that are known to have still really old stuff in /bin).
– Janis
Jun 1 '15 at 4:32

upvote

flag

@mikeserv; PS: The book mentions AT&T's "UNIX Programmers Manual Volumes 1, 2A, 2B" as source (but no manual date or UNIX release version, though; but must have been from the early 1980's, 1983, or so).
– Janis
Jun 1 '15 at 4:41

upvote

flag

Have you ever seen this? Unrelated - but I just found it today, and thought you might like it.
– mikeserv
Jun 1 '15 at 5:10

upvote

flag

I don't quite understand how this answer is different from mine. is it the presence of the -f flag to tar?
– strugee
Jun 1 '15 at 18:20

add a comment |
show 2 more comments

up vote
5
down vote

With GNU tar you can just do:

tar --totals -c . >/dev/null

...which will render output like...

Total bytes written: 5990400 (5.8MiB, 5.5GiB/s)

...on stderr. Similarly, with any tar (or stream) you can use dd to deliver a report on byte counts. This may or may not be preferable to wc, but dd defaults to a block-size of 512 bytes - which is identical to tar's block-size. If your system's PIPE_BUF is large enough, you can even expand dd's block-size to match tar's record size - which is 20 blocks, or 10240 bytes. Like this:

tar -c . | dd bs=bx20 >/dev/null
585+0 records in
585+0 records out
5990400 bytes (6.0 MB) copied, 0.0085661 s, 699 MB/s

This may or may not offer a more performant solution than wc.

In both the dd and tar use-cases you needn't actually dispose of the stream, though. I redirect to /dev/null above - but I could have as easily redirected to some file and still received the report on its size at the time it was written.

share|edit|flag

edited Apr 13 at 12:36

Community♦

1

answered Jun 1 '15 at 4:38

mikeserv

40.4k455128

upvote

flag

In case the file gets written to disk there would be no use to separately determine the size since that information is stored with the file. (+1) for the expected performance gain of dd (compared to wc).
– Janis
Jun 1 '15 at 4:53

1

upvote

flag

@Janis - possibly true in the simplest case - but imagine rather that dd's output is passed on - to a compressor, say - and for whatever reason you find it desirable to know both the raw size of the archive and the compressed one. Also useful is to get an instant report on the record counts - tar is not just an archive, but a stream format. It can be used in ways other than just saving to a group of files to some other file. It is often useful for blocking a stream before modifying it. At each of those record boundaries is a whole block of NULs.
– mikeserv
Jun 1 '15 at 5:03

add a comment |

up vote
1
down vote

The wording of your question lends itself to the tar ... | wc -c answers above. I originally read your question with a silent assumption that you wanted the size to be reported while it was creating the tar file (perhaps tar's output was then being piped over a network link?).

In which case, I'd suggest pv -- pipe viewer. I've seen reference to it but have not yet had a chance to play with it.

References

http://www.catonmat.net/blog/unix-utilities-pipe-viewer/

http://www.ivarch.com/programs/quickref/pv.shtml

share|edit|flag

edited Jun 2 '15 at 1:20

cuonglm

82.9k17150236

answered Jun 2 '15 at 0:38

Jeff Schaller

17.7k72960

add a comment |

Your Answer

draft saved

draft discarded

community wiki

discard

Not the answer you're looking for? Browse other questions tagged files pipe disk-usage stdin size or ask your own question.

asked

2 years, 2 months ago

viewed

1,635 times

active

2 years, 2 months ago

Featured on Meta

Control the types of email you receive via our new Email Settings feature

Linked

48

When is dd suitable for copying data? (or, when are read() and write() partial)

Related

30
Getting size with du of files only

3
Fastest way to get list of all file sizes

2
How to get the distribution of file sizes?

1
Redirecting stdin with stdout to file

11
How to get folder size ignoring hard links?

2
How do I make get-iplayer produce watchable quality without huge filesize?

2
How to list directory/file sizes in general on UNIX-like systems?

0
Get recursive file count (like `du`, but number of files instead of size)

-4
How can I show the size of all files in my ~/.cache directory?

1
macOS: possible that du can report the wrong size for disk usage?

Hot Network Questions

Breakers loose in main panel, does whole panel need to be replaced?

Partition sublists of a list

Why would GPS availability be reduced by high demand (or solar eclipse)?

Can a squib or a muggle be a seer?

Can squares of side 1/2, 1/3, 1/4, … be packed into three quarters of a unit square?

Travelling within Japan when Philppine passport expiry is within 6 months from the trip

Is there any equivalent for this Persian idiom: "Like a cow with a white forehead"?

Is there any significance to 'Fantastic Beasts' taking place in the year of Voldemort's birth?

Physical intuition for why proper time is an invariant quantity

How do you revise material that you already half-know, without getting bored and demotivated?

Do politicians use bug trackers or version control etc.?

How much weight can a light bulb socket hold?

Kill it With Fire

What will be the negation of this statement:

How do I begin to explain that my orcs may survive by eating soil?

How do I deal with someone people think is funny, but I think is mean?

Rocky desert or Sand desert - Which one to chose?

Is infinity larger than 1?

Do wizards need Muggles?

What is a person that works with make-up called?

Curious use of the definite article with body parts in dictionary definitions

How do seaplanes do run-up checks without brakes?

Why did Commodore fill their power supplies with epoxy?

Is the face on the front of the truck a Marvel approved likeness of the Green Goblin?

question feed

Unix & Linux

Tour

Help

Chat

Contact

Feedback

Mobile

Company

Stack Overflow

Stack Overflow Business

Developer Jobs

About

Press

Legal

Privacy Policy

Stack Exchange
Network

Technology

Life / Arts

Culture / Recreation

Science

Other

Stack Overflow

Server Fault

Super User

Web Applications

Ask Ubuntu

Webmasters

Game Development

TeX - LaTeX

Software Engineering

Unix & Linux

Ask Different (Apple)

WordPress Development

Geographic Information Systems

Electrical Engineering

Android Enthusiasts

Information Security

Database Administrators

Drupal Answers

SharePoint

User Experience

Mathematica

Salesforce

ExpressionEngine® Answers

Blender

Network Engineering

Cryptography

Code Review

Magento

Software Recommendations

Signal Processing

Emacs

Raspberry Pi

Programming Puzzles & Code Golf

Ethereum

Data Science

Arduino

more (26)

Photography

Science Fiction & Fantasy

Graphic Design

Movies & TV

Music: Practice & Theory

Worldbuilding

Seasoned Advice (cooking)

Home Improvement

Personal Finance & Money

Academia

Law

more (17)

English Language & Usage

Skeptics

Mi Yodeya (Judaism)

Travel

Christianity

English Language Learners

Japanese Language

Arqade (gaming)

Bicycles

Role-playing Games

Anime & Manga

Puzzling

Motor Vehicle Maintenance & Repair

more (32)

MathOverflow

Mathematics

Cross Validated (stats)

Theoretical Computer Science

Physics

Chemistry

Biology

Computer Science

Philosophy

more (10)

Meta Stack Exchange

Stack Apps

API

Data

Area 51

Blog

Facebook

Twitter

LinkedIn

site design / logo © 2017 Stack Exchange Inc; user contributions licensed under cc by-sa 3.0
with attribution required.
rev 2017.8.8.26722

Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group.
This site is not affiliated with Linus Torvalds or The Open Group in any way.