Help with sed commands
from orsetto@lemmy.dbzer0.com to linux@lemmy.ml on 15 Apr 20:41
https://lemmy.dbzer0.com/post/42286397

Hi all! I have always only used sed with s///, becouse I’ve never been able to figure out how to properly make use of its full capabilities. Right now, I’m trying to filter the output of df -h --output=avail,source to only get the available space from /dev/dm-2 (let’s ignore that I just realized df accepts a device as parameter, which clearly solves my problem).

This is the command I’m using, which works:

df -h --output=avail,source \
    | grep /dev/dm-2 \
    | sed -E 's/^[[:blank:]]*([0-9]+(G|M|K)).*$/\1/

However, it makes use of grep, and I’d like to get rid of it. So I’ve tried with a combiantion of t, T, //d and some other stuff, but onestly the output I get makes no sense to me, and I can’t figure out what I should do instead.

In short, my question is: given the following output

$ df -h --output=avail,source 
Avail Filesystem
  87G /dev/dm-2
 1.6G tmpfs
  61K efivarfs
  10M dev
...

How do I only get 87G using only sed as a filter?

EDIT:

Nevermind, I’ve figured it out…

$ df -h --output=avail,source \
    | sed -E 's/^[[:blank:]]*([0-9]+(G|M|K))[[:blank:]]+(\/dev\/dm-2).*$/\1/; t; /.*/d'
85G

#linux

threaded - newest

unlawfulbooger@lemmy.blahaj.zone on 15 Apr 20:58 next collapse

The easiest way is probably without sed, which you mentioned:

df -h --output=avail /dev/dm-2| tail -n1

But purely with sed it would be something like this:

df -h --output=avail,source | sed -n ‘/\/dev\/dm-2/s!/dev/dm-2!!p’

-n tells sed to not print lines by default

/[regex]/ selects the likes matching regex. We need to escape the slashes inside the regex.

s/// does search-and-replace, and has a special feature: it can use any character, not just a slash. So I used three exclamation points instead , so that I don’t need to escape the slashes. Here we replace the device with the empty string.

p prints the result

Check the sed man page for more details: linux.die.net/man/1/sed

orsetto@lemmy.dbzer0.com on 15 Apr 22:01 collapse

!/dev/dm-2!!p

This weired me out until I read the explanation. I’m so used to the slashes lol

In the end I’ve used the first command you wrote, because KISS, but I appreciate your explanation

Check the sed man page for more details

Yes that’s been my only source so far, but to be honest it’s really cryptic. it might just be that I’m used to syscalls man pages (also sed is kinda complex it can’t be easy to write a single man page for it)

unlawfulbooger@lemmy.blahaj.zone on 16 Apr 06:38 collapse

In the end I’ve used the first command you wrote, because KISS, but I appreciate your explanation

There’s no shame in combining multiple tools, that’s what pipelines are all about 😄.

Also there’s a different tool that I would use if I want to output a specific column: awk

df -h —output=avail,source | awk ‘/\/dev\/dm-2/ {print $1}’

For lines matching /dev/dm-2 print the first column. awk splits columns on whitespace by default.

But I would probably use grep+awk.

Sed is definitely a very powerful tool, which leads to complex documentation. But I really like the filtering options before using the search/replace.

You can select specific lines, with regex or by using a line number; or you can select multiple lines by using a comma to specify a range.

E.g. /mystring/,100s/input/output/g: in the lines starting from the first match of /mystring/ until line 100, replace input with output

orsetto@lemmy.dbzer0.com on 27 Apr 16:01 collapse

There’s no shame in combining multiple tools, that’s what pipelines are all about

Not at all, but some times it’s just funny

You can select specific lines, with regex or by using a line number; or you can select multiple lines by using a comma to specify a range.

Yep, learning this made sed even more useful to me.

I also gave awk a try and now i know what i’ve missed all these years

(Also, sorry for the 12 days old reply :))

unlawfulbooger@lemmy.blahaj.zone on 27 Apr 20:25 collapse

I get it, sometimes you just do something for the challenge.

It’s really great what you can accomplish when you know a little more than the bare minimum of the tools at your disposal (^^,)

And I had the same experience after learning a bit more about awk for the fist time, hahaha.

gnuhaut@lemmy.ml on 15 Apr 21:25 next collapse

You can do something like this to emulate grep:

sed -n '/myregex/p'

The -n suppresses auto-printing. That command should interpreted as: find line matching /myregex/ and then print (p) it.

You can then combine this with s (substitute):

sed '/myregex/ s/from/to/ p`

The complete command is then something like:

LANG=C df -h --output=avail,source | sed -n '/\/dev\/nvme0n1p2/ s/^\s*\([0-9.]\+[KMGT]\).*/\1/ p'

Note that the output can be something like 2.3G, but in my locale that would be 2,3G which is why I added LANG=C.

Easier IMHO is awk:

awk '/myregex/ {print $1}'

prints the first field.

orsetto@lemmy.dbzer0.com on 15 Apr 21:43 next collapse

The -n suppresses auto-printing

That’s something I was missing! It makes the command look definitely less complicated.

Also, what is the difference between separating expressions with a space vs with a semicolon?

Easier IMHO is awk

Eh, that’s another beast. For some reason tho I find sed more appealing.

gnuhaut@lemmy.ml on 15 Apr 21:51 collapse

What expressions?

I mean awk is more powerful, it has variables, function, can do arithmetic and format strings, and such proper scripting language features. And the GNU awk manual is rather well written.

orsetto@lemmy.dbzer0.com on 15 Apr 22:22 collapse

I meant to ask what is the difference between, i.e., sed ‘/myregex/ s/from/to/ p’ and sed ‘/myregex/ s/from/to/ ; p’, but while testing to explain what i meant I answered myself, and in the process I also understood what addresses are ehe

Right, awk is a proper programming language, right? that’ll be for another day…

gnuhaut@lemmy.ml on 15 Apr 22:44 collapse

Oh yeah that, so technically (and I was confused about this), the p in s/from/to/p is not the same as the p command, it’s a flag for the s command that tells it to print the output. You could do multiple commands like /re/ {s/a/b/;p} for the same result, by using a {} block.

If you do, say, /re/ s/a/b/; p those would be separate command, the first does the thing on lines matching /re/, while the p has no address range (e.g. regex) associated with it, so it gets executed for all the lines.

orsetto@lemmy.dbzer0.com on 16 Apr 00:45 collapse

I see. I guess what confused me was that i didn’t understand what addresses were.

Thank you for your explanations :)

gnuhaut@lemmy.ml on 16 Apr 01:43 collapse

Yeah no problem.

What’s maybe interesting is how sed came to be. Back in early days of Unix, they had ed as their editor (or, as some old manpage says, “Ed is the standard text editor.”)

Sed has basically the same commands as ed. You typed 3d to delete the third line, or 10,20p to print lines 10 to 20. They only had teletypes back then, which are basically a keyboard and dot-matrix printer with one of those continuous papers for output, prior to when hardware terminals with CRT screens were a thing.

Anyway, someone thought it would be useful if you could put ed-style editing commands inside shell pipelines, and ed doesn’t work for that. It is used for interactively editing files in place, and so gets commands from stdin. You can’t pipe any files into it. So the “stream editor”, sed, was born.

Also interesting: There were improved modified versions of ed going around like em and later ex. The original vi was a “visual mode” extension for ex, and you can still run ex/ed commands from vi by typing : first, e.g. you can delete line 3 by typing :3d inside vi.

orsetto@lemmy.dbzer0.com on 27 Apr 13:55 collapse

12 days late, but thanks for the bit of history, I always enjoy this stuff :)

bizdelnick@lemmy.ml on 16 Apr 10:27 collapse

You can then combine this with s (substitute):

sed '/myregex/ s/from/to/ p'

This is not combining commands. In your example p is a modifier to the s/// command.

gnuhaut@lemmy.ml on 16 Apr 22:15 collapse

Yeah you’re right, thanks. I did figure this out on my own already, see my other comment:

Oh yeah that, so technically (and I was confused about this), the p in s/from/to/p is not the same as the p command, it’s a flag for the s command that tells it to print the output.

smeg@feddit.uk on 15 Apr 23:36 next collapse

Are you opposed to using awk? Printing out a specific column is basically the only thing I actually know how to do with it: df -h --output=avail,source | awk ‘/dm-2/ {print $1}’

thingsiplay@beehaw.org on 16 Apr 00:01 next collapse

A perfect example when to use AWK.

orsetto@lemmy.dbzer0.com on 16 Apr 00:52 collapse

Are you opposed to using awk?

Not at all, I’m just not familiar with it so I find it confusing.

Although, looking at your command, i think I understand what it means

smeg@feddit.uk on 16 Apr 07:55 collapse

I had to check the syntax because I also don’t use it enough to be sure, definitely a bit weird. Basically just grep for a regex and print the specified column.

thingsiplay@beehaw.org on 16 Apr 08:59 collapse

Just a thumb of rule to make sense of it: A column in AWK is by default any space separated part. You can change the column separator to any other character too with -F “:” in example would be a double colon. There is also a way to print all columns, but with certain exceptions. In example print all, but the third and fourth columns: ls -l | awk ‘{$3=“”; $4=“”; print $0}’ . Admittely I forget this syntax often and have to look for it again.

bizdelnick@lemmy.ml on 16 Apr 10:20 collapse

Just disable printing out with the -n option and add the p modifier to the s/// command to print out lines where substitution has occured. sed -n ‘s/your-regexp/replacement/p’