The AWK utility is a data extraction and reporting tool that uses a data-driven scripting language consisting of a set of actions to be taken against textual data (either in files or data streams) for the purpose of producing formatted reports.

learn more… | top users | synonyms

38
votes
12answers
46k views

Do you have any useful awk and grep scripts for parsing apache logs?

I can use log analyzers, but often I need to parse recent web logs to see what's happening at the moment. I sometimes do things like to figure out top 10 ips that request a certain file cat foo.log ...
23
votes
8answers
43k views

how to find out mac addresses of all machines on network

Is there some easy way to find out mac address of all machines on my network rather than doing an SSH into each and ifconfig | grep HWaddr if there are 300 machines on network I really need some easy ...
20
votes
7answers
6k views

bash/sed/awk/etc remove every other newline

a bash commands outputs this: Runtime Name: vmhba2:C0:T3:L14 Group State: active Runtime Name: vmhba3:C0:T0:L14 Group State: active unoptimized Runtime Name: vmhba2:C0:T1:L14 Group State: active ...
19
votes
3answers
803 views

Best way to get the MAC of eth0?

Is there a more efficient way to retrieve the MAC address of a NIC in Linux? This works: ip link show dev eth0 | awk ' /link\/ether/ { print $2 }' but can it be found via something like: cat ...
16
votes
7answers
1k views

Can awk be used instead?

I would like to get the number from rating as output from this # nc localhost 9571 language: language:en_ZA.UTF-8 language:en_ZW.UTF-8 session-with-name:Ubuntu Classic (No effects):gnome-session ...
7
votes
2answers
971 views

bash sub-string extraction for specified start and end character

For example I have a log file having this entry: [Wed Aug 08 11:39:41 2012] [error] [client 155.94.70.224] ModSecurity: [file ...
7
votes
2answers
1k views

How to get all fingerprints for .ssh/authorized_keys(2) file

Is there a simple way to get a list of all fingerprints entered in the .ssh/authorized_keys || .ssh/authorized_keys2 file? ssh-keygen -l -f .ssh/authorized_keys will only return fingerprint of ...
7
votes
2answers
3k views

How to split a PEM file

Note : This is not really a question because I already found the answer but since I didn't find it easily here I will post it so that it can benefit others. Question : How to read a concatenated PEM ...
6
votes
4answers
4k views

Linux shell command to filter a text file by line length

I have a 30gb disk image of a borked partition (think dd if=/dev/sda1 of=diskimage) that I need to recover some text files from. Data carving tools like foremost only work on files with well defined ...
5
votes
3answers
12k views

How to delete all characters in one line after “]” with sed?

How to delete all characters in one line after "]" with sed ? Im trying to grep some file using cat, awk. Now my oneliner returns me something like 121.122.121.111] other characters in logs from ...
5
votes
1answer
314 views

File size limit exceeded in bash

I have tried this shell script on a SUSE 10 server, kernel 2.6.16.60, ext3 filesystem the script has problem like this: cat file | awk '{print $1" "$2" "$3}' | sort -n > result the file's size ...
4
votes
2answers
1k views

AWK and file names with space in it.

I'm trying to parse files with awk to change their names. Everything went well until im started to do this with files with space in file name. File names are something like 11237_712312955_2012-01-04 ...
4
votes
5answers
1k views

Is it worth while to learn Awk?

I am decent with bash scripting and I am catching on to regex, and a little sed usage. Is learning awk still worth while with all the alternatives out there. Opinions?
4
votes
3answers
378 views

Determine number of unique lines with awk or similar in bash

I am using AWK to read through a custom log file I have. The format is something like this: [12:08:00 +0000] 192.168.2.3 98374 "CONNECT 192.168.2.4:8091 HTTP/1.0" 200 Right now, I have AWK (from ...
4
votes
4answers
6k views

Parse and convert ini into bash array variables

I'm trying to convert an ini file into bash array variables. The sample ini is as below [foobar] session=foo path=/some/path [barfoo] session=bar path=/some/path so these become ...
4
votes
2answers
233 views

Perl search & replace

How can I use search and replace to change the contents of a file. I need to be able to change the 'all to none' and/or 'none to all' without having to manually do it every time for hundreds of ...
3
votes
3answers
439 views

Year Month day from file name in shell script

I hava file names like below adn_DF9D_20140515_0001.log adn_DF9D_20140515_0002.log adn_DF9D_20140515_0003.log adn_DF9D_20140515_0004.log adn_DF9D_20140515_0005.log adn_DF9D_20140515_0006.log ...
3
votes
2answers
1k views

Multiline data list (ldif file) manipulation with awk or other tool

I'm trying to manipulate an ldif file with multiple entries. My purpose is to parse this existing ldif file, extracting the "givenName" and "sn" attributes, so to generate a "mail" attribute. I was ...
3
votes
2answers
222 views

Extracting a line section of mysql backup using sed

I occasionally need to extract a single record from a mysqlbackup To do this, I first extract the single table I want from the backup... sed -n -e '/CREATE TABLE.*usertext/,/CREATE TABLE/p' ...
3
votes
2answers
2k views

Color awk output on commandline?

When i process a logfile in awk, is there a possibility to colorize the output on commandline?
3
votes
1answer
324 views

Alternatives to gnuplot+awk?

I've needed to do some more sophisticated graphing and plotting as of late. While I know that gnuplot is up to the task, and can do it all from the command line, are there other, simpler or more ...
3
votes
2answers
688 views

How do I use sed or awk to remove text?

I have the following line in a file: Linux Release............5.4.2.0-02 12_12_2011_07:31:23 How do I remove all characters before the first number with sed or awk? I wish to get the ...
3
votes
1answer
134 views

Unexpected Awk results with sh -c

I am trying to sudo a command that uses Awk, and it looks like awk works differently inside sh -c. echo '1 2' | awk '{print $2}' 2 sh -c "echo '1 2' | awk '{print $2}'" 1 2 Why is this ...
3
votes
2answers
902 views

Octal representation of file/folder permissions using awk?

We have various of machines here with all sorts of hardware and Operating Systems, most of them do regular tasks with bash that an agent executes. However it came to my attention that some Solaris ...
2
votes
5answers
659 views

How to remove a certain symbol for a bash script

I have a bash script where I get the disk usage, for example 60%. How can I remove the % symbol? Using grep or awk?
2
votes
2answers
1k views

Puppet & Facter - how to determine if running Cent 6 or Cent 5

How would you use facter and puppet to determine if the OS is running Cent 6.x or Cent 5.x ? facter operatingsystemrelease 6.4 I only care about the major release (6) I've thought about using ...
2
votes
3answers
172 views

linux + match only VALID IP from text file into other file

please advice how to match only the valid IPs ( 255.255.255.255 ) from the file.txt and insert only the valid IP into VALID_IP.txt file ( see VALID_IP.txt for example ) the solution should be ...
2
votes
3answers
101 views

awk ignores the record separator I set for the first record

I am trying to change the record separator but awk does not use the record separator I set until the second record and beyond. I get output like this: ARRAY /dev/md4 level=raid1 num-devices=2 ...
2
votes
7answers
9k views

How to show available disks on a Linux system?

If i run fdisk -l I get an output as: Disk /dev/sda: 120.0 GB, 120034123776 bytes 255 heads, 63 sectors/track, 14593 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: ...
2
votes
4answers
2k views

What command can I use to search sendmail logs for ALL message details involving a specific recipient.

CENTOS 5.x | Sendmail Occasionally I need to search through sendmail delivery logs to find out what happened to a missing message. This usually involves two (or more) steps: STEP 1: Search ...
2
votes
1answer
100 views

Memory usage REGEX

Could you help me to modify the following one-liner to get more precise memory usage - right now it's just an integer I would like to include first value after coma. free -m | awk '/Mem:/ {tot = $2;} ...
2
votes
2answers
152 views

Gathering IP's from a complicated log

I have a question regarding the use of some more advanced grep, awk, sed. I have a log file, for a proprietary MTA, that contains IP's in a string, delimited by [redacted]^~x.x.x.x^[redacted]. So far ...
2
votes
2answers
172 views

awk + perl + get two arguments in to awk syntax

please advice what wrong with my awk syntax and how to fix it ( this syntax is in my ksh script ) and I run my script on linux machine my target is to get from the file.txt only the lines that ...
2
votes
2answers
130 views

Display nth instance of grepped pattern

So here's my problem. I have an issue with a .csv file (Current.csv) in that there are commas randomly place throughout the data, so awk-ing the file doesn't give me constant column numbers for a ...
2
votes
3answers
377 views

Shell script to block proftp failled attempt

I want to filter and block failed attempt to access my proftp server. Here is an example line from the /var/log/secure file: Jan 2 18:38:25 server1 proftpd[17847]: spy1.XYZ.com ...
2
votes
2answers
1k views

How can I add a line break before a set of values going into a bash script?

I've got a set of data piping into a bash script. Here's an example of what that data looks like: "foo1": "Miscellaneous text", "foo2": "More text", "foo3": "blah blah blah", "foo4": ...
2
votes
3answers
1k views

awk on vmstat to get si,so

I am writing a shell script to check the vmstat si and so data at various time intervals vmstat 1 sample output: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b ...
1
vote
3answers
478 views

Passing variables problem - Bash

I have a following problem: #!/bin/bash NUM=`cat accounts | wc -l`; for i in {1..$NUM} do account=`awk "NR==$i" accounts`; echo -e "\nAccount: $account\n"; sudo ./backup_maildir $account; ...
1
vote
5answers
1k views

bash script - print from line N to EOF

I want to be able to print the number of lines from bash as such: Line number (as counted from the top) -> end of the file It seems tail will only count lines from the bottom. Does anyone know how ...
1
vote
3answers
135 views

drop ips with iptables from awk result [closed]

I have a log file like this : Frq:15 IP:0.0.0.0 Date: 2014-03-21 12:19:31 AM Frq:9 IP:198.252.206.25 Date: 2014-03-21 12:19:31 AM Now i get column of ip addresses with awk : cat /tmp/test | awk ...
1
vote
3answers
437 views

append to end of line sed / awk

This is probably easy and something is not clicking for me and my lack of coffee right now. I have a file with multiple lines that begin with a tab then the word GROUP something { Some of these lines ...
1
vote
3answers
158 views

alias to replace a text in file

here is a practical situation.I facing some of my files where there is a common string in these files.I usually will open vi and do some thing like :%s/text/replacement/g and I have to do it in ...
1
vote
4answers
356 views

double awk commands in same pipe?

Hello I'm trying to do something like this but it wont work, can you guys tell me help me out? /usr/bin/mysql -B -r -h ******** -u******* -p****** -D***** \ -e'SELECT `username`, `uid`, `gid`, ...
1
vote
2answers
346 views

Using wget and Awk to count similar expressions

I am trying to create a script that uses wget to download a data set and then awk to sort though the file and tell you the most common filter used which is $14 column. So far I have the wget function ...
1
vote
2answers
800 views

Grep to find lines starting at pattern A until pattern B is matched

I have a log that contains bits like this: [2012-04-16 15:16:43,827: DEBUG/PoolWorker-2] {'feed': {}, 'bozo': 1, 'bozo_exception': URLError(error(110, 'Connection timed out'),), 'entries': []} ...
1
vote
4answers
6k views

remove line break using AWK

I am facing some problem regarding my bash script. Below is my bash code : #!/bin/bash cd /root/Msgs/TESTNEW/new file=myfile.txt var1=$(awk '(NR==30){print $2}' $file) var2=$(awk 'NR>=38 ...
1
vote
5answers
265 views

Extracting lines from a file based on prefix

I need to extract all lines from file2 that begin with an id # prefix contained in file1. File 1 is single column like: 324 399 408 135236 321590 File 2 is multi-column like: 1 [tab] 108 [tab] ...
1
vote
4answers
852 views

Bash: delete all but most recent n files with prefix x

I have a directory of backups that looks like this: -rw-r--r-- 1 ftpuser ftponly 5610595 May 27 00:01 alpha-114.tar.asc -rw-r--r-- 1 ftpuser ftponly 50559368 May 27 00:04 beta-211.tar.asc ...
1
vote
3answers
2k views

using awk in a bash script with if

I'm doing a script in which I need to test a string and based on its result I'll decide if I go further or not. The command below works fine (if the string starts with "Clean" it will print 1, ...
1
vote
1answer
258 views

Get some meaning from apache logs with awk and grep [duplicate]

I am trying to have some meaning from my apache log files: I want to parse my access log and have some statistics about 200 status code hits (how many times each of them was hit). So I tried to learn ...