The AWK utility is a data extraction and reporting tool that uses a data-driven scripting language consisting of a set of actions to be taken against textual data (either in files or data streams) for the purpose of producing formatted reports.

learn more… | top users | synonyms

1
vote
1answer
66 views

How to write a shell script to search from a position in file instead to start from beginning of the file again

I am trying to write a shell script to find value stored after a repetitive word in a big file with millions of lines using for loop. After the finding the repetitive word how i make script to start ...
15
votes
5answers
12k views

How to split a PEM file

Note : This is not really a question because I already found the answer but since I didn't find it easily here I will post it so that it can benefit others. Question : How to read a concatenated PEM ...
10
votes
4answers
12k views

Linux shell command to filter a text file by line length

I have a 30gb disk image of a borked partition (think dd if=/dev/sda1 of=diskimage) that I need to recover some text files from. Data carving tools like foremost only work on files with well defined ...
0
votes
1answer
145 views

How to combine lsof stat and ps to get more info in one line bash command

I can use lsof to get top open files stat for processes, as below: $ lsof -n|awk '{print $2}'|sort|uniq -c|sort -nr | head -n 5 17955 11945 10282 2786 5980 32152 1920 27803 1786 32107 ...
28
votes
8answers
73k views

how to find out mac addresses of all machines on network

Is there some easy way to find out mac address of all machines on my network rather than doing an SSH into each and ifconfig | grep HWaddr if there are 300 machines on network I really need some easy ...
0
votes
2answers
66 views

Shell Script to check CPU Utilisation

I am trying to write a shell script which will check free memory of the server. I am trying below code top -M -n1 | grep "Mem" | awk '{print 0 + $7}' | awk '{ print $1 / 1024 }' | awk '{if ($1 < ...
0
votes
3answers
55 views

Matching first block of text from a file, between 2 strings

I'm trying to match only the first block of text between 2 strings: 'menuentry ' and '}' from a file as follows: Input file: (some irrelevant text) menuentry 'My Customized Linux, with Linux 3.10.0-...
7
votes
5answers
15k views

How to parse and convert ini file into bash array variables?

I'm trying to convert an ini file into bash array variables. The sample ini is as below: [foobar] session=foo path=/some/path [barfoo] session=bar path=/some/path so these become: session[foobar]=...
-1
votes
1answer
54 views

How to use the awk,sed Modify the text?

As follows, some text 16G 16G 1.9G 40G 4G 4G 952M 60G 16G 16G 1.6G 40G 5G 780M 5G 80G I want to change all the unit from 'M' to 'G', like this 16G 16G ...
0
votes
2answers
125 views

Prepend String using perl/awk/sed?

I'm trying to script a change to my apache config file (httpd.conf). I'm trying to match the following string: # # DirectoryIndex: sets the file that Apache will serve if a directory And PREPEND ...
17
votes
8answers
2k views

Can awk be used instead?

I would like to get the number from rating as output from this # nc localhost 9571 language: language:en_ZA.UTF-8 language:en_ZW.UTF-8 session-with-name:Ubuntu Classic (No effects):gnome-session --...
28
votes
8answers
14k views

bash/sed/awk/etc remove every other newline

a bash commands outputs this: Runtime Name: vmhba2:C0:T3:L14 Group State: active Runtime Name: vmhba3:C0:T0:L14 Group State: active unoptimized Runtime Name: vmhba2:C0:T1:L14 Group State: active ...
54
votes
12answers
83k views

Do you have any useful awk and grep scripts for parsing apache logs? [closed]

I can use log analyzers, but often I need to parse recent web logs to see what's happening at the moment. I sometimes do things like to figure out top 10 ips that request a certain file cat foo.log ...
0
votes
1answer
41 views

within awk how to check for a defined value before attempting a calculation

The following awk line will fail when c15 is not defined or is 0. How to modify it so that in such cases it does not attempt the calculation, but instead provides an error message or does nothing: ...
1
vote
2answers
539 views

Special char or wildcard in awk field separator

Would appreciate to know about the special char or wildcard in the following command: /sbin/ifconfig eth0 | awk -F ' *|:' '/inet addr/{print $4}' What does the code below mean? awk -F ' *|:' '
0
votes
1answer
35 views

Limit awk results to html requests

I'm using awk to analyze some access log files. I'm currently using the following: awk '($9 ~ /404/)' access_log | awk '{print $9,$7}' | sort | uniq -c | sort > 404.txt Which returns all of the ...
1
vote
1answer
59 views

How to using sed to change the last character from a match line

How to using sed to change the last character from a match line ? Exemple: /etc/network/interfaces auto eth0 iface eth0 inet static address 150.0.3.50 netmask 255.255.255.0 ...
2
votes
2answers
206 views

AWK + Host lookup

Currently using the following line to get top 20 ips sorted by requests: grep 'GET /' /var/log/nginx/access.log | awk '{ print $1 }' | sort -n | uniq -c | sort -rn | head -20 Output: 575 66.249.*.*...
1
vote
1answer
255 views

Running CAT in SCREEN no output to file

I am trying to code this command to run in screen along with 2 others that will be using cat to read a file then pipe it into awk for filtering. Here is the command; screen -d -m /bin/cat /var/www/...
2
votes
1answer
88 views

how to get the desired value in bash shell from the lshw output?

I want to extract the info from lshw output where it says: -cpu product: ...... I want only the product info of this -cpu. How can I do that with awk or sed?
0
votes
2answers
125 views

Print last column containing spaces and special characters

I need a full path/command thats been execued to start a process thats exposed on the network - listens on tcp/udp . I did manage to get all of that using below "one liner": netstat -nlp46|tail -n +3 ...
-1
votes
1answer
106 views

use awk to execute logrotate command when disk is full

What I'm trying to do is manually run logrotate when my disk is greater than 90% full. I'm trying to do this using df -P and awk. I can get it to display the percentage, if it's > 90, but I can't ...
2
votes
3answers
720 views

Passing variables problem - Bash

I have a following problem: #!/bin/bash NUM=`cat accounts | wc -l`; for i in {1..$NUM} do account=`awk "NR==$i" accounts`; echo -e "\nAccount: $account\n"; sudo ./backup_maildir $account; ...
1
vote
5answers
420 views

Extracting lines from a file based on prefix

I need to extract all lines from file2 that begin with an id # prefix contained in file1. File 1 is single column like: 324 399 408 135236 321590 File 2 is multi-column like: 1 [tab] 108 [tab] ...
2
votes
1answer
848 views

Why don't random numbers in AWK change after the first run?

I'm studying AWK, and when I use the following command for the second time why are the numbers always the same? First time run: awk 'BEGIN{for(i=1;i<=10;i++) print int(101*rand())}' 24 29 85 15 ...
4
votes
2answers
1k views

Split large log file WITHOUT keeping the original (splitting in-place)

I have a 226GB log file, and I want to split it up into chunks for easier xzing. Problem is I only have 177GB of space left in my workable space. Is there a way to split a file in half or into N ...
1
vote
0answers
101 views

how to filter sar -d disk output tidily

sar -D output-I would like a suitable command to run alongside stress -hdd but this is the output which sar -D 1 10 gives. I have filtered it using both awk |'NR > 1 {printf "%4s%8s%8s\n",$13,$14,$...
0
votes
1answer
149 views

Pipe lsof to grep to awk ps -ef to grep

I'm trying to get some info on a certain PID. I'm probably doing this in a manner that is way too difficult, but I'm trying to run this: lsof | grep grepstuff | awk '{ps -ef | grep $2;}' and I'm ...
0
votes
1answer
416 views

Executing command remotely on another linux server from a shell script

I have a shell-script which extract details from a log file between two dates and executes a command on the output to generate some report.The log files are on different server and scripts are ...
1
vote
1answer
95 views

How to substitute a string from an awk result?

I have a JSON file which contain multiple occurrence of replaceme. I only want to substitute an occurrence based on a specific string found a few line above. E.g., [...] "title" : { "...
-2
votes
1answer
689 views

How to insert text from a file into another in bash [closed]

The content in fileA is: Here’s the ip list. ip: 2.2.3.4 ip: 2.2.3.5 ip: 2.2.3.6 ip: 2.2.3.7 The content is fileB is ip: 1.2.3.4 ip: 1.2.3.5 ip: 1.2.3.6 ip: 1.2.3.7 How to use sed command or ...
2
votes
4answers
24k views

remove line break using AWK

I am facing some problem regarding my bash script. Below is my bash code : #!/bin/bash cd /root/Msgs/TESTNEW/new file=myfile.txt var1=$(awk '(NR==30){print $2}' $file) var2=$(awk 'NR>=38 &&...
0
votes
3answers
676 views

Modify config file settings from bash [closed]

I need to modify a config file from a bash script, changing the values of particular settings. I need to make the change using common Linux tools (sed/awk/tr/etc.). The config file can have ...
4
votes
2answers
9k views

Puppet & Facter - how to determine if running Cent 6 or Cent 5

How would you use facter and puppet to determine if the OS is running Cent 6.x or Cent 5.x ? facter operatingsystemrelease 6.4 I only care about the major release (6) I've thought about using ...
0
votes
2answers
277 views

Replace date in epoch (seconds) format to normal in a log file

I am on a ubuntu machine trying to read from log files and found the logs contain date and time in epoch format(seconds). How can I convert these dates in numbers into any readable format from command ...
-1
votes
1answer
1k views

filter body of incoming email in procmail [closed]

I've encountered some problem when configuring my linux machine. We are configuring fetchmail and procmail in order to receive the email from the external mail server.The problem start when we want ...
0
votes
3answers
90 views

Tinydns data file change ip with bash

I am using tinydns and need to dynamicly change some ip in data file. I want to use bash script for it. For example data file: +sub1.example.org:282.196.222.245:14400 +sub2.example.org:278.179.280....
0
votes
1answer
281 views

Regex matching specific details if first match exists, multiline

I'm struggling to get the correct regex to match specifically 'Contact' and 'User-Agent' only if the 'Contact' address matches 10.0.x.x in ~70GB of SIP messages. The SIP message will always contain a ...
15
votes
2answers
6k views

How to get all fingerprints for .ssh/authorized_keys(2) file

Is there a simple way to get a list of all fingerprints entered in the .ssh/authorized_keys || .ssh/authorized_keys2 file? ssh-keygen -l -f .ssh/authorized_keys will only return fingerprint of ...
0
votes
2answers
119 views

How to print regex match

My problme is two fold. First is printing a regex match. The second is the actual regex. Problem 1: I'm trying to print the output of yum grouplist --verbose into a format that I can put into a ...
3
votes
3answers
1k views

Year Month day from file name in shell script

I hava file names like below adn_DF9D_20140515_0001.log adn_DF9D_20140515_0002.log adn_DF9D_20140515_0003.log adn_DF9D_20140515_0004.log adn_DF9D_20140515_0005.log adn_DF9D_20140515_0006.log ...
0
votes
1answer
115 views

Eliminate need for grep use only awk

I am trying to generate a list of PID's to pass to strace I only want the PID's php-fpm. I can get the correct output by using grep with awk but am looking for an awk only answer. Here is what I ...
1
vote
1answer
613 views

Get some meaning from apache logs with awk and grep [duplicate]

I am trying to have some meaning from my apache log files: I want to parse my access log and have some statistics about 200 status code hits (how many times each of them was hit). So I tried to learn ...
1
vote
3answers
374 views

drop ips with iptables from awk result [closed]

I have a log file like this : Frq:15 IP:0.0.0.0 Date: 2014-03-21 12:19:31 AM Frq:9 IP:198.252.206.25 Date: 2014-03-21 12:19:31 AM Now i get column of ip addresses with awk : cat /tmp/test | awk '{...
2
votes
3answers
145 views

awk ignores the record separator I set for the first record

I am trying to change the record separator but awk does not use the record separator I set until the second record and beyond. I get output like this: ARRAY /dev/md4 level=raid1 num-devices=2 ...
0
votes
2answers
903 views

Script for search string inside the file for every one hour

I need to write a script where the particular string(alphanumeric) should be searched inside the file with every hour for whole day. And the search should not contain the previous hour results. Time ...
0
votes
1answer
290 views

Invalidating email addresses from sql dump before import

Synopsis I would like to import production data into the development environment so testing can be done in an almost accurate way. Of course I need to invalidate e-mail addresses as I don't want e-...
0
votes
1answer
75 views

Parse requests per OS type apache log

I'm trying to return a sorted list of requests per OS type, parsed from an Apache access_log file (not combined format as I need the user-agent info). Here's an example of the output I'm looking for: ...
1
vote
3answers
1k views

tail-pipe-awk script, fine on Centos but weird on Ubuntu

I wrote a script on CentOS and now I'm trying to run it on Ubuntu and it is behaving unexpectedly. Here's something like the script, which I run under bash: sudo tail -F /var/opt/my-application/log/...
-1
votes
2answers
256 views

How to find DEVICE on which read per second (rd_sec/s) is highest?

I am using awk to find out DEVICE on which read per second (rd_sec/s) is highest, (from SAR file) below is my command : # sar -d -f sa28 | awk '{print $3,$4}' |sort -k1,1n Output is showing on ...