Command linehardBash

Text Processing with Grep, Sed, and Awk

Powerful text manipulation commands for searching, filtering, and transforming text data.

01

The problem

You need to search through logs, extract specific data, or transform text files.

02

The solution

Bash
# Grep patterns
grep "pattern" file.txt
grep -r "error" /var/log/      # Recursive
grep -i "warning" file.txt     # Case insensitive
grep -v "exclude" file.txt     # Invert match
grep -E "pattern1|pattern2"    # Extended regex
grep -A 2 -B 2 "context"       # Show 2 lines before/after
grep -c "count" file.txt       # Count matches

# Sed stream editing
sed 's/old/new/g' file.txt     # Replace all occurrences
sed 's/old/new/2' file.txt     # Replace 2nd occurrence per line
sed '5,10d' file.txt           # Delete lines 5-10
sed -n '10,20p' file.txt       # Print lines 10-20 only
sed '/pattern/d' file.txt      # Delete lines matching pattern
sed -i.bak 's/old/new/' file   # In-place with backup

# Awk text processing
awk '{print $1}' file.txt      # Print first column
awk -F: '{print $1}' /etc/passwd # Custom delimiter
awk '/pattern/ {print $0}'     # Print matching lines
awk 'NR > 5 && NR < 10'        # Lines 6-9
awk '{sum+=$3} END {print sum}' # Sum column 3
awk '!seen[$0]++' file.txt     # Remove duplicates

# Combine commands
grep "error" log.txt | awk '{print $2, $5}' | sort | uniq -c
cat access.log | awk '{print $1}' | sort | uniq -c | sort -nr

# Cut columns
cut -d',' -f1,3 file.csv       # Comma delimited, fields 1&3
cut -f1-5 file.tsv             # Tab delimited

# Sort and unique
sort file.txt | uniq
sort -u file.txt               # Unique sort
sort -nr -k2 data.txt          # Numeric reverse by column 2

# Word count
wc -l file.txt                 # Line count
wc -w file.txt                 # Word count
wc -m file.txt                 # Character count

# Head and tail
head -20 file.txt
tail -f log.txt                # Follow (live view)
tail -n +100 file.txt          # Start from line 100

03

Put it to work

Example
# Extract IP addresses from logs
grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' access.log | sort | uniq -c | sort -nr

# Replace all tabs with commas
sed 's/\t/,/g' data.tsv > data.csv

# Sum values in second column
awk '{sum+=$2} END {print "Total:", sum}' sales.txt

# Find top 10 largest files in list
ls -la | awk '{print $5, $9}' | sort -nr | head -10

# Monitor error logs in real-time
tail -f /var/log/nginx/error.log | grep -i "error\|warning"

Worth knowing

Awk is a full programming language. Sed uses regex patterns. Use -E for extended regex in grep. Always test on copies before in-place editing.