Raju Ginni

wordpress tutorials seo hosting etc

You are here: Home / Linux sysadmin tutorials linux system administrator / Linux Text processing commands in with Examples

Linux Text processing commands in with Examples

Linux provides various text manipulation tools and commands for processing and modifying text files efficiently. Here are some of the most commonly used ones:

Table of Contents

1. Viewing and Extracting Text

  • cat file.txt – Display entire file content.
  • tac file.txt – Display file content in reverse.
  • less file.txt – View file one page at a time.
  • head -n 10 file.txt – Show the first 10 lines.
  • tail -n 10 file.txt – Show the last 10 lines.
  • cut -d':' -f1 /etc/passwd – Extract specific fields from a file.

2. Searching and Filtering

  • grep "pattern" file.txt – Search for a pattern in a file.
  • grep -i "pattern" file.txt – Case-insensitive search.
  • grep -r "pattern" /path/to/dir – Recursive search.
  • awk '{print $1}' file.txt – Print specific columns from a file.
  • sed -n '/pattern/p' file.txt – Print lines matching a pattern.

3. Modifying and Replacing Text

  • sed 's/old/new/g' file.txt – Replace text in a file.
  • awk '{ gsub("old", "new"); print }' file.txt – Replace globally using awk.
  • tr 'a-z' 'A-Z' – Convert lowercase to uppercase.
  • tr -d '[:digit:]' – Remove digits from input.

4. Sorting and Unique Operations

  • sort file.txt – Sort lines alphabetically.
  • sort -r file.txt – Sort in reverse order.
  • uniq file.txt – Remove duplicate lines.
  • sort file.txt | uniq -c – Count occurrences of unique lines.

5. Counting and Summarizing

  • wc -l file.txt – Count lines in a file.
  • wc -w file.txt – Count words in a file.
  • wc -c file.txt – Count characters in a file.

6. Combining and Splitting Files

  • paste file1.txt file2.txt – Merge lines from multiple files.
  • join file1.txt file2.txt – Join two files based on a common field.
  • split -l 100 file.txt part_ – Split a file into smaller chunks.

7. Formatting and Editing

  • fmt -w 80 file.txt – Format text to a specific width.
  • nl file.txt – Number lines in a file.
  • pr -h "Title" file.txt – Format file for printing.

8. Advanced Processing

  • awk '{sum+=$1} END {print sum}' file.txt – Sum values in a column.
  • sed '/pattern/d' file.txt – Delete lines matching a pattern.
  • perl -pe 's/old/new/g' file.txt – Perform regex-based replacements.

These commands are powerful when combined using pipes (|) and redirection (> and >>).

AWK Command SED Command Use Case
awk '{print}' file.txt sed '' file.txt Print entire file content.
awk '{print $2}' file.txt sed -n 's/.*\(pattern\).*/\1/p' file.txt Print a specific column (extract a pattern).
awk '/error/ {print}' file.txt sed -n '/error/p' file.txt Print lines matching "error".
awk '{print NR, $0}' file.txt sed '=' file.txt Print line numbers along with content.
awk 'NF > 5' file.txt sed -n '/\(\w\+\s\+\)\{5,\}/p' file.txt Print lines with more than 5 words.
awk '{sum += $2} END {print sum}' file.txt `sed -n ‘s/.* [0−9]\+[0-9]\+$/\1/p’ file.txt paste -sd+
awk 'NR==3 {print}' file.txt sed -n '3p' file.txt Print only the third line.
awk 'NR==3, NR==5 {print}' file.txt sed -n '3,5p' file.txt Print lines from 3 to 5.
awk '$2 > 100' file.txt sed -n '/ [1-9][0-9][0-9] /p' file.txt Print lines where column 2 is greater than 100.
awk '!seen[$1]++' file.txt sed -n '/^\(.*\)$/!d' file.txt Print unique values from the first column.
awk 'NR % 2 == 0' file.txt sed -n 'n;p' file.txt Print only even-numbered lines.
awk '{print tolower($0)}' file.txt sed 's/.*/\L&/' file.txt Convert text to lowercase.
awk '{print toupper($0)}' file.txt sed 's/.*/\U&/' file.txt Convert text to uppercase.
awk '{$1=""; print}' file.txt sed 's/^[^ ]* //' file.txt Remove the first column.
awk '{$NF=""; print}' file.txt sed 's/ [^ ]*$//' file.txt Remove the last column.
awk 'sub(/error/, "warning")' file.txt sed 's/error/warning/g' file.txt Replace "error" with "warning".
awk '{print $1, $3}' file.txt sed 's/\(\S\+\) \S\+/\1/' file.txt Print only columns 1 and 3.
awk 'NR>5 {print}' file.txt sed '1,5d' file.txt Delete the first 5 lines.
awk 'NR==10 {exit}' file.txt sed '10q' file.txt Stop processing after line 10.
awk '{printf "%-10s %-10s\n", $1, $2}' file.txt sed -E 's/(.*) (.*)/\1 \2/' file.txt Format output into a table.

Linux provides powerful text processing commands that allow you to manipulate, search, filter, and transform text files efficiently. Here are some of the most commonly used text processing commands along with examples:

1. cat (Concatenate and Display Files)

Example:

cat file.txt

Output: Displays the entire content of file.txt.

cat file1.txt file2.txt > merged.txt

Output: Merges file1.txt and file2.txt into merged.txt.

2. tac (Reverse File Content)

Example:

tac file.txt

Output: Displays the content of file.txt in reverse order (last line first).

3. less (View File Page by Page)

Example:

less file.txt

Usage: Navigate using arrow keys or q to quit.

4. head (Display First N Lines)

Example:

head -5 file.txt

Output: Shows the first 5 lines of file.txt.

5. tail (Display Last N Lines)

Example:

tail -10 file.txt

Output: Shows the last 10 lines of file.txt.

tail -f /var/log/syslog

Usage: Continuously monitors a log file in real time.

6. grep (Search for a Pattern in a File)

Example:

grep "error" file.txt

Output: Displays lines containing “error” in file.txt.

grep -i "warning" file.txt

Usage: Case-insensitive search for “warning”.

grep -r "failed" /var/log/

Usage: Recursively searches for “failed” in /var/log/ directory.

7. awk (Pattern Scanning and Processing)

Example:

awk '{print $1}' file.txt

Output: Prints the first column of each line in file.txt.

awk -F ':' '{print $1, $3}' /etc/passwd

Usage: Prints the first and third field of /etc/passwd, using : as a delimiter.

8. sed (Stream Editor for Text Replacement)

Example:

sed 's/old/new/g' file.txt

Usage: Replaces all occurrences of “old” with “new” in file.txt.

sed '/error/d' file.txt

Usage: Deletes lines containing “error” in file.txt.

9. cut (Extract Specific Columns)

Example:

cut -d':' -f1 /etc/passwd

Output: Extracts the first field (username) from /etc/passwd.

cut -c1-5 file.txt

Usage: Extracts the first five characters from each line.

10. tr (Translate or Delete Characters)

Example:

echo "hello world" | tr 'a-z' 'A-Z'

Output: Converts lowercase to uppercase: HELLO WORLD.

echo "hello123" | tr -d '[:digit:]'

Output: Removes numbers: hello.

11. sort (Sort Lines in a File)

Example:

sort file.txt

Output: Sorts the lines in ascending order.

sort -r file.txt

Usage: Sorts in descending order.

12. uniq (Remove Duplicate Lines)

Example:

sort file.txt | uniq

Output: Removes duplicate lines after sorting.

uniq -c file.txt

Usage: Counts occurrences of unique lines.

13. wc (Count Lines, Words, and Characters)

Example:

wc -l file.txt

Output: Counts the number of lines in file.txt.

wc -w file.txt

Usage: Counts words in file.txt.

14. paste (Merge Lines from Multiple Files)

Example:

paste file1.txt file2.txt

Output: Combines corresponding lines from file1.txt and file2.txt.

15. nl (Number Lines in a File)

Example:

nl file.txt

Output: Numbers each line in file.txt.

16. pr (Format File for Printing)

Example:

pr -h "Title" file.txt

Output: Adds a header “Title” to file.txt before printing.

17. split (Split a File into Smaller Files)

Example:

split -l 50 file.txt part_

Usage: Splits file.txt into multiple smaller files with 50 lines each.

18. tee (Write Output to File and Terminal)

Example:

ls -l | tee output.txt

Usage: Displays the ls -l output on the screen and saves it in output.txt.

19. diff (Compare Two Files)

Example:

diff file1.txt file2.txt

Output: Shows differences between file1.txt and file2.txt.

20. comm (Compare Sorted Files Line by Line)

Example:

comm file1.txt file2.txt

Output: Displays common and unique lines from two sorted files.

Linux Text Processing with awk and sed – Cheat Sheet with Examples

awk and sed are powerful command-line tools used for text processing in Linux. This guide provides detailed examples of both commands for filtering, replacing, and manipulating text.

📌 sed (Stream Editor) – Text Substitution & Manipulation

sed is used for searching, replacing, deleting, and modifying text in a stream or file.

1️⃣ Replace a Word in a File

bash
sed 's/oldword/newword/g' file.txt

🔹 Example: Replace "error" with "warning" in log.txt.

bash
sed 's/error/warning/g' log.txt

2️⃣ Replace Word Only in Specific Line

bash
sed '3s/error/warning/' file.txt

🔹 Example: Replace "error" with "warning" only on line 3.

3️⃣ Delete a Specific Line

bash
sed '5d' file.txt

🔹 Deletes line 5 from file.txt.

4️⃣ Delete Lines Containing a Specific Word

bash
sed '/error/d' file.txt

🔹 Deletes all lines containing "error".

5️⃣ Print Only Matching Lines

bash
sed -n '/error/p' file.txt

🔹 Prints only the lines that contain “error”.

6️⃣ Replace Text in a File and Save Output

bash
sed -i 's/old/new/g' file.txt

🔹 Modifies the file in place by replacing "old" with "new".

7️⃣ Remove Empty Lines

bash
sed '/^$/d' file.txt

🔹 Deletes all empty lines.

8️⃣ Insert Text at the Beginning of Each Line

bash
sed 's/^/Prefix: /' file.txt

🔹 Adds "Prefix: " at the beginning of each line.

9️⃣ Insert a Line Before a Match

bash
sed '/pattern/i\New Line Here' file.txt

🔹 Inserts "New Line Here" before lines matching "pattern".

🔟 Append a Line After a Match

bash
sed '/pattern/a\New Line Here' file.txt

🔹 Appends "New Line Here" after lines matching "pattern".

📌 awk (Pattern Scanning and Processing Language)

awk is used for filtering, formatting, and processing text in structured data.

1️⃣ Print the Entire File

bash
awk '{print}' file.txt

🔹 Prints all lines in file.txt.

2️⃣ Print a Specific Column

bash
awk '{print $2}' file.txt

🔹 Prints the second column.

3️⃣ Print Multiple Columns

bash
awk '{print $1, $3}' file.txt

🔹 Prints first and third columns.

4️⃣ Print Lines Matching a Pattern

bash
awk '/error/ {print}' file.txt

🔹 Prints only lines containing "error".

5️⃣ Print Line Numbers

bash
awk '{print NR, $0}' file.txt

🔹 Prints each line with a line number.

6️⃣ Print Lines with More than 5 Words

bash
awk 'NF > 5' file.txt

🔹 Prints lines with more than 5 fields (words).

7️⃣ Sum a Column of Numbers

bash
awk '{sum += $2} END {print "Total:", sum}' file.txt

🔹 Calculates the sum of the second column.

8️⃣ Find the Maximum Value in a Column

bash
awk 'NR == 1 || $2 > max {max = $2} END {print "Max:", max}' file.txt

🔹 Finds the largest number in the second column.

9️⃣ Find Unique Values in a Column

bash
awk '!seen[$1]++' file.txt

🔹 Prints only unique values from the first column.

🔟 Format Output Like a Table

bash
awk '{printf "%-10s %-10s\n", $1, $2}' file.txt

🔹 Aligns the first two columns into a table format.

📊 Combining awk and sed

You can combine both tools for powerful text processing.

🔹 Remove Empty Lines and Print Specific Columns

bash
sed '/^$/d' file.txt | awk '{print $1, $3}'

🔹 Removes empty lines and prints column 1 and 3.

🔹 Replace a Word and Print Specific Lines

bash
sed 's/error/warning/g' file.txt | awk '/warning/ {print}'

🔹 Replaces "error" with "warning" and prints only matching lines.