Linux provides various text manipulation tools and commands for processing and modifying text files efficiently. Here are some of the most commonly used ones:
1. Viewing and Extracting Text
cat file.txt– Display entire file content.tac file.txt– Display file content in reverse.less file.txt– View file one page at a time.head -n 10 file.txt– Show the first 10 lines.tail -n 10 file.txt– Show the last 10 lines.cut -d':' -f1 /etc/passwd– Extract specific fields from a file.
2. Searching and Filtering
grep "pattern" file.txt– Search for a pattern in a file.grep -i "pattern" file.txt– Case-insensitive search.grep -r "pattern" /path/to/dir– Recursive search.awk '{print $1}' file.txt– Print specific columns from a file.sed -n '/pattern/p' file.txt– Print lines matching a pattern.
3. Modifying and Replacing Text
sed 's/old/new/g' file.txt– Replace text in a file.awk '{ gsub("old", "new"); print }' file.txt– Replace globally usingawk.tr 'a-z' 'A-Z'– Convert lowercase to uppercase.tr -d '[:digit:]'– Remove digits from input.
4. Sorting and Unique Operations
sort file.txt– Sort lines alphabetically.sort -r file.txt– Sort in reverse order.uniq file.txt– Remove duplicate lines.sort file.txt | uniq -c– Count occurrences of unique lines.
5. Counting and Summarizing
wc -l file.txt– Count lines in a file.wc -w file.txt– Count words in a file.wc -c file.txt– Count characters in a file.
6. Combining and Splitting Files
paste file1.txt file2.txt– Merge lines from multiple files.join file1.txt file2.txt– Join two files based on a common field.split -l 100 file.txt part_– Split a file into smaller chunks.
7. Formatting and Editing
fmt -w 80 file.txt– Format text to a specific width.nl file.txt– Number lines in a file.pr -h "Title" file.txt– Format file for printing.
8. Advanced Processing
awk '{sum+=$1} END {print sum}' file.txt– Sum values in a column.sed '/pattern/d' file.txt– Delete lines matching a pattern.perl -pe 's/old/new/g' file.txt– Perform regex-based replacements.
These commands are powerful when combined using pipes (|) and redirection (> and >>).
| AWK Command | SED Command | Use Case |
|---|---|---|
awk '{print}' file.txt |
sed '' file.txt |
Print entire file content. |
awk '{print $2}' file.txt |
sed -n 's/.*\(pattern\).*/\1/p' file.txt |
Print a specific column (extract a pattern). |
awk '/error/ {print}' file.txt |
sed -n '/error/p' file.txt |
Print lines matching "error". |
awk '{print NR, $0}' file.txt |
sed '=' file.txt |
Print line numbers along with content. |
awk 'NF > 5' file.txt |
sed -n '/\(\w\+\s\+\)\{5,\}/p' file.txt |
Print lines with more than 5 words. |
awk '{sum += $2} END {print sum}' file.txt |
`sed -n ‘s/.* [0−9]\+[0-9]\+$/\1/p’ file.txt | paste -sd+ |
awk 'NR==3 {print}' file.txt |
sed -n '3p' file.txt |
Print only the third line. |
awk 'NR==3, NR==5 {print}' file.txt |
sed -n '3,5p' file.txt |
Print lines from 3 to 5. |
awk '$2 > 100' file.txt |
sed -n '/ [1-9][0-9][0-9] /p' file.txt |
Print lines where column 2 is greater than 100. |
awk '!seen[$1]++' file.txt |
sed -n '/^\(.*\)$/!d' file.txt |
Print unique values from the first column. |
awk 'NR % 2 == 0' file.txt |
sed -n 'n;p' file.txt |
Print only even-numbered lines. |
awk '{print tolower($0)}' file.txt |
sed 's/.*/\L&/' file.txt |
Convert text to lowercase. |
awk '{print toupper($0)}' file.txt |
sed 's/.*/\U&/' file.txt |
Convert text to uppercase. |
awk '{$1=""; print}' file.txt |
sed 's/^[^ ]* //' file.txt |
Remove the first column. |
awk '{$NF=""; print}' file.txt |
sed 's/ [^ ]*$//' file.txt |
Remove the last column. |
awk 'sub(/error/, "warning")' file.txt |
sed 's/error/warning/g' file.txt |
Replace "error" with "warning". |
awk '{print $1, $3}' file.txt |
sed 's/\(\S\+\) \S\+/\1/' file.txt |
Print only columns 1 and 3. |
awk 'NR>5 {print}' file.txt |
sed '1,5d' file.txt |
Delete the first 5 lines. |
awk 'NR==10 {exit}' file.txt |
sed '10q' file.txt |
Stop processing after line 10. |
awk '{printf "%-10s %-10s\n", $1, $2}' file.txt |
sed -E 's/(.*) (.*)/\1 \2/' file.txt |
Format output into a table. |
Linux provides powerful text processing commands that allow you to manipulate, search, filter, and transform text files efficiently. Here are some of the most commonly used text processing commands along with examples:
1. cat (Concatenate and Display Files)
Example:
Output: Displays the entire content of file.txt.
Output: Merges file1.txt and file2.txt into merged.txt.
2. tac (Reverse File Content)
Example:
Output: Displays the content of file.txt in reverse order (last line first).
3. less (View File Page by Page)
Example:
Usage: Navigate using arrow keys or q to quit.
4. head (Display First N Lines)
Example:
Output: Shows the first 5 lines of file.txt.
5. tail (Display Last N Lines)
Example:
Output: Shows the last 10 lines of file.txt.
Usage: Continuously monitors a log file in real time.
6. grep (Search for a Pattern in a File)
Example:
Output: Displays lines containing “error” in file.txt.
Usage: Case-insensitive search for “warning”.
Usage: Recursively searches for “failed” in /var/log/ directory.
7. awk (Pattern Scanning and Processing)
Example:
Output: Prints the first column of each line in file.txt.
Usage: Prints the first and third field of /etc/passwd, using : as a delimiter.
8. sed (Stream Editor for Text Replacement)
Example:
Usage: Replaces all occurrences of “old” with “new” in file.txt.
Usage: Deletes lines containing “error” in file.txt.
9. cut (Extract Specific Columns)
Example:
Output: Extracts the first field (username) from /etc/passwd.
Usage: Extracts the first five characters from each line.
10. tr (Translate or Delete Characters)
Example:
Output: Converts lowercase to uppercase: HELLO WORLD.
Output: Removes numbers: hello.
11. sort (Sort Lines in a File)
Example:
Output: Sorts the lines in ascending order.
Usage: Sorts in descending order.
12. uniq (Remove Duplicate Lines)
Example:
Output: Removes duplicate lines after sorting.
Usage: Counts occurrences of unique lines.
13. wc (Count Lines, Words, and Characters)
Example:
Output: Counts the number of lines in file.txt.
Usage: Counts words in file.txt.
14. paste (Merge Lines from Multiple Files)
Example:
Output: Combines corresponding lines from file1.txt and file2.txt.
15. nl (Number Lines in a File)
Example:
Output: Numbers each line in file.txt.
16. pr (Format File for Printing)
Example:
Output: Adds a header “Title” to file.txt before printing.
17. split (Split a File into Smaller Files)
Example:
Usage: Splits file.txt into multiple smaller files with 50 lines each.
18. tee (Write Output to File and Terminal)
Example:
Usage: Displays the ls -l output on the screen and saves it in output.txt.
19. diff (Compare Two Files)
Example:
Output: Shows differences between file1.txt and file2.txt.
20. comm (Compare Sorted Files Line by Line)
Example:
Output: Displays common and unique lines from two sorted files.
Linux Text Processing with awk and sed – Cheat Sheet with Examples
awk and sed are powerful command-line tools used for text processing in Linux. This guide provides detailed examples of both commands for filtering, replacing, and manipulating text.
📌 sed (Stream Editor) – Text Substitution & Manipulation
sed is used for searching, replacing, deleting, and modifying text in a stream or file.
1️⃣ Replace a Word in a File
🔹 Example: Replace "error" with "warning" in log.txt.
2️⃣ Replace Word Only in Specific Line
🔹 Example: Replace "error" with "warning" only on line 3.
3️⃣ Delete a Specific Line
🔹 Deletes line 5 from file.txt.
4️⃣ Delete Lines Containing a Specific Word
🔹 Deletes all lines containing "error".
5️⃣ Print Only Matching Lines
🔹 Prints only the lines that contain “error”.
6️⃣ Replace Text in a File and Save Output
🔹 Modifies the file in place by replacing "old" with "new".
7️⃣ Remove Empty Lines
🔹 Deletes all empty lines.
8️⃣ Insert Text at the Beginning of Each Line
🔹 Adds "Prefix: " at the beginning of each line.
9️⃣ Insert a Line Before a Match
🔹 Inserts "New Line Here" before lines matching "pattern".
🔟 Append a Line After a Match
🔹 Appends "New Line Here" after lines matching "pattern".
📌 awk (Pattern Scanning and Processing Language)
awk is used for filtering, formatting, and processing text in structured data.
1️⃣ Print the Entire File
🔹 Prints all lines in file.txt.
2️⃣ Print a Specific Column
🔹 Prints the second column.
3️⃣ Print Multiple Columns
🔹 Prints first and third columns.
4️⃣ Print Lines Matching a Pattern
🔹 Prints only lines containing "error".
5️⃣ Print Line Numbers
🔹 Prints each line with a line number.
6️⃣ Print Lines with More than 5 Words
🔹 Prints lines with more than 5 fields (words).
7️⃣ Sum a Column of Numbers
🔹 Calculates the sum of the second column.
8️⃣ Find the Maximum Value in a Column
🔹 Finds the largest number in the second column.
9️⃣ Find Unique Values in a Column
🔹 Prints only unique values from the first column.
🔟 Format Output Like a Table
🔹 Aligns the first two columns into a table format.
📊 Combining awk and sed
You can combine both tools for powerful text processing.
🔹 Remove Empty Lines and Print Specific Columns
🔹 Removes empty lines and prints column 1 and 3.
🔹 Replace a Word and Print Specific Lines
🔹 Replaces "error" with "warning" and prints only matching lines.