As a user of Hong Kong servers, mastering Linux shell text processing is a superpower that will skyrocket your productivity. The command line offers a rich toolset for slicing and dicing text data with surgical precision. In this article, we’ll dive into 15 must-know Linux shell text processing techniques that will help increase your productivity.

1. Grep: The Text Search Powerhouse

Grep is your go-to command for lightning-fast text search. Its basic syntax is:

grep "pattern" file.txt

But grep’s real power lies in its options. For example, to perform a case-insensitive search:

grep -i "pattern" file.txt

And to recursively search all files in a directory:

grep -r "pattern" directory/

2. Sed: The Stream Editor Extraordinaire

Sed is a potent stream editor that can perform complex text transformations. Its basic syntax for substitution is:

sed 's/old/new/g' file.txt

But sed can do much more, like deleting lines containing a pattern:

sed '/pattern/d' file.txt

Or inserting text after lines matching a pattern:

sed '/pattern/a New line' file.txt

3. Awk: The Text Processing Swiss Army Knife

Awk is a full-fledged text processing language packed into a command. It excels at handling structured text like CSV or TSV files. For example, to print the first and third columns of a CSV:

awk -F',' '{print $1, $3}' file.csv

Awk can also perform calculations and conditionals:

awk '{sum+=$1} END {print sum}' numbers.txt

4. Cut: The Column Cutter

Cut is handy for extracting specific columns from structured text. To cut the first and fourth columns of a CSV:

cut -d',' -f1,4 file.csv

5. Sort: The Line Sorting Machine

Sort is your friend when you need to sort lines of text. By default, it sorts alphabetically:

sort file.txt

But you can also sort numerically:

sort -n file.txt

Or sort by a specific column:

sort -k2 -t',' file.csv

6. Uniq: The Duplicate Destroyer

Uniq is essential for removing duplicate lines from a sorted file:

sort file.txt | uniq

7. Tr: The Character Transformer

Tr is perfect for translating or deleting characters. To convert lowercase to uppercase:

tr 'a-z' 'A-Z' < file.txt

Or to delete all digits:

tr -d '0-9' < file.txt

8. Paste: The File Combiner

Paste lets you combine multiple files side by side:

paste file1.txt file2.txt

9. Wc: The Word Counter

Wc counts lines, words, and characters in a file. To count lines:

wc -l file.txt

10. Tee: The Output Splitter

Tee splits command output, sending it both to a file and stdout:

command | tee output.txt

11. Find: The File Finder

Find is your ally for locating files based on various criteria. To find files by name:

find /path/to/directory -name "*.txt"

Or to find files modified within the last 7 days:

find /path/to/directory -mtime -7

12. Xargs: The Command Executor

Xargs allows you to build and execute commands from standard input. A common use case is pairing it with find:

find /path/to/directory -name "*.txt" | xargs grep "pattern"

13. Head: The File Header Viewer

Head displays the first part of a file. To view the first 10 lines:

head file.txt

Or specify the number of lines:

head -n 5 file.txt

14. Tail: The File Trailer Viewer

Tail shows the last part of a file. To view the last 10 lines:

tail file.txt

Or continuously monitor a file for changes:

tail -f log.txt

15. Diff: The File Comparator

Diff compares the contents of two files line by line:

diff file1.txt file2.txt

Armed with these 15 Linux shell text processing power tools, you’ll be ready to conquer any text manipulation challenge that comes your way.