Some files in Linux can get large. I mean really, really large. Certain .sql files for example, can easily span multiple GB. Even log files if left untended can start to eat up gigabytes of space. When files get that big, regular operations start to take longer and longer, and your usual commands don't work very well. For example, the "mail" file in my unused test server is itself 540 MB and here's how long it takes to view it via a regular editor like vi:
I closed vi immediately after loading, and that's 6 seconds just to open a file that's half a GB! Imagine the pain if you had to repeatedly view files several GB long? And god help you if you want to edit them. In this tutorial, I'm going to show you a few ways to view and edit large files as efficiently as possible. Often, there are no shortcuts other than to wait it out. But here's the best you can do.
Using "less" to View Files instead of "vi" or "vim"
We all know that "vi" is a fantastic editor. It's simple, and gets the job done. But as seen above, it can take ages to open large files. But why? The reason is that vi loads the entire file into memory before displaying it on screen. So you can already see the problem with large files. Not only does it take a lot of time, it's absolutely wasteful. Imagine having several GB of memory locked up by vi!
The alternative is to use a command called "less" instead. Unlike vi, it doesn't load the entire file into memory at once, but takes it in easily manageable chunks. It doesn't have the editing features of vi, but it makes it easy to open large files for viewing almost instantly. To open the same mail file (root-20170601) I opened earlier using "less, I just type:
Navigation is pretty simple.
- Press "f" to scroll to the next screen;
- Press "b" to scroll to the previous screen;
- Go forward and back one line at a time by using "j" and "k";
- Search for patterns by starting with "/" (just like vi);
- Go to a specific line number by typing the line number, followed by "g";
- Press "q" to quit.
Using less, here is the execution time for the same file that vi took 6 seconds to open:
That's just half a second - and it includes the time it took for me to press "q" to quit. So that makes it almost immediate! For viewing large files, less is definitely more!
Making Substitutions in Large Files
If viewing large files is problematic, editing them is even more so! Let's say you have a configuration file and you want to change a certain flag. To make the process as efficient as possible, the first step is finding out which line number you want to make the change in. We can display line numbers in less using the "-N" parameter like this:
less -N root-20170601
So let's say we're at line number 101 and we need to change the text "Successful Login" to "NOT Successful Login" as shown here:
Since we know the line number, we don't need to worry about the pattern finder changing the wrong occurrence. We quit less by pressing "q", and use the "sed" command to make the change:
sed -i '101s/Successful/NOT Successful/' root-20170601
The "-i" flag edits the file "in place" - meaning no new file is created. Note how I specify the line number 101. This command might take a while to execute since sed is creating a new file which will understandably be pretty large. Unfortunately, there's no shortcut around this. There's no way to make quick text changes to the middle of large files.
Once the command has executed, we can use less to view the file and go to line number 101 by typing "101g". As you can see below, the change has been made:
We can also use sed to quickly display a specific line number (in this case 101), like this:
sed '101q;d' root-20170601
This way, you don't need to open a text file and scroll or search.
Using "less" and "sed", can make your life a lot easier if you're dealing with large Linux files! Vi is fantastic for most situations, but sometimes you just need to find a more efficient way.