Every once in a while a programmer finds himself in need of a tool that allows him to edit very large text files. By large, I mean several gigabytes. For DBAs it is common, especially if you’re using MySQL dumps a lot. What do you do if you’re doing this on Windows?
If you’re using Notepad++ or any other Scintilla derivatives, you’re out of luck - those editors are not cut out for this kind of work. Using Visual Studio also won’t work. There are some partial solutions just for viewing, like LTFViewer – but it cannot handle large files without line breaks, something common in MySQL dumps. So what do you do?
The answer is simple and somewhat unexpected – use Vim for Windows. It opens a file of any size effortlessly and almost instantly – then you can browse through the file like it was a small script - it just works.
Few things to notice:
For files with very long lines, like dumps, consider using “set nowrap”.
Another thing is the fact that Vim keeps a temporary copy of the file for backup when saving an edited file, so not only it takes a while to save since it writes the entire buffer to the disk, but you also need to delete the temp file afterwards. To disable this behavior, use “set nobackup”.
Just one more reason why vi (and vim in specific) is still my favorite editor of all time. I spend at least a few hours per day in vim and I wouldn't trade it for anything in the world.
ReplyDeleteI do like notepad++ for quick editing on windows and for short files and scripts though. I always have two or three notepad++ open for jotting down notes, and for using as a temporary clipboard repository.
My MyQuery tool is currently using SCintilla. And I agree that what you say is an issue, and something that I wanted to adress in MyQuery. The Scintilla base to an extent limites what MyQuery can do, bt still, I do my own interpretation of scripts, so I have a few things I want to do eventually, beyond editing large files in Scintilla.
ReplyDeleteOne idea I have is to be able to run a script, statement by statement, bringing up each statement in Scintilla. Another is running statement by statement til we have an error, allow you to fix that that single statment using the Scintilla based editor, and then continue from there.
@swanhart: Well, back then, they built things to last :P
ReplyDelete@Karlsson: Scintilla is a great editor, but not meant for such work. I would assume it's because of the syntax parsing it does, etc.
Other editors like SQLyog are using their built-in editors, btw. SQLyog manages to open large files, but doesn't do that as gracefully as Vim though.
No Vim isn't good enough for this kind of work either. Maybe better than some, but still not ideal.
ReplyDeleteIf you just want to view the file ( often the case when its a large text file) just cat the file into less.
Find the spot you want to edit, and write a quick script to make the edit.
Or better yet, you could write a striped down editor that is specifically tailored to this problem, like I did :)
@water outbreaks
ReplyDeleteI agree it's not ideal (mostly since it rewrites the file), but it's the best available for Windows. Of course I would use "less" or "cat" if it was Linux...
I have a friend that says "Windows is for little girls", maybe he's right this time :)
What's that editor? Is it downloadable someplace?
On the other hand, this means using vi/vim, which in my book is marginally preferable to clawing your own eyes out. But what really amazes me is that the GNU head tool has no "skip n lines first" feature.
ReplyDeleteTo skip say 500 lines down, try typing 500 followed by the down arrow key.
DeleteIt isn't free (~$19.50), but I've used j-write to effortlessly open and edit files that were 7+GB in size.
ReplyDeleteAnother (open source) editor you might want to throw a very large file at is metapad.
Good discussion here:
ReplyDeletehttp://stackoverflow.com/questions/102829/best-free-text-editor-supporting-more-than-4gb-files
GigaEdit is also good and fast free text editor for big text file, it´s portable, but without undo function
ReplyDeleteNotepad++? It's the best free text editor to me, but... I tried to read really huge files (about 2GB or more) on it and I found "File is to big to be opened by Notepad++" error :(
ReplyDelete