Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 2 down vote favorite

I need to split a file. I usually use split but this time I need to have the splitted file of the same encoding type of the original. I have the original:

eianni@ianni-desktop:~/Desktop$ file FCAna.txt 
FCAna.txt: ISO-8859 text, with CRLF line terminators

while new ones are:

eianni@ianni-desktop:~/Desktop$ file xaa
xaa: ISO-8859 text, with CRLF line terminators
eianni@ianni-desktop:~/Desktop$ file xab
xab: Non-ISO extended-ASCII text, with CRLF line terminators

the second one is not ok. How can I solve this? The command executed is

split --lines=1588793 FCAna.txt
share|improve this question

2 Answers

up vote 4 down vote accepted

I think this could be down to the way file works. Reading from the manpage:

ASCII, ISO-8859-x, non-ISO 8-bit extended-ASCII character sets (such as those used on Macintosh and IBM PC systems), UTF-8-encoded Unicode, UTF-16-encoded Unicode, and EBCDIC character sets can be distinguished by the different ranges and sequences of bytes that constitute printable text in each set.

My interpretation of this statement is that file's ability to determine the encoding used is based off whether there are certain characters available in the text file that make it obvious what the encoding is. So for UTF the size of the bytes or existence of a BOM could be used. Your original text file may have used characters that could only be encoded in an extended ASCII character set (a pound symbol (£) maybe?) and so file was determining it was an ISO 8859 file. But now that the file is split, that symbol only appears in the first file and not the second. You should be able to test this hypothesis by searching in the text for "extended" characters and splitting at different points.

As a test I did the following:

[root@blah ~]# echo "this is a test of text encoding" > test_encoding.txt
[root@blah ~]# file test_encoding.txt
test_encoding.txt: ASCII text
[root@blah ~]# echo "£" >> test_encoding.txt
[root@blah ~]# file test_encoding.txt
test_encoding.txt: ISO-8859 text
[root@blah ~]#

Is there a reason you needed the file encodings reported by file to match?

share|improve this answer
Basically a client gave me this file and there's no single editor that is able to understand what the hell it is. E.g: Gedit ask me what encoding should use, Notepad++ chooses ASCII not extended and so on... – dierre Mar 13 '12 at 21:35
Hmmm, this is generally a problem with pure text files if you don't know the encoding (or you can't guess). Generally I would tell your text editor of choice to use iso-8859-1 (latin1) by default as it covers most western characters UNLESS you know the language they wrote it in. UTF-8 should get picked up automatically due to its encoding. I use emacs and just load files up and switch coding schemes with M-x set-buffer-file-coding-system until files look right. Here's a quick list terena.org/activities/multiling/ml-docs/iso-8859.html – webtoe Mar 13 '12 at 23:43
up vote 0 down vote

This is a line termination issue. Maybe tell split to break by line with the -l option?

share|improve this answer
split --lines=1588793 FCAna.txt this was my command. – dierre Mar 13 '12 at 11:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.