How to remove/delete CTRL-M (^M) characters from text files in Linux and UNIX systems
In this mini post, I’ll show you some methods/options to remove CTRL-M (^M) blue carriage return characters from a file in Linux and UNIX systems. Here’s you’ll find the exact answer for the following questions:
How to convert plain text files in DOS/MAC format to UNIX format?
Newly created SSL Certificates not working properly!
Certs invalid or not properly configured, agents unable to use!
Application reporting obscure syntax errors and other unanticipated, unforeseen errors!
Viewing the certificate files in Linux shows ^M characters appended to every line!
The file in question was created in Windows and then copied over to Linux!
^M is the keyboard equivalent to \r or CTRL-v + CTRL-m in vim!
See, There are many issues you may face because your text file have ^M appended to every line.
Before we start our solution methods, I’ll display the content of the file I’ll work with, it’s called “Mimastech.com” and it’s created by a windows program. I need to parse this text file in Linux and remove the duplicated lines, to display the content of the file, I used this command:
$ cat Mimastech.comanaproxy.comanaproxy.comwww.amazon.comwww.amazon.comyahoo.comyahoo.comamazon.comamazon.commimastech.commimastech.com
As you see, each line is duplicated once, So I used uniq command to remove the duplication in this file using the following command:
$ cat Mimastech.com |uniqanaproxy.comanaproxy.comwww.amazon.comwww.amazon.comyahoo.comyahoo.comamazon.comamazon.commimastech.commimastech.com
Wired, uniq didn’t remove the duplicated lines, but why?. I opened this file using vim editor and discovered that ever duplicated line has CTRL-M (^M) blue carriage return characters appended to it’s end, check the following output:
$ vim Mimastech.comanaproxy.comanaproxy.com^Mwww.amazon.comwww.amazon.com^Myahoo.comyahoo.com^Mamazon.comamazon.com^Mmimastech.commimastech.com^M
So, the line aren’t duplicated and uniq didn’t find any duplication in this file. So I wrote this post with some methods that fix this issue for further file parsing.
Option 1: Using dos2unix Command
I choose this command to be my first option because it’s simple and not need any options/regex/pattern with it. Unfortunately, this command not installed by default on most Linux/UNIX OSes. So, first install it.
- To install it on CentOS/RHEL/Fedora “rpm Linux family”, use the following command:
# yum -y install dos2unix- To install it on Debian/Ubuntu “Debian Linux family”, use the following commands:
$ sudo apt-get update$ sudo apt-get -y install dos2unix
Sure, if you a UNIX user, you’ll find the way to install dos2unix command on your UNIX OS.
Now, let’s proceed with our example, To remove the ^M using dos2unix command use this command:
$ dos2unix Mimastech.comdos2unix: converting file Mimastech.com to Unix format ...
Very simple, but we replaced the original file, to use dos2unix command and keep the original file use it with “-n” option. Now, I’ll check uniq output:
$ cat Mimastech.com |uniqanaproxy.comwww.amazon.comyahoo.comamazon.commimastech.com
Hooray, dos2unix worked correctly on the file.
Option 2: Using tr Command
This is our second option, it’s a very simple command and exists by default on all Linux/UNIX systems. To remove ^M from the text file Just run the following command:
$ tr -d '\r' < Mimastech.com > Mimastech.com-trWe saved the output file with name “Mimastech.com-tr”. Now, I’ll check uniq output:
$ cat Mimastech.com-tr | uniqanaproxy.comwww.amazon.comyahoo.comamazon.commimastech.com
Hooray, tr worked correctly on the file.
Option 3: Using sed Command
We can use sed command in two way as follow:
The first and easiest way to use sed syntax to remove carriage return in Unix or Linux:
$ sed -i 's/\r$//g' Mimastech.comIt’s output replaced the original file, Now, I’ll check uniq output:
$ cat Mimastech.com |uniqanaproxy.comwww.amazon.comyahoo.comamazon.commimastech.com
Hooray, sed worked correctly on the file.
The second way to use sed as follows (to get ^M type CTRL+V followed by CTRL+M i.e. don’t just type the carat symbol and a capital M. It will not work):
$ sed -i 's/^M//g' Mimastech.comIt’s output replaced the original file, Now, I’ll check uniq output:
$ cat Mimastech.com |uniqanaproxy.comwww.amazon.comyahoo.comamazon.commimastech.com
Hooray, sed worked correctly on the file.
Option 4: Using vi/vim Command
We can use vi/vim command to remove ^M character, Inside vi/vim [in ESC mode] type: :%s/^M//g
Note: To enter ^M, type CTRL-V + M. That is, hold down the CTRL key then press V and M in succession.as follow:
$ vim Mimastech.comanaproxy.comanaproxy.com^Mwww.amazon.comwww.amazon.com^Myahoo.comyahoo.com^Mamazon.comamazon.com^Mmimastech.commimastech.com^M~~~~~:%s/^M//g
Once you press the enter key, your file will be updated like this one:
anaproxy.comanaproxy.comwww.amazon.comwww.amazon.comyahoo.comyahoo.comamazon.comamazon.commimastech.commimastech.com~~~~~5 substitutions on 5 lines
Save and exit. Now, I’ll check uniq output:
$ cat Mimastech.com |uniqanaproxy.comwww.amazon.comyahoo.comamazon.commimastech.com
Hooray, vi/vim worked correctly on the file.
If You Appreciate What We Do Here On Mimastech, You Should Consider:
- Stay Connected to: Facebook | Twitter | Google+
- Support us via PayPal Donation
- Subscribe to our email newsletters.
- Tell other sysadmins / friends about Us - Share and Like our posts and services
We are thankful for your never ending support.
