How do I find and remove emojis in a text file?

I'm trying to remove all the emojis from a text file I'm parsing using mostly sed and some perl commands, and preferrably store them in a separate file but this isn't necessary.

Can I do this easily with bash or perl? Or should I use another language?

EDIT: Thank you to Cyrus and Barmar for pointing me in the right direction, towards this question. However, it doesn't tell me how to remove just the emojis from the text file. They use the bash line:

grep -P "[\x{1f300}-\x{1f5ff}\x{1f900}-\x{1f9ff}\x{1f600}-\x{1f64f}\x{1f680}-\x{1f6ff}\x{2600}-\x{26ff}\x{2700}-\x{27bf}\x{1f1e6}-\x{1f1ff}\x{1f191}-\x{1f251}\x{1f004}\x{1f0cf}\x{1f170}-\x{1f171}\x{1f17e}-\x{1f17f}\x{1f18e}\x{3030}\x{2b50}\x{2b55}\x{2934}-\x{2935}\x{2b05}-\x{2b07}\x{2b1b}-\x{2b1c}\x{3297}\x{3299}\x{303d}\x{00a9}\x{00ae}\x{2122}\x{23f3}\x{24c2}\x{23e9}-\x{23ef}\x{25b6}\x{23f8}-\x{23fa}]"  myflie.txt | more

which gets me all the lines containing an emoji.

grep -Pv will remove those lines from the input,

grep -Po will return just the emojis,

grep -Pov returns nothing.

Does anyone know how to remove those specific characters from the text?

Note: I am aware of this question, but my text file is not at all formatted. Emojis are mixed in with the rest of the text.

Solution

2020 UPDATE: Perl v5.32 uses Unicode 13 and supports several properties that deal with emoji. You can use the Emoji property:

Another update: This character class has some surprising matches, including the plain decimal digits, as @anon noted in his comment. See also Why do Unicode emoji property escapes match numbers?.

$ perl -le 'use open qw(:std :utf8); for(1..0xFFFF){ next unless chr() =~ /\p{Emoji}/; printf "%04x %s matches Emoji\n", $_, chr()}'
0023 # matches Emoji
002a * matches Emoji
0030 0 matches Emoji
0031 1 matches Emoji
0032 2 matches Emoji
0033 3 matches Emoji
0034 4 matches Emoji
0035 5 matches Emoji
0036 6 matches Emoji
0037 7 matches Emoji
0038 8 matches Emoji
0039 9 matches Emoji
00a9 © matches Emoji
00ae ® matches Emoji
... many more unsurprising results ...

This means that this program is a bit aggressive:

#!perl
use v5.32;
use utf8;
use open qw(:std :utf8);

while( <<>> ) {  # double diamond (from v5.26) 
    s/\p{Emoji}//g;
    print;
    }

However, Perl v5.18's regex set operations can fix this ((?[ ... ])). Subtract the ASCII range:

while( <DATA> ) {  # double diamond (from v5.26)
    s/(?[ \p{Emoji} - [\001 - \377] ])//g;
    print;
    }

As a one-liner, this turns into:

% perl -CS -pe 's/(?[ \p{Emoji} - [\001 - \377] ])//g' file1 file2 ...

You might want to knock out or include other characters, so you subtract or add those to the set.

Character classes for older Perls

In Perl, removing the emojis can be easy. At its core, this is very close to how you'd do it sed. Update the pattern and other details for your task:

#!perl
use utf8;
use open qw(:std :utf8);

my $pattern = "[\x{1f300}-\x{1f5ff}\x{1f900}-\x{1f9ff}\x{1f600}-\x{1f64f}\x{1f680}-\x{1f6ff}\x{2600}-\x{26ff}\x{2700}-\x{27bf}\x{1f1e6}-\x{1f1ff}\x{1f191}-\x{1f251}\x{1f004}\x{1f0cf}\x{1f170}-\x{1f171}\x{1f17e}-\x{1f17f}\x{1f18e}\x{3030}\x{2b50}\x{2b55}\x{2934}-\x{2935}\x{2b05}-\x{2b07}\x{2b1b}-\x{2b1c}\x{3297}\x{3299}\x{303d}\x{00a9}\x{00ae}\x{2122}\x{23f3}\x{24c2}\x{23e9}-\x{23ef}\x{25b6}\x{23f8}-\x{23fa}]";

while( <DATA> ) {  # use <> to read from command line
    s/$pattern//g;
    print;
    }

__DATA__
Emoji at end 😀
🗿 Emoji at beginning
Emoji 🙏 in middle

UTS #51 mentions an Emoji property, but it's not listed in perluniprop. Were there such a thing, you would simplify that removing anything with that property:

while( <DATA> ) {
    s/\p{Emoji}//g;
    print;
    }

There is the Emoticon property, but that doesn't cover your character class. I haven't looked to see if it would be the same as the Emoji property in UTS #51.

User-defined Unicode properties

You can make your own properties by defining a subroutine that begins is In or Is followed by the property name you choose. That subroutine returns a potentially multi-lined string where each line is either a single hex code number or two hex code numbers separated by horizontal whitespace. Any character in all of that is then part of your property.

Here's that same character class as a user-defined Unicode property. Note that I use the squiggly heredoc, mostly because I can write the program locally with leading space so I can paste directly into StackOverflow. The lines in IsEmoji cannot have leading space, though, but the indented heredoc takes care of that:

#!perl
use v5.26; # for indented heredoc
use utf8;
use open qw(:std :utf8);

while( <DATA> ) {  # use <> to read from command line
    s/\p{IsEmoji}//g;
    print;
    }

sub IsEmoji { <<~"HERE";
1f300 1f5ff
1f900 1f9ff
1f600 1f64f
1f680 1f6ff
2600 26ff
2700 27bf
1f1e6 1f1ff
1f191 1f251
1f004 1f0cf
1f170 1f171
1f17e 1f17f
1f18e
3030
2b50
2b55
2934 2935
2b05 2b07
2b1b 2b1c
3297
3299
303d
00a9
00ae
2122
23f3
24c2
23e9 23ef
25b6
23f8 23fa
HERE
}

__DATA__
Emoji at end 😀
🗿 Emoji at beginning
Emoji 🙏 in middle

You can put that in a module:

# IsEmoji.pm
sub IsMyEmoji { <<~"HERE";
1f300 1f5ff
...  # all that other stuff too
23f8 23fa
HERE
}

1;

Now you can use that in a one liner (the -I. adds the current directory to the module search path and the -M denotes a module to load):

$ perl -CS -I. -MIsEmoji -pe 's/\p{IsEmoji}//g' file1 file2

Beyond that, you're stuck with the long character class in your one-liner.