I am new to Rust and I was trying to split Devanagari (vowels and) bi-tri and tetra conjuncts consonants as whole while keeping the vowel sign and virama. and later map them with other Indic script. I first tried using Rust's chars()
which didn't work. Then I came across grapheme clusters. I have been googling and searching SO about Unicode and UTF-8, grapheme clusters, and complex scripts.
I have used grapheme clusters in my current code, but it does not give me the desired output. I understand that this method may not work for complex scripts like Devanagari or other Indic scripts.
How can I achieve the desired output? I have another code where I attempted to build a simple cluster using an answer from Stack Overflow, converting it from Python to Rust, but I have not had any luck yet. It's been 2 weeks and I have been stuck on this problem.
Here's the Devanagari Script and Conjucts wiki:
Devanagari Script: https://en.wikipedia.org/wiki/Devanagari
Devanagari Conjucts: https://en.wikipedia.org/wiki/Devanagari_conjuncts
Here's what I wrote to split:
extern crate unicode_segmentation;
use unicode_segmentation::UnicodeSegmentation;
fn main() {
let hs = "हिन्दी मुख्यमंत्री हिमंत";
let hsi = hs.graphemes(true).collect::<Vec<&str>>();
for i in hsi {
print!("{} ", i); // double space eye comfort
}
}
Current output:
हि न् दी मु ख् य मं त् री हि मं त
Desired ouput:
हि न्दी मु ख्य मं त्री हि मं त
My another try:
I also tried to create a simple grapheme cluster following this SO answer https://stackoverflow.com/a/6806203/2724286
fn split_conjuncts(text: &str) -> Vec<String> {
let mut result = vec![];
let mut temp = String::new();
for c in text.chars() {
if (c as u32) >= 0x0300 && (c as u32) <= 0x036F {
temp.push(c);
} else {
temp.push(c);
if !temp.is_empty() {
result.push(temp.clone());
temp.clear();
}
}
}
if !temp.is_empty() {
result.push(temp);
}
result
}
fn main() {
let text = "संस्कृतम्";
let split_tokens = split_conjuncts(text);
println!("{:?}", split_tokens);
}
Output:
["स", "\u{902}", "स", "\u{94d}", "क", "\u{943}", "त", "म", "\u{94d}"]
Desired ouput:
हि न्दी मु ख्य मं त्री हि मं त
I also checked other SO answers (links below) dealing issues with Unicode, grpahemes, UTF-8, but no luck yet.
Combined diacritics do not normalize with unicodedata.normalize (PYTHON)
what-is-the-difference-between-combining-characters-and-grapheme-extenders
Thanks for the above answers. Special thanks to those who helped in the comment section, it helped a lot.
I forgot to post the answer I did that time. Here's the rust implementation below. I also did a nextjs implementation too. My idea was to develop it for other similar languages. But got busy with other projects. :(
use unicode_segmentation::UnicodeSegmentation;
// Define a struct that holds a grapheme iterator
struct DevanagariSplitter<'a> {
graphemes: std::iter::Peekable<unicode_segmentation::Graphemes<'a>>,
}
// Implement Iterator trait for DevanagariSplitter
impl<'a> Iterator for DevanagariSplitter<'a> {
type Item = String;
fn next(&mut self) -> Option<Self::Item> {
// Get the next grapheme from the iterator
let mut akshara = match self.graphemes.next() {
Some(g) => g.to_string(),
None => return None,
};
// Check if the grapheme ends with a virama
if akshara.ends_with('\u{094D}') {
// Peek at the next grapheme and see if it starts with a letter
if let Some(next) = self.graphemes.peek() {
if next.starts_with(|c: char| c.is_alphabetic()) {
// Append the next grapheme to the current one
akshara.push_str(self.graphemes.next().unwrap());
}
}
}
// Return the akshara as an option
Some(akshara)
}
}
// Define a function that takes a string and returns an DevanagariSplitter
fn aksharas(s: &str) -> DevanagariSplitter {
// Use UnicodeSegmentation to split the string into graphemes
let graphemes = s.graphemes(true).peekable();
// Create and return an DevanagariSplitter from the graphemes
DevanagariSplitter { graphemes }
}
fn main() {
// Define an input string in devanagari script
let input = "हिन्दी मुख्यमंत्री हिमंत";
// Print each akshara separated by spaces using aksharas function
for akshara in aksharas(input) {
print!("{} ", akshara);
}
}
// The output of this code is:
// "हि न्दी मु ख्य मं त्री हि मं त"
In case anyone interested to check the next.js version, here's the link https://vinno.vercel.app/ (I don't know if it violets the SO TOS. If it does, please let me know, I will remove the link)