stringrustunicodedevanagarigrapheme

How to split Devanagari bi-tri and tetra conjunct consonants as a whole from a string?


I am new to Rust and I was trying to split Devanagari (vowels and) bi-tri and tetra conjuncts consonants as whole while keeping the vowel sign and virama. and later map them with other Indic script. I first tried using Rust's chars() which didn't work. Then I came across grapheme clusters. I have been googling and searching SO about Unicode and UTF-8, grapheme clusters, and complex scripts.

I have used grapheme clusters in my current code, but it does not give me the desired output. I understand that this method may not work for complex scripts like Devanagari or other Indic scripts.

How can I achieve the desired output? I have another code where I attempted to build a simple cluster using an answer from Stack Overflow, converting it from Python to Rust, but I have not had any luck yet. It's been 2 weeks and I have been stuck on this problem.

Here's the Devanagari Script and Conjucts wiki:

Devanagari Script: https://en.wikipedia.org/wiki/Devanagari
Devanagari Conjucts: https://en.wikipedia.org/wiki/Devanagari_conjuncts

Here's what I wrote to split:

extern crate unicode_segmentation;
use unicode_segmentation::UnicodeSegmentation;


fn main() {
    
    let hs = "हिन्दी मुख्यमंत्री हिमंत";
    let hsi = hs.graphemes(true).collect::<Vec<&str>>();
    for i in hsi { 
        print!("{}  ", i); // double space eye comfort
    }
} 

Current output:
हि न् दी मु ख् य मं त् री हि मं त

Desired ouput:
हि न्दी मु ख्य मं त्री हि मं त

My another try:

I also tried to create a simple grapheme cluster following this SO answer https://stackoverflow.com/a/6806203/2724286

fn split_conjuncts(text: &str) -> Vec<String> {
    let mut result = vec![];
    let mut temp = String::new();

    for c in text.chars() {
        if (c as u32) >= 0x0300 && (c as u32) <= 0x036F {
            temp.push(c);
        } else {
            temp.push(c);
            if !temp.is_empty() {
                result.push(temp.clone());
                temp.clear();
            }
        }
    }
    if !temp.is_empty() {
        result.push(temp);
    }
    result
}

fn main() {
    let text = "संस्कृतम्";
    let split_tokens = split_conjuncts(text);
    println!("{:?}", split_tokens);

}

Output:
["स", "\u{902}", "स", "\u{94d}", "क", "\u{943}", "त", "म", "\u{94d}"]


So, how can I get the desired output?

Desired ouput:
हि न्दी मु ख्य मं त्री हि मं त

I also checked other SO answers (links below) dealing issues with Unicode, grpahemes, UTF-8, but no luck yet.

Combined diacritics do not normalize with unicodedata.normalize (PYTHON)

what-is-the-difference-between-combining-characters-and-grapheme-extenders

extended-grapheme-clusters-stop-combining


Solution

  • Thanks for the above answers. Special thanks to those who helped in the comment section, it helped a lot.

    I forgot to post the answer I did that time. Here's the rust implementation below. I also did a nextjs implementation too. My idea was to develop it for other similar languages. But got busy with other projects. :(

    use unicode_segmentation::UnicodeSegmentation;
    
    // Define a struct that holds a grapheme iterator
    struct DevanagariSplitter<'a> {
        graphemes: std::iter::Peekable<unicode_segmentation::Graphemes<'a>>,
    }
    
    // Implement Iterator trait for DevanagariSplitter
    impl<'a> Iterator for DevanagariSplitter<'a> {
        type Item = String;
    
        fn next(&mut self) -> Option<Self::Item> {
            // Get the next grapheme from the iterator
            let mut akshara = match self.graphemes.next() {
                Some(g) => g.to_string(),
                None => return None,
            };
    
            // Check if the grapheme ends with a virama
            if akshara.ends_with('\u{094D}') {
                // Peek at the next grapheme and see if it starts with a letter
                if let Some(next) = self.graphemes.peek() {
                    if next.starts_with(|c: char| c.is_alphabetic()) {
                        // Append the next grapheme to the current one
                        akshara.push_str(self.graphemes.next().unwrap());
                    }
                }
            }
    
            // Return the akshara as an option
            Some(akshara)
        }
    }
    
    // Define a function that takes a string and returns an DevanagariSplitter
    fn aksharas(s: &str) -> DevanagariSplitter {
        // Use UnicodeSegmentation to split the string into graphemes
        let graphemes = s.graphemes(true).peekable();
        // Create and return an DevanagariSplitter from the graphemes
        DevanagariSplitter { graphemes }
    }
    
    fn main() {
        // Define an input string in devanagari script
        let input = "हिन्दी मुख्यमंत्री हिमंत";
    
        // Print each akshara separated by spaces using aksharas function
        for akshara in aksharas(input) {
            print!("{} ", akshara);
        }
    }
    
     
    // The output of this code is:
    
    // "हि न्दी  मु ख्य मं त्री  हि मं त"
    
    

    In case anyone interested to check the next.js version, here's the link https://vinno.vercel.app/ (I don't know if it violets the SO TOS. If it does, please let me know, I will remove the link)