I am working on a data migration where on the old system the users were allowed to enter their interests in a large text-field with no formatting instructions followed at all. As a result some wrote in bio format and others wrote in comma-separated list format. There are a few other formats, but these are the primary ones.
Now I know how to identify a comma-separated list (CSL). That is easy enough. But how about determining if a string is a CSL (maybe a short one with two terms or phrases) or just a paragraph someone wrote that contains a comma?
One thought that I have is to automatically ignore strings that contain punctuation and strings that don't contain commas. However, I am concerned that this won't be enough or will leave much to be desired. So I would like to query the community to see what you guys think. In the mean time I will try out my idea.
UPDATE: Ok guys, I have my algorithm. Here it is below...
MY CODE:
//Process our interests text field and get the list of interests
function process_interests($interests)
{
$interest_list = array();
if ( preg_match('/(\.)/', $interests) 0 && $word_cnt > 0)
$ratio = $delimiter_cnt / $word_cnt;
//If delimiter is found with the right ratio then we can go forward with this.
//Should not be any more the 5 words per delimiter (ratio = delimiter / words ... this must be at least 0.2)
if (!empty($delimiter) && $ratio > 0 && $ratio >= 0.2)
{
//Check for label with colon after it
$interests = remove_colon($interests);
//Now we make our array
$interests = explode($delimiter, $interests);
foreach ($interests AS $val)
{
$val = humanize($val);
if (!empty($val))
$interest_list[] = $val;
}
}
}
return $interest_list;
}
//Cleans up strings a bit
function humanize($str)
{
if (empty($str))
return ''; //Lets not waste processing power on empty strings
$str = remove_colon($str); //We do this one more time for inline labels too.
$str = trim($str); //Remove unused bits
$str = ltrim($str, ' -'); //Remove leading dashes
$str = str_replace(' ', ' ', $str); //Remove double spaces, replace with single spaces
$str = str_replace(array(".", "(", ")", "\t"), '', $str); //Replace some unwanted junk
if ( strtolower( substr($str, 0, 3) ) == 'and')
$str = substr($str, 3); //Remove leading "and" from term
$str = ucwords(preg_replace('/[_]+/', ' ', strtolower(trim($str))));
return $str;
}
//Check for label with colon after it and remove the label
function remove_colon($str)
{
//Check for label with colon after it
if (strstr($str, ':'))
{
$str = explode(':', $str); //If we find it we must remove it
unset($str[0]); //To remove it we just explode it and take everything to the right of it.
$str = trim(implode(':', $str)); //Sometimes colons are still used elsewhere, I am going to allow this
}
return $str;
}
Thank you for all your help and suggestions!
You could, in addition to the filtering you mentioned, create a ratio of number of commas to string length. In CSLs, this ratio will tend to be high, in paragraphs low. You could set some kind of a threshold, and choose based on whether or not the entry has a high enough ratio. Ones with ratios close to the threshold could be marked as prone to error, and could then be check by a moderator.