This has been one of the biggest obstacles in teaching new people ColdFusion.
When to use # is ambiguous at best. Since using them doesn't often create a problem it seems that most people gravitate to using them too much.
So, what are the basic rules?
I think it may be easier to say where NOT to use #. The only place is in cfif statements, and cfset statements where you are not using a variable to build a string in quotes. You would need to use the # sign in almost all other cases.
Example of where you are not going to use it:
<cfset value1 = 5>
<cfset value2 = value1/>
<cfif value1 EQ value2>
Yay!!!
</cfif>
<cfset value2 = "Four plus one is " & value1/>
Examples of where you will use #:
in a cfset where the variable is surrounded by quotes
<cfset value1 = 5>
<cfset value2 = "Four plus one is #value1#"/>
the bodies of cfoutput, cfmail, and cffunction (output="yes") tags
<cfoutput>#value2#</cfoutput>
<cfmail to="e@example.com" from="e@example.com" subject="x">#value2#</cfmail>
<cffunction name="func" output="yes">#value2#</cffunction>
in an attribute value of any coldfusion tag
<cfset dsn = "myDB"/>
<cfquery name="qryUsers" datasource="#dsn#">
<cfset value1 = 5>
<cfset value2 = 10/>
<cfloop from="#value1#" to="#value2#" index="i">
<cfqueryparam value="#value1#" cfsqltype="cf_sql_integer"/>
EDIT:
One oddball little thing I just noticed that seems inconsistent is conditional loops allow the variable name to be used with and without # signs.
<cfset value1 = 5>
<cfloop condition = "value1 LTE 10">
<cfoutput>#value1#</cfoutput><br>
<cfset value1 += 1>
</cfloop>
<cfset value1 = 5>
<cfloop condition = "#value1# LTE 10">
<cfoutput>#value1#</cfoutput><br>
<cfset value1 += 1>
</cfloop>