Powershell defaultparameterset not working. error

I am trying to use parameter set and use default parameter set. But the default parameter set doesn't seem to work for me. Any help is much appreciated. I can easily use validate set with default action but I want to know what I am doing wrong here.

Param([cmdletbinding(DefaultParametersetname="Directory")] 
      [Parameter(Mandatory=$false,ParameterSetName="File")]
      [switch]$file, 
      [Parameter(Mandatory=$false,ParameterSetName="Directory")]  
      [switch]$directory,

[Parameter(Mandatory=$false,ParameterSetName="File")] 
[Parameter(Mandatory=$false,ParameterSetName="Directory")] 
[string]$Source,
[Parameter(Mandatory=$true,ParameterSetName="File")]  
[Parameter(Mandatory=$true,ParameterSetName="Directory")] 
[string]$DestinationPath, 
[Parameter(Mandatory=$false,ParameterSetName="Directory")] 
[Parameter(Mandatory=$false,ParameterSetName="File")]
[array]$Servers

PS C:\> Test-Script -Source "c:\somedirectory" -DestinationPath "c:\someotherdirectory"

Test-Script : Parameter set cannot be resolved using the specified named parameters.
At line:1 char:1
+ Test-Script -Source "c:\somedirectory" -DestinationPath "c:\someotherdirectory"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Test-Script], ParameterBindingException
+ FullyQualifiedErrorId : AmbiguousParameterSet,Test-Script

The CmdletBinding() attribute needs to go outside the param block, immediately before the param keyword, otherwise it will simply be ignored:

[CmdletBinding(DefaultParametersetname="Directory")]
Param(
    [Parameter(Mandatory=$false,ParameterSetName="File")]
    [switch]$file, 

    [Parameter(Mandatory=$false,ParameterSetName="Directory")]
    [switch]$directory,

    [Parameter(Mandatory=$false,ParameterSetName="File")]
    [Parameter(Mandatory=$false,ParameterSetName="Directory")]
    [string]$Source,

    [Parameter(Mandatory=$true,ParameterSetName="File")]
    [Parameter(Mandatory=$true,ParameterSetName="Directory")]
    [string]$DestinationPath,

    [Parameter(Mandatory=$false,ParameterSetName="Directory")]
    [Parameter(Mandatory=$false,ParameterSetName="File")] 
    [array]$Servers
)