I want to throw error dialog (and fail the installation) if certain file exists on the system. Is it possible? pkg = flat file pkg
It's possible to customize the installer in various ways to get whatever functionality is needed generally. Typically in your scenario either a Pre-Installation script could be used or a Package Installer Plugin. The options would normally be defined in the installer package's distribution.dist which essentially functions as the schema it will follow.
See productbuild in the man pages:
--scripts scripts-path - The contents of scripts-path is added to the product archive for use by system.run() commands commands in the distribution. This is valid only for product archives targeted to the OS X Installer application.
--plugins plugins-path - The contents of plugins-path is added to the product archive for use by the OS X Installer application's plugin mechanism. It will normally contain a InstallerSections.plist file, and one or more plugin bundles.