You need to configure it properly. Here is directions to do it with Windows firewall. It should be similar for your firewall.

Go to Start -> Control Panel -> Firewall
Pick 'Advanced Settings' on the left bar
Click 'Outbound Rules' on the left, then 'New Rule' on the right
Select 'Program', Next
Browse to the program's .exe, Next
Select 'Block the connection', Next
Leave all the boxes ticked, Next
Give the rule a name, Finish