I used chestnut for a few days, and ran into the same problem... I can't remember what the fix is exactly, but IIRC, it needs to create a link called modem in /dev that points to the actual modem device.
My problem was that when ran as user, it did not have permisions to create it.
So, as root, I created the link, and then ran chestnut dialer.
Try running it from a terminal and post the output... it's easier to see what the actual problem is.