First I would not consider Vector to be a candidate for that, IMHO.
The approach of combining the ACLs from Windows to Linux
can be done, however you may need to add some things in the
kernel in order to do that.
Since you have asked. There are a couple approaches:
1.) Use Windows 2008 server for the entire Domain and setup
the other site accordingly. Then setup your permissions as
well. As far as file Locking I do not remember such a facility in
in Active Directory.
2.) Start in the direction you started and use a Linux server to provide
the File server and use a Document Management System for the
file locking. Then use a backup rsync server.
3.) Use Linux only to provide all the services.
Take your time in evaluating a solution. Install in stages so that all the kinks
can be worked out and alienate your users.
Maybe this will help with a good direction:http://www.linux.com/feature/114027