It is possible to installed the vCenter 5.1 without AD but from KB 2021202
If the system that you use for your vCenter Server installation belongs to a workgroup rather than a domain, not all functionality is available to vCenter Server. If assigned to a workgroup, the vCenter Server system is not able to discover all domains and systems available on the network when using some features. To determine whether the system belongs to a workgroup or a domain, right-click My Computer and click Properties, then click the Computer Name tab. TheComputer Name tab displays either a Workgroup label or a Domain label.
Before Installing vCenter 5.1 you need to install SSO and Inventory service and it is also possible to install SSO on a standalone system but this identity source is avilable only in SSO basic deploymenet option.
Suggest to read the below:
Installing vCenter Server 5.1 best practices
Comparing behavior of vCenter Single Sign-On with earlier versions of vCenter Server (2032135)