This response is based on the information provided.  We suggest you discuss with your contracting team, program manager and/or legal department as appropriate. 

It is not mandatory to attend; the contracting officer needs to make a sound business judgment in deciding whether to attend a site visit.  Your representative must be well versed in contractual requirements, particularly the PWS.  In any event, should questions arise needing contracting officer interpretation, your rep needs to relay the issue(s) to you immediately after the site visit so you can quickly respond.