There is no guidance in the FAR or DFARS relating to contractor return of CAC cards. There is one deviation, CD 2013-O0017, for the demobilization of personnel in Afghanistan that states:

" The Contractor shall ensure that all employees, including all subcontractor
employees at all tiers, return installation and/or access badges to the local Access
Control Badging Office for de-activation and destruction according to the approved
demobilization plan. The Contractor shall submit a Badge Termination Report to
ensure each record is flagged and the badge is revoked. If an employee's badge is
not returned, the Contractor shall submit a Lost, Stolen or Unrecovered Badgeheeport to the appropriate Access Control Badging Office. Contractor employees in
possession of a Common Access Card (CAC) shall be responsible for turning in the
CAC upon re-deployment through a CONUS Replacement Center in the United
States. Failure to comply with these requirements may result in delay of final
payment."

I would recommend putting a local clause in all soliciations and contracts stating within a certain time period (i.e. 72 hours)  after a contractor leaves, their Common Access Card must be returned to the government.  Creating a consequence such as delaying payment should ensure compliance.