The answer to this question revolves entirely around the fact that this is a new program funded with RDT&E. When determining the correct type of appropriation to use for any task, the reason for the effort will directly lead to the answer. The Background of this question clearly indicates that the AoA is for a new program funded with RDT&E.
FMR, Volume 2A, Chapter 1, paragraph 010213, section B.1.a states, “RDT&E will finance research, development, test and evaluation efforts performed by contractors and government installations, including procurement of end items, weapons, equipment, components, materials and services required for development of equipment, material, or computer application software; its Development Test and Evaluation (DT&E) and its Operational Test and Evaluation (OT&E) as provided for in paragraph C.5.” Traditionally, other than for purely theoretical and scientific research projects, the use of RDT&E funding begins following the Materiel Development Decision as defined in the DoD Instruction (DoDI) 5000.02.
DoDI 5000.02 section 5.d.(1)(a) says, “The Materiel Development Decision is based on a validated initial requirements document (an ICD or equivalent requirements document) and the completion of the Analysis of Alternative (AoA) Study Guidance and the AoA Study Plan. This decision directs execution of the AoA, and authorizes the DoD Component to conduct the Materiel Solution Analysis Phase.” Therefore, if the studies referenced in this question are being used to assess operational threats versus capabilities gaps or inform requirements formulation, O&M would be the correct appropriation. However, if the studies referenced in this question are in support of an AoA within the Materiel Solution Analysis Phase of the Defense Acquisition System (DAS), they would be most appropriately funded with RDT&E.
Conclusion: The most appropriate funding to be used for this new program's AoA would be RDT&E. However, I strongly suggest you look at the Budget Exhibits to understand which Appropriation the AoA funds were requested. The Budget Exhibits should show if the AoA was requested in the RDT&E Appropriation.
Note: FMR, Volume 2A, Chapter 1, paragraph 010213, section B states, “When after consideration of the following criteria, there is doubt as to the proper assignment of costs between appropriations, the issue should be resolved in favor of using RDT&E funding.”
Suggestions: Read DoD Financial Management Regulation 7000.14-R, Volume 2A, Chapter 1, paying particular attention to paragraph 010213. In addition, it is most strongly recommended that you contact your local comptroller organization, and legal counsel for more information and their policy interpretation of this issue.