The Kessel Run Experimentation Lab is not your typical DoD program - it's not located on a military installation, you won't see anyone wearing a suit and it's driven by an almost frenetic need to innovate. It's leaders wouldn't have it any other way.
Although the U.S. Air Force program is only a few years old, it is already shaking up how Service does things like create software and purchase information technology capabilities. It's non-standard approach is designed to do two things, speed up the acquisition process and turn the Air Force into a software company that happens to fly planes.
The imperative for change and the need to innovate is palpable. "Traditional acquisition and software development can't keep up with the rate of change or challenges from our enemies," said deputy director U.S. Air Force Lt. Col. Jeremiah Sanders during
an interview with the Defense Acquisition University. "We just can't keep up."
To overcome these challenges, U.S. Air Force Brig. Gen. Kevin Kennedy mentioned in
an interview with FedScoop that brings three skills into the conversation, "...an operator in that conversation who understands how to do software development, understands that it has some level of efficiency. I need an acquirer, someone from the acquisition community who understands how do we field these types of capabilities but also has some proficiency in it. And then I need a coder. That’s the ninja person who really knows how to do it quickly and can leverage those talents.”
ribbon cutting ceremony for the lab's new facility, Maj. Gen. Sarah Zabel, the Air Force’s director of information technology acquisition process development, called attention to just how different this organization is. “It’s one thing to say you’re going to do business differently, but look around and you can see that these airmen are learning. They’re building actual products, and they’re writing the book on how to be combat engineers for the information age,” she said.
Using this approach, Kessel Run has already been able to reduce the labor required to plan its missions and saves $750,000 to $1 million weekly.
Learn more in this video interview with Lt. Col. Jeremiah Sanders.
Video: The Agile Imperative with Lt. Col. Jeremiah SandersArticle: Kessel Run Lab Hits HyperdriveArticle: Air Force's New Software Lab in Boston Aims for High SpeedArticle: Air Force's Kessel Run has admirers elsewhere in the militaryKeywords: best practices, lessons learned
For more information about this story, or to submit your own Powerful Example - send an email to the DAU Powerful Examples Team at
Col. Donald Wols, J4, Director of Logistics at the United States Special Operations Command and Major Chris Baldwin of The J4 Logistics Integration Division share their success in rapidly developing a new tool for logistics and materiel solutions that will super-enable operators to support the "pointy end of the spear" in global Special Forces operations. Watch the short video or the podcast below to hear their story of how they developed truly innovative logisitics capabilities using analogies to Expedia and other tools to plan and deliver logistics support to their customers. Hear how they engaged key stakeholders and overcame funding and data sharing challenges to make things happen under a short timelines for a demanding customer.
DAU Powerful Examples Home
Keywords: best practices, lessons learned
For more information about this story, or to submit your own Powerful Example - send an email to the DAU Powerful Examples Team at email@example.com.
Not the phrase a Navy admiral wants to hear about his essential aircraft.
Time is critical when F/A-18 Super Hornet jets scramble for take-off. These carrier-based strike fighters, deploy on missions that range from power projection and forward deterrence to maritime security.
In his gut, Rear Admiral Jeffrey J. “Caesar” Czerewko knew too many non-mission capable aircraft took up precious space on flight decks. Non-mission capable is a readiness status that means an aircraft cannot execute its mission, and a carrier without mission-ready aircraft throws a wrench into naval aviation’s man-equip-train mandate. Now, Czerewko just needed the data to prove his theory.
Working with Dr. Adi Zolotov of CNA, Czerewko sought a solution that involved partnering his maintenance operators with CNA’s technical analysts to identify critical processes, then “iterate rapidly and continually” to improve maintenance.
“I was asking for something different than CNA would normally give us, which was a product and interim reporting,” said Czerewko.
In 2018, their efforts led to the genesis of the Force Readiness Analytics Group, or FRAG—naval aviation’s first office of data science. FRAG is forging a data analytics transformation in naval aviation, and accelerating a cultural change towards treating data as an asset to drive decision-making. The effort directly aligned with the 2018 National Defense Strategy and the Navy’s Business Operations Strategy, Agility and Accountability (FY 2019–2021).
Naval officers Lt. Cdr. Sean “Butterbean” Blackman and Cmdr. Jarrod “JROD” Groves were founding members of FRAG. At the start, their team spent months digging through data, assessing how to use that data to predict outcomes and inform decisions of senior leaders. To do so, they needed to alter the data environment to be more proactive and less reactive.
Blackman noted, “We didn’t have the right people with the right information able to act with good decisions and inform decision-makers with enough speed to really make sure that our hierarchies were protected and that we were able to be the most lethal force capable.”
They tackled the admiral’s request to be graded on mission capable readiness and flight hour execution. The group then hypothesized that the agile analytic data environment would need to identify levers of influence affecting readiness numbers.
Teams of operators and analysts worked together to employ machine learning--the science of causing computers to act without being explicitly programmed--and predictive advanced analytics--methods and tools that project future trends. They uncovered trends in maintenance shortfalls.
On their first, live prediction run, FRAG identified two squadrons that might struggle to meet readiness goals for the next quarter. They presented their predictions to Czerewko. Amazingly, FRAG’s forecasts about the two squadrons matched anecdotal inputs (hundreds of emails) that Czerewko had been receiving about the same two squadrons. The data provided an independent confirmation of his hunch. He was ready to act and directed squadron leaders to review manpower with a focus on positions that would improve the quality of maintenance. The result: fewer non-mission capable aircraft, precisely Czerewko’s intent.
As they continued to iterate with the massive amounts of data, FRAG moved from debating the validity of the data to debating solutions. In one situation, FRAG learned that customers needed visibility and transparency. Groves recalled that the “Air Boss” wanted to see his report card of readiness numbers on a weekly basis. FRAG created an online visualization process, capable of showing the status of all naval aviation on a daily basis.
As she coached FRAG, Dr. Zolotov assisted the commander in recognizing the “infinite opportunity in the space of data and analytics.”
Czerewko agreed and added that teaming maintenance operators with technical analysts allowed FRAG to bring “operational relevance” to the fleet as they continue to refine support to CNAF readiness levels.
Picture it: years of careful planning and meticulous strategizing to resolve an issue and improve it for future warfighters. Then imagine finally being ready to present your program acquisition strategy to the Service Acquisition Executive (SAE) for approval at Milestone B, only to be told to return with a new plan of action.
Such is the case with the B-52 Commercial Engine Replacement Program (CERP), a story they revealed at the DAU Alumni Association Acquisition Symposium on April 3, 2019. After briefing their acquisition strategy to Dr. William Roper, the U.S. Air Force Service Acquisition Executive, at Milestone B, which included plans for moving to a virtual system prototype to deliver residual operation capability, members of the CERP program found themselves at a red light. Dr. Roper then challenged the group to consider leveraging Section 804 authorities to plan the program with a Middle Tier of Acquisition strategy.
Click here to view a full-screen version of the video
Reflecting on how they overcame the hurdle, Abby Pogorzelski, CERP Program Manager, revealed they had to “develop our strategy from the bottom up.” To add to their already strenuous task, the group had only one night to “ensure we were following the intent of the law.”
In the end, all their efforts to save money while also providing new engines for the B-52 paid off. Pogorzelski added that the team learned to “take smart risks,” but did so under the guidelines of Section 804. This victory resulted in schedule and cost reduction, reduction of non-value added documentation, early virtual prototypes and the ability to release RFPs in milestones.
The shortened production time does not mean CERP is taking any shortcuts, Test Manager Bridget Durham, made sure to clarify. “We intend to run a full flight test program,” she stated, following up with the assurance that the program has “no intent of reducing rigor in our test program whatsoever.”
The team further elaborated that rigor is crucial in maintaining thorough documentation, which in turn provides CERP the means of rapid prototyping and rapid fielding. In other words, the ability to provide the best quality of B-52 engines without sacrificing production time.
“We owe it to the warfighter to get then the best they can possibly have,” agreed Michael Bredehoeft, Deputy Program Manager.
When asked if she had any advice to share for teams collaborating on new projects, Pogorzelski offered, “Think outside the box, engage stakeholders, bring all the right folks to the table and try to come up with something new and innovative.”
[RELATED CONTENT: 2019 DAU Acquisition Training Symposium Video]
[RELATED CONTENT: CERP slide deck from the 2019 DAU Acquisition Training Symposium]
[RELATED CONTENT: DAU Podcast with the CERP leadership team on getting started and leveraging Middle Tier Acquisition]
Do you have a powerful example of success or failure that you think others would benefit from? Share it with our team at firstname.lastname@example.org.
The Air Force’s C-130J, Super Hercules, Production & Development Program knows a thing or two about patience. When it comes to securing contracts and acquiring block upgrade kits, former Team Leader, Nathan Shrider, had plenty of experience to share. Shrider recalled a time when his team was tasked with acquiring block 8.1 upgrade kits in low-rate initial production in Lot 2.
“In the past, with Lot 1, it took nearly two years to get on contract, which was incredibly too long,” Shrider said.
He revealed that the biggest challenge his team faced was advancing from RFP (Request for Proposal) to contract award within 270 days.
“There’s typically a 60-90 day delay in their ability to create their proposal just due to solidifying those agreements,” Shrider explained. “This is one area that we recognized as an opportunity to be better at in our next acquisition.”
By using techniques available from DoD’s Sole Source Streamlining (SSS) Toolbox, such as implementing a memorandum of understanding between the government and contractor, Shrider and his team were able to streamline roles and responsibilities on both ends and cut this time-consuming process in half. Part of this victory entailed establishing early engagement with the Defense Contract Auditing Agency (DCAA) and connecting them with Supply Chain management within the contractor’s proposal team. This, Shrider said, reduced the proposal audit time from DCAA from 95 days to 45 days.
“When my team ventured off to try to reduce the procurement acquisition lead time down to 270 days, the biggest challenge was how we were going to get there through our internal gates,” Shrider said.
The team laid out a joint-integrated master schedule (IMS) with the contractor to quickly identify areas of opportunity. This schedule allowed for the C-130J team and the contractor to compare their internal processes prior to proposal submittal and define a timeline that accounted for both parties’ requirements.
“This joint IMS between both parties held both parties accountable and kept us on track throughout the acquisition process,” said Shrider.
It was important for the team to reflect on lessons gained from previous acquisitions. This helped the team identify past issues and in turn focus on overcoming issues they were facing with this current contract.
“Success for this program was reducing lead acquisition time from two years to 270 days,” Shrider shared. “Communication and transparency are paramount to having the same set of goals to get to the end result together.”
When you think about the phrase “augmented reality,” perhaps you think of video games or science fiction movies. It’s a concept straight out of the future, but at the Combat Capabilities Development Command (CCDC) Armament Center at Picatinny Arsenal, the future is now. Through the use of augmented reality assisted weapons, Army and Marine Corps programs hope to provide soldiers a new way of maintaining weapon systems.
“The use of augmented reality is going to enhance those products providing visual integration with a tactical system instead of having to go and reference separate system computer screens,” Joshua Zawislak, Project Lead of the CCDC Virtual Test & Training Environments, explained.
In learning to use this incredible technology for their acquisition research project, Zawislak’s team selected the M777A2 Howitzer. This particular weapons system was chosen for the project because models had already been developed for it in a 3D virtual trainer.
“Currently, complex weapon systems are maintained with paper manuals or digitized PDF manuals on a computer that are still on a screen,” Zawislak explained. “Some of the challenges with keeping these weapon systems up-to-date is maintaining information about different variants.”
This augmented reality technology, which has been fashioned into a pair of futuristic-looking glasses that the user can wear on-the-go, is expected to drastically improve the productivity of military operations. It is also intended to substantially decrease the associated risks that come with being in a war zone.
“Using augmented reality headsets, the user can have information provided through module windows, or highlighting of objects with text overlays, all through visual simulation,” Zawislak said.
Through various gestures, such as a pinching motion, users can select specific areas on a virtual schematic and highlight objects needing maintenance. Augmented reality technology also provides detailed videos and images associated with the object, along with a step-by-step diagram on how to complete the necessary functions, giving the user the ability to interact virtually with a component before working on it.
This concept of “augmented data” was largely inspired by BMW, Zawislak admitted. He spoke of BMW’s process for replacing a fan belt, and how the manufacturer’s use of this technology helped pave the way for maintenance on military equipment. Augmented reality technology has proven to be an instrumental tool in training new hands, as it allows users to fully map the repair process before actually dismantling any complicated equipment.
“The next step moving forward is looking at enhancing our requirements and developing a solid concept of operations for use of augmented reality for maintenance,” Zawislak stated.
System maintenance is not the only function this powerful tool is capable of, though. In the field, the headsets, using augmented reality technology, allow soldiers to view virtual maps and receive warnings of potential hazards in the area.
“We are looking to apply [the use of this emerging technology] to reduce the time of a repair, meaning that soldiers can have their systems available more in the field, keeping lethality at the maximum.”
The Kessel Run team gives us a glimpse of what life is like in a modern software development organization in DoD. Successful change management, and modern software practices such as DevSecOps and Extreme Programming, are critical capabilities that enable Kessel Run to continuously deliver value to the Warfighter.
Kessel Run team members (e.g., special projects director, release engineer, product manager, lab director, engineering practice lead, software engineer) discuss their culture, work environment and software practices such as pair programming, test driven development, and CI/CD through their software pipeline.
High performing software companies and industrial software factories have to release new code to the market at break-neck pace (within minutes or hours) just to maintain their competitive advantage. Kessel Run has been recognized for helping DoD move closer to achieving the Under Secretary of Defense’s goal to catch up to private sector software delivery performance. Kessel Run transformed a legacy system program that did not deliver capability for nearly a decade, into a modern organization that delivers war-winning capabilities to the fight every 12 hours. This involved scaling change and growing from a 20-person experiment -- to an organization of over 700 people that has inspired change across DoD. The efforts of the Kessel Run team, past and present, were recognized with new 2019 Software Innovation Team Award from the Under Secretary of Defense for Acquisition and Sustainment and the General Larry O. Spencer Innovation Award from Air Force Chief of Staff.
Enforcing change often takes time—especially where the government is involved.
For the hands behind the only Advanced Medium Range-Air-to-Air Missile (AMRAAM) production, time is something they can’t afford to lose if they hope to maintain a competitive advantage. This was the challenge the program recently faced when negotiating a contract for Lot 32. During an effort to insert a new variant of the AMRAAM missile into the field, which included a redesign of subsystems and replacement of electronic chips, obtaining the necessary funds without sacrificing too much lead time quickly became a challenge.
“Our job, as acquisition leaders, is to stay ahead of the game,” AMRAAM Program Manager, Colonel Brian Henson said.
“We needed a way to execute the AMRAAM production Lot 32 faster and more efficiently,” AMRAAM Lot 32 Production and Contracting Officer Jeff Mixson explained. “The DoD Sole Source Streamlining (SSS) Toolbox is a point-and-click interface that allows […] anyone involved in the acquisition process to streamline procurement action lead time without sacrificing quality or the art of the deal.”
Because of the similarities between Lots 31 and 32—same number of missiles, same type of variant, hardware and suppliers—the Lot 32 production team, by utilizing the previously analyzed data tool from the SSS Toolbox, were essentially able to skip an entire step of the process. This allowed the team to reduce the procurement action lead time by an estimated eight months, and their man time by about 3,000 hours—all while maintaining the competitive advantage.
“Success for us […] was the fact that we were able to get the contract awarded for us,” Col. Henson said. “Over six hundred missiles—within three months from the previous lot, rather than a year to a year and a half … That’s a year earlier we were able to focus on the next contract, which goes into also missiles being delivered earlier.”
Ultimately, the SSS Toolbox is intended to help acquisition specialists increase efficiency throughout the acquisition process. Of the 40 techniques offered by the SSS Toolbox, the team used techniques 3.5 and 3.7 during their evaluation of the Lot 32 proposal. The first of these techniques considers materiality and risks the government and program office is willing to take by conducting a top-level analysis of the Lot 32 proposal, while the latter streamlines cost analysis.
“The bottom line is: this tool box is there for a reason,” Mixson said. “Utilize it, utilize the techniques, understand your situation, make a case and present it to leadership and you will typically get a yes answer.
According to Col. Henson, “You can always challenge the status quo if you have a good reason.”
One of the biggest hurdles of managing data is keeping track of it.
The Army Leader Dashboard (ALD) will mitigate this challenge by organizing and integrating all Army data into one centralized location. It will provide leaders with easy accessibility to a comprehensive view of data that will aid in their strategic decision-making.
“There’s a great need to have that level of visibility, to have quality trusted data, to have informed, fact-based discussions instead of making decisions that impact the Army on assumptions,” said Strategic Initiatives Group Director and Project Lead, Lt. Col. Robert Wolfe from Program Executive Office Enterprise Information Systems (PEO EIS).
Wolfe hopes the ALD will address those visibility problems and the question of how to most effectively manage the Army’s data. In devising an innovative solution, Wolfe opted for Other Transaction Authorities (OTAs), a non-traditional contract method that Wolfe described as being “tailored to the need of each user.” By using this method, Wolfe has had organized access to 700 different data sources. However, Wolfe warned that “saving time is not one of the reasons to choose OTAs.”
The need for the ALD comes at the behest of the Army Chief of Staff, who stated that the enterprise suffers from the inability to “see self.” There is an abundance of data dispersed among numerous databases across the military Service. As that data continues to grow, the Army’s visibility dwindles.
“We currently don’t have that capability to look across the domains,” Wolfe said. “We have lots of great tools that can tell us everything about a person, we have lots of great tools that can tell us everything about training or logistics, but we don’t have a tool that gives us visibility to do cross-domain analysis to have insight-driven decisions.”
Among the system features the ALD will provide is data configurable to individual users, predictive analytics, plain text search capabilities, platform versatility and integrated private and public data. Wolfe believes these features will help drive the aforementioned data-driven decisions he spoke about, and eventually serve as a “common data platform” that Soldiers across the entire Army can access.
“We don’t know everything,” Wolfe admitted. “So we wanted to make sure that we leveraged industry, and we’re truly going to take advantage of best practices and cutting edge technology.”
Industry leader involvement was a major part of the process for executing the ALD. Wolfe revealed that 73 one-on-one sessions were conducted with industry leaders and partners in order to root out the data problem the Army was facing and determine a solution.
The success of the ALD will result in a competitive advantage for the Army by providing cost saving opportunities as the organization consolidates its analytics capabilities and increased productivity for workers.
When asked what advice he would share with new project managers, Wolfe kept it simple: “Don’t assume that you know everything.” Lastly, he advised others to trust the process and practice discipline.
Through Performance-Based Logistics, the Warfighter is given a leap in performance and transparency not offered by traditional acquisition arrangements.
In 2006, Naval Supply Systems Command Weapon Systems Support (NAVSUP WSS) began searching for a requisitions solution for the AMC computer program, a state-of-the-art system used in select high-performance jet aircraft that provides weapons targeting, digital imagery, and network-centric operations.
(Click here to watch a full screen version of the interview)
“We were having significant support issues in pre-PBL environment,” Larry Garvey of NAVSUP WSS said, during a roundtable discussion with Jeff Heron, NAVAIR, and DAU’s Betsy Lederer about Performance-Based Logistics (PBL) and the impact it’s had on solving readiness issues.
The search for a better way of doing acquisition led to Performance-Based Logistics, a weapon system sustainment strategy made to improve weapon system readiness and support and deliver items faster to the Warfighter. PBL leverages long-term performance-based agreements and incentivizes contractors to lean out the supply chain. The objective of PBL is to provide performance improvements for weapon systems, both new and legacy, as opposed to traditional sustainment models. Executed efficiently, PBL will generate substantial cost savings and improved capability.
“We were getting 37 percent materiel availability,” Garvey recounted. “Which means that only 37 percent of the time that we have an asset when we received a requisition from the fleet. At time of PBL award, we had 74 backorders.”
Through a partnership with the NAVAIR program office, NAVSUP WSS was able to secure a PBL to support the AMC computers just a couple of years later in 2008.
“We have seen significant improvements since the PBL was awarded,” said Garvey.
Garvey also explained that within six months of being awarded the PBL contract, materiel availability at NAVSUP WSS (Philadelphia, PA) improved to 100 percent and backorders were eliminated. Improvements have since been consistently maintained over the last ten years. Reliability of two of the key components has also been improved since instituting PBL, Garvey added. Advanced Mission Computer reliability is 29 percent better now than in the beginning, while the Mission Computer is 34 percent improved.
“Our goal at NAVSUP WSS is always to seek out the best possible support solution and PBL is one of the primary tools in our toolbox to do that,” Garvey stated.
In the end, both Garvey and Heron agreed that to procure the needs of the Warfighter, all parties involved need to come together to craft a solution.
[RELATED CONTENT: DAU Powerful Examples Homepage]
[RELATED CONTENT: Full-Length Podcast Interview with NAVSUP and NAVAIR Leaders]
[RELATED CONTENT: Life-Cycle Logistics Community of Practice]
[RELATED CONTENT: Life-Cycle Logistics Blog Landing Page]
For more information about this story, or to submit your own Powerful Example, send an email to the DAU Powerful Examples Team at email@example.com.
Key Words: Performance Based Logistics, NAVSUP, NAVAIR, Navy, DoD, Acquisition, Lessons Learned, Best Practices
More info: https://apps.dtic.mil/dtic/tr/fulltext/u2/1016070.pdf