Panel: On the Curation of Artifacts in the Era of AI/ML for Cybersecurity

Moderator: S. Jay Yang, Ph.D., ESL Global Cybersecurity Institute, Rochester Institute of Technology (Moderator)     SLIDES

Panelists:
David Balenson, USC Information Science Institute     SLIDES
Sebastián García, Stratosphere Laboratory, Czech Technical University in Prague
Robert Beverly, National Science Foundation     SLIDES
Emma Tosch, Northeastern University
Sagar Samtani, Indiana University

Abstract:
The emergence of AI/ML research for cybersecurity is meant to advance operation efficiency and reduction in cyber incidents. Unfortunately, the adoption of AI/ML advances for practical cyber defense has been slow and sporadic. With the various public and private support to transition research into practice, one would expect a much broader success by now. A key challenge to the limitation of practical impact of AI/ML research for cybersecurity is the lack of quality artifacts – the lack of know-how and recognized processes to curate, evaluate, share, and sustain the growth in development and uses of software, datasets, and experimental methodologies.

In recent years, researchers are increasingly producing and sharing their artifacts. Several conferences, workshops, journals are starting to encourage and recognize research artifacts and practitioners are trying to appreciate the value of research artifacts. However, many challenges and issues remain:

• The purposes, value, and quality of AI/ML for cybersecurity prototypes are not always well understood, documented, or aligned with practitioner needs.
• Researchers’ experiences in packaging and documenting the artifacts vary significantly. This becomes even more challenging with the strong correlation between software, datasets, and experimental methodologies for AI/ML research.
• There is a lack of effective mechanisms across the community to facilitate the adoption and deployment of quality artifacts, with support from the creators of the artifacts and the accumulation of knowledge in using the artifacts.
• There is a mismatch between public and private sectors on the expectations of cybersecurity software prototypes and datasets, which often translates into wasteful efforts in seemingly promising research results but not useful AI/ML solutions for cybersecurity operations.

Recent discussions through NSF 2021 CNS RFI and NSF 2022 SaTC PI Meeting have offered some insights to the challenges and opportunities for network system datasets and cybersecurity artifacts. Built upon these, this panel will invite researchers and practitioners from academia, industry, and government to discuss the current efforts, challenges, needs, and opportunities on curation of AI/ML software prototypes, datasets, and experimental methodologies for cybersecurity operations.