Methods for a blind analysis of isobar data collected by the STAR collaboration

J. Adam · L. Adamczyk · J. R. Adams · J. K. Adkins · G. Agakishiev ·M. M. Aggarwal · Z. Ahammed · I. Alekseev · D. M. Anderson ·A. Aparin · E. C. Aschenauer · M. U. Ashraf · F. G. Atetalla ·A. Attri · G. S. Averichev · V. Bairathi · K. Barish · A. Behera ·R. Bellwied0 · A. Bhasin · J. Bielcik · J. Bielcikova · L. C. Bland ·I. G. Bordyuzhin · J. D. Brandenburg · A. V. Brandin · J. Butterworth ·H. Caines · M. Caldero′n de la Barca Sa′nchez · D. Cebra · I. Chakaberia, ·P. Chaloupka · B. K. Chan · F-H. Chang · Z. Chang · N. Chankova-Bunzarova ·A. Chatterjee · D. Chen · J. Chen · J. H. Chen · X. Chen ·Z. Chen · J. Cheng · M. Cherney · M. Chevalier · S. Choudhury ·W. Christie · X. Chu · H. J. Crawford · M. Csana′d · M. Daugherity ·T. G. Dedovich · I. M. Deppner · A. A. Derevschikov · L. Didenko ·X. Dong · J. L. Drachenberg · J. C. Dunlop · T. Edmonds · N. Elsey ·J. Engelage · G. Eppley · S. Esumi · O. Evdokimov · A. Ewigleben ·O. Eyser · R. Fatemi · S. Fazio · P. Federic · J. Fedorisin · C. J. Feng ·Y. Feng · P. Filip · E. Finch · Y. Fisyak · A. Francisco · L. Fulek ·C. A. Gagliardi · T. Galatyuk · F. Geurts · A. Gibson0 · K. Gopal ·X. Gou · D. Grosnick0 · W. Guryn · A. I. Hamad · A. Hamed ·S. Harabasz · J. W. Harris · S. He · W. He · X. H. He · Y. He ·S. Heppelmann · S. Heppelmann · N. Herrmann · E. Hoffman0 ·L. Holub · Y. Hong · S. Horvat · Y. Hu · H. Z. Huang · S. L. Huang ·T. Huang · X. Huang · T. J. Humanic · P. Huo · G. Igo · D. Isenhower ·W. W. Jacobs · C. Jena · A. Jentsch · Y. Ji · J. Jia, · K. Jiang ·S. Jowzaee · X. Ju · E. G. Judd · S. Kabana · M. L. Kabir ·S. Kagamaster · D. Kalinkin · K. Kang · D. Kapukchyan · K. Kauder ·H. W. Ke · D. Keane · A. Kechechyan · M. Kelsey · Y. V. Khyzhniak ·D. P. Kiko?a · C. Kim · B. Kimelman · D. Kincses · T. A. Kinghorn ·I. Kisel · A. Kiselev · M. Kocan · L. Kochenda · L. K. Kosarzewski ·L. Kramarik · P. Kravtsov · K. Krueger · N. Kulathunga Mudiyanselage0 ·L. Kumar · S. Kumar · R. Kunnawalkam Elayavalli · J. H. Kwasizur ·R. Lacey · S. Lan · J. M. Landgraf · J. Lauret · A. Lebedev ·R. Lednicky · J. H. Lee · Y. H. Leung · C. Li · C. Li · W. Li ·W. Li0 · X. Li · Y. Li · Y. Liang · R. Licenik · T. Lin · Y. Lin ·M. A. Lisa · F. Liu · H. Liu · P. Liu · P. Liu0 · T. Liu · X. Liu ·Y. Liu · Z. Liu · T. Ljubicic · W. J. Llope · R. S. Longacre ·N. S. Lukow · S. Luo · X. Luo · G. L. Ma0 · L. Ma · R. Ma ·Y. G. Ma0 · N. Magdy · R. Majka · D. Mallick · S. Margetis ·C. Markert · H. S. Matis · J. A. Mazer · N. G. Minaev · S. Mioduszewski ·B. Mohanty · I. Mooney · Z. Moravcova · D. A. Morozov · M. Nagy ·J. D. Nam · Md. Nasim · K. Nayak · D. Neff · J. M. Nelson ·D. B. Nemes · M. Nie · G. Nigmatkulov · T. Niida · L. V. Nogach ·T. Nonaka · A. S. Nunes · G. Odyniec · A. Ogawa · S. Oh · V. A. Okorokov ·B. S. Page · R. Pak · A. Pandav · Y. Panebratsev · B. Pawlik0 ·D. Pawlowska · H. Pei · C. Perkins · L. Pinsky0 · R. L. Pinte′r ·J. Pluta · J. Porter · M. Posik · N. K. Pruthi · M. Przybycien ·J. Putschke · H. Qiu · A. Quintero · S. K. Radhakrishnan · S. Ramachandran ·R. L. Ray · R. Reed · H. G. Ritter · O. V. Rogachevskiy · J. L. Romero ·L. Ruan · J. Rusnak · N. R. Sahoo · H. Sako · S. Salur · J. Sandweiss ·S. Sato · W. B. Schmidke · N. Schmitz · B. R. Schweid · F. Seck ·J. Seger · M. Sergeeva · R. Seto · P. Seyboth · N. Shah · E. Shahaliev ·P. V. Shanmuganathan · M. Shao · A. I. Sheikh · W. Q. Shen0 ·S. S. Shi · Y. Shi · Q. Y. Shou0 · E. P. Sichtermann · R. Sikora ·M. Simko · J. Singh · S. Singha · N. Smirnov · W. Solyst ·P. Sorensen · H. M. Spinka · B. Srivastava · T. D. S. Stanislaus0 ·M. Stefaniak · D. J. Stewart · M. Strikhanov · B. Stringfellow ·A. A. P. Suaide · M. Sumbera · B. Summa · X. M. Sun · X. Sun ·Y. Sun · Y. Sun · B. Surrow · D. N. Svirida · P. Szymanski ·A. H. Tang · Z. Tang · A. Taranenko · T. Tarnowsky · J. H. Thomas ·A. R. Timmins0 · D. Tlusty · M. Tokarev · C. A. Tomkiel · S. Trentalange ·R. E. Tribble · P. Tribedy · S. K. Tripathy · O. D. Tsai · Z. Tu ·T. Ullrich · D. G. Underwood · I. Upsal, · G. Van Buren · J. Vanek ·A. N. Vasiliev · I. Vassiliev · F. Videb?k · S. Vokal · S. A. Voloshin ·F. Wang · G. Wang · J. S. Wang · P. Wang · Y. Wang · Y. Wang ·Z. Wang · J. C. Webb · P. C. Weidenkaff · L. Wen · G. D. Westfall ·H. Wieman · S. W. Wissink · R. Witt · Y. Wu · Z. G. Xiao ·G. Xie · W. Xie · H. Xu · N. Xu · Q. H. Xu · Y. F. Xu0 · Y. Xu ·Z. Xu · Z. Xu · C. Yang · Q. Yang · S. Yang · Y. Yang · Z. Yang ·Z. Ye · Z. Ye · L. Yi · K. Yip · Y. Yu · H. Zbroszczyk · W. Zha ·C. Zhang · D. Zhang · S. Zhang · S. Zhang0 · X. P. Zhang ·Y. Zhang · Y. Zhang · Z. J. Zhang · Z. Zhang · Z. Zhang ·J. Zhao · C. Zhong0 · C. Zhou0 · X. Zhu · Z. Zhu · M. Zurek ·M. Zyzak · STAR Collaboration

AbstractIn2018, theSTAR collaboration collected data fromto search for the presence of the chiral magnetic effect in collisions of nuclei. The isobar collision species alternated frequently betweenZr. In order to conduct blind analyses of studies related to the chiral magnetic effect in these isobar data, STAR developed a three-step blind analysis procedure. Analysts are initially provided a‘‘reference sample’’of data,comprised of a mix of events from the two species,the order of which respects time-dependent changes in run conditions. After tuning analysis codes and performing time-dependent quality assurance on the reference sample, analysts are provided a species-blind sample suitable for calculating efficiencies and corrections for individual ≈30-min data-taking runs.For this sample, species-specific information is disguised,but individual output files contain data from a single isobar species. Only run-by-run corrections and code alteration subsequent to these corrections are allowed at this stage.Following these modifications,the‘‘frozen’’code is passed over the fully un-blind data,completing the blind analysis.As a check of the feasibility of the blind analysis procedure, analysts completed a ‘‘mock data challenge,’’ analyzing data from Au+Au collisions atcollected in 2018. The Au+Au data were prepared in the same manner intended for the isobar blind data.The details of the blind analysis procedure and results from the mock data challenge are presented.

Keywords Blind analysis · Chiral magnetic effect ·

1 Introduction

For more than a decade, the STAR Collaboration has been searching for evidence of chiral magnetic effects(CME) [1–3]. CME [4, 5] refers to the induction of an electric current (Je) by the magnetic field (B) in a chiral system: Je ∝μ5B. A chiral system bears a nonzero μ5,which characterizes the imbalance of right-handed and lefthanded fermions in the system. The discovery of CME in high-energy heavy-ion collisions would confirm the simultaneous existence of ultra-strong magnetic fields,chiral symmetry restoration, and topological charge changing transitions in these collisions. On average, B is perpendicular to the reaction plane(ΨRP)that contains the impact parameter and the beam momenta.CME,therefore,will manifest a charge transport across the reaction plane.

2 Blinding techniques

2.1 General principle

Blind analyses often rely on a ‘‘reference sample’’ and an inability to differentiate two or more samples or a particular sample from the reference (see Ref. [9] for a brief overview of blind analyses in particle physics). The reference sample is often used either to tune an analysis without pre-determined bias or to provide a reference for evaluating the significance of a result, e.g. eliminating placebo effects or genetic conditions that may bias the result of medical studies.

2.2 Considerations

While many possibilities exist,the blinding method for a particular analysis should be well-matched to the specific needs of that analysis. For example, many of the typical methods (see Ref. [9] and references therein) do not specifically address the situation of comparing parallel analyses of two different but quite similar data samples.Among the specific considerations for analysis of the 2018 STAR isobar data are the following:

· The un-blind data should not be accessible by physics analysts prior to analysis tuning.

· Accounting for time-dependent detector fluctuations is a critical component of analysis quality assurance (Q/A).

· Accounting for run-by-run anomalies is a critical component of final analysis Q/A.

· Methods to blind by‘‘hiding’’or‘‘offsetting’’variables or information needed to gain sensitivity to signals are quite common in the literature, e.g. Refs. [10, 11]. We found many of these methods not well-suited to our analysis. In many cases, randomizing variables within an event may severely compromise the quality of analysis Q/A and associated corrections. For example,randomizing the sign of charged particle tracks would prevent charge-dependent efficiency corrections; and randomizing particle azimuthal angle would destroy correlations from secondary decays. Because of these considerations,such methods are not retained as part of this procedure.

· To ensure the isobar species have statistically comparable behaviors in terms of luminosity, event trigger composition, energy, vertex distribution, occupancy of tracks, etc., the 2018 RHIC run involved frequent switching of the isobar collision species.

· With this consideration in mind, it is feasible to interleave or ‘‘mix’’ events from the two collision species in a given output data file as an efficient method to disguise the collision species.

· Certain STAR experts, recused from blind physics analyses, may require isobar information during RHIC running to ensure data of sufficient quality to achieve target physics goals.

· Calibration experts, who are recused from conducting blind physics analyses, may need access to un-blind data to ensure sufficiently robust calibrations and corrections to achieve the desired physics goals.

· Runs of quality suitable for inclusion in physics analyses, e.g. not exhibiting large detector inefficiencies, must proceed prior to the mixing of events from different species.

For the blind analysis of isobar data collected in 2018,STAR adopted a three-step blinding procedure.For the first step,analysts are provided output data files that mix events from the two isobar collision species, while respecting the time-dependence of run conditions. Analysts use this data sample to perform time-dependent Q/A of the data and to tune analysis codes. At the conclusion of these studies,analysts commit their code to a repository. In the second step, analysts are provided an ‘‘unmixed-blind,’’ sample suitable for calculating corrections that vary according to individual ≈30-min data-taking runs. The run identification numbers are disguised,but the output data files do not mix events from different runs. Only these ‘‘run-by-run’’corrections (e.g. for changing detector efficiencies) and code alterations subsequent to these corrections are allowed during this step.At the conclusion of these studies,the final codes are committed to the repository, so that differences may be evaluated. After the analysis codes are verified, the final data analysis pass is completed using these final codes and the fully un-blind data released.

2.3 Initial procedure

Initial implementation of the analysis blinding procedure began prior to and during the 2018 RHIC run. To the extent possible,information pertaining to the isobar species was restricted during the run. Access to raw data for purposes of Q/A during the run was restricted to identified experts, approximately 5% of the collaboration, recused from blind physics analyses.To the extent possible,all raw data samples were limited in size below the level needed for sensitivity to a CME signal, e.g. less than 10,000 events. Un-blind experts produced species-blind performance plots to evaluate data quality for the run in-progress.

Prior to the software production of the blind data,it was necessary to set detector calibrations and determine an appropriate list of quality data-taking runs. Due to the importance of robust calibrations to the physics analyses,these calibrations were performed by the relevant experts using un-blind data.These calibration experts were recused from participation in blinded physics analyses. Additionally, a committee was designated to determine data-taking runs of sufficient quality for inclusion in physics analyses.Members of this run selection committee were also recused from participation in blinded physics analyses. Production of the blind data commenced after calibrations and the designation of good runs.

No physics analysis groups are provided with un-blinded data prior to completion of the un-blinding procedure.

2.4 Blind data production

In the blind production of data, the following information encoded in the data stream (DST) are obfuscated: the identification numbers for the event, its particular datataking run, and RHIC fill; the event timestamp; the event collision species; and the hit rates for the east and west STAR zero-degree calorimeters (ZDC) [12] and beambeam counters(BBC)[13],as well as their coincidence and background rates. All output data files are assigned a generic name and pseudo-run-number that monotonically increases with time. The exact start time of a data production is not known to ensure, e.g. that a particular pseudo-run-number is not trivially related to a particular isobar species. The mixing procedure and exact algorithm to re-assign pseudo-run numbers are encrypted and only known by two experts, who are recused from performing blind physics analyses. The reference sample, speciesseparated samples, and fully unblind samples are provided in a three-step process.

2.5 Step-1: ‘‘The Reference’’

Analysts are initially provided output files composed of events from a mix of the two isobar species. The mixing procedure is not a priori known. As much as possible, the order of events respects temporal changes in running conditions.Events showing peculiar discrepancies from the initial Q/A are excluded from the sample,and events from the two species are only combined if the detector performance, e.g. acceptance, was similar for the two events.Events are randomly rejected at the level of ～10%,so that the species cannot be determined, e.g. by counting the number of events associated with a particular run or event trigger and correlating it with information from the run log database.Analysis code and time-dependent Q/A are tuned on this reference sample, committed to the analysis code repository, and kept unchanged at this stage. Among other aspects, this step enables extraction of time-dependent spectra for Q/A, detection of time-dependent anomalies,detection of secondary decays and measurement of peak widths relevant to momentum resolution.

2.6 Step-2: ‘‘The run by run Q/A sample’’

After analysis of the reference data, analysts are provided an ‘‘unmixed-blind sample’’ comprised of files that obscure the true run number (and, hence, the isobar species) but do not mix events across different runs. The pseudo-run-number uniquely maps to one true run number and one(unknown)isobar species.The data are provided in such a way that a mix of files from each species appear in the same directory.As in the first step,a fraction of events from each run is rejected to ensure that simple counting of events could not decipher the species.This sample enables species-blind run-by-run Q/A.Only run-by-run corrections and code alteration directly resulting from these corrections are allowed at this stage. The number of events provided per file is tuned so that statistics are sufficient for robust corrections but insufficient for deciphering the isobar species.

2.7 Step-3: full un-blinding

Once Q/A is complete and analyses of the run-by-run Q/A data are final,full un-blinding proceeds.At this stage,physics results are produced with the previously tuned,vetted,and fixed analysis codes.In this data production,all information is un-blinded and restored to the data files.

3 Implementation and timeline for blinded analyses

No STAR physics analyses had access to species information prior to un-blinding. The timelines for unblinding are estimated by the blind analysts, who present regular updates to their respective physics working groups(PWG)to document progress and to inform adjustments to the timeline.Decisions to un-blind are based upon a review of thoroughly documented analysis procedures, codes, and analysis reports—including estimates of measurement uncertainty—by the relevant PWG. In addition, for blind analyses of the isobar data, so-called ‘‘godparent committees’’or‘‘GPCs,’’are set early and follow analyses closely throughout their development. The GPCs serve an important role in verifying that analyses are ready to proceed to the next stages of the blinding procedure. After the step-1 data are available, blind-data analysts estimate a timeline for completing the necessary analyses for advancing to step-2. Based on this input from the analysts, management approves a date for the beginning of the second step.Analysts present regular updates to document progress.Regardless of progress, un-blinding occurs no earlier than the original estimate unless all blind analyses are deemed ready to proceed by STAR Management. Based upon the progress reports,un-blinding may be delayed to ensure the quality of the final results. An analogous timeline procedure is done for the full un-blinding. Prior to the first unblinding step, analysts prepare detailed notes documenting the procedures, cuts, corrections, systematic uncertainties,and criteria for any future run-by-run cuts and corrections.Prior to the second un-blinding step, analysts ensure that the documentation is updated and complete, including the run-by-run portion of analyses. Prior to each un-blinding step, analysts provide analysis codes for vetting and Q/A by the GPC in addition to the standard vetting within the physics working groups.

When the GPC is satisfied that an analysis is ready for un-blinding, analysts present the status of their analyses to the physics working group conveners and the physics analysis coordinator. As the un-blinding date approaches,analysts discuss with STAR management any need for delays to un-blinding to ensure the quality of results. If an unresolved disagreement exists between analysts, the decision to un-blind or extend the date lies with STAR management. After physics results are produced with unblinded data,a review is conducted to verify that the frozen analysis code was used to produce the results.

Fig. 1 Mean transverse momentum of charged particle tracks associated with the primary collision vertex for three mock data samples: (top) ‘‘mixed-blind,’’ (middle) ‘‘unmixed-blind,’’ and (bottom) ‘‘un-blind.’’ The data were taken from Au+Au collisions at=27 GeV and provided to analysts using the computing machinery developed for the blind analysis of isobar data. Note that the three samples shown here do not contain identical sets of events.The‘‘mixed-blind’’data contain events from a mixture of data-taking runs, combined together in an output datafile for analysis. The‘unmixed-blind’’ samples do not mix across data-taking runs but do blind other information that could be used to identify the isobar species, e.g. the proper data-taking run ID number. The ‘‘mixedblind’’ and ‘‘unmixed-blind’’ datafiles are assigned psuedo-runnumbers (shown across the top of the upper two panels, where the actual run number is shown in the lower panel) that prevent the analysts from identifying the collision species from a run database.The‘‘un-blind’’samples are presented to analysts with no information obscured. The calculated observable is shown as a function of an arbitrary run index

While un-blinded data are not accessible to physics analyses until the blinding timeline is completed, management uses discretion in applying blinding to any calibration analysis. To ensure the integrity of calibrations,e.g. those of the beamline and TPC[14],STAR calibration experts may require access to un-blind data.Without robust calibrations, the physics analyses may not be able to achieve the required precision for deciphering a CME signal.Therefore,the relevant experts are allowed access to the un-blind data for these tasks.Furthermore,access to unblind data is restricted to these experts alone and the experts recuse themselves from participation in any blind physics analysis.

4 Mock data challenge

5 After un-blinding

After un-blinding, only changes to correct ‘‘mistakes,’’defined for this purpose as errors in arithmetic or unintended departures from the approved and documented analysis procedures, are allowed. If such a correction is made, the analysis results with the error will also be provided with a detailed explanation of the specific correction applied and why it was needed. On a case-by-case basis,the collaboration considers announcing the result from a blind analysis simultaneously with the submission of the corresponding paper to the journal and the preprint arXiv.Regardless, only one set of ‘‘final’’ results from the blind analysis will be released, e.g. there will be no set of‘‘preliminary’’results prior to the‘‘final’results.All STAR publications of 2018 results state explicitly whether the analysis followed the approved STAR blinding procedure.

6 Conclusion

The STAR Collaboration has developed a procedure to carry out blind analyses of isobar collision data, collected in 2018. The procedure described in this manuscript was accepted by the STAR Council in January 2018, prior to the isobar collision runs.The initial step in the procedure is an analysis of blinded data samples that interleave events from the two collision species, while the second step involves analysis of blinded data samples that do not mix events from the two collision species, followed by complete un-blinding of the data. Prior to commencing with analysis of the isobar data, a mock data challenge was successfully conducted to demonstrate the feasibility of the procedure both from an analysis standpoint and a computational standpoint. Analyses of the blind data are underway, following the procedure outlined in this manuscript.

AcknowledgementsWe would like to thank the BNL Nuclear and Particle Physics Program Advisory Committee for the initial suggestion and ongoing support for this blind analysis initiative. We thank the RHIC Operations Group and RCF at BNL, the NERSC Center at LBNL,and the Open Science Grid consortium for providing resources and support.This work was supported in part by the Office of Nuclear Physics within the U.S. DOE Office of Science, the U.S.National Science Foundation, the Ministry of Education and Science of the Russian Federation, National Natural Science Foundation of China, Chinese Academy of Science, the Ministry of Science and Technology of China and the Chinese Ministry of Education, the Higher Education Sprout Project by Ministry of Education at NCKU,the National Research Foundation of Korea, Czech Science Foundation and Ministry of Education, Youth and Sports of the Czech Republic,Hungarian National Research,Development and Innovation Office, New National Excellency Programme of the Hungarian Ministry of Human Capacities, Department of Atomic Energy and Department of Science and Technology of the Government of India,the National Science Centre of Poland, the Ministry of Science,Education and Sports of the Republic of Croatia,RosAtom of Russia and German Bundesministerium fur Bildung, Wissenschaft, Forschung and Technologie (BMBF), Helmholtz Association, Ministry of Education,Culture,Sports,Science,and Technology(MEXT)and Japan Society for the Promotion of Science (JSPS).