The logic of the event to an offline stream (in what follows called a logical stream) assignment is completely contained in the Stream Definition Table (SDT), which is a database table specifying the binary decision tree used for the assignment of a particular event to a certain stream. This binary tree completely defines the exclusive streams, as every event is assigned to one and only one stream as a result of the tree processing. The SDT should be linked to the Level 3 (L3) trigger database; it is desirable to change it as rarely as possible, but certainly not more often than the trigger menu itself.
The nodes of this decision tree use either L3 objects (such as an electron with the transverse energy in excess of a certain trigger threshold), or Level 3 trigger names, if a given trigger is to be mapped into a particular stream. The tree is parsed in order, from top to bottom, and parsing is terminated as soon as the decision to accept an event is reached at one of the nodes of the tree, which results in assigning a particular stream to the event.
The total number of physics streams should be small, of the order of half-a-dozen, which has a distinct advantage of having each logical stream mapped to a separate physical tape stream (in what follows, a physical stream). The online system at the moment allows for six such physical streams. Although desired, it is not necessary to have one-to-one logical stream to physical stream correspondence. Both the online system (via file families) and SAM allows for combination of several logical streams in one physical stream. While combining several streams on one tape in certain sense defeats the purpose of the exclusive streaming, sometimes it might become necessary, e.g. if one or more physical streams are taken by calibration or special runs. Combination of several logical streams on one tape also allows for future extensibility of the system by increasing the total number of streams or designating a special stream for a high-priority analysis (see below).
There are several considerations that should be taken into account when designing the SDT, both from the point of view of the online/offline systems and Run II physics. The following subsections discuss these considerations and possible SDT implementations.
One of the most important requirements to the offline streaming system is to be able to provide accurate integrated luminosity information for standard data sets. It is important to note that the luminosity per logical stream cannot be in general defined, as the only defined luminosities are the luminosities per trigger. Since events coming from a particular trigger could end up in more than one stream it is important to know for each trigger what streams the events from any given trigger can end up in. The tool that maps triggers to logical streams does exist, although has not been extensively tested.
The luminosity-tracking problem is thus reduced to gathering information about the luminosity blocks in each of the logical streams. The luminosity block corresponds to 60 seconds of data-taking, and has well defined instantaneous luminosity and scalars information for all the DØ triggers. Integrated luminosity is then calculated assuming constant instantaneous luminosity across the luminosity block. Each luminosity block has a unique number, and the online system guarantees that the events in a particular logical stream coming from one luminosity block are never split into two or more file partitions, i.e. the partitions are always opened and closed at the boundary of two luminosity blocks.
The information about the luminosity blocks is kept in the online luminosity database, which can be used to return information about the integrated luminosity corresponding to a given trigger and given range of luminosity blocks. The offline streaming system therefore should keep the list of luminosity blocks for each file in each of the logical streams in order to be able to construct a valid query to the luminosity database.
For a trigger mapped into several logical streams, one needs to process all the corresponding streams in order to obtain the luminosity information. It is possible, however, that certain range of luminosity blocks is missing from some of the streams, but present in the others. (A typical example of how this could happen is a damaged tape that holds several files from one of the logical streams.) In this case, for precise luminosity calculation, the streaming system has to mark the luminosity blocks missing from at least one of the logical streams as “bad.”
A user should eliminate these “bad” luminosity blocks from processing in any of the files used in the analysis. This is an approach that should be followed by the luminosity-sensitive analyses, such as measurement of a cross-section or a ratio of cross-sections. It is understood that this approach implies certain loss of statistics, but typical cross section measurements are systematic-dominated, and therefore a loss of small fraction of the statistics as a trade-off for precise knowledge of the integrated luminosity is worthwhile.
The other type of analyses, like searches for new physics, which are statistics-sensitive may want to process “bad” luminosity blocks in the streams where they do exist. It is understood, however, that such an action result in additional error in the integrated luminosity, related to an unknown fraction of lost data in the bad luminosity blocks missing from one of the streams. It is possible to estimate the amount of lost luminosity by calculation the fraction of events written to the streams with lost luminosity blocks in an adjacent luminosity block range, where no losses has occurred. Ideally, the streaming system would provide such an estimate with its uncertainty. However, given that the expected file lost rate is small (~1%, i.e. small fraction of the luminosity error), such a tool is not of as high priority as other tools needed for functional streaming system.
Finally, it is desirable to compress the luminosity database into a much smaller one where the information is integrated over the duration of each run. This database would be very manageable in size and can be stored offline for easy access for crude estimate of the integrated luminosity, perhaps sufficient for the preliminary stages of most of the analyses.
On one hand, while it’s desirable to have certain flexibility in the number of logical streams in order to decrease the amount of data required to be processed by a particular analysis, the exclusive streaming concept does not scale well with increased number of streams, as chances are that individual triggers will become split into more and more logical streams, therefore not saving any data processing time, and yet increasing complexity of the system.
Moreover, in order to add a new exclusive stream that contain only small fraction of the data, the corresponding node in the binary decision tree should be promoted very close to the top level. Unless we are talking about a very unique signature (like 4 electrons, ten jets, and missing energy), any such additional streaming branch close to the top of the decision tree is very likely to mess up the luminosity-sensitive triggers, such as inclusive electron or muon trigger used for W/Z cross section measurements. Therefore, introduction of such streams should be done with a great caution.
The other consideration against small and rare-count streams is the lifetime of the data file with the streamed data within the online system. If the event rate in such a file is small, it becomes hard to account for the missing luminosity blocks at the end of such a file, as the luminosity block of the last event in the file is no longer the last luminosity block of the file. Ideally, one would like to keep the streams large enough to have a few events per luminosity block written in each stream. Given the length of the luminosity block of 60 seconds and the rate-to-tape of ~10 Hz, that implies that no stream should be less than ~1% of the entire data set. Another disadvantage of having very small streams is that they are more vulnerable to the data loss disasters than their more frequent partners. The cost of a lost file or tape for such a stream is much higher, so if small streams are still desired, they probably have to be physically duplicated to prevent data loss offline. The vulnerability toward online data loss still remains high.
While the complication of having exclusive streams grows fast with their number, there exist another solution that enables low-rate streams, often desired by low-statistics search analyses. Not all the streams coming out of L3 are exclusive: there exist monitor triggers that are heavily prescaled, which need to be kept separately and handled as non-exclusive streams. Since the streaming system has to provide such a flexibility, one can use the same concept to create Express-Line Streams (ELS) that constitute a small (~5% or less) fraction of the entire data set that are non-exclusive. A typical use for such a stream would be to do a high-priority low statistics analysis by mapping a few specific triggers used for this analysis into the ELS, which would allow for their prioritized reconstruction and further processing. The ELS will consist of duplicates of the events that could be found in one of the inclusive streams, but it provides an efficient and flexible mechanism of prioritizing certain analyses without messing up the existing exclusive trigger setup. This approach also solves data vulnerability problems, as the missing events can be found elsewhere.
There are two issues related to the use of such non-exclusive stream: the first is that it size has to be kept under control very strictly, as the amount of tapes required for storing the Run II data set is very large, and therefore the experiment could only afford a small fraction of the data to be duplicated; the second is that the presence of non-inclusive streams opens a double-counting trap that users should not fall into. The first problem is entirely management issue, so it’s beyond the scope of this document; the second one could be solved in the software by not allowing queries that mix exclusive and non-exclusive streams. Typically, a user would be encouraged to use the ELS for preliminary stages of an analysis, with a final pass being done on a regular set of exclusive streams.
Given the above considerations, the following strawman streaming table is proposed for the initial stage of physics data-taking. It is based on the following physics concepts:
1) Keep the luminosity-sensitive W/Z cross section data in as few streams as possible;
2) Provide separate streams for main types of objects identified by the DØ detector;
3) Keep the number of streams small for possibility to store them on separate tapes;
4) Keep the size of the streams more or less equal for the most efficient streaming.
The first requirement is particularly important for Run II, as the W data is likely to be used as an ultimate luminosity calculation tool, given that its theoretical and experimental errors are much smaller than those for inelastic scattering process used in Run I.
In order to estimate frequency of various triggers, we perform a frequency test using Run I data. Note, that Run I detector was significantly different from the Run II detector, and so are the trigger rates. However the trigger frequencies in Run I reflect bandwidth allocation for various physics processes in Run I, and therefore could be used as a crude estimate for Run II. The absolute rate in Run I is not applicable to Run II, so we are only concerned with relative frequency of various triggers. The study was done on a representative trigger menu v10.2, spanning Run I runs 87804-89517. The sample corresponds to 5,671,746 events, or about 8% of the entire Run I statistics, and spans representative instantaneous luminosity range: (4-14)×1030 cm-2s-1, with a average of 8.1×1030 cm-2s-1. The results are listed in Table I.
|
Trigger name |
Brief Description |
Number of Events |
Prescale |
Effective fraction |
|
EM_1_MON |
PT(e) > 16 GeV |
28,154 |
100 |
33% |
|
MU_1_MAX |
PT(m) > 15 GeV |
136,792 |
1 |
2.4% |
|
MISSING_ET |
MET > 40 GeV |
115,410 |
1 |
2.0% |
|
MU_JET_CAL |
PT(m) > 10 GeV |
83,375 |
1 |
1.5% |
|
MU_ELE |
PT(m) > 8 GeV |
300,548 |
1 |
5.3% |
|
JET_MULTI |
ET(j)
> 10 GeV |
141,598 |
1 |
2.5% |
|
JET_4_MON |
ET(j)
> 10 GeV |
21,187 |
25 |
8.5% |
Table I: Frequency of various inclusive and semi-inclusive triggers in Run I.
Based on this table and the above 4 considerations, we propose the following “strawman” list of streams:
|
Priority |
Stream Name |
Stream Definition |
Comment |
|
1 |
MU_HIGH |
PT(m) > 5-10 GeV |
W/Z cross section in muon channel; New physics; B-physics; Top/Higgs |
|
2 |
ELE_HIGH |
PT(e) > 10-15 GeV |
W/Z cross section in electron channel; New physics; Top/Higgs |
|
3 |
PHOTON |
PT(g) > 10-15 GeV |
New physics, QCD, Electroweak |
|
4 |
TAU_B |
PT(e) > 3-5 GeV or PT(m) > 3-5 GeV or t-jet or Isolated track with PT > 5-7 GeV and low occupancy |
New physics; W/Z cross section in t-channel; B-physics; Top/Higgs |
|
5 |
MISSING_ET |
MET > 25-40 GeV |
New physics; Top/Higgs |
|
6 |
JET_MANY (or some other jetty stream, like HT) |
PT(j) > 15-20 GeV; N(j) > 3-4 |
New physics; Top/Higgs; QCD |
|
7 |
QCD |
Everything else |
QCD; Top/Higgs |
In this strawman list, each of the main objects, reconstructed by DØ is represented as a separate stream, with the decision-making order determined by the relative frequency of their appearance. The exact definition of the objects has to be tuned using real data and will match the definition in the corresponding triggers. Thus, for W/Z cross section analysis, the user only needs to process the MU_HIGH stream. In order to do a similar analysis in the electron channel, the user is required to process at most two streams (MU_HIGH and ELE_HIGH), or by designing the analysis in such a way that the events with high-pT muons are rejected from the cross section measurement in the electron channel (probably a smart idea, anyway), this list could be reduced to a single ELE_HIGH stream. Similarly, events with photons (that are being faked ~10 time more often than the electrons) can be found in the first three streams. The fourth stream is designed for t’s and b-jets. When the SVX trigger becomes available, tagged b-jets will also be routed into this stream. The MISSING_ET stream is inclusive stream for events with large MET, provided that they did not make it to one of the previous streams. Depending on the final MET threshold, it could be promoted somewhat up in the above hierarchy. Stream number 6 is somewhat optional stream designed to off-load the largest “Everything else,” or QCD stream. It can be defined more finely, perhaps accepting events with two jets and MET, or some combination of FPD triggers (although the latter belong more to special low-luminosity runs).
A number of smaller dedicated streams can be inserted in various places in the table (although for the purposes of not messing up the W/Z cross section measurements it is desirable to keep them below the first two rows in the above table). Alternatively, they can be arranged as non-exclusive streams, as discussed above.
The Run Summary Database should keep the version of the Stream Definition Table, and also could specify how to combine the logical streams into physics streams if the amount of free resources is not sufficient to assign a physical stream to each logical stream. It is desirable to make such a grouping manually, from the point of view of physics, rather than let COOR to decide how to group streams based solely on the load considerations. An example of such a grouping table is shown below:
|
# of free physical streams |
Grouping |
||||||
|
7 |
MU_HIGH |
ELE_HIGH |
PHOTON |
TAU |
MISSING_ET |
JET_MANY |
QCD |
|
6 |
MU_HIGH
+ ELE_HIGH |
PHOTON |
TAU |
MISSING_ET |
JET_MANY |
QCD |
|
|
5 |
MU_HIGH
+ ELE_HIGH |
PHOTON +
TAU |
MISSING_ET |
JET_MANY |
QCD |
||
|
4 |
MU_HIGH
+ ELE_HIGH |
PHOTON +
TAU |
MISSING_ET
+ JET_MANY |
QCD |
|||
|
3 |
MU_HIGH
+ ELE_HIGH |
PHOTON +
TAU |
MISSING_ET
+ JET_MANY + QCD |
||||
|
2 |
MU_HIGH
+ ELE_HIGH + PHOTON + TAU |
MISSING_ET
+ JET_MANY + QCD |
|||||
|
1 |
MU_HIGH
+ ELE_HIGH + PHOTON + TAU + MISSING_ET + JET_MANY + QCD |
||||||
(Another possibility is to store the grouping table together with the Stream Definition Table.)