Recommendation E.421

SERVICE QUALITY OBSERVATIONS ON A STATISTICAL BASIS

1	Definitions

1.1	service observation

F: observation de la qualité de service

S: observación de la calidad del servicio

Monitoring to obtain a complete or partial assessment of the quality of telephone calls, excluding test calls.

1.2	manual observation

F: observation manuelle

S: observación manual

Monitoring of telephone calls by an observer without using any automatic data-recording machine.

1.3	automatic observation

F: observation automatique

S: observación automática

Monitoring of telephone calls without an observer.

1.4	semi-automatic observation

F: observation semi-automatique

S: observación semiautomática

Monitoring of telephone calls using equipment which records some data automatically. For example, equipment in which information, such as

exchange being observed, number dialled by the subscriber, metering pulses and time of call, is recorded automatically on some means suitable for data processing. The observer merely has to key in a code indicating the condition observed.

2	Relative merits of manual, automatic and semi-automatic observations

2.1	The three methods mentioned above in SS 1.2, 1.3 and 1.4 are not exclusive; for example, automatic observations may be used to supplement observations taken by an operator. It was considered in 1968 that the need for automatic observations would increase in view of the heavy cost associated with manual or semiautomatic observations on the rapidly expanding international network. It was also considered that automatic observations would not entirely supersede observations taken by an observer within the foreseeable future.

The relative merits of the three methods can be assessed as follows:

2.2	Manual observation

Provides all the data required in Tables 1/E.422 and 1/E.423.

Observations can be carried out with a minimum of equipment.

Observations can permit the detection of a number of abnormalities which cannot be detected automatically, e.g. very poor speech transmission (item 5.2 of Table 1/E.422), or difficulty with audible tones encountered in the international service (item 6.4 of Table 1/E.422).

2.3	Semi-automatic observation

Provides all the data required in Tables 1/E.422 and 1/E.423.

There is a saving in staffing costs compared with manual observation.

Greater accuracy compared with manual observation is possible due to the fact that there is an automatic recording of the number dialled, the time of the call, etc.

It is possible for the observer to devote greater attention to the more critical conditions being checked during observations of calls.

The results are produced in a form suitable for subsequent mechanized analysis.

Owing to the reduction of costs it is possible to obtain a larger sample for the same expenditure.

Semi-automatic equipment may be converted, during certain hours of the day, to automatic operation.

2.4	Automatic observation

Operating cost is minimum (staff reduction).

Continuous observation is possible.

It is possible to have a larger sample.

Human error is eliminated.

Automatic processing of data is facilitated.

Conversational privacy is ensured.

Control of the time at which observations are made is facilitated.

Some of the differences between internal and external automatic observations are given below:

2.4.1	Internal automatic observations can be made in the switching centre itself, on the incoming side or the outgoing side or in between, according to the way the switching centre is engineered:

a)	Only line signals, such as seizure, answer, etc. can be monitored, and also register signals as long as they do not pass through the exchange in an end-to-end signalling procedure.

b)	Signals received are only monitored if the exchange itself operates correctly in that respect.

c)	Item b) applies also to outgoing signals. If there is a fault in the exchange it can happen that signals have not been sent in the appropriate way without the exchange being aware of it.

More information on this type of observation technique is given in Recommendation E.425.

2.4.2	External automatic observations are made by means of monitoring equipment which is supervising the traffic on incoming or outgoing lines:

-	All signalling signals can be monitored.

-	The detection of tones, speech and data is possible if advanced equipment is used.

-	This observation technique provides all the data required in Table 2/E.422 and Table 2/E.423.

-	The application is very flexible and can be used instead of manual or semi-automatic observation techniques.

3	Time of observations

The results of all observations taken over the whole day should be recorded in Table 1/E.422 or Table 2/E.422.

In the case where observations are not taken over the whole day the observation period is recorded under the heading "Time of observations" and should include the three busiest hours of the day.

4	Observation access points

4.1	Observations for Table 1/E.422 or Table 2/E.422 should be carried out from access points as close as possible to the outgoing international exchange.

The following access points can be considered:

For definitions of test access points see Recommendation M.700. See also Recommendation M.110.

i)	outgoing relay set of an international circuit ("exchange" side), i.e. international circuit access point1);

ii)	incoming relay set of a national circuit;

iii)	link circuits of the international exchange.

Observations will be made only while the call is being set up, and a few seconds after the called subscriber's reply.

When the circuit access point is used for observation of international calls it is possible that the service quality of the international exchange may not be checked by either international or national observation programmes.

Preferably, and where technically feasible for the most complete results, observations for Table 1/E.422 should be carried out as close as possible to the international exchange on the national side. This would be more representative of service to the subscriber, and allows observation of call failure at the outgoing international exchange. Where it is not possible to make the distinction between failures in the outgoing international exchange, and failures beyond this exchange, or where there is a meaningful advantage in doing so, observations should be taken on the outgoing side.

It is necessary to state in Table 1/E.422 or Table 2/E.422 the access point where the observations have been made, as observations obtained at each one of the three access points mentioned above are not comparable.

4.2	Observations for Table 1/E.423 must be carried out from access points on the operators' positions.

5	Number of observations

5.1	Service observing programmes should be established in such a manner that statistical results obtained be as reliable as practicable bearing in mind the cost of obtaining large samples.

5.2	According to the studies carried out by the CCITT in 1964-1968, the quantities shown below are considered the minimum quantities to provide a general indication of the quality of service.

5.2.1	Table 1/E.422

The minimum number of observations per outgoing circuit group for Table 1/E.422 should be 200 per month when more than 20 circuits are included in

a group, 200 per quarter when there are between 10 and 20 circuits in a group and 200 per year if there are less than 10 circuits in a group.

5.2.2	Table 1/E.423

The minimum number of observations for Table 1/E.423 should be 200 per quarter when there are more than 20 circuits in the group, 200 per semester when there are between 10 and 20 circuits and 200 per year when there are less than 10 circuits in the group.

5.2.3	Transit traffic

Where an outgoing circuit group also carries transit traffic it is desirable to obtain data for each destination country reached via this circuit group. In principle, the number of observations for each destination should be obtained as indicated above. To accomplish this, one should use for each destination country its corresponding number of erlangs and derive from these erlangs a theoretical number of circuits.

However, where only a very small amount of traffic is handled, e.g. less than 5 erlangs, each Administration may wish either to make a smaller number of observations or (e.g. in case of no complaints) no observations at all and rely on the information obtained at the transit exchange.

5.3	The number of observations specified above will provide a general indication of results on quality of service in certain broad categories. Administrations may desire more accurate results especially for the individual categories in Table 1/E.422.

Attention is drawn to Table 1/E.421 which gives the number of observations required to obtain a certain degree of accuracy.

TABLE 1/E.421

Expected percentage rate

Number of observations of a random sample required to predict with 95% confidence the true percentage of failure with an accuracy of:

of failure

25%

30%

35%

40%

45%

50%

2

3136

2178

1600

1225

1030

880

4

1536

1067

784

600

500

440

6

1003

696

512

392

330

290

8

736

511

376

288

245

215

10

576

400

294

225

195

170

12

469

326

239

183

150

132

14

393

273

201

154

128

112

16

336

233

171

131

112

98

18

292

202

149

114

95

80

20

256

178

131

100

85

70

30

149

104

76

60

50

42

40

96

67

50

38

30

24

50

64

44

33

25

20

16

Annex to Table 1/E.421

Examples of use of Table 1/E.421

Example 1 - It is estimated from previous results that a particular type of failure occurs on about 4% of calls. If it is required to confirm, with 95% confidence, that the existing failure rate is between 3% and 5% (i.e. 25% of 4%), then observations must be made on a random sample of 1536 calls.

Example 2 - For an expected failure rate of 2%, observations must be made on a random sample of about 1200 calls (1225 in the table) to predict, with 95% confidence, that the true percentage is between 1.2% and 2.8% (i.e. 40% of 2%). This means that when 200 observations are taken over a period it is necessary to take the "rolling average" of conditions over six periods. The rate of failure for a number of categories important from the maintenance point of view is expected to be about 2%.

Example 3 - After observations have been taken and the rate of failure in the sample has been calculated, the table may be used in a "backward" direction to give a rough indication of the accuracy of the result.

Suppose that out of a sample of 1000 observations, there were 29 failures due to cause "X" and 15 failures due to cause "Y". The rates of failure in the sample due to X and Y, respectively, are then 2.9% and 1.5%. From the table, it is apparent from this sample of 1000 calls that the true rate of failure due to X has an accuracy of about 35% (i.e. is between 1.9% and 3.9%), and that due to Y has an accuracy of about 50% (i.e. is between 0.8% and 2.3%).

6	Exchange and analysis of the results of observations

6.1	Exchange of the results of observations

The following periodicities are proposed for the exchange of results between Administrations:

Table 1/E.422 or Table 2/E.422 - a monthly exchange is desirable;

Table 1/E.423 or Table 2/E.423 - a quarterly exchange is desirable.

Nevertheless, in the case of small groups of circuits (less than 20 circuits) the information should be exchanged after 200 observations have

been made but never later than one year in any case; attention is drawn to the fact that less than 200 observations are of little value.

Results of observations will be reported without delay:

-	to the Administrations and the network analysis point of the country where observations are carried out;

-	to the Administrations and the network analysis point of the other country (including transit Administrations and their network analysis point when involved).

The benefits to be derived from service observations tend to decrease if there is any increase in the time taken to make the results available to those who can take action to bring about an improvement. The results of service observations according to Tables 1/E.422 and 1/E.423 should therefore be made available to the Administration in the countries of destination as soon as possible after completion of the observation period and in any case within six weeks.

6.2	Analysis of observation results

An analysis of the results should be carried out in the country of origin as well as in the country of destination.

Some Administrations have found it useful to distribute to other Administrations concerned, service observation statistics in the form of graphs.