<i>Data Survey</i>: The HILDA Survey: Progress and Future Developments

Abstract

As described in previous data survey articles in this journal (Wooden, Freidin and Watson 2002; Watson and Wooden 2004), the Household, Income and Labour Dynamics in Australia (HILDA) Survey is Australia's first and only large-scale, nationally representative household panel survey. Like all longitudinal surveys, the main purpose of the HILDA Survey is to identify changes in the behaviour of the sample units being observed (in this case, residents of a representative sample of private households) and, where possible, to quantify the magnitude of those changes. This requires the repetitive collection over time, from the same sample members, of like, if not identical, measures of characteristics and behaviour. It might also be expected that the data collection procedures used should be replicated at each survey wave. Nevertheless, the process of survey administration and data collection is also affected by behaviours and changes in the real world. Individuals are not always cooperative and hence all panel surveys are confronted by sample attrition. At the other end of the spectrum, populations change because of births and immigration, raising questions about sample representativeness and thus the desirability of incorporating new sample members each survey wave to reflect these changes in the population. Very differently, changes in technology can have implications for the way surveys are administered. Finally, longitudinal surveys, especially those concerned with the measurement of human behaviour, need to be responsive to changes in both policy settings and the wider society, suggesting the need for survey content to gradually evolve over time. A good panel study can thus not remain static; it must evolve to reflect broader changes in society. With 9 years of data collection behind it, the HILDA Survey is now a very different creature from the one originally conceived and implemented in 2001. The aim of this article is to provide an update on progress with the survey, including the evolution of the sample, changes in the way the survey is being administered, and the changing nature of the survey content. All longitudinal surveys have to confront the problem that with each successive survey wave, some sample members are lost, either because of a failure to locate sample members that have moved, or because sample members withdraw their cooperation. This can be problematic, both because of the risk that sample sizes will eventually become unacceptably small, and because sample loss is often non-random, leading to concerns about the representativeness of the sample. Additionally, survey designers have to decide ‘whether and how new entrants to the study population (births) should be included’ (Lynn 2009, p. 11). In the case of the HILDA Survey, all members of responding households in wave 1 (the interviewing for which was conducted over the period August to December 2001), regardless of age, as well as any subsequent children (both biological and adopted), form the sample pursued each year. In addition, any person who at a later survey wave is co-residing with an original (or permanent) sample member is also added to the sample. These persons, however, remain in the sample only for as long as they are co-resident with an original sample member. There are, however, two exceptions to this rule. First, persons who have a child with a permanent sample member are converted to permanent status. Second, from wave 9, any sample members who are identified as having been born overseas and arrived in Australia after 2001 (the year wave 1 of the HILDA Survey was administered) are being converted to permanent status. Details about the evolution of the responding sample over the first nine waves are provided in Table 1. This table shows that of the 13 969 persons originally interviewed in wave 1, 9245 (or 66 per cent) were re-interviewed in wave 9. This total includes persons known to have died or moved out of scope (that is, persons who have moved overseas long term). These number 1037 and, if excluded, would give a nine-wave sample retention rate of 71.4 per cent. Perhaps the most important feature of the sample revealed by Table 1 is the relatively large proportion of the responding sample at later waves that did not participate in wave 1. Just over 30 per cent of wave 9 respondents were not interviewed in wave 1. These respondents are divided fairly equally between original sample members (either children turning 15 years and thus becoming eligible for interview, or adult members of partially cooperating wave 1 households who were not interviewed in wave 1) and new sample members (most of whom are temporary sample members). It can also be seen that, with this design, it is not inevitable that overall sample size will decline given attrition. Indeed, sample growth as a result of changing household composition has, since wave 4, more than offset the loss coming from attrition. Further information about response rates are provided in Table 2. This table provides the annual wave-on-wave response for waves 2 to 9. These response rates are intended to provide an indication of how successful the HILDA Survey has been at obtaining high rates of response, and thus exclude from the denominator persons who have died or moved out of scope (including temporary sample members who move out of the sample households). The table distinguishes between previous wave respondents (who account for the large majority of in-scope sample members in any year), previous wave non-respondents, children turning 15 years of age and new sample members. It also reports figures for both all persons and persons with an attachment to a household that responded in the previous wave. As can be seen, the wave-on-wave response rate for previous wave respondents has gradually improved over time, rising from 86.8 per cent in wave 2 to 96.2 per cent in wave 9. Furthermore, and as illustrated in Figure 1, these response rates are very similar to those reported over the first nine waves in other leading household panel studies, such as the British Household Panel Survey (BHPS) and the German Socio-Economic Panel (GSOEP). Wave-on-Wave Attrition Rates, HILDA, BHPS and GSOEP Compareda Notes: (a) HILDA, Household, Income and Labour Dynamics in Australia Survey; BHPS, British Household Panel Survey; GSOEP, German Socio-Economic Panel.(b) Excludes proxies and short telephone interviews. The response rates among other types of respondents are much lower. New sample members have less knowledge of, and attachment to, the survey than previous wave respondents, and hence we would expect both lower response rates and no discernible upward trend over time. The response rates are indeed lower, but do show a marked jump upwards in wave 5. We suspect this reflects the changed incentive arrangements that were introduced in this wave (see Subsection 3.4). Among children turning 15 years of age, and hence becoming eligible for interview for the first time, response rates vary between 70 and 80 per cent. The population here, however, includes many households where the parents have long since discontinued their involvement in the survey. Once we restrict the population to just children of households that participated in the previous wave, the response rate exhibits the same upward trend characteristic of wave 1 respondents, rising from 80.4 per cent in wave 2 to 92.6 per cent in wave 9. Finally, Table 2 reports very low response rates for persons that had not responded in the previous wave. This is hardly surprising, and indeed many longitudinal surveys make no effort to reach non-respondents in later waves. This is not the practice in the HILDA Survey, and as a result every year a sizeable number of individuals are recruited back into the responding sample despite a year or more missed. The following rules used in the HILDA Survey go some way to ensuring the sample remains representative of the Australian population in a cross-sectional sense; however, there is one group that is largely overlooked—immigrants arriving in Australia after the original sample was selected. Watson (2006) estimated that in the 10 years since the sample was selected, approximately 1.5 million immigrants arrived in Australia, representing 7 per cent of the Australian population. Some of these immigrants have joined HILDA Survey households, and, as noted earlier, we have recently started following them on a permanent basis. This group, however, is far from a random sample of recent immigrants. This suggests the need for a strategy for augmenting the sample targeted specifically at recent immigrants. A range of options for such a top-up sample were canvassed (see Watson 2006) but because of difficulties in targeting the sample, especially when the sampling frame is quite dated, it was decided that the top-up sample would not be limited to recent immigrants but would instead be a random sample of people living in non-remote parts of Australia. This will have the added benefit of helping to alleviate, to some degree, the biases from non-random attrition. The top-up will result in the addition of 2000 households to the ongoing sample, commencing with wave 11 (that is, 2011). Consistent with the wave 1 sample design, a random sample of 125 Census Collection Districts (CDs) will be selected across Australia, with a probability proportional to the number of occupied dwellings in each CD based on the 2006 Census. A minimum household response rate of 66 per cent is projected, yielding, on average, 16 dwellings from each CD. Administration of the HILDA Survey in wave 1 involved two main elements: (i) face-to-face interviews (using pencil and paper methods) with all household members aged 15 years and over; and (ii) the distribution of a self-completion questionnaire (SCQ). If maintaining longitudinal consistency in the data were the primary objective, then these same administration methods would be used in every succeeding wave. Cost pressures, together with the perceived advantages offered by new technology, however, have led to significant changes in the mode of survey delivery. In addition, other changes, and notably a change in the fieldwork provider, have resulted from factors outside of our control. For waves 1 to 8, all fieldwork functions as well as many data processing functions (data entry and coding) were subcontracted to the Nielsen Company. In early 2008, Nielsen informed us that it had no intention of re-tendering for the next fieldwork subcontract (apparently because it had made a business decision to move away from face-to-face interviewing), necessitating a change in fieldwork provider. The successful tenderer was Roy Morgan Research (RMR), which has the contract to perform the fieldwork functions for waves 9 to 12. Although such changes during the life of a longitudinal survey must be anticipated and planned for, they do have the potential to significantly disrupt data continuity. Most other long-running comparable household panel studies conducted elsewhere in the world, for example, have been able to avoid making such changes.1 A detailed transition plan was thus developed with both RMR and Nielsen that covered issues such as the strategy for communicating with sample members, knowledge and materials transfer, interviewer recruitment and the scope for RMR involvement in wave 8. It is difficult to distinguish the impact of the change in provider from other changes introduced at the same time, and notably the change in survey mode (see Subsection 3.2), but most available indicators suggest the change in provider has had minimal, if any, detrimental impacts on the survey (although it is perhaps too early to assess whether data quality has suffered). Most obviously, and as we have already seen, most response rates continued to improve in wave 9. We suspect that a major contributing factor to this good outcome is the relatively large number of HILDA experienced staff that moved with the project. One of our main concerns stemmed from the potential change in interviewer workforce. It has, for example, been consistently demonstrated, both in the HILDA Survey and in other longitudinal surveys, that interviewer continuity matters for response (Watson and Wooden 2009a). Actively pursuing the employment of experienced HILDA Survey interviewers was thus made a priority for RMR, something that was facilitated by both the casual nature of the interviewer workforce and the cooperation received from Nielsen. During the wave 8 interviewer training sessions, all interviewers were offered the opportunity to have their contact details passed over to RMR so that they could continue to work on the HILDA project. As a result, 113 face-to-face interviewers (92 per cent) agreed to have their contact details handed over, and of these 89 eventually worked on wave 9. Interviewer attrition was still high, with 34 per cent of all face-to-face interviewers being new interviewers. Nevertheless, this rate was similar to the rate experienced in wave 2. At the same time as we changed fieldwork provider, we also implemented a major change in the way the personal interviews were to be delivered, shifting from pencil and paper methods to computer-assisted personal interviewing (CAPI). The advantages that CAPI offers, both in terms of cost-effectiveness and data quality, are very attractive and thus the only issue for us was not whether to change, but when. Indeed, the shift to CAPI had been under consideration for a number of years, with a split sample trial of CAPI methods undertaken in conjunction with the wave 7 dress rehearsal. The benefits that CAPI offers are well documented and include allowing more complex routing, automated checking of responses for logical and internal consistency, delivery of more timely information for monitoring fieldwork progress, and the elimination of a separate data entry phase (see de Leeuw, Hox and Snijkers 1995). CAPI also better facilitates the feeding forward of information from one survey wave to the next. Again it is too early to assess what effects the introduction of CAPI may have had on the data, but the of the wave 7 dress trial were very The from that trial that, to pencil and CAPI was with response rates that were different and lower rates of (Watson and It was also reported that mode effects were limited to a and were in the (that is, to less desirability the other and CAPI resulted in much interview however, to have been a of the of that a on and As a were in wave 9 by that used a than a It is also that all of the panel studies conducted overseas have had to go a similar change, and have reported of any major data and There has been a in the of personal interviews by the original HILDA Survey sample was (see Freidin and Watson time, the sample gradually the of sample members. In some where sample members move the reach of our interviewer these become and hence we are to to As in the of Table the proportion of interviews undertaken by telephone has from just per cent in wave 1 to more than 10 per cent in wave 8, to per cent in wave 9 (the result of a effort that year to restrict the proportion of interviews undertaken by in survey both over time and across sample units have the potential to consistency of respondents, for example, do not have to the same and that in the face-to-face Very differently, telephone interviews might be less to biases from the of in the household (including the of the We also that telephone interviews are than face-to-face 11 to per cent (see Watson and Wooden of less this might be equally well by less time being on in telephone interviews. such biases have any significant impact on the data is an issue under but are of in responses by mode (Watson and Wooden interviewing is, however, with a much lower proportion of self-completion personal interviews are conducted face-to-face the are handed to with arrangements made for the interviewers to the Indeed, in many interviewers are able to the away with them on the same the interview is conducted per cent of in wave 9 were on the same as the personal With telephone however, the has to be to with to the questionnaire in the As in Table response rates have been more or less since the HILDA Survey Table also suggests that much of this decline is by the over time in the of telephone The response rate among telephone interview respondents only per among face-to-face interview respondents it per the trend in the is still of a data after for interview wave as well as other interview and that telephone interview respondents are with an response rate that is lower than comparable face-to-face (Watson and Wooden The changes of have been in the of incentive offered to sample members for and in the way those are In waves 1 to all households were either or each year they with the only being when interviews were with all in-scope household members. The was made by to households after the fieldwork for that household had been In wave the incentive was changed to per personal interview, with a to households where all in-scope household members the personal As noted earlier, we suspect this change, which the incentive on in households as well as all household members than just the household was for the marked jump in response rates in wave especially among new sample entrants and children turning In wave 9 we made two changes. First, the of the incentive was to per with the household also to Second, and more for interviews conducted were no by well after the interview but in the form of following of the personal we suspect that these changes response rates what they would have This is perhaps not so in the response rate figures reported in Table we to the marked in the response rate among previous wave As already the main advantages of longitudinal data from measurement of the same there a of other content to be example, shifting policy and The HILDA Survey reflects with new content being developed and over relatively short The for new content has been in by more questionnaire and in by added on new have been introduced into the interview questionnaire on training (in wave and in wave on from work (in wave and on in wave is to new content across the different survey and it is this that is the of in this who to about the should Wooden and Watson The HILDA Survey has always been with the of scope in each wave for questions on that will not be covered each year. The for this content is in Table As can be seen, the HILDA Survey now has major that are on a into from wave 11). These household and and 8 and and on to a was for the first time in wave and and is still in the and not but to be for the first time in wave In addition, there have been a number of either because they far questions or because they are targeted at very The most significant of the is the on which is being conducted every years in wave Some of these are also being into the major The on and were recently into the wave 9 on and the on and can expect to be next in wave as of the and We now to each of the major the the is the and hence has already been the of much It to provide of total household by the of or (see and Wooden The scope of the was in wave with the for the first time of questions on The of was also with much more on different types of We also to relatively high of by from respondents to provide a more of the of a of or The and was implemented in to provide Australian data that would provide with the and Survey being implemented in many other The HILDA Survey was seen as an for given it was already information about This thus information about such as recent and whether they were other factors and the of employment following This is also by a of questions about which in wave 8 were the of detailed questions about with parents and who were not to quality of and the it has on and there has always been significant content about in the HILDA Survey, especially in the The however, provides an opportunity to on a wider of the implemented in wave 9 covered the following about difficulties by in wave private in wave of in wave and the of, and of by children in the Like questions on are in the HILDA Survey every year. Indeed, there is a to during the years Nevertheless, it is that that this is one where the HILDA Survey collection could be This was in wave 7 when a of questions about and were As noted we would expect these to be as of this issues that are to be include the collection of more details about as of and the of in the and the measurement of One of the relatively of the HILDA Survey is that, in addition to data by interview, all are also to a self-completion paper This questionnaire of questions that are difficult to in a in a personal interview or that respondents may in a face-to-face Furthermore, the content of this is much less with many in and out of the The types of covered in most waves include such as and quality and and time life and In recent waves a of new content has been facilitated in by the of the of the in wave from 16 to and for the first time in wave and every year the 10 of for the first time in wave 7 and expected to be every 2 first in wave and in wave measures of intended to quantify the of measures of and behaviour, for the first time in wave but to in with the of the in the interview a of for the first time in wave and to be every years in more in and and a household for the time in wave and in wave together with elsewhere in the HILDA Survey, such as provides of to 80 per cent of total household the provided is very for more are thus to the available from the HILDA Survey at We also about the being made of the data by and policy but that a of now already can be in Wooden and Watson Additionally, an of that make of the HILDA Survey can be on our at