The development of tau-PET allows paired helical filament tau pathology to be visualized in vivo. Increased knowledge about conditions affecting the rate of tau accumulation could guide the development of therapies halting the progression of Alzheimer’s disease. However, the factors modifying the rate of tau accumulation over time in Alzheimer’s disease are still largely unknown. Large-scale longitudinal cohort studies, adjusting for baseline tau load, are needed to establish such risk factors. In the present longitudinal study, 419 participants from four cohorts in the USA (Avid 05e, n = 157; Expedition-3, n = 82; ADNI, n = 123) and Sweden (BioFINDER, n = 57) were scanned repeatedly with tau-PET. The study participants were cognitively unimpaired (n = 153), or patients with mild cognitive impairment (n = 139) or Alzheimer’s disease dementia (n = 127). Participants underwent two to four tau-PET (18F-flortaucipir) scans with a mean (± standard deviation) of 537 (±163) days between the first and last scan. The change in tau-PET signal was estimated in temporal meta- and neocortical regions of interest. Subject specific tau-PET slopes were predicted simultaneously by age, sex, amyloid status (determined by amyloid-β PET), APOE ε4 genotype, study cohort, diagnosis and baseline tau load. We found that accelerated increase in tau-PET signal was observed in amyloid-β-positive mild cognitive impairment (3.0 ± 5.3%) and Alzheimer’s disease dementia (2.9 ± 5.7%), respectively, when compared to either amyloid-β-negative cognitively unimpaired (0.4 ± 2.7%), amyloid-β-negative mild cognitive impairment (-0.4 ± 2.3%) or amyloid-β-positive cognitively unimpaired (1.2 ± 2.8%). Tau-PET uptake was accelerated in females (temporal region of interest: t = 2.86, P = 0.005; neocortical region of interest: t = 2.90, P = 0.004), younger individuals (temporal region of interest: t = -2.49, P = 0.013), and individuals with higher baseline tau-PET signal (temporal region of interest: t = 3.83, P < 0.001; neocortical region of interest: t = 5.01, P < 0.001). Tau-PET slopes decreased with age in amyloid-β-positive subjects, but were stable by age in amyloid-β-negative subjects (age × amyloid-β status interaction: t = -2.39, P = 0.018). There were no effects of study cohort or APOE ε4 positivity. In a similar analysis on longitudinal amyloid-β-PET (in ADNI subjects only, n = 639), we found significant associations between the rate of amyloid-β accumulation and APOE ε4 positivity, older age and baseline amyloid-β positivity, but no effect of sex. In conclusion, in this longitudinal PET study comprising four cohorts, we found that the tau accumulation rate is greater in females and younger amyloid-β-positive individuals, while amyloid-β accumulation is greater in APOE ε4 carriers and older individuals. These findings are important considerations for the design of clinical trials, and might improve our understanding of factors associated with faster tau aggregation and spread.