TTL settings - The TTL settings can be configured for each DNS record in the zone file. TTL determines the interval which servers update their cache. The shorter the TTL interval, the earlier the name server will retrieve the updated DNS records from the zone file.
The ISP - ISPs follow their own practices to cache DNS records. In doing so, the ISP is able to minimize network traffic, which quickens browsing speed. There are ISPs who ignore TTL settings and update their cache following their own intervals.
Registry of Domain Name - As you make changes to your DNS zone file by changing the records, the updates are relayed to the registry of the domain name. The registry will have to update the changes to their root zone. Different registries will follow their own update intervals, which will affect propagation time.