Windows and Unix/Linux handle the effect of DST (daylight saving time) on time stamps differently

Most people would probably never notice this, if the time stamps of all files would be affected in the same way, and if both operating systems would use the same rule. Both is not the case.

This causes all sorts of problems, especially with software for backup and file sync, because they often incorrectly see a time difference between identical copies of files. Admins of Samba servers can even simultaneously see two different timestamps for the same file.

Everything is fine as long as you stay within one operating system and the same kind of file system (and if you don't know precisely which file should show which timestamp). Problems arise if you copy between different operating systems and/or different file systems. This happens often if you want to sync files from your digital camera or any other portable USB device, or between Windows and Linux, or with files inside an archive file or a truecrypt volume. If you want to sync again after DST switch, half or all files will appear to have changed, although they haven't.

Many people suggest to let your backup or sync software ignore time differences of 1 hour. This is a bad advice for several reasons. First: after several years if synching, you can easily have different copies of an identical file with different time stamps spanning more than just one hour. Second: imagine what happens if you change a file, run a backup or sync, then make another change to the same file within one hour and then don't change that file again. The last change will never be synced or be backed up. Nobody should accept such an unreliable behaviour of a backup or sync program.

Background

In theory the concepts of Windows and Linux for time stamps look great: Both usually store timestamps in UTC, which includes neither timezone, nor the DST offset. Linux even lets its system clock run in UTC. Only when they display a timestamp, they apply corrections based on timezone and DST. Backup and sync software is not supposed to be affected by differences in the conversion, because they are supposed to use the UTC timestamp.

Thus there shouldn't be any problem. But reality differs from the ideal world of the previous paragraph in three ways. First, many programs don't use UTC, but the time that the operating system displays. Second, time stamps are usuall stored in UTC, but not always. And third there is a subtle difference in the conversion math for UTC that Windows and Linux use.

Let's start with that last item, the conversion math.

Conversions

Linux treats DST different than changes of timezone. It adds the DST offset to a timestamp not based on the current time, but based on the timestamp itself. Linux adds the offset if DST is/was in place at the time that the timestamp of the file specifies. Files created during daylight saving time are always shown one hour ahead, the other files are never shown one hour ahead.

Windows doesn't care if a file was created during standard time or during daylight saving time. It adds the DST offset to the time stamp if the current system time includes the DST offset. Thus in Windows all displayed times change by one hour when DST starts or ends, because Windows treats DST and timezone in the same way.

Microsoft does it this way intentionally:
"The FileTimeToLocalFileTime() function and the LocalFileTimeToFileTime() function perform the conversion between UTC time and local time by using only the current time zone information and the DST information. This conversion occurs regardless of the timestamp that is being converted."
[http://support.microsoft.com/kb/932955/en-us]

An employe of Microsoft gives two reasons for this decision: First the functions to convert between local time and UTC would otherwise not be inverses of each other in the 'limbo hour' during the cutover from standard time to daylight time. This is because the switch to DST skips one hour, and during the switch back one hour of local time exists twice: before and after the switch. Second argument: the DST rules often change and the system knows only todays rules, not all historic rules, so it cannot apply the correct rule for old files.
[http://blogs.msdn.com/b/oldnewthing/archive/2003/10/24/55413.aspx]

I don't buy the first argument. A function to convert from local time to UTC always has this problem with the limbo hour. The only logic decision would have been to don't ever let DST affect the timestamp, or to abandon the daylight saving idea altogether (this would have been my favourite). Linux reduces the problem by running the system clock in UTC, thus reducing the need for a function that converts from local time to UTC. Converting from UTC to local time is unambiguous.

The second argument is valid. It means that Linux shows incorrect timestamps for some files that are older than the last change of the DST rules.

Let's move on and look what happens when both operating systems don't store timestamps as UTC.

Windows uses UTC for timestamps in NTFS file system, in entries in its log files, and on network shares. Linux uses UTC in its own file systems (e.g. ext2), not in logfiles, but also in network file systems. Both operating systems can also access FAT and FAT32 file systems. These file systems store timestamps as local time, not as UTC. NTFS is used by Windows on most internal hard disks. FAT32 is used on most USB drives and other external devices that store files, for example digital cameras, if their capacity is not above 2 Terabytes. FAT was used by DOS. FAT32 is an extension of FAT that allows long filenames.

If you let Windows access files on a FAT32 partition, it displays the timestamp from the file system, without adjustments. If does not add one hour during daylight saving time. Thus if you copy files from FAT32 to NTFS during standard time, and compare them during daylight saving time, you will see one hour time difference between original and copy. Wait half a year and the apparent difference is gone. But if you sync during daylight saving time, by copying the seemingly newer files over the seemingly older ones, then you preserve the new timestamps. Wait half a year, then the displayed time of the copies on NTFS jumps one hour back, but the newer timestamps on FAT32 stay. If you sync again, you will lose the original timestamps.

In Linux the situation is different. Linux adds one hour to the timestamps of all files that were created during daylight saving time, regardless of current date. Thus for approx. half of the files it shows the same time as the digital camera showed when it created the files, the other files are shown one hour later than the time when they were actually taken.

Note another difference: In Windows you have two versions of timestamps. One is UTC, the other is local time. UTC is stored in the file system, local time is UTC plus timezone offset plus DST offset. The same Local time is used for display and in FAT32 file system. Linux differentiates between three versions of each timestamp. One is the timestamp on a FAT32 file system. Another one is UTC, which is derived from local time by subtracting timezone offset, and used if Linux runs as sever, see below. And the third one is the displayed time. The displayed time is derived from the local time from the FAT32 file system by applying only the DST offset, but not the timezone offset.

What about accessing files on a server?

Both Windows and Linux apply their conversions for displaying timestamps also when they map (Windows) or mount (Linux) a remote file system.

Both report timestamps to clients in UTC format when they run as server. If they share files that are stored on FAT32, they convert timestamps to UTC by applying only timezone offset, no DST offset.

Thus the settings of date and DST on the server have no effect on the client. The timezone conversions cancel, if server and client are in the same timezone. The client add the DST offset according to his favourite rules, Windows does this different than Linux (see conversion math above).

Example: Create a file on a Linux server during standard time at 12pm. Wait for daylight saving time and look at that file twice: once from Linux, and also via Samba from Windows. Linux still displays 12pm, Windows displays 1pm. Two different time stamps for the same file.

What happens if you attach a FAT32 disk to a Windows server and access that from Linux? Windows converts the FAT32 timestamps to UTC, Linux converts them back, canceling the effect, and then it adds one hour for those timestamps from which it thinks that DST was in effect. The DST setting on Windows has no effect.

What happens if you mount a FAT32 partition on a Linux server and deliver the files via Samba to a Windows client? Linux assumes that the timestamps of FAT32 are local time, and converts them to UTC. For this is applies only the timezone offset, ignoring both its own DST and the DST of the timestamp. Then it delivers this UTC timestamp to Windows. Windows converts UTC back to local time. If you do this in the same timezone and during standard time, then both timezone-conversions cancel each other. You see the same timestamps as if you had attached the data medium directly to the Windows computer. But if you look during DST, then Windows will add one hour to each timestamp. This is the same what you would get for local files on NTFS, but you wouldn't get this if you had connected the device with FAT32 fipe system directly to the windows computer. Thus when you want to sync between NTFS and FAT32, connect the device with FAT32 to a Linux machine and export the files with Samba to Windows.

If you access files via FTP, then the server adjusts the timestamps. Thus if you run Windows and use FTP to download files from a Linux server, then Linux adds one hour if it thinks that DST was in effect when the file was created. Access the same file via Samba and the hour is not added.

If you use Windows to access files on an old Novell Netware server, Windows handles that server like a FAT32 partition, because Novell Netware sends timestamps in local time, not in UTC. Thus these timestamps don't change if DST starts or ends.

What happens if you use Linux to access a Novell Netware server? I don't know.

Remedy

Many backup programs don't look at the timestamp, they only look at the archive flag. Thus they are not affected by this mess.

Some backup programs for Windows actually do use the UTC timestamp, like they should. Sorry, I don't know which ones do that.

When Windows copies from FAT32 to FAT32, it preserves the timestamp, and these timestamps even don't change when DST starts or ends. So you could just use FAT32 for everything. But Linux will display different timestamps for half of the files. Also you will loose the benefits of NTFS. Ok, the pseudo-security is often only a hazzle, especially on portable disks. But the ability to have files larger than 4 gigabytes can be useful, and the ability to have partitions larger than 2 Terabytes will soon become important.

If you use only NTFS and network shares, Windows preserves the UTC timestamp, but all timestamps change when DST starts or ends.

If you want to stop Windows from change the displayed timestamps in NTFS and network shares, tell it to not automatically correct your clock for DST. Then it will also not add the DST offset to displayed timestamps any more. You must change the clock manually when DST starts and ends, and you must disable automatic time sync with servers. If you switch DST correction off during standard time, then displayed time stamps will from then on not change any more. If you switch it off during DST, then the display of all time stamps will change by one hour when you do the switch, and will then stay that way. But old timestamps created during DST will remain 1 hour off and it will not be more compatible with unix either.

If you have NTFS and FAT32, attach FAT32 to a Linux server with Samba and access it over the network, but don't look at it from inside Linux.

If you want to get rid of all these problems, you may come to the idea to set all computers to GMT and disable DST. Then no corrections are appliced and all timestamps are always consistent. But you cannot use a timesync service anymore, it would set the clock to a wrong value. Also if you travel or transfer files to people in other timezones, the automatic timezone correction can be useful. You lose that as well.

Linux offers the mount option tz=UTC when mounting FAT or FAT32 file systems. But to use that, you should set the clock in your digital camera to UTC and don't change it when DST starts or ends. Then you could always see the correct time when accessing these files from Linux. But probably you already have files on FAT32 and did manually adjust the clock when DST started and ended. In this case the effect described above will remain: files that were created during DST are shown as one hour ahead.

Conclusion: Ask you politician do abandon DST. They clearly had not the slightest idea when they were doing when they introduced this. And if they tell you that DST saves energy, tell them that they are liars, it's just not true.


© 2011 Pascal-Hacker