tailing a file in remote server

imgmoney

Hi All,

I am doing a real-time sync of a log file to a remote server. And I am tailing the same file on a remote server, but I cannot see the new changes happening to the file.

What I am missing here?

luckypenguin

How are you syncing it is the question, not all implementations will fwrite to your new file on a remote host. You are basically inventing rsyslog but in some weird way.

bulbasaur

Are they on a NFS mount?
What does a strace tail -f ... show?

imgmoney

@luckypenguin said: How are you syncing it is the question, not all implementations will fwrite to your new file on a remote host. You are basically inventing rsyslog but in some weird way.

I tried using lsycnd to do real-time sync of that file. And I can see that file is being synced in real-time.

But the issue is, tail -f does not show the new content which is added to that file.

@stevewatson301 said: Are they on a NFS mount?

What does a strace tail -f ... show?

Nope, they are copied via lsyncd.

newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=2369802, ...}, AT_EMPTY_PATH) = 0 fstatfs(3, {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=9899165, f_bfree=9143454, f_bavail=8733432, f_files=2427136, f_ffree=2369764, f_fsid={val=[0xdc17db02, 0x375e1b7c]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0 newfstatat(1, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0 newfstatat(AT_FDCWD, "/home/logs/nginx.log", {st_mode=S_IFREG|0644, st_size=2369802, ...}, AT_SYMLINK_NOFOLLOW) = 0 inotify_init() = 4 inotify_add_watch(4, "/home/logs/nginx.log", IN_MODIFY) = 1 newfstatat(AT_FDCWD, "/home/logs/nginx.log", {st_mode=S_IFREG|0644, st_size=2369802, ...}, 0) = 0 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=2369802, ...}, AT_EMPTY_PATH) = 0 pselect6(5, [4], NULL, NULL, NULL, NULL

This is shown at the end when I do strace.

Also, I find that, when I do rsyncd --append, I can see the changes in tail -f.

MeAtExampleDotCom

@imgmoney said: What I am missing here?

By default, lsyncd uses rsync for the actual updates. rsync creates a new file, unlinks² the original, and renames the new file to the original name¹.

So tail -f <filename> will keep monitoring the old file (now unlinked in the filesystem but kept alive by tail holding it open) not the updated one becasue as far as tail is concerned the updated file is entirely new - tail -f monitors the file by its file descriptor, not its name.

What you probably want here is tail -F <filename>³ or tail --follow=<filename>⁴ which tries to track the file by name instead.

---

[1] Slightly simplification here, rsync has numerous options that tweak this behaviour, but I don't think in ways that will affect the issue you are seeing.

[2] The difference between unlink and delete is significant. The file isn't actually deleted until all hard-links to it are unlinked and all file descriptors referencing it are closed.

[3] note the capitalisation

[4] -F filename is shorthand for --follow=filename --retry which is probably what you want, use the longer form if you don't want --retry to be in effect also or, indeed, just for clarity⁵.

[5] I try to avoid shorthand in scripts, using longer forms helps others understand my intent when they later have to maintain my mess.

MeAtExampleDotCom

@imgmoney said:
Also, I find that, when I do rsyncd --append, I can see the changes in tail -f.

Take note of the warnings about --append in rsync's documentation: https://download.samba.org/pub/rsync/rsync.1#opt--append

If you are happy the warnings don't apply to any of the other files you are syncing then it'll do the trick, otherwise try to resolve the issue at the tailing side instead as mentioned above.

imgmoney

Instead of fixing them in tail end as I used tail to debug why the other app is not able to read the log.

I want to fix it on the lsyncd end. Is it possible for the lsyncd to append to the existing file instead of overwriting it?

0xbkt

By default, rsync writes out changes to a .tmp file and then moves it onto the target for atomicity. The --inplace option applies changes to the target file directly, which avoids those create-on-write kind of overheads.

Are you looking for this?