214 reputation
18
bio website
location
age
visits member for 4 years, 9 months
seen Jun 19 at 10:37

Jan
3
comment Does a forward web proxy exist that checks and obeys robots.txt on remote domains?
I agree when you look at it like that, but arguably it's an issue of terminology. So, as different model, define "crawler" to be a system of components - a forward web proxy and numerous agents behind it. The whole thing can be dubbed "a crawler" because that is what it looks like to the outside world. But now we can have multiple, probably heterogeneous requesting agents at the back and a single point at the front responsible for obeying remote policy (i.e. robots.txt). It's interesting!
Jan
3
revised Does a forward web proxy exist that checks and obeys robots.txt on remote domains?
added 8 characters in body
Jan
3
comment Does a forward web proxy exist that checks and obeys robots.txt on remote domains?
A program to continually update a Squid config could be a good solution if nothing already does this - thanks.
Jan
3
comment Does a forward web proxy exist that checks and obeys robots.txt on remote domains?
Thanks for answering! My thought is that the proxy can act as a central, single point of obedience to any robots.txt rules encountered - it means that multiple crawlers can start from one network, go through a proxy, and all, in aggregate obey any robots.txt rules encountered.
Jan
3
awarded  Editor
Jan
3
revised Does a forward web proxy exist that checks and obeys robots.txt on remote domains?
added 146 characters in body
Jan
3
asked Does a forward web proxy exist that checks and obeys robots.txt on remote domains?
Feb
9
comment How can I sort du -h output by size
The relevant section of the manual: gnu.org/software/coreutils/manual/…
Jan
31
accepted On Linux, how can I see how far through a file a process is?
Jan
31
comment On Linux, how can I see how far through a file a process is?
Summarising, they are: 1. Use pv (before you start) 2. Use lsof -o (if offsets are supported) 3. look at value of wchar on /proc/<PID>/io , for the known PID of the cat process
Jan
31
comment On Linux, how can I see how far through a file a process is?
Actually, there are three great separate answers in this question now, so I will leave it as it is.
Jan
31
comment On Linux, how can I see how far through a file a process is?
I only just saw your edit! This works perfectly given the absence of (i) pv in the chain, (ii) lsof -o working. Thanks!
Jan
31
comment On Linux, how can I see how far through a file a process is?
The lsof FAQ, mirrored at gd.tuwien.ac.at/utils/admin-tools/lsof/FAQ, implies in point/question 3.43.2.3 that lsof -o is not possible on Linux! I assume this is over-broad, as other users appear to be using lsof -o
Jan
31
comment On Linux, how can I see how far through a file a process is?
Yes, tried both. I am searching for reasons lsof warns -o as not available now...
Jan
31
comment On Linux, how can I see how far through a file a process is?
I like pv - when I remember to use it! Thanks.
Jan
31
comment On Linux, how can I see how far through a file a process is?
This is the lsof option that I had looked at. But: what if lsof -o is not allowed or unavailable on the system?
Jan
31
comment On Linux, how can I see how far through a file a process is?
Thank you for the answers so far! However, I should have been clearer: the command is already running.
Jan
31
asked On Linux, how can I see how far through a file a process is?
Dec
2
comment How to stop SSH from falling back to password?
Thanks! This is what I was thinking of.
Dec
2
accepted How to stop SSH from falling back to password?