[Yum-devel] [UG] proxy support

[Michael Stenner]

OK, everybody throw a big thanks Jeremy's way. I set up a test squid
server and started playing around with it. I made up some simple unit
tests. This led to the finding and fixing of two fairly serious
bugs. It turns out all those people bitching that proxies often broke
were probably not on crack.

1) there was a bug in the proxy caching that made it so the first
time the application (ie yum) tried to use a given set of
proxies, they would not really get used. You might never notice
(if the proxy is not required) or you might catch it on a retry.
This one is our fault (Ryan's actually, but I won't point fingers)

2) urllib2 under python 2.2 is a bit broken, and the upshot is that
if you tried to do ftp or non-keepalive http through a proxy, it
just never worked. It always tried to connect normally. I've
now worked around this in urlgrabber.

These are significant bugfixes, and so I'll probably push a new
version in the next few days. Ryan, perhaps you could take a look at
that zero-range issue related to Seth's request for modification-time
based stuff. I noticed poking around that I didn't actually make FTP
support check_timestamp mode for reget, but it falls back to
reget=None mode.

-Michael

[seth vidal]

Post by Michael Stenner
2) urllib2 under python 2.2 is a bit broken, and the upshot is that
if you tried to do ftp or non-keepalive http through a proxy, it
just never worked. It always tried to connect normally. I've
now worked around this in urlgrabber.
These are significant bugfixes, and so I'll probably push a new
version in the next few days. Ryan, perhaps you could take a look at
that zero-range issue related to Seth's request for modification-time
based stuff. I noticed poking around that I didn't actually make FTP
support check_timestamp mode for reget, but it falls back to
reget=None mode.

Sounds good and does explain a few odd proxy-related issues.
proxy heisenbugs are deep weird.

Any idea when y'all are thinking about heading into a beta release?

-sv

[Michael Stenner]

Post by seth vidal

Post by Michael Stenner
These are significant bugfixes, and so I'll probably push a new
version in the next few days. Ryan, perhaps you could take a look at
that zero-range issue related to Seth's request for modification-time
based stuff. I noticed poking around that I didn't actually make FTP
support check_timestamp mode for reget, but it falls back to
reget=None mode.

Sounds good and does explain a few odd proxy-related issues.
proxy heisenbugs are deep weird.

Yeah, that's why I'm so damned happy to have a test proxy set up
(thanks again Jeremy!). Also, (and this is mostly for Ryan), I've now
included instructions in the urlgrabber cvs repo for setting up vsftp
and squid test servers. The unittests just skip those if the servers
aren't available, though.

Post by seth vidal
Any idea when y'all are thinking about heading into a beta release?

OK, here's my thinking on that.

alpha -> beta means the API has stabilized
beta -> release means the code is reliable and tested

Frankly, I think urlgrabber is currently pretty reliable and
well-tested. The only this is a lingering API question related to
batch grabbing (for example, with threads, although a select-based
approach may still be appealing). Ryan and I both spent a bunch of
time on this a while back, and it's a tough problem to solve well.

It might be best to work toward a 3.0.0 release WITHOUT batch stuff,
and then do batch stuff in 3.1.x, allowing for a possible API break to
3.2.0 or 4.0.0 (depending on the size of the break). If we go that
route, then I'd be perfectly happy calling it beta right now, and
would expect to call it stable very shortly.

How does that sound?
-Michael

[Ryan Tomayko]

Post by Michael Stenner
Yeah, that's why I'm so damned happy to have a test proxy set up
(thanks again Jeremy!). Also, (and this is mostly for Ryan), I've now
included instructions in the urlgrabber cvs repo for setting up vsftp
and squid test servers. The unittests just skip those if the servers
aren't available, though.

Great. I'll take a look.

Post by Michael Stenner
... The only this is a lingering API question related to
batch grabbing (for example, with threads, although a select-based
approach may still be appealing). Ryan and I both spent a bunch of
time on this a while back, and it's a tough problem to solve well.

I spent quite a bit of time on this and never came close to somewhere I
thought was stable.

Post by Michael Stenner
It might be best to work toward a 3.0.0 release WITHOUT batch stuff,
and then do batch stuff in 3.1.x, allowing for a possible API break to
3.2.0 or 4.0.0 (depending on the size of the break). If we go that
route, then I'd be perfectly happy calling it beta right now, and
would expect to call it stable very shortly.

I think that's a good idea. I know I'm not going to have the time to
figure out all the gaps with batch grabbing and then have it working
properly in any kind of reliable timeframe. Michael, if you or others
can take this up then cool but I just don't have the bandwidth right
now.

Thanks,
Ryan

[Michael Stenner]

Post by Ryan Tomayko

Post by Michael Stenner
It might be best to work toward a 3.0.0 release WITHOUT batch stuff,
and then do batch stuff in 3.1.x, allowing for a possible API break to
3.2.0 or 4.0.0 (depending on the size of the break). If we go that
route, then I'd be perfectly happy calling it beta right now, and
would expect to call it stable very shortly.

I think that's a good idea. I know I'm not going to have the time to
figure out all the gaps with batch grabbing and then have it working
properly in any kind of reliable timeframe.

I'm not overwhelmed with free time these days either, so lets go with
the above plan. The 2.9.x line is hereby beta, with 3.0.0 to be the
next stable release. It will not include batch grabbing.

Post by Ryan Tomayko
Michael, if you or others can take this up then cool but I just
don't have the bandwidth right now.

Perhaps you and I can chat some time, or you can send me what you've
done. I am generally interested in the problem. It's just that it's
not a simple one :) If there's anybody else that wants to take a
crack at this, let me know. It's a cool problem, but it's not easy,
and is not a good "my first program" experience :) Basically, serious
offers only, please...

-Michael

[Jeremy Lyon]

Post by Michael Stenner
Yeah, that's why I'm so damned happy to have a test proxy set up
(thanks again Jeremy!).

I'm glad I could help. I've managed a squid environment for a 60,000 user
base for the past 5 years. I can probably help with any proxy related
issues you run into.

Jeremy Lyon
Systems Mgmt Integ Prof-Adv
IBM Global Services - Qwest
720.540.1083
***@us.ibm.com