next »

[inp o҉rtb]

This script is now hosted on Launchpad: https://launchpad.net/py-htfilter

I wrote a Python script for fun, reminiscent of Jonas Wagner's HTTP Ripper. Download htfilter here. Download htfilter2 here.

htfilter is a lightweight HTTP proxy server that can monitor, filter, and cache HTTP requests and responses. It has no GUI, but it's extensible using custom Python code. You can hook into the intercepted HTTP conversations and modify them as appropriate. This makes htfilter a useful debugging tool.

This script is designed to be a protocol testing tool. If you're looking to mess with other people, this is not for you.

Included are some sample hooks:

• RewriteYouTube - detects YouTube videos, redirects to their MP4 versions, and archives them with meaningful filenames
• RewritePandora - detects Pandora music, archives them with meaningful filenames, and writes tags
• RewriteExample - blocks Microsoft websites, JPEG images, and cookies

To run htfilter, you need:

• python - I have 2.5.2, but other versions may work
• python-twisted - a Python networking library
• python-mutagen - a Python audio tagging library used by the RewritePandora hook

Just download, unpack, and run the htfilter Python script. Configuration can be adjusted in config.py; by default, the server runs on port 8080 and the YouTube and Pandora hooks are loaded. Data are cached in cache1 and archived in cache2. To create your own hooks, refer to the examples in server/rewriter. By default, the RewriteYouTube and RewritePandora hooks are active.

What I need:

• help testing the script
• help packaging the script for Windows users who may not have all the dependencies

Enjoy!

[Mop (Gb)]

This = epic.

Mac instructions: You need to install mutagen.

Code:

wget -O ~/Desktop/mutagen.tar.gz http://www.sacredchao.net/~piman/software/mutagen-1.14.tar.gz
cd ~/Desktop/
tar -xzf ~/Desktop/mutagen.tar.gz
cd mutagen-1.14
python setup.py build
sudo python setup.py install

(It will ask for your password on the last step, because it has to access system directories to install it in the right spot)

That will install mutagen. Twisted, is already install. (I did this on Leopard, older versions, I don't know.)

So then you can start it normally.

One more note for mac users, this will NOT work if you use it as a global proxy. It will work if you use it as a proxy in Firefox's proxy settings. So then it will work (In FireFox) Fixed, works in the global setting or any program's specific setting.

[inp o҉rtb]

Hey Mop, thanks for testing! The YouTube hook has been optimized and fixed to ignore global proxy settings, so the latest package should work just fine.

As a note, on Ubuntu, you can do the following to install the dependencies:
sudo apt-get install python python-twisted python-mutagen

[Mop (Gb)]

Hey Mop, thanks for testing! The YouTube hook has been optimized and fixed to ignore global proxy settings, so the latest package should work just fine.

Awesome, I'll edit my post.

[inp o҉rtb]

If anyone has innovative uses for the script or ideas on a new hook, I'd love to hear them

[Ðaz]

If anyone has innovative uses for the script or ideas on a new hook, I'd love to hear them

Replacing every image with goatse? O.o I only suggest that because at some hacker convention someone did that with airpwn I think. But, you said it wasn't for messing with people so feel free to disregard

[inp o҉rtb]

hmm... that's doable, but the people you pwn have to expressly route their traffic through your proxy; so you'd have limited success except in an environment where you run an intercepting proxy server. Also, I've made no attempts to disguise the server; all the error messages emitted to the browser contain tracebacks and can potentially be used to identify the server.I've committed another revision to SVN. The new version should be more streamlined; it makes a PID file, so you can track it down more easily; the example hooks have been reconfigured to move instead of copy to save space. There's a mysterious memory leak that I'm hunting, however. You can download the new version at the same address.
Actually... there was no memory leak to begin with. Memory usage increases at first, but levels out and fluctuates slightly. It's been running for about 4 hours on my machine so far on just under 7MB.

[antimatter15]

rofl at "blocks microsoft sites"

And... Mmmm python. I wonder if it'll run on my python enabled iphone

[inp o҉rtb]

Mop tried that; he says the iP* does not have Twisted, so you gotta install that manually.So I had some time to think about application design, and htfilter2 is currently in the pipeline. In fact, I'm about to test my first POST operation through it.

What's new?

• improved object-oriented, event-based hook API (warning: not backwards-compatible)
• no longer depends on Twisted, so should be much more portable (read: iP*)
• still single-threaded and extremely performant
• now supports HTTP/1.1, but still no keepalive connections
• it appears to be leaking memory, but I'm working on it
• CPU usage is a little bit higher at ~2.5%, but I'm working on shaving it down

I'm running a benchmark at the moment. I'll be fixing some bugs and finalizing the API before releasing this. I guess I should also port my old plugins.Yep, POST/1.1 works. There's no noticeable performance difference between proxied and unproxied transfers for normal browsing, so that's good.

Incidentally, the iPhone might not work with this script because proxy servers need to run in the background. On Android, however... I've successfully used SSH tunnels so this should be easy when Python becomes available.

[antimatter15]

does it run on py3k?

[inp o҉rtb]

I haven't tested. But I write portable code

[antimatter15]

OH CRAP! I just realized why they call it portable code! I thought it was something like putting it on an iphone, which seemed strange, but no! it's the ease to port things (right?)

[inp o҉rtb]

Well, there are different meanings of 'portable' depending on context. But yeah, I meant exactly what you said

And dang, there's a pretty serious memory leak; running queries through it all night ran ram usage from 4mb to 17mb :XThat leak was taken care of... as it turned out, circular references were keeping garbage from being collected.

I implemented a proxy-chaining mechanism through which htfilter could be configured to use yet another proxy for outgoing requests. In config.py, look for the tuple labeled 'chain'. Simply remove this line to disable chaining.