T9323 /disc/ How does link grabbing work?

Thread 9323

Age: 3108.66d Health: 92% Posters: 4 Posts: 12 Replies: 10 Files: 0

Anonymous 18jun2016(sa)10:16 No.34555 OP P1

How does link grabbing work?
Supposedly swfchan scans the threads for urls ending in .swf and that is how some of the flashes end up here without any wiki records.

What are the limitations on that?

!///SWFAnts #ADMIN# 18jun2016(sa)10:55 No.34556 SWF ~~P2R1~~

Most of the flashes without a wiki record are here because I've written custom scanners for many sites. They crawl their purposed site, finds flashes, puzzles together file names from tags/descriptions and downloads them. These are individual modules to the core of swfchan and there aren't really any limitations since each one is custom built.

Flashes found from a thread on 4chan should always have a thread on its wiki (the one where the link was found). However sometimes the system bugs and the connection between the linked flash and the thread isn't made. In rare cases flashes found on 4chan itself has its thread dropped as well (due to some bug). I tried to improve things relatively recently so the buggy behaviour shouldn't happen as often now.

Some flashes from 4chan ends up with no thread if the server was down when they were posted and I manually add them to the archive later. Flashes from 7chan/420chan are always added manually nowadays every once in a while so they never get a thread.

Anonymous 18jun2016(sa)11:05 No.34558 OP ~~P3R2~~

Hmm, well I'm monitoring a thread right now with a link in it and the flash hasn't appeared to be downloaded yet.

I'm assuming that the link scanner waits until the thread is closed to download all of the flashes it found in the links besides the OP flash?

!///SWFAnts #ADMIN# 18jun2016(sa)11:11 No.34560 SWF ~~P4R3~~

>>34558
Could be a bandwidth issue, sadly sometimes the server gets too poor connection to wherever to actually download the flash in a timely fashion.

I'm assuming this is the thread? http://boards.4chan.org/f/thread/3100445/a-gem-am ongst-the-rough
And this is the flash? http://locker.phinugamma.org/magick.swf
Looks like it has discovered the second link but havsn't been able to download the second flash: http://swfchan.net/34/U0S08ST.shtml

Anonymous 18jun2016(sa)11:14 No.34561 OP ~~P5R4~~

>>34556
Oh also, is there a list of sites that swfchan has crawled? I might want to either visit them myself or suggest some that have not been visited (if you want the functionality).

Anonymous 18jun2016(sa)11:18 No.34562 OP P6

>>34560
Ah, there isn't an auto-updater here, but that's okay.

Yes that's the thread.

If the server that downloads the flash from that particular link, the flash it grabs will depend upon the IP.

!///SWFAnts #ADMIN# 18jun2016(sa)12:04 No.34563 SWF ~~P7R5~~

>>34561
There's no list because I reckon some site owners wouldn't appreciate the fact that they have been scanned. I'm always open for suggestions of sites though, if you have URLs to tip about. Maybe they'll get scanned, maybe not, maybe they've already been scanned. I've got a list of my own already but I'd be glad to add new URLs to it.

Anonymous 18jun2016(sa)21:20 No.34572 OP ~~P8R6~~

>>34563
That's a good thing to keep in mind, since you might also give them traffic that they didn't want.

Most of the sites I'm suggesting aren't around anymore or have changed to not a flash site. As a result they are only avalible in the wayback machine.

!///SWFAnts #ADMIN# 19jun2016(su)10:22 No.34596 SWF ~~P9R7~~

Thanks! We're lucky to have the Wayback Machine, I hope it stays up forever.

I've tweaked the old code for the link grabber so it should be better to discover and download secondary flashes in threads from now on.

Anonymous 23sep2016(fr)06:24 No.39298 B ~~P10R8~~

Hey I found two sites that have flashes, don't know if they're already taken care of though.

http://web.archive.org/web/20071016083427/http:// wtfux.org/flash/

https://lulz.net/flash/wakaba.html

Anonymous 29jul2017(sa)17:17 No.51901 C ~~P11R9~~

I guess this thread will work.
http://swfchan.net/38/XJYNLL6.shtml
Hey SWFAnts, I made this thread but in the post where I link to a bunch of flashes didn't grab them at all. Is there a reason for this?

!///SWFAnts #ADMIN# 29jul2017(sa)21:03 No.51904 SWF ~~P12R10~~

>>51901
Server must have timed out either when connecting or downloading the files. Tried it now from a VPN and it couldn't connect to the host of toolzbydaman.com. There's a guy posting in the thread you linked to that says he couldn't access the website as well.

There's a number of reasons why the server can't connect to the website while you could. For example the bandwidth quota could have been exceeded or the host might only accept connections from a certain country. Perhaps it has some kind of anti-hotlink implementation, reading cookies or something.

The most reliable way to add stuff to the archive is to upload it directly to /show/, it's slow but then you can at least rename the flashes before you upload them from undescriptive numbers to something more useful, like "toolzbydaman 01.swf".