Posts Tagged ‘SimplePie’

I’ve spent yet more time fine tuning my miniature painting news aggregator. Time I probably could have used more productively. I tend to put a lot of time and effort into ‘side projects’. Side projects that don’t earn income or help solve any of my problems.

The first week or so was spent wrestling with PHP, particularly with SimplePie and other related code. I still have an outstanding issue with that codebase that no one seems to want to acknowledge or suggest any solution. I definitely did not want to become an expert in PHP, RSS, and Atom feeds.

I also spent way too much time gathering more RSS and Atom feeds. I originally just used the blogs I subscribed to, but FtW Ron had visions of 100s if not a thousand blog feeds being aggregated and merged together. It was while building that, that I discovered the issue with merging multiple blogspot feeds. I don’t know why people particularly in the miniature painting hobby have stuck with blogspot and continue to create more Blogger blogs when so many other free options exist.

The hobby is much more commercial than it was 15 years ago, back then everyone was a hobbyist except a few people who worked for one of a few miniature companies. Now there are many, many mini manufactures, many online stores, professional miniature painters, professional miniature sculptures, and numerous semi-pro hobbyists who supplement their income through their website and the hobby. I’ve always avoided going down that route, but I don’t really feel qualified to give career advice, given the mess I’ve made of mine. I’ve tried to keep the miniature painting news aggregator non-commercial, but maybe I’ll finally breakdown and put an ad on one of my ‘side projects’. I’ve also chosen to focus on miniature painting rather than a specific manufacture or game system, just like I tried to do with this blog. Focus can be difficult to maintain.

The miniature painting news aggregator looks and operates a lot like AllTop. That isn’t a coincidence. AllTop is one of the more successful news aggregation sites. My aggregator is much more specialized, even personalized. It is random, deterministic, algorithmic, chronological, and social. It sources information from Internet darlings like Twitter and Pinterest. You can pin, tweet, or like the featured story/photo. Information is pulled from major hobby sites like Tabletop Gaming News, Beasts of War, and Cool Mini Or Not. News is also pulled from an eclectic collection of smaller hobby blogs and websites. Information is also found by search agents, such as the YouTube painting tutorials, but other less mainstream search engines are used like Topsy. I even added Reddit’s miniature painting section.

I’ll try to continue to ensure it runs OK, I don’t think it’ll ever be fast unless I reduce the amount of content or do something else drastic. I’ll try out new feeds and I hope sites like Chest of Colors fixes one of their feeds, but mainly it should just be there for bored hobbyists who want to see cool stuff. Checking more than hourly will be futile, but hopefully it can become a useful tool for finding cool miniature painting news online.

Although I clearly should be studying or doing something else more productive, I spent a lot of the last week building a miniature painting news aggregator. I didn’t want to build a miniature painting news aggregator. When I want to look at painted miniatures I have a few sites I go to. I subscribe to quite a few blog feeds. I frequent the odd forum. But one site I used to waste time on was From the Warp, particularly the From the Warp sidebars which contained the From the Warp Blogger Group.¬†Miniature Painting News

It was a good way to waste time. It was also a good way to find other blogs to follow. There are other blogging networks targeting gamers or miniature painters, but I thought this one was the best. It certainly was the best curated. It was the one I chose to join when I split my miniature painting exploits off from Muskblog. With no progress seeming to be happening I inquired with Ron what was up. I did some research. I looked at Yahoo Pipes, but the tool I thought could do the job was SimplePie.

I’d previously suggested to Ron to move From the Warp to WordPress even WordPress.com as Blogger/blogspot seems to be limiting what he wants to do with his blog. I still don’t know when the From the Warp Blog Network will be up and functioning. I tried to help. In the mean time I think I’ll pay more attention to the TGN Blog Network and I decided to see what I could build with SimplePie using the feeds I personally follow plus a few more.

Not all feeds are created equal. There are technical problems even bugs in SimplePie. Ideally they’ll get fixed and new release will solve all my problems. In the mean time I decided to let other people poke around and see if they have any suggestions, maybe they are PHP wizards.

As I knew someone would ask, I took a screen shot to try and explain how it works. Most everything is pulled from RSS feeds. Flickr, TGN, even Dakka have working RSS feeds that SimplePie can work with. The rest of the information is pulled from two text files containing a list of feeds. If the From the Warp Network comes back online I’ll try to work it in, but given all the trouble I had with feeds from blogspot, I have my doubts that software will handle 1000 plus feeds effectively and gracefully.

The data in the red boxes to the left comes from one text file. The data on the lower right comes from another text file. The text file that controls the data on the left is going to be smaller and much more exclusive. There is no policy. I didn’t want to create yet another blog network, I just wanted a place to go to look at cool stuff. The most recent post appears at the top and one random feed is featured in the bottom box. I create a thumbnail from the first image in the post. This is perhaps the most intensive and problematic feature of my news aggregator. I’d prefer people just uploaded their finished miniatures to Flickr or a distant second Dakka Dakka. Cool Mini Or Not has no feeds for their gallery and has become more and more of an online store, although there are a lot of nice minis their site doesn’t appeal that much to me. I’m interested in knowing how something was done. Ditto Bell of Lost Souls which has too many writers and just a way too busy site design now. I just want to look at the minis not fight with their web design or told to go fuck myself.

If people want to know how to get their blog included, I suggest they join the TGN Blog Network. If they want to know how they get their website added to one of the two text files I created I suggest they create awesome content. If you consistently create awesome content I or someone else will notice and then I may add your feed assuming you have one that functions. The text file that controls the content on the left I intend to be more exclusive, difficult to get on to, easy to get taken off of. The other text file I’m more willing to just throw a website on, but I’m big on quality content. I can handle the odd picture of your new baby, but if you go on too many tirades or post too much stuff that should end up Facebook or more likely in a private message I’ll be unsubscribing from your feed, removing you from my sidebar and definitely deleting you from the text file that controls the miniature painting news aggregator.

I hope the news aggregator is useful to someone other than me, it took a long time to make. I also hope it isn’t too expensive to maintain so I can keep it lean and mean. Some people would like it to run faster and there are some optimizations that may be possible, but cutting the content in half may make it run twice as fast but is that really desirable?¬†Finding the right balance and the coolest content is the key. If you know of an RSS (or Atom) feed I should include let me know via email or in a comment. Remember I’m a quality over quantity guy and I want painted miniatures or painting advice primarily not your opinion of why Swooping Hawks suck.